Fortinet FortiGate Site-to-Site IPsec VPN Troubleshooting

Fortinet FortiGate Site-to-Site VPN Troubleshooting

Before diving into the troubleshooting methods for Site-to-Site IPsec VPN on FortiGate devices, it’s important to understand the foundational concepts of IPsec itself.

IPsec (Internet Protocol Security) is a suite of protocols designed to securely authenticate and encrypt IP packets between two peers. The three key protocols within the IPsec suite are:

IKE (Internet Key Exchange): Responsible for negotiating the tunnel, performing mutual authentication, maintaining the connection, and handling disconnection.
ESP (Encapsulation Security Payload): Provides data encryption and ensures integrity of the transmitted packets.
AH (Authentication Header): Offers integrity and authentication of data, but does not provide encryption.

Note: FortiGate firewalls use only ESP to encapsulate and transport IPsec payloads. The AH protocol is not supported or used by FortiGate devices.

Now let’s focus on IKE, the protocol responsible for establishing and maintaining the IPsec tunnel.

IKE operates on UDP port 500, and if NAT Traversal (NAT-T) is enabled, it also uses UDP port 4500.
IKE is used to negotiate Security Associations (SAs) between the peers. These SAs define how data will be encrypted, decrypted, and authenticated.
The IKE process is split into two phases:
- Phase 1: Establishes a secure, authenticated channel using either main mode or aggressive mode.
- Phase 2: Negotiates IPsec SAs to define how traffic will be encrypted across the tunnel. It uses only quick mode.
Each Phase 2 SA is unidirectional, two SAs are needed for bidirectional communication.

Understanding these building blocks is critical for configuring a stable and secure Site-to-Site IPsec VPN on FortiGate, as well as effectively troubleshooting when issues arise.

FortiGate Site-to-Site VPN Topology Comparison

When designing a Site-to-Site IPsec VPN architecture with Fortinet FortiGate devices, selecting the right topology is a critical decision that impacts scalability, performance, and management complexity.

Three common topologies in enterprise networks:

Hub-and-Spoke
Partial Mesh
Full Mesh

Each has its own strengths and trade-offs depending on the size of your network and how the different locations need to communicate.

The table below provides a clear comparison of these topologies, helping you choose the right model for your environment.

Topology	Description	Typical Use Case	Advantages	Disadvantages
Hub-and-Spoke	A central FortiGate (Hub) connects to multiple remote locations (Spokes). Spokes do not talk directly.	Branch offices connecting to headquarters (HQ) via a central point.	– Easy to configure and manage- Centralized control- Simple routing policies	– No direct communication between spokes- Hub is a single point of failure
Partial Mesh	Some sites have direct tunnels between each other, in addition to hub links.	Medium-sized networks with specific inter-branch communication needs	– Reduces hub load- Improved latency between selected sites- More flexibility	– More complex configuration- Needs careful planning- Additional tunnels to manage
Full Mesh	Every site has a direct VPN tunnel to every other site.	Environments where all sites need full communication (e.g., data centers).	– Best performance for any-to-any communication- No dependency on a single node	– Very complex to manage- Poor scalability- High overhead (many tunnels required)

✅ Partial Mesh is a hybrid model that offers flexibility by only connecting the sites that need direct access to each other — ideal when balancing performance and manageability.

IPsec Deployment

When deploying Site-to-Site VPNs, especially between FortiGate and third-party devices, pay close attention to compatibility. Many vendors don’t support wildcard selectors like 0.0.0.0/0, or they require subnet matching with the same prefix length. Because of this, each pair of local and remote subnets may need a separate Security Association (SA) and Phase 2 configuration.

This increases configuration time and complexity on the FortiGate side.

What to Do If the Tunnel Does Not Come Up – Troubleshooting

If the tunnel fails to establish, start by checking for misconfigurations. Most VPN issues are caused by mismatched settings between the two peers.

First, confirm:

Matching IKE versions (v1 vs v2)
Identical encryption and hash algorithms
Same pre-shared keys
Matching local and remote subnets

If everything looks correct but the tunnel still doesn’t come up, use IKE real-time debug to trace the problem.

1. Check Phase 1 Status

diagnose vpn ike gateway list

This shows whether Phase 1 successfully established. If not, check:

IP addresses and interface bindings
Pre-shared key or certificate mismatch
Incorrect IKE version (v1 vs v2)
Port blocking (UDP 500 or 4500)

2. Check Phase 2 Selectors

If Phase 1 is up but traffic doesn’t pass, the issue is likely in Phase 2. Use:

diagnose vpn tunnel list

Look for selector mismatches. The local and remote subnets must match exactly on both sides.

3. Monitor Real-Time Logs

Run this command to view IKE negotiation in real time:

diagnose debug application ike -1
diagnose debug enable

After testing, stop debugging with:

diagnose debug disable

4. Check Encryption and Hash Settings

Ensure that both sides use compatible:

Encryption algorithms (e.g., AES256, AES128)
Authentication (e.g., SHA1, SHA256)
DH group settings (e.g., Group 14, 5)

Even one mismatch can cause Phase 2 failure.

5. Use `sniffer` for Traffic Analysis

Verify that traffic is actually hitting the FortiGate:

diagnose sniffer packet any 'host <remote peer IP>' 4

This helps confirm if traffic arrives at the expected interface and if the response is being sent.

Run IKE Real-Time Troubleshooting

This debug provides a step-by-step view of both Phase 1 and Phase 2 negotiations. It’s one of the most effective tools for identifying tunnel negotiation failures.

Follow these steps:

1. Set a Debug Filter for the Peer IP

Create a filter:

diagnose vpn ike log-filter dst-addr <peer_public_IP>

2. Enable IKE Debugging

diagnose debug application ike -1
diagnose debug enable

Now try to manually bring the tunnel up while the debug is running.

3. Review the Output

Look for failure messages such as NO_PROPOSAL_CHOSEN, AUTH_FAILED, or INVALID_ID_INFORMATION. These often point directly to the root cause.

4. Stop Debugging

diagnose debug disable
diagnose debug reset

⚠️ Important: Do not leave real-time debug running for extended periods — it can impact performance and flood the logs.

Pro Troubleshooting Tips

Always double-check that both ends use the same subnet mask (e.g., /24 vs /16).
If NAT is involved, ensure NAT-T is enabled and both peers use UDP port 4500.
Keep VPN configurations consistent in naming and structure — this helps a lot during troubleshooting.
Enable log settings on the VPN phase 1 interface for better diagnostics in the GUI.

Common IPsec VPN Issues and How to Fix Them

Issue	What You Might See in Debug Logs	Possible Cause	Recommended Action
Tunnel won’t come up	`negotiation failure`	Phase 1 or Phase 2 settings don’t match	Double-check IKE and IPsec proposal settings on both sides
	`no SA proposal chosen`	Proposal mismatch	Verify encryption, authentication, and DH group settings
	FortiGate selects wrong VPN tunnel	Wrong or missing local ID	Use unique local IDs or aggressive mode if you have multiple dial-up tunnels to the same gateway
	`XAUTH failure` or `connection expired`	Invalid username, password, or wrong group	Confirm remote user credentials and group mapping
	`peer has not completed XAUTH exchange`	XAUTH is disabled on the client	Enable XAUTH or use certificate-based auth depending on your setup
Tunnel flaps (goes up and down)	DPD packets not received	Packet loss or unstable WAN	Check ISP connection quality or increase DPD timeout
Tunnel is up, but traffic fails	`no matching IPsec selector, drop`	Phase 2 selectors mismatch NAT or routing misconfigured	Ensure local and remote subnets match exactly Disable NAT if not needed
	Tunnel up, but no traffic passing	Route or policy is missing	Route traffic correctly (IPsec interface or policy-based)