I’m going to give a very short and sweet summary of this article for those in a hurry that happened across this page by way of some quality Google searches.  I had two different groups of users behind one Fortigate.  I needed some of these users to reach a particular remote network across the internet with NAT.  I needed others to reach the same network through a site to site VPN without NAT.  Policy-based routing initially did not seem to work.  Turns out, it was because one of the remote networks being routed to also contained the site to site VPN destination IP.  The solution was a /32 static route for just the remote firewall’s IP, still using the tunnel device (seems weird/wrong), and then a broader policy-based route sending the appropriate traffic over the same tunnel device, with the next hop specified as being the tunnel target.  This let one group’s traffic go internet, and the other’s go VPN.

Here’s the longer version:

I encountered an odd situation recently.  I had a remote office with multiple local networks, internet access for all of them behind NAT, and a Fortigate to Fortigate site to site VPN configuration with multiple destination networks that went through the tunnel.  This is all pretty typical of companies with remote offices; remote users can access the corporate network across the VPN, but internet-bound traffic just goes out the local connection with NAT.

On the VPN config side, this is a Fortigate to Fortigate VPN, which means I was handling the VPN traffic with a single tunnel definition where the phase2 local and remote addresses were left as 0.0.0.0/0 so the firewalls could figure it out based on policy.  If it were not Fortigate to Fortigate, you would of course have to define each local and remote network pair individually in the phase 2 settings.

With the VPN configured, networks that were supposed to go through the tunnel had relevant lines in the firewall policy permitting it, and then there were static route definitions where the ‘Device’ used to reach those external networks was set to the IPSec tunnel device.  No next hop is specified in this configuration once the tunnel device is selected.  This setup worked great, traffic from the remote users to the internet was permitted and NAT’d, traffic from the remote users to the corporate networks across the tunnel was permitted and not NAT’d.

A unique combination of events and circumstances resulted in an odd situation developing:

• A group of staff were moved from the central office to the remote office.
• They used web-based applications which were written to behave differently for users coming from a central office IP range.
• Their IP range was RFC1918 space; i.e. private networks.
• The pre-existing remote office users should continue to access the apps via public internet, or more importantly, not private network space, since that would result in the apps behaving in a way they should not see.

So, what to do.  Well, moving the central office team, moving their private network and keeping their access to the web apps as if they had not even moved was the easy part.  We’ll use the following definitions:

• Users that are moving, 10.100.2.0/24
• Web-apps hosted on public range 1.2.3.0/24 which is on site at central office
• Pre-existing remote office users, 172.16.0.0/24
• We’ll call the IPSec tunnel defined on both Fortigates ‘Site2SiteVPN’
• Central office Fortigate external interface (i.e., the VPN target IP) is 1.2.3.4 (notice this is on the same network as the public web apps being accessed by Internet users)

The move steps:

• Power down the users on 10.100.2.0/24 that will be moving.
• Delete 10.100.2.0/24 from central office core router
• Add static route on central office core router sending 10.100.2.0/24 to the local Fortigate.
• Create a VLAN for them at the remote office, create router interface, put their specific 10.100.2.0/24 network on it.
• VPN already exists between the two sites so no creation of a tunnel is needed
• VPN is Fortigate to Fortigate so no adjustment or addition of IKE phase 2 networks is needed
• Add a policy entry on remote office Fortigate saying traffic coming from the relevant interface, whether it be physical or vlan, from 10.100.2.0/24, is permitted to go out device Site2SiteVPN with destination 1.2.3.0/24 with NO NAT.
• Add a policy entry on central office Fortigate saying traffic coming in from interface Site2SiteVPN, source address 10.100.2.0/24, destination address 1.2.3.0/24, is permitted.
• Power the migrated users back up.

Okay, at this point, I kind of wondered if things would actually work because of the fact that I needed to send traffic for 1.2.3.0/24 across the Site2SiteVPN ‘device’ via the static route, but the remote VPN target itself was on that network at 1.2.3.4.  How could you have a static route sending a particular network’s traffic through a device that would not be in an ‘up’ state if that device was not reached across the internet first instead of via the tunnel?

Well, the Fortigate perhaps anticipates this, knows what you really intended to do, or who knows what, but everything seemed to be working great.  The migrated users had their same network as before, the tunnel remained up with IPSec packets leaving via default gateway destined for 1.2.3.4, and any other traffic coming in from behind the Fortigate for the 1.2.3.0/24 network going through the tunnel.  The migrated users were able to access the central office web ap from their private IP’s as if they had never moved since their traffic was going across the tunnel without NAT, and the remote end knew how to get the return traffic back to them through the tunnel.

All was not actually okay though.  This configuration broke access to the 1.2.3.0/24 network for the pre-existing users at the remote office.  The reason is the static route for 1.2.3.0/24 through the tunnel.  Those pre-existing users have no policy related to accessing the 1.2.3.0/24 network; they had just been accessing the web apps on that network by way of their internet access policy rule stating “Remote office users on VLAN 123 are permitted to access ‘any’ via interface ‘wan1’ with NAT enabled.”  They did not have a policy allowing their traffic to cross the VPN, but the static route was forcing the traffic to only go across the VPN.  I also could not add such a policy because these users were intentionally supposed to only access the web apps via a public IP because the app behaved differently for private IP users, which would only be the central office folks.

So what to do, I knew Fortigate units supported policy based routing, but I didn’t see that option anywhere.  I discovered it needs to be enabled by way of turning on the ‘Advanced Routing’ feature.  You can do this in the GUI by editing the ‘Features’ settings on the status screen.  Okay, so that’s on, now to configure.  My initial attempt at configuring was to delete the 1.2.3.0/24 static route using device Site2SiteVPN, and adding a policy based static route that said: “ANY protocol, coming in on VLAN 456 (the migrated users), source address 10.100.2.0/24, destination address 1.2.3.0/24, outgoing interface Site2SiteVPN, gateway address, 1.2.3.4.”  I thought it was odd that it asked me for a gateway address after specifying the VPN tunnel device Site2SiteVPN, since in the traditional static route dialogue, the next hop disappears when you select a tunnel device.  I actually wasn’t sure what to put there so the 1.2.3.4 VPN target IP was just a wild guess.

It didn’t work.  Traffic for all users was now going across the internet via the regular users can access the internet with NAT policy rule.  I thought policy routing was broken.  I tinkered with the next hop, could not find a value that changed anything.  How could I configure a Fortigate policy route where the next hop goes through a VPN tunnel?  I thought to myself, even though it doesn’t entirely make sense, what if I add a more specific static route just for the VPN target?  So I add a 1.2.3.4/32 static (not policy based static) route, switch device to Site2SiteVPN, which of course takes away the next hop box, and now everything is working great.  Regular users accessing 1.2.3.0/24 have no special policy, and no special route, so their traffic is NAT’d and leaves via the internet, the migrated users on 10.100.2.0/24 have their traffic hit the policy route, matching firewall policy is found, their traffic goes across the tunnel without NAT.  Remote web app living in 1.2.3.0/24 sees the original remote office users as public IP, and the migrated users as the private IP’s, everyone is happy.

So I’m still not sure why the policy-based static route was ignored until I added the more specific tunnel target static route back in, and I’m also still not sure how a Fortigate internally processes a static route directive that tells it to use a tunnel device for a destination IP, that could only be reached via the internet prior to the tunnel coming up, but it works.  Perhaps the code has some logic that says static routes using a tunnel device do not apply to IPSec packets destined for that same target that would be needed to bring the tunnel up?  That still doesn’t explain why the policy route was ignored until the static route was added.

Anyway, if someone does have this answer and wants to add comments, please do, would love a better explanation.