BGP in a small network: when dynamic routing should replace static routes

Head office and two branches are connected by IPsec tunnels, and each branch has a second tunnel over LTE. Routing is static: a primary route and a backup route with a worse administrative distance. The fibre at one branch goes down. The branch router switches to the backup tunnel, but head office keeps replying through the primary tunnel, because its tunnel interface has not changed state. Packets from the branch reach head office, and the replies are lost.

Dynamic routing is not reserved for service providers. With several sites and backup paths, a BGP session acts as a health check shared by both ends of the tunnel: when it drops, both sides withdraw their routes almost simultaneously. The condition is filters and limits from day one, because without them BGP passes on everything it receives.

When static routes are entirely sufficient

A single site with a single internet exit does not need dynamic routing: workstations have a default gateway, and the firewall has a default route to the provider. A second link does not change that, because outbound failover is handled by link health checks, described in the post on two internet links. Two sites with one tunnel mean one static route on each side.

Typical signs that static routes are no longer enough:

• Every new subnet requires changes on several routers, and missing one produces a network that works “almost everywhere”.
• Backup tunnels rely on routes with a worse administrative distance, switched only by interface state.
• Each end of a tunnel assesses the path on its own, so failover is sometimes one-sided.

! primary route through the tunnel on the fibre link
ip route 10.20.0.0/16 10.255.1.2
! backup route through the LTE tunnel, administrative distance 250
ip route 10.20.0.0/16 10.255.2.2 250

The backup route enters the routing table only when 10.255.1.2 becomes unreachable, usually after the tunnel interface goes down. Tunnel interfaces on Linux (VTI, XFRM) stay up even when IPsec is not working, and on many firewalls the interface state depends on dead peer detection (DPD). A good addressing plan, with one /16 block per site, keeps the number of entries low, but does not change the fact that a static route does not know whether the path works.

Why BGP and not OSPF

BGP establishes a TCP session on port 179 with a neighbour named in the configuration. It needs neither multicast nor a shared network segment, so it works over any tunnel that carries ordinary IP traffic. Neighbours entered by hand mean you know exactly whom the router exchanges routes with.

The second advantage is per-neighbour policy. A prefix list defines what may be accepted and what may be announced. Local-preference marks the preferred exit within a single autonomous system (AS). AS-path prepend, which adds your own AS number to the path several times, makes a route look longer to neighbours in other ASes. BGP is supported by firewalls (FortiGate, MikroTik, pfSense and OPNsense through the FRR package), Linux with FRR and the VPN gateways in AWS and Azure.

OSPF is a fair alternative within a single site with several routers or L3 switches: neighbours discover each other, and the interface cost sets path preference. However, routers in an area must share the same link-state database, so a route cannot be filtered from a single neighbour, and every topology change spreads across the whole area, including over LTE links. Redistributing routes between OSPF and BGP is a separate source of errors, so a small company should choose one protocol.

Head office as a route reflector

The simplest design is a single private AS (e.g. 65001) for the whole company and iBGP sessions over the tunnels. A router does not pass a route learned from one iBGP neighbour on to other iBGP neighbours, so without extra configuration every branch would need a session with every other branch. A route reflector at head office passes the branches’ routes on, and each branch has a session with it over each of its tunnels.

Addresses in the examples: head office 10.10.0.0/16, branch A 10.20.0.0/16, branch B 10.30.0.0/16, primary tunnels in 10.255.1.0/24, backup tunnels in 10.255.2.0/24, one /30 network per tunnel.

! head office range in the routing table, so the network command has something to announce
ip route 10.10.0.0/16 blackhole
!
ip prefix-list PL-BRANCH-A seq 10 permit 10.20.0.0/16
ip prefix-list PL-BRANCH-B seq 10 permit 10.30.0.0/16
ip prefix-list PL-TO-BRANCHES seq 10 permit 10.10.0.0/16
ip prefix-list PL-TO-BRANCHES seq 20 permit 10.20.0.0/16
ip prefix-list PL-TO-BRANCHES seq 30 permit 10.30.0.0/16
!
! routes from the backup tunnel: lower preference
route-map RM-BACKUP-IN permit 10
 set local-preference 50
!
router bgp 65001
 bgp router-id 10.10.0.1
 bgp log-neighbor-changes
 neighbor BRANCHES peer-group
 neighbor BRANCHES remote-as 65001
 neighbor BRANCHES password <session-password>
 ! BFD requires the bfdd daemon to be running
 neighbor BRANCHES bfd
 ! branch A: primary and backup tunnel
 neighbor 10.255.1.2 peer-group BRANCHES
 neighbor 10.255.2.2 peer-group BRANCHES
 ! branch B: primary and backup tunnel
 neighbor 10.255.1.6 peer-group BRANCHES
 neighbor 10.255.2.6 peer-group BRANCHES
 !
 address-family ipv4 unicast
  network 10.10.0.0/16
  neighbor BRANCHES route-reflector-client
  neighbor BRANCHES next-hop-self force
  neighbor BRANCHES prefix-list PL-TO-BRANCHES out
  neighbor BRANCHES maximum-prefix 20
  neighbor 10.255.1.2 prefix-list PL-BRANCH-A in
  neighbor 10.255.2.2 prefix-list PL-BRANCH-A in
  neighbor 10.255.2.2 route-map RM-BACKUP-IN in
  neighbor 10.255.1.6 prefix-list PL-BRANCH-B in
  neighbor 10.255.2.6 prefix-list PL-BRANCH-B in
  neighbor 10.255.2.6 route-map RM-BACKUP-IN in
 exit-address-family

The network command announces a prefix only when exactly that prefix is in the routing table, hence the blackhole route for the head office range. Traffic to non-existent subnets is dropped instead of looping. bgp log-neighbor-changes writes every session state change to the log.

A backup tunnel with lower preference

Local-preference is one of the first criteria in best-path selection, ahead of AS path length. The higher value wins, and the default is 100. Head office assigns 50 to routes from the backup tunnel, and the branch does the same on its side. When the session on the primary tunnel drops, both ends remove its routes and move to the backup almost simultaneously, and once the session is re-established they move back by themselves. The same policy on a branch with a FortiGate:

config router route-map
    edit "RM-BACKUP-IN"
        config rule
            edit 1
                set set-local-preference 50
            next
        end
    next
end
config router bgp
    set as 65001
    set router-id 10.20.0.1
    config neighbor
        edit "10.255.1.1"
            set remote-as 65001
        next
        edit "10.255.2.1"
            set remote-as 65001
            set route-map-in "RM-BACKUP-IN"
        next
    end
    config network
        edit 1
            set prefix 10.20.0.0 255.255.0.0
        next
    end
end

The branch range must also be in the routing table, for example as a static route with set blackhole enable. Filters are created under config router prefix-list and attached with the neighbour fields prefix-list-in and prefix-list-out, and the session password goes in the password field. In the FortiOS CLI the ? character opens context help, so it is easier to generate the password without it.

BGP with a provider

A BGP session with an internet provider, that is announcing your own addresses over two links, requires a public AS number and provider-independent addresses. Both are assigned by the RIPE NCC, usually through a LIR, and an AS number request needs a justification, most often connections to at least two networks. The smallest IPv4 network widely accepted on the internet is a /24, and the RIPE NCC IPv4 pool is exhausted, so addresses are bought or leased. For a small company two links with failover are usually enough, and BGP stays inside its own network.

Session safeguards

Without filters, a mistake at one branch, such as announcing a default route, reaches every router.

Rule	Protects against	FRR
Inbound prefix filter	a default route or someone else’s range from a neighbour	prefix-list … in
Outbound prefix filter	route leaks, e.g. management networks	prefix-list … out
Prefix limit	flooding the table after a filter error	maximum-prefix 20
Session password	a session from an unauthorised device	password
BFD or shorter timers	minutes of traffic into a dead tunnel	bfd, timers 10 30
Private AS number	collisions with public numbers	64512–65534

Exceeding the prefix limit closes the session by default: a dropped session at one branch is a smaller problem than hundreds of wrong routes. redistribute connected announces every connected network, including management and tunnel networks, so it should not be used without a route map.

The session password is the TCP MD5 option (RFC 2385), supported by firewalls and FRR. The newer TCP-AO (RFC 5925) is more secure, but support needs checking on both sides. The password does not encrypt the session; inside a tunnel, IPsec does that.

Default BGP timers in many implementations are 60 s between keepalive messages and 180 s before a neighbour is declared unreachable, so without BFD failover can take three minutes. BFD in FRR sends packets every 300 ms by default and declares the path dead after three are missed. It must be enabled on both sides of the session.

Static, OSPF or BGP

Criterion	Static routes	OSPF	BGP
Typical scope	1–2 sites, a single path	one site, several L3 routers	several sites, backup tunnels
Path failure detection	interface state or a separate monitor	built in, optionally BFD	built in, optionally BFD
Consistency of both ends	each side separately	shared link-state database	one session for both sides
New subnet	change on many devices	automatic	no change within the site range
Per-neighbour filtering	not applicable	only at the area border	full
Path selection	administrative distance	interface cost	local-preference, AS path
Cloud VPN gateways	no	usually not	yes
Entry threshold	lowest	medium	highest, filters from the start

Check your own environment

On a router with FRR, these commands show session state, the routes installed by BGP and the prefixes announced to a neighbour.

# vtysh requires root or membership of the frrvty group
vtysh -c "show bgp summary"
vtysh -c "show ip route bgp"
# all paths to branch A's range, with local-preference and the best path marked
vtysh -c "show bgp ipv4 unicast 10.20.0.0/16"
# prefixes announced to branch A over the primary tunnel
vtysh -c "show ip bgp neighbors 10.255.1.2 advertised-routes"

Every neighbour should show a number in the State/PfxRcd column; Active, Connect or Idle means there is no session. A short time in the Up/Down column for a neighbour nobody has touched points to an unstable tunnel. The best path (best) to a branch should run through the primary tunnel with local-preference 100, and the announced prefixes should match PL-TO-BRANCHES.

On a FortiGate, the get commands show the same information.

get router info bgp summary
get router info routing-table bgp
get router info bgp neighbors 10.255.1.1 advertised-routes

A good result on branch A: a number in State/PfxRcd for both neighbours, routes to 10.10.0.0/16 and 10.30.0.0/16 via 10.255.1.1, and only 10.20.0.0/16 in the list of announced prefixes.

From a workstation at the branch you can see which way traffic to head office actually goes.

Find-NetRoute -RemoteIPAddress 10.10.1.10
Get-NetRoute -AddressFamily IPv4 | Where-Object NextHop -ne '0.0.0.0'
tracert -d -h 6 10.10.1.10

A workstation most often has only a default route to the branch router. The second hop in tracert is usually the head office address on the primary tunnel (10.255.1.1); the backup tunnel address (10.255.2.1) with no known outage means traffic is going over LTE.

ip route
ip route get 10.10.1.10
traceroute -n -m 6 10.10.1.10

ip route get shows the route chosen for a specific address, and traceroute reads like tracert.