There are multiple P2P MPLS client networks that hang off of South-Client-Handoff and North-Core which must pass all of their up/downstream traffic through MPLS-Handoff. There are also non-P2P MPLS client networks whose up/downstream traffic must pass through Edge via VLAN 15.
One of the big take-home points here is that we cannot trunk VLAN 24 between South-Core and Edge. We could create an SVI for VLAN 15 (let's say, 192.168.15.6/28) on South-Client Handoff and put it into OSPF Area 0, as seen below:
Pre-VLAN 15 SVI addition to South-Client-Handoff:
----------------------------------------------
North-Core#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 192.168.24.4 on Vlan24, 00:08:54 ago
Routing Descriptor Blocks:
* 192.168.24.4, from 4.4.4.4, 00:11:17 ago, via Vlan24
Route metric is 2, traffic share count is 1
Edge#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 3, type intra area
Last update from 192.168.15.3 on Vlan15, 00:00:12 ago
Routing Descriptor Blocks:
* 192.168.15.4, from 4.4.4.4, 00:00:12 ago, via Vlan15
Route metric is 3, traffic share count is 1
192.168.15.3, from 4.4.4.4, 00:00:12 ago, via Vlan15
Route metric is 3, traffic share count is 1
Post-VLAN 15 SVI addition to South-Client-Handoff:
-----------------------------------------------
North-Core#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 192.168.15.6 on Vlan15, 00:00:06 ago
Routing Descriptor Blocks:
* 192.168.24.4, from 4.4.4.4, 00:02:29 ago, via Vlan24
Route metric is 2, traffic share count is 1
192.168.15.6, from 4.4.4.4, 00:00:06 ago, via Vlan15
Route metric is 2, traffic share count is 1
Edge#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 192.168.15.6 on Vlan15, 00:00:34 ago
Routing Descriptor Blocks:
* 192.168.15.6, from 4.4.4.4, 00:00:34 ago, via Vlan15
Route metric is 2, traffic share count is 1
But, then our P2P MPLS Client Networks will get the default route out through Edge on VLAN 15 unless we get creative with a distribution-list to filter them away from VLAN 15 and keep them routing in/out of VLAN 24 through our MPLS-Handoff. At that point, it becomes less of a hassle for us to just keep the odd switched path through North-Core that I'll be covering below.
With OSPF max-paths enabled for multi-path traffic forwarding, downstream traffic for non-P2P MPLS clients hanging off of South-Client-Handoff is split switched either to SVI 15 (192.168.15.4) on South-Core or SVI 15 (192.168.15.3) on North-Core, then routed onto VLAN 24 to go to the South-Client-Handoff:
Edge#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 3, type intra area
Last update from 192.168.15.3 on Vlan15, 00:01:26 ago
Routing Descriptor Blocks:
* 192.168.15.4, from 4.4.4.4, 00:01:26 ago, via Vlan15
Route metric is 3, traffic share count is 1
192.168.15.3, from 4.4.4.4, 00:01:26 ago, via Vlan15
Route metric is 3, traffic share count is 1
Edge#traceroute 10.17.12.1This is because the non-P2P MPLS clients' assigned public ranges are advertised in OSPF by South-Client-Handoff via VLAN 24, as seen here:
Type escape sequence to abort.
Tracing the route to 10.17.12.1
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.15.3 5 msec
192.168.15.4 4 msec
192.168.15.3 4 msec
2 192.168.24.4 5 msec 6 msec 4 msec
South-Core#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 192.168.24.4 on Vlan24, 00:34:47 ago
Routing Descriptor Blocks:
* 192.168.24.4, from 4.4.4.4, 00:34:47 ago, via Vlan24
Route metric is 2, traffic share count is 1
North-Core#sh ip route 10.17.12.1While this isn't horrible, it's certainly a waste of bandwidth, as downstream traffic for our non-P2P clients is, in part, switched out e0/3 of South-Core, onto North-Core, then is routed off VLAN 15 onto VLAN 24 and egresses e0/1 back onto South-Core before finally egressing e0/2 onto South-Client-Handoff.
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 2, type intra area
Last update from 192.168.24.4 on Vlan24, 00:10:03 ago
Routing Descriptor Blocks:
* 192.168.24.4, from 4.4.4.4, 00:10:03 ago, via Vlan24
Route metric is 2, traffic share count is 1
We could aleviate this by setting OSPF max-paths to 1, but then we'll still have to trust that OSPF on Edge will have its route through South-Core to 10.17.12.1 as the more stable (oldest) LSA (since they're equal cost paths) to avoid looping traffic through North-Core. Lucky for us, SVIs are pretty stable (unless you go and purge VLANs from the config, or start messing with VLAN suspending), but we can't guarantee that SVI 15 on South-Core (192.168.15.4) will always be stable (what if we only have that VLAN tagged on one circuit and it flaps) or that North-Core will have its current SVI 15 IP address. Let's see what happens when we set max-path to 1 and change North-Core's SVI 15 to 192.168.15.5.
Edge(config)#router ospf 1Because max-path clears the RIB, we get equal cost/equal age LSAs, and Edge selects North-Core's new, higher SVI (192.168.15.5) Let's see what happens if we shut e0/3 and e0/1 on North-Core to force Edge to only have the LSA for 10.17.12.1 from South-Core with a lower SVI (192.168.15.4), but then bring North-Core back in with a higher SVI but a shorter LSA age (less stability).
Edge(config-router)#maximum-path 1
Edge#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 3, type intra area
Last update from 192.168.15.5 on Vlan15, 00:00:10 ago
Routing Descriptor Blocks:
* 192.168.15.5, from 4.4.4.4, 00:00:10 ago, via Vlan15
Route metric is 3, traffic share count is 1
North-Core(config)#int range e0/1 , e0/3
North-Core(config-if-range)#shut
*Mar 7 00:54:10.390: %LINK-5-CHANGED: Interface Ethernet0/1, changed state to administratively down
*Mar 7 00:54:10.390: %LINK-5-CHANGED: Interface Ethernet0/3, changed state to administratively down
*Mar 7 00:54:11.395: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
*Mar 7 00:54:11.395: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/3, changed state to down
North-Core(config-if-range)#
*Mar 7 00:54:40.295: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Vlan15 from FULL to DOWN, Neighbor Down: Dead timer expired
*Mar 7 00:54:40.612: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Vlan24 from FULL to DOWN, Neighbor Down: Dead timer expired
North-Core(config-if-range)#
*Mar 7 00:54:46.425: %OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on Vlan24 from FULL to DOWN, Neighbor Down: Dead timer expired
North-Core(config-if-range)#
*Mar 7 00:54:47.820: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on Vlan15 from FULL to DOWN, Neighbor Down: Dead timer expired
Edge#sh ip route 10.17.12.1
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 3, type intra area
Last update from 192.168.15.4 on Vlan15, 00:11:11 ago
Routing Descriptor Blocks:
* 192.168.15.4, from 4.4.4.4, 00:01:11 ago, via Vlan15
Route metric is 3, traffic share count is 1
North-Core(config-if-range)#no shut
Edge#sh ip route 10.17.12.1Look at that. LSA stability (oldest in database) is preferred over North-Core's SVI 15 (192.168.15.5). But, that still doesn't afford us much comfort in our environment. Who wants to rely solely on LSA stability? Let's say we want to keep max-path as > 1, but we still don't want to split traffic.
Routing entry for 10.17.12.1/32
Known via "ospf 1", distance 110, metric 3, type intra area
Last update from 192.168.15.5 on Vlan15, 00:00:02 ago
Routing Descriptor Blocks:
* 192.168.15.4, from 4.4.4.4, 00:04:31 ago, via Vlan15
Route metric is 3, traffic share count is 1
Since we can't build a route-map with a prefix-list of Non-P2P MPLS Client Networks to increment the LSA cost received on North-Core from neighbor South-Client-Handoff, and we're not using point-to-multipoint so per-neighbor bandwidth cost is out, our best alternative to trusting that the LSA through South-Core is more stable is increasing the OSPF Cost on North-Core's SVI 24 (192.168.24.2) to 2 from 1 so Edge prefers the more direct path through South-Core.
Edge(config)#router ospf 1
Edge(config-router)#no max-path 1
Edge#sh ip route ospf
10.0.0.0/32 is subnetted, 1 subnets
O 10.17.12.1 [110/3] via 192.168.15.4, 00:00:15, Vlan15
[110/3] via 192.168.15.3, 00:00:15, Vlan15
192.168.24.0/28 is subnetted, 1 subnets
O 192.168.24.0 [110/2] via 192.168.15.4, 00:00:15, Vlan15
[110/2] via 192.168.15.3, 00:00:15, Vlan15
North-Core(config)#int vlan 24
North-Core(config-if)#ip ospf cost 2
Edge#sh ip route ospf
10.0.0.0/32 is subnetted, 1 subnets
O 10.17.12.1 [110/3] via 192.168.15.4, 00:14:46, Vlan15
192.168.24.0/28 is subnetted, 1 subnets
O 192.168.24.0 [110/2] via 192.168.15.4, 00:14:46, Vlan15
So, our best solution in order to keep multiple equal-cost paths, but not rely on LSA stability to avoid additional bandwidth utilization, appears to be altering the OSPF Cost on North-Core's VLAN 24 SVI. Keep in mind, though, that this will apply to all advertised networks that North-Core gets via VLAN 24 from South-Client-Handoff.
Luckily, we have relatively large pipes in our production environment, so we can handle the extra switched bandwidth without worrying about altering per-SVI OSPF Cost. A minor bump in the road for a simplified configuration, I'd say.
