BGP in an Arista Data Center

The following is a practical analysis of the use of BGP in the DC on Arista platforms based largely on Petr Lapukhov's work with BGP in hyperscale DCs

Why Layer 3 (L3)?

There are several reasons to run a L3 routing protocol over legacy layer 2 (L2) designs in the data center. Leveraging standards-based routing protocols to avoid vendor lock-in, provide for faster convergence, minimize L2 fault domains, and provide for better traffic engineering.

Extension of L2

Naturally something that comes into question in a L3 switch fabric is, “What if I need L2 adjacency between hosts?” For Arista, the extension of L2 services across a L3 switch fabric is provided by Virtual eXtensible LAN (VXLAN). While closely related, in-depth discussion of the “network overlay” provided by VXLAN is outside the scope.

Why BGP?

Some might question the use of BGP within the data center due to it being designed for, and in the past primarily leveraged as, an EGP. However, BGP provides several benefits in a data center switch fabric, such as:

• Less complexity in protocol design
• Relies on TCP rather than adjacency formation/maintenance and/or flow control
• Less “chatty”
• Supports third-party (recursively-resolved) next-hops
• With proper ASN usage, built-in loop prevention via AS_PATH
• Better support for traffic engineering

General Operations Within the Fabric

ASN Scheme Option 1(Per-Rack Leaf AS)

Below is a diagram derived mostly from an IETF Draft that is a joint effort by Facebook, Google, and Arista Networks for very large data centers. It is shown in entirety to help visualize the reuse of ASNs at the leaf layer.

Notice that ASNs are reused at the leaf layer for each “cluster” or “pod”. This is achievable by configuring the leaf switches with allowas-in, and minimizes the need for 4-Byte ASNs which not all vendors support. Loop prevention is provided at the spine layer by placing spine switches within the same AS at each cluster/pod.

Going forward, focus will be on the spine/leaf elements.

ASN Scheme Option 2 (Leaf Layer AS)

Another option could be to use the same AS at every leaf:

Advantages of this design include less ASN usage, simple configuration templatization, and with that, better support for automation. A disadvantage is losing the ability to trace routes via AS_PATH (At least without additional configuration). These topics are further discussed in later sections.
Intra-AS Spine-to-Spine Reachability

Due to the spine switches being in the same AS, direct reachability from spine-to-spine within the same AS is not supportable without breaking loop prevention within the architecture itself. However, the impact of this is very minimal and should mostly be ignored.

Dynamic Neighbor Discovery

As a data center scales out, it is possible that many more leaf switches will be added. To make life easier, it is very desirable to automate and reduce configuration however possible. BGP dynamic neighbors configured at the spine layer provides a mechanism to dynamically add new leaf switches as BGP peers based on an IP address range. To support this, care must be taken in IP addressing as described in the “IP Addressing and Prefix Advertisement” section.

BGP Dynamic Neighbors with Per-Rack Leaf AS

Below is a diagram depicting the utilization of a separate AS at each rack:

In this configuration, a bgp listen router configuration command is added for each new rack with the IP address range accounting for the routed links going to each leaf switch within the rack. This reduces the amount of required configuration versus static configuration for each individual leaf.

BGP Dynamic Neighbors with Leaf Layer AS

Below, the second option leverages the same AS at every leaf:

In this configuration, a single bgp listen router configuration command is all that is required at each spine. No further configuration is needed at the spine as additional leaf switches are added to the environment. This makes the configuration easy to templatize and automate across all spine switches.

Making it Easy to Automate

It is much easier to automate configuration when it can be applied to a container of devices rather than having to have specific, per-device configuration.

For example, Ansible is a YAML-based automation platform that can be leveraged to push configuration to Arista switches. When configuration is simple and repeatable on as many devices as possible, that configuration can be compartmentalized into a template more easily. Ansible uses “playbooks” which are basically a list of resources and what tasks should be performed on those resources. Below is an excerpt from a playbook showing a single template configuration that could be applied to all spine switches in a per-rack Leaf AS design (Given a whopping two racks, but you get the idea):

In summary, the more configuration you can compartmentalize and template, the less items need to be defined as variables while still being able to be repeated on multiple devices - and still meet requirements of course - the better.

Route Source Tracing via AS_PATH with Leaf Layer AS

As mentioned earlier, a disadvantage to using the same AS at every leaf is not having the ability to trace routes via AS_PATH:

To get around this, an ASN could be prepended using a route map on each leaf switch, for example:

BGPDC-LEAF1(config)#route-map BGP_PREPEND
BGPDC-LEAF1(config-route-map-BGP_PREPEND)#set as-path prepend 65101

BGPDC-LEAF2(config)#route-map BGP_PREPEND
BGPDC-LEAF2(config-route-map-BGP_PREPEND)#set as-path prepend 65102

Then on both leaf switches, apply the route-map to all advertised routes:

router bgp 65000
neighbor ARISTA route-map BGP_PREPEND out

Now from LEAF3’s perspective, it is easier to see where the routes were sourced from:

IP Addressing and Prefix Advertisement

One challenge that resides in having a layer three (L3) switch fabric in the data center is how to handle your point-to-point routed links between spine and leaf switches. Not advertising them can increase troubleshooting difficulty, but advertising them inefficiently can lead to a large number of /31 entries in the Routing Information Base (RIB) that may become resource-intensive.

As a hypothetical example, if an Arista 7050QX-32 switch were to be used as a spine switch, assuming no ports are used for uplinks to the core or edge, and only 40Gb links are used to connect leaf switches, you would have a total of 32 links with /31 addressing. This means they could be aggregated as follows:

• 16 /30s
• 8 /29s
• 4 /28s
• 2 /27s
• 1 /26

A single /26 can represent all possible point-to-point connections that the particular spine switch could have. So if you started with 192.168.255.0 for your IP addressing on your point-to-point links, you could allocate contiguous blocks of IP addresses starting with 192.168.255.0/26 for the first spine switch, 192.168.255.64/26 for the second spine switch, and so on. Then on each spine switch, only advertise its loopback and the aggregate with the aggregate-address summary-only BGP router configuration command.

What this will ultimately look like from the leaf layer perspective is an optimized RIB containing the loopbacks of all switches in the fabric, and a single additional entry per spine switch to reach point-to-point IP addresses for troubleshooting purposes.

To help visualize, consider a scenario where two spine switches are connected to just three leaf switches. The two spine switches have been configured to advertise every leaf connection, like so:

BGPDC-SPINE1(config-router-bgp)#sh active
router bgp 64600
router-id 192.168.254.1
bgp log-neighbor-changes
distance bgp 20 200 200
maximum-paths 32 ecmp 32
bgp listen range 192.168.255.0/24 peer-group ARISTA remote-as 65000
neighbor ARISTA peer-group
neighbor ARISTA fall-over bfd
neighbor ARISTA password 7 6x5GIQqJNWigZDc2QCgeMg==
neighbor ARISTA maximum-routes 12000
network 192.168.254.1/32
network 192.168.255.0/31
network 192.168.255.2/31
network 192.168.255.4/31

Looking at the routing table of LEAF1, there are a number of /31 entries learned via BGP:

As you can imagine, in larger environments, the number of entries could be excessive. Instead, the spine switches can configured with the aggregate-address router configuration command:

BGPDC-SPINE1(config-router-bgp)#sh active
router bgp 64600
router-id 192.168.254.1
bgp log-neighbor-changes
distance bgp 20 200 200
maximum-paths 32 ecmp 32
bgp listen range 192.168.255.0/24 peer-group ARISTA remote-as 65000
neighbor ARISTA peer-group
neighbor ARISTA fall-over bfd
neighbor ARISTA password 7 6x5GIQqJNWigZDc2QCgeMg==
neighbor ARISTA maximum-routes 12000
network 192.168.254.1/32
aggregate-address 192.168.255.0/28 summary-only


The same output from LEAF1 now shows no /31 entries learned from BGP:

Instead, there is a single /28 entry from each spine learned via BGP:

BGP at the Spine Layer

The Need for Fast Failure Detection

In a BGP switch fabric it is critical for failures to be detected as quickly as possible at the spine layer. Consider the following hypothetical scenario:

NODE1 is communicating with NODE2. LEAF1 and LEAF2 are MLAG peers. In an ECMP-routed switch fabric, asymmetric routing can occur, where NODE1’s traffic flow is taking the path through LEAF2, while the return traffic is taking the path through LEAF1. This is fine for now, but let’s consider what happens when LEAF1 is rebooted:

With default behavior of MLAG, when LEAF1 finishes rebooting and re-establishes the MLAG session with LEAF2, LEAF1 will place all non-peer-link interfaces into the err-disable state for 300 seconds (five minutes) in order to allow the topology to reconverge. So including the reboot, you’re looking at ~7 minutes or so before the switch begins forwarding again. Even though you have a redundant gateway provided by LEAF2, the reboot of LEAF1 would actually create a problem when combined with the fact that the spine switches are using default configuration in this particular situation of asymmetric routing. Default BGP keepalive and hold timers are 30 seconds and 180 seconds, respectively, so for 180 seconds (3 minutes), the spine switches will still consider LEAF1 an established BGP neighbor, and still consider the path to LEAF1 a viable route even though it’s clearly not. So for any flow that gets hashed to this path, the traffic is effectively blackholed. BGP timers could be modified to provide some relief, but failover time may still be considered unacceptable, in addition to the possibility of introducing instability of the network to BGP peer flapping.

By default, Arista switches will immediately tear down an eBGP neighbor adjacency when the interconnecting link goes down. This alleviates the issue described above. To further improve failure detection, Bidirectional Forwarding Detection (BFD) should also be leveraged. BFD is a very simple, low overhead Hello protocol that works in conjunction with BGP to provide sub-second failure detection. BFD adds additional protection in “gray” failure conditions such as would possibly be seen with a degraded fiber strand. The interface status may remain up and therefore not trigger the eBGP neighborship tear-down, but not pass traffic intermittently or at all - which in those cases network performance can suffer while the BGP keepalive/hold-down mechanism by itself may or may not be of much help.

BGP at the Leaf Layer

MLAG - To Peer or not to Peer

When running a pair of leaf switches in an MLAG configuration with VARP, you want to peer the two leaf switches. Although unlikely to actually happen, consider the following failure scenario:

NODE1 is communicating with NODE2. LEAF1 and LEAF2 are MLAG peers and are configured to run VARP to provide an active/active redundant first hop gateway for NODE1, but they are not BGP peers. On LEAF1, both of its uplinks have failed. While both SPINE1 and SPINE2 would certainly notice this failure and reconverge, some things will not be affected, such as:

• MLAG running between LEAF1 and LEAF2 would not notice any change, and continue functioning as normal, which in turn means that the port-channel between NODE1 and LEAF1/LEAF2 would remain up and function as normal
• VARP would continue to function as normal

This is important because traffic leaving NODE1 that gets hashed to LEAF2 would be fine, but any traffic hashed to LEAF1 would effectively be blackholed:

Peering LEAF1 and LEAF2 alleviates this. Traffic hashed to LEAF1 would follow the only remaining route pointing to LEAF2, and then be ECMP-routed to the spine layer:

Additionally, when this failure scenario doesn’t exist, you don’t have to worry about having a suboptimal path because without the failure described above, the path through the peering leaf switch is longer than those directly connected to the spine layer:

Optimizations

update wait-for-convergence

The update wait-for-convergence BGP router configuration command basically prevents BGP from programming routes into hardware and from advertising routes until a convergence event is resolved. A convergence event can occur when initializing BGP for the first time, clearing the BGP process, terminating the Rib agent, etc. The benefit this provides is reduced CPU churn from repeatedly programming routes into and out of hardware during a convergence event.

Best practice is to only use this on spine switches. Enabling this on leaf switches (especially leaf switches acting as host gateways) can result in traffic being blackholed until BGP finishes installing routes in hardware.

NOTE: Enabling this feature in vEOS will cause routes not to be advertised.

update wait-install

The update wait-install BGP router configuration command prevents routes from being advertised to peers until they are programmed in hardware, and that hardware sends an “Ack” to software. This prevents traffic being dropped in situations where routes are advertised, but they aren’t programmed yet in hardware.

NOTE: Enabling this feature in vEOS will cause routes not to be advertised.

ip hardware fib next-hop update event bfd

The ip hardware fib next-hop update event bfd global configuration command allows BFD to remove an interface that went down from next-hop entries of routes using it in hardware. The benefit is reduced traffic loss compared to the normal operation of BFD which is to bring down the BGP neighborship and recompute the next-hop for all routes using the interface which went down and then reprogram hardware. This is a default configuration whenever BFD is enabled.

ip hardware fib route unprogrammed parent-drop

The ip hardware fib route unprogrammed parent-drop global configuration command makes it so that if a more specific route cannot be programmed in hardware for whatever reason, its parent route will be programmed to point to a drop index (similar to null0). This is default behavior.

NOTE: This is not really an optimization, and is instead more of a feature. It should be noted that this will result in traffic being black-holed rather than being forwarded on to possibly undesirable destinations which is the “benefit” of this feature. An unfortunate side effect is that if there is no other parent route, the default route will be pointed to a drop index instead. In summary, if there isn’t a need to prevent traffic being routed using parent routes when a more specific route cannot be programmed, it is recommended to disable this feature with the “no” form of the same command.

Sources

• RFC 4271, “A Border Gateway Protocol 4 (BGP-4)”, https://tools.ietf.org/html/rfc4271
• “Use of BGP for routing in large-scale data centers”, http://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/
• “Understanding BGP Convergence”, http://blog.ine.com/2010/11/22/understanding-bgp-convergence/

Configuration Excerpt Examples

Configuration examples are from a small-scale switch fabric created using virtual Extensible Operating System (vEOS) version 4.15.0F.

Per-Rack Leaf AS (Option 1)

SPINE1

interface Ethernet1
no switchport
ip address 192.168.255.0/31
!
interface Ethernet2
no switchport
ip address 192.168.255.2/31
!
interface Ethernet3
no switchport
ip address 192.168.255.4/31
!
interface Loopback0
ip address 192.168.254.1/32
!
ip routing
no ip hardware fib route unprogrammed parent-drop
!
router bgp 64600
router-id 192.168.254.1
update wait-for-convergence
update wait-install
bgp log-neighbor-changes
distance bgp 20 200 200
maximum-paths 32 ecmp 32
bgp listen range 192.168.255.0/30 peer-group ARISTA remote-as 65000
bgp listen range 192.168.255.4/31 peer-group ARISTA remote-as 65001
neighbor ARISTA peer-group
neighbor ARISTA fall-over bfd
neighbor ARISTA password p4ssw0rd
network 192.168.254.1/32
aggregate-address 192.168.255.0/28 summary-only

LEAF1

vlan 10
name HOST_10
!
vlan 4093
name BGP_PEER
trunk group BGP_PEER
!
vlan 4094
name MLAG
trunk group MLAG
!
interface Port-Channel1
switchport mode trunk
switchport trunk group BGP_PEER
switchport trunk group MLAG
!
interface Ethernet5
channel-group 1 mode active
!
interface Ethernet6
no switchport
ip address 192.168.255.1/31
!
interface Ethernet7
no switchport
ip address 192.168.255.17/31
!
interface Loopback0
ip address 192.168.254.3/32
!
interface Vlan10
ip address 192.168.10.2/24
ip virtual-router address 192.168.10.1/24
!
interface Vlan4093
ip address 1.1.1.2/31
!
interface Vlan4094
ip address 1.1.1.0/31
!
ip routing
no ip hardware fib route unprogrammed parent-drop
!
!
route-map BGP_PREPEND permit 10
set as-path prepend 65101
!
router bgp 65000
router-id 192.168.254.3
update wait-install
bgp log-neighbor-changes
distance bgp 20 200 200
maximum-paths 32 ecmp 32
neighbor ARISTA peer-group
neighbor ARISTA remote-as 64600
neighbor ARISTA allowas-in 1
neighbor ARISTA fall-over bfd
neighbor ARISTA route-map BGP_PREPEND out
neighbor ARISTA password p4ssw0rd
neighbor 1.1.1.3 peer-group ARISTA
neighbor 1.1.1.3 remote-as 65000
neighbor 1.1.1.3 next-hop-self
neighbor 192.168.255.0 peer-group ARISTA
neighbor 192.168.255.16 peer-group ARISTA
network 192.168.10.0/24
network 192.168.254.3/32

Leaf Layer AS (Option 2)

SPINE1

interface Ethernet1
no switchport
ip address 192.168.255.0/31
!
interface Ethernet2
no switchport
ip address 192.168.255.2/31
!
interface Ethernet3
no switchport
ip address 192.168.255.4/31
!
interface Loopback0
ip address 192.168.254.1/32
!
ip routing
no ip hardware fib route unprogrammed parent-drop
!
router bgp 64600
router-id 192.168.254.1
update wait-for-convergence
update wait-install
bgp log-neighbor-changes
distance bgp 20 200 200
maximum-paths 32 ecmp 32
bgp listen range 192.168.255.0/24 peer-group ARISTA remote-as 65000
neighbor ARISTA peer-group
neighbor ARISTA fall-over bfd
neighbor ARISTA password p4ssw0rd
neighbor ARISTA maximum-routes 12000
network 192.168.254.1/32
aggregate-address 192.168.255.0/28 summary-only

LEAF1

vlan 10
name HOST_10
!
vlan 4093
name BGP_PEER
trunk group BGP_PEER
!
interface Port-Channel1
switchport mode trunk
switchport trunk group BGP_PEER
switchport trunk group MLAG
!
interface Ethernet5
channel-group 1 mode active
!
interface Ethernet6
no switchport
ip address 192.168.255.1/31
!
interface Ethernet7
no switchport
ip address 192.168.255.17/31
!
interface Loopback0
ip address 192.168.254.3/32
!
interface Vlan10
ip address 192.168.10.2/24
ip virtual-router address 192.168.10.1
!
interface Vlan4093
ip address 1.1.1.2/31
!
ip routing
no ip hardware fib route unprogrammed parent-drop
!
route-map BGP_PREPEND permit 10
set as-path prepend 65101
!
router bgp 65000
router-id 192.168.254.3
update wait-install
bgp log-neighbor-changes
distance bgp 20 200 200
maximum-paths 32 ecmp 32
neighbor ARISTA peer-group
neighbor ARISTA remote-as 64600
neighbor ARISTA allowas-in 1
neighbor ARISTA fall-over bfd
neighbor ARISTA route-map BGP_PREPEND out
neighbor ARISTA password p4ssw0rd
neighbor 1.1.1.3 peer-group ARISTA
neighbor 1.1.1.3 remote-as 65000
neighbor 1.1.1.3 next-hop-self
neighbor 192.168.255.0 peer-group ARISTA
neighbor 192.168.255.16 peer-group ARISTA
network 192.168.10.0/24
network 192.168.254.3/32

SPAN Destination ports and VLAN Membership

Recently at work, a discussion sprouted up around how to handle/configure local session Switched Port Analyzer (SPAN) destination ports.  A suggestion was made to create a new VLAN just for these SPAN destination ports and place them there.  The justification was that they would be out of VLAN 1, and easily identifiable.  Personally I thought it was a waste of a VLAN for a few simple SPAN destination ports, as SPAN destination ports do not participate in spanning tree, and do not forward traffic.  However, ultimately in this case it was a good decision due to security requirements.
Some key characteristics to know about SPAN destination ports:

• A destination port can be a physical port that is assigned to an EtherChannel group, even if the EtherChannel group has been specified as a SPAN source. The port is removed from the group while it is configured as a SPAN destination port.
  
• The port does not transmit any traffic except that traffic required for the SPAN session unless learning is enabled.
  
• The state of the destination port is up/down by design. The interface shows the port in this state in order to make it evident that the port is currently not usable as a production port.
  
• If ingress traffic forwarding is enabled for a network security device. The destination port forwards traffic at Layer 2.
  
• A destination port does not participate in spanning tree while the SPAN session is active.
  
• When it is a destination port, it does not participate in any of the Layer 2 protocols (STP, VTP, CDP, DTP, PagP).
  
• A destination port that belongs to a source VLAN of any SPAN session is excluded from the source list and is not monitored.
  
• A destination port receives copies of sent and received traffic for all monitored source ports. If a destination port is oversubscribed, it can become congested. This congestion can affect traffic forwarding on one or more of the source ports.
  
• When you configure a port as a SPAN destination, it is dedicated for use only by the SPAN feature.
  
• Destinations, by default, cannot receive any traffic. With Release 12.2(33)SXH and later releases, you can configure Layer 2 destinations to receive traffic from any attached devices.
  
• Destinations, by default, do not transmit anything except SPAN traffic. Layer 2 destinations that you have configured to receive traffic can be configured to learn the Layer 2 address of any devices attached to the destination and transmit traffic that is addressed to the devices.

Sources:
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml#charac_dest
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.html#wp1020380
Some of the points above seem like repeats and that was done on purpose to emphasize the point.  Probably the most important point is the one about the port being in an Up/Down status.  To me, this would naturally indicate that it is doing nothing outside of its SPAN function.

Spanning Tree Exercise and Revisiting Root Guard

This was actually spurned from a comment I received on another one of my blog posts that you can find here.  Seeing that comment, I white boarded it and realized that I may have been completely wrong in regards to how Root Guard could “break a network”. 

Let’s say we have the following topology:

• Core 1 is the root for VLAN 10 with a configured priority of 4096, and is the secondary root for VLAN 20 with a configured priority of 8192.  We alternate this with Core 2 in order to load balance VLAN traffic.
• Access 3 and 4 are left in default configuration regarding spanning tree.
• Two workstations are present – one in VLAN 10, and another in VLAN 20.  Their default gateways are SVIs that are on the Core switches.
• For simplicity, switch MAC addresses are the number contained in their names.  Example: Access 4’s MAC address is “4”.
• All link costs are the same.
• All links between switches are trunks transporting all VLANs.

Let’s work through the spanning tree topologies.

Core 1 – Root bridge for VLAN 10.  All ports designated.

Core 2 – Port 1 will be a root port due to lowest cost to root. Since the cost to reach root for Core 2, Access 3, and Access 4 are equal, and a root port exists, ports 2 and 3 will be designated due to Core 2 having the lowest bridge ID (STP tie-breaker).  We know this by calculating the bridge IDs:

• Core 2  - MAC Address (2) + VLAN ID (10) + Priority (8192) = 8204
• Access 3 – MAC Address (3) + VLAN ID (10) + Priority (32768) = 32781
• Access 4 – You get the idea.

Access 3 and 4 – Port 1 will be the root port leaving port 2 as blocking since there are already designated ports on those links belonging to Core 2.

VLAN 10	Port 1	Port 2	Port 3
Core 1	Designated	Designated	Designated
Core 2	Root	Designated	Designated
Access 3	Root	Blocking	N/A
Access 4	Root	Blocking	N/A

And for VLAN 20:

VLAN 20	Port 1	Port 2	Port 3
Core 1	Root	Designated	Designated
Core 2	Designated	Designated	Designated
Access 3	Blocking	Root	N/A
Access 4	Blocking	Root	N/A

Now we can see how traffic will flow from each workstation to each other.

VLAN 10 Workstation to VLAN 20 Workstation:

1. Traffic leaves VLAN 10 Workstation and hits VLAN 10 SVI on Core 1
2. Multilayer switched to VLAN 20 within Core 1.
3. Hit Core 2 since the link directly to Access 4 is blocked on VLAN 20.
4. Traverse Access 4 to reach VLAN 20 Workstation.

And now we move to the million dollar question from my previous blog article.  Let’s say we have Root Guard configured on ports 2 and 3 on both Core 1 and Core 2.  What happens when we lose the link between the Core switches? First it may warrant to discuss what happens without Root Guard in place:

Core 1 – Root bridge for VLAN 10 – all remaining ports remain designated.

Core 2 – Since the cost to root is the same going through either Access 3 or Access 4, port 2 will transition to a root port due to, again, Access 3 having the lowest bridge ID. Port 3 will transition to a blocking state since Access 4 now has the lowest cost to reach root for that link.

Access 3 – Port 1 remains a root port.  Port 2 transitions to a designated port since Access 3 has the lowest cost to reach root on that link.

Access 4 – Port 1 remains a root port.  Port 2 transitions to a designated port since Access 4 has the lowest cost to reach root on that link.

VLAN 10	Port 1	Port 2	Port 3
Core 1	Down	Designated	Designated
Core 2	Down	Root	Blocking
Access 3	Root	Designated	N/A
Access 4	Root	Designated	N/A

And for VLAN 20:

VLAN 20	Port 1	Port 2	Port 3
Core 1	Down	Root	Blocking
Core 2	Down	Designated	Designated
Access 3	Designated	Root	N/A
Access 4	Designated	Root	N/A

This has some pretty drastic results:

The path for the blue Workstation to the red Workstation is pretty straight-forward.  But check out the path from the red Workstation to the blue one:

1. Traffic from the VLAN 10 Workstation hits the VLAN 10 SVI on Core 1 as usual.
2. Multilayer switched to VLAN 20 within Core 1, again, as usual.
3. Here’s where it gets interesting.  The link between Core 1 and Core 2 is physically down, and STP is blocking on port 3 for VLAN 20, so this traffic must go back down to Access 3.
4. Traffic traverses Core 2 and Access 4 before finally reaching its destination.

Now let’s see what happens when we enable Root Guard on Core 1 and Core 2’s ports 2 and 3:

• Core 1 begins receiving superior BPDUs on VLAN 20 on ports 2 and 3, so it places those ports in root-inconsistent.
• Core 2 begins receiving superior BPDUs on VLAN 10 on ports 2 and 3, so it places those ports in root-inconsistent.

VLAN 10	Port 1	Port 2	Port 3
Core 1	Down	Designated	Designated
Core 2	Down	Root-Inconsistent	Root-Inconsistent
Access 3	Root	Designated	N/A
Access 4	Root	Designated	N/A

 

VLAN 20	Port 1	Port 2	Port 3
Core 1	Down	Root-Inconsistent	Root-Inconsistent
Core 2	Down	Designated	Designated
Access 3	Designated	Root	N/A
Access 4	Designated	Root	N/A

Now lets take a look at traffic flow from the red Workstation to the blue:

1. Traffic leaves the VLAN 10 Workstation and hits the VLAN 10 SVI on Core 1
2. Multilayer switched to VLAN 20 within Core 1
3. Uh oh – port 1 is physically down and ports 2 and 3 are down for VLAN 20 thanks to Root Guard.  The traffic has no where to go!

Welp – guess I wasn’t wrong after all.  This appears to be a corner case scenario because if it were only a single instance of spanning tree there would be no issue.  Even in this scenario, intra-VLAN communication would be fine, but this does break inter-VLAN communication.

OSPF Adjacency Building Process

Ever curious regarding how two routers configured for OSPF become fully adjacent?  The following diagram of the process was modeled directly from RFC 2328, and the steps described gleaned from the Routing TCP/IP Vol I book.  Since we can see mention of a DR, this example must be based on a multi-access network.

1. RT1 becomes active and sends a Hello.  At this point, RT1 hasn’t seen any neighbors, so it reports such and sets its DR and BDR fields to 0.0.0.0.
2. Upon receipt by RT2, RT2 will build a data structure for RT1 and set RT1’s state to Init.  RT2 will then send a Hello packet reporting that it has seen RT1, and will report itself as the DR.
3. RT1 now sees its own RID in the received Hello packet from RT2, so RT1 will now create a data structure for RT2 and set its state to ExStart.  RT1 then begins Master/Slave negotiation with a DD packet with a sequence number of “x”, the Init bit set  to indicate that it is the start of an exchange, the More bit set to indicate that it is not the last DD packet to be sent, and lastly with the MS-bit set to indicate that RT1 wants to be the Master.  No LSA summaries are provided just yet – this is purely for Master/Slave negotiation.
4. Upon receipt of RT1’s DD packet, RT2 sets RT1’s state to ExStart.  RT2 then sends a DD with a sequence number of “y”, and the MS-bit set since RT2 has a higher RID in this example.
5. RT1 agrees that RT2 is the Master and sets its state to Exchange. RT1 informs RT2 of its agreement by sending a DD packet with a sequence number “y” that matches what was sent from RT2 in step 4.  The MS-bit will now be set to 0, and here is where we’ll start to see LSA summaries provided by RT1.
6. RT2 now sets RT1’s state to Exchange as well and sends a DD packet containing LSA headers from its Link State Summary list and increments the DD sequence number to “y+1”.
7. RT1 acknowledges this by sending an acknowledgement with the same sequence number that it received in the DD packet from RT2.  It also begins taking the LSA summaries it received and building a Link State Request list.  This process repeats until RT2 sends the last of its LSA summaries and sets the More bit to 0.
8. Once RT1 receives the last of RT2s LSA summaries and acknowledges it knowing that it has sent the last of its own LSA summaries, the Exchange state process is complete.  Now RT1 will begin processing the LSA summaries it has added to its Link State Request list and transition to the Loading state.
9. Once RT2 receives RT1’s last DD packet, RT2 then sets RT1s’s state to Full because RT2 has no entries in its own Link State Request list.
10. RT1 will send Link State Requests for all entries in its Link State Request list and RT2 will send Link Sate Update packets until RT1’s list is empty.  At that point, it sets RT2 to Full, and the adjacency is complete.

OSPF Link State Advertisements (LSAs) and Areas – Part II

For a table describing the different LSA types, check out the first post of this series.

In the first part of the series, we looked at LSA Types 1, 2, and 3 – Router, Network, and Network Summary, respectively.  To move on to the next two LSA types, we need to bring in another Autonomous System (AS).  In the diagram below, we’ve added R5 which has an interface in EIGRP AS 1, and is redistributing that into OSPF Area 4.  The fact that R5 has an interface inside of the OSPF AS, as well as the EIGRP AS, makes R5 an Autonomous System Boundary Router (ASBR). 

The EIGRP-oriented subnet that is being redistributed is considered an external route to the OSPF domain, so a Type 5 LSA, or ASBR External, is flooded into OSPF Area 4 containing a LSID and netmask of the subnet, plus the External Type. This important because it tells other routers whether or not to add the internal link costs within the OSPF domain to the metric to reach that subnet.  A type 2 external route specifies that only the external cost is taken into consideration.

When R2 catches wind of this, it will generate a Type 4 LSA, or ASBR Summary, and flood it into Area 0 as well as the Type 5 LSA.  The contents of the Type 4 LSA are mostly just the LSID of the ASBR’s RID and the metric reach it.

For  other routers to reach this External Type 2 route, they just take metric provided in the Type 5 LSA (20) – no questions asked.

However if it were an External Type 1 route, each router must determine its cost to reach it taking into account the link costs along the way.

So in this example, if R1 needed to reach 172.16.5.0/24 as an External Type 1 route, it would first add the cost to reach R2 (The ABR) [Similar to what it would do to determine cost to an intra-area (IA) route utilizing a Type 3 LSA] to the cost to reach the ASBR as listed in the Type 4 LSA.  Finally, it would then add the cost listed in the Type 5 LSA for that subnet.

Cost to reach ABR (1) + Cost to reach ASBR (2) + Cost in Type 5 LSA (20) = 23

Stubby.. Totally Stubby… Not-So-Stubby… Totally Not-So-Stubby….

While there are several iterations of a stubby area, with varying mechanics behind them, they all serve mostly a single purpose – to reduce overhead.

  

Once R6 is added and area 6 is configured as a stubby area, two major things happen:

• R3 (An ABR for area 6) will still receive, but stop flooding Type 5 LSAs into area 6.  Notice that R6’s routing table does not include R5’s EIGRP-oriented subnet (172.16.5.0/24)
• R3 injects a default route into area 6

So we can see here that the logic is, hey, there’s only going to be one way to get out of this area, so let’s reduce the size of the LSDB and make things easier by just injecting a default route into the area and calling it a day.  Imagine if R5 had 100 subnets hanging off of it in the EIGRP AS – if R6 wasn’t in a stubby area, those 100 routes would be added to its LSDB and in the routing table.  Why do that if we can just inject a default route and not advertise these external routes into the area – effectively accomplishing the same thing?  If R6 needs to hit any of these external routes, it just uses the default route to the ABR who knows how to get there, and greatly reduce the size of R6’s LSDB and routing table all at the same time!

A “totally” stubby area takes this concept even further by also not flooding Type 3 LSAs.  What’s kind of interesting about that statement though is that the default route R3 injects into the stubby area is a Type 3 LSA.

A Not-So-Stubby Area (NSSA) simply allows you to redistribute external routes if needed.  A special type of LSA is used in this situation – a Type 7, or NSSA External.

This will be generated by the now ASBR, R6, who is redistributing its connected 192.168.6.0/24 subnet.  As depicted above, once it reaches the ABR, R3, it will convert this LSA into a Type 5 before flooding it into area 0, so you will only see Type 7 LSAs in a NSSA.

 

 

Exploring OSPF Messages in a Multi-access Network

 

The following network is configured with OSPF with all interfaces in area 0.  Since this is a multi-access network, a Designated Router (DR) is elected which improves OSPF performance by reducing the amount of LSA flooding. R3 is the current DR, with R2 as the BDR.  R4’s interface to SW1 has been configured as a passive interface to prevent an adjacency from forming and simulate R4 being a “new” router on the network.  Wireshark is monitoring the link between R4 and SW1.

I won’t go into all the details regarding Wireshark output and the OSPF process.  If you want a more detailed analysis, take a look at my previous blog article here.  In this article, we’ll only be taking a closer look at what happens specifically in a multi-access environment.

Upon re-enabling R4’s interface for OSPF, we see R4 sends a Hello packet to the All OSPF Routers multicast address (224.0.0.5) and that no DR or BDR is listed.  R4 is “new” to the network as far as OSPF is concerned, so it has no idea about the current topology.

R1, R2, and R3 all send Hello packets with the routers they have seen, which now includes the newly added R4. It also provides information regarding the established DR and BDR.  Since there is already an established DR/BDR, R4 will not attempt to participate in an election or usurp the current DR/BDR.  Also note that this Hello packet, and the communication that follows afterward between R4 and routers R1, R2, and R3 is unicast – not multicast.

What follows next are the usual Database Descriptor (DD) packets utilized to establish Master/Slave relationships, followed by the exchange of summarized LSAs.  Notice that this communication is only occurring between R4 and routers R2 and R3 – the BDR and DR, respectively.  Communication between R4 and R1 stops after they have reached the 2-way state, since R1’s role in this environment is DROther (Not the DR or BDR).  

Of interest in this scenario is the Network LSA provided by R2 and R3 in addition to the usual Router LSAs.  This is a type 2 LSA that represents a subnet where a DR has been elected.  Notice that the advertising router is the DR, and the  Link State ID is the DR’s IP address on that subnet.  An interesting note is that the SPF process treats this as an individual node in its “mathematical model”, so this type of LSA is sometimes referred to as a pseudonode.

Since R4 will attempt to establish full adjacency with R2 and R3, R4 follows the usual process of requesting full LSAs from R2 and R3 for any LSAs it does not already have or that it may have but are not up to date.  In addition, R2 and R3 will request at least R4’s Router LSA since they will obviously not have it. Once this process completes, R3 as a DR will send LSA Updates to the All OSPF Routers multicast address. In the picture below, R3 advertises R4’s Router LSA.  This will in turn update R1 – the DR acts as a relay of sorts for updates in the OSPF environment.

R4 will acknowledge the update utilizing the unicast IP address of the DR, mirroring the LS Sequence number.  R4 has an updated LSA however, so it reports this to the DR utilizing the All OSPF DR Routers multicast address, 224.0.0.6. Notice the updated LS Sequence number.

R3 then provides the updated LSA to the rest of the world, acknowledgements are sent back, and everything settles into place.

Looking at the capture, there are two points I’m a little confused on. 

In one packet, R4 sends an acknowledgement using 224.0.0.6, and there are two Network LSAs contained within. Why are there two of the same network LSAs with different LS Seq numbers in the same LS Acknowledge?

Another is that I see LS Acknowledgements sourced from R4 to the unicast address of the DR, as well as to the multicast address 224.0.0.6.  What determines when an LS Acknowledgement is sent to the DR unicast address or to the multicast address?

To be continued….

OSPF Link State Advertisements (LSAs) and Areas – Part I

If every router in an enterprise environment was in a single OSPF area, at some point you’re going to encounter scalability issues due to any changes in the environment causing an SPF recalculation in all routers in that single area.

LSAs and their use within areas provide a mechanism for maximizing performance in OSPF by logically segmenting groups of contiguous links so that every router in the entire autonomous system does not have to have exact copies of the Link State Database (LSDB) and to reduce the amount of LSA flooding.  SPF calculations are also isolated to each individual area rather than the entire environment.  Different LSAs are used in different situations, and are treated differently depending on the type of OSPF area involved.

The following table represents the different LSA types, and was taken from the CCIE R&S OCG.

TYPE	NAME	DESCRIPTION
1	Router	One per router containing its RID and all interface IP addresses; also represents stub networks.
2	Network	One per transit network.  Created by the DR and represents the subnet and router interfaces connected in the subnet.
3	Network Summary	Created by Area Border Routers (ABRs) to represent one area’s type 1 and 2 LSAs when being advertised into another area.  This also defines the links (subnets) in the origin area, and cost, but no topology data.
4	ASBR Summary	Similar to a type 3 LSA, but it advertises a host route to reach an Autonomous System Boundary Router (ASBR)
5	AS External	Created by ASBRs for external routes injected into OSPF
6	Group Membership	Used for Multicast Open Shortest Path First (MOSPF).  This is an extension to OSPF that allows multicast routing, allowing routers to share information about group memberships.  This is not supported by Cisco.
7	NSSA External	Created by ASBRs inside of a NSSA area instead of a type 5 LSA.
8	External Attributes	Not implemented in Cisco routers.
9-11	Opaque	Used as generic LSAs to allow for easy future extension of OSPF; as an example, type 10 has been adapted for MPLS traffic engineering.

The diagram below depicts a simple point-to-point network with all interfaces within the same area.

In this situation, there will only be Type 1 LSAs exchanged between routers.  The LSA will contain the RID of the advertising router, connected links (subnets), and the RIDs of any other routers it has seen. Below is Wireshark output taken from a different network than the one diagrammed above. The link types seen here are:

• PTP – Point-to-point connection
• Stub – a subnet on which a router has not formed any neighbor relationships
  • *QUESTION* The CCIE R&S OCG defines a stub network this way, but Wireshark packet capture lists connected subnets that HAVE formed neighbor relationships as stubs… what gives?  The entries that contain the RIDs of the other routers are listed as PTP and the loopback of the advertising router is listed as a Stub
    • *ANSWER* SPF treats these two “stub” entries as  separate data structures from the PTP entries that actually specify the connected neighboring router.  The more important of the two entries is the PTP entry. I don’t know exactly what that means, but I’ll push the “I Believe” button for now.

When we talk about a multi-access environment, a DR is elected and here is where we can expect to see a Type 2 LSA – a Network LSA. 

 

This type of LSA is  generated by the DR and describes the subnet as well as the RIDs of the DR’s connected neighbors.  The Link State ID will be the interface IP address of the DR within that subnet.  It may seem odd that it lists itself as an attached router.  This is because this LSA is treated as a pseudonode – not an actual physical device, but a logical node on the SPF mathematical model.  So think of it as a separate entity from the actual physical router that is the acting DR.

You will find Type 1 and 2 LSAs inside of a single area.  These are the most computation-intensive LSA types and they do not traverse areas.  When you bring in multiple areas, this is where you encounter a Type 3 – a Network Summary LSA.  Since R2 now has interfaces in two areas – area 0 and 4, it is now an Area Border Router (ABR).

This type of LSA is created by ABRs to represent subnets that the ABR can reach.  Notice that it not only contains its directly connected subnets in area 4, but also the route it learned from R4.  The nice thing about this is that a router, for example R3, doesn’t have to run the SPF algorithm for these routes.  All it does is add its cost to reach the ABR, to the metric value gleaned from the LSA from the ABR.  So in this case, the cost for R3 to reach 192.168.4.0 would be 3 (1 for the cost of R3’s fa0/0 interface, plus 2 for the metric advertised in the Type 3 LSA).

If you haven’t picked up on it already, an interesting note is that this is distance vector behavior because R3 in this example depends on R2 to tell it the metrics to reach routes in area 4 instead of R3 determining the cost on its own.  So OSPF is a link state routing protocol, but when it comes to inter-area operations it uses a distance vector algorithm.  Pretty cool.  Something I found out from the TCP/IP Routing Vol I book is that this is the reason why there is a backbone area and that all areas must be connected to it.  This essentially creates a hub-and-spoke topology and prevents the route-loops that distance vector protocols are so vulnerable to.

Continue to Part II.