Quality of Service (QoS) – Policing and Shaping Notes

Policers and shapers identify traffic violations in an identical manner, but treat them differently.  Policers perform instantaneous checks and immediately take action when a violation occurs.  Actions can include marking, dropping, and even just transmitting the packet.  Shapers on the other hand are traffic-smoothing tools.  Its objective is to send all traffic out a given interface, but to smooth it out so that it never exceeds a given rate – usually in order to meet SLAs.  Excess traffic is buffered and delayed until the traffic once again dips below the defined maximum rate.

Policer	Shaper
Causes TCP resends as traffic is dropped	Delays traffic; involves less TCP resends
Inflexible; makes instant drop decisions	Adapts to network congestion by queuing excess traffic
Ingress or egress interface tool	Typically egress only
Rate limiting – no buffering	Rate limiting with buffering

While policing and shaping tools are not employed to directly provide QoS for real-time traffic, they do regulate/stabilize traffic flows so that unexpected bursts in data traffic do not induce jitter and latency that adversely affects real-time traffic.

Policers determine whether each packet conforms, exceeds, or violates the policies configured for traffic, and takes the prescribed action in each case.

• Conforming – traffic that falls within the rate configured for the policer
• Exceeding – traffic that is above the policer rate, but still within the burst parameters
• Violating – traffic that is above both the policer rate and burst parameters

It is not productive to police voice traffic or call-signaling traffic because the incoming rates of these traffic types should be controlled at their origin by call admission control (CAC) mechanisms.

You can also use a policer as a marker to re-mark traffic upon an exceed and/or violate action rather than just drop it.

Although a policer can be deployed ingress or egress, it is typically deployed at the network edge on traffic ingress.  If packets will be dropped, there is little point in spending CPU cycles routing these packets.  Policers are also often deployed at the traffic egress interface to control bandwidth used or allocated to a particular class of traffic.

As mentioned earlier, shapers are similar to policers in that they also limit the transmission rate of packets but they do so by delaying packets that exceed the CIR.  This allows for conformance to SLAs.  Shaping is crucial for non-broadcast multi-access (NBMA) topologies such as ATM and Frame Relay, or potentially anywhere else where a speed mismatch may exist.  Examples of this would be line speed mismatches, aggregated traffic oversubscription, and SLA enforcement by a carrier.

EtherChannel – Quick and Dirty

EtherChannel allows you to aggregate several switch links into a single, fast, fault-tolerant, logical interface. 16 links can be defined for an EtherChannel, however, a maximum of 8 will be active at any one time.  The other links are placed on standby.

While having multiple links between two switches can possibly create bridging loops, EtherChannel avoids this by bundling the links into a single logical interface.  This logical interface can be configured as an access or trunk interface.

For ports to be members of the same EtherChannel, there are some restrictions. Ports must:

• Belong to the same VLAN
• Have identical STP settings
• Have identical speed/duplex settings
• Note: In addition, if the EtherChannel is to be used as a trunking interface, all ports must be in trunking mode, have the same native VLAN, and pass the same set of VLANs.

The full duplex maximum bandwidth for 8 links is as follows:

• Fast EtherChannel (FEC): 1600 Mbps
• Gigabit EtherChannel (GEC): 16Gbps
• 10-Gigabit EtherChannel (10GEC): 160Gbps
• Note:  This is theoretical; maximum bandwidth is not likely to be achieved due to unequal load balancing, and other factors.

Load Balancing

 

EtherChannel load balancing across the links can occur in a number of configurable methods for optimization in your environment. IP addresses, MAC addresses, and TCP/UDP port numbers can be leveraged. The complete list is:

• Source IP (src-ip)
• Destination IP (dst-ip)
• Source and Destination IP (src-dst-ip)
• Source MAC (src-mac)
• Destination MAC (dst-mac)
• Source and Destination MAC (src-dst-mac)
• Source Port (src-port)
• Destination Port (dst-port)
• Source and Destination Port (src-dst-port)

When more than one item is utilized in the load balancing method,  an XOR operation occurs, and for 2 links, the last bit is utilized.  Four links uses the last two bits, and eight links use the last three.  Below shows two switches with an EtherChannel with four links, configured to use the Source and Destination IP (src-dst-ip) method of load balancing.  The four different examples show how the links are used as different devices communicate across the two switches.

 

For best results, it is recommended to consider using MAC addresses or the Source IP address as your load-balancing method.  However, this all depends on your environment. For example, a router always uses it’s burned-in MAC address, so the destination MAC address remains the same for all frames destined through the router.  When two routers are forwarding traffic to each other, MAC addresses remain constant, so only one link is used.  Using IP addresses as the load-balancing method instead may be a better idea.  If most of the traffic is between the same two IP addresses, use IP port numbers instead.

If EtherChannel traffic consists of non-IP traffic, distribution according to MAC address is recommended.

If a frame can’t meet load-balancing criteria, switch reverts to “next lowest” method. For instance is MAC traffic is sent across an EtherChannel that’s configured to load-balance by IP addressing, MAC addresses will be used instead.

To prevent loops, inbound (received) broadcasts and multicasts are not sent back out any of the links.  Outbound frames are load-balanced normally.

 

EtherChannel Negotiation: PAgP vs. LACP

 

There are two EtherChannel negotiation protocols.  Port Aggregation Protocol (PAgP) is a Cisco-proprietary protocol, while Link Aggregation Control Protocol (LACP) is standards based.

PAgP dynamically modifies the EtherChannel if one of the ports’ VLAN, speed, etc. is changed so that all of the links in the EtherChannel match. PAgP can be configured in active mode (desirable), which actively attempts negotiation.  Passive mode (auto, the default) only negotiates an EtherChannel if the far end initiates it.

LACP assigns roles to end points.  The switch with the lowest system priority makes decisions about what ports will participate in the EtherChannel at any given time.  If you’re familiar with STP, this is similar to the way the Root Bridge is elected.  Ports are selected and become active in the EtherChannel according to their port priority. LACP Active mode (active) – actively negotiates, while passive mode (passive) negotiates only if the far end initiates it.

Lastly, the “on” mode forces the EtherChannel to be formed; no PAgP/LACP negotiation occurs when this mode is utilized.

Here’s a configuration Video:

PAgP EtherChannel Configuration

Brocade Auth-Change-Wait-Time

 

The other day I was at work doing an interoperability test with Cisco and Brocade multilayer switches, and we ran into a strange issue that really highlighted my “tunnel view” to the Cisco world.

We were setting up basic OSPF stuff using md5 authentication and we couldn’t get the Cisco and Brocade to form an adjacency.  A debug ip ospf adjacency command on the Cisco switch revealed that the Cisco was using “type 2” authentication, and the Brocade was using “type 0”. 

Here’s a quick breakdown of the authentication types:

Type 0	No authentication
Type 1	Clear text authentication
Type 2	md5 authentication

I set up a SPAN on the Cisco switch and sure enough, we were getting the OSPF Hello packets from the Brocade with no authentication.

After some digging, it turns out the Brocade has an Auth-Change-Wait-Time command in interface configuration mode.  This is set to 300 seconds (5 minutes) by default.  While I don’t quite understand it, the description states it allows for graceful authentication implementation.  So after you enable md5 on the interface, it waits 300 seconds before actually sending OSPF Hellos with authentication.  We toyed around with it and took a packet capture to confirm the behavior, and then set it to 0 to immediately start sending packets with authentication and we were good to go.

Here’s a screenshot of the behavior in Wireshark with the parameter set to 20 seconds.  You’ll see the OSPF adjacency start forming at almost exactly 20 seconds.

OSPF LSA Manipulation Vulnerability – 8/1/2013

Vulnerability Details

OSPF LSA Manipulation Vulnerability in Multiple Cisco Products

· Summary

Multiple Cisco products are affected by a vulnerability involving the Open Shortest Path First (OSPF) Routing Protocol Link State Advertisement (LSA) database. This vulnerability could allow an unauthenticated attacker to take full control of the OSPF Autonomous System (AS) domain routing table, blackhole traffic, and intercept traffic.
The attacker could trigger this vulnerability by injecting crafted OSPF packets. Successful exploitation could cause flushing of the routing table on a targeted router, as well as propagation of the crafted OSPF LSA type 1 update throughout the OSPF AS domain.
To exploit this vulnerability, an attacker must accurately determine certain parameters within the LSA database on the target router. This vulnerability can only be triggered by sending crafted unicast or multicast LSA type 1 packets. No other LSA type packets can trigger this vulnerability.
OSPFv3 is not affected by this vulnerability. Fabric Shortest Path First (FSPF) protocol is not affected by this vulnerability.

· Affected Products

Cisco devices that are running Cisco IOS Software and configured for OSPF are vulnerable. Devices that do not have OSPF enabled are not affected by this vulnerability.

Cisco devices that are running Cisco IOS XE Software and configured for OSPF are vulnerable. Devices that do not have OSPF enabled are not affected by this vulnerability.

The version of Cisco IOS-XE Software that is running on a Cisco device can be determined using the show version command from the Command Line Interface (CLI).

· Workarounds

The use of OSPF authentication is a valid workaround. OSPF packets without a valid key will not be processed. MD5 authentication is highly recommended, due to inherent weaknesses in plain text authentication. With plain text authentication, the authentication key will be sent unencrypted over the network, which can allow an attacker on a local network segment to capture the key by sniffing packets.
Refer to http://www.cisco.com/en/US/tech/tk365/technologies_configuration_example09186a0080094069.shtml for more information about OSPF authentication.
Additionally, an OSPF Time To Live (TTL) security check can be applied as a partial workaround.
Note: This workaround is valid to protect against remotely triggered attacks and does not protect against attackers that are layer 2-adjacent to vulnerable devices.
For more information about general Interior Gateway Protocol (IGP) hardening, refer tohttp://www.cisco.com/en/US/tech/tk365/technologies_configuration_example09186a0080094069.shtml.

Additional mitigations that can be deployed on Cisco devices within the network are available in the Cisco Applied Mitigation Bulletin companion document for this advisory, which is available at the following link:http://tools.cisco.com/security/center/viewAMBAlert.x?alertId=29974

Quality of Service (QoS) Congestion-Avoidance Notes

Congestion-avoidance tools are complementary to, and dependent upon, queuing algorithms. Queuing/scheduling algorithms manage the front of a queue, while congestion-avoidance mechanisms manage the tail of a queue.

Congestion-avoidance tools are designed for TCP traffic, because TCP has built-in flow-control mechanisms that operate by gradually increasing traffic flows until packet loss has occurred.  Once packet loss has occurred, the transmission rate is reduced before slowly ramping up again.  This means that if no mechanism is in place to control TCP, any particular flow has the ability to eat up all available bandwidth.

When there are no congestion-avoidance tools in place, and queues fill, tail drop occurs, which means all traffic is dropped. 

In a constricted channel without congestion-avoidance tools, TCP connections eventually synchronize with each other – they ramp up together, lose packets together, and back off together.  This is called global synchronization and basically results in “waves” of TCP traffic.

Congestion-avoidance tools has no real benefit or use for UDP traffic, because UDP traffic does not have any retry logic.

Random Early Detection (RED)

RED combats global synchronization by preemptively and randomly dropping packets before queues fill.  Instead of waiting for the queues to fill, RED causes the router to monitor the buffer depth and perform early drops on random packets when the defined minimum queue threshold has been exceeded.

RED drops occur within the bounds of TCP retry timers, which slows the transmission rate of sessions but prevents them from starting slow.  This optimizes network throughput.

It should be noted that Cisco IOS doesn’t support RED, only Weighted RED (WRED).  When you utilize the random-detect command in a queue, it actually enables WRED.  However, if there are no separate IPP or DSCP markings within a given class of traffic, then the effective policy is simply RED.

Weighted Random Early Detection (WRED)

WRED is an enhancement to RED that allows you to control how packets are selected to be “randomly” dropped.  A configured minimum threshold determines when packets of a given IPP value begin to be dropped.  The configured maximum threshold determines at what queue depth that all packets of that value will be dropped. The mark probability denominator determines how aggressively that packets of a given IPP value are dropped.  For example, a denominator of 10 means that up to 1 of every 10 packets will be randomly dropped for that IPP value.  The maximum rate of 1 out of every 10 packets being dropped in this example occurs at the configured maximum threshold.  Past the maximum threshold, all packets of that value are dropped (tail drop).

By default, packets with lower IPP values are dropped sooner than packets with higher IPP values. Also, WRED is dependent on queuing, so a queuing option (usually either bandwidth or fair-queue) has to be enabled on the traffic class before you can utilize WRED.

DSCP values can also be used, and this is simply called DSCP-Based WRED.  It pretty much works the same way.  It uses AF drop-preference values (the second digit in the AF code, ex: In “AF21”, the “1”) to determine what packets will be dropped. For example, when WRED is enabled on an interface, packets with a higher drop precedence value, i.e. “AF23” would be dropped more often than those with lower drop precedence values, i.e. “AF21”.

Explicit Congestion Notification (ECN)

Traditionally, the only way to inform sending hosts that there was congestion on the network so they would slow their transmission rates was to drop TCP packets.  ECN was developed to combat this by marking the final 2 bits of the Type of Service (ToS) byte of the IP header.  These two bits are:

• ECN-Capable Transport (ECT) bit – indicates whether ECN is supported on the device
• Congestion Experienced (CE) bit – used in tandem with the ECT bit to signal that congestion was experienced en route.

When congestion occurs WRED drops packets when the configured threshold value is exceeded.  ECN is an extension to WRED, in that ECN marks packets instead of dropping them to communicate the existence of congestion.  Routers configured with the WRED ECN feature (Introduced in IOS 12.2(8)T), use this marking to know that the network is congested.  This allows TCP to be controlled  without dropping packets or at least with dropping fewer packets.

WRED ECN takes the following actions based on the bit settings:

• If the number of packets in a queue are below the configured threshold, packets are transmitted (Normal operation).
• If the number of packets is between the configured minimum and maximum thresholds:
  • If ECT – 1, CE – 0  or ECT – 0, CE – 1 and WRED determines packet should be dropped based on drop probability, the ECT and CE bits are changed to 1 and the packet is transmitted.
  • If ECT and CE bits are 0, this indicates that the sending device is not capable of ECN and the packet then can be dropped based on WRED drop probability.
  • If both ECT and CE bits are set to 1, the packet indicates that there is network congestion, the packet is transmitted and no further marking is required.
• If the number of packets in the queue is above the maximum threshold, all packets are dropped.

Dynamic Buffer Limiting (DBL)

This was actually something I didn’t find out about until we started figuring out how to do QoS on a Catalyst 4500.  I went digging on Cisco’s website and from what I saw initially seemed like it was pretty awesome:

Industry’s First! Cisco innovation! High-speed hardware implementation!  Of course I want more info, so I clicked on Full Story:

Bummer – guess it can’t be found on the Kanye West - I mean cambeywest website.  Even putting in “Dynamic Buffer Limiting” into the search box on Cisco.com came up with nothing. On to Google….

Active Queue Management (AQM), which informs you of congestion before you run into a buffer overflow situation, utilizes DBL to track the queue length for each traffic flow. DBL tracks the queue length for each traffic flow in a switch.  When the queue length exceeds its limit, DBL drops packets or sets the ECN bits in the packet headers.

DBL classifies flows into two categories:

• adaptive – reduce the rate of packet transmission once it receives congestion notification
• aggressive – do not take any corrective action in response to congestion notification

For every active flow, the switch maintains two parameters - “buffersUsed” and “credits”.

Quality of Service (QoS) Congestion Management Notes

Of all the tools within the QoS toolset, congestion management tools, also known as queuing tools, provide the biggest impact on application service levels.  Whenever packets enter a device faster than can exit it, congestion exists and this is where queuing tools come into play.  Queuing tools are only engaged when congestion exists, otherwise packets are sent as soon as they arrive.  When congestion does exist, packets must be buffered, or queued, to mitigate dropping.

Packet markings, or lack thereof, affect queuing policies, so queuing policies are complementary and have a dependence on classification and marking policies.

Scheduling vs. Queuing

These two terms are often incorrectly used interchangeably – they are two different things.  Scheduling determines how a frame or packet exits a device. Whenever packets enter a device faster than they can exit it, as is the case with speed mismatches (ex. Gigabit Ethernet traffic heading to a WAN interface), congestion can occur.  Devices have buffers that allow the temporary storing and subsequent scheduling of these backed-up packets, and this process is called queuing.

Inbound traffic > Queuing (During congestion) > Scheduling > Outbound traffic

• Queuing – orders packets in linked output buffers. Only engaged when there is congestion
• Scheduling – decides which packet to transmit next.  This occurs whether there is congestion or not (Although the scheduling decision is of course much simpler when there is no congestion).

During congestion, the scheduler has to make a decision of what queue to service first based on various types of scheduling logic algorithms:

• Strict Priority – Lower-priority queues are served only if higher-priority queues are completely empty.  This can potentially starve out lower priority queues. Strict priority is good for real-time, delay-sensitive traffic.
• Round-robin – Services queues in a sequence.  Doesn’t have the potential to starve traffic, but may not provide the level of service that delay-sensitive traffic needs that Strict Priority scheduling would be able to provide.
• Weighted-fair – Packets in queues are weighted so that some queues are serviced more frequently than others.  Addresses the cons of strict priority and round-robin, but doesn’t guarantee the bandwidth that real-time flows may require.

Congestion Management vs. Congestion Avoidance

The amount of buffer space (memory) for queues is of course limited.  Once the buffer is overrun, packets may be dropped as they arrive (tail drop), or proactively beforehand.  The selective, proactive dropping of packets is called congestion avoidance.  Congestion avoidance works best with TCP-based applications since the selective dropping causes the TCP windowing mechanism to engage and throttle back the rate of traffic flow to a manageable state.  The relationship between this and congestion management is that the scheduling algorithms of congestion management manage the front of a queue, where with congestion avoidance, the mechanisms manage the tail of a queue.

Legacy L3 Queuing Mechanisms

These are considered legacy, but are what newer mechanisms are built upon:

• Priority queuing (PQ)
• Custom queuing (CQ)
• Weighted Fair Queuing (WFQ)

Newer queuing mechanisms used combinations of these while also attempting to minimize drawbacks, such as:

• Class-based Weighted Fair Queuing (CBWFQ)
• Low latency queuing (LLQ)

Priority Queuing

• Only consists of 4 queues (high, medium, normal/default, low)
• Scheduler empties high queue first before servicing lower queues.
• So, similar to strict priority queuing, handles real-time traffic well but risks starving other queues.

Custom Queuing

• Introduced a round-robin scheduler based on byte counts.
• Prevented bandwidth starvation and introduced bandwidth guarantees
• Supports up to 16 queues
• No capability to provide strict priority

Weighted Fair Queuing

• Built to expand upon principle of fairness that CQ introduced
• Simply divided interface bandwidth by number of flows
• Added a fixed weight based on IPP for bandwidth calculation to favor higher-priority flows, based on that IPP marking
• No ability to provide bandwidth guarantees due to bandwidth allocation changing as flows are added and ended

Currently Recommended L3 Queuing Mechanisms

Enhanced mechanisms were developed to utilize the strengths of the legacy mechanisms while minimizing their weaknesses.

Class-Based Weighted Fair Queuing

• Hybrid queuing algorithm that combines guaranteed bandwidth (from CQ) with the ability to dynamically ensure fairness to other flows within a class of traffic (from WFQ)
• Up to 256 classes of traffic with reserved queues
• Each queue is serviced based on assigned bandwidth
• Minimum bandwidth is explicitly defined and enforced
• Uses Modular QoS CLI (MQC)-based class maps for classification

CBWFQ lacks the ability to provide strict-priority queuing for real-time applications.  To service real-time applications, a strict-priority queue was added to the CBWFQ algorithm, and low-latency queuing (LLQ) was born.

Low Latency Queuing

• Enhanced combination of PQ, CQ, and WFQ.
• Basically CBWFQ with a strict PQ.
• Has a built-in policer to to prevent the strict-priority queue from starving lower-priority traffic
• Only engages when there is congestion, so it is important to provision priority classes properly

Bandwidth Provisioning in LLQ

• General best practice is to provide at least 25% of a link’s bandwidth to class-default
• Limit the sum of all priority class traffic to no more than 33% of a link’s capacity.
• All bandwidth guarantees within LLQ should be no more than 75% link capacity.
• When the percentage-remaining (bandwidth remaining percent) form of LLQ is used, this rule goes out the window because it utilizes a percentage of the remaining bandwidth after the PQ is serviced rather than a set value.

Quality of Service (QoS) Classification and Marking Notes

The first part of building a QoS policy is to identify the traffic that you need to treat preferentially (give better priority), or differentially.  This is accomplished via classification and marking.

• Classification – sorts packets into different traffic types that policies can then be applied to.
• Marking (or re-marking) – establishes a trust boundary on which scheduling tools later utilize.  The edge of the network where markings are either accepted or rejected is known as the trust-boundary.
• Classifier tools – Inspect one or more fields in a packet to identify the type of traffic that is being carried. After being identified, it is passed to the appropriate mechanism to handle that type of traffic class.
• Marking tools – actually write a field within the packet (or frame, cell, label) to preserve the classification decision.  By marking traffic at a trust boundary, subsequent nodes do not have to perform the same in-depth analysis to determine how to treat the packet.

Classification Tools

These tools can examine a number of criteria within layers 1, 2, 3, 4, and 7.

• L1 – Physical interface, subinterface, PVC, port
• L2 – MAC, 802.1Q/p CoS, VLAN, MPLS EXP, ATM Cell Loss Priority (CLP), Frame Relay DE
• L3 – IPP, DSCP, source/dest IP address
• L4 – TCP/UDP Ports
• L7 – Application signatures and URLs in packet headers or payload

Marking Tools

The primary marking tools used currently are class-based marking and marking done via class-based policing.  Legacy marking techniques include committed access rate (CAR) and policy-based routing (PBR).  Voice gateway packet marking is also an option for IPT applications.

• L2 Marking Fields – 802.1Q/p CoS, MPLS EXP, ATM CLP, Frame Relay DE
• L3 Marking Fields – IPP or DSCP

Cisco Catalyst switches perform scheduling based on L2 CoS, however DSCP is the preferred marking method for end-to-end QoS, because L2 marking is lost whenever the L2 media changes.  So it is important to ensure that L2 markings are translated to and from L3 markings consistently throughout the environment for end-to-end QoS.