Integrated Routing and Bridging

In the traditional data center design, inter-subnet forwarding is provided by a centralized router, where traffic traverse across the network to a centralized routing node and back again to its final destination. In a large multi-tenant data center environment this operational model can lead to inefficient use of bandwidth and sub-optimal forwarding.

To provide a more optimal forwarding model and avoid traffic tromboning, the EVPN inter-subnet draft (draft-sajassi-l2vpn-evpn-inter-subnet-forwarding) proposes Integrating the Routing and Bridging (IRB) functionality directly onto the VTEP, thereby allowing the routing operation to occur as close to the end host as possible. The draft proposes two forwarding models for the IRB functionality, which are termed asymmetric IRB and symmetric IRB, these two models are described in the following sections.

In the asymmetric IRB model, the inter-subnet routing functionality is performed by the ingress VTEP, with the packet after the routing action being VXLAN bridged to the destination VTEP. The egress VTEP only then needs to remove the VXLAN header and forward the packet onto the local Layer 2 domain based on the VNI to VLAN mapping. In the return path, the routing functionality is reversed with the destination VTEP now performing the ingress routing and VXLAN bridging operation, hence the term asymmetric IRB.

Figure 1. EVPN Asymmetric IRB

To provide inter-subnet routing on all VTEPs for all subnets, an anycast IP address is utilized for each subnet and configured on each VTEP. The anycast IP acts as the default gateway for the hosts, therefore regardless of where the host resides the directly attached VTEPs can act as the host’s default gateway. The host MAC and MAC to IP bindings are learned by each VTEP based on a combination of local learning/ARP snooping and type-2 route advertisement from remote VTEPs.

In a typical implementation, the optional MAC and IP, type-2 route is advertised separately from the MAC only type-2 route. This is done so that if the MAC and IP route is cleared, for example the ARP flushed, or the ARP timeout is set to less than the MAC timeout, then the MAC only route will still exist.

The format of the two advertised type-2 routes for Server-1 are illustrated below, where the RD IP-A:1010 and route-target 1010:1010 are used to distinguish the uniqueness of the route and allow the route to be imported into the correct remote MAC-VRF based on the route-target import policy of the VTEP.
Figure 2. EVPN Comparison of MAC & MAC+IP Type 2 Route in Asymmetric IRB

For the traffic flow between Server-1 in subnet-10 and Server-4 in subnet-11, the ingress VTEP (VTEP-1) locally routes the packet into subnet-11/VNI 1011 and then VXLAN bridges the frame, inserting the VNI 1011 into the VXLAN header with an inner DMAC equal to the destination host, Server-4. This requires the receiving VTEP, (VTEP-4) to only perform a local Layer 2 lookup, based on the VNI to VLAN mapping, for the DMAC of Server-4.

Figure 3. EVPN Asymmetric IRB VxLAN Data-plane Forwarding Detail
For the asymmetric model to operate the sending VTEP needs the information for all the tenant’s hosts (MAC and MAC to IP binding), to route and bridge the packet. This means the VTEP needs to be member of all the tenant’s subnets/VNI and have an associated SVI with anycast IP for all the subnets, and this will be required on all VTEPs participating in the routing functionality for the tenant. This introduces scaling issues on multiple fronts.
  • VNI Scaling: The number of VNIs supported on a hardware VTEP will be finite, so not all VNIs can reside on all VTEPs. This is especially true in data-center deployments, where the TOR’s have traditionally been more resource constrained than chassis-based edge systems.
  • Forwarding memory scaling: The VTEPs needs to store all host MACs and ARP entries for all subnets in the network, on leaf switch this is hardware resource which again will be a finite resource defined by the specific hardware platform deployed at the leaf.

Symmetric IRB

To address the scale issues of the asymmetric model, in the symmetric model the VTEP is only configured with the subnets that are present on the directly attached hosts. Connectivity to non-local subnets on a remote VTEP is achieved through an intermediate IP-VRF. The subsequent forwarding model for symmetric IRB is illustrated in the figure below, for traffic between Server-1 on subnet-10 (Green) and Server-4 on the remote subnet-11 (Blue). In this model, the ingress VTEP routes the traffic between the local subnet-10) and the IP-VRF, which both VTEPs are a member of, the egress VTEP then routes the frame from the IP-VRF to the destination subnet. The forwarding model results in both VTEPs performing a routing function, hence the term symmetric IRB.

Figure 4. EVPN Symmetric IRB

To provide the inter-subnet routing, when the subnet is stretched across multiple VTEPs, an anycast IP address is utilized for each subnet, but only configured on the VTEP’s where the subnet exists. The host MAC and MAC to IP bindings are learned by each VTEP based on a combination of local learning/ARP snooping and type-2 route advertisements.

For the symmetric IRB model the type-2 (MAC and IP) route is advertised with two labels and two route-targets corresponding to the MAC-VRF the MAC address is learned on and the IP-VRF. Remote VTEP’s receiving the route, import the IP host route into the corresponding IP-VRF based on the IP-VRF route-target and if the corresponding MAC-VRF exists on the VTEP the MAC address is imported into the local MAC-VRF based on the MAC-VRF’s Route-Target. The import behavior for the type-2 route is illustrated in the diagrams below for the host Server-1.

If the MAC-VRF exists locally on the receiving router, both the IP host route will be installed in the IP-VRF, and the MAC address will be installed in the MAC-VRF. With both a MAC route in the MAC-VRF and an IP host route in the IP-VRF, the VNI used in the data-path will depend on whether the traffic is being VXLAN bridged between hosts in the same VNI (1010) or VXLAN routed (VNI 2000).

Figure 5. EVPN Type 2 Route in Symmetric IRB - MAC-VRF on Both VTEPs

Compare this to the figure below, where the MAC-VRF does not exist on the receiving VTEP (VTEP-2). In this case the MAC route is not installed and ignored, as there is no corresponding Route Target on the VTEP. In this scenario, only the IP-VRF host route is installed on VTEP-2. Traffic from VTEP-2 destined to hosts on subnet-10, are therefore always VXLAN routed via the IP-VRF, VNI 2000.

Figure 6. EVPN Type 2 Route in Symmetric IRB - MAC-VRF Only Exists on Sending VTEP
The symmetric IRB type-2 route contains a number of additional extended community attributes over the asymmetric IRB type-2 route, the salient fields of the route are summarized below.
  • Multi-protocol Reachable NLRI (MP_REACH_NLRI) attribute is used to carry the next-hop hop for the advertised route. In the context of a VXLAN forwarding plane, this will be the source address of the advertising VTEP.
  • Route Distinguisher of the advertising node’s MAC-VRF. For Server-1 in the example above this would be IPA:1010.
  • MAC address field contains the 48-bit MAC address of the host being advertised. For Server-1 in the example above this would be MAC-1.
  • IP address and length field contain the IP address and 32-bit mask for the host being advertised. For Server-1 in the example above this would be IP-1.
  • MAC-VRF label, this contains the VNI number (label) corresponding to the local Layer 2 domain/MAC-VRF the host MAC was learned on. For Server-1 in the example above this would be VNI 1010.
  • IP-VRF label, this contains the VNI number (label) corresponding to the MAC-VRF’s associated lP-VRF. For MAC-VRF 10 in the example above this would be IP-VRF 2000.
  • Extended community Route Target for the IP-VRF. This contains the route-target of the IP-VRF associated with the learned MAC address.
  • Extended community Router MAC. This field advertises the system MAC of the advertising VTEP and is used as the DMAC for any packet sent to the VTEP via the IP-VRF.
  • Extended community Route Target for the MAC-VRF. This contains the route-target of the MAC-VRF associated with the learned MAC address.