Managing Switches and Interfaces

This chapter describes how to manage switches and interfaces after installing the monitoring fabric switches.

 

Connecting Directly to a Switch

To install the Switch Light OS individually on a switch in a different Layer 2 domain or troubleshoot the switch, use telnet or SSH to connect to the switch.

To allow SSH to a switch, if using ZTF for installing switches from the DANZ Monitoring Fabric (DMF) Controller in the same Layer 2 domain, configure an IPAM IP address pool. Alternatively, use the connect switch switch-name command to connect to the switch CLI. 
controller-1(config)# connect switch DMF-FILTER-SWITCH-1
Warning: Permanently added the RSA host key for IP address 'fe80::3617:ebff:fef2:cfc4%em1' to the
list of known hosts.
Last login: Mon Sep 18 02:32:35 2017 from fe80::46a8:42ff:fe35:29f7%ma1
SwitchLight ZTN Manual Configuration. Type help or ? to list commands.
After connecting to the switch, enter the debug admin command at the displayed ztn-config prompt. 
(ztn-config) debug admin
DMF-FILTER-SWITCH-1>
This command provides access to the switch CLI prompt for directly managing or troubleshooting the switch. The following commands are available in the ZTN console:
  • controller: adds, removes, sets, or clears the L3 ZTN Controller list
  • debug: Special command to access the full switch CLI
  • help: Displays CLI help
  • interface: sets the ma1 address parameters
  • reboot: restarts the switch
  • setup: performs interactive setup
  • show: displays the current settings

Manually Configuring Enhanced Hashing for Load Distribution

In some scenarios, it may be desirable to manually select the bytes in each packet that are used for load distribution among the members in a LAG.

Enter the hash-type enhanced command from the config-switch-lag-if submode to manually configure enhanced hashing.

Use the lag-enhanced-hash command to enter the config-switch-hash submode and use the hash command to identify the values to use for load distribution.
Note: For a list of the switch platforms that support enhanced hashing (including symmetric hashing), refer to the DANZ Monitoring Fabric Hardware Compatibility List.

Changes in hash configuration do not affect the LAG configuration, so there is no need to reconfigure LAGs after changing the hash type.

Configuring Enhanced Hashing

To configure enhanced hashing, use the lag-enhanced-hash command to enter the config-switch-hash submode. Use the hash command to identify the hash type and the specific fields for load distribution. 
controller-1(config-switch)# lag-enhanced-hash
controller-1(config-switch-hash)#

The hash command has the following syntax:

[no] hash gtp header-first-byte <GTP header first byte> header-first-byte-mask <GTP header first byte mask> | gtp port-match <UDP tunnel port match entry number> {dst-port <GTP tunnel UDP destination port> {and | or} src-port <GTP tunnel UDP source port> | src-port <GTP tunnel UDP source port>} | ipv4 {[dst-ip] [l4-dst-port] [l4-src-port] [protocol] [src-ip] [vlan-id]} | ipv6 {[dst-ip] [l4-dst-port] [l4-src-port] [nxt-hdr] [src-ip] [vlan-id]} | l2 [dst-mac] [eth-type] [src-mac] [vlan-id] l2gre {inner-l2 [dst-mac] [eth-type] [src-mac] [vlan-id] | inner-l3 [dst-ip] [l4-dst-port] [l4-src- port] [protocol] [src-ip] [vlan-id]} mpls {[label-1] [label-2] [label-3] [label-hi-bits] [payload-dst-ip] [payload-src-ip]} seeds { <First hash seed> [<Second hash seed>]} symmetric {enable | disable}

Symmetric Load Balancing

Enhanced hashing supports symmetric load balancing (enabled by default) for switch platforms that support this feature.
Note: DANZ Monitoring Fabric (DMF) supports symmetric hashing on specific switches for IP and Fibre Channel over Ethernet (FCoE) traffic. Symmetric hashing on MPLS traffic labels is not supported.
With symmetric load balancing, the link selected for distributing traffic in one direction is also used for traffic in the other direction. Arista Networks recommends enabling hashing on the source IP address and destination IP address for optimal symmetric behavior.
Note: In some scenarios, using Layer 4 protocol ports can improve load-balancing efficiency. However, these fields are not used by default because they cannot be used if packet fragmentation is likely to occur.

GTP Hashing

Generic Tunneling Protocol (GTP) hashing provides a more even distribution of GTP-encapsulated packets among the members of a port-group. When GTP hashing is enabled, DANZ Monitoring Fabric (DMF) includes the Tunnel endpoint identifier (TEID) value in the GTP packets in its hashing algorithm for outbound traffic. This applies only to GTP user data tunneling packets (udp port 2152). GTP control traffic (udp port 2123) is not affected.

To enable hashing with Generic Tunneling Protocol (GTP), use the hash gtp command. This command sets enhanced hash parameters for distributing traffic on port-channel member ports for which enhanced hashing is enabled. The command syntax is as follows: 
hash gtp port-match <port-match> {dst-port <dst-port> {and | or} src-port <src-port> | dst-port
<dst- port> | src-port <src-port>}

The GTP command specifies the packet fields to identify GTP traffic. When enabled, DMF uses the TEID in the GTP header for hashing GTP traffic instead of using L4 ports—Configure l4-dst-port and l4-src-port in hash ipv4 or hash ipv6 for proper operation.

Overriding the Default Switch Configuration

After completing the switch installation, further switch configuration, including software upgrades, is managed from the DANZ Monitoring Fabric (DMF) Controller.
CAUTION: Make all configuration changes related to fabric switches using the Controller CLI or the Controller GUI, which provides DMF-level configuration options in the config-switch submode for each switch. Do not log in to the switch to make changes directly using the switch CLI.
In general, the configuration options set on the DMF Controller are pushed to each connected switch, eliminating the need for box-by-box configuration. However, merging or overriding the default configuration pushed from the DMF Controller with switch-specific configuration for some parameters is possible.These parameters are as follows:
  • Clock
  • SNMP and SNMP Traps
  • Logging
  • TACACS+
DMF supports two types of overriding mechanisms:
  • override-global: Only the switch-specific configuration is applied.
  • merge-global: The global config and switch-specific configuration are merged and then applied.
In the merge mode, the effective switch configuration is determined by the following rules:
  • Stand-alone values:If the key only exists in one of the configurations; take it as-is in the resultant configuration, otherwise:
  • If the key exists in both global and switch configurations: the value of the key from the switch-config takes precedence (over its value from the global-config).
  • Lists: If the list only exists in one of the configurations; take that list as-is in the result configuration, otherwise:
  • If it exists in both global and per-switch configuration, then merge with this rule.
  • If the global and switch-specific configuration has an entry with the same key, the switch-specific list entry completely replaces the entry from the global-config, otherwise:
  • All entries from the switch-specific configuration are appended to the global-config (with de-duplication). The configurations that occur as lists for the above overridable parameters are indicated below:
  • ntp
    • server <- list
    • time-zone
  • snmp-server
    • community <- list
    • contact
    • enable
    • host <- list
    • location
    • switch trap
    • user <- list
  • logging
    • controller
    • remote
    • remote server <- list
  • tacacs
    • server <- list

GUI Procedure

  1. Select Fabric > Switches from the main menu.
    The system displays the Switches page, which lists the switches connected to the DMF controller.
  2. To override any of the default switch configuration settings, click the Menu control next to the switch and select Configure from the pull-down menu that appears.
    Figure 1. Configure Switch (Page 1)
    This dialog provides access to a series of dialogs used to override the default configuration that is pushed from the DMF Controller to the switch.
  3. To advance to another page, click the numbered link for the page or click Next.
     
  4. After making any changes required, click Submit.
    CLI Procedure
    To override the default configuration for a specific switch, enter the config-switch submode for the specific switch and use the commands available, viewable by entering the help command or by using tab completion. 
    controller-1# config
controller-1(config) switch DMF-FILTER-SWITCH-1
controller-1(config-switch) <Tab>
admin lag-interface sflow switch-group
banner logging show
 tacacs
description mac shut down
 tunnel-interface
interface ntp snmp
lag-enhanced-hash role snmp-server
controller-1 (config-switch)#

    Use the shutdown command to shut down a switch from the Controller, in which case all the interfaces of the switch are put in admin down mode and the switch is black-holed.

Configuring Switch Interfaces

To use the GUI to configure switch interfaces, complete the following steps.

Procedure

  1. Select Fabric > Interfaces from the GUI Main menu.
    Figure 2. Fabric Interfaces Option

     

    This page allows monitoring and configuring the interfaces on switches connected to the DANZ Monitoring Fabric (DMF) Controller. To display details about a specific interface, click the Expansion control to the left of the interface, and the entry expands to show any Tunnel Interfaces or Core Links currently using the interface.
    To configure an interface, click the Menu control next to the interface and select Configure from the pull-down menu.
    Figure 3. Configuring Interface Settings - Page 1
    This dialog provides access to three pages.
    Note: Page 3 is the DMF page configuration settings for interfaces to use in policies. For further information, refer to the DANZ Monitoring Fabric User Guide.

     

    Page 1: The Port dialog provides the following options:
    • Admin Status: Enable or disable the switch administratively.
    • Enable Optics: Change the default to cause the optical laser to be left on after the port goes down.
    • MAC Loopback Mode: Returns traffic to the originating interface.
    • Force Link Up: This is useful to enable when only the transmit fiber is connected.
    • Description: Assign a description for the interface.
    Tip:
    1. Ideally, only apply a force link-up configuration on a delivery interface.
    2. This configuration allows L1 on the port to stay up, even when the optical fiber cable is connected only in the TX direction.
    3. This feature helps to black hole traffic if applied on links between switches.

     

  2. When finished, click Save. To configure traffic options, click Next.
    After clicking Next, the system displays the following page.
    Page 2:
    Figure 4. Configuring Interface Settings - Page 2
    This page provides the following options.
    • Forward Error Correction
    • Auto-Negotiation
    • Breakout
    • Speed
    • Rate Limit
  3. After making any changes required, click Save.

Forward Error Correction (FEC)

Use Page 2 of the Edit Interface dialog to explicitly enable or disable forward error correction or to restore the default.

Figure 5. FEC Configuration

 

CLI Procedure

To use the CLI to explicitly enable or disable forward error correction or restore the default, use the following commands in config-switch mode: 
controller-1(config-switch)# interface ethernet10
controller-1(config-switch-if)# forward-error-correction
disable Force disable interface forward-error-correction
enable Force enable interface forward-error-correction
enable-fire-code Force enable interface fire-code forward-error-correction
enable-reed-solomon Force enable interface reed-solomon forward-error-correction
enable-reed-solomon544 Force enable interface reed-solomon544 forward-error-correction
controller-1(config-switch-if)# forward-error-correction enable
controller-1(config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface ethernet10
autoneg disable
forward-error-correction enable
controller-1(config-switch-if)# forward-error-correction disable
controller-1(config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface ethernet10
autoneg disable
forward-error-correction disable
controller-1(config-switch-if)

 

The following summarizes the effect of each forward error correction keyword option:
  • disabled – Disable if possible in the current port context.
  • enabled – Enable if possible in the current port context.
  • enable-fire-code – Request Fire-Code FEC (CL74) in the port context.
  • enable-reed-solomon – Request Reed-Solomon FEC (CL91, CL108) in the port context.
  • enable-reed-solomon544 – Force Reed-Solomon544 in the port context.
Note: Switch platforms with the Tomahawk ASIC cannot support Reed-Solomon FEC (CL108) on 25G interfaces. FEC options supported on 25G interfaces are limited Fire-Code FEC or FEC Disabled. This limitation does not apply to 100G interfaces.

 

400GBASE-ZR Modules

400GBASE-ZR modules utilize coherent FEC: concatenated FEC (C-FEC) or open FEC (O-FEC), which are available amongst the FEC options in the Edit Interface window.

Figure 6. Coherent FEC Configuration (for example OFEC)

 

To use the CLI to explicitly configure coherent FEC, use the following commands in config-switch mode: 
controller-1(config-switch)# interface ethernet31/1
controller-1(config-switch-if)# forward-error-correction 
disableForce disable interface forward-error-correction
enable Force enable interface forward-error-correction
enable-cfecForce enable interface concatenated forward-error-correction
enable-fire-code Force enable interface fire-code forward-error-correction
enable-ofecForce enable interface open forward-error-correction
enable-reed-solomonForce enable interface reed-solomon forward-error-correction
enable-reed-solomon544 Force enable interface reed-solomon544 forward-error-correction

controller-1(config-switch-if)# forward-error-correction enable-ofec
controller-1(config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface Ethernet31/1
forward-error-correction enable-ofec
transceiver-frequency 192100000
controller-1(config-switch-if)# forward-error-correction enable-cfec
controller-1(config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface Ethernet31/1
forward-error-correction enable-cfec
transceiver-frequency 192100000
controller-1(config-switch-if)

 

Transceiver Frequency Configuration

All coherent transceiver modules, including 400GBASE-ZR, require their laser frequency to be explicitly configured. To configure the frequency in 400GBASE-ZR, use the transceiver frequency command under the defined interface, specifying the value in MHz in the range 191300-196100 GHz.

controller-1(config-switch-if)# transceiver-frequency
<Core/switch-config/interface/transceiver-frequency-mhz> Enter frequency in MHz

controller-1(config-switch-if)# transceiver-frequency 193100000
controller-1(config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface Ethernet31/1
forward-error-correction enable-cfec
transceiver-frequency 193100000
controller-1(config-switch-if)# transceiver-frequency 192100000
controller-1(config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface Ethernet31/1
forward-error-correction enable-cfec
transceiver-frequency 192100000
controller-1(config-switch-if)

 

Use Page 1 of the Edit Interface dialog to configure the frequency.

Figure 7. Transceiver Frequency Configuration

 

 

Autonegotiation

The following summarizes the effect of each option:
  • autoneg enabled – Enable if possible in the current port context.
  • autoneg disabled – Disable if possible in the current port context.
Autonegotiation can be enabled or disabled for the following interface configurations:
  • 100 GbE DAC in 100 GbE mode
  • 25 GbE
  • 1G-BASE-SX
  • 1G-BASE-LX

GUI Procedure

Use Page 2 of the Edit Interface dialog to enable or disable autonegotiation, or to restore the default.

CLI Procedure

To use the CLI to explicitly enable or disable autonegotiation, or to restore the default, enter the following command from config-switch mode for any fabric switch: 
controller-1(config-switch)# interface ethernet10
controller-1 (config-switch-if)# autoneg enable
controller-1 (config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface ethernet10 autoneg enable
controller-1 (config-switch-if)# autoneg disable
controller-1 (config-switch-if)# show this
! switch
switch DMF-FILTER-SWITCH-1
!
interface ethernet10
autoneg disable
controller-1 (config-switch-if)#

Replace intf-port-list by the interface name or port list.

Manually Setting the Interface Speed

To manually set the interface speed for an interface, from config-switch-if submode, enter the speed command, which has the following syntax.

[no] speed [{100G | 25G | 200G | 1G | 10G | 40G | 400G | 50G}]

The following are the options supported.
  • 100G Set interface speed to 100 Gbps
  • 10G Set interface speed to 10 Gbps
  • 1G Set interface speed to 1 Gbps
  • 200G Set interface speed to 200 Gbps
  • 25G Set interface speed to 25 Gbps
  • 400G Set interface speed to 400 Gbps
  • 40G Set interface speed to 40 Gbps
  • 50G Set interface speed to 50 Gbps

Using Breakout Cables

DANZ Monitoring Fabric (DMF) supports using breakout (splitter) cables to split a single 40 GbE, 100 GbE, 200 GbE, and 400 GbE port into individual sub-interfaces.

For a list of supported switches, ports, and breakout cables, refer to the DANZ Monitoring Fabric 8.5 Hardware Compatibility List. The breakout cables listed in the Hardware Compatibility List are broken out automatically; manually entering the breakout command is not required.

GUI Procedure

To use the GUI to manually enable the use of multiple interfaces on a single switch port with a breakout cable, select Fabric > Interfaces , select Edit from the menu control for an interface, and use the settings on the Traffic page (Page 2) of the Edit Interface dialog.

To enable the use of multiple interfaces on a single switch port with a breakout cable, complete the following steps.

CLI Procedure

Use the breakout mode command to configure the breakout property for the current interface to configure the force breakout on an interface. When the config is applied, the interface, if it supports the breakout for the mode, is broken out into sub-interfaces based on the specified mode. Auto is the default option, which lets the switch automatically select the mode for the breakout. The following are the modes supported.
  • 2x100G Breakout to 2 sub-interfaces of 100G each
  • 2x200G Breakout to 2 sub-interfaces of 200G each
  • 2x40G Breakout to 2 sub-interfaces of 40G each
  • 2x50G Breakout to 2 sub-interfaces of 50G each
  • 4x100G Breakout to 4 sub-interfaces of 100G each
  • 4x10G Breakout to 4 sub-interfaces of 10G each
  • 4x1G Breakout to 4 sub-interfaces of 1G each
  • 4x25G Breakout to 4 sub-interfaces of 25G each
  • 4x50G Breakout to 4 sub-interfaces of 50G each
  • 8x10G Breakout to 8 sub-interfaces of 10G each
  • 8x25G Breakout to 8 sub-interfaces of 25G each
  • 8x50G Breakout to 8 sub-interfaces of 50G each
  1. To verify the breakout-capable ports on the switch, enter the show switch interfaces command, which has the following syntax: 
    show switch switch-name interfaces
    To locate breakout-capable ports from the Controller, enter the following command: 
    controller-1> show switch DMF-F1 interfaces
#IF NameMAC Address ConfigStateAdv. Features Curr Features Supported Features
-- |---------- |------------------------------ |------ |----- |-------------------------- |-------------------------- |--------------------------------------------------- |
1 ethernet1 5c:16:c7:13:d5:9f (Big Switch)upup autoneg, fec, 1g, 10g, 25gcopper, autoneg, fec, 25g copper, autoneg, fec, 1g, 10g, 25g
2 ethernet2 5c:16:c7:13:d5:a0 (Big Switch)upup autoneg, fec, 1g, 10g, 25gcopper, autoneg, fec, 25g copper, autoneg, fec, 1g, 10g, 25g
...
...
49 ethernet49 5c:16:c7:13:d5:d5 (Big Switch)upup 40g fiber, 40gfiber, bsn-breakout-capable, 1g, 10g, 25g, 40g, 100g
50 ethernet50 5c:16:c7:13:d5:d6 (Big Switch)upup 40g fiber, 40gfiber, bsn-breakout-capable, 1g, 10g, 25g, 40g, 100g
51 ethernet51 5c:16:c7:13:d5:d7 (Big Switch)upup 40g fiber, 40gfiber, bsn-breakout-capable, 1g, 10g, 25g, 40g, 100g
52 ethernet52 5c:16:c7:13:d5:d8 (Big Switch)upup 40g fiber, 40gfiber, bsn-breakout-capable, 1g, 10g, 25g, 40g, 100g
53 ethernet53 5c:16:c7:13:d5:d9 (Big Switch)upup 40g fiber, 40gfiber, bsn-breakout-capable, 1g, 10g, 25g, 40g, 100g
54 ethernet54 5c:16:c7:13:d5:d0 (Big Switch)upup 40g fiber, 40gfiber, bsn-breakout-capable, 1g, 10g, 25g, 40g, 100g

    Each breakout-capable port is identified by the string bsn-breakout-capable in the Supported Features column. In this example, ports ethernet49 through ethernet54 are breakout-capable.

     

  2. Enter the enable and configure command to enter config mode, as in the following example. 
    controller-1> en
controller-1#
conf controller-1(config)#
  3. Enter the config-switch submode and then the config-switch-if submode to enter the breakout command, as in the following example. 
    controller-1(config)# switch DMF-FILTER-SWITCH-1
controller-1(config-switch)# interface ethernet54
controller-1(config-switch-if)# breakout mode 4x10G
  4. Enter the show switch switch-name interface command to verify the operation, as in the following example. 
    controller-1(config-switch-if)# show switch DMF-FILTER-SWITCH-1 interfaces
#IF NameMAC AddressConfig State Adv. FeaturesCurr FeaturesSupported Features
--|------------|------------------------------|------|-----|--------------------------|--------------------------|----------------------------------|
1ethernet15c:16:c7:13:d5:9f (Big Switch) upup autoneg, fec, 1g, 10g, 25g copper, autoneg, fec, 25gcopper, autoneg, fec, 1g, 10g, 25g
2ethernet25c:16:c7:13:d5:a0 (Big Switch) upup autoneg, fec, 1g, 10g, 25g copper, autoneg, fec, 25gcopper, autoneg, fec, 1g, 10g, 25g
...
...
54 ethernet54/1 5c:16:c7:13:d5:d5 (Big Switch) upup 40gfiber, 10g fiber, 1g, 10g, 25g, 40g, 100g
55 ethernet54/2 5c:16:c7:13:d5:d6 (Big Switch) upup 40gfiber, 10g fiber, 1g, 10g, 25g, 40g, 100g
56 ethernet54/3 5c:16:c7:13:d5:d7 (Big Switch) upup 40gfiber, 10g fiber, 1g, 10g, 25g, 40g, 100g
57 ethernet54/4 5c:16:c7:13:d5:d8 (Big Switch) upup 40gfiber, 10g fiber, 1g, 10g, 25g, 40g, 100g

     

  5. The breakout ports are named ethernetx/1 through ethernetx/4. The example output shows four 10G ports where there was previously a single 40G port (ethernet54/1 through ethernet54/4).

Verifying Switch Configuration

GUI Procedure

Use the Fabric > Interfaces option to view the interfaces table, which provides information about the configuration and activity on each interface of the switches connected to the DANZ Monitoring Fabric (DMF) controller.

CLI Procedure

To view the configuration or activity for a specific interface, use the show switch switchname interfaces command. The detail option provides additional information about the interface, including the up and down counts, indicating if the interface has been flapping. The output also indicates if the interface supports breakout interfaces.

The following is an example. 
controller-1# show switch DMF-DELIVERY-SWITCH-1 interfaces ethernet49 detail
# IF NameMAC Address Config State Adv. FeaturesCurr FeaturesSupportedFeatures
- |------------ |------------------------------ |------ |----- |------------------- |------------- |------------------------------- |
1ethernet495c:16:c7:13:d5:d8 (Big Switch)updown fec, 25g, 50g, 100gfecfec,bsn-breakout-capable, 100g
To view the interface descriptions, use the show switch switchname interface description command as shown in the following example. 
controller-1(config-switch-if)# show switch DMF-DELIVERY-SWITCH-1 interface description
# Switch Name IF NameDescription
-|---------------------|------------|---------------|
1 DMF-DELIVERY-SWITCH-1 ethernet1100g-to-SFO
2 DMF-DELIVERY-SWITCH-1 ethernet2100g-to-NYC

Disabling the TX Direction on an Interface

DANZ Monitoring Fabric (DMF) supports disabling the TX direction of an interface, which is useful for example on a DMF switch whose filter interfaces are directly connected to a 40 GbE bidirectional tap using 40 GbE BiDi optics.
Note: when using bidirectional TAPs, either you need to use RX-only BiDi optics or you need to disable the TX direction on the DMF interfaces; otherwise, the optic transmits light back to the TAP and interferes with the production link. This is the case with all bidirectional TAPs: the outputs of the bidirectional TAPs should be connected to transceivers that don’t transmit because there is no way for the TAP to filter out light coming back from the packet broker.

 

If a filter interface is connected to a bidirectional TAP using a BiDi optic, you should configure that filter interface with the disable-xmit command to avoid optical interference with the TAP:

Example of command usage:

dmf-controller-1(config)# switch sw-filter1
dmf-controller-1(config-switch)# interface ethernet2
dmf-controller-1(config-switch-if)# disable-xmit