Telemetry Collector

Deployment

The diagram above shows a DMF deployment with an Active/Standby Controller cluster. In this environment, each Controller collects metrics from all the devices it manages and the controller nodes. Each Controller establishes a gNMI connection to all the devices and the other Controller in a fabric to collect telemetry streams. gNMI is a gRPC-based protocol to configure and access states on network devices. The REST API exposes this information in the /api/v1/data/controller/telemetry/data subtree. The Controller exposes several metrics about CPU, memory, disk utilization, interface counters, and sensor states of all devices in a fabric. All metrics are fetched at a 10-second frequency except those associated with the sensors, which are collected every minute.

Configuration

No additional configuration is necessary on the DMF Controller to enable the metric collection. However, to read these metrics, the user must be an admin user or configured with the privilege, category:TELEMETRY. To set the telemetry permission for a custom group and associate the user with this group, use the following commands:

dmf-controller(config)# group group_name
dmf-controller(config)# permission category:TELEMETRY privilege read-only
dmf-controller(config)# associate user username 

Connection Status

The Controller uses the gNMI protocol to collect telemetry data from all devices. DMF reports the status of these connections in both the REST API and the CLI.

REST API

The REST API subtree /api/v1/data/controller/telemetry/connection reports the telemetry connection state. The API Schema browser of the GUI provides more details.

CLI

The following show commands display the connection details. All these commands support filtering the output with a device name.

show telemetry connection device-name | all 
show telemetry connection device-name | all details
show telemetry connection device-name | all last-failure

The show telemetry connection device-name all command shows the state and the latest state change of the connection between the Controller and the devices.

dmf-controller# show telemetry connection all
# Name                         State Last state change              
-|----------------------------|-----|------------------------------|
1 c2                           ready 2023-11-10 13:10:02.718000 UTC
2 core2                        ready 2023-11-10 13:22:56.311000 UTC
<snip>

The show telemetry connection device-name all command displays the state and the latest state change of the connection between the Controller and the devices.

dmf-controller# show telemetry connection all details 
# Name  State Last state change              Target              Connection type Last message time              
-|-----|-----|------------------------------|-------------------|---------------|------------------------------|
1 core2 ready 2023-11-10 13:22:56.311000 UTC 10.243.255.102:6030 clear-text      2023-11-10 13:59:35.437000 UTC
<snip>

The show telemetry connection device-name all last-failure command displays more details about a connection failure. The time of the latest failure and the potential reason appear in the output. If the connection is still in the failed state, this output also shows when the next reconnection is attempted.

dmf-controller# show telemetry connection all last-failure 
# Name  Fail time                      Fail type       Root cause                Next retry in 
-|-----|------------------------------|---------------|-------------------------|-------------|
1 core2 2023-11-10 13:19:34.237000 UTC unavailable     UNAVAILABLE: io exception 0
<snip>

Limitations

• Software interfaces (for example, loopback, bond, and management) do not report counters for broadcast and unicast packets.
• The reported interface names are the raw physical interface name (e.g., et1) rather than the user-configured name associated with the role of an interface (e.g., filter1).
• Resetting the interface counter does not affect the counter values stored at the /telemetry/data path. The value monotonically increases and corresponds to the total count since the device was last powered up. This value only gets reset when rebooting the device.

Usage Notes

• DMF uses the configured name of a managed device (e.g., switch, recorder node, etc.) on the Controller as the value of the key name for the node device for all the metrics corresponding to it. In the case of a Controller, DMF uses the configured hostname as the key. Thus, these names must be unique in a specific DMF deployment.
• The possibility exists that metrics are not collected from a device for a short period. This data gap may happen when rebooting the device or when the Controllers experience a failover event.
• If the gNMI connection between the Controller and the device is interrupted, the Controller attempts a new connection after 2 minutes. The retry timeout for a subsequent connection attempt increases exponentially and can go up to 120 minutes. Upon a successful reconnection, this timeout value resets to 2 minutes.
• There might be gNMI warning messages in the floodlight log during certain events, e.g., when first adding a device or it is reloading. Ignore these messages.
• This feature enables an OpenConfig agent on switches running EOS to collect telemetry.

Telemetry Availability

As a DMF fabric consists of different types of devices, the metrics of each vary. The following outlines the metrics collected from each device type by the Controller and typically made available over its REST API. However, some specific platforms or hardware might not report a particular metric. For brevity, the following list mentions the leaves that can correspond to a metric. For more details, use the API Schema browser of the GUI.

telemetry
+-- data
     +-- device
         +-- interface
         |  +-- oper-status                Ctrl, SWL, EOS, SN, RN
         |  +-- counters
         |     +-- in-octets               Ctrl, SWL, EOS, SN, RN
         |     +-- in-pkts                 Ctrl, SWL, EOS, SN, RN
         |     +-- in-unicast-pkts         Ctrl, SWL, EOS, SN, RN
         |     +-- in-broadcast-pkts       Ctrl, SWL, EOS, SN, RN
         |     +-- in-multicast-pkts       Ctrl, SWL, EOS, SN, RN
         |     +-- in-discards             Ctrl, SWL, EOS, SN, RN
         |     +-- in-errors               Ctrl, SWL, EOS, SN, RN
         |     +-- in-fcs-errors           Ctrl, SWL, EOS, SN, RN
         |     +-- out-octets              Ctrl, SWL, EOS, SN, RN
         |     +-- out-pkts                Ctrl, SWL, EOS, SN, RN
         |     +-- out-unicast-pkts        Ctrl, SWL, EOS, SN, RN
         |     +-- out-broadcast-pkts      Ctrl, SWL, EOS, SN, RN
         |     +-- out-multicast-pkts      Ctrl, SWL, EOS, SN, RN
         |     +-- out-discards            Ctrl, SWL, EOS, SN, RN
         |     +-- out-errors              Ctrl, SWL, EOS, SN, RN
         +-- cpu
         |  +-- utilization                Ctrl, SWL, EOS, SN, RN
         +-- memory
         |  +-- total                      Ctrl, SWL, SN, RN
         |  +-- available                  Ctrl, SWL, EOS, SN, RN
         |  +-- utilized                   Ctrl, SWL, EOS, SN, RN
         +-- sensor
         |  +-- fan
         |  |  +-- oper-status             Ctrl, SWL, EOS, SN, RN
         |  |  +-- rpm                     Ctrl, SWL, EOS, SN, RN
         |  |  +-- speed                   SWL
         |  +-- power-supply
         |  |  +-- oper-status             Ctrl, SWL, EOS, SN, RN
         |  |  +-- capacity                EOS
         |  |  +-- input-current           Ctrl, SWL, EOS, SN, RN
         |  |  +-- output-current          SWL, EOS
         |  |  +-- input-voltage           Ctrl, SWL, EOS, SN, RN
         |  |  +-- output-voltage          SWL, EOS
         |  |  +-- input-power             Ctrl, SWL, SN, RN
         |  |  +-- output-power            SWL, EOS
         |  +-- thermal
         |     +-- oper-status             Ctrl, SWL, SN, RN
         |     +-- temperature             Ctrl, SWL, EOS, SN, RN
         +-- mount-point
         |  +-- size                       Ctrl, SWL, SN, RN
         |  +-- available                  Ctrl, SWL, SN, RN
         |  +-- utilized                   Ctrl, SWL, SN, RN
         |  +-- usage-percentage           Ctrl, SWL, SN, RN
         +-- control-group
         +-- memory                     Ctrl, SWL, SN, RN
         +-- cpu                        Ctrl, SWL, SN, RN
            
* Ctrl = Controller, SWL = A switch running SwichLight OS, 
EOS = A switch running Arista EOS, SN = Service Node, RN = Recorder Node