Deployment

The aforementioned diagram shows a DMF deployment with an Active/Standby controller cluster. In this environment, each controller collects metrics from all the devices it manages as well as from both the controller nodes. It then exposes them via the telemetry endpoint /api/v1/metrics/prometheus. This is an authenticated endpoint that listens on port 8443 and supports both Prometheus and OpenMetrics exposition formats.

Even though each controller is capable of serving the telemetry information, it is recommended to use cluster's virtual IP (VIP) in the Prometheus configuration in order to achieve seamless continuity in the event of a controller failover.

Configuration

No additional configuration is necessary on the DMF controller to enable the metric collection. However, to allow the Prometheus service to access the new HTTP telemetry endpoint, a user access-token needs to be created. An admin user can choose an existing user or create a dedicated user with correct privileges and generate a token in order for Prometheus to fetch metrics from a fabric. The following sections describe the necessary configurations.

Permission

The group that the user belongs to needs to have sufficient permission to query the Prometheus endpoint. The following table summarizes the behavior.

To set telemetry permission for a custom group, use the following commands:

dmf-controller(config)# group group_name
dmf-controller(config)# permission category:TELEMETRY privilege read-only
dmf-controller(config)# associate user username

Access Token

Generate an access token for the user using the following command.

dmf-controller(config)# user username
dmf-controller(config-user)# access-token descriptive name for the access-token
access-token : ZxyHXL0QyOhDUogT8wjZj7ouSiVtWNB3

Prometheus Service

The following configuration needs to be added to the Prometheus server to fetch metrics from the controller's /api/v1/metrics/prometheus endpoint periodically.

scrape_configs:
- job_name: <job_name>
scheme: https
authorization:
type: Bearer
credentials: <access-token>
metrics_path: /api/v1/metrics/prometheus
scrape_interval: <interval>
static_configs:
- targets:
- <vip>:8443
tls_config:
insecure_skip_verify: true

The table below depicts the recommended configurations for this feature.

Please refer to the configuration guidelines of the specific Prometheus version you are using in the production.

Limitations

• Every device does not support every metric. Check the Metrics Summary section for more details.
• Software interfaces (for example, loopback, bond, and management) do not report counters for broadcast and unicast packets.
• The reported interface names are the raw physical interface name (e.g., et1) rather than the user configured name associated with the role of an interface (e.g., filter1).
• Resetting the interface counter does not have any effect on the counter values reported by the telemetry. The value is monotonically increasing and corresponds to the total count since the device was last powered up. This value only gets reset when the device is rebooted.

Notes

• The configured name of a managed device (e.g., switch, recorder node etc.) on the controller is used as the value of the device_name label for all the metrics corresponding to it. In the case of a controller, the configured hostname is used in the device_name label. Thus, these names are expected to be unique in a specific DMF deployment.
• It is possible that no metrics are collected from a device for a short time period. This may happen when the device is rebooting or when the controllers experience a failover event.
• Prometheus will add additional metrics, e.g., scrape_duration_seconds, scrape_samples_post_metric_relabeling, scrape_samples_scraped, and scrape_series_added. These metrics do not collect any data from the DMF fabric and can be ignored.
• A metric representing a status that can take a fixed set of values is represented as a StateSet metric. Each possible state is reported as a separate metric. The current state is reported with value 1 and the other states with value 0. The state itself is reported as a label with the same name as that of the metric. For example, device_psu_oper_status will display multiple metrics for the operational status of a PSU (Power Supply Unit). If a PSU, psu1 is in the failed state, then the metric device_psu_oper_status{name="psu1", device_psu_oper_status="failed"} will report value 1. At the same time, we will see the metric device_psu_oper_status{name="psu1", device_psu_oper_status="ok"} with value 0 for another state ok.
• Internally, all the metrics are fetched at 10 sec frequency except the ones associated with the sensors. These are currently collected at every minute.

Troubleshooting

If no metric is collected or no change in the metrics is visible on Prometheus for a few minutes, please follow the following troubleshooting steps:

• The controller cluster is reachable over its VIP and there is an elected active controller.
• You can retrieve the telemetry states in Prometheus exposition format using the token you created. Use command curl -k -H "Authorization: Bearer <token>" https://<vip>:8443/api/v1/metrics/prometheus
• The telemetry connection to the devices is active by running show telemetry connection command. Check the Telemetry Collector section of the DANZ Monitoring Fabric User Guide for more details.

Resources

• Prometheus - Monitoring system & time series database
• OpenMetrics Specification

Appendix

The following table shows the details of each metric exposed by the DMF fabric. The supported metric column shows that if a metric is generally supported on the device type. However, some specific platforms or hardware might not report a specific metric.

* Ctrl = Controller, SWL = A switch running Switch Light