DMF Recorder Node

This chapter describes configuring the DANZ Monitoring Fabric (DMF) Recorder Node (RN) to record packets from DMF filter interfaces. For related information, refer to the following:

Overview

The DANZ Monitoring Fabric (DMF) Recorder Node (RN) integrates with the DMF for single-pane-of-glass monitoring. A single DMF Controller can manage multiple RNs, delivering packets for recording through Out-of-Band policies. The DMF Controller also provides central APIs for packet queries across one or multiple RNs and for viewing errors, warnings, statistics, and the status of connected RNs.

A DMF out-of-band policy directs matching packets for recording to one or more RNs. An RN interface identifies the switch and port used to attach the RN to the fabric. A DMF policy treats these as delivery interfaces and adds them to the policy so that flows matching the policy are delivered to the specified RN interfaces.

Configuration Summary

At a high level, perform the following steps to configure the Recorder Node (RN).

Step 1: Add the RN instance to the DMF Controller.

Step 2: Define DMF policy to select the traffic to forward to the RN.

Step 3: View and analyze the recorded traffic.

Refer to the Recorder Node User Interface section and the Arista Analytics User Guide for more information.

Indexing Configuration

The Recorder Node (RN) indexing configuration defines the fields used to query packets on the RN. By default, DMF enables all indexing fields in the indexing configuration. Selectively disable the specific indexing fields not required in RN queries.

Disabling indexing fields has two advantages. First, it reduces the index space required for each packet recorded. Second, it improves query performance by reducing unnecessary overhead. Arista recommends disabling unnecessary indexing fields.

The RN supports the following indexing fields:
  • MAC Source
  • MAC Destination
  • VLAN 1: Outer VLAN ID
  • VLAN 2: Inner/Middle VLAN ID
  • VLAN 3: Innermost VLAN ID
  • IPv4 Source
  • IPv4 Destination
  • IPv6 Source
  • IPv6 Destination
  • IP protocol
  • Port Source
  • Port Destination
  • MPLS
  • Community ID
  • MetaWatch Device ID
  • MetaWatch Port ID
Note: Enable the Outer VLAN ID indexing field to query the RN using a DANZ Monitoring Fabric (DMF) policy name or a DMF filter interface name.

To understand leveraging an indexing configuration, consider the following examples:

Example 1: To query packets based on applications defined by unique transport ports, disable all indexing fields except source and destination transport ports, saving only transport ports as metadata for each packet recorded. This technique greatly reduces per-packet index space consumption and increases RN query speed.

However, this will impact an effective query on any other indexing field because that metadata was not saved when the packets were recorded.

Example 2: The RN supports community ID indexing, a hash of IP addresses, IP protocol, and transport ports that identify a flow of interest. Suppose the RN use case is to query based on community ID. In that case, indexing on IPv4 source and destination addresses, IPv6 source and destination addresses, IP protocol, and transport port source and destination addresses might be redundant.

Pre-buffer Configuration and Events

The Recorder Node (RN) pre-buffer is a circular buffer recording received packets. When enabled, the pre-buffer feature allows for the retention of the packets received by the RN for a specified length of time prior to an event that triggers the recording of buffered and future packets to disk. Without an event, the RN will record into this buffer, deleting the oldest packets when the buffer reaches capacity.

When an RN event is triggered, DMF saves packets in the pre-buffer to disk. The packets received from the time of the event trigger to the time of the event termination are saved directly to disk upon termination of the event. However, the received packets are also retained in the pre-buffer until the next event is triggered. By default, the pre-buffer feature is disabled, indicated by a value of zero minutes.

For example, when configuring the pre-buffer to thirty minutes, the buffer will receive up to thirty minutes of packets. When triggering an event, DMF records the packets currently in the buffer to disk, and packets newly received by the RN bypass the buffer and are written directly to disk until the termination of the event. When terminating the event, the pre-buffer resets, accumulating received packets for up to the defined thirty-minute pre-buffer size.

The packets affiliated with an event can be queried, replayed, or analyzed using any RN query. Each triggered event is identified by a unique, user-supplied name, used in the query to reference packets recorded in the pre-buffer before and during the event.

Using an Authentication Token

When using a DANZ Monitoring Fabric (DMF) Controller authentication token, the Recorder Node (RN) treats the DMF Controller as an ordinary client, requiring it to present valid credentials either in the form of an HTTP basic username and password or an authentication token.

Static authentication tokens are pushed to each RN as an alternative form of authentication in headless mode when the DMF Controller is unreachable or by third-party applications that do not have or do not need Controller credentials.

Recorder Node User Interface

Overview

DANZ Monitoring Fabric (DMF) 8.7.0 introduces a redesigned Recorder Node (RN) UI with an improved configuration workflow, monitoring page, and query features.

Common Features and Functions

Recorder Node Dashboard Layout

Features and Functions

There are several functions used in the DMF Recorder Node that are used throughout the user interface (UI).

These include:

 
Search and Filtering   Cancel
Column Sorting

Ascending or Descending

   Window Controls

Move, Minimize, Expand, Close
Refresh Data Immediately   Information
Create Policy   Edit
Collapse Extra Settings   Delete
Show Extra Settings   Error Condition
Create   Warning Condition
Unit Display

Bit Rate

Packet Rate

Utilization

   Expand Window
Export Data   Show or Hide Columns
Save

Settings or Configuration

      

Recorder Node Dashboard Layout

Active Queries: Expand the Active Queries widget to view ongoing queries.

Figure 1. Active Queries
Note: Active Queries fetch queries periodically; if a query completes quickly, the query may not appear in the widget.

Top Policy Utilization: The chart displays up to 5 top policies associating the Recorder Node interfaces with their respective bit rates.

Figure 2. Top Policy Utilization

Alerts: The Alerts widget displays Recorder Node warnings or errors. Any errors appear in the Alerts drop-down in the menu bar.

Figure 3. Alert Messages
Figure 4. Fabric Health Status

Top Filter Interfaces: Top Filter Interfaces displays the Top Filter Interfaces (up to five) attached to a policy with associated Recorder Node interfaces. Select the Unit drop-down to update the data with the selected unit type. The selection persists until changed. Hovering over a bar displays the throughput for the interface.

Figure 5. Top Filter Interfaces
Figure 6. Top Filter Interfaces - Units

Top Recorder Node Interfaces: Top Recorder Node Interfacesdisplays the Top Recorder Node Interfaces (up to five) attached to a policy with associated Recorder Node interfaces. Select the Unit drop-down to update the data with the selected unit type. The selection persists until changed. Hovering over a bar displays the throughput for the interface.

Figure 7. Top Recorder Node Interfaces

Inventory Table:

The Inventory Table is located in the lower section of the Recorder Nodes dashboard and displays all Recorder Node instances configured on the DMF Fabric. Selecting RN Interfaces displays all Recorder Node Interfaces configured in DMF.

Note: Even if a Recorder Node Interface is not configured, but the fabric detects a Recorder Node, it will show a partial entry in the Recorder Node Interfaces table.
Figure 8. Inventory - Recorder Nodes
Figure 9. Inventory - Recorder Node Interfaces

Select Edit to modify the Edit Recorder Node Interface or Edit Recorder Node configuration.

Figure 10. Edit Recorder Node Interface

 

Figure 11. Edit Recorder Node
Show / Hide Columns determine the information displayed in the dashboard and are user-selectable. Selections persist until changed.
Note: The Name column cannot be hidden.
Figure 12. Show / Hide Columns

Onboard a Recorder Node

To configure a Recorder Node (RN) or update the configuration of an existing RN, perform the following steps:
  1. Select Monitoring > Recorder Nodes from the main menu bar of the DANZ Monitoring Fabric (DMF) GUI.
    Figure 13. Recorder Nodes
  2. To add a new RN, select the Add Recorder Node (Recorder Nodes tab active) in the Inventory section.
    Figure 14. Provision Recorder Node
  3. Enter the following information in the required fields:
    • Assign a name to the RN.
    • Set the MAC Address of the RN. Obtain the MAC address from the chassis ID of the connected device, using the Fabric > Connected Devices .
      Figure 15. Connected Devices
  4. Configure the following options as needed:
    • Recording: Recording is enabled by default. To disable recording on the RN, move the Recording toggle switch to Off. When recording is enabled, the RN records the matching traffic directed from the filter interface defined in a DMF policy.
    • Disk Full Policy: The default packet removal policy is Continuous operating as FIFO (First In First Out), which means the oldest packets are deleted to make room for newer packets. This occurs only when the RN disks are full. The alternative removal policy is Stop, which causes the RN to stop recording when the disks are full and wait until disk space becomes available. Disk space can be made available by leveraging the RN delete operation to remove all or selected time ranges of recorded packets.
    • Backup Disk Policy: Specify the disk backup policy to as desired. Select from one of the three following options:
      • No Backup: This is the default option and is also the recommended option when no extra disk is available.
      • Remote Extend: In this option, recording is performed on the local disks. When full, the recording continues on a remote Isilon cluster mounted over NFS. In this mode, the remote disks are called backup disks. With regard to the Disk Full Policy, if set to:
        • Stop: Recording stops when both local and remote disks become full.
        • Continuous: When the configured threshold is reached, the oldest files from both disks are removed until the disk usage returns below the threshold number.
      • Local Fallback: In this option, recording is performed on a remote Isilon cluster mounted over NFS. If the connection between the Recorder Node and the remote cluster fails, the recording is performed on the local disks until the failure is resolved. In this mode, the local disks are called backup disks. With regard to the Disk Full Policy, if set to:
        • Stop: Recording stops when the remote disks become full.
        • Continuous: When the configured threshold is reached, the oldest files from both disks are removed until the disk usage returns below the threshold number.
      Note: A connection failure should not occur due to a misconfiguration of the NFS server on the DMF Controller. In such cases, the recording stops until the Controller’s configuration is fixed.
    • Max Packet Age: Change the Max Packet Age to set the maximum number of minutes that recorded packets are kept on the RN. Packets recorded are discarded after the specified number of minutes. This defines the maximum age in minutes of any packet in the RN. It can be used in combination with the Disk Full Policy to control when packets are deleted based on age rather than disk utilization alone. When unset, Max Packet Age is not enforced.
    • Incident Look Ahead Buffer: Assign the number of minutes the RN pre-buffer allows for windowed retention of packets received by the RN for a specified length of time. By default, the Incident Look Ahead Buffer is set to zero minutes (disabled). With a nonzero Incident Look Ahead Buffer setting and triggering a recorder event, any packets in the pre-buffer are saved to disk, and any packets received by the recorder after the trigger are saved directly to disk. When terminating an ongoing recorder event, a new pre-buffer is established in preparation for the next event.
    • Max Disk Utilization: Specify the maximum utilization allowed on the index and packet disks. The Disk Full Policy is enforced at this limit. If left unset, then the disks space are used to capacity.
    • Parse MetaWatch Trailer: Determine the parsing of the MetaWatch trailer.
      • Off: When set to Off, the RN will not parse the MetaWatch trailer, even if it is present in incoming packets.
      • Auto: When set to Auto, the RN will look for a valid timestamp in the last 12 bytes of the packet. If it matches the system timestamp closely enough, the trailer is parsed by the RN.
      • Always: When set to Always, RN will assume the last 12 bytes of packet is a MetaWatch trailer and parse it, even if it did not find a valid timestamp.
    • Indexing: All the indexing options are enabled by default. To disable any of the indexing behaviors, select Indexing and deselect from the list, as required. These include:
      • MAC Source
      • MAC Destination
      • VLAN 1
      • VLAN 2
      • VLAN 3
      • IPv4 Source
      • IPv4 Destination
      • IPv6 Source
      • IPv6 Destination
      • IP Protocol
      • Port Source
      • Port Destination
      • MPLS
      • Community ID
      • MetaWatch Device ID
      • MetaWatch Port ID

      For more details, see the Indexing Configuration section.

  5. Configure network and storage settings selecting Network & Storage Config.
    In order to store packets on external storage using an NFS mount, connect the Recorder Node's (RN) auxiliary interface to the same network and subnet where the NFS storage resides, as displayed in the figure below.
    Figure 16. Topology to Use External Storage
    Note: Create the volume for the index and packet on the NFS storage first. Refer to the vendor-specific NFS storage documentation about creating the volume (or path).
    Figure 17. Network & Storage Settings
    1. Auxiliary NIC Configuration: Move the toggle switch to On.
      Enter the IP Address and Mask. These are required fields.
      Figure 18. Auxiliary NIC Configuration
    2. Index Disk Configuration: Move the toggle switch to On.
      Enter the NFS Server as an IP address or hostname. For external NFS storage, such as Isilon, connect the auxiliary interface of the RN to a network and subnet that is reachable to Isilon NFS storage. When recording to an Isilon cluster over NFS with SmartConnect, the name of the storage pool can be specified. SmartConnect requires this name be registered as an A record pointing to the SmartConnect VIP in the DNS server used by the recorder node. The recorder will establish many mount points to the provided volume name in this storage pool. Each mount point will resolve to a different node in the storage pool, allowing the recorder to distribute recorded packets and metadata across multiple nodes in parallel.
      Enter the Volume: Specify the storage location or path on the NFS server for the Index Disk.
      Enter the Transport Port of NFS Service details. DMF uses the default value (2049) if no value is specified. Specify a value if the NFS storage has been configured to use something other than the default value.
      Figure 19. Index Disk Configuration
    3. Packet Disk Configuration: Move the toggle switch to On.
      Enter the NFS Server as an IP address or hostname. For external NFS storage, such as Isilon, connect the auxiliary interface of the RN to a network and subnet that is reachable to Isilon NFS storage. When recording to an Isilon cluster over NFS with SmartConnect, the name of the storage pool can be specified. SmartConnect requires this name be registered as an A record pointing to the SmartConnect VIP in the DNS server used by the recorder node. The recorder will establish many mount points to the provided volume name in this storage pool. Each mount point will resolve to a different node in the storage pool, allowing the recorder to distribute recorded packets and metadata across multiple nodes in parallel.
      Enter the Volume: Specify the storage location or path on the NFS server for the Packet Disk.
      Enter the Transport Port of NFS Service details. DMF uses the default value (2049) if no value is specified. Specify a value if the NFS storage has been configured to use something other than the default value.
      Figure 20. Packet Disk Configuration
  6. Select Add Recorder Node to save and close the configuration page.
    Figure 21. Provision Recorder Node-Indexing
Note:When editing the configuration of a previously added RN to use external storage versus local storage or vice versa, reboot the RN. 
dmf-controller-1># restart recorder-node rn-name application

Configure a Recorder Node Interface

To record packets to a recorder node using a DANZ Monitoring Fabric (DMF) policy, configure a DMF Recorder Node (RN) interface that defines the switch and interface in the monitoring fabric where the RN is connected. The DMF RN interface is referenced by name in the DMF policy as the destination for traffic matched by the policy. To configure a DMF RN interface, perform the following steps:
  1. To add a new RN interface, select the RN Interfaces tab in the Inventory section.
    Figure 22. Create DMF Recorder Node Interface
  2. Select Create RN Interface.
    Figure 23. Create Recorder Node Interface
  3. Enter or select the following parameters:
    • Recorder Node Interface Name: Assign a name to the interface.
    • Switch Name: Select a switch from the drop-down list containing the interface connecting the RN to the monitoring fabric.
    • Interface Name: Select an interface from the drop-down list connecting the RN to the monitoring fabric.
    • Description: (optional) Enter a description of the RN interface.
  4. Select Create RN Interface to add the configuration to the DMF Controller.

Edit a Recorder Node Interface

To edit a DMF RN interface, perform the following steps:
  1. Select the RN Interfaces tab in the Inventory section.
    Figure 24. Create DMF Recorder Node Interface
  2. Select an interface from the Inventory list and use the Edit icon.
    Figure 25. Create Recorder Node Interface
  3. Edit the following parameters:
    • Switch Name: Select a switch from the drop-down list containing the interface connecting the RN to the monitoring fabric.
    • Interface Name: Select an interface from the drop-down list connecting the RN to the monitoring fabric.
    • Description: (optional) Edit the description of the RN interface.
  4. Select Edit RN Interface to modify and update the interface configuration.
  5. To delete an interface, use the Delete icon and Confirm.
    Figure 26. Confirm Delete

Recorder Node Details

The Recorder Node Detail page displays specific information about the Recorder Node.

In the Inventory window (Recorder Nodes tab active), select the Recorder Node Name to navigate to the details page.

Figure 27. Recorder Nodes Details
The upper dashboard displays a brief summary of the status of the Recorder Node. A Recorder Nodes link returns to the Recorder Nodes dashboard and a drop-down speed navigation to other RN instances.
Figure 28. Brief Summary
The middle dashboard displays any active Alerts on the selected Recorder Node and Storage details:
Figure 29. Alerts and Storage
  • Storage
    • Disk Index
    • Backup Disk Index
    • Disk Packet
    • Backup Disk Packet
  • Index Mount
    • Volume, Mount, File System, and Health
  • Packet Mount
    • Volume, Mount, File System, and Health
  • Virtual Disk Health
    • Name, Size, State, Block Health, and Raid Level
    • Use the + icon to obtain granular information on the Virtual Disk Packet and Index health.
      • Slot #, Device ID, Type, Size, State, Temp, Predicted Failure Count, Media Error Count, and Other Error Count.
        Figure 30. Virtual Disk Health Details

The lower dashboard displays live graphs of the Recorder Node Ingress Rate, Recording Errors, and Packet Capture Frame Count, and Events.

Figure 31. Recorder Node Rates, Errors and Counts.

Hovering over the graph displays Timestamps and Ingress Rate.

Figure 32. Timestamps and Ingress Rate

Hovering over a bar displays Timestamps and Frame Count.

Figure 33. Timestamps and Frame Count
Show / Hide Additional Tabs determine the information displayed in the Event section. They are selectable and persist until changed. The UI displays up to four tabs.
Note: The Events table cannot be hidden.
Figure 34. Show / HideAdditional Tabs

 

Recorder Node Events

The Recorder Node Details dashboard displays Events in the lower section.

Figure 35. Recorder Nodes Details
Figure 36. Events Dashboard
Select Create Event to create an event. Enter the required information:
  • Name: Enter a name for the RN event.
  • Incident Look Ahead Buffer: Enter a value (in minutes).
Figure 37. Create Recorder Node Event
Select Create to create a Recorder Node Event.
Figure 38. Added Event

Use the X icon to stop the event and confirm.

Figure 39. Confirm Stop
Show / Hide Additional Tabs determine the information displayed in the Event section. They are selectable and persist until changed. The UI displays up to four tabs.
Note: The Events table cannot be hidden.
Figure 40. Show / Hide Additional Tabs
Additional tabs include:

CPU Core and Memory Info

The CPU Core and Memory Info dashboard displays Memory Info and CPU Status.

Figure 41. CPU Core and Memory Info
Memory Info summary information includes:
  • Collection Time
  • Total Bytes
  • Used Bytes
  • Free Bytes
  • Shared Bytes
  • Buffer Bytes
  • Cache Bytes
  • Available Bytes
CPU Status tabular information includes:
  • Core Number
  • Name
  • User Utilization
  • User Low Priority Utilization
  • Kernel Utilization
  • I/O Wait Utilization
  • Hard Interrupt Utilization
  • Soft Interrupt Utilization
  • Idle Utilization

Stenographer Info

The Stenographer Info dashboard displays Stenographer Info and Recording Threads.

Figure 42. CPU Core and Memory Info
Stenographer Info summary information includes:
  • Collection Time
  • Initialized
  • Tracked Files
  • Cached Files
  • Maximum Cached Files
Recording Threads tabular information includes:
  • Instance
  • Tracked Files
  • Cached Files
  • Maximum Cached Files

Recording Info

The Recording Info dashboard displays Recording Threads.
Figure 43. Recording Info
Recording Threads tabular information includes:
  • CPU Core
  • Disk
  • Dropped Packets
  • Total Packets

  • Collection Start Time

Assign a Recorder Node Interface to a Policy

To forward traffic to a Recorder Node (RN), include one or more RN interfaces as a delivery interface in a DANZ Monitoring Fabric (DMF) policy. Two methods exist to create a Policy:

Using Recorder Nodes to Create a Policy

Using Monitoring Policies to Create a Policy

 

 

Using Recorder Nodes to Create a Policy

When creating a new policy or editing an existing policy, select the RN interfaces from the Monitoring > Recorder Nodes .
Note: For more information on configuring Policies refer to the Managing DMF Policies section.
Figure 44. Recorder Nodes
To create a policy, select + Create Policy followed by Destination Tools > Add Ports(s) .
Figure 45. Recorder Node - Create Policy
Use RN Fabric Interface to select a previously configured RN interface. Select or drag the Interfaces or Recorder Nodes.
Figure 46. Selected Interface
Select Add n Interface to add to Destination Tools.
Figure 47. Destination Tools

Select Create Policy.

Note: The RN interface can only be selected and not created in the create policy dialogue.

For more information on configuring Policies refer to the Managing DMF Policies section.

Using Monitoring Policies to Create a Policy

When creating a new policy or editing an existing policy, select the RN interfaces from the Monitoring > Policies dialog, as shown in the following screen.
Note: For more information on configuring Policies refer to the Managing DMF Policies section.
Figure 48. DMF Policies
Note: If no RN fabric Interfaces appear, proceed to the Monitoring > Recorder Nodes > RN Interfaces tab in the Inventory section to create a RN interface.
To create a policy, select + Create Policy followed by Destination Tools > Add Ports(s) .
Figure 49. Recorder Node - Create Policy
Use RN Fabric Interface to select a previously configured RN interface. Select or drag the Interfaces or Recorder Nodes.
Figure 50. Selected Interface
Select Add n Interface to add to Destination Tools.
Figure 51. Destination Tools

Select Create Policy.

Note: The RN interface can only be selected and not created in the create policy dialogue.

For more information on configuring Policies refer to the Managing DMF Policies section.

Recorder Node Query

Use the options in the Query Recorder Nodes section to create a query and submit it to the RN for processing.

Initiate the Query Recorder Node workflow from the Recorder Nodes or Query History pages.

Figure 52. Recorder Nodes

Select Query Recorder Nodes to open the Query Recorder Node window.

Figure 53. Active Queries - Query Recorder Nodes
The Recorder Node (RN) records all the packets received on a filter interface that match the criteria defined in a DMF policy. The RN can recall or analyze recorded packets using various queries. Use the options described in the Query Recorder Node section to create a query and submit it to the RN for processing.
Figure 54. Query Action Parameters
The following query actions are supported:
  • Interval: Retrieve the oldest and most recent recorded packet timestamps. While hovering over the info icon provides information about the oldest and newest timestamp, to perform a query, you must enter the query time range using one of the following:
    • Quick Windows: A time range relative to the current time in which look for packets.
    • Select Range: A specific time range in which to look for packets.
  • Recorder Nodes: Select a single or multiple Recorder Nodes from the drop-down list.
  • IP Protocol: If required, select the IP protocol from the drop-down list or specify the numeric identifier of the protocol.
  • Size: Provides the number of packets and their aggregate size in bytes that match the filter criteria specified.
  • AppID: Performs deep packet inspection to identify applications communicating with the packets recorded and that match the filter criteria specified.
  • Packet Data: Retrieves all the packets that match the filter criteria specified.
  • Packet Objects: The packet object query extracts unencrypted HTTP objects from packets matching the given stenographer filter.
  • Replay: Replays selected packets and transmits them to the specified delivery interface.
  • Flow Analysis: Analyzes TCP flows for information such as maximum RTT, retransmissions, throughput, etc.
  • Traffic:
    • Any IP: Include packets with the specified IP address in the IP header (either source or destination).
    • Unidirectional: Include packets with the specified source and/or destination IP address in the IP header.
  • Traffic Pair - Source Destination:
    • IP/CIDR or Mac: Select packets with a specific source and destination IP or MAC address.
    • Src Port: Include packets with the specified protocol port number in the Src Port field in the IP header.
    • Dst Port: Include packets with the specified protocol port number in the Dst Port field in the IP header.
  • VLAN: Select packets with a specific VLAN ID.
    • Inner VLAN
    • Inner Inner VLAN
    • Outer VLAN
  • Filter Interfaces: Select the filter interfaces to restrict the query to those interfaces.
  • Policies: Select the policies to restrict the query to those policies.
  • Coalesce: Defines whether or not the data is coalesced if from multiple Recorder Nodes.
  • Fast Fail: For multi-packet-recorder queries, if one packet recorder fails, fail the entire query immediately. Otherwise, continue obtaining a partial result from the remaining packet recorders.
  • Timeout: Specify a timeout interval in Seconds, Minutes, or Hours.
  • Max Size: This option is only available for packet queries. Specify the maximum number of bytes returned by a packet query in a PCAP file.
  • Max Packets: This option is only available for packet queries. Specify the maximum number of packets returned by a packet query in a PCAP file.
  • Dedup time Window: Refer to Deduplicate Packets for more information.

To query the Recorder Nodes, enter the required information. Interval details are mandatory, while other fields are optional. The current computed Stenographer Query string displays under Query Preview.

Figure 55. Query Recorder Nodes

Hovering over the info icon for the Interval field displays the range of packets found.

Figure 56. Interval Details

The Replay query type has an additional mandatory field: Delivery Interfaces. Select Delivery Interfaces or Delivery Interface Groups from the drop-down.

Figure 57. Replay Fields
The Flow Analysis query type has an additional drop-down field of configuration parameters:
  • DNS: Analyzes any DNS packets, extracting query and response metadata.
  • HTTP: Analyzes HTTP packets, extracting request URLs, response codes, and statistics.
    • HTTP Request
    • HTTP Response
    • HTTP Stat
  • Hosts: Identifies all the unique hosts that match the filter criteria specified.
  • IPv4: Identifies and dissects distinct IPv4 flows.
  • IPv6: Identifies and dissects distinct IPv6 flows.
  • RTP Streams: Characterizes the performance of Real Time Protocol streaming packets.
  • SIP Correlate:
  • SIP Health:
  • TCP: Identifies and dissects distinct TCP flows.
  • TCP Flow Health: Analyzes TCP flows for information such as maximum RTT, retransmissions, throughput, etc.
  • UDP: Identifies and dissects distinct UDP flows.
Figure 58. Flow Analysis

Upon entering the information and selecting Query, the window closes, and an info notification appears. The Active Queries are populated (unless the query completes quickly).

Figure 59. Info Notification

When the query completes, a success notification appears. Selecting the link goes to the Query History page.

Figure 60. Success Message

Select Re-Query from the Query History page to prepopulate certain querying fields.

Figure 61. Re-Query Option
Note: The system does not repopulate all fields because it does not retain all of a query’s details. However, this entry point quickly modifies existing queries or speeds up commonly used queries.
Figure 62. Re-Query Configuration

Viewing Query History

Navigate to Monitoring > Recorder Nodes and scroll down to the Query History section.

Figure 63. Recorder Nodes

From Active Queries, select View Query History.

Figure 64. View Query History

The Query History dashboard appears.

Figure 65. Query Dashboard

 

While on the Query History page, use Recorder Nodes to return to the Recorder Nodes page.

Multiple Queries

To run queries on recorded packets by the RN, navigate to the Monitoring > Recorder Nodes page.

Under the Active Queries section, select Query Recorder Nodes to select the type of analysis to run on the recorded packets.

After selecting the query type, use filters to limit or narrow the search to obtain specific results. Providing specific filters also helps to complete the query analysis faster. In the following example, the query result for the TCP query type will return the results for IP address 10.240.30.24 for the past 15 minutes.

Figure 66. Query Type

After entering the desired filters, select Query. Query status windows appear.

To view the results select Active Queries. If the query has finished, view the results using Query History.

Select View Details under the More Options button.
Figure 67. More Options

 

Figure 68. Query Details

Query Results

To view Recorder Node Query Results, navigate to Query History.

Figure 69. Query History
Select View Details under the More Options button.
Figure 70. More Options

View Details displays the results of the query.

Figure 71. View Details
Selecting Download begins downloading the JSON data.
Figure 72. Downloaded Data

Deduplicate Packets

For Recorder Node queries, the recorded packets matching a specified query filter may contain duplicates when packet recording occurs at several different TAPs within the same network; i.e., as a packet moves through the network, it may be recorded multiple times. The dedup feature removes duplicate packets from the query results. By eliminating redundant information, packet deduplication improves query results' clarity, accuracy, and conciseness. Additionally, the dedup feature significantly reduces the size of query results obtained from packet query types.

Navigate to Monitoring > Recorder Nodes > Query Recorder Nodes .
Figure 73. Query Recorder Nodes

The Query Recorder Nodes configuration window appears.

Figure 74. Query Recorder Nodes

Deduplication is off by default for these queries. To enable deduplication, perform the following steps:

  1. Choose a Query Type.

    Packet deduplication is available for the Size, AppID, Packet Data, Packet Objects, Replay, and Flow Analysis query types.

  2. Specify a time window (in milliseconds) by entering an integer between 0 and 999 (inclusive) in the Dedup Time Window field.
  3. Select Query to continue.
The following example illustrates enabling deduplication for a Size query specifying a Dedup Time Window value of 200 ms.
Figure 75. Dedup Parameters

Manage Access to the Recorder Node

Use Role-Based Access Control (RBAC) to manage access to the DANZ Monitoring Fabric (DMF) Recorder Node (RN) by associating the RN with an RBAC group.

To restrict access for a specific RN to a specific RBAC group, use the following instructions.

RBAC Configuration

  1. Select Security > Groups , and select Edit from the Actions and select + Create Group.
    Figure 76. Create Security Group
  2. Enter a Group Name.
    Figure 77. Create Group
  3. Under the Role Based Access Control section select Add Recorder Node.
  4. Select the Recorder Node from the selection list, and assign the permissions required.
    • Read: The user can view recorded packets.
    • Use: The user can define and run queries.
    • Configure: The user can configure packet recorder instances and interfaces.
    • Export: The user can export packets to a different device.
    Figure 78. Associate Recorder Node
  5. Select Create.

Rename a RBAC Group

This topic describes the workflow for renaming a Group Name in DMF.

Overview

Navigate to Security > Groups and select Groups.

Figure 79. Security Groups

 

Figure 80. Groups Dashboard
Renaming a Group Name

To update a Group Name, such as changing test-group to test-group-updated, select Edit in the row menu action.

Figure 81. Edit

An Edit Group window displays.

Figure 82. Edit Group

Enter the new Group Name.

Figure 83. Updated Group Name

Select Save to apply the change.

DMF updates the Group Name.

Figure 84. Updated Group Name

Enabling Egress sFlow® on Recorder Node Interfaces

Enable egress sFlow®* to sample traffic sent to any DANZ Monitoring Fabric (DMF) Recorder Node (RN) attached to the fabric. Examining these sampled packets on a configured sFlow collector allows the identification of post-match-rule flows recorded by the RNs without performing a query against the RNs. While not explicitly required, Arista Networks highly recommends using the DMF Analytics Node (AN) as the configured sFlow collector, as it can automatically identify packets sampled utilizing this feature.

Platform Compatibility

All platforms apart from the following series:

  • DCS-7280R
  • DCS-7280R2
  • DCS-7500R
  • DCS-7020
  • DCS-7050X4

Configuration

After configuring the fabric for sFlow and setting up the sFlow collector, navigate to Monitoring > Recorder Nodes .

Figure 85. Query Recorder Nodes

Select Edit Configuration and the configuration menu appears.

Figure 86. sFlow Configuration

Set Enable sFlow to Yes.

Figure 87. sFlow Enabled

Select Save.

DMF Analytics Node

When using a DMF Analytics Node as the sFlow collector, it has a dashboard to display the results from this feature. To access the results:

  1. Navigate to the sFlow dashboard from the Fabric dashboard.
  2. Select the disabled RN Flows filter.
  3. Select the option to Re-enable the filter.
Figure 88. Re-enable sFlow

Troubleshooting Egress sFlow Configurations

Switches not affiliated with a sFlow collector (either a global sFlow collector or a switch-specific sFlow collector) do not have an active feature even if the feature is enabled. Ensure the fabric is set up for sFlow and a configured sFlow collector exists. To verify that a configured global sFlow collector exists, use the command:

Controller-1# show sflow default

A configured collector appears as an entry in the table under the column labeled collector. Alternatively, to verify a configured collector exists for a given switch, use the command:

Controller-1#show switch switch-name table sflow-collector

This command displays a table with one entry per configured collector.

A feature-unsupported-on-device warning appears when connecting an unsupported switch to an RN. The feature does not sample packets passing to an RN from an unsupported switch. View any such warnings using the GUI or using the following CLI command:

Controller-1#show fabric warnings feature-unsupported-on-device

To verify the feature is active on a given switch, use the command:

Controller-1#show switch switch-name table sflow-sample

If the feature is enabled, the entry values associated with the ports connected to an RN would include an EgressSamplingRate(number) with a number greater than 0. The following example illustrates Port(1) on switch-name connecting to an RN.

Controller-1# show switch <switch-name> table sflow-sample
#Sflow-sample Device nameEntry key Entry value
--|------------|---------------|---------|----------------------------------------------------------------------------------|
5352 <switch-name>Port(1) SamplingRate(0), EgressSamplingRate(10000), HeaderSize(128), Interval(10000)

Empty State

Users installing DMF 8.7.0 for the first-time who have not migrated from a previous DMF release will see an empty-state Recorder Nodes dashboard, as shown in the following example.
Figure 89. Recorder Node - Empty State

Please refer to the DMF Deployment Guide for more information on installing and configuring a DMF Recorder Node.

Using the Recorder Node Command Line Interface

 

Manage the DMF Recorder Node

 

Basic Configuration

To perform basic Recorder Node (RN) configuration, perform the following steps:
  1. Assign a name to the RN device. 
    controller-1(config)# recorder-node device rn-alias
  2. Set the MAC address of the RN. 
    controller-1(config-recorder-node)# mac 18:66:da:fb:6d:b4
    Determine from the chassis ID of connected devices if the management MAC is unknown.
  3. Define the RN interface name. 
    controller-1(config)# recorder-fabric interface Intf-alias
controller-1(config-pkt-rec-intf)#

    Assign any alphanumeric identifier for the recorder node interface name, which changes the submode to config-pkt-rec-intf, to provide an optional description. This submode allows specifying the switch and interface where the RN is connected.

  4. Provide an optional description and identify the switch interface connected to the RN. 
    controller-1(config-pkt-rec-intf)# description 'Delivery point for recorder-node'
controller-1(config-pkt-rec-intf)# recorder-interface switch Switch-z9100 ethernet37
  5. (Optional) Recording: Enabled by default. To disable recording, enter the following commands: 
    controller-1(config)# recorder-node device rn-alias
controller-1(config-recorder-node)# no record
  6. (Optional) Disk Full Policy: By default, Disk Full Policy is set to rolling-fifo, deleting the oldest packets to make room for newer packets when RN disks are full. This configuration can be changed to stop-and-wait, allowing the RN to stop recording until disk space becomes available. Enter the commands below to configure Disk Full Policy to stop-and-wait. 
    controller-1(config)# recorder-node device rn-alias
controller-1(config-recorder-node)# when-disk-full stop-and-wait
  7. Backup Disk Policy: Define the backup disk policy to select the secondary volume and select one of the following three options: 
    controller-1(config-recorder-node)# backup-volume
local-fallback Set local disk as backup when remote disk is unreachable
no-backupDo not use any backup volume (default selection)
remote-extendSet remote volume to extend local main disk
    The no-backup mode is the default mode. The other two modes require that the Recorder Node have a set of recording disks and a connection to an Isilon cluster mounted via NFS. Configure this remote storage from the DMF Controller.
  8. (Optional) Max Packet Age: This defines the maximum age in minutes of any packet in the RN. By default, Max Packet Age is unset, which means no limit is enforced. When setting a Max Packet Age, packets recorded on the RN are discarded after the minutes specified. To set the maximum number of minutes that recorded packets will be kept on the RN, enter the following commands: 
    controller-1(config)# recorder-node device rn-alias
controller-1(config-recorder-node)# max-packet-age 30
    This sets the maximum time to keep recorded packets to 30 minutes.
    Note: Combine Max Packet Age with the packet removal policy to control when packets are deleted based on age rather than disk utilization alone.
  9. (Optional) Max Disk Utilization: This defines the maximum disk utilization as a percentage between 5% and 95%. The Disk Full Policy (rolling-fifo or stop-and-wait) is enforced when reaching this value. If unset, the default maximum disk utilization is 95%; however, configure it, as required, using the following commands: 
    controller-1(config)# recorder-node device rn-alias
controller-1(config-recorder-node)# max-disk-utilization 80
  10. (Optional) Disable unused or unneeded indexing configuration fields in subsequent recorder node queries. DMF enables all indexing fields by default. To disable a specific indexing option, enter the following commands from the config-recorder-node-indexing submode. To re-enable a disabled option, enter the command without the no prefix.
    Use the following command to enter the RN indexing submode: 
    controller-1(config-recorder-node)# indexing
controller-1(config-recorder-node-indexing)#
    Use the following commands to disable any unused fields in subsequent queries:
    • Disable MAC Source indexing: no mac-src
    • Disable MAC Destination indexing: no mac-dst
    • Disable outer VLAN ID indexing: no vlan-1
    • Disable inner/middle VLAN ID indexing: no vlan-2
    • Disable innermost VLAN ID indexing: no vlan-3
    • Disable IPv4 Source indexing: no ipv4-src
    • Disable IPv4 Destination indexing: no ipv4-dst
    • Disable IPv6 Source indexing: no ipv6-src
    • Disable IPv6 Destination indexing: no ipv6-dst
    • Disable IP Protocol indexing: no ip-proto
    • Disable Port Source indexing: no port-src
    • Disable Port Destination indexing: no port-dst
    • Disable MPLS indexing: no mpls
    • Disable Community ID indexing: no community-id
    • Disable MetaWatch Device ID: no mw-device-id
    • Disable MetaWatch Port ID: no mw-port-id
    For example, the following command disables indexing for the destination MAC address: 
    controller-1(config-recorder-node-indexing)# no mac-src
  11. Identify the RN interface by name in an out-of-band policy. 
    controller-1(config)# policy RecorderNodePolicy
controller-1(config-policy)# use-recorder-fabric-interface intf-1
controller-1(config-policy)#
  12. Configure the DANZ Monitoring Fabric (DMF) policy to identify the traffic to send to the RN. 
    controller-1(config-policy)# 1 match any
controller-1(config-policy)# # filter-interface FilterInterface1
controller-1(config-policy)# # action forward
    This example forwards all traffic received in the monitoring fabric on filter interface FilterInterface1 to the RN interface. The following is the running-config for this example configuration: 
    recorder-fabric interface intf-1
description 'Delivery point for recorder-node'
recorder-interface switch 00:00:70:72:cf:c7:cd:7d ethernet37
policy RecorderNodePolicy
action forward
filter-interface FilterInterface1
use-recorder-fabric intf-1
1 match any

Authentication Token Configuration

Static authentication tokens are pushed to each Recorder Node (RN) as an alternative form of authentication in headless mode when the DANZ Monitoring Fabric (DMF) Controller is unreachable or by third-party applications that do not have or do not need DMF controller credentials to query the RN.

To configure the RN with a static authentication token, use the following commands: 
controller-1(config)# recorder-node auth token mytoken
Auth : mytoken
Token : some_secret_string <--- secret plaintext token displayed once here
controller-1 (config)# show running-config recorder-node auth token
! recorder-node
recorder-node auth token mytoken $2a$12$cwt4PvsPySXrmMLYA.Mnyus9DpQ/bydGWD4LEhNL6xhPpkKNLzqWS <---hashed token shows in running config
The DMF Controller uses its hidden authentication token to query the RN. To regenerate the Controller authentication token, use the following command: 
controller-1(config)# recorder-node auth generate-controller-token

Configuring the Pre-buffer

To enable the pre-buffer or change the time allocated, enter the following commands: 
controller-1(config)# recorder-node device name
controller-1(config-recorder-node)# pre-buffer minutes

Replace name with the recorder node name. Replace minutes with the number of minutes to allocate to the pre-buffer.

Triggering a Recorder Node Event

To trigger an event for a specific Recorder Node (RN), enter the following command from enable mode:

controller-1# trigger recorder-node name event event-name

Replace name with the RN name and replace event-name with the name to assign to the current event.

Terminating a Recorder Node Event

To terminate a Recorder Node (RN) event, use the following command: 
controller-1# terminate recorder-node name event event-name

Replace name with the RN name and replace event-name with the RN event name to terminate.

Viewing Recorder Node Events

To view recorder node events, enter the following command from enable mode: 
controller-1# show recorder-node events
# Packet Recorder Time Event
-|---------------|------------------------------|-------------------------------------------------------------------|
1 pkt-rec-740 2018-02-06 16:21:37.289000 UTC Pre-buffer event my-event1 complete. Duration 3 minute(s)
2 pkt-rec-740 2018-02-06 20:23:59.758000 UTC Pre-buffer event event2 complete. Duration 73 minute(s)
3 pkt-rec-740 2018-02-07 22:39:15.036000 UTC Pre-buffer event event-02-7/event3 complete. Duration 183 minute(s)
4 pkt-rec-740 2018-02-07 22:40:15.856000 UTC Pre-buffer event event5 triggered
5 pkt-rec-740 2018-02-07 22:40:16.125000 UTC Pre-buffer event event4/event-02-7 complete. Duration 1 minute(s)
6 pkt-rec-740 2018-02-22 06:53:10.216000 UTC Pre-buffer event triggered

Run Recorder Node Queries

Note: The DANZ Monitoring Fabric (DMF) Controller prompt is displayed immediately after entering a query or replay request, but the query continues in the background. Attempting to enter another replay or query command before the previous command is completed, an error message is displayed.

Packet Replay

Enter the replay recorder-node command from enable mode to replay the packets recorded by a Recorder Node (RN). 
controller-1# replay recorder-node name to-delivery interface filter stenographer-query
[realtime | replay-rate bps ]
The following are the options available with this command.
  • name: Specify the RN from which to replay the recorded packets.
  • interface: The DMF delivery interface name receiving the packets.
  • stenographer-query: The filter used to look up desired packets.
  • (Optional) real-time: Replay the packets at the original rate recorded by the specified RN. The absence of this parameter will result in a replay up to the line rate of the RN interface.
  • (Optional) replay-rate bps: Specify the number of bits per second used for replaying the packets recorded by the specified RN. The absence of this parameter will result in a replay up to the line rate of the RN interface.
The following command shows an example of a replay command using the to-delivery option. 
controller-1# replay recorder-node packet-rec-740 to-delivery eth26-del filter 'after 1m ago'
controller-1#
Replay policy details:
controller-1# show policy-flow | grep replay
1 __replay_131809296636625 packet-as5710-2 (00:00:70:72:cf:c7:cd:7d) 0 0 6400 1
in-port 47 apply: name=__replay_131809296636625 output: max-length=65535, port=26

Packet Data Query

Use a packet query to search the packets recorded by a specific Recorder Node (RN). The operation uses a Stenographer query string to filter only the interesting traffic. The query returns a URL to download and analyze the packets using Wireshark or other packet-analysis tools.

From enable mode, enter the query recorder-node command. 
switch # query recorder-node name packet-data filter stenographer-query
The following is the meaning of each parameter:
  • name: Identify the RN.
  • packet-data filter stenographer-query: Look up only the packets that match the specified Stenographer query.
The following example illustrates the results returned:

Packet Object Query

The packet object query extracts unencrypted HTTP objects from packets matching the given stenographer filter. To run a packet object query, run the following query command: 
switch# query recorder-node bmf-integrations-pr-1 packet-object filter 'after 5m ago'
The following example illustrates the results returned: 
switch# query recorder-node bmf-integrations-pr-1 packet-object filter 'after 1m ago'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Packet Object Query Results ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Coalesced URL : /pcap/__packet_recorder__/coalesced-bmf-2022-11-21-14-27-56-67a73ea9.tgz
Individual URL(s) : /pcap/__packet_recorder__/bmf-integrations-pr-1-2022-11-21-14-27-55-598f5ae7.tgz

Untar the folder to extract the HTTP objects.

Size Query

Use a size query to analyze the number of packets and the total size recorded by a specific Recorder Node (RN). The operation uses a Stenographer query string to filter only the interesting traffic.

Enter the query recorder-node command from enable mode to run a size query. 
# query recorder-node name size filter stenographer_query
The following is the meaning of each parameter:
  • name: Identify the RN.
  • size filter stenographer-query: Analyze only the packets that match the specified Stenographer query.
The following example illustrates the results returned: 
switch# query recorder-node hq-bmf-packet-recorder-1 size filter "after 1m ago and src host 8.8.8.8"
~ Summary Query Results ~
# Packets : 66
Size: 7.64KB
~ Error(s) ~
None.

Window Query

Use a window query to analyze the oldest and most recent available packets recorded by a specific Recorder Node (RN).

Enter the query recorder-node command from enable mode to run a window query.

switch# query recorder-node name window
The following is the meaning of each parameter:
  • name: Identify the RN.
The following example illustrates the results returned: 
switch# query recorder-node hq-bmf-packet-recorder-1 window
~~~~~~~~~~~~~ Window Query Results ~~~~~~~~~~~~~
Oldest Packet Available : 2020-07-30 05:01:08 PDT
Newest Packet Available : 2020-10-19 08:14:21 PDT
~ Error(s) ~
None.

Stopping a Query

Use the abort recorder-node command to stop the query running on the specified Recorder Node (RN). From enable mode, enter the following command: 
controller-1# abort recorder-node name filter string
Replace name with the RN name, and use the filter keyword to identify the specific filter used to submit the query. If the specific running query is unknown, use an empty-string filter of "" to terminate any running query. 
controller-1# abort recorder-node hq-bmf-packet-recorder-1 filter ""
Abort any request with the specified filter? This cannot be undone. enter "yes" (or "y") to
continue:
yes
Result : Success
~ Error(s) ~
None.

Viewing Query History

View Recorder Node (RN) submitted queries using the CLI.

To display query history, enter the following command: 
dmf-controller> show recorder-node query-history
# Packet Recorder QueryType StartDuration
---|---------------|--------------|------------------------|------------------------------|--------|
1 HW-PR-2 after 10m agoanalysis-hosts 2019-03-20 09:52:38.021000 PDT 3428
2 HW-PR-1 after 10m agoanalysis-hosts 2019-03-20 09:52:38.021000 PDT 3428
3 HW-PR-2 after 10m agoabort2019-03-20 09:52:40.439000 PDT 711
4 HW-PR-1 after 10m agoabort2019-03-20 09:52:40.439000 PDT 711
---------------------------------output truncated---------------------------------------------------

Using RBAC to Manage Access to the DMF Recorder Node

Use Role-Based Access Control (RBAC) to manage access to the DANZ Monitoring Fabric (DMF) Recorder Node (RN) by associating the RN with an RBAC group.

To restrict access for a specific RN to a specific RBAC group, use the CLI as described in the following instructions.

RBAC Configuration Using the CLI

  1. Identify the group to associate the Recorder Node (RN).
    Enter the following command from config mode on the active DANZ Monitoring Fabric (DMF) controller: 
    controller-1(config)# group test
controller-1(config-group)#
  2. Associate one or more RNs with the group.
    Enter the following CLI command from the config-group submode: 
    controller-1(config-group)# associate recorder-node device-name
    Replace device-name name with the RN name, as in the following example: 
    controller-1(config-group)# associate recorder-node HW-PR-1

View Information About a Recorder Node

This section describes monitoring and troubleshooting the Recorder Node (RN) status and operation. The RN stores packets on the main hard disk and the indices on the SSD volumes.

Viewing the Recorder Node Interface

To view information about the RN interface information, use the following command: 
controller-1(config)# show topology recorder-node
# DMF IF Switch IFName State SpeedRate Limit
-|------------|----------|----------|-----|------|----------|
1 RecNode-Intf Arista7050 ethernet1up25Gbps -

Viewing Recorder Node Operation

controller-1# show recorder-node device packet-rec-740 interfaces stats
Packet Recorder Name Rx Pkts Rx BytesRx DropRx Errors Tx PktsTx Bytes Tx Drop Tx Errors
---------------|----|-------------|---------------|--------|---------|--------|----------|-------|---------|
packet-rec-740pri1 2640908588614 172081747460802 84204084 0 24630503 3053932660 0 0
Information about a Recorder Node (RN) interface used as a delivery port in a DANZ Monitoring Fabric (DMF) out-of-band policy appears in a list. It lists RN interfaces as dynamically added delivery interfaces. 
Ctrl-2(config)# show policy PR-policy 
Policy Name                            : PR-policy
Config Status                          : active - forward
Runtime Status                         : installed
Detailed Status                        : installed - installed to forward
Priority                               : 100
Overlap Priority                       : 0
# of switches with filter interfaces   : 1
# of switches with delivery interfaces : 1
# of switches with service interfaces  : 0
# of filter interfaces                 : 1
# of delivery interfaces               : 1
# of core interfaces                   : 0
# of services                          : 0
# of pre service interfaces            : 0
# of post service interfaces           : 0
Push VLAN                              : 1
Post Match Filter Traffic              : 1.51Gbps
Total Delivery Rate                    : 1.51Gbps
Total Pre Service Rate                 : -
Total Post Service Rate                : -
Overlapping Policies                   : none
Component Policies                     : none
Installed Time                         : 2023-09-22 12:16:55 UTC
Installed Duration                     : 3 days, 4 hours
~ Match Rules ~
# Rule        
-|-----------|
1 1 match any

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Filter Interface(s)  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF      Switch              IF Name   State Dir Packets     Bytes          Pkt Rate Bit Rate Counter Reset Time             
-|-----------|-------------------|---------|-----|---|-----------|--------------|--------|--------|------------------------------|
1 Lab-traffic Arista-7050SX3-T3X5 ethernet7 up    rx  97831460642 51981008309480 382563   1.51Gbps 2023-09-22 12:16:55.738000 UTC

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Delivery Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF          Switch              IF Name    State Dir Packets     Bytes          Pkt Rate Bit Rate Counter Reset Time             
-|---------------|-------------------|----------|-----|---|-----------|--------------|--------|--------|------------------------------|
1 PR-intf Arista-7050SX3-T3X5 ethernet35 up    tx  97831460642 51981008309480 382563   1.51Gbps 2023-09-22 12:16:55.738000 UTC

~ Service Interface(s) ~
None.

~ Core Interface(s) ~
None.

~ Failed Path(s) ~
None.
Ctrl-2(config)# 

Viewing Errors and Warnings

The following table lists the errors and warnings a recorder node may display. In the CLI, display these errors and warnings by entering the following commands:
  • show fabric errors
  • show fabric warnings
  • show recorder-node errors
  • show recorder-node warnings
Table 1. Errors and Warnings
Type Condition Cause Resolution
Error Recorder Node (RN) management link down. RN has not received controller LLDP. Wait 30s if the recorder node is newly configured. Verify it is not connected to a switch port that is a DANZ Monitoring Fabric (DMF) interface.
Error RN fabric link down. Controller has not received RN LLDP. Wait 30s if recorder node is newly configured. Check it is online otherwise.
Warning Disk/RAID health degraded. Possible hardware degradation. Investigate specific warning reported. Could be temperature issue. Possibly replace indicated disk soon.
Warning Low disk space. Packet or index disk space has risen above threshold. Prepare for disk full soon.
Warning Disk full. Packet or index disk space is full. Packets are being dropped or rotated depending on removal policy. Do nothing if removal policy is rolling-FIFO. Consider erasing packets to free up space otherwise.
Warning Recorder misconfiguration on a DMF interface. A recorder node has been detected in the fabric on a switch interface that is configured as a filter or delivery interface. Remove the conflicting interface configuration, or re-cable the recorder node to a switch interface not defined as a filter or delivery interface.

Changing the Recorder Node Default Configuration

Configuration settings are automatically downloaded to the Recorder Node (RN) from the DANZ Monitoring Fabric (DMF) Controller, eliminating the need for box-by-box configuration. However, the option exists to override the default configuration for the RN from the config-recorder-node submode for any RN.
Note:These options are available only from the CLI, not the DMF Controller GUI.
To change the CLI mode to config-recorder-node, enter the following command from config mode on the active DMF controller: 
controller-1(config)# recorder-node device instance

Replace instance with the alias to use for the RN. This alias is affiliated with the MAC hardware address using the mac command.

Use any of the following commands from the config-recorder-node submode to override the default configuration for the associated RN:
  • banner: Set the RN pre-login banner message
  • mac: Configure the MAC address for the RN
Additionally, the option exists to override the configurations shown below to use values specific to the RN or used in a merge-mode along with the configuration inherited from the DMF controller:
  • ntp: Configure RN to override default timezone and NTP parameters.
  • snmp-server: Configure RN SNMP parameters and traps.
  • logging: Enable RN logging to Controller.
  • tacacs: Set TACACS defaults, server IP address(es), timeouts and keys.
Use the following commands from the config-recorder-node submode to change the default configuration on the RN:
  • ntp override-global: Override global time configuration with RN time configuration.
  • snmp-server override-global: Override global SNMP configuration with RN SNMP configuration.
  • snmp-server trap override-global: Override global SNMP trap configuration with RN SNMP trap configuration.
  • logging override-global: Override global logging configuration with packet recorder logging configuration.
  • tacacs override-global: Override global TACACS configuration with RN TACACS configuration.
To configure the RN to work in a merge mode by merging its specific configuration with that of the DMF Controller, execute the following commands in the config-recorder-node submode:
  • ntp merge-global: Merge global time configuration with RN time configuration.
  • snmp-server merge-global: Merge global SNMP configuration with RN SNMP configuration.
  • snmp-server trap merge-global: Merge global SNMP trap configuration with RN SNMP trap configuration.
  • logging merge-global: Merge global logging configuration with RN logging configuration.

TACACS configuration does not have a merge option. It can either be inherited from the DMF Controller or overridden to use only the RN-specific configuration.

Large PCAP Queries

Access the RN via a web browser to run large PCAP queries to the Recorder Node (RN). This allows running packet queries directly to the RN without specifying the maximum byte or packet limit for the PCAP file (which is required when executing the query from the DANZ Monitoring Fabric (DMF) Controller).

To access the RN directly, use the URL https://RecorderNodeIP in a web browser, as shown below:
Figure 90. URL to Recorder Node
The following page will be displayed:
Figure 91. Recorder Node Page
  • Recorder Node IP Address: Enter the target RN IP address.
  • DMF Controller Username: Provide the DMF Controller username.
  • DMF Controller Password: Provide the password for authentication.
  • Stenographer Query Filter: Use the query filter to filter the query results to look for specific packets. For example, to search for packets with a source IP address of 10.0.0.145 in the last 10 minutes, use the following filter: 
    after 10m ago and src host 10.0.0.145
  • Stenographer Query ID: Starting in DMF 8.0, a Universally Unique Identifier (UUID) is required to run queries. To generate a UUID, run the following command on any Linux machine and use the result as the Stenographer query ID: 
    $ uuidgen
b01308db-65f2-4d7c-b884-bb908d111400
  • Save pcap as: Provide the file name for this PCAP query result.
  • Submit Request: Sends a query to the specified RN and saves the PCAP file with the provided file name to the default download location for the browser.

Recorder Node Management Migration L3ZTN

After completing the first boot (initial configuration), remove the Recorder Node (RN) from the old Controller and point it to a new Controller via the CLI in the case of a Layer-3 topology mode.
Note:For appliances to connect to the DANZ Monitoring Fabric (DMF) Controller in Layer-3 Zero Touch Network (L3ZTN) mode, configure the DMF Controller deployment mode as pre-configure.

To migrate management to a new Controller, follow the steps below:

  1. Remove the RN and switch from the old Controller using the commands below: 
    controller-1(config)# no recorder-node device RecNode
    controller-1(config)# no switch Arista7050
  2. Add the switch to the new Controller.
  3. SSH to the RN and configure the new Controller IP using the zerotouch l3ztn controller-ip command: 
    controller-1(config)# zerotouch l3ztn controller-ip 10.2.0.151
  4. After pointing the RN to use the new Controller, reboot the RN.
  5. Once the RN is back online, the DMF Controller receives the ZTN request.

  6. After the DMF Controller has received a ZTN request from the RN, add it to the DMF Controller running-configuration using the below command: 
    controller-1(config)# recorder-node device RecNode
controller-1(config-recorder-node)# mac 24:6e:96:78:58:b4
  7. Verify the addition of the RN to the new DMF Controller using the command below:

Recorder Node Show Commands

The following commands are available from the Recorder Node (RN):

Use the show version command to view the version and image information that RN is running on. 
RecNode(config)# show version
Controller Version : DMF Recorder Node 8.1.0 (bigswitch/enable/dmf-8.1.x #5)
RecNode(config)#
Use the show controllers command to view the connected DANZ Monitoring Fabric (DMF) controllers to the recorder node.
Note: All cluster nodes appear in the command output if the RN is connected to a DMF Controller cluster.
RecNode(config)# show controllers
controllerRole State Aux
---------------------|------|---------|---|
tcp://10.106.8.2:6653 master connected 0
tcp://10.106.8.3:6653 slaveconnected 0
tcp://10.106.8.3:6653 slaveconnected 1
tcp://10.106.8.3:6653 slaveconnected 2
tcp://10.106.8.2:6653 master connected 1
tcp://10.106.8.2:6653 master connected 2
RecNode(config)#

Ability to Deduplicate Packets - Query from Recorder Node

For Recorder Node queries, the recorded packets matching a specified query filter may contain duplicates when packet recording occurs at several different TAPs within the same network; i.e., as a packet moves through the network, it may be recorded multiple times. The dedup feature removes duplicate packets from the query results. By eliminating redundant information, packet deduplication improves query results' clarity, accuracy, and conciseness. Additionally, the dedup feature significantly reduces the size of query results obtained from packet query types.

Deduplicate Packets

In the DANZ Monitoring Fabric (DMF) Controller CLI, packet deduplication is available for the packet data, packet object, size, and replay query types. Deduplication is off by default for these queries. Add the dedup option to the end of the query command after all optional values (if any) have been selected to enable deduplication.

The following are command examples of enabling deduplication.

Enabling deduplication for a size query:

controller# query recorder-node rn size filter “before 5s ago” dedup

Enabling deduplication for a packet data query specifying a limit for the size of the PCAP file returned in bytes:

controller# query recorder-node rn packet-data filter “before 5s ago” limit-bytes 2000 dedup

Enabling deduplication for a replay query:

controller# replay recorder-node rn to-delivery dintf filter “before 5s ago” dedup

Enabling deduplication for a replay query specifying the replay rate:

controller# replay recorder-node rn to-delivery dintf filter “before 5s ago” replay-rate 100 dedup

Specify a time window (in milliseconds) for deduplication. The time window defines the time required between timestamps of identical packets to no longer be considered duplicates of each other. For example, for a time window of 200 ms, two identical packets with timestamps that are 200 ms (or less) apart are duplicates of each other. In contrast, if the two identical packets had timestamps more than 200 ms apart, they would not be duplicates of each other.

The time window must be an integer between 0 and 999 (inclusive) with a default time window of 200 ms when deduplication is enabled and no set time window value.

To configure a time window value, use the dedup-window option followed by an integer value for the time window after the dedup option.

controller# query recorder-node rn size filter “before 5s ago” dedup dedup-window 150

Enable Egress sFlow

Enable egress sFlow®* to sample traffic sent to any DANZ Monitoring Fabric (DMF) Recorder Node (RN) attached to the fabric. Examining these sampled packets on a configured sFlow collector allows the identification of post-match-rule flows recorded by the RNs without performing a query against the RNs. While not explicitly required, Arista Networks highly recommends using the DMF Analytics Node (AN) as the configured sFlow collector, as it can automatically identify packets sampled utilizing this feature.

Platform Compatibility

All platforms apart from the following series:

  • DCS-7280R
  • DCS-7280R2
  • DCS-7500R
  • DCS-7020
  • DCS-7050X4

Configuration

The egress sFlow feature requires a configured sFlow collector. After configuring the sFlow collector, enter the following command from the config mode to enable the feature:

Controller-1(config)# recorder-node sflow

To disable the feature, enter the command:

Controller-1(config)# no recorder-node sflow

Considerations and Limitations

 

Deduplication Limitations

Expect a query with packet deduplication enabled to take longer to complete than packet deduplication disabled. Hence, packet deduplication, by default, is off.

The maximum time window value permitted is 999 ms to ensure that TCP retransmissions are not regarded as duplicates, assuming that the receive timeout value for TCP retransmissions (of any kind) is at least 1 second. If the receive timeout value is less than 1 second (particularly, exactly 999 ms or less), then it is possible for TCP retransmissions to be regarded as duplicates when the time window value used is larger than the receive timeout value.

Due to memory constraints, removing some duplicates may not occur as expected. This scenario is likely to occur if a substantial amount of packets match the query filter, which all have timestamps within the specified time window from each other. We refer to this scenario as the query having exceeded the packet window capacity. To mitigate this from occurring, decrease the time window value or use a more specific query filter to reduce the number of packets matching the query filter at a given time.

Guidelines and Limitations for Enabling Egress sFlow

Consider the following guidelines and limitations while enabling Egress sFlow:

  • The Egress sFlow support for the Recorder Nodes (RN) feature requires a configured sFlow collector in a fabric configured to allow sFlows.
  • If a packet enters a switch through a filter interface with sFlow enabled and exits through a port connected to an RN while the feature is enabled, only one sFlow packet (i.e., the ingress sFlow packet) is sent to the collector.
  • The Egress sFlow feature does not identify which RN has recorded a given packet in a fabric when there are multiple RNs. This is fine in a normal case as the queries are issued to the RNs in aggregate rather than to individual RNs, and hence, the information that any RN has received a packet is sufficient. In some cases, it may be possible to make that determination from the outport of the sFlow packet, but that information may not be available in all cases. This is an inherent limitation of egress sFlow.
  • An enabled egress sFlow feature captures the packets sent to any RN with recording enabled, regardless of whether the RN is actively recording or not.

Recorder Node Recording State API Limitations

The ready state only occurs after the recording application has finished initializing if no recordable traffic has been received yet. The recording application must undergo its initialization process whenever the RN is rebooted, restarted, or after restarting the RN application from the DMF Controller. If the RN is in the active state and stops receiving packets, it will not regress into the ready state; it will remain in the active state.

*sFlow® is a registered trademark of Inmon Corp.
*sFlow® is a registered trademark of Inmon Corp.