• Configure a stream job using topic_indexer. Access the setting via the Kibana dashboard in the analytics node.
• Locate the topic_indexer configuration on the Fabric Dashboard: Analytics > Fabric > System > Analytics Configuration, as shown in the following screenshots.

• 

• Another view:

  

• The configuration for a topic to be consumed can be specified as described in the design section.

• 

• To perform the topic_indexer configuration, select the System > Configuration > Fabric page and open the Analytics Configuration panel:

  

• 

• topic: Kafka topic name; type string and is a mandatory field.
• broker_address: Broker address(es), this is of type array; this will be of format [IPv4|hostname:Port number] and is a mandatory field.
• consumer_group: This is an optional field; however, there is always a consumer group if not specified explicitly in the configuration. We create it as topic_name + index_name. Setting this field will be particularly useful when ingesting multi-partitioned topics from the client's end.
• index: A dedicated index name we want for the topic; type string. In Elastic Search (ES), it will be created as topic_indexer_<index_name> and is a mandatory field.
• field_group: An optional JSON field mapping to specify any column rename/format transformations.

  Specifies format for modifications to incoming data.

• type: This is to set timestamp-field as the type.
• source_key: This is the source field name in the incoming data.
• indexed_key: This is the destination field name inserted in the outgoing ES index.

  The indexed_key may be a @timestamp field of an ES index. If we do not specify a @timestamp field, topic_indexer automatically picks the time the message was received as the @timestamp of that message.

• format: Data format for the field (ISO8601).

• my-topic-name
• my_topic_name
• itlabs.mytopic.name
• topic123
• 123topic
• my-index-name

• myTopicName
• ITLabs-Website-Tracker
• 12435
• MY-Index-name

• A broker address in Kafka comprises two values: IPv4 address and Port Number.
• While entering the broker address, use the format: IPv4:PORT.

POST <stream-name>/_rollover 

• Cross-index visualization - two data streams that need cross-querying:

• 

• Step 1. To view the documents in these indexes, create an index pattern (e.g., topic*spend) in Kibana.
• Step 2. View the data in the Discover dashboard.

  

• Step 3. Create a common field (runtime field) between the two data streams by applying an API in Dev Tools.

  

  Note: Setting rollover policy on runtime fields can also be done in Dev Tools, as shown in the following examples:

  POST /topic-indexer-service-spend/_rollover
  POST /topic-indexer-product-spend/_rollover

  Note: These changes are not persistent. You must reapply them after any restart of AN.
• Step 4. Finally, create a visualization using this common field, for example, Customer. The illustration below shows the Top 5 customers with the highest spending across products and services.

  

an> debug bash
admin@an$ cd /var/log/analytics/
admin@an:/var/log/analytics$
admin@an:/var/log/analytics$ ls -ls topic_indexer.log
67832 -rw-rwxr-- 1 remoteuser root 69453632 Apr 27 11:05 topic_indexer.log

1. The Save button in Topic Indexer Config is disabled.
   When editing the configurations of topic_indexer in the Kibana User interface, default validations appear to ensure the correctness of the values entered in the fields. Specific standards and requirements are associated when filling in the config for topic indexer as stated in the above section linked: Topic Indexer on Arista Analytics . As illustrated below, you may encounter validation errors when entering an invalid value in the configuration field. Topic Indexer on Arista Analytics

   

   In such an event, the edited configuration will not Save. Therefore, before saving the configuration, validate the fields and ensure there is no visible validation error in the topic_indexer configuration editor.

2. Index for topic indexer is not created.

   After entering the correct fields in the topic_indexer configuration, the topic_indexer service will start to read the Kafka topic as documented in the configuration and load its data into the ElasticSearch index entered by the index field. The name of the index is prefixed by topic_indexer_.

   There is a wait time of several minutes before the index is created and loaded with the data from the Kafka topic. In the event the index is not created, or there is no index shown with the name topic_indexer_<index_name> value, Arista Networks recommends using the following troubleshooting steps:

   1. Check the configurations entered in the topic_indexer editor once again to see whether the spellings for the topic name, broker address configuration, and index name are correct.
   2. Verify the broker address and the port for the Kafka topic are open on the firewall. Kafka has a concept of listeners and advertised.listeners. Validate if the advertised.listeners are entered correctly in the configuration. For more details, please review the following links:
      1. Kafka 3.5 Documentation
      2. Kafka Listeners – Explained | Confluent
   3. If all the above steps are correct, you should now check the logs in the Analytics Node for the topic indexer.
      Steps to reach the topic_indexer.log file in AN node:

      1. Secure remote access into the AN using the command line: ssh <user>@<an-ip>
      2. Enter the password for the designated user
      3. Enter the command debug bash to enter into debug mode
      4. Use the sudo user role when entering into the AN node: hence the sudo su command.
      5. Topic indexer logs reside in the following path: /var/log/analytics/topic_indexer.log
      6. Since this log file can be more extensive, you should use the tail command.
      7. Validate if the log file shows any visible errors related to the index not being created.
      8. Report any unknown issues.
3. Data is not indexed as per the configuration.
4. Data ingestion is paused.

   When experiencing issues 3 or 4 (described above), use the topic_indexer log file to validate the problem.

5. Index pattern for topic indexer is missing.

   In the Kibana UI, we create a default topic_indexer_* index pattern. If this pattern or a pattern to fetch the dedicated index for a topic is missing, create it using the Kibana UI as described in the following link:

   Create an index pattern | Kibana Guide [7.17] | Elastic