• Configure a stream job using topic_indexer. Access the setting via the Kibana dashboard in the analytics node.
• Locate the topic_indexer configuration on the Fabric Dashboard: Analytics > Fabric > System > Analytics Configuration, as shown in the following screenshots.

• 

• Another view:

  

• The design section shows the configuration for a topic.

• 

• To perform the topic_indexer configuration, select the System > Configuration > Fabric page and open the Analytics Configuration panel:

  

• 

• topic: Kafka topic name; type string and is a mandatory field.
• broker_address: Broker address(es), this is of type array; this will be of format [IPv4|hostname:Port number] and is a mandatory field.
• consumer_group: This is an optional field; however, there is always a consumer group if not specified explicitly in the configuration. It is topic_name + index_name. Setting this field is particularly useful when ingesting multi-partitioned topics from the client's end.
• index: A dedicated index name for the topic; type string. In Elastic Search (ES), it is created as topic_indexer_<index_name> and is a mandatory field.
• field_group: An optional JSON field mapping to specify any column rename/format transformations. It specifies the format for modifications to incoming data.
• type: To set the timestamp field as the type.
• source_key: The source field name in the incoming data.
• indexed_key: The destination field name inserted in the outgoing ES index.

  The indexed_key may be a @timestamp field of an ES index. If you do not specify a @timestamp field, topic_indexer automatically picks the received time of the message as the @timestamp of that message.

• format: Data format for the field (ISO8601).

• my-topic-name
• my_topic_name
• itlabs.mytopic.name
• topic123
• 123topic
• my-index-name

• myTopicName
• ITLabs-Website-Tracker
• 12435
• MY-Index-name

• A broker address in Kafka comprises two values: IPv4 address and Port Number.
• When entering the broker address, select the format IPv4:PORT.

POST <stream-name>/_rollover 

• Cross-index visualization - two data streams that need cross-querying:

• 

• Step 1. To view the documents in these indexes, create an index pattern (e.g., topic*spend) in Kibana.
• Step 2. View the data in the Discover dashboard.

  

• Step 3. Create a common field (runtime field) between the two data streams by applying an API in Dev Tools.

  

  Note: Setting rollover policy on runtime fields can also be done in Dev Tools, as shown in the following examples:

  POST /topic-indexer-service-spend/_rollover
  POST /topic-indexer-product-spend/_rollover

  Note: These changes are not persistent. Reapply is a must after any restart of AN.
• Step 4. Finally, create a visualization using this common field, for example, Customer. The following illustration shows the Top 5 customers with the highest spending across products and services.

  

an> debug bash
admin@an$ cd /var/log/analytics/
admin@an:/var/log/analytics$
admin@an:/var/log/analytics$ ls -ls topic_indexer.log
67832 -rw-rwxr-- 1 remoteuser root 69453632 Apr 27 11:05 topic_indexer.log

1. The Save button in the topic_indexer config is disabled.
   When editing the configurations of topic_indexer in the Kibana User interface, default validations appear to ensure the correctness of the values entered in the fields. Specific standards and requirements are associated with filling in the configuration for topic_indexer, as stated in the earlier section. It may encounter validation errors when entering an invalid value in the configuration field as follows.

   

   In such an event, the edited configuration will not save. Therefore, before saving the configuration, validate the fields and ensure there is no visible validation error in the topic_indexer configuration editor.

2. The index for the topic_indexer has not been created.

   After entering the correct fields in the topic_indexer configuration, the topic_indexer service will start to read the Kafka topic as documented in the configuration and load its data into the ElasticSearch index entered by the index field. The topic_indexer_ prefixes the name of the index.

   There is a wait time of several minutes before the index is created and loaded with the data from the Kafka topic. In the event the index is not created, or there is no index shown with the name topic_indexer_<index_name> value, Arista Networks recommends using the following troubleshooting steps:

   1. Check the configurations entered in the topic_indexer editor again to see whether the spellings of the topic name, broker address configuration, and index name are correct.
   2. Verify the broker address and the port for the Kafka topic are open on the firewall. Kafka has a concept of listeners and advertised.listeners . Validate if the advertised.listeners are entered correctly into the configuration. Review the following links for more details:
      1. Kafka 3.5 Documentation
      2. Kafka Listeners – Explained | Confluent
   3. If all the earlier steps are correct, check now for the logs in the Analytics Node for the topic_indexer.
      Steps to reach the topic_indexer.log file in the AN node:

      1. Secure remote access into the AN using the command line: ssh <user>@<an-ip>
      2. Enter the password for the designated user.
      3. Enter the command debug bash to enter into debug mode.
      4. Use the sudo user role when entering the AN node, hence the sudo su command.
      5. topic_indexer logs reside in the following path: /var/log/analytics/topic_indexer.log
      6. Since this log file can be more extensive, you should use the tail command.
      7. Validate if the log file shows any visible errors related to the index not being created.
      8. Report any unknown issues.
3. Data is not indexed as per the configuration.
4. Data ingestion is paused.

   When experiencing issues 3 or 4 (described earlier), use the topic_indexer log file to validate the problem.

5. The index pattern for the topic_indexer is missing.

   In the Kibana UI, it creates a default topic_indexer_* index pattern. If this pattern or a pattern to fetch the dedicated index for a topic is missing, create it using the Kibana UI as described in the following link:

   https://www.elastic.co/guide/en/kibana/8.15/data-views.html