Watcher Alerts

The DMF 8.7.0 release supports the following connector integrations:

• Gmail via the Email Connector
• Google Chat and Slack via the Webhook Connector.

HTTP header Content-Type is mandatory with the value application/json; charset=UTF-8.

Under Test > Create an action > Body enter the following test:

{"text": "Message from Kibana Connector"}

Ensure the result is successful and confirm that the respective chat window receives the message.

The screen shows the following message.

Under Test > Create an action > Body enter the following test:

{"text": "Message from Kibana Connector"}

The screen shows the following message.

A sample watcher script is as follows with an action type of Google Chat and Slack using the Webhook connectors.

"webhook_googlechat": {
 "webhook": {
 "scheme": "http",
 "host": "169.254.16.1",
 "port": 8000,
 "method": "post",
 "params": {},
 "headers": {},
 "body": """{"request_body": "{\"text\": \"The Elasticsearch cluster status is {{ctx.payload.status}}.\"}","kibana_webhook_connector": "<google-chat-webhook-connector-name>"}"""
 }

• Add a webhook action with the following fields to select Google Chat and Slack Webhook Connectors.
  • Method: POST
  • Scheme: HTTP
  • Host: 169.254.16.1
  • Port: 8000
  • Specify the Body field as follows:
    • kibana_webhook_connector: The name of the Kibana connector (of type webhook) in string format. It is case-sensitive.
    • request_body: Enter the required fields: The connector gets the HTTP request body (in string format as text).
      Note: request_body value is specific to the connector’s specification.

Google Chat webhook accepts JSON formatted body with a mandatory text field. Hence, request body must have a JSON formatted string with text field. For example,

"{\"text\": \"The Elasticsearch cluster status is {{ctx.payload.status}}.\"}"

A sample watcher script with action type Google Chat Webhook is:

{
 "trigger": {
 "schedule": {
 "interval": "1h"
 }
 },
 "input": {
 "http": {
 "request": {
 "scheme": "https",
 "host": "10.10.10.10",
 "port": 443,
 "method": "get",
 "path": "/es/api/_cluster/health",
 "params": {},
 "headers": {
 "Content-Type": "application/json"
 },
 "auth": {
 "basic": {
 "username": "admin",
 "password": "::es_redacted::"
 }
 }
 }
 }
 },
 "condition": {
 "script": {
 "source": "ctx.payload.status != 'green'",
 "lang": "painless"
 }
 },
 "actions": {
 "webhook_googlechat": {
 "webhook": {
 "scheme": "http",
 "host": "169.254.16.1",
 "port": 8000,
 "method": "post",
 "params": {},
 "headers": {},
 "body": """{"request_body": "{\"text\": \"The Elasticsearch cluster status is {{ctx.payload.status}}.\"}","kibana_webhook_connector": "BSN-Analytics-AppTest-GH-connector"}"""
 }
 }
 }
}

Slack webhook accepts JSON formatted body with fields:

• text: (mandatory)
• channel : (optional) The channel name must match the same Slack channel for which the webhook is enabled.
• username: (optional) You can select any username.

For Example:

"{\"channel\": \"test-channel\", \"username\": \"webhookbot\", \"text\": \"The Elasticsearch cluster status is {{ctx.payload.status}}.\"}"

{
 "trigger": {
 "schedule": {
 "interval": "1m"
 }
 },
 "input": {
 "http": {
 "request": {
 "scheme": "https",
 "host": "10.10.10.1",
 "port": 443,
 "method": "get",
 "path": "/es/api/_cluster/health",
 "params": {},
 "headers": {
 "Content-Type": "application/json"
 },
 "auth": {
 "basic": {
 "username": "admin",
 "password": "::es_redacted::"
 }
 }
 }
 }
 },
 "condition": {
 "script": {
 "source": "ctx.payload.status != 'green'",
 "lang": "painless"
 }
 },
 "actions": {
 "webhook_slack": {
 "webhook": {
 "scheme": "http",
 "host": "169.254.16.1",
 "port": 8000,
 "method": "post",
 "params": {},
 "headers": {},
 "body": """{"request_body": "{\"channel\": \"test-webhook\", \"username\": \"webhookbot\", \"text\": \"The Elasticsearch cluster status is {{ctx.payload.status}}.\"}","kibana_webhook_connector": "test-slack-webhook-connector"}"""
 }
 }
 }
}

If the Watcher is not sending an alert to the configured Webhook connector when the condition is already there:

• Check Kibana Connector, which is a type of Webhook.
• Check that the Kibana Connector is properly configured by running tests from UI. Check connector-specific configurations.
• Check for properly configured Watcher’s trigger conditions.
• Make sure all required parameters are present in the connector's watcher configuration.
• To debug execution in CLI:
  • SSH to AN node
  • Log in to CLI mode command: debug bash
  • Review logs in /var/log/analytics/webhook_listener.log for any clues. Command: tail -f /var/log/analytics/webhook_listener.log
  • To execute service in debug mode:
    • Login as root command: sudo su
    • Stop service command: service webhook-listener stop.
    • Edit web-service in your preferred editor and set the logger to debug mode command: vi /usr/lib/python3.9/site-packages/webhook_listener/run_webhook_listener.py Change Line: LOGGER.setLevel(logging.INFO) to LOGGER.setLevel(logging.DEBUG).
    • Start service command: service webhook-listener start.