Link

Elastic APM is an application performance monitoring system built on the Elastic Stack. It allows one to monitor software services and applications in real-time, by collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more.

Open Distro for Elasticsearch provides a powerful, easy-to-use event monitoring and alerting system, enabling you to monitor your data and send notifications automatically to your stakeholders. With an intuitive Kibana interface and powerful API, it is easy to set up and manage alerts. Build specific alert conditions using Elasticsearch’s query and scripting capabilities. Alerts help teams reduce response times for operational and security events.

With the Zenduty-OpenDistro integration, you will be able to create new Incidents/Alerts in Zenduty whenener any rule in Opendistro’s Monitors are breached/violated.

To integrate OpenDistro with Zenduty, complete the following steps:

In Zenduty:

  1. To add a new OpenDistro integration, go to “Teams” on Zenduty and click on the “Manage” button corresponding to the team you want to add the integration to.

  2. Next, go to “Services” and click on the “Manage” button corresponding to the relevant Service.

  3. Go to “Integrations” and then “Add New Integration”. Give it a name and select the application “ElasticSearch Hosted(Opendistro)” from the dropdown menu.

  4. Go to “Configure” under your Integrations and Copy the generated Webhook URL.

In Elastic APM:

  1. Log into Elastic, and launch the APM application.

  2. Click the Sidebar which opens up the navigation and navigate to “Alerting” under “OpenDistro For ElasticSearch”.

  3. Now under “Alerting” tab, navigate to “Destinations”.

  4. Create a new Desination by clicking the “Add Destination” button.

  5. Name the destination as you see appropriate, and then select the “Custom Webhook” Type.

  6. Paste the previously copied URL under “Define endpoint by URL” and set the method under “Method Selection” to “POST”.

  7. Leave the Header Information as is (Content-Type:”application/json”) & Create the destination.

  8. Proceed by Creating/Selecting a monitor from which alerts are to be generated from. A monitor can be created by -
    • Clicking the Create Monitor button, and putting an appropriate Monitor name under “Configure Monitor”
    • Selecting a method of definition under “Define Monitor” with a specific index that is to be Monitored.
    • Creating a specific query to graph the monitor for the index selected under timestamped values.
    • Proceed by selecting a frequency for which the monitor query should run.
    • More information on creation of a monitor can be obtained here.
  9. After selecting the Monitor, In the “Triggers” sub-section, create a new Trigger.

  10. Define a appropriate Trigger Name, Severity level and a Trigger condition based on the monitor defined.
    • Severity levels help you organize your triggers and actions. A trigger with a high severity level might page a specific individual, whereas a trigger with a low severity level might email a list.
  11. Under “Configure Actions”, Define the Action name and select the Destination as the Webhook that was previously created.

  12. For the body of the Alert being sent, Copy and paste the following JSON below :
        {
    "title":"[OpenDistro (ElasticAPM)] - {{ctx.trigger.name}}",
    "id":"{{ctx.trigger.name}}",
    "description":"{{ctx.monitor.name}} just entered Triggered status from {{ctx.periodStart}} Please investigate the issue.",
    "alert_id":"{{ctx.alert.id}}",
    "status":"Triggered",
    "period_start":"{{ctx.periodStart}}",
    "period_end":"{{ctx.periodEnd}}"
    }
    

    You can add some more key value pairs to this payload that will be shown in your Alert payload, and can also be used in your alert rules.

    Available variables that may be used in Opendistro, can be found here.

    • Action Throttling may be enabled to reduce the spamming of alerts onto Zenduty.
    • You can also test the Alert/Incident Creation by using the “Send Test Message” function.
  13. Proceed to create the Trigger.

  14. OpenDistro (For OpenSourced Elastic APM) is now integrated with Zenduty.

Respond to OpenDistro (For Elastic APM) Integration alerts faster

Looking for a better way to get real-time alerts from OpenDistro (For Elastic APM) Integration, setup a solid incident escalation and incident response pipeline and minimize response and resolution times for OpenDistro (For Elastic APM) Integration incidents?

Signup for a free trial


Copyright Zenduty 2020. Product of YellowAnt