Link

Alert rules let you create complex conditions for an integration, by matching fields like alert_type, summary, description of the alert, and payload sent by the alert. Alert Rules also lets you take custom actions like assign a custom EP or a user to an integration. Custom actions include:

  • Change alert type
  • Change incident urgency
  • Add note to incident
  • Assign incident tag
  • Assign incident role
  • Change entity_id
  • Change incident title
  • Change incident summary
  • Assign incident to a user
  • Route incident to an escalation policy other than the default policy associate with the service
  • Suppress the incident

The basics of Alerts and Incidents in Zenduty

Every alert coming from an integration source has an entity_id.
An entity_id points to a resource or an entity in the source application. For example, the entity_id of an alert from Jira will be its Ticket ID, an alert from Datadog will be its Alert ID in Datadog, an alert from Grafana will be its Rule ID.
Multiple alerts can have the same entity_id. entity_id is how Zenduty groups multiple alerts about the same problem into one single incident.

Each alert comes with an alert_type. There are six types of alert types:

  • critical
  • acknowledged
  • resolved
  • info
  • warning
  • error

If the alert source is sending an entity_id, and the alert type is either critical, error or warning, then that alert(depending on your integration level settings) will trigger a single incident. Every alert after that with the same entity_id will be attached to the earlier created incident. An acknowledged alert_type from the source will automatically acknowledge an incident and a resolved alert_type will automatically resolve the incident.

For example:

Alert 1: entity_id:database-alpha, alert_type:critical, message:DB is down

The above alert will create a new incident(‘I’). Incident ‘I’ is now in the triggered state.

Alert 2: entity_id:database-alpha, alert_type:critical, message:DB is still down

The above alert will not create a new incident, but will be appended as an alert to incident ‘I’. Incident ‘I’ will continue to be in the triggered state.

Alert 3: entity_id:database-alpha, alert_type:resolved, message:DB is up

The above alert will automatically resolve the incident ‘I’ and will also be appended as an alert to incident ‘I’.

Alert 4: entity_id:database-alpha, alert_type:critical, message:DB is down

The above alert will create another new incident - ‘I2’. Incident ‘I2’ is now in the triggered state.

Alert Fields

The following fields can be used to build your alert rules -

Alert FieldDescription
Alert TypeThe class of the alert received. Includes Critical, Acknowledged, Resolved, Error, Warning and Info
Incident UrgencyThe urgency of the incident. Can be high or low.
MessageTitle of the alert.
SummarySummary of the alert as generated by Zenduty.
Alert Time(UTC)Time when the alert was received.
Alert Date(UTC)Date when the alert was received.
Day of WeekDay of the week on which the alert was received.
Entity IDUnique identifier used to club alerts together, into a single incident.
Seconds since last similar incidentSeconds since the last incident with similar identifiers, useful in supressing noise when multiple similar incidents start getting triggered.
Payload(Value Match)Match a particular key’s value from the alert payload to a given pattern.
Payload(Key Search)Search the payload for the existence of a particular key.

Actions

The following actions can be triggered when a rule is matched on an incoming alert -

Action
Change alert type to (Critical, Error, Warning, Info, Acknowledged, Resolved)
Add Note
Suppress
Route to escalation policy
Assign User
Change Incident Urgency
Change Alert Message
Change Alert Summary
Change Alert Entity ID
Assign Role to User
Assign incident tag
Assign SLA
Assign Priority

Adding an Alert Rule

  1. Click on the service your integration is in.

  2. Go to the ‘integrations’ tab and choose the integration for which you want to add Alert Rules.

  3. Click on the ‘Alert Rules’ tab from the secondary sidebar and then the ‘Create Alert Rule’ button.

  4. Give your alert a name and set the conditions and actions as intended. Save to apply.

You can assign payload with specific rules to your alerts. Follow the JSON path syntax here while describing these rules. For example, if the payload sent by the alert is {"project":"zd-test", "release":{"version":"2.3.1"}}, then to compare the release version, your JSONPath will be $.release.version.

For time window based operations that cross into the next day, for eg: 8PM Monday to 6AM Tuesday, split the time in the conditions as 20:00 to 23:59:59 and 00:00:00 to 06:00:00. So, if you’re adding a condition to check if the timing of the alert is between two times say A and B, please ensure that the B>A. If not, then split the alert rule into 2 conditions, with an OR operator between them:

A <= T <= "23:59:59" OR "00:00:00" >= T >= B

Example: If we want to create an alert rule where the alert time is between 7PM and 5AM, the two conditions with an OR operator would be:

  1. Alert time between 7:00:00PM and 11:59:59PM
  2. Alert time between 12:00:00AM and 05:00:00AM

If we want to create an alert rule where the time is between 9AM and 5PM, we can do that with a single condition:

  • Alert time between 09:00:00 am and 05:00:00pm

Modifying incident title and summary, adding notes

You can modify the incident titles and summaries by using the alert data(message, summary, payload, entity_id) within placeholders like The incident message is , or The incident summary is or High priority incident in release .

[Important] The Add Note action is only applied for the first alert that triggered the incident. For any alert that does not directly trigger an incident, the Add Note action will not be applied.

Example: Modifying incident details - changing alert summary

Example: Adding notes, assigning roles and incident urgency

Example: Assigning incident tags and SLAs


Copyright © 2023 Zenduty