Design an alert policy

Understand the lifecycle

A reliable alert policy starts by separating several states that are often collapsed into one:

• Observation: one protocol check from one region.
• Regional state: the result of applying consecutive failure or success thresholds to a region.
• Investigation: evidence has begun to accumulate, but regional confirmation has not been reached.
• Confirmed incident: enough regions have reached the failing state.
• Resolved incident: the recovery policy has been satisfied.
• Delivery: a notification attempt for an incident event.

This separation prevents one transient timeout from paging the whole team while preserving the evidence needed to understand what happened.

If the terms below are new, start with the recommended values shown in Create your first monitor. Return to this guide after you have seen several days of normal checks.

Set the failure threshold

The failure threshold accepts 1–10 consecutive failures. A value of one reacts to the first failed observation. Higher values filter short faults but delay detection.

Use a value of two as a starting point for a 60-second production monitor. Set one only for services that require immediate reaction and have very stable checks. Consider three or more for noisy external dependencies where brief failures are expected and the impact of a later alert is acceptable.

Avoid compensating for a poor assertion with a high threshold. If normal, healthy responses intermittently fail the check, first correct the timeout, expected status, keyword, DNS assertion, or target.

Require regional confirmation

Confirmation accepts 1–16 regions and cannot exceed the monitor’s selected regions. For three globally distributed probes, two-region confirmation usually distinguishes a local path failure from broader impact.

The earliest approximate confirmation time is:

interval × failure threshold

The actual time includes scheduling and request duration, and the incident cannot confirm until the final region required for quorum reaches its threshold.

Define recovery separately

Recovery thresholds also accept 1–10 consecutive successes. Do not assume the best failure threshold is automatically the best recovery threshold.

A recovery threshold of two helps prevent a single good response during an unstable period from resolving the incident prematurely. For services with cache warmup or deployment convergence, use a larger recovery threshold only if the resulting delay is acceptable to responders and customers.

Decide when to notify

Alert delivery begins from confirmed incident state, not from every failed check. The optional alert wait can delay delivery by 0–120 minutes after confirmation.

Use no added delay for critical services with a staffed response path. A short delay can make sense for lower-severity dependencies where the team wants confirmation recorded immediately but does not want interruption unless the condition persists.

Separate urgency from audience:

1. Identify who owns the service.
2. Choose delivery channels appropriate to the expected response time.
3. Subscribe destinations only to the event types they need.
4. Keep customer communication separate from internal diagnostic detail.
5. Test the destination before relying on it.

Use maintenance and muting correctly

Maintenance and muting solve different problems.

Scheduled maintenance describes planned work during a defined time range. Affected monitor failures do not open a new incident during an active window, and published windows can appear on matching public status pages.

Muting temporarily stops alert delivery for a monitor. Use it for a short operational pause when the work does not need a customer-facing schedule. Muting should not be a permanent fix for noisy checks.

Always record a reason and an end condition. Forgotten suppression is more dangerous than alert noise because it can hide real customer impact.

Choose a starting policy by criticality

These are baselines, not service-level guarantees. Calculate timing from the actual interval, thresholds, and region behavior.

Test the policy safely

Test against a staging monitor or a controlled endpoint. Confirm each transition:

1. A single failed observation appears in check evidence.
2. The region becomes failing only after the configured streak.
3. An investigation remains unconfirmed until regional quorum is reached.
4. Confirmed delivery is recorded in the incident timeline.
5. One success does not resolve the incident when recovery requires two.
6. The resolved event reaches destinations subscribed to recovery.

Do not create a production outage merely to test alerting. A dedicated synthetic endpoint can return controlled statuses or timeouts without affecting customers.

Investigate noisy alerts

When an alert appears incorrect, use the check log and incident timeline rather than raising thresholds immediately.

• Failures from one region across many monitors suggest probe or route health.
• Keyword failures with successful HTTP status suggest an assertion or application response change.
• TLS failures before expiry may indicate chain, hostname, or handshake problems.
• Alternating pass/fail results can indicate load-balancer inconsistency.
• Timeouts clustered around p95 latency suggest the timeout is below normal tail behavior.
• Failures during planned work indicate the monitor was not included in the maintenance scope or the window timing was wrong.

Adjust one policy dimension at a time, then review the next week of evidence. This keeps the reason for improvement measurable.

Review alert quality

Track confirmed incidents, acknowledged incidents, delivery outcomes, time to confirmation, time to recovery, and false-positive investigations. The goal is not zero alerts; it is an evidence chain that responders trust.

Next, configure a Slack webhook or organize customer communication with a status page.