Threshold, Anomaly, and Multi-Signal Alerts

Threshold, anomaly, and multi-signal alerts represent different ways to decide when the system should interrupt a human. A threshold alert fires when a value crosses a defined line. An anomaly alert fires when behavior deviates materially from its recent pattern. A multi-signal alert combines evidence from several conditions before escalating.

None of these strategies is universally best. Thresholds are easier to reason about and are often ideal for explicit objectives such as error rate or latency targets. Anomaly alerts can help when natural patterns vary by hour or day. Multi-signal alerts are useful when one signal alone is too noisy, but a combination tells a stronger story.

    flowchart TD
	    A["Observed telemetry"] --> B{"Alert strategy"}
	    B --> C["Threshold"]
	    B --> D["Anomaly"]
	    B --> E["Multi-signal"]
	    C --> F["Simple and explicit"]
	    D --> G["Pattern-aware"]
	    E --> H["Stronger evidence before paging"]

Choose Strategy From Signal Behavior

A good rule of thumb:

• use thresholds when the boundary is meaningful and understandable
• use anomaly detection when the baseline varies naturally and hard thresholds are weak
• use multi-signal logic when one metric alone is noisy but combinations are trustworthy

 1alerts:
 2  - name: api_error_rate_page
 3    type: threshold
 4    condition: "error_rate > 2% for 10m"
 5  - name: traffic_drop_anomaly
 6    type: anomaly
 7    condition: "request_volume deviates materially from expected pattern"
 8  - name: degradation_page
 9    type: multi_signal
10    condition:
11      - "p95_latency > 500ms"
12      - "error_rate > 1%"
13      - "traffic > minimum_active_load"

Simplicity Is Usually Stronger Than Cleverness

A frequent mistake is building sophisticated alert logic too early. If a simple threshold on a user-visible symptom works well, that is often better than an opaque anomaly model nobody trusts. Complex alert logic earns its place only when it clearly improves signal quality or reduces false positives without hiding real incidents.

This is why alert tuning should be empirical. Teams should review:

• how often the alert fired
• how often it represented real impact
• whether responders could explain why it fired

Design Review Question

If latency rises every morning during a known traffic ramp and a fixed threshold pages the team daily even though the behavior is expected, which alerting strategy may deserve review?

The stronger answer is either a better threshold design or an anomaly or multi-signal approach that understands the system’s normal rhythm more accurately.

Quiz Time

Design Review Question

If this capability were weak during a live incident, what uncertainty would remain unresolved, and which team would be unable to act with confidence?