Alerts worth having

For about a year my alert channel received roughly forty messages a week. I eventually stopped reading them, which is functionally identical to having no alerting at all, plus the noise.

The only test that matters

An alert has to answer yes to this: does a human need to do something, right now?

If the answer is “no, it’ll sort itself out”, it isn’t an alert. If it’s “yes, but Monday morning is fine”, it isn’t an alert either: it’s a ticket. Everything else belongs on a dashboard, not in a notification.

Alert on symptoms, not causes

The mistake I made for a long time: alerting on node_cpu_seconds_total > 90%. A CPU at 90% isn’t a problem in itself: it might be a well-sized machine doing work. The problem is the service no longer responding fast enough.

groups:
  - name: symptoms
    rules:
      - alert: HighApiLatency
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 1
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "p95 latency above one second for 10 minutes"
          runbook: "https://ops.toulliou.org/notes/alerts-worth-having"

Two details that change everything:

• for: 10m filters out transient spikes. An alert that resolves itself in two minutes should never have fired.
• runbook points at the procedure. An alert with no attached procedure hands over the problem without handing over the means to fix it.

Predict saturation instead of observing it

A disk at 100% is an incident. A disk that is going to fill up is a schedulable task. Prometheus’ linear prediction is more than enough:

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 24 * 3600) < 0

In words: at the rate of the last six hours, this filesystem will be full in four days. Plenty of time to clean up without working under pressure.

The result

About three alerts a week now, and I read all of them. The difference isn’t that the infrastructure became more reliable. It’s that the other forty messages never told me anything I couldn’t read off a graph at a time of my choosing.