Implement multi-window multi-burn-rate SLO alerting using Prometheus recording rules and Sloth

domain: sloth.dev · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Define an SLO spec in Sloth YAML format with slo.service, slo.slos[].name, slo.slos[].objective (e.g., 99.9), and slo.slos[].sli.events specifying error and total metric selectors
  2. Run sloth generate -i slo-spec.yaml -o slo-rules.yaml to output Prometheus recording rules and multi-window multi-burn-rate alerting rules conforming to the Google SRE Workbook alert method
  3. Apply the generated rules to Prometheus via your Kubernetes PrometheusRule CRD or by adding to prometheus.rules config; rules create metrics like slo:sli_error:ratio_rate5m and slo:burnrate5m
  4. The generated alerts come in page and ticket severity pairs: a page alert fires when the short window AND long window both exceed the burn-rate threshold (e.g., 14.4x budget consumed in 1h/5m); ticket alert fires at lower thresholds (e.g., 6x in 6h/30m)
  5. Wire the page-severity alert to your PagerDuty or Opsgenie integration via Alertmanager routes using the severity label set by Sloth
  6. Validate burn rate calculations: at a 14.4x burn rate for a 99.9% SLO, the error budget would exhaust in approximately 1 hour — use this as a sanity check for threshold tuning

Known gotchas

Related routes

Implement multi-window multi-burn-rate SLO alerting in Prometheus following the Google SRE Workbook model
prometheus.io · 6 steps · unrated
Implement multi-window multi-burn-rate alerting for an SLO in Prometheus Alertmanager
prometheus.io · 6 steps · unrated
Implement SLO error budget burn rate alerting with multi-window alerts using Prometheus alerting rules
prometheus.io · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans