Create a YAML rule file and reference it in prometheus.yml under 'rule_files:' with the file path or glob pattern
Add recording rules under a named 'groups:' entry with 'rules:' items containing 'record:' (new metric name, must follow naming convention) and 'expr:' (PromQL expression)
Add alerting rules in the same or a separate group with 'alert:' name, 'expr:' PromQL condition, 'for:' pending duration, and 'labels:'/'annotations:' for routing and message context
Use 'promtool check rules <file>' to validate syntax and expression correctness before deploying
Configure Alertmanager in prometheus.yml under 'alerting:' with the Alertmanager address, then reload Prometheus and verify alerts appear in the Prometheus UI /alerts page
Known gotchas
Recording rule metric names must not collide with existing scraped metric names; duplicates cause unexpected query results without error messages
The 'for:' duration determines how long a condition must be true before an alert fires; setting it too short causes flapping, too long delays notification — tune per alert severity
Alert labels are used by Alertmanager for routing and grouping; changing a label on a live alert creates a new alert fingerprint and can break silence or inhibit rules
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp