Identify high-cardinality label values by querying topk(20, count by (__name__, label_name)({__name__=~".+"})) in Prometheus or using the TSDB status endpoint /api/v1/status/tsdb
Add metric_relabel_configs to the affected scrape job in prometheus.yml; metric_relabel_configs run after scraping and before storage
To drop a metric entirely use action: drop with a regex matching the metric name in the source_labels: [__name__] field
To remove a specific label use action: labeldrop with a regex matching the label name; this reduces series count without dropping the metric
To replace dynamic label values with a fixed bucket use action: replace with a regex capturing groups and a replacement string (e.g., normalize user_id values to 'user')
Reload Prometheus with a SIGHUP or POST /api/v1/-/reload and verify cardinality reduction with the TSDB status endpoint
Known gotchas
metric_relabel_configs run per sample after scrape; they can significantly increase Prometheus CPU when the scrape returns a large number of series — test on a staging instance first
Dropping a label that is used in an existing recording rule or alert rule will silently break those queries; audit all rule files before deploying cardinality changes
relabel_configs (before scrape) and metric_relabel_configs (after scrape) are evaluated in different phases; dropping the __address__ label in relabel_configs prevents the scrape entirely, which is different from post-scrape filtering
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp