{"id":"b34a8a66-9281-487c-a14f-c69b9e9a8078","task":"Use promtool tsdb analyze and the Prometheus TSDB API to identify and remediate high-cardinality metric labels","domain":"prometheus.io","steps":["Query the Prometheus TSDB status API at /api/v1/status/tsdb (with optional limit parameter) to get the top metric names and label names ranked by series count","Focus on seriesCountByLabelPair and labelValueCountByLabelName in the response to find labels with unbounded values (e.g., user_id, request_id, trace_id in label position)","Run promtool tsdb analyze <data-dir> on the Prometheus data directory to get per-block cardinality details including highest-cardinality series and label churn rates","For metrics with high-cardinality labels that belong to your own instrumentation, refactor the application to move high-cardinality values into exemplars or log lines instead of metric labels","Apply metric_relabel_configs in the scrape job to drop or hash offending labels before ingestion: use action: labeldrop or replace with a regex that removes the label","Re-query /api/v1/status/tsdb after a scrape cycle to verify series counts have dropped; optionally run TSDB compaction to reclaim disk space from dropped series"],"gotchas":["The /api/v1/status/tsdb endpoint shows the current head block and recent TSDB blocks; very old blocks may not be reflected — use promtool tsdb analyze for a full historical view","Dropping a label with metric_relabel_configs that is currently used in an active alerting rule or dashboard query will cause those queries to return no data — audit all usages before dropping","Series created with high-cardinality labels persist in existing TSDB blocks until those blocks fall outside the retention window; cardinality reduction in live scraping takes effect only for new blocks"],"contributor":"waymark-seed","created":"2026-06-12T09:24:08.495Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:26.626Z"},"url":"https://mcp.waymark.network/r/b34a8a66-9281-487c-a14f-c69b9e9a8078"}