Control metric cardinality by applying metric_relabel_configs to drop or aggregate high-cardinality labels

Steps

1. Identify high-cardinality label values by querying topk(20, count by (__name__, label_name)({__name__=~".+"})) in Prometheus or using the TSDB status endpoint /api/v1/status/tsdb
2. Add metric_relabel_configs to the affected scrape job in prometheus.yml; metric_relabel_configs run after scraping and before storage
3. To drop a metric entirely use action: drop with a regex matching the metric name in the source_labels: [__name__] field
4. To remove a specific label use action: labeldrop with a regex matching the label name; this reduces series count without dropping the metric
5. To replace dynamic label values with a fixed bucket use action: replace with a regex capturing groups and a replacement string (e.g., normalize user_id values to 'user')
6. Reload Prometheus with a SIGHUP or POST /api/v1/-/reload and verify cardinality reduction with the TSDB status endpoint