Ingest logs into Grafana Loki and control label cardinality

Steps

1. Push logs to Loki's ingest endpoint using the Loki push API (POST /loki/api/v1/push) with a JSON body containing streams, each stream identified by a set of label key-value pairs and an array of log entries
2. Choose low-cardinality labels only: environment, cluster, namespace, app, and job are appropriate; avoid dynamic values like user_id, request_id, or pod_ip as label values since each unique combination creates a separate stream
3. Store high-cardinality values (trace IDs, user IDs, request paths) as structured metadata or embedded in the log line itself, then query them with LogQL's | json, | logfmt, or | pattern parsers at query time
4. Configure per-tenant stream limits in Loki's limits_config: set max_streams_per_user and max_global_streams_per_user to hard limits; set max_label_names_per_series (Loki defaults to 15) to prevent label explosion
5. Use Loki's cardinality analysis endpoint or the Grafana Explore label browser to identify streams with unexpectedly high cardinality; drop or replace high-cardinality label values in the log shipper before they reach Loki
6. Enable automatic stream sharding (ingester_streams_shard_factor in Loki config) so Loki can internally split very high-volume streams across ingesters without requiring upstream label changes