Tune the OpenTelemetry Collector memory_limiter and batch processor for stable high-throughput operation

Steps

1. Add memory_limiter as the first processor in every pipeline; set limit_mib or limit_percentage (e.g., 80% of container memory limit) and spike_limit_percentage (e.g., 25% of limit) with check_interval: 1s
2. Set the GOMEMLIMIT environment variable on the Collector to approximately 80% of the hard container memory limit so the Go runtime's GC triggers before the memory limiter's hard limit is breached
3. Place the batch processor after memory_limiter in the pipeline; configure send_batch_size (e.g., 8192) and timeout (e.g., 10s) — batches are sent when either threshold is reached first
4. For very high throughput, set send_batch_max_size to cap the maximum batch size and prevent oversized payloads from overwhelming downstream exporters
5. Monitor otelcol_processor_refused_spans (or the equivalent metric for logs/metrics) to detect when the memory limiter is actively backpressuring; this signals the Collector is under memory pressure
6. Scale horizontally by deploying additional Collector replicas behind a load balancer rather than increasing per-instance memory limits beyond the point where GC pause latency becomes noticeable