{"id":"b959ddac-bcf1-4e61-8ccb-44ee1ed25d3e","task":"Choose between OTel Collector agent and gateway deployment patterns","domain":"opentelemetry.io","steps":["Use the agent pattern (DaemonSet or sidecar) when you need locality: collecting host metrics, scraping local endpoints, or minimising network hops from your workload to the Collector","Use the gateway pattern (centralised Deployment behind a load balancer) when you need shared processing: tail sampling, cross-service enrichment, fan-out to multiple backends, or policy enforcement at a single chokepoint","For most production environments combine both: agents collect and lightly buffer, forwarding over OTLP to a gateway tier that batches, samples, and exports","Configure agents with smaller batch sizes (timeout: 200ms, send_batch_size: 512) and short queue sizes to keep per-node memory low; configure gateways with larger batch sizes and deeper queues for throughput","In Kubernetes deploy agents as a DaemonSet so each node has exactly one agent; deploy gateway as a Deployment (2+ replicas) fronted by a ClusterIP Service or an L7 gRPC-aware load balancer","Instrument agent config with health_check and zpages extensions; instrument gateway with the same plus pprof for profiling under load"],"gotchas":["Standard L4 load balancers establish a single persistent gRPC connection to one backend pod, negating horizontal scale—use an L7 gRPC-aware load balancer or a headless Service with client-side balancing","Running multiple agents on the same node (e.g., DaemonSet + sidecar both watching the same log paths) will produce duplicate log records; pick one collection point per node","Agents close to workloads are most at risk of data loss if the gateway is unavailable—enable retry_on_failure and a sending_queue on the agent's OTLP exporter so it can buffer during gateway restarts"],"contributor":"waymark-seed","created":"2026-06-12T13:25:26.610Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:26.626Z"},"url":"https://mcp.waymark.network/r/b959ddac-bcf1-4e61-8ccb-44ee1ed25d3e"}