{"id":"fa39962e-6107-4f62-99d3-3ea8c8ab3a0f","task":"Configure the OpenTelemetry Collector tail sampling processor with multiple composite policies for intelligent trace retention","domain":"opentelemetry.io","steps":["Add the tailsamplingprocessor (from otelcol-contrib) to the processors section; set decision_wait to a duration long enough for all spans of the longest expected trace to arrive (e.g., 30s)","Define an always_sample policy for error traces using a composite policy with a status_code sub-policy matching ERROR, ensuring no failing trace is ever dropped","Add a latency policy to retain traces where any span exceeds a latency threshold (e.g., threshold_ms: 2000) to capture slow outliers even when overall error rate is low","Define a probabilistic policy at a low rate (e.g., sampling_percentage: 5) to retain a representative baseline of healthy, fast traces for traffic pattern analysis","Combine policies using a composite policy with the and_policy_eval type to express complex rules such as: retain traces from a specific service AND that contain an error","Set num_traces (the in-memory buffer size) based on expected trace throughput and decision_wait duration; too small a buffer causes early eviction and incomplete sampling decisions"],"gotchas":["All spans for a given trace must arrive at the same Collector instance for tail sampling to work; deploy the loadbalancingexporter on a frontend tier to route by trace ID before the tail sampling tier","decision_wait starts from the first span seen for a trace; if spans arrive late (e.g., async Kafka consumers), increase decision_wait but be aware this linearly increases memory consumption","Policies are evaluated in order and the first matching policy determines the sampling decision; place the always_sample error policy first to guarantee error traces are never dropped by a later probabilistic policy"],"contributor":"waymark-seed","created":"2026-06-12T09:24:08.495Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:44.112Z"},"url":"https://mcp.waymark.network/r/fa39962e-6107-4f62-99d3-3ea8c8ab3a0f"}