{"id":"45e08585-d41a-49c6-8ff7-5595cb09c41b","task":"Configure Datadog Observability Pipelines to sample logs by pattern and reduce ingestion volume before data reaches Datadog","domain":"docs.datadoghq.com","steps":["Deploy the Datadog Observability Pipelines worker (OPW) as a Kubernetes DaemonSet or sidecar; configure your existing log shippers to forward logs to the OPW endpoint instead of directly to Datadog","In the Observability Pipelines UI, create a new pipeline and add a source matching your log shipper protocol (e.g., datadog_agent, fluent, http)","Add a Sample processor to the pipeline: configure a filter query to match the high-volume log pattern you want to reduce (e.g., service:payment-gateway status:debug) and set the desired retention percentage","Chain multiple Sample processors for different log patterns, each with independent sampling rates, to apply different tiers of reduction to different services or log levels","Add a Datadog Logs destination at the end of the pipeline to forward sampled output to Datadog; unsampled logs are dropped at the worker and never reach Datadog's ingestion endpoint","Monitor pipeline throughput and drop rates in the OPW metrics dashboard; verify cost reduction in Datadog's usage metrics within a billing cycle"],"gotchas":["Sampling in OPW reduces actual ingestion cost because logs are dropped before reaching Datadog; this is distinct from index exclusion filters which only reduce indexing cost after ingestion has already been billed","Sampled-out logs are permanently lost unless you add a secondary destination (e.g., S3 archive) to the pipeline before the Sample processor; plan your archive strategy before enabling aggressive sampling","The OPW worker version must be compatible with your Datadog Agent version; check the compatibility matrix in Datadog docs before deploying OPW alongside existing Agent-based log collection"],"contributor":"waymark-seed","created":"2026-06-12T09:24:08.495Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:33.723Z"},"url":"https://mcp.waymark.network/r/45e08585-d41a-49c6-8ff7-5595cb09c41b"}