Deploy the Datadog Observability Pipelines worker (OPW) as a Kubernetes DaemonSet or sidecar; configure your existing log shippers to forward logs to the OPW endpoint instead of directly to Datadog
In the Observability Pipelines UI, create a new pipeline and add a source matching your log shipper protocol (e.g., datadog_agent, fluent, http)
Add a Sample processor to the pipeline: configure a filter query to match the high-volume log pattern you want to reduce (e.g., service:payment-gateway status:debug) and set the desired retention percentage
Chain multiple Sample processors for different log patterns, each with independent sampling rates, to apply different tiers of reduction to different services or log levels
Add a Datadog Logs destination at the end of the pipeline to forward sampled output to Datadog; unsampled logs are dropped at the worker and never reach Datadog's ingestion endpoint
Monitor pipeline throughput and drop rates in the OPW metrics dashboard; verify cost reduction in Datadog's usage metrics within a billing cycle
Known gotchas
Sampling in OPW reduces actual ingestion cost because logs are dropped before reaching Datadog; this is distinct from index exclusion filters which only reduce indexing cost after ingestion has already been billed
Sampled-out logs are permanently lost unless you add a secondary destination (e.g., S3 archive) to the pipeline before the Sample processor; plan your archive strategy before enabling aggressive sampling
The OPW worker version must be compatible with your Datadog Agent version; check the compatibility matrix in Datadog docs before deploying OPW alongside existing Agent-based log collection
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp