Authenticate API calls with the headers DD-API-KEY and DD-APPLICATION-KEY (application key must have the logs_write_pipelines scope)
Create a pipeline with POST /api/v1/logs/config/pipelines; provide a name, filter.query string (e.g., source:nginx) to scope which logs enter the pipeline, and an is_enabled flag
Add processors inside the pipeline by including a processors array in the create or update body; each processor object requires a type (e.g., grok-parser, date-remapper, attribute-remapper, string-builder-processor), a name, and type-specific config fields
Update an existing pipeline (to add, reorder, or modify processors) with PUT /api/v1/logs/config/pipelines/{pipelineId}; this replaces the entire pipeline definition, so always GET the current state first and merge your changes
Control pipeline execution order with PUT /api/v1/logs/config/pipeline-order; the body is {"pipeline_ids": ["id1", "id2"]} in the desired evaluation order—pipelines are evaluated top-to-bottom and the first matching pipeline's processors run
Manage pipelines as code using the Datadog Terraform provider (resource datadog_logs_custom_pipeline) or Pulumi to version-control and review pipeline changes through normal CI/CD workflows
Known gotchas
The PUT pipeline endpoint replaces the entire pipeline including all processors; a race condition between two concurrent PUT calls can cause one to overwrite the other—use the GET-then-PUT pattern with optimistic locking or serialise pipeline updates in your automation
Pipeline filter.query uses the same Datadog log search syntax as the Log Explorer; an empty or overly broad query will match all logs and run all processors on your entire log volume, which can affect indexing performance
Pipelines only process logs at ingest time; changes to processors do not retroactively re-process historical logs—reprocess by rehydrating from a Datadog archive if you need to apply new parsing to past data
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp