Create an ingest pipeline using the PUT /_ingest/pipeline/{pipeline_id} API with a processors array; each processor object has a single key (the processor type) and its config
Use the grok processor with field and patterns keys to extract structured fields from unstructured log bodies; reference built-in Grok patterns (%{IP:client_ip}) or define custom patterns in pattern_definitions
Add a geoip processor to enrich an IP field with city, country, and coordinates from the bundled MaxMind GeoLite2 database; set target_field to avoid overwriting the source IP field
Use the set processor to add computed or static fields, the rename processor to normalise field names to ECS (Elastic Common Schema), and the remove processor to delete sensitive or redundant fields before indexing
Attach the pipeline to an index at index creation time by setting index.default_pipeline in the index settings, or apply it per-request by including ?pipeline={pipeline_id} in the bulk or index API call
Test changes with the POST /_ingest/pipeline/{pipeline_id}/_simulate endpoint, passing sample documents to verify processor output before attaching the pipeline to a live index
Known gotchas
Processor failures by default abort indexing and return an error for the document; wrap error-prone processors in an on_failure block to redirect failed documents to a dead-letter index or to log the failure and continue
Ingest pipelines run on the ingest node (or any node with the ingest role); heavy regex-based grok processing at high ingest rates can saturate ingest nodes—consider moving parsing upstream to Logstash or Fluent Bit
Pipelines are not automatically versioned; use the version field in the PUT request to track revisions and prevent accidental overwrites via if_version in concurrent updates
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp