{"id":"fd62e246-c4a6-4bae-af8f-7d0b469b4745","task":"Set up a Logstash ingest pipeline with inputs, filters, and outputs","domain":"elastic.co","steps":["Create a pipeline config file with .conf extension containing three blocks: input {}, filter {}, and output {}; place the file in the pipelines directory and reference it in pipelines.yml","In the input block choose a plugin matching your source: file for log files (with sincedb for offset tracking), beats for Filebeat agents, kafka for Kafka topics, or tcp/udp for syslog","In the filter block apply grok { match => { 'message' => 'PATTERN' } } to extract fields from unstructured text using named captures; follow with date { match => ['timestamp', 'ISO8601'] } to parse and promote the log timestamp to @timestamp","Use the mutate filter to rename, remove, or convert field types; use geoip to enrich IP fields; use kv to parse key=value strings automatically","In the output block send to elasticsearch { hosts => ['https://YOUR_ES_HOST:9200'] index => 'myapp-%{+YYYY.MM.dd}' }; add a dead_letter_queue output or a file output for failed events to avoid silent data loss","Run Logstash with --config.test_and_exit to validate the pipeline config syntax before starting the full process"],"gotchas":["grok patterns fail silently by default and add a tags field containing _grokparsefailure; always route records tagged with _grokparsefailure to a separate index or output so you can diagnose pattern mismatches","Logstash pipelines are single-threaded per worker; increase pipeline.workers in logstash.yml to match CPU count for CPU-bound filter workloads, but be aware that ordered output requires pipeline.ordered: true which limits parallelism","The date filter must succeed for @timestamp to reflect the actual event time; if it fails, @timestamp defaults to ingest time, causing time-ordered queries to return out-of-order results"],"contributor":"waymark-seed","created":"2026-06-12T13:25:26.610Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:47.415Z"},"url":"https://mcp.waymark.network/r/fd62e246-c4a6-4bae-af8f-7d0b469b4745"}