Build a trace analytics pipeline into OpenSearch using Data Prepper

domain: opensearch.org · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Run an OpenTelemetry Collector that exports OTLP traces to Data Prepper's `otel_trace_source`
  2. Write a Data Prepper `pipeline.yaml` with an entry pipeline that fans out to a raw-trace pipeline (using the `otel_traces` processor) and a service-map pipeline (using the `service_map` processor)
  3. Configure OpenSearch sinks for each pipeline with `index_type: trace-analytics-raw` and `trace-analytics-service-map` respectively
  4. Run Data Prepper pointed at your OpenSearch cluster and confirm span documents land in the `otel-v1-apm-span-*` indices
  5. Open OpenSearch Dashboards' Observability plugin to view traces and the generated service map
  6. Scale Data Prepper instances as trace volume grows, since it does the trace/service-map processing OpenSearch itself does not do natively

Known gotchas

Related routes

Ingest pipeline metadata and dataset lineage into OpenLineage-compatible backends from a custom Python job
openlineage.io · 5 steps · unrated
Amazon SageMaker Pipelines: build a pipeline with Processing, Training, and RegisterModel steps using the SageMaker Python SDK
ml-ops · 6 steps · unrated
build a CSAT/NPS webhook pipeline that lands survey responses into a data warehouse
general-cx-orchestration · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans