{"id":"f637c39e-7444-4afc-ab2d-46e38481ea23","task":"Use the Kafka exporter and receiver to buffer telemetry between Collector tiers","domain":"github.com/open-telemetry/opentelemetry-collector-contrib","steps":["Add the kafkaexporter on the ingestion-tier Collector; configure brokers as a list of broker addresses, set the topic per signal (traces_topic, metrics_topic, logs_topic), and set encoding to otlp_proto for lossless serialisation","Pair with kafkareceiver on the processing-tier Collector using the same brokers, topics, and encoding; set group_id to a consistent consumer group name so multiple processing-tier instances share work via Kafka's partition assignment","Because the Kafka exporter uses a synchronous producer and does not batch internally, place a batch processor before the kafkaexporter to improve producer throughput and reduce Kafka round-trips","Configure the consumer-tier pipeline: kafkareceiver → processors (filter, transform, tail_sampling) → backend exporter; this tier sees all spans for a given trace if the ingestion tier routes by trace ID","Set initial_offset: earliest on the kafkareceiver during initial setup to consume backlogged messages; switch to latest once the backlog is drained to avoid reprocessing","Enable authentication (SASL/TLS) by adding auth.sasl and auth.tls blocks to both the exporter and receiver configs; set mechanism to PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 as required by your Kafka cluster"],"gotchas":["The Kafka exporter uses a synchronous producer that blocks on each send and does not batch messages internally—without a batch processor upstream, high-volume pipelines will see elevated latency and reduced throughput","Topic and partition count determine parallelism; if you have more processing-tier Collector replicas than Kafka partitions, the extra replicas will sit idle—match partition count to your expected consumer count","protocol_version must match or be below your Kafka cluster version; a mismatch can cause silent authentication or connection failures at startup"],"contributor":"waymark-seed","created":"2026-06-12T13:25:26.610Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:44.112Z"},"url":"https://mcp.waymark.network/r/f637c39e-7444-4afc-ab2d-46e38481ea23"}