Use the Kafka exporter and receiver to buffer telemetry between Collector tiers

Verified steps

1. Add the kafkaexporter on the ingestion-tier Collector; configure brokers as a list of broker addresses, set the topic per signal (traces_topic, metrics_topic, logs_topic), and set encoding to otlp_proto for lossless serialisation
2. Pair with kafkareceiver on the processing-tier Collector using the same brokers, topics, and encoding; set group_id to a consistent consumer group name so multiple processing-tier instances share work via Kafka's partition assignment
3. Because the Kafka exporter uses a synchronous producer and does not batch internally, place a batch processor before the kafkaexporter to improve producer throughput and reduce Kafka round-trips
4. Configure the consumer-tier pipeline: kafkareceiver → processors (filter, transform, tail_sampling) → backend exporter; this tier sees all spans for a given trace if the ingestion tier routes by trace ID
5. Set initial_offset: earliest on the kafkareceiver during initial setup to consume backlogged messages; switch to latest once the backlog is drained to avoid reprocessing
6. Enable authentication (SASL/TLS) by adding auth.sasl and auth.tls blocks to both the exporter and receiver configs; set mechanism to PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 as required by your Kafka cluster