Migrate a Kafka consumer group to the KIP-848 next-generation rebalance protocol
domain: kafka · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Ensure all brokers are on Kafka 4.0 or later (KIP-848 reached GA in 4.0); verify with kafka-broker-api-versions.sh
Update client libraries to a version that supports the new consumer group protocol (look for ConsumerGroupHeartbeat API support in the client changelog)
Set group.protocol=consumer in the consumer configuration to opt the client into the KIP-848 server-side coordinator
Deploy consumers with rolling restart; the new protocol is incremental — each consumer heartbeats individually and receives only its delta assignment without triggering a stop-the-world rebalance
Monitor the group state via kafka-consumer-groups.sh --describe; under KIP-848 the group should show state 'Stable' continuously rather than cycling through 'PreparingRebalance' and 'CompletingRebalance'
If mixed old/new protocol clients are detected the broker falls back to the classic protocol; ensure all consumers in the group upgrade before expecting incremental rebalancing
Known gotchas
KIP-848 uses a broker-side group coordinator that maintains assignment state; the broker resource cost is higher per-group compared to the classic protocol, so monitor GroupCoordinator heap on heavily loaded brokers
The group.protocol=consumer setting is a client-level toggle; setting it on only some consumers in the same group causes the broker to reject the mixed state and fall back to classic mode
Cooperative-sticky (group.protocol=classic with partition.assignment.strategy=CooperativeStickyAssignor) is the incremental option within the classic protocol and remains available for brokers below 4.0; it is distinct from KIP-848
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?