Press "Enter" to skip to content

Kafka Consumer Group Assignment

David Brinegar discusses how consumers within an Apache Kafka consumer group get assigned work:

Or one might want some assignment that results in uniform workloads, based on the number of messages in each partition.  But until we have pluggable assignment functions, the reference implementation has a straightforward assignment strategy called Range Assignment.  There is also a newer Round Robin assignor which is useful for applications like Mirror Maker, but most applications just use the default assignment algorithm.

The Range Assignor tries to land on a uniform distribution of partitions, at least within each topic, while at the same time avoiding the need to coordinate and bargain between nodes.  This last goal, independent assignment, is done by each node executing a fixed algorithm:  sort the partitions, sort the consumers, then for each topic take same-sized ranges of partitions for each consumer.  Where the sizes cannot be the same, the consumers at the beginning of the sorted list will end up with one extra partition.  With this algorithm, each application node can see the entire layout by itself, and from there take up the right assignments.

Click through to see an example of how this is implemented.