Kafka Consumer Group Assignment

Kevin Feasel



David Brinegar discusses how consumers within an Apache Kafka consumer group get assigned work:

Or one might want some assignment that results in uniform workloads, based on the number of messages in each partition.  But until we have pluggable assignment functions, the reference implementation has a straightforward assignment strategy called Range Assignment.  There is also a newer Round Robin assignor which is useful for applications like Mirror Maker, but most applications just use the default assignment algorithm.

The Range Assignor tries to land on a uniform distribution of partitions, at least within each topic, while at the same time avoiding the need to coordinate and bargain between nodes.  This last goal, independent assignment, is done by each node executing a fixed algorithm:  sort the partitions, sort the consumers, then for each topic take same-sized ranges of partitions for each consumer.  Where the sizes cannot be the same, the consumers at the beginning of the sorted list will end up with one extra partition.  With this algorithm, each application node can see the entire layout by itself, and from there take up the right assignments.

Click through to see an example of how this is implemented.

Related Posts

Event Sourcing On Kafka

Adam Warski shows how you can use Apache Kafka as your event sourcing data source: There’s a number of great introductory articles, so this is going to be a very brief introduction. With event sourcing, instead of storing the “current” state of the entities that are used in our system, we store a stream of events that relate to these […]

Read More

The Basics Of Kafka Security

Stephane Maarek has a nice post covering some of the basics of securing an Apache Kafka cluster: Once your Kafka clients are authenticated, Kafka needs to be able to decide what they can and cannot do. This is where Authorization comes in, controlled by Access Control Lists (ACL). ACL are what you expect them to be: […]

Read More


October 2016
« Sep Nov »