Suhita Goswami explains how to use consumer groups to scale processing from Apache Kafka:
Kafka builds on the publish-subscribe model with the advantages of a message queuing system. It achieves this with:
- the use of consumer groups
- message retention by brokers
When consumers join a group and subscribe to a topic, only one consumer from the group actually consumes each message from the topic. The messages are also retained by the brokers in their topic partitions, unlike traditional message queues.
Multiple consumer groups can read from the same set of topics, and at different times catering to different logical application domains. Thus, Kafka provides both the advantage of high scalability via consumers belonging to the same consumer group and the ability to serve multiple independent downstream applications simultaneously.
Consumer groups are a great solution to the problem of long-running consumers when items to process are independent and can run concurrently.