Swapnil Gosavi walks us through the process of creating a custom partitioner in Apache Kafka:
Assume we are collecting data from different departments. All the departments are sending data to a single topic named department. I planned five partitions for the topic. But, I want two partitions dedicated to a specific department, named IT, and the remaining three partitions for the rest of the departments. How would you achieve this?
You can solve this requirement, and any other type of partitioning needs by implementing a custom partitioner.
This is quite useful when you don’t necessarily want topic explosion but you do want more than what the classic partitioner allows.