Press "Enter" to skip to content

Key Metrics for Kafka Clusters

Paul Brebner tracks some measurements:

Like a steam locomotive, Apache Kafka also has lots of “moving parts”. A Kafka cluster is actually a distributed system, a cluster, which consists of multiple brokers (servers)—messages are sent to Kafka with producers, and consumers receive messages. Topics are used to direct messages between producers and consumers—producers write to selected topics, and consumers subscribe and read from selected topics. Topics are divided up into partitions, and the partitions are distributed over the available brokers for high availability and concurrency. Partitions are replicated to other partitions (followers) from the leader partition (3 is common for production clusters).  

Click through for the description of metrics on brokers, topics, consumer groups, and more, as well as an extended analogy to steam locomotives which works surprisingly well.