Category: Kafka / Flink

Jack Vanlightly continues a series on Dimster. First up is a benchmark of consumer groups versus share groups:

In this first share group benchmarking post, we’re going to use share groups as they are not intended to be used, but for a good reason. Share groups allow you to move past partitions as the unit of parallelism by allowing multiple consumers to read from the same partition, using message queue semantics. We’ll run those kinds of tests in the next post. In this post I just want to understand if the mechanics of how share groups work add any additional overhead compared to consumer groups. So we’ll use share groups as if they were consumer groups (by capping consumer count to partition count).

Objective: Use synthetic tests to measure the overhead of share groups compared to consumer groups in identical conditions.

After that, Jack simulates processing time:

In this post we’re going to simulate processing time in the consumers to make these benchmarks more realistic and show the utility of share groups (namely the ability to parallelize processing beyond the partition count).

We’ll see how the following two configurations play an important role in parallelizing consumption with share groups:

• max.poll.records (consumer config)
• group.share.partition.max.record.locks (broker-side config)

And there’s one more post in the series so far:

In the last post we used simulated consumer processing time to reveal how important it is to set an appropriate value for max.poll.records to ensure the consumer parallelism that we expect. With a uniform distribution of messages over partitions, the rule of thumb was a value somewhat lower than:

group.share.partition.max.record.locks / number of consumers per partition

But there’s more to parallel consumption than max.poll.records. The size of producer batches also plays a role when using the default share.acquire.mode (batch_optimized).

Stay tuned for the next post in the series.

Leave a Comment

Paul Brebner extends a metaphor:

I’ve been tracking the progress of Apache Kafka “Diskless Topics” for a while now. It’s a topic that sparks curiosity—mostly because the name itself sounds like an oxymoron. How can a topic be diskless? Where does the data go? 

With the recent voting on KIP-1150, I decided it was time to dive deep into the architectural changes. There are several related Kafka Improvement Proposals (KIPs) floating around, but KIP-1150 is dependent on KIP-1163 and KIP-1164, and the designs are still in flux. Consider this blog post a “theory” in the true scientific sense: a best-guess model based on current evidence that will almost certainly evolve. 

Click through for your moment of zen.

Comments closed

Jaisen Mathai lays out some changes:

Hey, fellow Apache Kafka® developers! Let’s look at the important updates across Confluent’s client ecosystem, from the core librdkafka to the wrappers for Python, Go, .NET, and JavaScript. The last couple of months have been focused on laying down some solid architectural foundations and adding key quality-of-life features.

I’m curious to see what changes with IBM purchasing Confluent.

Comments closed

Confluent has an announcement:

We are excited to announce that Confluent has entered into a definitive agreement to be acquired by IBM. After the transaction is closed (subject to customary closing conditions and regulatory approvals), together, IBM and Confluent will aim to provide a platform that unifies the world’s largest enterprises, unlocking data for cloud/microservices, accelerating time-to-value, and building the real-time data foundation required to scale AI across every organization. 

Whelp. I suppose it was bound to happen at some point, but I definitely can’t say this news pleases me.

Comments closed