Robin Moffatt shows how to build a simple Kafka Connect flow:
This is pretty cool – the
update_tscolumn is managed automagically by MySQL (other RDBMS have similar functionality), and Kafka Connect’s JDBC connector is using this to pick out new and updated rows from the database.As a side note here, Kafka Connect tracks the offset of the data that its read using the
connect-offsetstopic. Even if you delete and recreate the connector, if the connector has the same name it will retain the same offsets previously stored. So if you want to start from scratch, you’ll want to change the connector name – for example, use an incrementing suffix for each test version you work with. You can actually check the content of the connect-offsets topic easily:
This is part 1 of a mini-series, but does show you how to build connections to stream data from MySQL into Kafka and then into a flat file.