A few observations if I may. Yup KafkaCat is good for this sort of thing, perfect for testing and general mucking around and learning.

Once you get into a production scenario then all cards are basically off the table. Consumers are rarely written this way, they are there to run and consume. If you miss messages then you'll be rolling the offset back of the consumer group to a specific offset or timestamp, but you'll be reading everything back.

The key skill, I firmly believe, is writing good consumers and producers that embrace idempotency, force good ordering (either with the Transaction API or using a single partition(!)) and being able to handle previously read messages on the consumer group.

And if you're using Kafka Connect then make side effecting issues your friend, how you handle error tolerance and dead letter topics will define how much sleep you'll get LOL.

I do though encourage everyone to study the core CLI tools, not every organisation is allowed to use KafkaCat in production because of security issues and open source licenses etc.

Great post.