Kafka Is A Team Sport Redux — #kafka #apachekafka #streamingdata #data #confluentkafka

The feedback from Saturday’s blog post Kafka Is A Team Sport was very positive. I still felt I hadn’t quite got my point across though, well not clearly enough. Then it hit me.

The Three Functions

I see the current ecosystem as three distinct parts. Development, DevOps Engineering and Data Engineering. They all have different functions and, in my eyes, operate differently.

Excuse the Apple Pencil, they don’t allow sharp objects in here….. and it was early and I’ve only had one cup of tea.

Clients

The application development area. The Client APIs whether that’s Java, Go, Python, Clojure, PHP, C++ or anything else I can think of. The main thing is that there is a development task that needs writing. It might be a producer, a consumer, a streaming job or even a KSQL query.

I honestly don’t need to care how Kafka works at this point, I just want to get a message or read a message. And I can’t program in any of those languages I have the REST proxy to help me too. HTTP still has it’s place in the world, just be careful with the security.

Tooling

Jobs that require setting up for things like Kafka Connect and Replicator/Mirror Maker are more tooling jobs. When I say tooling I really mean “sorting out configuration” and sending it over REST (especially with Connect). Do I need to know how to program? No, not really.

Backend

Brokers, latency, capacity planning and (oh no!) Zookeeper, are admin functions, or DevOps if you will. I’d never ask a software developer get into the minute detail of the server.properties file or look at volume throughput. In the same way I’d never ask them to tune a PostgreSQL database (in fact I wouldn’t touch that either, I’ve no idea).

Originally published at http://dataissexy.wordpress.com on November 9, 2020.