The Best Books to Learn Apache Kafka

A practitioner’s journey to enlightenment

Apache Kafka is the proverbial runaway freight train of event-driven and Big Data-centric architectures, as well as this microservices thing that everyone seems to be rattling on about. Its tremendous power and flexibility have given it unstoppable momentum in both the industry and the open-source community; so much so that Kafka has become the default scan term of recruiters, along with other buzzwords such as ‘full-stack’, ‘micro frontends’ and ‘polyglot’. By some estimates, as much as 40% of Fortune 500 companies are using Kafka. Some might argue that these estimates are conservative.

Effective Kafka: A Hands-on Guide to Building Robust and Scalable Event-Driven Applications

Me, before reading Effective Kafka

Kafka: The Definitive Guide

Contrasting The Definitive Guide with Effective Kafka

  • The Definitive Guide is laser-focused on the technology platform, while Effective Kafka also educates you on architecture and best-practices and even goes into some of the concepts of distributed computing.
  • Effective Kafka covers the core open-source platform components — Kafka and ZooKeeper. The Definitive Guide also touches on some of Confluent’s contributions, such as Replicator and Schema Registry.
  • The Definitive Guide is a lot smaller — roughly 280 pages of content versus 460 pages of Effective Kafka. I hadn’t realised how short it was in comparison until I read it a second time. You’ll get through The Definitive Guide faster, if that matters to you.
  • I would argue that The Definitive Guide limits its audience from beginner to intermediate level. Effective Kafka can appeal to a broader range — from beginner to advanced, and beyond.
  • Effective Kafka discusses deployment topologies and networking in a lot more detail, even touching on technologies such as Docker and Kubernetes. The Definitive Guide is an older publication, preceding Effective Kafka by about three years. Perhaps for that reason, it omits any mention of Docker and Kubernetes.
  • Both books have their fair share of code examples written in Java. Effective Kafka has a broader API coverage of producer and consumer clients and is backed by a GitHub repository.
  • Effective Kafka has a broader coverage of tooling, not only the CLI tools that come with Kafka, but also third-party web-based UIs, such as Kafdrop. (Unsurprising, seeing that the author is a Kafdrop maintainer.)
  • Rebalance listeners — they do get a mention, but only in passing. Depending on your application, you may need complete knowledge of rebalance listeners to implement proper consumer exclusivity.
  • Transactions — the book was written before Kafka transactions were a thing. Effective Kafka dedicates a whole chapter to this topic.
  • Exactly-once delivery — gets a mention in the context of idempotent writes, but is incomplete without a discussion on transactions.
  • Resilience patterns, such as retries, dead-letters, timeouts, and so forth, are not given due attention. By comparison, Effective Kafka dedicates an entire chapter to dealing with failures. From experience, this is also one of the most overlooked areas in real-life systems built on Kafka.
  • Advertised listeners are omitted altogether. This is somewhat disappointing, being an intermediate concept that all self-respecting operators and developers should be well aware of. I think this is, in part, due to how Kafka used to be commonly deployed in 2017 — on virtual machines or bare metal. Perhaps having multiple listeners configured wasn’t as crucial back then, but it definitely is now. These days most developers run Kafka and ZooKeeper locally with Docker Compose.
  • Quotas are only mentioned from a monitoring standpoint. When operating shared Kafka infrastructure in a medium-large organisation, having a quota-free setup just doesn’t cut it. Effective Kafka dedicates an entire chapter to quotas and rightly so.
  • Security is covered in all of three paragraphs in The Definitive Guide. There is hardly any mention of SASL, which is the dominant broker authentication protocol. There is no mention of certificates and Access Control Lists (ACLs). Effective Kafka spends… get ready for this… 80 pages on the blasted topic. This is bonkers! It’s like having a book inside a book! And just about all of it is useful.
  • Partition assignment strategy is briefly touched on in The Definitive Guide, showing the differences between RoundRobinAssignor and RangeAssignor. Effective Kafka covers all four built-in assignors in detail and provides nice examples comparing their behaviour.
  • JMX monitoring is omitted from Effective Kafka. The Definitive Guide has a couple of paragraphs on it.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

  • Persistence models (B-Trees, LSMs).
  • Data encoding formats (JSON, XML, Avro, Thrift, Protobuf, etc.). It’s probably worth mentioning: Martin is one of the maintainers of Avro.
  • Replication — leader-follow architectures, lag, multi-leader replication, write conflicts and quorum replication.
  • Transactions — isolation levels, read and write skew, concurrency control mechanisms (serial, pessimistic and optimistic).
  • Faults in distributed systems — timings and clock synchronization, Byzantine failures, timeouts and asynchrony.
  • Distributed algorithms and areas such as consistency models, ordering, distributed transactions and consensus.
  • Batch processing, MapReduce and stream processing.
  • I only heard of consistency models in the context of eventual consistency, thinking that there is some strong form and a weak form and that’s that. Little did I know that consistency takes many forms and it’s not even an ordered spectrum — some consistency models allow different phenomena (or side effects, if you like) without necessarily being stronger or weaker.
  • I always associated distributed systems with microservices and saw them as an evolutionary step from monoliths. But I never appreciated the difficulties of dealing with failures where only parts of a system may fail or become unreachable. Just let it time out and retry… What can possibly go wrong?

Other books on Apache Kafka?

I have read a few more books that were related to Kafka than what’s listed here, and you may be wondering whether there are others that I would recommend. The answer, unfortunately, is an emphatic no.

I’m an experienced software engineer specialising in all manner of event-driven and microservices applications, polyglot stacks and performance tuning.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store