by Datadog
Operating a complex distributed system such as Apache Kafka is no easy task. There are many moving parts that need to be understood when something goes wrong. Properly scaling Apache Kafka can be challenging: with brokers, partitions, leaders, consumers, producers, offsets, consumer groups, and more. Ensuring your setup is secure adds to the challenge.
From managing consistency, scaling to the correct number of partitions, and understanding under-replicated partitions, to the challenges of setting up proper security, this talk we will review common issues and solutions seen from the trenches helping teams around the globe with their Kafka infrastructure.
Founded by the team that originally created Apache Kafka, Confluent runs some of the world's largest Kafka clusters. We'll share what we've learned in the field. By the end of this talk you will have a collection of strategies to detect and prevent common issues with Apache Kafka; in a nutshell you will leave with more peace of mind, fewer nights of lost sleep for you, and more happiness for your users.