Operations | Monitoring | ITSM | DevOps | Cloud

Never lose quorum in Apache Kafka - and what to do when you do

Apache Kafka's documentation and operational lore for KRaft are clear on one point: you shall never lose controller quorum. It is good advice, because a Raft quorum that can no longer elect a leader cannot commit metadata, and a Kafka cluster without a functioning controller cannot create topics, change ACLs, elect partition leaders, or otherwise make progress on its control plane.

Diskless Topics on Aiven for Apache Kafka

Diskless moves Kafka replication into object storage, removing cross-AZ fees. See how diskless topics work, then create one live on Aiven for Apache Kafka. AIVEN DATA PLATFORM The Aiven Platform is more than a collection of open source services for streaming, storing and analyzing data. The platform ensures that all services run reliably and securely in the clouds of your choice, are observable, and can easily be integrated with each other and with external 3rd party tools.

Troubleshoot Kafka issues across every layer of your stack with Kafka Console

Kafka is a crucial and widely used technology: 80% of the Fortune 100 rely on the event streaming platform as part of their stack, according to Apache. But Kafka issues can be complex to manage and even more difficult to troubleshoot, as the same symptom can point to very different problems. Suppose consumer lag on your checkout-events topic suddenly exceeds its SLA.

Deploying Apache Kafka Streams next to your data

If you’re using Apache Kafka for data streaming, then you’re likely to want to take data from one topic and write an amended version of that data to another topic, for instance: for filtering (give me the items that have been delivered), enrichment (add some more information to messages, from another data source) or anomaly detection (show me potential signs of trouble). The premier solution in this space is the Apache Kafka Streams library.

Skills as Guardrails: Contributing to Apache Kafka with AI, Without Knowing Every Module

Let me start with something most Kafka contributors think but rarely say out loud: nobody understands all of Kafka. I'm not a core committer and have only contributed a few times, but those contributions I have made have been in part thanks to using coding assistants. There are some issues with this approach though, the Apache Kafka project is huge. It's split into many parts: the core, the server, the client libraries, the streams engine, the storage layer, the consensus code, and more.

Engineer Gleb Lesnikov on Why Splitting a Monolith into Microservices May Sink Your Business

Gleb Lesnikov has been developing the system behind Dodo Pizza, an international fast-food chain, since joining the company in 2015. Over the years, the local business turned into a company running over 1,600 restaurants. When Dodo expanded rapidly, Gleb had to take key technical decisions that allowed the company to scale and succeed. Looking back, he reflects on the period when the company moved to a microservices architecture and explains how he managed to solve problems that could have caused the company to collapse.

What I got wrong about ClickHouse as a Kafka Person

Kafka is brilliant at moving events around, but sooner or later someone wants to actually query those events, perhaps aggregations, dashboards, or ad-hoc analytics over billions of rows. That is where ClickHouse comes in. It's the option for when stream processing is more than you need, but warehouse query latency is more than you'll tolerate.