Operations | Monitoring | ITSM | DevOps | Cloud

Understand how Kafka integrates with Apache Iceberg and experiment locally with Docker and Spark The streaming data landscape is evolving rapidly, and one of the most exciting developments is the integration between Apache Kafka and Apache Iceberg. While Kafka excels at real-time data streaming, organizations often struggle with the complexity of moving streaming data into analytical systems.

Developers love tools that make their lives easier. Whether it’s debugging, visualizing data, or managing clusters, the right tools can transform hours of manual work into just a few clicks. When it comes to Kafka - one of the most powerful but complex streaming platforms - having the right interface is critical. That’s exactly why we built our new Kafka UI Plugin.

When your telemetry pipeline starts falling behind, the countdown to production impact has already begun. One Bindplane customer operating a large-scale log ingestion pipeline built on the OpenTelemetry Collector and Kafka hit that breaking point. Instead of keeping pace with incoming data, their pipeline was ingesting just 12,000 events per second (EPS) per partition/collector—and this Kafka topic had 16 partitions. In aggregate, that was roughly 192K EPS.

We’re rolling out table-formatted topics for Apache Kafka, enabling open-source users to ingest, process, and emit data natively as Apache Iceberg tables. We’ve developed a new class of topics in Apache Kafka that stream data into lakehouse formats such as Apache Iceberg. The developer experience, similar to Diskless, is flexible, and the topics are opt-in via cluster configuration.

Finding yourself migrating and weighing the options when it comes to moving your entire Apache Kafka setup to a new cloud? This blog will point you to a solution to efficiently migrate your workloads to an Aiven for Apache Kafka cluster circumventing the headache that usually comes with cloud migration.

Diskless topics reward high-throughput workloads with large batches but can struggle with low-throughput patterns. Note: This analysis is based on testing with Diskless Kafka 4.0.0-rc15. Diskless topics are available for you to start experimenting with via the Inkless fork but the feature is still in development, and performance characteristics may change significantly as the technology matures. If you're: This post is for you!