Kafka and the ELK Stack — usually these two are part of the same architectural solution, Kafka acting as a buffer in front of Logstash to ensure resiliency. This article explores a different combination — using the ELK Stack to collect and analyze Kafka logs. As explained in a previous post, Kafka plays a key role in our architecture. As such, we’ve constructed a monitoring system to ensure data is flowing through the pipelines as expected.
At Honeycomb, we’re pretty engineering-focused. We think our product is something every developer can benefit from, and we’re working to build something we would want to (and do!) use every day to make our own jobs easier.
Let’s start with the happy ending — after a long search, we managed to identify a Netty memory leak in one of our log listeners and were able to troubleshoot and fix the issue on time before the service crashed.
Grafana v6.0 Beta1 was released this week, and it’s packed with new features! This is one of the biggest updates to Grafana and introduces a new way to explore your data, support for log data, and includes tons of other enhancements. We hope you’ll give v6.0 Beta1 a try and let us know what you think. You can see a few of the highlights below, but check out all of the updates in the What’s new in Grafana v6 documentation.
