Operations | Monitoring | ITSM | DevOps | Cloud

Roughly a year ago, I left Honeycomb’s SRE team to join the newly formed Tenant team, which works on our Private Cloud offering. This team held some significant challenges on its roadmap if it wanted to demonstrate that the offering was possible, would be worth the cost, and could be done without representing a heavy tax on the rest of the organization.

A year ago, I wrote “It’s the End of Observability (and I Feel Fine).” The upshot of that post was that AI was about to fundamentally change the way we approach systems design and operation in the future. In the grand tradition, I’d like to revisit my claims from then and see how my predictions panned out.

In this two-part blog series, I give a detailed report-out on how our Honeycomb engineering team 2.5x-ed our throughput using AI without breaking everything or lowering our standards for quality. Part 1 explains how we did it and shows data about how that ramp-up happened. Part 2 shares what we learned.

In this two-part blog series, I give a detailed report-out on how our Honeycomb engineering team 2.5x-ed our throughput using AI without breaking everything or lowering our standards for quality. Part 1 explains how we did it and shows data about how that ramp-up happened. In this blog, I share what we learned. The “platform engineering” frame and the “autonomy, ownership, feedback loops” frame are the same frame, spoken in two different vocabularies.

We recently wrapped up a large-scale, multi-month Kafka migration project. We used to run self-hosted Confluent Platform and ZooKeeper as clusters of AWS EC2 instances, and now all of our Kafka clusters run open-source Apache Kafka 4.1.1 running in KRaft mode and deployed to AWS EKS.

The world is especially hard right now. The future of the software engineering profession looks more uncertain than ever. Execs are under heavy pressure to turn AI into magic results, and teams are fighting product competition and AI-induced burnout on one side, melting mental models and hellish oncall on the other side. Observability was supposed to be a solved problem by now.

One of the hardest challenges facing platform teams is wrangling the rising volume of PRs looking to add drift to the systems we've invested in. It's impossible to catch them all, so it's more important than ever to invest in building stronger guardrails so our product teams can keep building quickly and catch issues before they merge to main. Linters are a great tool to reach for first.

AI applications generate far more than model outputs. Every request includes prompts, retrieval, tool calls, agent steps, latency, token usage, and evaluation signals that all contribute to the final response. When something goes wrong, engineering teams need to understand what happened, why it happened, what it cost, and whether the outcome met quality expectations.

Monitoring tells you when an event you predicted has actually happened. Observability lets you investigate behavior you may not have predicted at all. For most of the past decade, that distinction was something teams could afford to treat as a philosophical debate, because their systems failed in expected ways that had been seen before. A memory leak, a bad deploy, a saturated connection pool. You could build a dashboard and alerts for each and sleep reasonably well.

This post was co-written with Staff Software Engineer Martin Holman. Honeycomb Canvas is a collaborative investigation environment. When something goes wrong in production, multiple engineers might join the same Canvas to debug it together. Each person has their own AI agent, so they can pursue their own conversation thread and line of inquiry. This creates an opportunity for coordination.