Operations | Monitoring | ITSM | DevOps | Cloud

Welcome to the third and final part of our series on AI norms and values. Parts of this doc were extracted and published separately on substack; as a whole, they describe the principles we hold pertaining to technology and AI, and the ethical commitments we make to each other and our customers. We set out to write about AI, and ended up writing about ourselves. These documents are not meant to be aspirational ones; they are derived from how we do our work every day in honeycomb.

As agentic AI workflows gain traction within organizations, those organizations are asking how to account for their behavior while keeping costs manageable. Some are sticking with the old three pillars of observability approach: take a measurement to create a metric, record output to a log, and track serial progress with a trace. Each of these is useful, but treating them as distinct formats from the start means paying for them distinctly too. Separate storage doesn't come cheap.

I'm investigating repeated errors in my e-commerce application, and I need to get enough context in a single Honeycomb query to piece the entire picture together. Each query returns events based on the event's WHERE clauses, but I want to know several things from outside of the event that recorded an error. Things like: Those attributes are all over the trace. That's going to make a single query tough, right? Wrong!

It's been one year since we issued an AI mandate inside Honeycomb, and we've been doing a lot of reflecting internally on how far we've come and how far we have yet to go.

Sampling is a core skill that everyone who runs an observability pipeline at scale will learn. There are lots of tradeoffs within the various decisions you'll make from reducing bandwidth, CPU, and memory, to reducing costs and making the observability backend's performance better for users. Historically, there have only been three mechanisms, each with their own tradeoffs: However, there is a secret fourth option: adaptive tail sampling—which changes those tradeoffs.

A few months ago, Darragh Curran, CTO at Fin (formerly Intercom) set a public goal to double engineering productivity and nearly tripled it instead. They did so by pulling a few levers: AI writing code at scale, building an AI-driven PR review system, leveraging observability as a trust mechanism, and with leadership becoming more hands-on through the transition.

It's been almost exactly one year since we issued our AI mandate here at Honeycomb, and we've been doing some reflection. When we issued our mandate, it's not like we hadn't been using AI. We were the first in the industry to bake a feature powered by AI into our product, way back in May of 2024. Many of us had been experimenting and using these tools in our spare time. But we believe that software is the killer app for AI.

When our company pushed everyone to start using AI tools, I thought about what it would mean for my team. As a remote company, we are already challenged by the lack of organic human connection. Every connection is planned and takes effort, and now, AI adds another layer. People now spend part of their day collaborating with a tool rather than with a person, which can take away from the time we spend learning from each other.

AI model drift is when an AI system's performance and accuracy degrades over time because the data, user behavior, or business environment has changed since the model was trained or evaluated. Even if latency, uptime, and infrastructure metrics remain healthy, model quality can quietly decline, leading to less accurate predictions, inconsistent responses, and reduced user trust.

BubbleUp has always been the fastest way to figure out what a group of outliers have in common. Draw a box around a band of slow traces, a cluster of errors, or any set of events you're interested in, and BubbleUp compares that selection to the baseline across every dimension you've sent us. It's how Honeycomb users find the "unknown unknowns" that dashboards can’t show you.