Operations | Monitoring | ITSM | DevOps | Cloud

I used to joke that I got my best parenting ideas from management books, and my best management ideas from parenting books. Now, I'm extending it to agents. Hear me out. When you're teaching a teen how to drive, you don't hand them the car keys on day one. You start in an empty parking lot. You ride along and ask them to talk through what they're doing.

The best alert is the one that never fires. Picture an idyllic Tuesday afternoon: last night's release nudged checkout p99 up 41%, but an agent noticed. Nobody got paged, and your SLOs didn't start burning. Instead, the agent acted and already narrowed it down to one retry change. It filed a ticket for the Payments team and flagged the PR. By the time anyone would have noticed, the fix is in review. That's what SRE was always supposed to feel like.

Yesterday, my partner told me about his new automated pipeline from a conversation recording to publication of a whole podcast, with social media blurbs and graphics and everything. It did more than he'd predicted the agent could do. "We are limited by what we can imagine," he said. Cheesy, but it kinda gave me goosebumps because it's (almost) true. Most of the experienced developers I know are concerned.

We just wrapped O11yDay London, and one theme ran through every session: writing code is no longer the hard part. Review queues, attention, operating agents in production, runaway token bills, and proving any of it is worth the money are where teams are getting stuck. The teams making progress are the ones who can see what's happening in production. Here are some highlights. We'll have videos available in the near future.

We're excited to share that Honeycomb has achieved yet another Amazon Web Services (AWS) Competency—the Resilience Technology Competency—which recognizes our ability to help customers detect, investigate, and recover from failures in production systems running on AWS. While receiving the competency designation is a fantastic recognition of the capabilities of the Honeycomb platform (we also received one in Financial Services earlier this year), it's ultimately not the point.

Your LLM application emits telemetry unlike anything else in your stack. Model calls, tool invocations, retrieval steps, and token usage arrive as spans whose attributes carry entire prompts and completions. That data is bulky, it's full of user content you may not be allowed to export, and depending on which instrumentation each service uses, the same fact can arrive under different attribute names.

Back in July, I wrote about how the Tenant team (the team behind Honeycomb Private Cloud (HPC)) has embraced the code review bottleneck to focus more of its work. One of the other challenges we have is that we're downstream of almost all the other teams at Honeycomb, meaning that we have to package up everyone's code and services, and how it gets provisioned! This is something impossible to handle through code review since there are so many engineers on other teams, and so few of us.