Operations | Monitoring | ITSM | DevOps | Cloud

It's been almost exactly one year since we issued our AI mandate here at Honeycomb, and we've been doing some reflection. When we issued our mandate, it's not like we hadn't been using AI. We were the first in the industry to bake a feature powered by AI into our product, way back in May of 2024. Many of us had been experimenting and using these tools in our spare time. But we believe that software is the killer app for AI.

Every AI agent architecture diagram shows the same five layers: perception, memory, reasoning, action, and feedback. Each layer assumes the one before it worked correctly, and none of them can confirm that once the agent runs against live production data. Runtime context is the sixth layer most designs leave out, and it’s the one that decides whether any of the other five can be trusted.

Say you get a support escalation about a page in the app that won’t load. But when you pull up your synthetic checks, they're all green: 100% uptime, probes are passing. Something's not adding up, but which one do you trust? If you’ve run Grafana Cloud Synthetic Monitoring, you’ve been on both sides of this. Sometimes it's the ticket: real users hit a wall on the path but your checks pass cleanly. Other times, it’s the inverse.

What it takes to prove AI value with LLM observability Over six months, the Elastic IT team ran internal AI applications that returned $2.5 million in operational time to the business.1 A conversational support assistant moved us from zero digital resolution, where anything complex became a ticket, to 30% of support interactions closing without one.

When our company pushed everyone to start using AI tools, I thought about what it would mean for my team. As a remote company, we are already challenged by the lack of organic human connection. Every connection is planned and takes effort, and now, AI adds another layer. People now spend part of their day collaborating with a tool rather than with a person, which can take away from the time we spend learning from each other.

AI agents are only as effective as the context they can access. The problem is that most of that context lives outside your observability platform.

Summary: Machine learning-based anomaly detection improves observability by learning normal system behavior instead of relying only on static thresholds. This article explains how vmanomaly, its MCP server, purpose-built skills, and an LLM-powered UI copilot help engineers explore telemetry, investigate anomalies, build MetricsQL queries, select suitable models, apply business constraints, and validate configurations through natural language.

AI model drift is when an AI system's performance and accuracy degrades over time because the data, user behavior, or business environment has changed since the model was trained or evaluated. Even if latency, uptime, and infrastructure metrics remain healthy, model quality can quietly decline, leading to less accurate predictions, inconsistent responses, and reduced user trust.