Operations | Monitoring | ITSM | DevOps | Cloud

Knowing what you're spending on observability is useful. Knowing which team, service, or project is driving that spend is what actually lets you act on that information. Cost attribution is a core part of how Grafana Cloud approaches cost management and optimization.

Roughly a year ago, I left Honeycomb’s SRE team to join the newly formed Tenant team, which works on our Private Cloud offering. This team held some significant challenges on its roadmap if it wanted to demonstrate that the offering was possible, would be worth the cost, and could be done without representing a heavy tax on the rest of the organization.

As we entered 2026, enterprises raced to build internal AI engineering agents, automating incident response, code review, and support. The investment was real, but 88% of these pilots never reached production, and teams are now in rebuild mode, trying to understand why. Live runtime validation was the key architectural decision skipped in these v1 agents and it’s still missing from many v2 designs. Agents need to verify their reasoning against production before they act.

AI agents fail differently from the software we spent the last two decades learning to monitor. We hear some version of the same story from teams shipping agents to production: an agent starts producing wrong answers. Not obviously broken: confident, well-formatted, plausible wrong. The logs are clean, latency looks healthy, and error rates sit at zero. Nothing flags a problem. A user eventually does.

When an alert lands, does your team already see the full story, or does the work start with pulling scattered data together from one tool after another? For many organizations it's the second one, where the incident itself takes a backseat while analysts hunt across dashboards. The evidence is right there, scattered across platforms that don't share context. Security observability exists to close that gap.

Decisions in each SDLC phase from planning to design, development, testing, deployment, and maintenance are made without sight of live production behavior. AI coding agents are widening that visibility gap faster, working faster than human engineers ever could. This piece maps exactly how this gap presents at each phase, and the harm that this brings.

This post is a joint effort by the OpenLIT and VictoriaMetrics teams. OpenLIT brings the OTel-native GPU collector for NVIDIA, AMD, and Intel hardware, while VictoriaMetrics provides the storage and query layer for the resulting metrics. We wrote it together to show how the two projects fit into a single, self-hosted observability pipeline, and to share the queries and rules that worked well for us along the way.