Operations | Monitoring | ITSM | DevOps | Cloud

Grafana AI Observability is our new database and platform for observing AI Agents. Over the past year at Grafana Labs, we built Agents and we needed a way to understand how they are performing, what are the costs associated with them, what's the error rate or time to the first token as well as how they are behaving. Grafana Staff Engineer, Ivana Hučková provides a deep dive demo on how Grafana AI Observability connects our experience building Agents with our experience building observability systems.

Context Offloading is a pipeline solution for managing Observability with AI Agents. If you are building AI Agents that work with real data, the context window can very easily get filled with bloated context that the Agent does not really need. Sven demonstrates "Context Offloading", a solution that stores the JSON result and sends only the summary of the JSON blob, making the LLM loop performance much quicker and keeping your context window small.

MCP vs CLI: which one should your AI agent actually use? We get into it with Grafana's cloud and OSS MCP servers and gcx.

Grafana Assistant is going places you might not expect — including healthcare. Golden Grot winner Oren Lion from TeleTracking reveals how Grafana Cloud supports their systems that help keep patient care moving — and how Assistant enables teams to get from “what happened?” to “here’s why” faster. From moon landings to patient care, Grafana is everywhere. Congratulations to Oren, Chris Johnson, Mark Munson, and the entire TeleTracking team on winning this year's Golden Grot Award for Pioneering AI in Observability!

This is an excerpt from a real AI team weekly meeting where we talk about the stuff we build and occasionally also demo them! In this one, Principal Software Engineer Sven Großmann demos how he integrated AI Observability into the OTel demo, complete with the guards feature he introduced last week, and Principal Software Engineer Yas Ekinci gives a rare glimpse of LLMSpec, the internal counterpart of the o11ybench benchmark that we use to evaluate Assistant.

Tempo 3.0 introduces a major architectural shift that decouples the read and write paths, with Kafka handling durability on the write side and a new live store serving recent traces on the read side. Blocks are now written at a replication factor of one instead of three, significantly reducing storage overhead. This release also brings TraceQL metrics to general availability, adds comparison operators for filtering metric results at query time, and introduces a new Tempo CLI redact command for removing sensitive trace data on demand without waiting for retention to expire.

In the May edition of the Kubernetes Monitoring Helm chart office hours, we discuss the version 4.1 release, the upcoming 4.2 feature release, and we discuss the upcoming deprecation of the 1.x and 2.0 versions.

AI agents are only as useful as the context they can access. With gcx, your coding agents can connect to Grafana and query real-time production telemetry from your Cloud, Enterprise, or OSS environment. The best part: it avoids the upfront context bloat that can come with loading tools before you even send a prompt. gcx uses a CLI approach, so there’s zero token cost until your agent actually needs to run a query.

When a compromised GitHub Actions workflow targets your CI/CD pipeline, how do you respond — and what do you change so it never happens again? Nick and David from Grafana Security walk through a real supply chain incident triggered by a pull_request_target misconfiguration, showing exactly what broke, what tools caught it, and what the team rebuilt afterward.

This is an excerpt from a real AI team weekly meeting where we talk about the stuff we build and occasionally also demo them! In this one, Principal Software Engineer Sven Großmann shows a new feature he's working on for AI Observability, called "guards". We're showing parts of our team meetings to build in public in some small way and give you a sneak preview of what's to come. But not all features we show may make it to production! You've been warned. :)