Operations | Monitoring | ITSM | DevOps | Cloud

Agent Observability Deep Dive Demo | Grafana Cloud

Grafana AI Observability is our new database and platform for observing AI Agents. Over the past year at Grafana Labs, we built Agents and we needed a way to understand how they are performing, what are the costs associated with them, what's the error rate or time to the first token as well as how they are behaving. Grafana Staff Engineer, Ivana Hučková provides a deep dive demo on how Grafana AI Observability connects our experience building Agents with our experience building observability systems.

SaaS vs Self-Hosted Observability (Including BYOC): Cost, Latency, Control and Governance

Most teams don't choose an observability deployment model, they inherit one. Someone signed up for a SaaS trial three years ago, telemetry volume grew 40x, and now Finance is asking why the observability line item is the fourth-largest infrastructure cost on the P&L. Or the opposite: a self-hosted Prometheus and Elasticsearch stack was stood up when the company had twelve services, and now it needs a team of two just to keep it upright while everyone else waits on slow dashboards during incidents.

Comprehensive Postgres observability: is your database healthy, and will it stay that way?

A Postgres database is one of the most complex pieces of software you run. It stores your data durably so a power loss doesn't lose a committed transaction. It hands out real transactions while thousands of queries fight over the same rows. It stays available when a node dies, lets you rewind to a point in time after someone drops the wrong table, and constantly tunes itself with background jobs. Two questions matter about all of that machinery: is it healthy right now, and will it stay healthy?

Honeycomb Named a Visionary in the 2026 Gartner Magic Quadrant for Observability Platforms

For the third consecutive year, Honeycomb has been recognized for its Ability to Execute and Completeness of Vision, and we believe for its strong vision around fast, flexible, high-cardinality querying that helps engineers understand not just that something broke, but why. The software development lifecycle has collapsed. The neat sequence of plan, build, test, and ship that teams have relied on for 20 years is now happening in a single afternoon. AI writes a large share of the code.

What Comes Next: Honeycomb Demos at O11yCon 2026

In this session at O11yCon, Purvi Kanal, Jamie Danielson, and Martin Holman demoed the new Canvas. Canvas now understands OpenTelemetry GenAI semantic conventions, and can show agent invocations, LLM calls, and tool calls all in one trace view. Humans and agents can work in the same place, with skills that let each team encode their own expertise so both the agent and their colleagues can use it. Multiplayer support means you can see your teammates' cursors and share charts.

Reproducing split brain on CloudNativePG

We run Postgres under an operator for automatic failover. That is a promise about what happens during a failure, so the only way to know you have it is to cause the failure and watch. The docs tell you what should happen. A config review tells you which knobs are set. Neither tells you how long an isolated primary keeps accepting writes after its replacement has been promoted, and that number decides whether a failover is clean or leaves you with two versions of your data.

The Gremlin app for Dynatrace: resilience testing and reliability scoring, built on the observability you already trust

Dynatrace gives engineering teams deep, real-time visibility into every service they run. That visibility is the foundation of every effective reliability practice, and it's exactly the foundation Gremlin is built to extend. Once you can see how your distributed systems behave today, the next step is knowing how they'll behave under failure tomorrow—and to do it before those failures happen.

How to Choose an Observability Platform: A Growth-Stage Framework

Every engineering team eventually hits the same wall: the monitoring stack that was fine six months ago now can't answer the questions you're actually asking during an incident. This isn't a tooling failure, it's a growth mismatch. The right observability platform at 10 engineers is rarely the right one at 100, and the cost of getting that wrong isn't the subscription fee, it's the migration you'll be forced into later, usually mid-incident, usually under pressure.

It was surprisingly hard to break CloudNativePG replication

I wanted a Postgres replica that falls behind its primary. Sounds easy. It wasn't. Every time I cut the replica off from the primary, its lag jumped for a moment and then dropped right back to zero. It just would not stay behind. Chasing down why turned into a fun little tour of how a Postgres standby keeps itself alive, and what CloudNativePG quietly sets up behind the scenes.

Why Static Reachability Isn't Enough for CVE Remediation

Most CVE remediation tools can tell you that a vulnerability could be exploited. Few can confirm whether it actually is. A scanner flags the same CVE in two services and marks both as vulnerable. Only one of them ever runs the flawed code in production. That gap, reachable in theory versus reachable in fact, is the real problem, and static analysis alone cannot close it.

On Release Days We Wear Teal Episode for release 4.19

In this episode, Leon explores some of the new features, functions, updates, and improvements in release 4.19, which includes a raft of AI-enabled features including the Cribl Apps, integrated MCP server, and the fact that AI features are now turned on by default. For more information, check out these links.

Cost attribution in Grafana Cloud: Manage spend across observability and testing workflows

Knowing what you're spending on observability is useful. Knowing which team, service, or project is driving that spend is what actually lets you act on that information. Cost attribution is a core part of how Grafana Cloud approaches cost management and optimization.

Embracing the Code Review Bottleneck

Roughly a year ago, I left Honeycomb’s SRE team to join the newly formed Tenant team, which works on our Private Cloud offering. This team held some significant challenges on its roadmap if it wanted to demonstrate that the offering was possible, would be worth the cost, and could be done without representing a heavy tax on the rest of the organization.

Why Internal Agents Must Be Rebuilt with Runtime Context

As we entered 2026, enterprises raced to build internal AI engineering agents, automating incident response, code review, and support. The investment was real, but 88% of these pilots never reached production, and teams are now in rebuild mode, trying to understand why. Live runtime validation was the key architectural decision skipped in these v1 agents and it’s still missing from many v2 designs. Agents need to verify their reasoning against production before they act.

Build a Docker Monitoring Dashboard in Minutes with Claude MCP + Uptrace

In this video, we use Claude MCP to create and merge Docker container dashboards in Uptrace — directly from the terminal, no manual clicking required. What you'll see: CPU, memory, network, and disk I/O dashboards created with plain text prompts Two dashboards merged into one unified view Dashboard exported as YAML for version control.

Let's break autovacuum in Postgres: reproducing failures to make it observable

Autovacuum is one of those Postgres background jobs that quietly keeps your database healthy. It cleans up the dead row versions that every UPDATE and DELETE leaves behind, and it keeps the database away from a hard transaction-ID limit that would take it offline. Most of the time you don't think about it, because it just works.

Introducing Harness AgentTrace: An Observability and Guardrail Framework for AI Agents | Harness Blog

AI agents fail differently from the software we spent the last two decades learning to monitor. We hear some version of the same story from teams shipping agents to production: an agent starts producing wrong answers. Not obviously broken: confident, well-formatted, plausible wrong. The logs are clean, latency looks healthy, and error rates sit at zero. Nothing flags a problem. A user eventually does.

Security Observability: Pillars, Use Cases, and How It Works

When an alert lands, does your team already see the full story, or does the work start with pulling scattered data together from one tool after another? For many organizations it's the second one, where the incident itself takes a backseat while analysts hunt across dashboards. The evidence is right there, scattered across platforms that don't share context. Security observability exists to close that gap.

SDLC Phases and the Reliability Gap AI Can't Close

Decisions in each SDLC phase from planning to design, development, testing, deployment, and maintenance are made without sight of live production behavior. AI coding agents are widening that visibility gap faster, working faster than human engineers ever could. This piece maps exactly how this gap presents at each phase, and the harm that this brings.

GPU Observability with the OpenLIT Collector and the VictoriaMetrics observability stack

This post is a joint effort by the OpenLIT and VictoriaMetrics teams. OpenLIT brings the OTel-native GPU collector for NVIDIA, AMD, and Intel hardware, while VictoriaMetrics provides the storage and query layer for the resulting metrics. We wrote it together to show how the two projects fit into a single, self-hosted observability pipeline, and to share the queries and rules that worked well for us along the way.

Instant Kubernetes Observability with Proxymock #speedscale #kubernetes #ebpf #devops #cloudnative

Learn how to get instant observability into your Kubernetes cluster by installing the Speedscale operator and proxymock tool. In this step-by-step tutorial, we walk you through setting up the operator to capture live network traffic (including encrypted traffic, API calls, and database calls) without complex instrumentation or manual configuration.

Adding Routing Intelligence To Your Observability Stack

Observability has a blind spot, and for most teams it sits at the network layer. You instrument your services, scrape metrics into Prometheus, ship logs somewhere searchable, and build dashboards that tell you when something is wrong inside your infrastructure. But the routing that carries traffic to and from that infrastructure often lives entirely outside the stack, watched through separate tools that do not talk to your alerting. This piece looks at why routing belongs in your observability pipeline and what it takes to get it there.

Best PR Review Tools for AI-Generated Code (2026)

Most PR review tools answer one question: does this code look correct? They scan the diff, flag known patterns, catch security violations, and post inline comments before merge, which is useful, but only half the review. The other half is behavioral: does the code actually behave correctly once it’s running against your live system?

Why partners love working with Cribl

Hear directly from Cribl partners—including AWS—about what it’s really like to work together. This short is for technology and cloud partners, consulting firms, and customers who want a quick, human view of Cribl’s partner ecosystem and the value it delivers. In under two minutes, partners highlight Cribl’s partner program, the people they work with, and the outcomes they’re delivering for joint customers. You’ll hear about the FedRAMP opportunity, why “it’s all about the data” for AWS, and how Cribl helps get data where it needs to be for shared customers.

Beyond performance monitoring: Understand the user experience with Grafana Cloud Frontend Observability

You've optimized your Largest Contentful Paint. Your Time to First Byte is under 200ms. Your Lighthouse scores are green. And yet, your checkout conversion rate is quietly dropping. A segment of users in Southeast Asia is churning. Your support team is fielding tickets about a form that "just doesn't work" and you have no idea which one. Traditional frontend performance monitoring tells you whether your application is fast. It doesn't tell you whether people are actually succeeding when using it.

What Is LLM Observability? A Complete Guide

If you run LLM features in production, your most dangerous failures are the ones your monitoring never flags. Your LLM feature passed every test, and the demo went great. Three weeks after launch, a support ticket lands: the chatbot quoted a refund policy that does not exist. The dashboards are all green, and the same prompt answers correctly when you retry it. This is the blind spot LLM observability exists to close. Your existing tools saw the request come back fast with a clean status code.

30 to 70 PRs a Day: How We Managed to Not Wreck Our Systems

In this two-part blog series, I give a detailed report-out on how our Honeycomb engineering team 2.5x-ed our throughput using AI without breaking everything or lowering our standards for quality. Part 1 explains how we did it and shows data about how that ramp-up happened. Part 2 shares what we learned.

AI Amplifies Your Existing Practices: Lessons from Our Shift to an AI-First Strategy

In this two-part blog series, I give a detailed report-out on how our Honeycomb engineering team 2.5x-ed our throughput using AI without breaking everything or lowering our standards for quality. Part 1 explains how we did it and shows data about how that ramp-up happened. In this blog, I share what we learned. The “platform engineering” frame and the “autonomy, ownership, feedback loops” frame are the same frame, spoken in two different vocabularies.

Datadog named Leader in 2026 Gartner Magic Quadrant for Observability Platforms

We are thrilled to announce that Datadog has been named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms, for the sixth consecutive year. We believe this recognition reflects our continued focus on helping customers observe, secure, and act on everything that matters across their technology stack. Datadog was positioned highest in Ability to Execute in the 2026 Gartner Magic Quadrant for Observability Platforms.

Grafana Labs named a Leader again in the 2026 Gartner Magic Quadrant for Observability Platforms

We’re delighted to share that Grafana Labs has been named a Leader in the Gartner Magic Quadrant for Observability Platforms for the third consecutive year. Notably, we’re also positioned furthest in “Completeness of Vision” for the second year in a row.

5 Things to Know About Context Engineering

Software systems are getting better at understanding themselves. The mix of richer telemetry, smarter pipelines, and agentic AI is shifting observability from a passive record of events into something more active and useful. That shift is what we mean by context engineering. We recently partnered with O’Reilly on a report by David Beale that introduces the discipline. Before you read it, here are five things worth knowing.

What is AI cost observability? A guide to tracking LLM and AI spend

AI cost observability is the practice of measuring, attributing, and analyzing AI workload costs at the request, model, and workflow level in real time. It connects cloud infrastructure spend, inference and token costs, and business attribution (cost per feature, team, customer, or product) so engineering, finance, and product teams can see where AI spend goes and whether it creates value.

AI is Exposing Observability's Dirty Secret

The 3 pillars of observability are breaking. For years, dev teams relied on Logs, Metrics, and Traces to know when something went wrong. But now? AI agents are writing, deploying, and changing code in real-time. When an AI hallucination pushes a bug to production, standard monitoring sees nothing wrong.To survive the AI era, we need a 4th Pillar of Observability. Watch to find out what it is and why the old way of monitoring just became obsolete.

Shipping Is Your Company's Heartbeat: A Letter from a CTO

The world is especially hard right now. The future of the software engineering profession looks more uncertain than ever. Execs are under heavy pressure to turn AI into magic results, and teams are fighting product competition and AI-induced burnout on one side, melting mental models and hellish oncall on the other side. Observability was supposed to be a solved problem by now.

Lattice Watch: Smarter Guardrails for Design System Observability

One of the hardest challenges facing platform teams is wrangling the rising volume of PRs looking to add drift to the systems we've invested in. It's impossible to catch them all, so it's more important than ever to invest in building stronger guardrails so our product teams can keep building quickly and catch issues before they merge to main. Linters are a great tool to reach for first.

Observability: The Complete Guide (2026)

When something breaks in a distributed system, "is it down?" is the easy question. "Why is it down, and where exactly?" is the one that actually costs engineering teams time. Observability is the practice and the tooling built to answer that second question, and it's become one of the most important disciplines in modern software operations.

Observability vs. Monitoring for AI Systems

Monitoring tells you when an event you predicted has actually happened. Observability lets you investigate behavior you may not have predicted at all. For most of the past decade, that distinction was something teams could afford to treat as a philosophical debate, because their systems failed in expected ways that had been seen before. A memory leak, a bad deploy, a saturated connection pool. You could build a dashboard and alerts for each and sleep reasonably well.

Network Observability Tools: Complete Guide for Cloud-Native Applications

Modern IT ecosystems have undergone a profound transformation. Organizations have shifted from monolithic applications running on static infrastructure to highly distributed, cloud-native environments powered by microservices, containers, and Kubernetes. This shift has unlocked unprecedented scalability and agility, but it has also introduced new layers of complexity that traditional monitoring tools were never designed to handle.

What Is Observability 2.0? Meaning, Key Features, and How to Adopt It

How many tools does your team need to answer one question about production? For most enterprise IT teams the honest count is four: a metrics dashboard, a log analyzer, a tracing tool, and the spreadsheet where someone stitches the other three together during an incident. Each of those tools stores its own copy of the truth and sends its own bill.

15 Best AI Observability Tools for Production Teams in 2026

AI applications generate far more than model outputs. Every request includes prompts, retrieval, tool calls, agent steps, latency, token usage, and evaluation signals that all contribute to the final response. When something goes wrong, engineering teams need to understand what happened, why it happened, what it cost, and whether the outcome met quality expectations.

Multi-Agent Collaboration on a Shared Canvas

This post was co-written with Staff Software Engineer Martin Holman. Honeycomb Canvas is a collaborative investigation environment. When something goes wrong in production, multiple engineers might join the same Canvas to debug it together. Each person has their own AI agent, so they can pursue their own conversation thread and line of inquiry. This creates an opportunity for coordination.

Driving Value from Puppet Metrics: Puppet Observability Data Connector

The Puppet Observability Data Connector is a premium Forge module included with Puppet Enterprise Advanced (PEA). This module provides a deeper dive into your Puppet agent reports. Visualizing these metrics gives you a great way to identify what is healthy and unhealthy in your environment.

Deterministic vs Probabilistic AI Engineering Explained

Deterministic processes carry one guarantee: the same input will produce the same output. That guarantee built the entire observability stack. AI broke that contract by reasoning in terms of probability. The same input can now produce different outputs, whether from AI-generated code that carries assumptions invisible in staging, or from distributed systems where timing creates failures that no pre-captured telemetry can anticipate.

From Prototype to Production With AWS AgentCore

"Hello world, this is your agent speaking!" The agent loop! The LLM is calling tools, the answers are sensible, and the sky's the limit. Now, as you look forward to production, you look for a composable toolset, something that can grow with your use case and system needs. That's what we created with Honeycomb Canvas: a collaborative investigation space where AI agents help you understand, fix, and learn about your system.

Tech Talk: Observability Simplified, APM and Network Behavior

Participants are welcomed to a session titled "Observability Simplified," focusing on user experience, application performance, and network behavior. This second part of a three-part series highlights how the Splunk Observability Cloud and Cisco ThousandEyes can create a unified view of applications, infrastructure, and network performance. Key discussions include addressing siloed troubleshooting, enhancing visibility, and a live demo showcasing how to identify network issues affecting application performance. Attendees are encouraged to participate in the Q&A and are reminded that the session will be recorded for future reference.

Observability for LLM Apps and Agents: OpenLIT SDK + VictoriaMetrics observability stack

Many “LLM observability with OpenTelemetry” tutorials stop at a single chat.completions span. That works for a demo, but it leaves gaps once an agent fans out into 30 tool calls, two vector-DB queries, three handoffs, and a 90-second tail latency you need to attribute. This post wires the OpenLIT SDK (50+ instrumentations, OTel GenAI semantic conventions, one line of code) into the full VictoriaMetrics observability stack and shows query examples that turn agent telemetry into decisions.

Unified Observability: Moving IT Teams from Reactive to Predictive

What does it take to stop an outage before it starts? In many cases, the warning signs are already there, scattered across different monitoring tools, which makes it difficult to see the full picture before issues escalate. When an incident occurs, engineers often spend valuable time piecing together metrics, logs, traces, and alerts to determine the root cause. Every minute spent investigating extends the outage and increases its business impact.

How SRE Practices Improve Trust in Digital Finance and Healthcare Platforms

Trust used to be a brand problem. Now it's an uptime problem, a latency problem, a data integrity problem, and sometimes a "why is the payment button spinning again?" problem. For digital finance and healthcare platforms, users don't separate the service from the system behind it. If the app fails, the business feels careless. If records lag, confidence drops. If a transaction disappears for even a few seconds, panic arrives fast.

Could vs. Should: The First Year Managing an SRE Team

As of today, I’ve drafted this post upwards of 10 times – it’s old enough that the version I first started working on was called “Reflections on 1 Year of SRE Management” (I’m currently at 2.5 years). But everything I learned during that first year became critical for the next.

Why Modern IT Incident Response Needs Social Sentiment Analysis

IT operations teams face an ongoing battle against alert fatigue. Despite running sophisticated telemetry and baseline Application Performance Monitoring, engineers are often bombarded with notifications that lead nowhere. Relying purely on internal dashboards creates a massive visibility gap, and when critical incidents slip through the cracks, the financial damage is swift and severe. To close this gap, DevOps professionals are increasingly looking beyond traditional server metrics and turning to a surprising source for early warning signals: public social sentiment.

How AI Agents Are Changing Each Agile SDLC Phase

The Agile software development lifecycle was designed to surface problems early, with short sprints, iterative testing, and continuous integration built on the premise that faster feedback loops produce better software. AI coding tools have changed the velocity equation across every phase of that loop, but the phases designed to catch failures are struggling to keep up because build speed and validation capacity have not accelerated at the same rate, and the gap between them is widening with every sprint.