Operations | Monitoring | ITSM | DevOps | Cloud

The world is especially hard right now. The future of the software engineering profession looks more uncertain than ever. Execs are under heavy pressure to turn AI into magic results, and teams are fighting product competition and AI-induced burnout on one side, melting mental models and hellish oncall on the other side. Observability was supposed to be a solved problem by now.

One of the hardest challenges facing platform teams is wrangling the rising volume of PRs looking to add drift to the systems we've invested in. It's impossible to catch them all, so it's more important than ever to invest in building stronger guardrails so our product teams can keep building quickly and catch issues before they merge to main. Linters are a great tool to reach for first.

When something breaks in a distributed system, "is it down?" is the easy question. "Why is it down, and where exactly?" is the one that actually costs engineering teams time. Observability is the practice and the tooling built to answer that second question, and it's become one of the most important disciplines in modern software operations.

How many tools does your team need to answer one question about production? For most enterprise IT teams the honest count is four: a metrics dashboard, a log analyzer, a tracing tool, and the spreadsheet where someone stitches the other three together during an incident. Each of those tools stores its own copy of the truth and sends its own bill.

AI applications generate far more than model outputs. Every request includes prompts, retrieval, tool calls, agent steps, latency, token usage, and evaluation signals that all contribute to the final response. When something goes wrong, engineering teams need to understand what happened, why it happened, what it cost, and whether the outcome met quality expectations.

Monitoring tells you when an event you predicted has actually happened. Observability lets you investigate behavior you may not have predicted at all. For most of the past decade, that distinction was something teams could afford to treat as a philosophical debate, because their systems failed in expected ways that had been seen before. A memory leak, a bad deploy, a saturated connection pool. You could build a dashboard and alerts for each and sleep reasonably well.

Modern IT ecosystems have undergone a profound transformation. Organizations have shifted from monolithic applications running on static infrastructure to highly distributed, cloud-native environments powered by microservices, containers, and Kubernetes. This shift has unlocked unprecedented scalability and agility, but it has also introduced new layers of complexity that traditional monitoring tools were never designed to handle.

This post was co-written with Staff Software Engineer Martin Holman. Honeycomb Canvas is a collaborative investigation environment. When something goes wrong in production, multiple engineers might join the same Canvas to debug it together. Each person has their own AI agent, so they can pursue their own conversation thread and line of inquiry. This creates an opportunity for coordination.

The Puppet Observability Data Connector is a premium Forge module included with Puppet Enterprise Advanced (PEA). This module provides a deeper dive into your Puppet agent reports. Visualizing these metrics gives you a great way to identify what is healthy and unhealthy in your environment.

Deterministic processes carry one guarantee: the same input will produce the same output. That guarantee built the entire observability stack. AI broke that contract by reasoning in terms of probability. The same input can now produce different outputs, whether from AI-generated code that carries assumptions invisible in staging, or from distributed systems where timing creates failures that no pre-captured telemetry can anticipate.