Operations | Monitoring | ITSM | DevOps | Cloud
The Place Where Modern Operations & Technology Come Together
Institutional knowledge doesn't scale: Building an agentic data analyst
We’ve previously written about how deeply embedded data is in people’s day-to-day work at incident.io, and I’d have it no other way — demand for data is undoubtedly a good thing. What risks breaking at scale, however, is everything downstream of that demand: data-team capacity gets stretched thin, dashboard sprawl outpaces anyone's ability to maintain it, and stakeholders can't reach an answer without going through the data team.
Dashboards aren't (quite) dead
Historically, non-technical stakeholders would’ve had most of their data questions answered either through pre-built dashboards or by asking their Data team (or equivalent). Self-serve analytics tools went a step further by offering safe, governed datasets built by Data teams which let non-technical users dig into data without having to worry about how it joins together, how metrics like “revenue” are defined, and so on.
Don't add a read replica until you've read this
As the size and complexity of their relational database workload grows, every company eventually goes through the process of off-loading work on a read replica. It comes with lots of benefits, but at a cost of increased complexity. This article is about how we dealt with that, a lot of learnings, and some useful techniques. incident.io is an incident management product relied on by thousands of customers to be the thing that supports them through anything from a minor blip to a full outage.
How Zendesk ditched 15 years of patchwork tooling, in 10 weeks
Zendesk replaced 15 years of homegrown incident tooling and PagerDuty by migrating 1,200 engineers across 150 teams onto incident.io in just 10 weeks, cutting mean time to triage by 32%, saving $500k+ in year one, and eliminating 800+ hours of annual toil, with zero incidents on go-live day. Tom Monaghan (VP of Engineering Productivity & Product Reliability) and Anna Roussanova (Engineering Manager) share how they pulled it off and what's next as Zendesk helps build Investigations, our AI agent that starts digging into incidents the moment an alert fires.
Customers over control: how we measure On-call reliability
Our On-call product has a lot of great features: configuring escalation paths, viewing rotas and schedules, requesting cover, etc. However, when framing its reliability, we reduce it down to two critical pieces of functionality: It’s not that we’re happy if only these parts are working, but they are the most important parts. In this post, I'll go into more detail on how we think about their reliability.
Engineering teams in 2027
There's a conversation I keep having with our design partners at incident.io. It starts when I ask "what are you doing with AI internally?" and lands in a similar place every time. The shape of how their engineering teams work is changing fast. Not in vague "AI is transforming everything" ways, but in concrete, repeatable patterns. Different companies are building the same things. The frontier teams are six to twelve months ahead of the average, and they're describing the same future.
PagerDuty Rescue Program
We're announcing the PagerDuty Rescue Program. PagerDuty worked. For a long time, it was the standard. But the world's changed, and PagerDuty hasn't. The single biggest reason teams stay on PagerDuty isn’t the product - it’s the pain of leaving. So, we’ve removed every barrier. You've wanted out for a while. Now, nothing is stopping you.
Humans aren't fast enough for 4 9's
When thinking about Service Level Objectives (SLOs) and contractual Service Level Agreements (SLAs) for availability, I always like to put the percentages into concrete numbers. It’s easy to lose track of what’s meant when saying “99.95%” availability, and even more is lost when thinking how much harder it is to achieve 99.99% compared to 99.95%. On a monthly basis, and in concrete terms, 99.95% availability means you get 21 minutes and 55 seconds of downtime.
Behind-the-scenes: Building Post-mortems | incident.io team
We rebuilt our post-mortems from the ground up. In this video, Pete and the engineering team talk through how they built it: the decisions they made, the problems they were solving, and what it took to ship AI-native post-mortems.