Operations | Monitoring | ITSM | DevOps | Cloud

I don't write SQL. Not by hand anyway. That might sound like a confession, but it's true of most analysts (and engineers) now. In the 9 months since joining incident.io I have built headcount models for hiring decisions, a Slack bot for assigning new customers, guidance for our AI data analyst, and a couple of internal web apps. If you asked me which of those counted as "analysing data", I'd struggle to say. A lot of that comes down to the "commercial" part of the job title.

Our cloud agents now test mobile pull requests on cloud simulators and leave screenshots and a recording behind. The agent collects the evidence, and a person decides whether the change is right. More and more of the code in that app is written by coding agents. Of the 144 pull requests opened against it in the last two weeks, 99 were opened with Claude Code or Cursor. The agents are good at writing the change. What they couldn't do was show it to you.

At incident.io we are huge fans of Postgres; we've written about it a lot over the years, including how to choose the right indexes and how we're proud of being boring (The Pet Shop Boys). We use Postgres as our primary transactional database, which as of today has ~900 tables, and counting! The vast majority of our codebase does something along the following lines: read some data from Postgres, execute some business logic, then write that data back to Postgres. It is not, however, always that simple.

At incident.io, we've spent the last two years building Investigations, our AI SRE. When you get paged, it starts investigating straight away, looking across your telemetry, recent deploys, past incidents, docs and code, and posts what it's found in your incident channel (or on your phone, if it's 2am and you're still deciding whether you need to get out of bed). By the time you open your laptop, you're starting at step six of triage rather than step one.

Six weeks into my internship, I was handed the biggest project I'd ever worked on: WhatsApp notifications for on-call paging. It was bigger by a large margin. When we scoped it out it broke into about a dozen chunks, each roughly the size of a whole project I'd done before. This is what I learned from it, and what it was like leading a project of that size as one of the most junior engineers at the company.

Like many modern software stacks, the incident.io platform is predominantly event-driven. For example, whenever you send us an alert, post a message to our agent on Slack, or update an entry in your Catalog - these are all events that then get enqueued on a message topic, meaning any of our downstream components that are interested in that event can subscribe and react asynchronously, such as sending a push notification or posting a reply to you in Slack.

We’ve previously written about how deeply embedded data is in people’s day-to-day work at incident.io, and I’d have it no other way — demand for data is undoubtedly a good thing. What risks breaking at scale, however, is everything downstream of that demand: data-team capacity gets stretched thin, dashboard sprawl outpaces anyone's ability to maintain it, and stakeholders can't reach an answer without going through the data team.

Historically, non-technical stakeholders would’ve had most of their data questions answered either through pre-built dashboards or by asking their Data team (or equivalent). Self-serve analytics tools went a step further by offering safe, governed datasets built by Data teams which let non-technical users dig into data without having to worry about how it joins together, how metrics like “revenue” are defined, and so on.

As the size and complexity of their relational database workload grows, every company eventually goes through the process of off-loading work on a read replica. It comes with lots of benefits, but at a cost of increased complexity. This article is about how we dealt with that, a lot of learnings, and some useful techniques. incident.io is an incident management product relied on by thousands of customers to be the thing that supports them through anything from a minor blip to a full outage.

Our On-call product has a lot of great features: configuring escalation paths, viewing rotas and schedules, requesting cover, etc. However, when framing its reliability, we reduce it down to two critical pieces of functionality: It’s not that we’re happy if only these parts are working, but they are the most important parts. In this post, I'll go into more detail on how we think about their reliability.