Today marks a particularly challenging day for incident responders across the globe. As many of you may have noticed, a recent update from CrowdStrike has triggered widespread disruptions, causing chaos in various sectors. The ripple effects have been far-reaching and severe: While the technical specifics of the issue might not be the focus here—and indeed, there are experts better suited to dissect the cause—what's crucial is understanding the impact on those who manage such crises.

Our On-call product has been in the wild for a few months now, and in this post I want to talk about building a time-sensitive system and what we did to handle some of the challenges. I’ll cover what our scheduler is responsible for, the basics of working with time, and talk a bit about how we tested our system.

Arguably the most important part of an on-call product is knowing that you will be notified when things break, wherever you are. When it comes to SMS and phone call notifications, we have to leave the familiar realm of the internet and JSON responses, and deal with systems that provide limited observability and insight into what’s gone wrong.

A significant part of being on-call is the ability to respond to pages and handle escalations on the go. In the early stages of developing incident.io On-call, we considered whether a Minimum Viable Product (MVP) could rely solely on SMS and phone calls. However, we quickly realized that a fully featured mobile app was going to be essential to the on-call experience. This led us to the question: how should we build this mobile app?

We’re excited and honored (and might even be blushing a little) to share our Summer 2024 accolades from G2, including being ranked #1 in G2’s Relationship Index! There are several factors that go into determining this ranking, including: While all of these awards are special to us, Best Relationship means a lot because, well, our customers mean a lot.

March 5th was a big day for incident.io as we released our on-call product to the world. Nine months of listening to our customers, coding, fixing, testing, and polishing came together for our biggest product launch to date. Releasing On-call was a huge milestone and represented the next step in our journey as a company.

At incident.io we use infrastructure as code for configuring everything we can, and we feel that there’s no reason we should exclude our own product from that. As well as configuring things like Google Cloud Platform, Sentry and Spacelift via our infrastructure repo, we also configure incident.io. On your first day as an engineer here, the first PR that you make is to our infrastructure repo.

Whenever a new customer joins incident.io, we make notes on what made them chose to buy our product and, if we were in a competitive process, why they chose us over other providers they were evaluating. It’s a lot of messy data and raw notes, but contained within is a veritable treasure trove of customer feedback. Summarising large amounts of data? Sounds like the perfect job for an LLM.

We approached building a Terraform integration with a sense of trepidation. One of the things that motivates us is building features we think people are going to love using, and Terraform integrations are often not that. Terraform integrations have a number of common pitfalls. Building resources by hand is tedious, and requires deep understanding of their specification. Importing and managing existing resources is also often painful.

Clean, clear, and ready to be customized to suit your needs. Google Docs Having a dedicated incident post-mortem is just as important as having a robust incident response plan. The post-mortem is key to understanding exactly what went wrong, why it happened in the first place, and what you can do to avoid it in the future.