|
By Navo Das
Historically, data teams have not been closely involved in the incident management process (at least, not in the traditional “get woken up at 2AM by a SEV0” sense). But with a growing involvement of data (and therefore data teams) in core business processes, decision making, and user-facing products, data-related incidents are increasingly common, and more important than ever.
|
By Stephen Whitworth
Today marks a particularly challenging day for incident responders across the globe. As many of you may have noticed, a recent update from CrowdStrike has triggered widespread disruptions, causing chaos in various sectors. The ripple effects have been far-reaching and severe: While the technical specifics of the issue might not be the focus here—and indeed, there are experts better suited to dissect the cause—what's crucial is understanding the impact on those who manage such crises.
|
By Henry Course
Our On-call product has been in the wild for a few months now, and in this post I want to talk about building a time-sensitive system and what we did to handle some of the challenges. I’ll cover what our scheduler is responsible for, the basics of working with time, and talk a bit about how we tested our system.
|
By Leo Sjöberg
Arguably the most important part of an on-call product is knowing that you will be notified when things break, wherever you are. When it comes to SMS and phone call notifications, we have to leave the familiar realm of the internet and JSON responses, and deal with systems that provide limited observability and insight into what’s gone wrong.
|
By Rory Bain
A significant part of being on-call is the ability to respond to pages and handle escalations on the go. In the early stages of developing incident.io On-call, we considered whether a Minimum Viable Product (MVP) could rely solely on SMS and phone calls. However, we quickly realized that a fully featured mobile app was going to be essential to the on-call experience. This led us to the question: how should we build this mobile app?
|
By incident.io
We’re excited and honored (and might even be blushing a little) to share our Summer 2024 accolades from G2, including being ranked #1 in G2’s Relationship Index! There are several factors that go into determining this ranking, including: While all of these awards are special to us, Best Relationship means a lot because, well, our customers mean a lot.
|
By Henry Course
March 5th was a big day for incident.io as we released our on-call product to the world. Nine months of listening to our customers, coding, fixing, testing, and polishing came together for our biggest product launch to date. Releasing On-call was a huge milestone and represented the next step in our journey as a company.
|
By Pip Taylor
At incident.io we use infrastructure as code for configuring everything we can, and we feel that there’s no reason we should exclude our own product from that. As well as configuring things like Google Cloud Platform, Sentry and Spacelift via our infrastructure repo, we also configure incident.io. On your first day as an engineer here, the first PR that you make is to our infrastructure repo.
|
By Chris Evans
Whenever a new customer joins incident.io, we make notes on what made them chose to buy our product and, if we were in a competitive process, why they chose us over other providers they were evaluating. It’s a lot of messy data and raw notes, but contained within is a veritable treasure trove of customer feedback. Summarising large amounts of data? Sounds like the perfect job for an LLM.
|
By Lisa Karlin Curtis
We approached building a Terraform integration with a sense of trepidation. One of the things that motivates us is building features we think people are going to love using, and Terraform integrations are often not that. Terraform integrations have a number of common pitfalls. Building resources by hand is tedious, and requires deep understanding of their specification. Importing and managing existing resources is also often painful.
|
By Incident.io
In this episode, Norberto (VP of Engineering) and Lawrence (Product Engineer) delve into the recent CrowdStrike incident that began on July 19th. Rather than focus on technical specifics, they provide a thoughtful exploration of key aspects that matter to us at incident.io, such as effective communication, overall response strategies, and proactive problem-solving during crises.
|
By Incident.io
Gone are the days when incidents were manual to resolve, invisible to customers, and overall viewed with a negative lens. This is part two of the virtual event series as we dive into our fresh take on what incidents should look like, The Incident Way, and hear from customer stories putting these principles into practice.
|
By Incident.io
Scaling incident management processes can present massive challenges for an organization as large and complex as Netflix. And for Netflix, whose brand has become synonymous with dependability, there’s a lot at stake. Since its introduction to a specific set of Netflix teams, incident.io has been organically adopted far and wide across Netflix Engineering, highlighting just how indispensable and impactful the tool has become.
|
By Incident.io
During a recent episode of The Debrief, we spoke with Jeff Forde, Architect on the Platform Engineering team at Collectors, about building an incident management program at various stages of growth. In that episode, we called it growth from zero to one, one to two, and two to three. But what happens once you’ve scaled beyond three and answers to question you may have become that much harder to find.
|
By Incident.io
In this event uncover the common pains associated with legacy incident management norms and why they don’t meet the mark for modern needs.
|
By Incident.io
This week, we have a really fun conversation lined up. For this episode, we chatted with Toby Jackson, Global SRE Team Lead at Future, about why it’s a bad idea to take a cookie-cutter approach to incident management or, put another way, why it’s not a good idea to treat all incidents alike. In our conversation, we discuss what’s wrong with this approach, some situations where this might actually make sense, how psychological safety factors into this conversation, and a whole lot more.
|
By Incident.io
In this clip, Pete explains why we've taken the approach of "exoskeletons, not robots" when building with AI. It’s fair to say that AI is here to stay. So, as companies grapple with this reality, they’re putting their best foot forward to build AI features that really make a difference for their customers. But should you be building these features if there’s no obvious fit in your product? And even if there is, are you making sure to stay true to your product principles?
|
By Incident.io
It’s fair to say that AI is here to stay. So, as companies grapple with this reality, they’re putting their best foot forward to build AI features that really make a difference for their customers. But should you be building these features if there’s no obvious fit in your product? And even if there is, are you making sure to stay true to your product principles? The reality is that deciding to build AI into your product isn’t a decision you make on a whim.
|
By Incident.io
It’s no secret that teamwork is one of those things that, when done right, can make a world of a difference. So sometimes, when responding to a particularly complicated incident, it can be best to bring a team together to figure out what’s going on and work towards a fix. But it’s not enough to just jam a bunch of folks into a room and hope for the best. You need a framework in place to ensure that everyone stays focused, diagnoses the issue and resolves it as quickly as possible.
- July 2024 (10)
- June 2024 (8)
- May 2024 (13)
- April 2024 (18)
- March 2024 (15)
- February 2024 (18)
- January 2024 (9)
- December 2023 (10)
- November 2023 (5)
- October 2023 (10)
- September 2023 (16)
- August 2023 (3)
- July 2023 (8)
- June 2023 (6)
- May 2023 (4)
- April 2023 (8)
- March 2023 (2)
- February 2023 (5)
- January 2023 (5)
- December 2022 (3)
- November 2022 (4)
- October 2022 (10)
- September 2022 (7)
- August 2022 (11)
- July 2022 (6)
- June 2022 (3)
- May 2022 (2)
- April 2022 (3)
- March 2022 (6)
- February 2022 (7)
- January 2022 (2)
- December 2021 (5)
- November 2021 (5)
- October 2021 (2)
Create, manage and resolve incidents directly in Slack. Leave the admin and reporting to us.
Improving your incident response, visibility, and ability to learn:
- Less faffing, more fixing: We take care of the admin during incidents, so you can save your brainpower for the decisions that matter.
- Divide and conquer: We make sure everyone’s role is clear, track who’s working on what, and help you escalate if you need extra help.
- Get up to speed, at speed: Get everyone on the same page from the moment they join the incident, and help stakeholders stay in the loop.
- Timelines, in no time: Constructing an incident timeline for review is important, but time consuming. We’ll build one for you in real-time, and keep it constantly up to date.
- Data and insights you can trust: You’ve already paid for your incidents. By surfacing the data you need to make decisions, we help you get your money’s worth.
Incident response for your whole organisation.