If there’s one essential thing we’ve learned from being in the business of digital operations for more than 13 years, it’s that every business has a unique approach to building resilience with its bespoke tech stacks and processes. Many PagerDuty customers around the world are starting to provide direct access to their on-call teams with Live Call Routing (LCR).
Regardless of your organization’s size, industry, or security measures, you will inevitably face IT incidents. But what do you do if an incident affects a critical system and your on-call responders can’t resolve it? Does your team have a set of clearly outlined next steps they should take to handle the issue? Answering these questions can be complicated, even more so for large organizations that rely on cloud-based services to fuel their IT environment.
Enterprise Alert® is the leading enterprise-class software for automated, targeted, and traceable alerting. But what does “escalation” mean in this case? In the course of my work, be it with new customers, existing customers or even prospects, I repeatedly find that the term escalation is often defined very differently. Therefore, I would like to clarify in the context of this blog what we mean by escalation.
We recently sat down with Luis Mineiro, Head of Site Reliability Engineering at Zalando, during a session on our 99PercentVisible: DevOps Reliability tech talks. "Are we on the same page? Let’s fix that" discusses the practice of alerting and how it can go from a routine task to team burn out if not done correctly. We’ll recap the conversation and do a deep dive on what is known as “The Christmas Tree Effect” and how Adaptive Paging is the correct solution for it.
Reacting to alerts can be a pain, however, there are ways to be proactive and decrease frustration concerning IT Alerting. Developing an alerting strategy saves IT Operations and Development teams time, money, and eliminates notifications from low priority alerts. Keep reading for more information on routing and escalation chains, fielding alerts, and how to communicate an alerting strategy to management.
The idea behind an escalation matrix is simple: the situation requires greater authority to resolve. Authority can take many forms, including experience with a particular toolset or simply the proper permissions to flip the right switches. Therefore, escalation must involve putting the proper information into the right person’s hands (well, device).
Uptime.com includes several advanced check options to provide the flexibility organizations need in creating a response plan to downtime. Maintenance and planned downtime for patches and updates don’t typically create severe downtime events. With escalations, teams have an automated alert system that contacts designated senior-level personnel with relevant technical data.
Let’s set the scene – an IT professional just received a critical alert, notifying him of an urgent matter. Unfortunately, due to a distracting social event, this professional is completely unaware of the notification. Even worse, he receives incident alerts via email, a communication method which doesn’t have the ability to forward the alert to another person if he doesn’t respond within a short time frame. Maybe he should stop attending social events.
