Operations | Monitoring | ITSM | DevOps | Cloud

On Call

The engineering on-call experience: misconceptions, lessons learned, and how to prepare

The on-call experience is sometimes a dreaded one for software engineers. Those late-night alerts and frantic Slack messages, after all, don’t exactly sound pleasant. But what’s an on-call shift really like? Is that perception of constant fire-fighting and 3 AM wake-up calls actually realistic? Michael Mandrus and Owen Smallwood, both senior software engineers here at Grafana Labs, wanted to set the record straight.

How Squadcast's Snooze Incidents Promotes Focussed On Call Shifts

Dealing with a flood of incidents, each with varying degrees of urgency, can be a daily struggle for Incident Response teams. Suppose a low-priority alert pings while you're tackling a critical incident. This pulls your focus away from the urgent issue. This constant alert bombardment can: How do engineers ensure that high-severity issues take precedence? Don't they want to avoid being bothered or bombarded with notifications while addressing critical matters? They sure do.

Improving your on-call schedule with runbooks

Incidents are a stressful time for your team: your service isn't working the way you expect and your customers/stakeholders want to know what's going on. The last thing you want to do is let your team improvise everything when it comes to responding to incidents. Google's own SRE book has great overall tips for incident management, part of which involves "develop(ing) and document(ing) your incident management procedures in advance", which this article dives into.

We've launched incident.io On-call

It’s 3am. You wake up to a blaring alarm, the sound burned into your soul from countless sleepless nights. You reach for your phone, ‘press 4 to acknowledge’ and bleary eyed, you open your laptop, grab a coffee and get to work. The next hour is a whirlwind—bringing services back online, keeping colleagues in the loop, maintaining a list of action items, updating a status page that will be seen by millions of customers. Potentially for the fifth time this month.

The Debrief: Introducing incident.io On-call

This is on-call as it should be. The secret's out. The world can finally know. incident.io On-call is here. Naturally, a lot of you may be wondering: why and why now. So to help answer those questions, we sat down with Chris and Pete, two of our co-founders here at incident.io to get a bit of background on this project: This episode will not only get you excited about this huge week, it'll get you pumped for what's ahead for on-call.

Finally: alerting and on-call scheduling for how you actually work

TL;DR You deserve a better alerting and on-call tool. So we built Signals. In our early days, we often used the tagline, “You just got paged. Now what?” It encapsulated how FireHydrant solved for all of the messy bits that come after your alert is fired, from incident declaration all the way through to retrospective. At the time, we saw alerting and on-call scheduling as a solved problem.

Best Practices For Building A Resilient On-Call Framework

Whether a business is small scale, medium-sized, or a large enterprise, downtime issues can affect any organization as no business is exempt from experiencing downtime. However, the swifter the acknowledgment of an issue, the quicker the response, resulting in a reduced impact on business. An effective On-Call framework not only aids in prompt issue resolution but also plays a vital role in minimizing the overall downtime impact on business operations.

How to set up on-call compensation

Once you set up an on-call team, the next step is to decide their compensation. There might be several questions in your mind right now: "How do we fairly value on-call time?" "Is it a flat rate or hourly?" and a few others. So we are here to help you set up an on-call compensation system because we know compensating people fairly lays the foundation of a healthy business. Are you still stuck on setting up an on-call team? Read this guide first: 7 steps to set up an on-call team.

Use ilert mobile app to take someone else's on-call shift

Use the ilert mobile app to receive push notifications about alerts and gain access to essential incident management features so that you can take immediate action from anywhere. The app also allows you to quickly take over your colleague's on-call shift while on the go. Check out the video to learn more about this feature.