As an IT manager you would have often heard from your line manager or user ask “Let’s drill down to find the root cause.”? As dreaded a question as it may seem, it is really the most important answer to understand IT outages. IT infrastructure availability is highly dependent on isolating problems, so the deciding variable in a problem can be fixed without putting the entire system at a halt. This is where RCA can be of tremendous help.
Our instrumentation uses built-in extension mechanisms where possible, such as Django’s database instrumentation. But often libraries have no such mechanisms, so we resort to wrapping third party libraries’ functions with our own decorators. For example, we instrument jinja2 ’s Template.render() function with a decorator to measure template rendering time. We value the correctness of our instrumentation a lot so that we do not affect our users’ applications.
Scaling sort of sneaks up on you, doesn’t it? One day, you’re carefree, the next you start to notice something is off… Maybe it’s the crashing, or the frequent dips in performance. Could it be the new hire? It’s not DNS. Is it DNS? Scaling is a natural part of the business process, and your infrastructure will start to change completely as your userbase doubles and triples.
IT applications are vital for today’s digital economy and for the business to succeed, these applications must be highly available and performing well. Application performance degradations can occur for several reasons. There may be code-level issues, database slowness, or network bandwidth constraints. IT applications run on servers and if the server is not sized correctly or is under-performing, application performance will degrade as well.
On Star Trek, there’s an incredibly useful device called the universal translator. As you’d expect, it allows everyone to understand each other. For example, if Captain Jean Luc Picard bumped into a race of aliens that bore a striking resemblance to Commander Riker’s beard, then they could set a date for some Earl Grey tea (hot) thanks to the universal translator. Without it, there might be grave misunderstandings and the firing of photon torpedoes.
For IT professionals, an enduring outage is one of the worst things that can happen. There goes your credibility, and here comes the executive team enraged and impatient. Now of course, given the distributed, multi-sourced nature of IT infrastructure, some outages are simply not preventable. The details are still emerging from the June 9th outage on IBM Cloud. ITPro Today interviewed Forrester analyst Dave Bartoletti for some mitigation advice.