Operations | Monitoring | ITSM | DevOps | Cloud

The task of monitoring and managing an entire network, including all the servers and applications that run on it, is by no means easy. With so many components of varying complexity, the volume of performance data coming at you can be overwhelming. This information overload increases the chances of missing data that could help discover performance inefficiencies.

As an IT manager you would have often heard from your line manager or user ask “Let’s drill down to find the root cause.”? As dreaded a question as it may seem, it is really the most important answer to understand IT outages. IT infrastructure availability is highly dependent on isolating problems, so the deciding variable in a problem can be fixed without putting the entire system at a halt. This is where RCA can be of tremendous help.

Our instrumentation uses built-in extension mechanisms where possible, such as Django’s database instrumentation. But often libraries have no such mechanisms, so we resort to wrapping third party libraries’ functions with our own decorators. For example, we instrument jinja2 ’s Template.render() function with a decorator to measure template rendering time. We value the correctness of our instrumentation a lot so that we do not affect our users’ applications.

Scaling sort of sneaks up on you, doesn’t it? One day, you’re carefree, the next you start to notice something is off… Maybe it’s the crashing, or the frequent dips in performance. Could it be the new hire? It’s not DNS. Is it DNS? Scaling is a natural part of the business process, and your infrastructure will start to change completely as your userbase doubles and triples.