The task of monitoring and managing an entire network, including all the servers and applications that run on it, is by no means easy. With so many components of varying complexity, the volume of performance data coming at you can be overwhelming. This information overload increases the chances of missing data that could help discover performance inefficiencies.
Because of its popularity and widespread adoption, Kubernetes has become the industry’s de facto for deploying a containerized app. Google Kubernetes Engine (GKE) is Google Cloud Products’ (GCP) managed Kubernetes service. It provides out-of-the-box features such as auto-scaling nodes, high-availability clusters, and automatic upgrades of masters and nodes. In addition, it offers the most convenient cluster setup workflow and the best overall developer experience.
As an IT manager you would have often heard from your line manager or user ask “Let’s drill down to find the root cause.”? As dreaded a question as it may seem, it is really the most important answer to understand IT outages. IT infrastructure availability is highly dependent on isolating problems, so the deciding variable in a problem can be fixed without putting the entire system at a halt. This is where RCA can be of tremendous help.
Our instrumentation uses built-in extension mechanisms where possible, such as Django’s database instrumentation. But often libraries have no such mechanisms, so we resort to wrapping third party libraries’ functions with our own decorators. For example, we instrument jinja2 ’s Template.render() function with a decorator to measure template rendering time. We value the correctness of our instrumentation a lot so that we do not affect our users’ applications.
Scaling sort of sneaks up on you, doesn’t it? One day, you’re carefree, the next you start to notice something is off… Maybe it’s the crashing, or the frequent dips in performance. Could it be the new hire? It’s not DNS. Is it DNS? Scaling is a natural part of the business process, and your infrastructure will start to change completely as your userbase doubles and triples.
