Operations | Monitoring | ITSM | DevOps | Cloud

It can be quite challenging for an SRE team to maintain the well-being of a large-scale Kubernetes based system with hundreds or thousands of services. In this blog post, Gigi Sayfan, author of “Mastering Kubernetes”, outlines the SRE challenge and how we can achieve the ultimate goal of automated SRE with Kubernetes operators.

Nowadays, a working digital infrastructure is the lifeblood of almost any organization. The impact of a major IT incident can go far beyond the IT department, affecting a company’s revenue or incur costs in other areas of the business caused by service disruption. Therefore, in addition to the technical response to a major incident from the IT department, business stakeholders need to be involved as well, so they can prepare the business response.

DevOps pipelines enable teams to implement continuous software development processes, often by using automation and collaboration tooling. The overall goal is to quickly release software products, updates, and fixes. To ensure a DevOps pipeline works well, teams add management and monitoring tooling to the pipeline. This includes incident alert management, which supports the team’s efforts in monitoring the security of various software and environment components.

The ongoing pandemic and resulting economic downturn have led to dramatically changing market conditions. As a consequence, technology teams have become increasingly concerned with the need to minimize their financial risk and reduce costs to mitigate the effects of abruptly pivoting to a fully remote working environment. For some, there has been a struggle to maintain business continuity—i.e., keeping the physical components of the business running when everyone is working from home.

Efficient team communication requires the proper set of tools and processes, ensuring that the right persons receive timely messages. This way, recipients are well-informed of a critical issue, while having time to address the incident. Unfortunately, a large helicopter services company relied on time-wasting procedures to communicate with stakeholders, resulting in delayed incident response and resolution.

With many companies utilizing remote-working during the COVID-19 pandemic, effective communication is more important than ever. Unfortunately, being in the middle of responding to a global pandemic will not prevent your organization from suffering from other business disruptions. Likely disruptions you may face are: Cyber/ phishing attacks – these attacks can cripple your regular communication methods such as email, or may exploit ineffective communications to extract illegal payments.

Every NOC engineer will tell you that the first thing they look for in an outage is “what changed?”. And they are right to look. While every organization is unique, Gartner reports that on average about 80% of IT incidents today are caused by changes in infrastructure and/or software.