Ask most SREs how many incidents they’d have to respond to in a perfect world, and their answer would probably be “zero.” After all, making software and infrastructure so reliable that incidents never occur is the dream that SREs are theoretically chasing. Reducing actual incidents by as much as possible is a noble goal. However, it’s important to recognize that incidents aren’t an SRE’s number one enemy.

SRE requires teamwork and planning. Be like Aiven, get it right.

Wondering about development velocity? We explain what development velocity is, how to measure it, and how to balance the need for fast development and reliable products.

Chaos testing is a way to test the integrity of a system. Its purpose is to simulate failures that could crash a production system in a controlled environment. This helps to identify failures before they cause unplanned downtime that disrupts the user experience. Unlike standard testing, which tests a system response against a predefined result, chaos testing does not have a predefined result. Rather, the entire purpose of the experiment is to find out new information about the system.

Digital business is an imperative for 21st-century companies. Increasingly, organizations are directing investments toward technologies that deliver outcomes fast and enable more resilient digital business models. In this landscape, incidents such as software bugs, power outages, or downed networks have major consequences that affect both revenue and customer loyalty.

Site reliability engineering continues to gain traction in software development and IT. SRE is at the crossroads of software development and IT operations. In Ben Treynor’s words, SRE is “what happens when you ask a software engineer to design an operations function.” Site reliability engineering is a way for developers to actively build services and functions to improve the resilience of people, processes and technical systems.

Toil is a term coined by Google which describes the repetitive and tedious tasks associated with running a production service. Toil tends to be manual and devoid of any long-term value. Toil is not just ‘work I do not like to do’. Each time an engineer engages with a production system, it represents time devoted to toil. These types of tasks get worse as your service grows even more extensive. Site Reliability Engineers (SRE) should spend less time on toil.

Distributed caching is an important aspect of cloud based applications, be it for on-premises, public or hybrid cloud environments. It facilitates incremental scaling, allowing the cache to grow and incorporate the data growth. In this blog we will explore distributed caching on cloud and why it is useful for environments with high data volume and load.

“It’s too expensive!” “Do we really need another tool?” “Our APM works just fine.” With strapped tech budgets and an abundance of tooling, it can be hard to justify a new expense—or something new for engineers to learn. Especially when they feel their current tool does the job adequately. But, does it?

Summer means things are heating up. And things are definitely heating up at Blameless! We’ve been hard at work delivering new features and capabilities to our customers, so today I wanted to share a quick summary of all the latest. Here are 4 exciting product updates that enhance the way teams manage incidents and deliver reliable products to their customers.

As systems continue to become more complex, reliability is becoming an increasingly important requirement. Organizations are quickly realizing that making reliability a critical part of their service means that other organizations will be less likely to cut costs on them. As a result of this, the field of service reliability engineering (SRE) has grown rapidly over the past few years.

SLOs or Service Level Objectives are the foundation of Site Reliability Engineering (SRE). To correctly understand SLOs, the first step is to understand Service Level Indicators or SLIs. SLIs are metrics that measure the vitals of the service. These vitals are chosen based on two conditions. First, they are the features that the user is primarily concerned about. Second, they allow the engineering team to get an overview of the system’s health.

By investing in a CI/CD pipeline, it’s entirely possible to automate a large part of the software development life cycle – letting businesses deliver high-quality, high-efficiency outputs with a faster time to market. But there are multiple elements to the CI/CD process, including the all-seeing eye that is continuous validation. So what exactly is continuous validation, and why should software developers bother to engage with it?

Continuous documentation is the process of creating and maintaining code documentation incrementally throughout a project in a way that seamlessly incorporates it into the development workflow. It is a key part of improving reliability within an organization. It’s not just new features that need to be documented – anything useful from bug fixes, to how to get started using the code should be documented. It should also be updated frequently to ensure that it stays relevant.

There is a distinctive gap opening up between the top engineers and the rest. The elite engineers represent the top few percent of engineering teams and are making incredible gains year on year in velocity, reliability, and human compatibility, whilst the bottom 50% are losing ground. The loss has nothing to do with engineering ability.

At CircleCI, CI has a second meaning: Continuous Improvement. We continuously seek out feedback not only to improve our code but to improve our processes and get better at our jobs along the way. This Continuous Improvement starts with one important company value: a blameless culture. Our blameless culture extends into every part of how we operate.

Wondering about retrospective templates? We give a complete explanation of what a retrospective should include and best practices for using one.

OKR stands for Objectives and Key Results (OKRs). They are essential frameworks for establishing and monitoring goals and outcomes. They also facilitate discussions regarding the alignment of an employee’s job with the company’s objectives. Many companies such as Google use OKRs to improve engineering team culture and productivity. OKRs require a strong, open, and creative workplace culture to take root. OKRs offer you focus, alignment, commitment, and goal tracking.

No buzzwords were harmed in the making of this post Let’s take one of the most overloaded terms, DevOps, and mix it with the haziest of topics, security. What do you get, apart from confusion? SecDevOps. Or maybe it’s DevSecOps. If you’re not sure what either means, you’re not alone. Even the industry at large can’t decide what they should call it. And so they - we - came up with a new term altogether.

We live in an age where the internet and digital data drive modern day markets, which results in huge amounts of data being generated and consumed. Hence, it has become very important for online platforms to manage this traffic and serve their customers more efficiently. In this blog we will explore the Amazon Route 53 service and see how it addresses domain name system routing and health check problems.

Curious about incidents vs. problems? We explain the differences and how to handle each one. ‍

Distributed systems are responsible for many different tasks and processes that need to be monitored and managed. In this article, we will explore what you should monitor in a distributed system, including network communication, resources, and performance.

System reliability is the probability that a system performs as it is expected to under a set of specified conditions throughout a specified period. Organizations use reliability engineering to help to make products more reliable in a cost-effective way. The key objectives of reliability engineering are to reduce the frequency of failures, identify the causes of failures and correct them, figure out ways of coping with failures when they do occur, and estimate the likely reliability of new designs.

For the past four years, Catchpoint and various partners have been running a yearly SRE Survey. This year, Blameless is excited to partner with Catchpoint for the fifth annual survey. We want to hear from you if you are in a DevOps or SRE role or even if you work on reliability with some other title or role. There are tremendous, valuable learnings when we listen closely to practitioners.

For the past four years, Catchpoint and various partners have been running a yearly SRE Survey. This year, Blameless is excited to partner with Catchpoint for the fifth annual survey. We want to hear from you if you are in a DevOps or SRE role or even if you work on reliability with some other title or role. There are tremendous, valuable learnings when we listen closely to practitioners.

Curious about the incident priority matrix? We discuss how to determine the impact and urgency of an incident, and how to create a matrix that helps prioritize incidents.

Storage systems are an integral part of IT infrastructure. Given that modern markets are highly competitive and demanding, businesses strive for 24/7 availability. This in turn sets higher expectations for storage systems to be operational all the time. But just like other IT components, even storage systems are prone to incidents. Hence, it is important to have an efficient communication process, to manage alerts during system failures/disasters.

We discuss what software engineers and site reliability engineering are and explain their differences and their importance in the software development process.

As infrastructure engineers, there’s so much you can learn from studying past incidents. Luckily, Blameless Reliability Insights helps you find patterns that better equip you to deal with incidents to come. If you’ve never used it before and you’re curious what it looks like, you can watch a video demo here! These statistical insights give you the power to learn everything you can when something goes wrong. ‍