If you don't test in production, you're missing risks

Testing in production can be scary, but it’s necessary to improve reliability. Check out this clip from when Gremlin Co-founder and CEO Kolton Andrus sat down with Stephen Townshend on the Slight Reliability podcast!

FULL TRANSCRIPT:

KOLTON -  
When it comes to production, the way I like to explain is like there's this pie chart of everything that can go wrong. And only like half or two thirds of it lives in staging. You're never gonna find a set of failures until you test in production. Why? Production has a plethora of customers and customer behavior, and customers always do something unexpected, so you may require a customer to trigger that unexpected behavior.

We have security groups, we have load balancers, we have DNS. We have multiple layers of racks. We're doing deployments. Every service behind the scenes is changing regularly. Some of those things you will just never catch in staging. And so should we test in staging first? Yes. Go find everything in staging or in dev that you can. Fix all of those.

That's just smart engineering, but you gotta go to production in the end. The other piece I hear a lot is like, we're not mature enough to test in production.

STEPHEN -
I was just about to ask about that.

KOLTON -
If you're afraid to get started, you're never gonna get better. I have a New Year's resolution and what I want to do is get in better shape, but what I've decided is I need to lose 20 pounds before I start going to the gym. That's a bit like what we're saying here, and the answer is it's a Chaos Engineering practice, and you need to go out and you need to start doing it because that's how you're gonna gain maturity.

That's how you're gonna gain comfort. The thing that really scares you is you don't understand your system as well as you think you do. And you're gonna be surprised. And guess what? That's okay. Everybody's surprised. Nobody understands everything perfectly. But until you go out and start poking at it and asking questions and analyzing the result, you're never gonna have a better understanding, and therefore, you're never gonna be more comfortable.

And you're never gonna get anywhere.