We test our own critical dependencies

Even if you know a dependency is critical, you still should test it. Otherwise, who knows what will happen if it goes down?

Full transcript:
A favorite anecdote that I have about some of these tests happened with one of our critical dependencies, which we absolutely run tests against as well, even when we know that things are gonna fail. We had a dependency on a critical AWS service. We know when that service is unreachable, when the network is unreliable, we are gonna see problems, we're gonna see failures. But when we ran tests against this dependency, we actually didn't see the failures we expected. We assumed, "Hey, we're gonna fail fast.

We'll see a ton of errors. Those errors will feed into our alerts, and we're gonna be notified right away when that happens." It actually turns out a ton of downstream tasks became really slow when we made the network unreliable there and the latency around all of our processing shot way up.   And as it turns out, the reason for that latency had to do with a default timeout configuration that we had on this dependency, which was way higher than what we considered reasonable. And this test pointed us right to that problem to say, "Hey, you need to lower your timeouts in order to fail faster the way that you expect."