One of the most common reasons for system failures is changes to the underlying infrastructure. Amazon CloudTrail does a great job of recording when actions are taken but a lot of organizations don’t take advantage of it. FireHydrant now includes this data, giving you visibility into changes to your infrastructure while you’re investigating an incident.
In the past I’ve written about how to use informers in Kubernetes for particular resources, but what if you need to be able to receive events for any Kubernetes resource dynamically? Well, there’s a client-go package for that too. At FireHydrant, we recently updated our Kubernetes integration to watch changes for any resource you configure and I wanted to write down how we made it at a high level.
Ever go to a status page and it says everything is operational when it definitely isn’t? You refresh maddeningly thinking it might be you. You ponder if the bill for the internet has been paid. Then, as a last resort, you check Twitter only to discover hundreds of people are experiencing the same problem. This is common, and because of it, we’re happy to release out integration with Statuspage.io!
Incidents happen all the time because of bad code deploys. You write some code that passes code review, it then is automatically shipped to production after a test suite passes, and BAM, an outage happens. This fairly common occurrence has ways to prevent it entirely. Using some simple ideas we can defend ourselves from the hidden mistakes that code reviews and chaos engineering sometimes won’t catch.
We’re launching a new feature today that allows anyone in your organization to kick off your incident response process with an appropriate severity level attached from Slack. Often people are afraid to open an incident or even share that they’re aware of something going wrong with your applications. When everything is important, nothing is important; users frequently overestimate the impact of an incident and assign an inappropriately high severity level.
So you’ve signed up to give a tech talk, awesome! You’re a subject matter expert in something and want to share you knowledge, that’s what helps make a community awesome. You’re going to be speaking in front of a room of people that you don’t know in a place you’ve likely never been, talking about something you confidently know. Sounds easy, right?
The conference started off with a heady fulfilling talk from John Allspaw about the impact of human performance in automatable systems. Wrapping the crux of his argument around the 3 Mile Island nuclear disaster, he drew parallels to how automation in software suffers from those same issues and how the automators should attack these problems.
We recently removed webpacker from our Rails 5 application. While it was a great way to bootstrap a new Rails app with a significant amount of Javascript we encountered some serious performance problems that we felt really impacted our productivity. We build our application images after every git push and we’d like to be able to deploy them quickly.
Imagine you are driving a car on a freeway. Your speedometer is telling you you’re going 62 mph. But you “gotta go fast”. Faster than then 65 mph speed limit. So you go for it: first 68mph, then 75mph, then 80mph. Then you pass a police officer hiding in a speed trap. To your dismay, they pull you over and give you a ticket. All is not lost: there is a silver lining here.
If you’re running a production application, you need metrics. There are great products out there that allow you to gain visibility into how your application is performing, give some nice graphs, and charge you for it. In the Rails community, this is commonly achieved by using NewRelic and Skylight. But for some of us, we achieve visibility by using Prometheus and Grafana that we build and host ourselves.
