When Ticketmaster started their Kubernetes migration, they had to address a huge problem: whenever ticket sales opened for a popular event, as many as 150 million visitors flooded their website, effectively causing distributed denial of service (DDoS) attacks. With new events happening every 20 minutes and $7.6 billion in revenue at stake, outages could mean hundreds of thousands in lost sales.
It's the middle of the night when your phone goes off. You rub your eyes and unlock the screen to see a SEV 1 alert from your incident management tool. The application is down, multiple cloud server instances are offline, and the remaining instances are being overwhelmed by the sudden increase in demand. You jump out of bed and start trying to troubleshoot. You log into your cloud provider and try to provision systems manually, only to find out you can't.
You can subscribe to Break Things on Purpose wherever you get your podcasts. In this episode of the Break Things on Purpose podcast, we speak with Sam Rossoff, Principal Software Engineer at Gremlin.
For many years, system uptime was the primary measure of reliability, especially when the most popular method of running software was on bare metal, on-premises servers. If a server was shutdown, rebooted, or otherwise became unavailable, downtime was expected until a system administrator could manually restart it.
Time for a bit of a review! Join Jason as he looks back on some previous guests who have shared some opinions that range from the idiosyncratic to down right unpopular. Pulling from a handful of “Breaking Things” interviews, Jason covers death to VPNs, to the validity of “AI Ops,” check out the litany!
