Operations | Monitoring | ITSM | DevOps | Cloud

On July 19, 2024, a major IT disaster struck when a CrowdStrike update caused widespread chaos. CrowdStrike, a cybersecurity firm, inadvertently pushed a faulty “sensor configuration update” for its Falcon Sensor software. This update caused 8.5 million Windows devices to crash. The impact was severe, affecting airlines, banking systems, and healthcare networks, and the recovery process was laborious, requiring manual intervention for impacted devices.

Historically, data teams have not been closely involved in the incident management process (at least, not in the traditional “get woken up at 2AM by a SEV0” sense). But with a growing involvement of data (and therefore data teams) in core business processes, decision making, and user-facing products, data-related incidents are increasingly common, and more important than ever.

Reliability testing and Chaos Engineering are essential for finding reliability risks and improving the resiliency of systems. Gremlin makes it easy to do so, but not every engineer needs access to the same experiments, systems, or services. That’s why we released customizable role-based access controls (RBAC), letting Gremlin customers control which actions your users can perform in Gremlin.