Never-firing alerts: What they are and how to deal with them
Alerting is one of the main reasons for having a monitoring system. It is always better to be notified about an issue before an unhappy user or customer gets to you. For this, engineers build systems that would check for certain conditions all the time, day and night. And when the system detects an anomaly - it raises an alert. Monitoring could break, so engineers make it reliable. Monitoring could get overwhelmed, so engineers make it scalable. But what if monitoring was just poorly instructed?