Alert Fatigue Analyzer

Alert fatigue occurs when on-call engineers are overwhelmed by high volumes of alerts — many of which are noise, duplicates, or self-resolving. The danger: engineers start ignoring or bulk-dismissing alerts, which means real critical incidents get missed. Studies show that teams with >50% noisy alerts have significantly higher MTTR because genuine signals are buried. Alert fatigue also causes burnout and increased on-call attrition.

A healthy alert environment has an actionable rate of 75%+, meaning most alerts require human action. Noise ratio below 25% is the target. Teams with noise ratios above 50% should immediately audit their alerting rules — start with flapping alerts (self-resolving within 5 minutes) and alerts that have been ACKd without action for 30+ days.

Flapping alerts fire and resolve within minutes (typically <5 minutes) repeatedly, often without meaningful human intervention. Common causes: thresholds set too close to normal operating levels, missing hysteresis (the threshold for alerting vs. clearing should differ), or genuinely intermittent infrastructure issues. Fix by: adding minimum-duration requirements (alert only if condition persists 5+ minutes), adding hysteresis, or addressing the underlying instability.

A pragmatic fatigue score factors in: noise ratio (high noise = high score), acknowledgement time (slow ack = engineers are overwhelmed), resolution time relative to ack time (long gap = poor runbooks or unclear alerts), and flapping rate. A score above 70/100 indicates severe fatigue requiring immediate intervention. Below 30 is healthy.

Key strategies: (1) Alert on symptoms, not causes — 'user checkout failing' not 'high CPU on checkout-service-3'. (2) Set meaningful thresholds with hysteresis. (3) Group related alerts into single incidents (alert correlation). (4) Add minimum duration requirements. (5) Regular alert reviews — monthly audit to delete alerts unacted on in 30 days. (6) Ownership — every alert must have a clear owner who maintains it.