Prometheus Alert Rules Validator

Prometheus evaluates alert rules at regular intervals (typically 1m). When a PromQL expression returns results, the alert becomes 'pending'. After the 'for' duration passes, it fires and is sent to Alertmanager, which routes notifications to PagerDuty, Slack, email, etc. Alert rules are defined in YAML files and loaded by Prometheus via the rule_files config.

The 'for' clause specifies how long a condition must be true before the alert fires. Without it, alerts fire immediately (and may flap). Example: 'for: 5m' means the CPU must stay above 80% for 5 continuous minutes. Use it to avoid alerting on brief spikes. Set it to 0s or omit it only for critical alerts where you want instant notification.

CPU usage: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode='idle'}[5m])) * 100) > 80. Memory: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10. Disk: (node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes * 100 > 85. Use rate() for counters and instant value for gauges.

Use promtool: promtool check rules /path/to/alerts.yaml. For unit testing, write test files with promtool test rules. In Grafana, use the Alerting > Alert rules page to preview firing alerts. You can also query the Prometheus /api/v1/rules endpoint to see current rule status and use /graph to test PromQL expressions interactively.

Required labels: severity (critical/warning/info) for Alertmanager routing, team or service for ownership. Required annotations: summary (one-line description, use {{ $labels.instance }} for dynamic data), description (detailed context including {{ $value }} for the metric value, runbook_url for links to runbooks). These appear in your notification messages.

Set appropriate 'for' durations (5-15 minutes for non-critical). Use inhibition rules in Alertmanager (e.g. suppress warning when critical fires). Group related alerts. Set proper severity levels and route only critical to PagerDuty. Use dead man's switch alerts for silence detection. Review and tune thresholds regularly — start conservative and tighten based on incident history.