Prometheus

The open-source monitoring standard for cloud-native systems. Prometheus scrapes /metrics endpoints, stores time-series data locally, and evaluates alert rules via Alertmanager. Paired with Grafana, it forms the backbone of the open-source observability stack.

What is Prometheus?

Prometheus is an open-source monitoring system and time-series database originally built at SoundCloud and now a CNCF graduated project. It uses a pull model: Prometheus periodically makes HTTP requests to a /metrics endpoint on each monitored service, parsing the Prometheus text exposition format into time-series data stored in its local TSDB.

Prometheus is the de facto standard for Kubernetes observability. The kube-state-metrics and node-exporter exporters instrument your cluster, and the Prometheus Operator lets you manage Prometheus configuration via Kubernetes CRDs. In non-Kubernetes environments, client libraries for Go, Python, Java, Node.js, and Ruby let you instrument any service in minutes.

Metric Types

prometheus.yml Configuration

# prometheus.yml
global:
  scrape_interval: 15s      # How often to scrape targets
  evaluation_interval: 15s  # How often to evaluate rules
  scrape_timeout: 10s

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

# Rule files (recording rules + alert rules)
rule_files:
  - 'rules/*.yml'

# Scrape targets
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node-exporter
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: app
    metrics_path: /metrics
    static_configs:
      - targets: ['app:3000']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

PromQL Examples

# PromQL Examples

# HTTP request rate (requests per second, 5m window)
rate(http_requests_total[5m])

# Error rate as a percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100

# 95th percentile request latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# CPU usage by instance
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
/ node_memory_MemTotal_bytes * 100

# Number of pods in a Kubernetes namespace
count(kube_pod_info{namespace="production"})

Alert Rules

# rules/alerts.yml
groups:
  - name: application
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
          > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency above 1 second"

Installation

# Docker
docker run -p 9090:9090 \
  -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

# Kubernetes (kube-prometheus-stack — includes Grafana + Alertmanager)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

# Binary (Linux)
wget https://github.com/prometheus/prometheus/releases/latest/download/prometheus-*.linux-amd64.tar.gz
tar xvf prometheus-*.linux-amd64.tar.gz
./prometheus --config.file=prometheus.yml