What is Grafana Tempo?

Grafana Tempo is a distributed tracing backend designed for high scale and low cost. Like Loki’s approach to logs, Tempo stores traces directly in object storage without indexing trace content. Instead, traces are looked up by Trace ID — making it extremely storage-efficient.

Tempo is the tracing component of the Grafana observability stack (Loki for logs, Mimir/Prometheus for metrics, Tempo for traces, Grafana for visualization).

Quick Start

# Docker Compose with full Grafana stack
cat > docker-compose.yml << 'EOF'
version: "3"
services:
  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/tmp/tempo
    ports:
      - "3200:3200"   # Tempo UI (via Grafana)
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/ds.yaml

volumes:
  tempo-data:
EOF

Tempo Configuration

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 48h  # How long to keep traces

storage:
  trace:
    backend: local  # Use s3 for production
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal

# For production with S3:
# storage:
#   trace:
#     backend: s3
#     s3:
#       bucket: my-tempo-traces
#       region: us-east-1

TraceQL: Querying Traces

Tempo’s TraceQL is a trace query language for finding traces by span attributes:

# Find slow payments
{ .service.name = "payments-service" && duration > 1s }

# Find traces with errors
{ status = error }

# Find database calls slower than 500ms
{ span.db.system = "postgresql" && duration > 500ms }

# Find traces with specific HTTP status codes
{ .http.status_code = 500 }

# Complex query: find user checkout flows with payment errors
{ .service.name = "checkout-service" } && { .service.name = "payments-service" && status = error }

Grafana Integration

Configure Tempo as a Grafana data source:

# grafana-datasources.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: prometheus  # For service graph
      nodeGraph:
        enabled: true
      lokiSearch:
        datasourceUid: loki  # Jump from traces to logs

In Grafana Explore:

Query traces with TraceQL
Click a trace to view flame graph
Click a span with log.trace_id to jump to related logs in Loki
Click a service in the service graph to view Prometheus RED metrics

This correlation — moving seamlessly between metrics, logs, and traces — is Tempo’s primary value proposition.

Service Graph Metrics

Tempo can automatically generate service graph metrics (request rate, error rate, duration) from trace data:

# tempo.yaml
metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
  processors:
    - service-graphs
    - span-metrics

This creates Prometheus metrics like:

traces_service_graph_request_total{client, server, status_code}
traces_service_graph_request_duration_seconds{client, server}
traces_spanmetrics_duration_seconds{service, span_name, status_code}

Kubernetes Deployment

# Helm installation
helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo \
  --namespace monitoring \
  --set tempo.storage.trace.backend=s3 \
  --set tempo.storage.trace.s3.bucket=my-tempo-traces \
  --set tempo.storage.trace.s3.region=us-east-1

Tempo is the right choice when you’re already using Grafana for metrics and Loki for logs — it completes the observability stack with minimal additional infrastructure cost.

Grafana Tempo — Scalable Distributed Tracing Backend

✅ Pros

⚠️ Cons