DVC: Data Version Control for ML Projects & Pipelines

Initialize DVC & Track Data Files

Install with pip install dvc[s3], then add DVC to an existing Git repo:

# Initialize DVC (creates .dvc/ directory alongside .git/)
git init my-ml-project && cd my-ml-project
dvc init
git commit -m "Initialize DVC"

# Track a large dataset — DVC moves it to cache, creates a pointer file
dvc add data/train.csv         # creates data/train.csv.dvc
dvc add models/model.pkl       # works for model files too

# Commit the pointer files to Git (not the data itself)
git add data/train.csv.dvc data/.gitignore models/.gitignore
git commit -m "Add training data and model v1"

# Configure S3 remote storage
dvc remote add -d myremote s3://my-ml-bucket/dvc-store
dvc remote modify myremote region us-east-1
git add .dvc/config && git commit -m "Configure S3 remote"

# Push data to S3
dvc push             # uploads data/.dvc/cache to S3

# On another machine: restore exact data version
git clone https://github.com/org/my-ml-project
dvc pull             # downloads from S3 using .dvc pointers

DVC Pipelines: Reproducible ML Stages

Define a pipeline in dvc.yaml — DVC only re-runs stages whose inputs changed:

# dvc.yaml
stages:
  preprocess:
    cmd: python src/preprocess.py
    deps:
      - src/preprocess.py
      - data/raw/train.csv            # if this changes, stage re-runs
    outs:
      - data/processed/train_clean.csv
    params:
      - params.yaml:
          - preprocess.test_size
          - preprocess.random_state

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/processed/train_clean.csv
    outs:
      - models/classifier.pkl
    metrics:
      - reports/metrics.json:         # tracked as metrics, not just artifact
          cache: false
    plots:
      - reports/feature_importance.csv:
          cache: false

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/classifier.pkl
      - data/processed/test_clean.csv
    metrics:
      - reports/eval_metrics.json:
          cache: false

# Run the pipeline (only changed stages execute)
dvc repro

# Compare metrics across Git commits/branches
dvc metrics diff main feature/new-model
# Path                    Metric    HEAD      main      Change
# reports/metrics.json    accuracy  0.9421    0.9187    0.0234

# Visualize training plots
dvc plots show reports/feature_importance.csv

DVC Experiments: Branch-Free Experiment Tracking

DVC Experiments let you run and compare multiple experiment variants without creating Git branches:

# Run an experiment (creates a temporary Git stash under the hood)
dvc exp run --set-param train.learning_rate=0.01

# Run multiple experiments with parameter grid
dvc exp run --set-param train.lr=0.001,0.01,0.1 --queue
dvc queue start --jobs 3    # run 3 in parallel

# Compare all experiments
dvc exp show
# ┏━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
# ┃ Experiment   ┃ accuracy┃ learning_rate  ┃ f1_score ┃
# ┡━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
# │ main         │ 0.9187  │ 0.001          │ 0.9102   │
# │ exp-abc12    │ 0.9421  │ 0.01           │ 0.9380   │ ← best
# │ exp-def34    │ 0.9301  │ 0.1            │ 0.9244   │

# Promote best experiment to a Git branch
dvc exp branch exp-abc12 feature/best-lr

DVC vs Git LFS vs LakeFS vs Pachyderm