Overview

Buildkite takes a hybrid approach to CI/CD: the control plane (scheduling, UI, API) is Buildkite’s cloud, but the actual build execution happens on agents you run on your own infrastructure. This means your code and secrets never touch Buildkite’s servers — only job metadata and results do. It’s popular at companies like Shopify, Airbnb, and Canva that run tens of thousands of builds per day.

Basic Pipeline

# .buildkite/pipeline.yml
steps:
  - label: ":npm: Install"
    command: npm ci
    key: install

  - label: ":test_tube: Test"
    command: npm test
    depends_on: install
    parallelism: 4  # Split across 4 agents

  - label: ":hammer: Build"
    command: npm run build
    depends_on: test
    artifact_paths: "dist/**/*"

  - wait  # All above must pass before continuing

  - label: ":rocket: Deploy"
    command: ./scripts/deploy.sh
    branches: "main"

Dynamic Pipelines

Buildkite’s killer feature: generate pipeline steps at runtime:

#!/bin/bash
# .buildkite/generate-pipeline.sh

# Determine which services changed
CHANGED=$(git diff --name-only origin/main...HEAD)

cat << EOF
steps:
EOF

if echo "$CHANGED" | grep -q "^services/api/"; then
  cat << EOF
  - label: "Test API"
    command: "cd services/api && npm test"
EOF
fi

if echo "$CHANGED" | grep -q "^services/web/"; then
  cat << EOF
  - label: "Test Web"
    command: "cd services/web && npm test"
EOF
fi

# .buildkite/pipeline.yml
steps:
  - label: ":pipeline: Generate pipeline"
    command: .buildkite/generate-pipeline.sh | buildkite-agent pipeline upload

Running Agents

Install the Buildkite agent on any machine:

# Linux/macOS
curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh | bash

# Configure
vim /etc/buildkite-agent/buildkite-agent.cfg
# token="your-agent-token"
# name="my-agent-%n"
# tags="queue=default,os=linux"

# Start
buildkite-agent start

Elastic CI on AWS

Auto-scale agents with official CloudFormation stack:

# Deploy auto-scaling agent pool
aws cloudformation create-stack \
  --stack-name buildkite-agents \
  --template-url https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml \
  --parameters \
    ParameterKey=BuildkiteAgentToken,ParameterValue=xxx \
    ParameterKey=MaxSize,ParameterValue=20 \
    ParameterKey=MinSize,ParameterValue=0 \
    ParameterKey=InstanceType,ParameterValue=c5.2xlarge \
    ParameterKey=SpotPrice,ParameterValue=0.20

Plugin Ecosystem

steps:
  - label: ":docker: Build"
    plugins:
      # Official Docker plugin
      - docker-compose#v4.0.0:
          run: app
          config: docker-compose.test.yml

      # Secrets from AWS SSM
      - aws-ssm#v1.0.0:
          parameters:
            - path: /myapp/prod/db-password
              env: DB_PASSWORD

      # Slack notifications
      - slack-notification#v1.6.0:
          channels: ["#deployments"]
          message: "Deploy complete: ${BUILDKITE_BUILD_URL}"

Test Analytics

# Upload JUnit XML results for flaky test detection
npm test -- --reporter junit --outputFile test-results.xml

# Upload via Buildkite CLI
buildkite-agent artifact upload test-results.xml

Buildkite Test Analytics tracks test duration trends, flaky tests, and which tests fail most — helping prioritize stability work.

Best For

Large engineering organizations that need to run thousands of builds per day on their own hardware (Mac minis for iOS, GPU servers for ML, bare metal for performance testing) without paying cloud CI per-minute rates
Security-sensitive industries (finance, healthcare, defense) where source code and build artifacts cannot leave the company’s own infrastructure
Mobile teams running iOS and Android builds on self-hosted Mac agents that other CI platforms charge a premium for or don’t support well
Teams with specialized build hardware — custom GPU machines, FPGAs, or lab equipment that must run build agents on-site

Buildkite vs. Alternatives

	Buildkite	GitHub Actions	CircleCI	Jenkins
Infrastructure	Bring your own	GitHub-hosted or self-hosted	Cloud or self-hosted	Self-hosted
Orchestration	Cloud (Buildkite)	Cloud (GitHub)	Cloud (CircleCI)	Self-hosted
Cost model	Per-seat, flat	Per-minute (hosted)	Per-minute	Free (infra cost)
Plugins	YAML-based ecosystem	Action marketplace	Orbs	1,800+ plugins
Best for	Large orgs, custom hardware, security	GitHub projects, open source	Mid-size teams	Legacy enterprise

Buildkite’s hybrid model (their cloud orchestrates, your agents run) hits the sweet spot between Jenkins (all self-hosted, high ops burden) and GitHub Actions (easy but expensive at scale). It’s the right choice when you need to own your build infrastructure without owning the CI control plane.

Buildkite — Scalable CI/CD on Your Own Infrastructure

✅ Pros

⚠️ Cons