Airbyte: Open-Source Data Integration Platform Guide

Airbyte is the leading open-source ELT platform with 350+ connectors that you can self-host for free. Move data from any source to any destination — databases, SaaS tools, APIs, and files — with full control over your infrastructure and connector code.

What is Airbyte?

Airbyte is an open-source ELT (Extract, Load, Transform) data integration platform founded in 2020. It provides a catalog of 350+ pre-built connectors — covering databases (Postgres, MySQL, MongoDB, SQL Server), SaaS applications (Salesforce, HubSpot, Stripe, Google Ads, Zendesk), cloud storage (S3, GCS), and data warehouses. The open-source version is free to self-host on Docker Compose or Kubernetes, making it an attractive alternative to Fivetran and Stitch for cost-conscious teams.

Airbyte's architecture separates the platform (UI, scheduler, metadata DB) from the connector runtime (Docker containers for each connector). Each sync runs a source connector and a destination connector in Docker, extracting and loading data in Airbyte's internal JSON format. The platform handles scheduling, state management (incremental cursors), and error retry. Airbyte Cloud is the managed offering; for most teams scaling beyond a single developer, Kubernetes deployment with the Helm chart is the recommended production path.

Self-Hosted Deployment

# Deploy Airbyte self-hosted with Docker Compose
# Requirements: Docker, Docker Compose, 4GB+ RAM

git clone https://github.com/airbytehq/airbyte.git
cd airbyte
./run-ab-platform.sh

# Airbyte UI available at: http://localhost:8000
# Default credentials: airbyte / password
#
# For production, use Airbyte on Kubernetes:
helm repo add airbyte https://airbytehq.github.io/helm-charts
helm install airbyte airbyte/airbyte \
  --namespace airbyte \
  --set global.edition=community \
  --set webapp.service.type=LoadBalancer

# Environment variables for custom config:
# .env
AIRBYTE_VERSION=0.63.0
DATABASE_PASSWORD=your_secure_password
SECRET_PERSISTENCE=GOOGLE_SECRET_MANAGER  # or AWS_SECRET_MANAGER
LOG_LEVEL=INFO

Managing Connections via Airbyte API v2

# Airbyte API v2 — manage connections programmatically
# pip install requests

import requests

BASE_URL = "https://api.airbyte.com/v1"
CLIENT_ID = "your_client_id"
CLIENT_SECRET = "your_client_secret"

# 1. Get access token
token_resp = requests.post(
    "https://api.airbyte.com/v1/applications/token",
    json={"client_id": CLIENT_ID, "client_secret": CLIENT_SECRET},
)
token = token_resp.json()["access_token"]
HEADERS = {"Authorization": f"Bearer {token}"}

# 2. List available source definitions
sources = requests.get(
    f"{BASE_URL}/sources",
    headers=HEADERS,
).json()

# 3. Create a Postgres source
new_source = requests.post(
    f"{BASE_URL}/sources",
    headers=HEADERS,
    json={
        "name": "Production Postgres",
        "workspaceId": "your_workspace_id",
        "definitionId": "decd338e-5647-4c0b-adf4-da0e75f5a750",  # Postgres
        "configuration": {
            "sourceType": "postgres",
            "host": "db.example.com",
            "port": 5432,
            "database": "production",
            "username": "airbyte",
            "password": "secret",
            "ssl_mode": {"mode": "require"},
            "replication_method": {
                "method": "CDC",
                "publication": "airbyte_pub",
                "replication_slot": "airbyte_slot",
            },
        },
    },
).json()
source_id = new_source["sourceId"]
print(f"Source created: {source_id}")

Building a Custom Connector with CDK

# Airbyte CDK — build a custom connector in Python
# pip install airbyte-cdk

from airbyte_cdk.sources import AbstractSource
from airbyte_cdk.sources.streams.http import HttpStream
from airbyte_cdk.models import SyncMode
from typing import Any, Iterable, List, Mapping, Optional, Tuple
import requests

class OrdersStream(HttpStream):
    url_base = "https://api.example.com/"
    primary_key = "order_id"

    def path(self, **kwargs) -> str:
        return "orders"

    def next_page_token(self, response: requests.Response) -> Optional[Mapping]:
        data = response.json()
        if data.get("next_cursor"):
            return {"cursor": data["next_cursor"]}
        return None

    def request_params(self, next_page_token=None, **kwargs) -> Mapping:
        params = {"limit": 100}
        if next_page_token:
            params["cursor"] = next_page_token["cursor"]
        return params

    def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]:
        yield from response.json().get("orders", [])

class MyApiSource(AbstractSource):
    def check_connection(self, logger, config) -> Tuple[bool, Any]:
        try:
            resp = requests.get(
                "https://api.example.com/health",
                headers={"Authorization": f"Bearer {config['api_key']}"},
            )
            return resp.status_code == 200, None
        except Exception as e:
            return False, str(e)

    def streams(self, config: Mapping) -> List:
        auth = {"Authorization": f"Bearer {config['api_key']}"}
        return [OrdersStream(authenticator=auth)]

Airbyte vs Fivetran vs Stitch

Production Best Practices

• Use Incremental Deduped sync mode for tables with primary keys to avoid data duplication
• Configure CDC (Debezium) for high-volume database sources — far lower source load than key-based polling
• Deploy on Kubernetes with dedicated worker nodes to isolate connector memory from platform services
• Store secrets in AWS Secrets Manager or Google Secret Manager instead of the Airbyte DB for production security
• Use the Airbyte Terraform provider to manage connections as code — version control your data pipeline configuration
• Monitor sync lag and failure rates in the Airbyte UI or export metrics to Datadog/Prometheus via the Airbyte metrics endpoint