Data Pipeline Diagram Generator

ETL (Extract, Transform, Load) transforms data before loading it into the destination — the transform step runs in a dedicated processing layer (Apache Spark, dbt on a staging DB, custom Python). This is common for legacy data warehouses with limited compute. ELT (Extract, Load, Transform) loads raw data first into the destination (Snowflake, BigQuery, Redshift), then transforms it using the warehouse's compute power. ELT is now dominant in modern cloud data stacks because cloud warehouses are cheap and powerful. dbt (data build tool) is the de facto ELT transform layer.

Source stages: database (Postgres, MySQL), file (S3, GCS, SFTP), API (REST, GraphQL), or stream (Kafka, Kinesis). Transform stages: filter (remove rows), join (combine datasets), aggregate (GROUP BY operations), normalize (standardize formats/types), enrich (add data from external sources), validate (data quality checks), deduplicate (remove exact/fuzzy duplicates). Sink stages: operational database, analytical data warehouse (Snowflake, BigQuery), file output (Parquet on S3), downstream API, or event stream.

Mermaid is natively supported in GitHub Markdown (use ```mermaid code blocks), GitLab, Notion, Confluence (with plugin), Obsidian, and many wikis. For standalone rendering, use mermaid.live (online editor with live preview), or install the Mermaid CLI: npm install -g @mermaid-js/mermaid-cli, then mmdc -i diagram.mmd -o diagram.png. For React/Vue apps, use the mermaid npm package to render diagrams dynamically.

Lambda Architecture: dual batch + streaming paths that merge into a serving layer — complex but handles both historical and real-time data. Kappa Architecture: streaming-only, simpler but requires replayable streams (Kafka). Modern Data Stack: dbt + Fivetran/Airbyte + Snowflake/BigQuery — ELT with managed connectors. Medallion Architecture (used in Databricks): Bronze (raw) → Silver (cleaned) → Gold (aggregated) layers in a data lakehouse. For most teams starting in 2026, a simple ELT pipeline with Fivetran/Airbyte + dbt + BigQuery is the fastest path to production.