Overview

Apache Iceberg is the table format that Netflix built for petabyte-scale tables with thousands of concurrent readers and writers. It’s now the dominant open standard, supported natively by Snowflake, BigQuery, AWS Athena, and Databricks — making it the most portable choice for multi-cloud data lakehouses.

Core Advantages Over Plain Parquet

Hidden Partitioning: Query without specifying partition column:

-- Without Iceberg (Hive-style partitioning): you must filter by partition
SELECT * FROM orders WHERE year = 2026 AND month = 4 AND day = 2;

-- With Iceberg hidden partitioning: just filter by the natural column
SELECT * FROM orders WHERE created_at >= '2026-04-02';
-- Iceberg automatically uses the underlying day partition

Partition Evolution: Change how data is partitioned as it grows:

-- Start with monthly partitioning
ALTER TABLE orders ADD PARTITION FIELD months(created_at);

-- Switch to daily as data grows (old data keeps monthly, new data gets daily)
ALTER TABLE orders DROP PARTITION FIELD months(created_at);
ALTER TABLE orders ADD PARTITION FIELD days(created_at);

PySpark + Iceberg

spark = SparkSession.builder \
    .appName("Iceberg Example") \
    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.glue_catalog.warehouse", "s3://data-lake/iceberg/") \
    .config("spark.sql.catalog.glue_catalog.catalog-impl",
            "org.apache.iceberg.aws.glue.GlueCatalog") \
    .getOrCreate()

# Create table
spark.sql("""
    CREATE TABLE glue_catalog.analytics.orders (
        order_id        STRING,
        customer_id     STRING,
        created_at      TIMESTAMP,
        status          STRING,
        amount          DECIMAL(10,2),
        category        STRING
    )
    USING iceberg
    PARTITIONED BY (days(created_at))
""")

# Write
orders_df.writeTo("glue_catalog.analytics.orders").append()

# Merge (upsert)
spark.sql("""
    MERGE INTO glue_catalog.analytics.orders AS target
    USING new_orders AS source
    ON target.order_id = source.order_id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

Time Travel

-- Query historical version
SELECT * FROM orders
FOR SYSTEM_TIME AS OF TIMESTAMP '2026-04-01 00:00:00';

-- By snapshot ID
SELECT * FROM orders
FOR VERSION AS OF 8516875965979439684;

-- Show table history
SELECT * FROM orders.history;

-- Show snapshots
SELECT * FROM orders.snapshots;

Schema Evolution

-- Add column (safe, never breaks readers)
ALTER TABLE orders ADD COLUMN discount_code STRING;

-- Rename column (safe)
ALTER TABLE orders RENAME COLUMN amount TO total_amount;

-- Change type (safe promotions only, e.g., int → long)
ALTER TABLE orders ALTER COLUMN total_amount TYPE DECIMAL(12,2);

Iceberg vs Delta Lake

	Apache Iceberg	Delta Lake
Governance	Apache Foundation	Linux Foundation
Hidden partitioning	✅	❌
Multi-engine	Excellent	Good
Databricks	Supported	Native
Snowflake	Native	Via connector
BigQuery	Native	No
AWS Athena	Native	Via manifest
Best for	Multi-cloud, multi-engine	Databricks-centric stacks

For Databricks-centric organizations, Delta Lake is simpler. For multi-cloud or multi-engine architectures, Iceberg’s open standard and broader engine support make it the better long-term choice.

Best For

Multi-engine data architectures where different teams use Spark, Trino, Flink, and Athena to query the same data — Iceberg’s open spec ensures compatibility across all engines
Multi-cloud organizations who need to read data from S3, GCS, and Azure Blob with a single table format that doesn’t lock them into one cloud
Teams adopting a lakehouse with AWS Glue, Snowflake, or BigQuery as the catalog — all natively support Iceberg as a first-class table format
Data mesh architectures where each domain team publishes data products as Iceberg tables consumable by any engine across the organization

Apache Iceberg vs. Delta Lake vs. Apache Hudi

	Apache Iceberg	Delta Lake	Apache Hudi
Multi-engine support	Excellent	Growing	Partial
Databricks native	Supported	✅ Native	Supported
AWS Athena native	✅	Via manifest	✗
Snowflake native	✅	✗	✗
BigQuery native	✅	✗	✗
Streaming writes	✅	✅	✅ (specialty)
Best for	Multi-cloud, multi-engine	Databricks-centric	Record-level CDC at scale

If your stack is Databricks-only, Delta Lake is the simpler choice. Choose Iceberg for multi-engine or multi-cloud architectures where open standards and broad engine compatibility matter most.

Apache Iceberg — Open Table Format for Huge Analytic Datasets

✅ Pros

⚠️ Cons