Apache Iceberg is the open table format for huge analytical datasets that provides ACID transactions, time travel, and schema evolution on top of object storage — originally created at Netflix to solve the reliability and performance problems of Hive Metastore partitioning at petabyte scale, now the engine-agnostic standard for data lakehouse table formats.

What Is Apache Iceberg?

• Definition: A high-performance table format specification for storing large analytical datasets in object storage — defining how table metadata (schemas, partitioning, snapshots) is stored alongside Parquet/ORC/Avro data files, enabling multiple compute engines to reliably read and write the same table.
• Origin: Created by Netflix engineers Ryan Blue and Daniel Davids to solve production problems with Hive Metastore — specifically the inability to atomically update petabyte-scale tables and the listing overhead of discovering which files belong to a query.
• Engine-Agnostic: Unlike Delta Lake (optimized for Spark/Databricks), Iceberg is a neutral specification — supported natively by Apache Spark, Trino, Presto, Apache Flink, Hive, DuckDB, and cloud engines like Athena, BigQuery Omni, and Snowflake.
• Catalog: Iceberg tables are tracked via a catalog (Hive Metastore, AWS Glue, Nessie, REST catalog) that stores the current metadata pointer — enabling atomic table updates that all engines see simultaneously.
• Adoption: Netflix, Apple, LinkedIn, Adobe, Expedia — production deployments at petabyte+ scale using Iceberg as the foundational table format.

Why Iceberg Matters for AI/ML

• Multi-Engine Flexibility: ML teams using Spark for training, Trino for exploration, and DuckDB for local analysis can all read the same Iceberg table — no vendor lock-in to a single compute engine.
• Hidden Partitioning: Iceberg partitions data transparently without requiring users to include partition columns in every query — the table format handles partition pruning automatically based on the query predicate.
• Time Travel for Reproducibility: Query training data as of any past snapshot — guaranteed to return identical results for model reproduction regardless of subsequent table modifications.
• Schema Evolution Without Rewrites: Add columns, rename columns, or change types in a large feature table without rewriting any data files — Iceberg handles column mapping between old and new schemas at read time.
• Row-Level Deletes: Iceberg v2 supports row-level position deletes and equality deletes — enabling GDPR compliance (delete a user's data) and CDC upserts on analytical tables.

Core Iceberg Features

Snapshot-Based Architecture:

• Every table write creates a new snapshot (immutable set of data files)
• Readers always see a consistent snapshot — no dirty reads during concurrent writes
• Snapshots retained for configurable period enabling time travel

Time Travel: -- Query historical data SELECT FROM orders FOR SYSTEM_TIME AS OF TIMESTAMP '2024-01-01 00:00:00'; SELECT FROM orders FOR SYSTEM_VERSION AS OF 5234567890;

-- Rollback table to previous snapshot CALL catalog.system.rollback_to_snapshot('db.orders', 5234567890);

Partition Evolution: -- Change partitioning strategy without rewriting data ALTER TABLE orders REPLACE PARTITION FIELD year(order_date) WITH month(order_date);

Metadata Pruning:

• Column-level min/max statistics in manifest files
• Queries skip entire data files based on predicates without reading them
• Orders of magnitude faster than Hive for selective queries on large tables

Iceberg vs Alternatives

Apache Iceberg is the open standard for analytical table formats that liberates data from single-engine lock-in — by defining a precise, engine-agnostic specification for storing metadata and data files, Iceberg enables any compute engine to reliably read, write, and time-travel on the same petabyte-scale tables with ACID guarantees.