Apache Iceberg is the open table format for huge analytical datasets that provides ACID transactions, time travel, and schema evolution on top of object storage — originally created at Netflix to solve the reliability and performance problems of Hive Metastore partitioning at petabyte scale, now the engine-agnostic standard for data lakehouse table formats.
What Is Apache Iceberg?
- Definition: A high-performance table format specification for storing large analytical datasets in object storage — defining how table metadata (schemas, partitioning, snapshots) is stored alongside Parquet/ORC/Avro data files, enabling multiple compute engines to reliably read and write the same table.
- Origin: Created by Netflix engineers Ryan Blue and Daniel Davids to solve production problems with Hive Metastore — specifically the inability to atomically update petabyte-scale tables and the listing overhead of discovering which files belong to a query.
- Engine-Agnostic: Unlike Delta Lake (optimized for Spark/Databricks), Iceberg is a neutral specification — supported natively by Apache Spark, Trino, Presto, Apache Flink, Hive, DuckDB, and cloud engines like Athena, BigQuery Omni, and Snowflake.
- Catalog: Iceberg tables are tracked via a catalog (Hive Metastore, AWS Glue, Nessie, REST catalog) that stores the current metadata pointer — enabling atomic table updates that all engines see simultaneously.
- Adoption: Netflix, Apple, LinkedIn, Adobe, Expedia — production deployments at petabyte+ scale using Iceberg as the foundational table format.
Why Iceberg Matters for AI/ML
- Multi-Engine Flexibility: ML teams using Spark for training, Trino for exploration, and DuckDB for local analysis can all read the same Iceberg table — no vendor lock-in to a single compute engine.
- Hidden Partitioning: Iceberg partitions data transparently without requiring users to include partition columns in every query — the table format handles partition pruning automatically based on the query predicate.
- Time Travel for Reproducibility: Query training data as of any past snapshot — guaranteed to return identical results for model reproduction regardless of subsequent table modifications.
- Schema Evolution Without Rewrites: Add columns, rename columns, or change types in a large feature table without rewriting any data files — Iceberg handles column mapping between old and new schemas at read time.
- Row-Level Deletes: Iceberg v2 supports row-level position deletes and equality deletes — enabling GDPR compliance (delete a user's data) and CDC upserts on analytical tables.
Core Iceberg Features
Snapshot-Based Architecture:
- Every table write creates a new snapshot (immutable set of data files)
- Readers always see a consistent snapshot — no dirty reads during concurrent writes
- Snapshots retained for configurable period enabling time travel
Time Travel:
-- Query historical data
SELECT * FROM orders FOR SYSTEM_TIME AS OF TIMESTAMP '2024-01-01 00:00:00';
SELECT * FROM orders FOR SYSTEM_VERSION AS OF 5234567890;
-- Rollback table to previous snapshot
CALL catalog.system.rollback_to_snapshot('db.orders', 5234567890);
Partition Evolution:
-- Change partitioning strategy without rewriting data
ALTER TABLE orders REPLACE PARTITION FIELD year(order_date) WITH month(order_date);
Metadata Pruning:
- Column-level min/max statistics in manifest files
- Queries skip entire data files based on predicates without reading them
- Orders of magnitude faster than Hive for selective queries on large tables
Iceberg vs Alternatives
| Format | Engine Agnostic | Multi-Writer | Row Deletes | Best For |
|--------|----------------|-------------|-------------|---------|
| Iceberg | Yes | Yes (v2) | Yes (v2) | Multi-engine, open standard |
| Delta Lake | Partial | Yes | Yes | Databricks/Spark focus |
| Hudi | Partial | Yes | Yes | Streaming upserts |
| Hive | No | No | No | Legacy only |
Apache Iceberg is the open standard for analytical table formats that liberates data from single-engine lock-in — by defining a precise, engine-agnostic specification for storing metadata and data files, Iceberg enables any compute engine to reliably read, write, and time-travel on the same petabyte-scale tables with ACID guarantees.