databricks
**Databricks** is the **unified data intelligence platform founded by the creators of Apache Spark that combines data engineering, data warehousing, and machine learning** — pioneering the Lakehouse architecture that merges the flexibility of data lakes with the reliability of data warehouses, while providing managed Spark clusters, Delta Lake storage, MLflow experiment tracking, and large-scale LLM training via MosaicML.
**What Is Databricks?**
- **Definition**: A cloud data platform founded in 2013 by the creators of Apache Spark at UC Berkeley — providing managed Spark clusters (Databricks Runtime), the Delta Lake open table format, the MLflow ML experiment tracking standard, and the Unity Catalog data governance layer as a unified platform on AWS, Azure, and GCP.
- **Lakehouse Architecture**: Databricks invented and popularized the "Data Lakehouse" — storing data in open formats (Parquet + Delta Lake) on cheap object storage (S3/ADLS/GCS) while providing ACID transactions, schema enforcement, and SQL analytics performance previously requiring separate data warehouse products.
- **Spark Standard**: Databricks is the primary commercial distribution of Apache Spark — the team that wrote Spark continues to develop it, so Databricks customers get the most optimized Spark runtime with proprietary enhancements (Photon vectorized engine, Delta Engine).
- **Open Source Stewardship**: Databricks created and maintains MLflow (experiment tracking), Delta Lake (ACID table format), Apache Spark (distributed computing), and Koalas (Pandas on Spark) — core infrastructure for the modern data stack.
- **MosaicML Acquisition**: Acquired MosaicML in 2023 for $1.3B — integrating enterprise LLM training, fine-tuning, and deployment capabilities including the DBRX open-source model.
**Why Databricks Matters for AI**
- **Unified Analytics + ML**: Run SQL analytics, Python data science, and ML training on the same data without ETL between systems — a data scientist can query production data in SQL then feed it directly into PyTorch training in the same notebook.
- **Delta Lake Foundation**: ACID transactions on petabyte-scale datasets enable reliable ML training pipelines — concurrent writes, time travel for reproducible dataset versions, schema evolution without data rewrites.
- **Spark for Data Preprocessing**: Process terabytes of training data with distributed Spark — tokenize, deduplicate, and format datasets for LLM training at scales impossible on single machines.
- **MLflow Native Integration**: Experiment tracking, model registry, and deployment integrated directly into Databricks notebooks — every training run automatically logged to the shared MLflow server.
- **Enterprise Governance**: Unity Catalog provides column-level access control, data lineage tracking, and audit logs across all Databricks workspaces — critical for regulated industries.
**Databricks Key Components**
**Databricks Notebooks**:
- Collaborative Jupyter-like notebooks supporting Python, SQL, R, Scala
- Attach to Spark clusters or single-node GPU instances
- Real-time collaboration (like Google Docs for data science)
- MLflow auto-logging: training runs logged automatically
**Databricks Clusters**:
- Managed Apache Spark clusters: define cluster size, auto-terminate on idle
- Interactive clusters: persistent for development
- Job clusters: ephemeral clusters for scheduled workloads
- GPU clusters: for PyTorch/TensorFlow training (A10, A100 instances)
**Delta Lake**:
from delta.tables import DeltaTable
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Write training data as Delta table
df.write.format("delta").save("s3://bucket/training-data/")
# Time travel: read dataset as of specific version
df_v1 = spark.read.format("delta").option("versionAsOf", 1).load("s3://bucket/training-data/")
# MERGE (upsert) for streaming data ingestion
DeltaTable.forPath(spark, "s3://bucket/training-data/").merge(
updates_df, "target.id = source.id"
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()
**MLflow Integration**:
import mlflow
mlflow.autolog() # Automatically logs params, metrics, artifacts
with mlflow.start_run():
model = train_model(lr=0.001, epochs=10)
mlflow.log_metric("val_accuracy", 0.95)
mlflow.pytorch.log_model(model, "model")
**Databricks SQL (Warehouse)**:
- ANSI SQL interface over Delta Lake tables
- Photon vectorized query engine: 2-12x faster than standard Spark SQL
- BI tool integration: Tableau, Power BI, Looker via JDBC/ODBC
**Unity Catalog**:
- Unified governance across all data assets (tables, files, ML models, dashboards)
- Fine-grained access control: row-level, column-level, tag-based
- Automated data lineage: track data transformations end-to-end
**LLM Capabilities (MosaicML)**:
- Train custom LLMs from scratch on Databricks GPU clusters
- Fine-tune open-source models (Llama, Mistral) on proprietary data
- Serve LLMs via Databricks Model Serving (llm/ endpoint namespace)
- DBRX: Databricks' own open-source mixture-of-experts LLM
**Databricks vs Alternatives**
| Aspect | Databricks | Snowflake | AWS SageMaker | dbt + BigQuery |
|--------|-----------|---------|--------------|---------------|
| Data Processing | Spark (best) | SQL only | SageMaker Processing | dbt SQL |
| ML Training | Native GPU | Via partner | Native | External |
| Table Format | Delta Lake | Proprietary | S3 + Glue | BigQuery native |
| Governance | Unity Catalog | Good | Lake Formation | Limited |
| Best For | Unified data+ML | Pure SQL analytics | AWS ML | Analytics-first |
Databricks is **the unified platform where data engineering and machine learning converge on a lakehouse architecture** — by providing managed Spark for massive-scale data processing, Delta Lake for reliable open-format storage, and integrated MLflow for experiment tracking and model management, Databricks enables data teams to move from raw data to production AI models without context-switching between disconnected tools.