Git LFS (Large File Storage) is a specialized Git extension protocol that transparently replaces large binary files (trained model weights, datasets, compiled binaries, high-resolution assets) in a Git repository with lightweight text pointer files — offloading the actual massive binary content to a dedicated, separate storage server while preserving the standard Git workflow of add, commit, push, and pull.
The Git Binary File Catastrophe
- The Fundamental Design: Git was architected for tracking changes in text source code files. It stores the complete content of every version of every file as a compressed object in the local repository's hidden .git/ directory.
- The Explosion: When a developer commits a $2$ GB neural network model file (model.bin), Git stores the entire $2$ GB blob. When they retrain and commit a slightly modified version, Git stores another complete $2$ GB blob (binary files cannot be efficiently delta-compressed). After $10$ retraining cycles, the .git/ directory contains $20$ GB of model history. Every git clone by every team member downloads the full $20$ GB, even if they only need the latest version.
The LFS Pointer Mechanism
1. Track: The developer configures Git LFS to track specific file patterns: git lfs track ".bin" ".h5" "*.onnx".
2. Add/Commit: When git add model.bin is executed, Git LFS intercepts the operation. It computes a SHA-256 hash of the file, uploads the actual binary blob to a dedicated LFS storage server (GitHub LFS, GitLab LFS, or a self-hosted server), and commits only a tiny (~130 byte) text pointer file containing the hash and file size.
3. Push: The pointer is pushed to the Git remote. The binary blob is pushed separately to the LFS server.
4. Clone/Pull: When a collaborator clones the repository, Git downloads the tiny pointer files instantly. Git LFS then transparently downloads only the binary blobs that are actually needed (the latest version), reconstructing the full files in the working directory.
The ML Limitation
Git LFS solves the binary bloat problem but lacks the sophisticated dependency graph, data versioning, and pipeline reproducibility features of dedicated ML experiment tracking tools like DVC (Data Version Control). Git LFS treats large files as opaque blobs with no understanding of the training pipeline that generated them.
Git LFS is the heavy cargo freight service — keeping the Git repository's core package light and fast by shipping the massive industrial payloads through a separate, dedicated logistics channel.