conda

Conda is a package manager that installs not just Python libraries but the entire binary dependency stack—C libraries, CUDA toolkits, compilers, R runtimes—by maintaining a curated repository of pre-built binaries for every supported platform, solving the fundamental limitation that pip can only install Python packages whose C extensions are already compiled for your machine. ```svg ``` **Conda's defining capability is resolving the binary ABI compatibility stack before any package is downloaded: when you conda install pytorch cudatoolkit=11.8, the solver finds the exact CUDA runtime version, cuDNN build, PyTorch wheel, and supporting C libraries (libstdc++, glibc) that are mutually ABI-compatible—a constraint that pip's resolver cannot express because pip's package index has no concept of non-Python shared library dependencies.** The repodata.json file for conda-forge/linux-64 alone is ~100–300 MB uncompressed (~15–40 MB gzipped), encoding every package's build metadata including ABI tags, run dependencies, and sha256 of every artifact. The legacy conda SAT solver parsed this graph in ~30–120 s for complex ML environments; the libmamba C++ solver (conda 23.1+) performs the same resolution in ~5–15 s by implementing a version-aware backtracking SAT algorithm in compiled code with lazy repodata loading. **Channels are conda's versioned package repositories and their priority order determines which build of a package is installed: the defaults channel (Anaconda Inc.) ships ~500 curated packages built against Intel MKL, while conda-forge ships ~22,000 community-maintained packages built against OpenBLAS, and mixing channels without setting strict priority (the default since conda 4.7) causes the solver to produce inconsistent environments where packages from different channels link against different C runtime versions.** Setting channel_priority: strict in ~/.condarc ensures a package from a higher-priority channel is never replaced by a newer build from a lower-priority channel, preventing the classic "conda environment works locally but breaks on CI" failure mode. Miniforge installs with conda-forge as the sole default channel, eliminating defaults/conda-forge mixing entirely—the recommended starting point for most users since 2022. **The MKL advantage in conda's defaults channel is real and measurable: numpy linked against Intel MKL executes DGEMM (matrix multiply) on a 1,000×1,000 float64 matrix in ~30 ms versus ~40 ms for OpenBLAS-linked numpy (conda-forge) and ~100 ms for the reference BLAS linked by pip's binary wheel—a 3.3× performance gap on BLAS-heavy workloads like PCA, linear regression, and eigendecomposition.** MKL dispatches to processor-specific SIMD kernels at runtime (AVX-512 on Intel, AVX2 on AMD) and uses multi-threaded BLAS with OpenMP; conda's defaults numpy installs MKL automatically as a dependency, while conda-forge numpy links against OpenBLAS which is competitive on AMD hardware but ~25% slower than MKL on Intel microarchitectures. For GPU-heavy PyTorch workloads, the BLAS gap is irrelevant since matrix operations run on CUDA cores, not the CPU. **Conda's CUDA toolkit management is its exclusive differentiator from pip: conda install cudatoolkit=11.8 cudnn downloads NVIDIA's pre-built binaries (~1–2 GB for toolkit, ~600 MB for cuDNN) from the nvidia or conda-forge channel, linking them into the environment's lib/ directory and setting LD_LIBRARY_PATH—enabling a PyTorch GPU environment without CUDA installed system-wide—critical for multi-user shared machines where system-wide CUDA installations would conflict.** pip install torch installs a self-contained PyTorch wheel with bundled CUDA libraries for a specific toolkit version, which works but requires downloading a 2 GB wheel per PyTorch version per CUDA version and cannot install a CUDA toolkit version different from what the wheel embeds. The conda approach is more composable: toolkit and framework are separate packages, so PyTorch 2.0 and TensorFlow 2.12 can coexist in separate environments each pinned to CUDA 11.8. **The conda package format evolved from .tar.bz2 (bzip2-compressed tarball, ~2–5 s extract time per 50 MB package) to .conda (a zip file containing an inner .tar.zst using zstd compression, ~0.5–1 s extract)—a 4× extraction speedup that meaningfully reduces environment creation time for large environments containing 100+ packages, where extraction time dominates over solver time.** The .conda format stores package metadata in a separate info-.tar.zst archive inside the outer zip, enabling package managers to read metadata without extracting the full package payload—critical for the libmamba solver's lazy repodata strategy. Packages are verified by sha256 hash against the repodata record before extraction, making conda environments bit-reproducible given the same repodata state. **Conda-lock generates platform-specific lockfiles containing exact package URLs, build strings, and sha256 hashes for every transitive dependency—turning conda's environment.yml (which specifies version constraints, not exact builds) into a fully reproducible specification that produces byte-identical environments on any machine running the same OS and architecture.** environment.yml export without --no-builds includes platform-specific build strings (py311h1234567_0) that are non-portable across OS families; --no-builds exports version constraints only, which conda-lock then solves per-platform into separate linux-64, osx-arm64, and win-64 lockfiles. conda env create -f environment.yml takes ~30–120 s for a scientific ML stack (solve + download + extract), while conda env create from a conda-lock lockfile skips the solve step and takes ~60–90 s (download + extract only). | Distribution | Size | Default channel | numpy BLAS | Solver | Best for | |---|---|---|---|---|---| | Anaconda | ~3 GB | defaults | MKL | legacy | Beginners, offline use | | Miniconda | ~50 MB | defaults | MKL (optional) | legacy or libmamba | CI, Docker, experienced | | Miniforge | ~50 MB | conda-forge | OpenBLAS | libmamba | Modern, open-source stack | | Mamba | ~50 MB | any | any | libmamba | Drop-in conda replacement | ``` [CONDA ENVIRONMENT CREATION — solve → download → extract] $ conda create -n myenv python=3.11 pytorch cudatoolkit=11.8 Phase 1: Repodata fetch (per channel × platform) conda-forge/linux-64/repodata.json ~15–40 MB (cache: ~0.1 s) defaults/linux-64/repodata.json ~5–15 MB (cache: ~0.1 s) Cache miss: ~5–30 s total Phase 2: Solve (constraint satisfaction) legacy solver: ~30–120 s (Python SAT, pycosat) libmamba solver: ~5–15 s (C++, lazy repodata) Output: exact package list with URLs + sha256 Phase 3: Download + verify 100+ packages × 10–200 MB each → ~1–10 GB total sha256 verified before each extract Time: ~60–300 s (network) Phase 4: Extract + link .tar.bz2: ~2–5 s/pkg | .conda (zstd): ~0.5–1 s/pkg Hard-link from cache if already downloaded Sets CONDA_PREFIX, activates env vars conda activate myenv: ~200–500 ms (PATH + LD_LIBRARY_PATH) ``` Read conda through a *binary ABI contract* lens rather than a *Python package manager* lens: conda's value is not that it downloads packages—pip does that—it is that every package in a conda channel was compiled against a known, versioned set of shared libraries, and the solver enforces that no two installed packages in an environment require incompatible versions of those libraries. The MKL linkage, the CUDA toolkit download, the strict channel priority, and the .conda format's sha256 manifest are all mechanisms for maintaining this contract. An environment.yml that works today but breaks in six months is a conda contract violation—packages were rebuilt against newer shared libraries in the channel, breaking the ABI guarantee. Conda-lock is the fix: it freezes the contract at a point in time by recording exact URLs and hashes, turning a constraint specification into a reproducible binary agreement.