llama cpp
**llama.cpp** is a **C/C++ library for running large language model inference on consumer hardware with high performance** — created by Georgi Gerganov to demonstrate that Meta's LLaMA models could run on a MacBook, it has grown into the most widely used local LLM inference engine, powering Ollama, LM Studio, GPT4All, and dozens of other tools through its efficient CPU/GPU inference, 4-bit quantization (GGUF format), and zero-dependency design that requires no Python or PyTorch installation.
**What Is llama.cpp?**
- **Definition**: A plain C/C++ implementation of LLM inference (no PyTorch, no Python required) that loads quantized model weights in GGUF format and generates text using optimized CPU and GPU kernels — supporting LLaMA, Mistral, Mixtral, Phi, Gemma, Qwen, and virtually every open-weight model architecture.
- **Key Innovation — Quantization**: llama.cpp popularized 4-bit quantization for practical use — compressing a 70B parameter model from 140 GB (FP16) to ~40 GB (Q4_K_M) with minimal quality loss, making it runnable on a Mac Studio or high-RAM PC.
- **Zero Dependencies**: Download the binary and a GGUF model file — that's it. No Python environment, no CUDA toolkit, no pip install. This simplicity is why llama.cpp became the foundation for user-friendly tools like Ollama.
- **Hardware Support**: CPU (AVX2, AVX-512, ARM NEON), NVIDIA GPU (CUDA), Apple GPU (Metal), AMD GPU (ROCm/Vulkan), Intel GPU (SYCL) — the widest hardware support of any local inference engine.
**Key Features**
- **GGUF Model Format**: Self-describing model files containing weights, tokenizer, and metadata — download a single `.gguf` file and run it immediately. Thousands of GGUF models available on Hugging Face Hub.
- **Server Mode**: `llama-server` provides an OpenAI-compatible REST API — drop-in replacement for OpenAI API in applications, enabling local inference with zero code changes.
- **Speculative Decoding**: Use a small draft model to propose tokens, verified by the large model — 2-3× speedup for generation with no quality loss.
- **Grammar-Constrained Generation**: GBNF grammar support forces output to match a specified format — guaranteed valid JSON, SQL, or any structured output.
- **Continuous Batching**: Serve multiple concurrent requests efficiently — the server batches requests together for higher throughput on GPU.
- **Context Extension**: RoPE scaling and YaRN support for extending context length beyond the model's training length — run 8K models at 32K+ context.
**llama.cpp Model Compatibility**
| Model Family | Supported | Popular GGUF Variants |
|-------------|-----------|----------------------|
| LLaMA 2/3 | Yes | Q4_K_M, Q5_K_M, Q8_0 |
| Mistral/Mixtral | Yes | Q4_K_M, Q5_K_M |
| Phi-2/3 | Yes | Q4_K_M, Q8_0 |
| Gemma/Gemma 2 | Yes | Q4_K_M, Q5_K_M |
| Qwen 1.5/2 | Yes | Q4_K_M, Q5_K_M |
| Command R | Yes | Q4_K_M |
| StarCoder 2 | Yes | Q4_K_M, Q8_0 |
**llama.cpp is the inference engine that democratized local LLM access** — by providing efficient C/C++ inference with aggressive quantization and zero dependencies, llama.cpp made it possible for anyone with a modern laptop to run powerful language models privately, spawning an entire ecosystem of user-friendly tools built on its foundation.