llama.cpp is a C/C++ library for running large language model inference on consumer hardware with high performance — created by Georgi Gerganov to demonstrate that Meta's LLaMA models could run on a MacBook, it has grown into the most widely used local LLM inference engine, powering Ollama, LM Studio, GPT4All, and dozens of other tools through its efficient CPU/GPU inference, 4-bit quantization (GGUF format), and zero-dependency design that requires no Python or PyTorch installation.

What Is llama.cpp?

• Definition: A plain C/C++ implementation of LLM inference (no PyTorch, no Python required) that loads quantized model weights in GGUF format and generates text using optimized CPU and GPU kernels — supporting LLaMA, Mistral, Mixtral, Phi, Gemma, Qwen, and virtually every open-weight model architecture.
• Key Innovation — Quantization: llama.cpp popularized 4-bit quantization for practical use — compressing a 70B parameter model from 140 GB (FP16) to ~40 GB (Q4_K_M) with minimal quality loss, making it runnable on a Mac Studio or high-RAM PC.
• Zero Dependencies: Download the binary and a GGUF model file — that's it. No Python environment, no CUDA toolkit, no pip install. This simplicity is why llama.cpp became the foundation for user-friendly tools like Ollama.
• Hardware Support: CPU (AVX2, AVX-512, ARM NEON), NVIDIA GPU (CUDA), Apple GPU (Metal), AMD GPU (ROCm/Vulkan), Intel GPU (SYCL) — the widest hardware support of any local inference engine.

Key Features

• GGUF Model Format: Self-describing model files containing weights, tokenizer, and metadata — download a single .gguf file and run it immediately. Thousands of GGUF models available on Hugging Face Hub.
• Server Mode: llama-server provides an OpenAI-compatible REST API — drop-in replacement for OpenAI API in applications, enabling local inference with zero code changes.
• Speculative Decoding: Use a small draft model to propose tokens, verified by the large model — 2-3× speedup for generation with no quality loss.
• Grammar-Constrained Generation: GBNF grammar support forces output to match a specified format — guaranteed valid JSON, SQL, or any structured output.
• Continuous Batching: Serve multiple concurrent requests efficiently — the server batches requests together for higher throughput on GPU.
• Context Extension: RoPE scaling and YaRN support for extending context length beyond the model's training length — run 8K models at 32K+ context.

llama.cpp Model Compatibility

llama.cpp is the inference engine that democratized local LLM access — by providing efficient C/C++ inference with aggressive quantization and zero dependencies, llama.cpp made it possible for anyone with a modern laptop to run powerful language models privately, spawning an entire ecosystem of user-friendly tools built on its foundation.