ollama
**Ollama** is the **easiest way to run open-source large language models locally, packaging model download, quantization, and serving into a single CLI tool** — providing a Docker-like experience where `ollama pull llama3` downloads a model and `ollama run llama3` starts an interactive chat session, with a built-in OpenAI-compatible REST API that enables local LLM integration into any application without cloud API costs, internet dependency, or data privacy concerns.
**What Is Ollama?**
- **Definition**: A local LLM runtime that wraps llama.cpp in a user-friendly package — handling model downloading, GGUF format management, GPU detection, memory allocation, and API serving so users never interact with raw model files or compilation flags.
- **One-Line Install**: `curl -fsSL https://ollama.com/install.sh | sh` on Linux/Mac — a single command installs the Ollama daemon, CLI, and all dependencies. Windows installer also available.
- **Docker-Like Model Management**: `ollama pull` downloads models, `ollama list` shows installed models, `ollama rm` removes them — the same mental model as Docker images, making it immediately familiar to developers.
- **Model Library**: Ollama hosts a curated library of pre-quantized models — Llama 3, Mistral, Mixtral, Phi-3, Gemma, CodeLlama, Qwen, Command R, and dozens more, each available in multiple size variants (7B, 13B, 70B) and quantization levels.
- **OpenAI-Compatible API**: `http://localhost:11434/v1/chat/completions` — applications using the OpenAI SDK can switch to local inference by changing the base URL, with zero code changes to the application logic.
**Key Features**
- **Automatic GPU Detection**: Ollama detects NVIDIA (CUDA), AMD (ROCm), and Apple Silicon (Metal) GPUs automatically — no manual CUDA configuration or driver management.
- **Model Customization (Modelfile)**: Create custom model configurations with system prompts, temperature settings, and parameter overrides — `FROM llama3` + `SYSTEM "You are a helpful coding assistant"` creates a specialized variant.
- **Concurrent Requests**: The Ollama server handles multiple simultaneous requests with automatic batching — suitable for multi-user development teams sharing a single GPU server.
- **Embedding API**: `ollama.embeddings(model="nomic-embed-text", prompt="text")` generates embeddings locally — enabling fully local RAG pipelines without any cloud API calls.
- **Multimodal**: Support for vision models (LLaVA, Llama 3.2 Vision) — send images alongside text prompts for local multimodal inference.
**Ollama Model Library (Popular Models)**
| Model | Sizes | Use Case | RAM Required (Q4) |
|-------|-------|----------|-------------------|
| llama3.1 | 8B, 70B, 405B | General chat, reasoning | 5 GB / 40 GB / 230 GB |
| mistral | 7B | Fast general purpose | 4.5 GB |
| mixtral | 8x7B | High quality, MoE | 26 GB |
| phi3 | 3.8B, 14B | Small, efficient | 2.5 GB / 8 GB |
| gemma2 | 9B, 27B | Google's open model | 5.5 GB / 16 GB |
| codellama | 7B, 13B, 34B | Code generation | 4.5 GB / 8 GB / 20 GB |
| nomic-embed-text | 137M | Embeddings | 0.3 GB |
**Ollama vs Alternatives**
| Feature | Ollama | LM Studio | GPT4All | llama.cpp (raw) |
|---------|--------|----------|---------|----------------|
| Interface | CLI + API | GUI | GUI + API | CLI |
| Setup | 1 command | Installer | Installer | Compile from source |
| Model management | Docker-like | Hub browser | Built-in | Manual GGUF files |
| API | OpenAI-compatible | OpenAI-compatible | REST API | llama-server |
| GPU support | Auto-detect | Auto-detect | CPU focus | Manual flags |
| Customization | Modelfile | UI settings | Limited | Full control |
| Target user | Developers | Non-technical | Non-technical | Power users |
**Ollama is the tool that made local LLM inference as simple as running a Docker container** — wrapping the complexity of model management, quantization, and GPU configuration into a familiar pull/run workflow with an OpenAI-compatible API that lets developers build privacy-preserving AI applications without cloud dependencies.