constrained decoding
**Constrained Decoding** is a **generation technique that forces LLM output to strictly conform to a predefined grammar, schema, or regular expression** — filtering the vocabulary at each generation step to allow only tokens that produce valid completions according to the constraint (JSON schema, SQL syntax, function signatures), guaranteeing syntactically correct output for downstream program consumption without relying on the model to "learn" the output format through prompting alone.
**What Is Constrained Decoding?**
- **Definition**: A modification to the LLM decoding process where, at each token generation step, the set of allowed next tokens is restricted to only those that would produce a valid partial completion according to a formal grammar or schema — invalid tokens have their probabilities set to zero before sampling.
- **Grammar-Based Masking**: A context-free grammar (CFG) or regular expression defines the valid output space — at each step, the decoder determines which tokens are valid continuations of the current partial output according to the grammar, and masks all other tokens.
- **JSON Mode**: The most common constrained decoding application — ensures output is valid, parseable JSON by restricting tokens to those that maintain valid JSON syntax at each generation step. Many LLM APIs now offer built-in JSON mode.
- **Schema Enforcement**: Beyond syntactic validity, constrained decoding can enforce semantic schemas — ensuring output matches a specific JSON Schema with required fields, correct types, and valid enum values.
**Why Constrained Decoding Matters**
- **Eliminates Parsing Failures**: Without constraints, LLMs occasionally produce malformed JSON, incomplete structures, or invalid syntax — constrained decoding guarantees 100% syntactic correctness, eliminating retry loops and error handling for parsing failures.
- **Type Safety**: Constrained decoding ensures output matches expected types — strings where strings are expected, numbers where numbers are expected, valid enum values from a predefined set.
- **Reduced Token Waste**: Without constraints, models may generate explanatory text, markdown formatting, or preamble before the actual structured output — constraints force immediate generation of the target format.
- **Program Integration**: AI outputs that feed into downstream programs (APIs, databases, code execution) must be syntactically valid — constrained decoding bridges the gap between probabilistic text generation and deterministic software interfaces.
**Constrained Decoding Libraries**
- **Outlines**: Open-source library for structured generation — supports JSON Schema, regex, CFG, and custom constraints with efficient token masking.
- **Guidance (Microsoft)**: Template-based constrained generation — interleaves fixed text with model-generated content within defined constraints.
- **LMQL**: Query language for LLMs — SQL-like syntax for specifying output constraints, types, and control flow.
- **JSONFormer**: Specialized JSON generation — fills in values within a predefined JSON structure.
- **vLLM + Outlines**: Production-grade integration — Outlines constraints with vLLM's high-throughput serving for constrained generation at scale.
| Feature | Unconstrained | JSON Mode | Full Schema Constraint |
|---------|-------------|-----------|----------------------|
| Syntax Validity | Not guaranteed | JSON guaranteed | Schema guaranteed |
| Type Safety | No | Partial | Full |
| Retry Needed | Often | Rarely | Never |
| Token Efficiency | Low (preamble) | Medium | High |
| Latency Overhead | None | Minimal | 5-15% |
| Library | None | API built-in | Outlines, Guidance |
**Constrained decoding is the technique that makes LLM output reliably machine-readable** — enforcing grammatical, schema, and type constraints at the token level during generation to guarantee syntactically correct structured output, eliminating the parsing failures and retry loops that plague unconstrained LLM integration in production software systems.