json mode

**JSON mode** is a feature offered by LLM APIs that forces the model to generate output that is **guaranteed to be valid JSON**. This eliminates the common problem of LLMs producing malformed, truncated, or incorrectly formatted JSON when you need structured data output. **How JSON Mode Works** - **Token-Level Enforcement**: At each generation step, the system only allows tokens that would result in valid JSON according to a **JSON grammar parser**. Invalid tokens are masked out before sampling. - **Structure Tracking**: The system maintains a parse state tracking whether it's inside a string, object, array, etc., and only permits tokens valid for the current position. - **Automatic Completion**: If the model would naturally stop generating, JSON mode ensures all open braces, brackets, and strings are properly closed. **API Implementations** - **OpenAI**: Set `response_format: { type: "json_object" }` — model output is guaranteed valid JSON. With **Structured Outputs**, you can provide a JSON Schema and the output will conform to that exact schema. - **Anthropic**: Uses tool/function calling with JSON schemas for structured outputs. - **Open-Source**: Libraries like **Outlines**, **Guidance**, and **llama.cpp** with GBNF grammars provide JSON mode for local models. **JSON Mode vs. JSON Schema Enforcement** - **Basic JSON Mode**: Guarantees valid JSON but not a specific structure. The model might return `{"answer": 42}` or `{"result": "yes", "confidence": 0.9}` — valid JSON but unpredictable structure. - **Schema Enforcement**: Guarantees the output matches a specific **JSON Schema** with exact field names, types, and required properties. Much more useful for production applications. **Best Practices** - Always include "respond in JSON" in your prompt even when using JSON mode — the instruction helps the model produce better-structured content. - Define a clear schema and provide an example in the prompt for consistent field names. - Use JSON mode with **function/tool calling** for the most reliable structured outputs. JSON mode has become a critical capability for building **reliable AI applications** where LLM outputs feed into downstream code that expects structured data.