JSON mode is a feature offered by LLM APIs that forces the model to generate output that is guaranteed to be valid JSON. This eliminates the common problem of LLMs producing malformed, truncated, or incorrectly formatted JSON when you need structured data output.

How JSON Mode Works

• Token-Level Enforcement: At each generation step, the system only allows tokens that would result in valid JSON according to a JSON grammar parser. Invalid tokens are masked out before sampling.
• Structure Tracking: The system maintains a parse state tracking whether it's inside a string, object, array, etc., and only permits tokens valid for the current position.
• Automatic Completion: If the model would naturally stop generating, JSON mode ensures all open braces, brackets, and strings are properly closed.

API Implementations

• OpenAI: Set response_format: { type: "json_object" } — model output is guaranteed valid JSON. With Structured Outputs, you can provide a JSON Schema and the output will conform to that exact schema.
• Anthropic: Uses tool/function calling with JSON schemas for structured outputs.
• Open-Source: Libraries like Outlines, Guidance, and llama.cpp with GBNF grammars provide JSON mode for local models.

JSON Mode vs. JSON Schema Enforcement

• Basic JSON Mode: Guarantees valid JSON but not a specific structure. The model might return {"answer": 42} or {"result": "yes", "confidence": 0.9} — valid JSON but unpredictable structure.
• Schema Enforcement: Guarantees the output matches a specific JSON Schema with exact field names, types, and required properties. Much more useful for production applications.

Best Practices

• Always include "respond in JSON" in your prompt even when using JSON mode — the instruction helps the model produce better-structured content.
• Define a clear schema and provide an example in the prompt for consistent field names.
• Use JSON mode with function/tool calling for the most reliable structured outputs.

JSON mode has become a critical capability for building reliable AI applications where LLM outputs feed into downstream code that expects structured data.