Structured Outputs
How it works
Step by step: 1) The application passes a JSON Schema describing the desired output structure. 2) The schema is compiled into a formal grammar (context-free grammar / finite-state machine); compilation is typically one-time and cached (in Claude, for 24 hours). 3) During autoregressive decoding, at each generation step a logit mask is computed: only tokens leading to a grammar-valid state are allowed, all others are set to zero probability. 4) The model samples the next token exclusively from the allowed set, so every partial output stays schema-conformant. 5) The finished output is deterministically parseable; SDK libraries additionally validate it against the original schema (including constraints not natively enforced by the grammar). The same mechanism applies to tool-call arguments (strict tool use), guaranteeing correct tool names and input fields.
Problem solved
Freely generated LLM text is often inconsistent with the expected format: fields are missing, types are wrong, out-of-vocabulary enum values appear, or the JSON is syntactically invalid. This forces expensive error-handling parsers, retries, and application-side validation. Structured Outputs removes this problem by guaranteeing the output satisfies the schema, which simplifies machine-to-machine integrations, tool calls, and agentic workflows.
Components
A declarative description of the desired output structure (types, required fields, enums, nesting).
Transforms the JSON Schema into a context-free grammar or finite-state machine plus an index over the model vocabulary.
Official
At each generation step applies a mask to the logits, zeroing tokens that would violate the grammar.
On the SDK side deserializes the output and validates it against the original schema, including constraints not enforced by the grammar.
Official
Implementation
When the input does not match the schema, the model may fabricate values to fill required fields.
Hitting max_tokens stops generation, yielding incomplete JSON despite the constraints.
Numeric constraints (minimum/maximum), string lengths, regex patterns, recursive schemas, and external $ref are typically dropped or rejected.
The first request with a new schema incurs extra grammar compilation cost.
Changing the output format can invalidate the prompt cache for a thread, raising cost.
An overly rigid schema can degrade answer quality by removing room for step-by-step reasoning.
Evolution
Open implementation of grammar-constrained sampling for open-weight models.
Willard and Louf formalize guided generation as finite-state machine transitions with an index over the vocabulary, enabling efficient constrained decoding.
On 6 August 2024 OpenAI ships Structured Outputs for gpt-4o-2024-08-06 with a strict: true option and a json_schema format, guaranteeing schema conformance.
Claude exposes JSON outputs (output_config.format) and strict tool use, with grammar compilation cached for 24 hours.
Hyperparameters (configurable axes)
Enables guaranteed schema validation (e.g. for tool calls).
Must be set to false to forbid fields outside the schema.
List of fields whose presence is guaranteed in the output.
Selection of the structured output mode (e.g. json_schema).
Computational complexity
Time complexity: O(1) na token (amortyzowane, z prekompilowanym indeksem FSM). Space complexity: O(|Q| · |V|).
Compute bottleneck
The first request incurs grammar compilation cost; every decoding step adds computing and applying the logit mask.
Execution paradigm
A decoding-time technique layered on an existing model; it does not change model weights or architecture.
No expert routing; the “conditional” aspect refers to schema- and prefix-dependent token masking.
Parallelism
Grammar compilation and index construction can be parallelized offline.
Hardware requirements
Constrained decoding is a layer over any inference engine; it works regardless of the accelerator type.
In practice it works alongside GPU LLM inference; efficient backends (e.g. XGrammar) minimize masking overhead.