Key takeaways
- The Structured Output Benchmark evaluated 21 models and found near-perfect schema compliance alongside value accuracy of 83.0% on text, 67.2% on images and 23.7% on audio. The envelope is correct and the contents are not.
- Constrained decoding enforces shape, not truth. A schema cannot state that an order ID exists, that a date is in the past, or that an amount matches the invoice.
- There are 3 places to enforce a tool schema: the prompt, the API's strict mode, and the decoder. Only the last two remove malformed output as a failure class, and none of them removes wrong values.
- JSONSchemaBench evaluated 6 constrained decoding frameworks against roughly 10,000 real-world schemas. Coverage of real schemas, not toy ones, is where the frameworks separate.
- Our recommendation: pair every strict schema with at least 1 semantic check the schema cannot express, and count those two failure types separately in telemetry.
A tool call that parses is not a tool call that is right. The Structured Output Benchmark put 21 frontier and open-weight models against that distinction and found near-perfect schema compliance sitting on top of 83.0% value accuracy for text inputs.
For anyone running agents, that gap is the whole story. The failure you engineered away is malformed JSON. The failure you still have is a well-formed object containing the wrong order number, and it arrives looking exactly like a success.
What does a structured output actually guarantee?
Shape. Nothing beyond shape.
OpenAI’s Structured Outputs guide is precise about this: strict mode ensures the model generates responses that adhere to the supplied JSON Schema. Adherence means the declared fields are present, the declared types hold, and nothing extra appears. It is a statement about the container.
A JSON Schema has no vocabulary for the things that actually go wrong in an agent. It cannot say that order_id refers to an order that exists. It cannot say that refund_amount is less than what the customer paid. It cannot say that scheduled_for is a date in the future rather than a plausible date in the past. Those are all type-valid and all wrong.
The same guide notes one genuinely useful exception to the guarantee: if the model declines the request, the response carries a programmatically detectable refusal field instead of the structure. That is the one case where a strict schema tells you something semantic, and it tells you only that the model chose not to answer.
| Enforcement layer | Removes malformed JSON | Removes schema mismatch | Removes wrong values |
|---|---|---|---|
| Prompt instruction only | No | No | No |
| JSON mode | Yes | No | No |
| Strict schema, constrained decoding | Yes | Yes | No |
| Semantic check after parsing | Not its job | Not its job | Partly |
The fourth row is the one most pipelines are missing, and it is the only row that touches the 17% of text-input values the Structured Output Benchmark found wrong.
How large is the gap between compliance and correctness?
Large, and it widens with input difficulty rather than with schema difficulty.
| Input modality | Value accuracy | Values wrong | Schema compliance |
|---|---|---|---|
| Text | 83.0% | 17.0% | Near-perfect |
| Images | 67.2% | 32.8% | Near-perfect |
| Audio | 23.7% | 76.3% | Near-perfect |
Read the last column, then the first. The structural metric is flat at near-perfect across a range where the semantic metric falls by 59.3 points. A dashboard built on schema validity would show three green lines over an extraction pipeline that is wrong three times in four.
The benchmark’s own explanation is that longer context makes extraction substantially harder, which is why audio, once transcribed into text, performs so much worse than text written to be read. That is the same degradation pattern that makes long agent runs lose accuracy as context accumulates, arriving here through a different door.
Why the metric you already have is the flattering one
Schema validity is cheap to measure. It is a boolean your parser already produces, it needs no ground truth, and it can be computed on every request in production.
Value accuracy needs a labelled answer. That is why most teams report the first and assume the second, and why the first looks so good.
Which layer should enforce the schema?
The decoder, when you can, because it converts a runtime failure into an impossibility.
Constrained decoding restricts token generation so that only continuations valid under the grammar can be emitted. A malformed argument stops being something you catch and becomes something the sampler cannot produce. That is a real change in failure profile, and it is the argument for paying the throughput cost.
The caveat is coverage of real schemas. JSONSchemaBench evaluated six frameworks, Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini, against roughly 10,000 real-world JSON schemas rather than hand-written examples. Real schemas carry conditionals, references and composition keywords that toy schemas do not, and coverage of those is where implementations diverge. A framework that silently reports success on a schema it did not fully enforce is worse than one that refuses the schema outright, because the first produces confident invalid output.
So the practical ordering for a production tool call is: use strict schema mode where the provider offers it, keep the schema inside the subset the provider documents as supported, and treat anything the schema cannot express as a separate check rather than adding it to the schema and hoping.
What should happen when a value is wrong rather than malformed?
It escalates differently, because it is a different failure class.
A malformed argument is a repair candidate. You have the validator’s error, you can inject it, and one more attempt often succeeds. A wrong-but-valid value has no validator output to inject, so there is nothing to repair with, and re-running the same prompt reproduces the same answer at a higher price.
Three things follow for the orchestrator:
- Count the two separately. One counter for schema failures and one for semantic-check failures. Merged into a single error rate they are indistinguishable, and the mix is the only thing that tells you where to spend.
- Put the semantic check where it can see live data. Existence, range and cross-field consistency all require a lookup the model did not do. A check that runs only against the model’s own output verifies nothing.
- Make the irreversible tools require the semantic check to pass. A refund with a valid schema and a wrong amount is still a refund. This is the same asymmetry that governs what an agent may attempt without a person watching.
The question for your next tool-call review
Not “what share of our tool calls parse”. Ask: for the last 1,000 tool calls, how many passed the schema and then failed a check the schema could not express? If nobody can answer, the pipeline is measuring the container and reporting it as the contents.
Frequently asked questions
Does OpenAI structured outputs guarantee correct tool arguments?
No. Strict mode guarantees the response conforms to the supplied JSON Schema. It makes no claim about whether the values inside that structure are correct, and benchmarks measuring value accuracy separately find a large gap.
What is the difference between JSON mode and structured outputs?
JSON mode guarantees only that the output parses as JSON. Structured outputs, or strict schema mode, additionally guarantees the output matches your declared schema: the right fields, the right types, nothing extra.
Is constrained decoding worth the performance cost?
Usually yes, because it converts a recurring runtime failure into a compile-time constraint. The cost is real but bounded, while malformed output costs a full retry with the failed attempt still sitting in context.
How do you validate tool arguments that a JSON Schema cannot express?
With a separate semantic check after parsing: existence lookups, range checks against live data, and cross-field consistency. Treat a failure here as a different class from a schema failure, because the two need different handling.
Method and sources
- The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models, arXiv:2604.25359. Source of the 21-model sample and of the value accuracy figures of 83.0% on text, 67.2% on images and 23.7% on audio.
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models, Geng and others, arXiv:2501.10868. Source of the roughly 10,000 real-world schemas and the six frameworks compared: Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini.
- OpenAI, Structured Outputs guide, consulted for what strict mode guarantees, for the programmatically detectable refusal field, and for the stated difference from JSON mode.
- We did not run these benchmarks ourselves and we report no success rate of our own. The recommendation to pair a strict schema with a semantic check is our editorial judgement, not a measured result.