LLM Structured Outputs: JSON Schema, Tool Calls, and Validators

A language model left to its own devices will happily narrate its answer, apologize for the delay, and bury the one JSON field your application needs somewhere in paragraph three. Structured outputs exist to make that impossible: the API constrains the model so that what comes back is valid JSON matching a schema you declared — or a tool call with typed arguments — instead of prose you have to scrape. This guide covers the four mechanisms that produce structured output from modern LLM APIs — JSON mode, JSON Schema constrained decoding, tool/function calling, and constrained grammar generation — plus the validation and retry layers that turn "mostly correct JSON" into "always correct data." If your goal is a full extraction pipeline (documents in, records out), this is the mechanism guide; our data extraction guide covers the end-to-end pipeline that sits on top of these primitives.

Why Structured Outputs Matter

Every downstream consumer of an LLM — a database insert, a UI state update, an API call to a third party, a billing system — expects data in a shape it understands. Unstructured prose forces you into a fragile parse-then-guess loop: regexes, string slicing, and hope. The costs of that loop are measurable and ugly:

• Silent corruption — a mis-parsed field writes wrong values into the database without any error, and wrong data is worse than no data.
• Latency tax — a parse failure means a regeneration round-trip, doubling effective response time and token cost.
• Engineering tax — every prompt change risks breaking your parser, so the prompt freezes and the product stalls.
• Security surface — unvalidated model output flowing into SQL, HTML, or shell commands is an injection vector; validated, schema-checked data is not.

Structured outputs move the contract from prose to types. The model still does the reasoning; the API and your validator guarantee the shape. The result is an integration you can trust without reading every response.

The Four Mechanisms, Compared

The choice is not "one mechanism forever" — production systems commonly use tool calling for agent loops, JSON Schema for data contracts, and a validator on top of whichever one is active. Each section below shows working code for one mechanism.

JSON Mode: The Floor, Not the Ceiling

JSON mode tells the model "respond with valid JSON" and the API guarantees the output parses. It is the oldest and simplest mechanism, available on virtually every OpenAI-compatible endpoint. The catch: JSON mode guarantees valid JSON, not correct JSON — the model can return the right fields with the wrong types, or invent extra fields, or nest things differently than you expected.

# JSON mode — guaranteed parseable, not guaranteed correct
r = requests.post("https://api.dr-ai.top/v1/chat/completions",
    headers={"Authorization": "Bearer " + KEY},
    json={
        "model": "gpt-5-mini",
        "response_format": {"type": "json_object"},
        "messages": [
            {"role": "system", "content": "Extract the summary, sentiment, and category as JSON."},
            {"role": "user", "content": "Analyze: 'Support took 4 days to reply and charged me twice.'"}
        ],
    })
data = json.loads(r.json()["choices"][0]["message"]["content"])
print(data["sentiment"])   # may be "negative", "Negative", "neg", or missing — no guarantee

Two operational rules for JSON mode. First, the word JSON must appear in your prompt (most APIs reject or ignore the mode otherwise) and the instruction should describe the shape, not just demand JSON. Second, always run the result through a validator — JSON mode without validation is just a nicer failure mode. If your integration can tolerate field-level drift, JSON mode is the cheapest mechanism available; if it cannot, move to JSON Schema below.

JSON Schema Constrained Output: The Production Contract

JSON Schema constrained output (sometimes called structured outputs or response_format: json_schema) lets you declare the exact shape — field names, types, required fields, enums, nested objects, arrays — and the API guarantees the model output validates against it. This is the mechanism to use for anything that touches a database or a typed API boundary.

# JSON Schema constrained output — the shape is enforced by the API
schema = {
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
        "category": {"type": "string", "enum": ["billing", "support", "product", "other"]},
        "action_items": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["summary", "sentiment", "category", "action_items"],
    "additionalProperties": False,
}

r = requests.post("https://api.dr-ai.top/v1/chat/completions",
    headers={"Authorization": "Bearer " + KEY},
    json={
        "model": "gpt-5-mini",
        "response_format": {"type": "json_schema", "json_schema": {
            "name": "analysis",
            "strict": True,
            "schema": schema,
        }},
        "messages": [{"role": "user", "content": "Analyze this ticket: " + ticket_text}],
    })
result = json.loads(r.json()["choices"][0]["message"]["content"])
# result now has exactly the four declared fields, with enum-constrained values

Three design rules make schema-constrained output reliable in practice:

• Keep schemas shallow and explicit — strict mode rejects extra properties and enforces required lists, but deeply nested or recursive schemas burn tokens and occasionally confuse weaker models. Flat objects with enums outperform deep trees.
• Let the enum do the reasoning — constraining sentiment to an enum forces the classification decision into the model where it belongs, instead of leaving it to string matching downstream.
• Still validate on your side — the API guarantees JSON Schema conformance, but your runtime should still parse and type-check before using the data. Providers occasionally ship schema-handling bugs; a jsonschema or Pydantic check costs microseconds and catches them all.

Tool Calling: Structured Output as an Action

Tool (function) calling wraps structured output in an action envelope: the model decides to call one of your declared functions with typed arguments. It is the backbone of agent loops and the most flexible structured-output mechanism, because the "schema" is your function signature and the output is executable.

# Tool calling — the model emits a typed call, your code executes it
tools = [{
    "type": "function",
    "function": {
        "name": "submit_order",
        "description": "Submit a customer order for fulfillment",
        "parameters": {
            "type": "object",
            "properties": {
                "customer_id": {"type": "string"},
                "items": {"type": "array", "items": {
                    "type": "object",
                    "properties": {"sku": {"type": "string"}, "qty": {"type": "integer"}},
                    "required": ["sku", "qty"]}},
            },
            "required": ["customer_id", "items"],
        },
    },
}]

r = requests.post("https://api.dr-ai.top/v1/chat/completions",
    headers={"Authorization": "Bearer " + KEY},
    json={"model": "gpt-5-mini", "tools": tools,
          "tool_choice": "auto",   # or {"type": "function", "function": {"name": "submit_order"}} to force
          "messages": [{"role": "user", "content": "Place an order for 2 units of SKU-A1 for customer 8821."}]})

msg = r.json()["choices"][0]["message"]
if msg.get("tool_calls"):
    call = msg["tool_calls"][0]["function"]
    args = json.loads(call["arguments"])   # typed arguments, guaranteed to match the schema
    print(call["name"], args)

Three tool-calling patterns separate reliable agents from flaky demos:

• Force the call when the step is mandatory — tool_choice with an explicit function name removes the model's option to narrate instead of acting. Use it for the final step of a workflow where an action is required.
• Validate arguments with the same schema you sent — parse arguments and validate before executing. Models occasionally emit near-valid arguments (a float where an integer belongs, a missing field the API should have enforced); your validator is the last line of defense before side effects.
• Handle the multi-call case — modern models can emit several tool calls in one response. Loop over tool_calls, execute in parallel where safe, and feed every result back into the conversation for the next turn.

The full agent-loop pattern — tool calls, results, continuation — is covered in depth in our function calling guide.

Constrained Decoding: Token-Level Guarantees

JSON mode, JSON Schema, and tool calling are API-level guarantees: the provider restricts sampling so the output conforms. Constrained decoding does the same thing at the token level on models you run yourself, using libraries like Outlines, Guidance, or xgrammar. The sampler only proposes tokens that keep the output valid against a grammar — JSON Schema, a regex, or a full CFG. There is no "mostly valid": validity is enforced one token at a time.

# Constrained decoding with Outlines (local models)
import outlines

model = outlines.models.transformers("Qwen/Qwen3-8B")
schema = {
    "type": "object",
    "properties": {"company": {"type": "string"}, "revenue": {"type": "number"}},
    "required": ["company", "revenue"],
}
generator = outlines.generate.json(model, schema)
result = generator("Extract the company and revenue from: 'Acme Corp grew to $4.2M in 2026.'")
print(result)   # {"company": "Acme Corp", "revenue": 4.2} — always

Constrained decoding is the strongest guarantee available, and it is the right tool when validity is a hard requirement and the model runs under your control. The costs are real: constrained sampling is slower than free sampling (the sampler must re-check the grammar state on every step), the libraries are local-first, and hosted APIs generally do not expose grammar hooks — which is why most teams use it for their self-hosted models and JSON Schema mode for hosted ones.

Validating Outputs with Pydantic

Every mechanism above can still hand you something you did not intend — a provider schema bug, a tool call with borderline arguments, a JSON mode response with the right shape and the wrong content. The validation layer is where structured output becomes trustworthy data. Pydantic is the standard tool in Python: declare a model, parse the LLM output, and get typed objects or a clear error you can act on.

# Pydantic validation — typed data or a retryable error
from typing import Literal
from pydantic import BaseModel, Field, ValidationError

class Analysis(BaseModel):
    summary: str = Field(min_length=3, max_length=2000)
    sentiment: Literal["positive", "neutral", "negative"]
    category: Literal["billing", "support", "product", "other"]
    action_items: list[str] = Field(default_factory=list, max_length=10)

def parse_analysis(raw_json: str) -> Analysis:
    obj = json.loads(raw_json)          # JSON mode: parse first
    return Analysis.model_validate(obj)  # then validate — raises on any mismatch

# Schema-constrained mode: parse the already-conforming JSON
result = Analysis.model_validate_json(completion.content)

Validation does more than catch errors — it defines the contract. The Pydantic model is the single source of truth shared by your prompt description, your JSON Schema (generate it with Analysis.model_json_schema()), and your runtime checks. When the model, the schema, and the validator all derive from one declaration, drift between them becomes impossible by construction.

Retry Strategies for Failed Validation

Even with constrained decoding, validation failures happen — especially with weaker models, long outputs, or hostile input. The retry layer decides whether a 2% failure rate costs you 2% of requests or 0.002%. The proven pattern is a bounded, escalating retry loop that feeds the error back to the model:

# Escalating retry: regenerate with the validation error as feedback
def generate_structured(client, messages, max_attempts=3):
    for attempt in range(max_attempts):
        r = client.chat.completions.create(
            model="gpt-5-mini",
            response_format={"type": "json_schema", "json_schema": SCHEMA},
            messages=messages,
        )
        raw = r.choices[0].message.content
        try:
            return Analysis.model_validate_json(raw)
        except ValidationError as e:
            # Feed the error back so the model can fix its own output
            messages = messages + [
                {"role": "assistant", "content": raw},
                {"role": "user", "content": f"Your previous response failed validation: {e}. "
                                            "Fix it and return only valid data."},
            ]
    raise RuntimeError("structured output failed after %d attempts" % max_attempts)

Three refinements make the loop production-ready. Budget it — cap attempts (2-3 is the sweet spot; each retry costs a full generation) and consider a cheaper fallback model for the retry, since most failures are trivial to fix. Log the failure mode — a persistent "missing required field" pattern at 5% of traffic is a prompt problem, not a model problem; retries will not fix it. Know when to stop — if the same request fails three times with feedback, route it to a queue for review or a more capable model rather than looping forever. The full retry-and-fallback taxonomy lives in our error handling guide.

Choosing Between the Mechanisms

Note the architecture that runs through every row: constraint at generation time, validation at consumption time. No mechanism removes the validator; the best mechanisms make validation a formality instead of a gamble.

Cost and Latency Considerations

Structured output is not free. JSON Schema and tool calling add tokens to every response — the schema is sent with the request, and tool calls wrap the answer in an envelope — typically 5-15% more output tokens on short responses. Constrained decoding adds sampling overhead on local models. The practical levers: prefer json_schema over tool calling when you do not need the action envelope (saves the wrapper tokens), keep schemas minimal (a 40-field schema costs more than a 12-field one on every call), and let retries use a cheaper model. Measured against the alternative — a parser rewrite every time the model drifts — the overhead is the best money an LLM integration spends. Run the numbers for your own workload with a per-request cost calculator before optimizing.

Summary

Structured outputs turn a language model from a narrator into a typed function: JSON mode guarantees parseable output, JSON Schema constrains it to your contract, tool calling makes it executable, and constrained decoding enforces validity at the token level. On top of whichever mechanism you choose, a Pydantic validator and a bounded, feedback-driven retry loop close the reliability gap. Start with JSON Schema plus validation for data contracts, graduate to tool calling for agent actions, and reserve constrained decoding for your self-hosted models. Every layer you add moves the same failure — malformed, wrong-shaped model output — further from your users and closer to a logged, retried, and eventually fixed edge case.