LLM Response Schema vs Fine-Tuning for Clinical Extraction

Every major model platform now offers some way to force a language model to answer in JSON that matches a schema you supply. Google calls it a response schema, OpenAI calls it Structured Outputs, Anthropic ships structured outputs for Claude, and open-weight serving stacks such as vLLM and llama.cpp do the same job with grammars. For a team building clinical document extraction, the pitch is attractive: write the schema once, point a general model at a discharge summary, and never parse broken JSON again.

The catch is that these features solve one problem very well and leave a second one almost untouched. They control the shape of the answer. They do not control whether the medication in the medications field is the one the letter actually names. OpenAI's own launch note reported that, on its evals of complex JSON schema following, gpt-4o-2024-08-06 with Structured Outputs scored a perfect 100% against less than 40% for gpt-4-0613 (OpenAI, 6 August 2024). That is a schema-following result. It says nothing about extraction accuracy, and the post does not present it as such. Accuracy on your own documents is the question our approach starts from.

This guide is for the Head of Data Science or ML lead who has to choose between two methods: constraining a general model at inference time, or fine-tuning a small model on the schema itself. It compares what each method guarantees, where each guarantee stops, and how to measure the part neither method guarantees. It deliberately cedes two neighbouring arguments. Migrating off Google's deprecated clinical API is covered in our Healthcare NL API migration guide, and the full breakdown of Mira-Q2's published test sets lives in the 782-document evaluation. Here we hold the method comparison.

This guide is shipped by the team behind Dilr Mira, a family of small private language models that read clinical documents on hardware you control and return schema-valid JSON for a person to review. Or see DATS, the five-stage consulting system that takes an extraction pilot into production.

What does an LLM response schema actually guarantee in clinical extraction?

An LLM response schema guarantees syntax and structure: the model's reply parses as JSON, carries the required keys and uses the declared types. In clinical extraction that removes a whole class of pipeline failures, such as truncated objects, stray prose around the JSON and invented field names. It does not guarantee that any value is correct, present in the source document, or clinically sensible.

The mechanism explains the limit. A model normally samples its next token from the whole vocabulary, which is why it can emit a closing brace too early or wrap the answer in a friendly sentence. OpenAI describes the fix plainly in its launch post: to force valid outputs, the model is constrained to only the tokens that would be valid according to the supplied schema. OpenAI's post notes that implementations differ, compiling the schema into a context-free grammar or a finite state machine, but the principle is the same. The constraint acts on which tokens are allowed next. It never consults the source document, which is why source grounding has to be checked separately.

Google's documentation draws the line between hint and guarantee carefully. The Vertex AI structured output page says that setting the response type to JSON without a schema acts only as a strong hint and does not ensure 100% valid JSON, and that both a response_schema and a JSON response type are required for that guarantee. Anthropic's structured outputs documentation makes the same promise for Claude through constrained decoding. The promise is real, and for a pipeline engineer it is worth having. It is just narrower than the word "structured" suggests to a clinical buyer, which is why the DATS methodology separates the integration test from the accuracy test before any pilot starts.

Where does a response schema stop helping?

A response schema stops helping at three boundaries: the parts of JSON Schema the platform does not enforce, the schemas too complex for it to compile, and every question about meaning. Google and OpenAI each document a supported subset of JSON Schema, Anthropic documents limitations, and Google says its API strips and ignores some keywords, so a schema that looks strict on paper can be looser in production.

The unenforced keywords are the first trap for clinical teams, because they are exactly the ones people reach for to express data quality rules. Google's Gemini API structured output guide lists minLength, maxLength, minProperties, maxProperties, uniqueItems, contains, minContains and maxContains as not enforced by its constrained decoding grammar engine, and says the API strips and ignores them. A rule that a medication list must not be empty, or that diagnosis entries must be unique, can therefore vanish between your Pydantic model and the model's decoder. OpenAI's Structured Outputs guide likewise states that it supports a subset of the JSON Schema language. Read the supported list for whichever platform you use, and re-implement anything missing as a post-hoc validator. Our AI operating model work treats that validator as an owned component, not a script someone wrote during the pilot.

Complexity is the second boundary. The Vertex AI page warns that a complex schema can return an InvalidArgument 400 error, and names the usual causes: long property names, long array length limits, enums with many values and objects with many optional properties. A clinical schema that enumerates every specialty, every document type and every unit of measure hits all four. OpenAI's launch post adds a latency cost: typical schemas take under 10 seconds to process on the first request, and complex ones up to a minute, before the processed schema is cached.

The third boundary is the one that matters most, and the vendors say so themselves. The Gemini API guide tells developers that while the output is syntactically correct JSON, they should always validate values in the application and handle schema-compliant but semantically incorrect outputs. OpenAI's documentation is shorter still:

"Structured Outputs can still contain mistakes." -- OpenAI API documentation, Structured Outputs guide

Two smaller edges are worth knowing before a pilot. OpenAI notes that a model can decline to follow the schema when it refuses an unsafe request, returning a refusal value instead, so the pipeline needs a branch for that case. Vertex AI warns that examples in the prompt must present properties in the same order as the schema, because a mismatch can confuse the model and lead to incorrect or malformed output. Neither is a reason to avoid response schemas. Both are reasons to test the whole path, which is what our evaluation harness engineering post describes in depth.

How does constrained decoding work on open-weight models?

Constrained decoding on open-weight models works the same way as on hosted APIs, except the team runs the grammar engine itself. vLLM exposes structured outputs through backends such as XGrammar and Guidance, and llama.cpp uses GBNF grammars, which it can generate from a JSON schema. For clinical teams this matters, because the document and the decoder both stay inside their own network.

The vLLM structured outputs documentation is a useful reminder that this tooling is still moving. It names XGrammar and Guidance as its backends, marks the older LM Format Enforcer backend as deprecated from version 0.33.0 and due for removal in 0.35.0, and records that the older guided_json style fields were removed in version 0.12.0 in favour of a structured_outputs parameter. A team that pins a serving stack for a clinical safety case needs to re-test whenever it moves version, not just when the model changes.

The llama.cpp grammar guide describes GBNF as a format for defining formal grammars to constrain model outputs, names forcing valid JSON as one use, and explains that llama.cpp can convert JSON schemas to grammars ahead of time or at each request. That route suits a fully air-gapped deployment, where the model, the grammar and the documents all sit on one controlled machine. It is the same deployment shape our clinical document extraction guide argues for on data protection grounds.

The engines are not interchangeable, and the field now has a benchmark that treats them as products to be tested. JSONSchemaBench (Geng and colleagues, January 2025) pairs 10K real-world JSON schemas with the official JSON Schema Test Suite and evaluates six constrained decoding frameworks: Guidance, Outlines, Llamacpp, XGrammar, OpenAI and Gemini. Its stated dimensions are efficiency, coverage of different constraint types, and the quality of the generated output. The practical lesson for a buyer is simple: coverage of your particular schema is something to test, not assume.

There is one more caution, and it should be read narrowly. A study titled Let Me Speak Freely? (Tam and colleagues, August 2024) reported a significant decline in reasoning ability when models were restricted to structured formats, with stricter constraints generally causing greater degradation on reasoning tasks. That finding is about reasoning benchmarks, not document extraction, and it should not be stretched into a claim that schemas make extraction worse. It is a reason to measure accuracy with and without the constraint on your own documents, rather than to assume the constraint is free.

What does fine-tuning a small model buy that a schema cannot?

Fine-tuning a small model buys behaviour rather than syntax. A model trained on thousands of documents paired with correct outputs learns what belongs in each field, when to leave a field empty and how to follow a schema it was trained to expect. A response schema can only forbid malformed tokens; it cannot teach a general model where a discharge summary keeps its medication changes.

Dilr Mira's published lineage shows both what that training changes and what it does not. According to the Mira-Q2 model card, the untrained base model, Qwen2.5-3B, invents its own schema and scores 0% validity against the required one. Mira-Q1, trained on 3,438 examples, reached 98% on a 50-example evaluation. Mira-Q2, trained on 8,400 examples, reached 100% validity on a 200-example held-out evaluation. The two versions were tested on different numbers of documents, so the steps are indicative rather than a controlled comparison.

The training data is where the schema behaviour comes from. The card lists 6,400 gold-by-construction examples plus 2,000 schema variants, built by renaming fields, dropping fields and using minimal schemas. The live Dilr Mira product page says that because Mira-Q2 was trained on those variants, it follows a schema passed in its prompt; out of the box it returns ten required fields, from document_type and patient through to allergies and an extraction_notes field for the human reviewer. DILR.AI has published no field-level accuracy for schemas other than the one in its test set. Pointing Mira at any schema without setup is listed for Mira-3, which is in development and not available.

The honest limit is that fine-tuning does not, on its own, guarantee shape. The same card reports JSON validity of 94.7% on 150 real physician documents that match the schema, and 85.8% on 282 real physician documents across 39 specialties. The live page explains why: Mira-Q2 was trained on synthetic data, learned it well, and struggles with document types it never saw, such as operative notes and physical examinations. A model under a response schema returns schema-valid JSON unless it refuses or the schema is rejected outright. A fine-tuned model without a grammar can return malformed output. That is the trade the next diagram sets out.

Can you fine-tune and constrain the same model?

You can fine-tune a model and then constrain its decoding, and on open-weight stacks the two layers are independent: the grammar engine in vLLM or llama.cpp does not care how the weights were trained. Hosted platforms are less settled. Google's Vertex AI documentation warns that using structured output on tuned Gemini models can result in decreased model quality, so combining the two there needs its own measurement.

That warning, in the Vertex AI structured output page, is the most useful single sentence in this debate, because it comes from a vendor that sells both features. It confirms that the two methods interact, and that adding a constraint to a tuned model is a change to be tested rather than a free upgrade. The open-weight case is different in mechanism, since the serving engine masks the tokens the grammar forbids before each token is sampled, but the measurement duty is the same. If the constraint pushes a fine-tuned model away from the tokens it would have chosen, accuracy can move in either direction, and only your own labelled documents will show which.

For Mira-Q2 specifically, the published model card reports the fine-tuned model's own validity and does not describe results under a decoding grammar. DILR.AI has published no figures for Mira-Q2 served with vLLM structured outputs or a llama.cpp grammar, so we make no claim about how the two combine. What can be said from the published numbers is narrower: on documents like the ones it was trained on, the fine-tuned model already produced valid JSON on every held-out document, and the remaining validity gap sits on document types it was not trained on, which is a training data problem first, and the kind of question a DATS engagement settles with your own documents.

How should a team measure correctness, not just shape?

A team should measure correctness by scoring each extracted field against a labelled gold copy of the same document, reporting the result per field and per document type, and keeping JSON validity as a separate integration metric. Correctness needs labelled documents; validity does not. Treating the cheaper metric as the acceptance test is how an extraction pilot passes review and still ships wrong values.

Dilr Mira's own scorecard is a worked example of keeping the two apart. The Mira-Q2 card reports field-F1 of 1.000 on the 200-document held-out gold set, which is synthetic and drawn from the same distribution as the training data, with a 95% interval of 0.999 to 1.0, and reports field-F1 as "n/a (unlabeled)" on the three other test sets, because those documents have no gold labels to score against. Read plainly, the published card measures validity on all four sets and field-level accuracy only on that synthetic held-out set; no field-level accuracy is yet published for real documents. That is not a weakness to hide. It is the normal state of any clinical extraction project before a design partner supplies labelled documents, and the card says that retraining on real documents is what closes the gap.

The measurement plan that follows from this has four parts, and each one is cheap compared with a failed rollout:

• Validity rate. The share of outputs that parse and pass the full schema, including the rules your decoder ignores. Under a response schema this should be at or near 100%, so it mainly tests your pipeline.
• Field-level accuracy. Precision, recall or F1 per field, against a gold set labelled by clinicians or coders from your own document mix. Report medications and allergies separately from administrative fields; an error in one is not an error in the other.
• Grounding rate. The share of extracted values that can be traced to a span in the source text. A value that appears nowhere in the document is a fabrication, whatever its format.
• Abstention behaviour. How often the model leaves a field empty when the document is silent, and how often it fills one anyway. A schema that makes every field required pushes a constrained model towards filling it.

That last point deserves emphasis because it is where schema design and correctness collide. If a schema marks a field as required and the document does not contain it, a constrained model must still produce something of the right type. Design required fields to allow an explicit empty value, and score how often the model uses it correctly. Our evaluation scorecard for Mira-Q2 shows the published sets in detail; the point here is the method, which applies to any model.

The methods side by side look like this:

The same diagnostic logic underpins our AI execution office, where a senior team owns the scorecard, the gold set and the release decision after the pilot ends.

Which approach fits which clinical extraction job?

The right approach depends on three facts about the job: whether documents may leave your network, whether you have labelled examples to train and test on, and how stable the schema is. A hosted response schema suits cloud-permitted, schema-light work; a fine-tuned small model suits private, high-volume, stable-schema work; most regulated teams end up needing both layers.

Start with where the documents may go. If your information governance allows documents to be processed by a cloud model provider under contract, a hosted model with a response schema is the fastest route to a working pipeline, and the schema guarantee removes most integration work. If the documents must stay on infrastructure you control, which is common for special category health data, the choice narrows to open-weight models, either general ones with a grammar or small fine-tuned ones. In an NHS setting, the clinical safety standards DCB0129 and DCB0160 bind the manufacturer and the deploying organisation respectively whichever method is chosen, so the hazard log needs to name both the shape controls and the accuracy controls.

Next, ask what labelled data exists. Fine-tuning needs examples paired with correct outputs, and correctness testing needs a gold set whichever method you choose. A team with no labelled documents is not ready to judge any method on accuracy, only on validity, and that is a conversation worth having with our team before a vendor demo sets the expectation. It is also the gap our AI placement diagnostic is designed to surface early, before a pilot with a demo and a parser but no gold set is mistaken for a tested system.

Finally, look at schema stability and document mix. A schema that changes monthly favours a general model with a response schema, because each change is a configuration edit rather than a training run. A stable schema over a narrow, high-volume document mix, such as lab reports or discharge summaries, favours a small model trained on that mix, which is the kind of document Mira-Q2's live page says it is strongest on. A wide, unpredictable mix of narrative letters is hard for both, and is where human review carries the most weight.

What is the best way to get reliable clinical JSON in 2026?

The best way to get reliable clinical JSON in 2026 is to treat shape and correctness as separate problems with separate controls. Use a response schema or grammar so the shape never breaks, use training or careful prompting so the content is right, and measure field-level accuracy against labelled documents before anyone relies on the output. No single feature does all three.

Judged on criteria rather than brand, the options separate cleanly. On integration effort and schema guarantees, hosted platforms from Google, OpenAI and Anthropic win today: their constrained decoding is managed and documented, so the team does not run a grammar engine itself, and for documents that may be processed in the cloud they are the sensible first choice. On data residency, open-weight models served with vLLM or llama.cpp win, because nothing leaves the building. On behaviour for a fixed schema and document mix, a small fine-tuned model is the stronger fit, and Dilr Mira was built for that case, by a UK team that publishes its gaps alongside its scores.

It is worth conceding where a fine-tuned small model is the wrong answer. If your schema is still being designed, if your document mix is broad narrative correspondence, or if you have no labelled examples and no prospect of getting them, a general model with a response schema will get you to a measurable pilot faster. The OpenBioLLM comparison makes a similar point about large general biomedical models: the right choice depends on the job, not the model family.

Does a response schema stop a model inventing clinical values?

A response schema does not stop a model inventing clinical values. It constrains which tokens can form the JSON structure, so the output always parses, but any string or number the schema allows is still chosen by the model. An invented dose in the correct format passes the schema exactly as easily as a real one, which is why grounding checks and human review remain necessary.

Does Dilr Mira use constrained decoding?

The published Mira-Q2 model card does not describe constrained decoding. Dilr Mira's validity figures are for the fine-tuned model's own output: 100% on held-out documents like its training data and 85.8% on general real physician notes. DILR.AI says Mira-Q2 follows a schema passed in its prompt, having been trained on 2,000 schema variants, and every output is a draft for human review.

The full figures, with their intervals, are on the Mira-Q2 model card and summarised in our plain-English Dilr Mira explainer.

Is JSON validity a good acceptance metric for an extraction pilot?

JSON validity is a good integration metric but a poor acceptance metric for an extraction pilot. It proves the pipeline can parse what the model returns, and under a response schema it should sit near 100% by design. Acceptance should rest on field-level accuracy against labelled documents, grounding to the source text, and correct handling of fields the document does not contain.

Want to see this on your own documents? Read about Dilr Mira, see how an AI execution office runs the release decision, see how an AI operating model assigns ownership of the gold set, or read our approach to placing AI inside regulated systems. More posts in this series sit under AI strategy.

Written by the Dilr.ai engineering team, practitioners who ship enterprise AI in production. Follow us on LinkedIn for shipping notes, or subscribe via the RSS feed.