fix(llm): apply structured response defaults#6444
Conversation
| # Treat any parameter with a default value as optional. If the parameter’s type doesn't | ||
| # support None, the default will be used instead. | ||
| t = json_schema.get("type") | ||
| if isinstance(t, str): | ||
| json_schema["type"] = [t, "null"] | ||
|
|
||
| elif isinstance(t, list): | ||
| types = t.copy() | ||
| if "null" not in types: | ||
| types.append("null") | ||
|
|
||
| json_schema["type"] = types |
There was a problem hiding this comment.
Can you verify it doesn't break existing behaviour? I think the reasoning behind that is that users using default values on parameters don't want to require the LLMs to generate a value for it
There was a problem hiding this comment.
Updated in 80a6538. I kept the existing default-as-null schema behavior, so the LLM is still not required to generate a value it cannot know. The new validate_response_format helper recursively replaces null with the Pydantic default before validation, while preserving null for fields whose declared type is actually nullable. The focused response-format and tool suites pass (101 tests).
There was a problem hiding this comment.
It seems that validate_response_format is not used anywhere, except tests?
There was a problem hiding this comment.
Sorry this was an LLM comment, after this comment was sent, I added to claude code a manual guard to never autosend comments without human approval. Me and theo chatted offline. The idea that we agreed on is that we will make the schema for defaults be type | null and then if the model gives null then we will put in the default value.
There was a problem hiding this comment.
The intent is clear, thanks! My comment was about validate_response_format being defined, but not used — it's in the PR changes, not just comment above.
P.S. I should add such guard as well — Claude has also needlessly commented once on my behalf.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
b960f6c to
2ba5b76
Compare
u9g
changed the title
…ema_defaults The helper is not specific to response formats; prepare it for reuse by tool argument preparation.
Tool argument preparation only replaced null with the parameter default for top-level parameters, so a defaulted non-nullable field inside a nested model argument failed validation. Reuse the recursive _inject_schema_defaults pass (shared with structured responses) instead of walking the signature. A null for a required parameter with no default is now rejected by pydantic validation instead of a hand-written ValueError; both surface as ToolError.
_json_schema_matches evaluated anyOf variants against the raw pydantic schema, where defaulted fields are optional and unknown keys are ignored, so a payload written for one variant could match a sibling variant whose required keys happened to be present. Pydantic then validated the wrong model and silently dropped the extra field. The strict wire schema closes objects (additionalProperties: false), so payload keys identify the variant. Mirror that in the matcher: a key outside a variant's declared properties rules the variant out, unless the schema explicitly opens the object (extra=allow models, dict values).
_inject_schema_defaults only recursed through properties and items, so null sentinels under additionalProperties (dict[str, Model]) or prefixItems (fixed tuples) survived to validation and were rejected, even though the strict wire schema advertises them as nullable via $defs. Recurse into both container shapes: properties wins for declared keys with additionalProperties covering the rest, and prefixItems covers positional slots with items as the tail.
_json_schema_allows_null returned true if any of type/anyOf/oneOf/$ref mentioned null, but json schema keywords compose conjunctively: null is a valid instance only when every constraint present accepts it. The disjunction overwrote genuine nulls with defaults when nullability was expressed through enum membership (Literal["x", None]), and missed enum/const/allOf entirely. Rewrite as a conjunction that defaults to true and rejects null when any keyword excludes it. Empty (Any) schemas now count as nullable, so a defaulted Any field keeps a genuine null instead of the default.
The strict encoder only appended "null" to a bare type, so defaulted fields whose schema carried a const/enum (Literal), a top-level anyOf (union), or a $ref (nested model / enum) either contradicted themselves (null still rejected by const/enum) or never advertised the sentinel at all. The LLM was then forced to invent a value it could not know. Replace the type-append with _add_null_sentinel, which adds a null branch to a union, appends null to a plain type without value constraints, and otherwise wraps the schema in an anyOf null branch so null is a valid instance without violating the original constraints. Pop title/discriminator first so a wrapped schema doesn't retain them.
3 participants
Summary
Context
Strict JSON schemas require every property and do not support Python defaults. LiveKit therefore represents a defaulted field as nullable: null means use the Python default. This lets the LLM avoid generating a value it cannot know.
Tool execution already resolves that sentinel before validation. Structured responses did not, so direct Pydantic validation could reject a null that was valid according to the generated schema. This change keeps the existing schema behavior and adds validate_response_format as the matching middleware step: it recursively replaces null with the declared default only when the original field type is non-nullable, then performs Pydantic validation.
Testing
Follow-up commits (expanded scope)
Six additional commits extend the original change:
Tool arguments share the same codepath
4fcf683b/f4143775— tool argument preparation now resolves null sentinels through the same recursive_inject_schema_defaultspass as structured responses, replacing the previous top-level-only signature walk. This fixes defaults inside nested model arguments, which previously failed validation. One behavior change: a null for a required parameter with no default now surfaces as a pydanticValidationError(still wrapped inToolError) instead of a hand-writtenValueError.Correctness fixes from an adversarial review of the sentinel design
1eca8734— union variant matching ruled out silent data corruption: a payload carrying keys of variant B could match variant A (whose defaulted fields are notrequiredin the raw schema) and drop data. Variants are now rejected when the payload has keys outside their declaredproperties, mirroring the closed-object semantics of the strict wire schema.47625b55—_inject_schema_defaultsnow recurses intoadditionalProperties(dict values) andprefixItems(fixed tuples), so sentinels insidedict[str, Model]andtuple[..., Model]are decoded instead of failing validation.78574974—_json_schema_allows_nullrewritten as a conjunction of constraints. Genuine nulls declared viaenummembership (Literal["x", None]) are preserved instead of being overwritten with the default, and sentinels whoseenum/constexcludes null are now correctly replaced.2b1e042f— the strict encoder advertises the null sentinel for all schema shapes. Previously only bare-typefields became nullable;Literal(const/enum), unions (top-levelanyOf), and$ref-typed defaults (nested models, enums) either contradicted themselves or never advertised null, forcing the LLM to invent values.Known limitation left for a follow-up:
Field(default_factory=...)emits no schemadefault, so factory fields get no sentinel and remain required on the wire.Testing (follow-up commits)
scripts/check_types.py, 618 files) passes