feat(core): honor per-model provider overrides in the models.dev gateway#18928
Conversation
The models.dev gateway resolved every model of a provider through a single provider-level url + request shape (chosen by provider id). This ignores models.dev's per-model `provider` block, which lets a provider serve individual models over a different endpoint/shape/SDK (e.g. a model on the OpenAI Responses API while the provider default is chat-completions). Capture per-model `provider` overrides (api/shape/npm) in fetchProviders, prefer a per-model `api` in buildUrl, and route `shape: "responses"` models via the OpenAI Responses API in resolveLanguageModel. Providers without overrides are unchanged.
andrelandgraf
requested review from
CalebBarnes,
TylerBarnes,
abhiaiyer91 and
wardpeet
as code owners
🦋 Changeset detectedLatest commit: 07b1067 The changes in this PR will be included in the next version bump. This PR includes changesets to release 22 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@andrelandgraf is attempting to deploy a commit to the Mastra Team on Vercel. A member of the Team first needs to authorize it. |
WalkthroughAdds per-model provider override support to the models.dev gateway in ChangesPer-model provider overrides
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PR triageThis PR needs to fix an existing issue. Please link an issue in the PR description, for example with Applied label: PRs without a linked issue will automatically close after 1 day(s) with the label. Changed test gateChanged Test Gate is pending. The |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.changeset/models-dev-per-model-provider-overrides.md (1)
1-8: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winMissing code example for a new feature (potential_issue).
As per path instructions, "If the change is a breaking change or is adding a new feature, ensure that a short code example is provided. This code example should show the public API usage." This changeset documents a new feature (per-model routing overrides) but includes no code example.
Consider adding a short snippet showing the (unchanged) model string usage that now benefits from correct routing, e.g.:
const agent = new Agent({ model: 'neon/gpt-5-mini', // now routed via the OpenAI Responses API per models.dev override });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.changeset/models-dev-per-model-provider-overrides.md around lines 1 - 8, The changeset for the new models.dev per-model provider override feature is missing a public API usage example. Add a short code example to the document, near the description of the feature, showing the unchanged model string usage through Agent or the relevant public entry point so readers can see how a model like neon/gpt-5-mini is now routed via the per-model provider override. Refer to the models.dev gateway behavior and the resolveLanguageModel/buildUrl routing changes in the text so the example clearly matches the new feature.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.changeset/models-dev-per-model-provider-overrides.md:
- Around line 1-8: The changeset for the new models.dev per-model provider
override feature is missing a public API usage example. Add a short code example
to the document, near the description of the feature, showing the unchanged
model string usage through Agent or the relevant public entry point so readers
can see how a model like neon/gpt-5-mini is now routed via the per-model
provider override. Refer to the models.dev gateway behavior and the
resolveLanguageModel/buildUrl routing changes in the text so the example clearly
matches the new feature.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 49d417a7-6995-4b0c-9148-dba62eba8b37
📒 Files selected for processing (4)
.changeset/models-dev-per-model-provider-overrides.mdpackages/core/src/llm/model/gateways/base.tspackages/core/src/llm/model/gateways/models-dev.test.tspackages/core/src/llm/model/gateways/models-dev.ts
|
Closing this PR because it has had the needs-issue label for at least 1 days without a linked issue. Please open or link an issue first, then reopen this PR when it is ready. |
2 participants
Summary
The models.dev gateway currently resolves every model of a provider through a single provider-level URL + request shape, chosen by provider id (
resolveLanguageModelswitches onproviderId;buildUrluses the provider-levelurl). This ignores models.dev's per-modelproviderblock, which lets a provider serve individual models over a different endpoint / request shape / SDK than the provider default.A concrete case: a provider whose default is OpenAI-compatible chat-completions, but whose GPT‑5 models are served over the OpenAI Responses API (
shape: "responses") at a different path. Today Mastra sends those to chat-completions and they 404 / can't use Responses-only features (e.g. theimage_generationbuilt-in tool).This PR makes the models.dev gateway honor per-model overrides generically (not tied to any specific provider):
fetchProviderscaptures each model'sproviderblock (api,shape,npm) into a new optionalProviderConfig.modelOverridesmap. It serializes into the generated registry like any other field, so it flows through the existing sync pipeline.buildUrlprefers a per-modelapiover the provider-levelurl(explicit*_BASE_URLenv override still wins).resolveLanguageModelroutes a model withshape: "responses"viacreateOpenAI(...).responses(modelId)(for any provider), and prefers a per-modelnpmover the provider default in the openai-compatible dispatch.Providers without per-model overrides are completely unaffected.
Why
models.dev already supports per-model
provideroverrides in its schema (shape: "responses" | "completions",api,npm) and providers use them (e.g. Amazon Bedrock pointsopenai.gpt-5.5at a/openai/v1Responses endpoint). Mastra was the one consumer dropping them on the floor. This came up while modeling the Neon AI Gateway, which serves its GPT‑5 models over the OpenAI Responses route while the rest of the catalog uses a chat-completions endpoint — companion models.dev change for context: anomalyco/models.dev#3021.Test plan
packages/coreunit tests pass (vitest run src/llm/model/gateways/models-dev.test.ts— 40/40), including 3 new tests: fetchProviders capturesmodelOverrides; ashape: "responses"model resolves viacreateOpenAI().responses()on a chat-completions provider;buildUrlprefers the per-model endpoint.vitest run src/llm/model --exclude '**/*.e2e.test.ts'— 318/318, no type errors).@mastra/coreminor).Notes
provider-registry.jsonpicks up per-model overrides on the next registry sync (or live fetch); the field is optional, so existing cached registries and providers without overrides behave exactly as before.ELI5
This change lets each model choose its own “instructions” instead of forcing every model from the same provider to use the same setup. That means the gateway can now send some models to a different URL, use a different request style, or pick a different SDK when needed.
Summary
models.devsupport, includingapi,shape, andnpm.ProviderConfig.modelOverrideswhile fetching providers.shape: "responses"routes through the OpenAI Responses API for any provider.npmpackage over the provider default.@mastra/corerelease.