studio: deterministic backend tool-calling wiring test#6836
studio: deterministic backend tool-calling wiring test#6836danielhanchen wants to merge 4 commits into
Conversation
Add a deterministic, download-free test that exercises the shared tool-calling seam both inference backends use. InferenceBackend (transformers) and MLXInferenceBackend both render the prompt through apply_chat_template_for_generation(..., tools=...) and stream cumulative text into run_safetensors_tool_loop. The existing test_safetensors_tool_loop.py covers the parser and the loop state machine with fake generators but does not cover the backend's own tool-injection seam, so a regression that drops the tool schema before the tokenizer, or fails to feed a tool result back into generation, would slip through. The test drives that seam with fakes: a tokenizer that records the tools it is handed, a canned tool-call generation, and a stub executor. It asserts the full chain: tools reach the chat template, the loop parses the call, the tool is dispatched once with the parsed arguments, the result is fed back, generation re-enters, and the final answer streams after the tool result. It also guards that the raw tool-call markup never leaks to the client as content. The test imports no torch, unsloth, or mlx, so it runs in the portable Backend CI alongside the tool-call parser tests and stays sub-second. Follow-up to the parser test PRs #5620 and #5704.
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new deterministic backend-wiring test suite (test_safetensors_toolcall_wiring.py) to verify the safetensors and MLX tool-calling path. It utilizes a fake tokenizer and a stub executor to validate the agentic tool loop end-to-end without external dependencies. There are no review comments, so no feedback is provided.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40c46cd709
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| active_tools_seen.append(active_tools) | ||
| apply_chat_template_for_generation(tok, conversation, tools=active_tools) | ||
| text = next(turns) |
There was a problem hiding this comment.
This fake hard-codes the behavior the test is meant to protect: it passes active_tools directly into apply_chat_template_for_generation and then returns the next canned turn regardless of the conversation. A regression in the production wiring (for example InferenceOrchestrator.generate_chat_completion_with_tools or the backend generation method dropping tools before rendering, or failing to append the tool result before the second turn) would still pass because this single_turn does not use those production paths or require the tool result to be present in conversation. Make the fake depend on the received conversation/tools, or drive the actual backend wrapper seam, so the test fails for the wiring regressions it claims to cover.
Useful? React with 👍 / 👎.
Strengthen the wiring test so single_turn records each turn's conversation and the test asserts the tool result message is present in the conversation handed to the final generation turn. Event ordering alone did not catch a loop that stops appending the tool output before re-entering generation, because the fake generation ignores the conversation; this closes that gap.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd01929ded
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # then yield cumulative snapshots (each yield is all text so far, per the loop contract). | ||
| active_tools_seen.append(active_tools) | ||
| conversations_seen.append([dict(m) for m in conversation]) | ||
| apply_chat_template_for_generation(tok, conversation, tools = active_tools) |
There was a problem hiding this comment.
When the regression is in the production _single_turn wiring (for example, InferenceOrchestrator.generate_chat_completion_with_tools or InferenceBackend.generate_chat_completion_with_tools stops passing tools = turn_tools), this test would still pass because the local fake performs the apply_chat_template_for_generation(..., tools = active_tools) call itself. That means the new test does not actually guard the backend seam it was added to cover; consider driving the real backend/orchestrator method with a stubbed generation backend/tokenizer instead of reimplementing the forwarding in the test.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd01929ded
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # then yield cumulative snapshots (each yield is all text so far, per the loop contract). | ||
| active_tools_seen.append(active_tools) | ||
| conversations_seen.append([dict(m) for m in conversation]) | ||
| apply_chat_template_for_generation(tok, conversation, tools = active_tools) |
There was a problem hiding this comment.
In the production transformers path, the forwarding being guarded happens in InferenceBackend.generate_chat_completion_with_tools → _generate_chat_response_inner (studio/backend/core/inference/inference.py:786-805, then :970-974), and MLX has its own helper calls (mlx_inference.py:360-364/:470-474). This fake bypasses all of those call sites and hands active_tools to apply_chat_template_for_generation itself, so dropping tools=turn_tools or tools=tools in the real backend would still leave this test green. To make this a backend-wiring test, patch/call the backend method and observe the tokenizer from there.
Useful? React with 👍 / 👎.
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 804bfd4377
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # then yield cumulative snapshots (each yield is all text so far, per the loop contract). | ||
| active_tools_seen.append(active_tools) | ||
| conversations_seen.append([dict(m) for m in conversation]) | ||
| apply_chat_template_for_generation(tok, conversation, tools = active_tools) |
There was a problem hiding this comment.
Because this fake single_turn itself invokes apply_chat_template_for_generation(..., tools=active_tools), the test still passes if the real safetensors backend stops forwarding tools=turn_tools before calling generate_chat_response / _generate_chat_response_inner; it only proves the loop passes active_tools to this local fake and that the helper forwards what it was given. In that regression scenario the production backend never injects schemas, which is the seam this test is meant to guard, so this should exercise the real backend method with generation/helper monkeypatched or assert at that boundary.
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Follow-up to the tool-call parser test PRs #5620 and #5704.
What
Adds one deterministic, download-free backend test:
studio/backend/tests/test_safetensors_toolcall_wiring.py.No production code changes. The file is auto-discovered by the existing Backend CI
pytest matrix (
studio-backend-ci.yml), needs no workflow edit, and runs sub-second.Why
test_safetensors_tool_loop.pyalready coversparse_tool_calls_from_textand thecumulative-text state machine in
run_safetensors_tool_loopexhaustively, using fakesingle-turn generators. What it does not cover is the backend own tool-injection seam:
both
InferenceBackend(transformers) andMLXInferenceBackendrender the prompt throughthe shared
apply_chat_template_for_generation(..., tools=...)helper and then streamcumulative text into the shared
run_safetensors_tool_loop. A regression that drops thetool schema before it reaches the tokenizer, or that fails to feed a tool result back into
generation, would pass the existing suite but break real tool calling.
Because MLX and safetensors share the exact same injection helper and loop, one CPU test of
that seam guards both paths, including the macOS MLX path, without importing mlx.
What it asserts
The test mirrors the real backend
_single_turnwiring with fakes (a tokenizer that recordsthe
toolsit is handed, a canned tool-call generation, a stub executor) and checks the fullchain end to end:
chat-template fallback)
tool_start, one dispatch with the parsed arguments, onetool_endwith the resultre-entered generation)
It imports no torch, unsloth, or mlx, so it stays in the portable Backend CI next to the parser
tests. It reuses the existing fake-tokenizer and stub-executor patterns from
test_safetensors_tool_loop.py.Testing
cd studio/backend && python -m pytest tests/test_safetensors_toolcall_wiring.py -qpasses in~0.2s with only
studio.txt+ pytest installed (no torch/unsloth/mlx).tests/test_safetensors_tool_loop.pyalongside it: no regressions.ruff checkclean.