fix(voice/realtime): add tool_reply_after_audio to prevent duplicate speech#6480
Open
Utkarshsteve wants to merge 2 commits into
Open
fix(voice/realtime): add tool_reply_after_audio to prevent duplicate speech#6480Utkarshsteve wants to merge 2 commits into
Utkarshsteve wants to merge 2 commits into
Conversation
The `with_azure` factory set `model="gpt-4o"` by default, but Azure OpenAI uses `azure_deployment` to select models — the `model` parameter is not used by the API. This caused the `.model` property to return "gpt-4o" regardless of the actual deployment. Change `model` to default to `None` and fall back to `azure_deployment` so that `.model` reflects the real deployment name. Closes livekit#6209
…licate speech When a realtime model bundles audio output and a function_call in one response, the framework's post-tool reply fires and the agent answers twice. This adds a configurable `tool_reply_after_audio` policy on `ToolHandlingOptions`: - "always" (default): backwards-compatible, always generate reply - "skip": suppress post-tool reply when audio was already delivered Closes livekit#6411
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Sanjays2402
reviewed
Jul 20, 2026
| if len(tool_output.output) > 0: | ||
| speech_handle._num_steps += 1 | ||
|
|
||
| response_had_audio = started_speaking_at is not None and any( |
There was a problem hiding this comment.
started_speaking_at is also set by _TextOutput.first_text_fut in _on_first_frame, so this becomes true for a text-only response with no audio at all. With transcription enabled and tool_reply_after_audio="skip", that suppresses the required post-tool reply even though nothing was spoken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tool_reply_after_audiopolicy toToolHandlingOptionsthat controls whether the framework generates a post-tool reply when the realtime response already produced audible audio"always"(backwards-compatible); setting"skip"suppresses the duplicate replyresponse_had_audioonFunctionToolsExecutedEventso user handlers can inspect the signalProblem
When a realtime model (e.g. xAI Grok) bundles audio output and a
function_callin one response, the framework fires a post-tool reply after executing the tool — causing the agent to audibly answer twice (#6411).Solution
Detection: after tool execution, check whether the triggering response produced audio that was actually played (
started_speaking_at is not Noneand at least one message output wasn't skipped).Guard: when
tool_reply_after_audio == "skip"and audio was produced, skip the post-tool reply generation.Test plan
ToolHandlingOptionsconfig dictionaryAgentSessionconfig storage (default, skip, always, empty dict)make checkpasses (format, lint, mypy strict)uv run pytest --unit— all 1132 + 10 new tests passCloses #6411