Support adaptive interruption for realtime models#6488
Merged
Conversation
longcw
changed the title
longcw
marked this pull request as ready for review
July 21, 2026 03:06
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| # only honor the verdict while this turn's overlap is unresolved so a late verdict | ||
| # can't leak into the next turn; an interruption supersedes a prior backchannel | ||
| if self._overlap_in_current_turn: | ||
| self._turn_backchannel_over_agent = not ev.is_interruption |
Member
There was a problem hiding this comment.
The overlap speech also ends when agent finishes speaking first, so we trigger a synthesized is_interruption=False, but the user speech might not be done, and we might carry over the wrong flag (til the end of turn).
Contributor
Author
There was a problem hiding this comment.
added a flag agent_ended to distinguish two types of is_interruption=False
Member
There was a problem hiding this comment.
if there are one backchannel ev + one agent ends first ev during the agent speech, self._turn_backchannel_over_agent would be stale, right?
longcw
force-pushed
the
longc/realtime-barge-in
branch
from
July 22, 2026 01:49
a557e26 to
2926544
Compare
A realtime model transcribes internally and commits turns manually, so barge-in can gatekeep the overlapping user turn by withholding the commit instead of holding and flushing STT transcripts. Relax _resolve_interruption_detection so a realtime model with server-side turn detection disabled no longer requires a separate STT; the STT pipeline path is unchanged.
When a realtime model runs client-side barge-in with server-side turn detection disabled, an overlapping utterance the adaptive detector classifies as a backchannel must not create a user turn. Track the backchannel verdict per turn and skip the end-of-turn commit; clear the model's audio buffer so the backchannel audio doesn't leak into the next turn.
Move the backchannel verdict from an AgentActivity flag reset on VAD start-of-speech to a turn-scoped signal owned by the audio recognition layer and surfaced on _EndOfTurnInfo. A second start-of-speech within a logical turn no longer resets the verdict, and a late verdict can no longer leak into the next turn (it is only recorded while the current turn has an unresolved overlap). Clear the buffered input audio at the drop point rather than at the verdict, so it also covers the case where the agent is still speaking and the verdict has not arrived, and no longer truncates a turn that escalates from a backchannel into an interruption.
Clear the turn's backchannel verdict when a new speech segment starts without overlapping agent speech, so a real turn following a backchannel in the same logical turn commits instead of being dropped. A stutter while the agent is still speaking keeps the verdict, so pure backchannels are still dropped.
…irst When the agent finishes speaking while the user is still overlapping, the overlap is force-ended and a synthesized is_interruption=False event is emitted. That isn't a real verdict — the user may still be mid-turn — but it was latched as a backchannel and carried to the end of the turn, dropping genuine speech. Tag the agent-driven overlap-end via agent_ended on the sentinel and OverlappingSpeechEvent, and skip the backchannel latch for it so the turn falls back to committing.
The backchannel verdict persisted a True from an earlier overlap into a later inconclusive overlap within the same agent turn, so a user turn that continued past the agent could be wrongly dropped. Reset it on every speech onset so each overlap re-derives its own verdict.
A confirmed backchannel over the agent left its audio in the realtime input buffer, so it prefixed the next committed turn. Clear the buffer when the backchannel is confirmed while the user is between segments; deferring when the user is already mid next-segment avoids clipping a real turn. This also applies when an STT is attached, since the realtime session buffers input regardless.
longcw
force-pushed
the
longc/realtime-barge-in
branch
from
July 23, 2026 06:01
82a241c to
2b70881
Compare
chenghao-mou
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables client-side adaptive interruption (barge-in) for OpenAI-compatible realtime models when the model's own server-side turn detection is disabled — so a backchannel ("uh-huh") while the agent is speaking doesn't trigger a spurious reply, without requiring a separate STT.
The gate. With no STT there's no transcript to gatekeep, so the audio recognition layer tracks a turn-scoped signal — whether the turn's speech overlapped agent speech and was classified a backchannel — and surfaces it on the end-of-turn info. When set, the user turn is dropped and the buffered input audio is cleared, so no reply is generated.
Difference from the STT gate. The STT path is timestamp-windowed: it drops the backchannel portion of a turn and re-emits the rest. Realtime is all-or-nothing —
clear_audio()wipes the whole input buffer and the model buffers the turn as one unit, so there's no per-timestamp split. It's correct for the dominant cases (pure backchannel, real interruption) and drops only on a confirmed backchannel; a slow verdict after the agent has stopped falls back to committing rather than silently dropping a possible real barge-in.STT + realtime. If you want the finer-grained STT gate, pass an STT alongside the realtime model — the realtime drop path is skipped when an STT is present and transcript-based backchanneling takes over.