fix(voice): keep realtime turn after chat ctx timeout#2083
fix(voice): keep realtime turn after chat ctx timeout#2083rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 083d5a4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 38 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| export class RealtimeError extends Error { | ||
| constructor(message: string, options?: ErrorOptions) { | ||
| super(message, options); | ||
| this.name = 'RealtimeError'; | ||
| Error.captureStackTrace(this, RealtimeError); | ||
| } | ||
| } |
There was a problem hiding this comment.
🟡 New public error type ships without documentation
A new publicly exported class is added (RealtimeError at agents/src/llm/realtime.ts:74) without the TypeDoc comment that the repository requires for every new class/interface/method.
Impact: Public API documentation is incomplete for consumers of the framework.
CONTRIBUTING.md documentation requirement
CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc... and every new addition has to be properly documented." The newly exported RealtimeError class (agents/src/llm/realtime.ts:74-80, re-exported in agents/src/llm/index.ts:76) has no doc comment.
| export class RealtimeError extends Error { | |
| constructor(message: string, options?: ErrorOptions) { | |
| super(message, options); | |
| this.name = 'RealtimeError'; | |
| Error.captureStackTrace(this, RealtimeError); | |
| } | |
| } | |
| /** | |
| * Error raised by a realtime provider when an operation fails or times out | |
| * (e.g. a chat-context update or reply generation). | |
| */ | |
| export class RealtimeError extends Error { | |
| constructor(message: string, options?: ErrorOptions) { | |
| super(message, options); | |
| this.name = 'RealtimeError'; | |
| Error.captureStackTrace(this, RealtimeError); | |
| } | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| exception(): unknown { | ||
| if (!this.doneFut.done) { | ||
| throw new Error('SpeechHandle is not done yet'); | ||
| } | ||
|
|
||
| return this._error; | ||
| } |
There was a problem hiding this comment.
🟡 New public method added without documentation
A new public method is added (exception() at agents/src/voice/speech_handle.ts:187) without the TypeDoc comment that the repository requires for every new method.
Impact: Public API documentation is incomplete and the throwing behavior of the method is undocumented for callers.
CONTRIBUTING.md documentation requirement
CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them." The new public exception() method on SpeechHandle throws when the handle is not yet done, which is a behavior worth documenting.
| exception(): unknown { | |
| if (!this.doneFut.done) { | |
| throw new Error('SpeechHandle is not done yet'); | |
| } | |
| return this._error; | |
| } | |
| /** | |
| * Returns the error that caused this SpeechHandle to complete, if any. | |
| * | |
| * @throws Error if the SpeechHandle is not done yet. | |
| */ | |
| exception(): unknown { | |
| if (!this.doneFut.done) { | |
| throw new Error('SpeechHandle is not done yet'); | |
| } | |
| return this._error; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Ports livekit/agents#6445 to JS: realtime user turns are no longer dropped when the pre-reply chat context push times out with a provider-reported realtime error.
Changes:
RealtimeErrorfor realtime provider timeout/failure classification.RealtimeErrorfromAgentActivity.realtimeReplyTaskchat-context push as best-effort and still callgenerateReply.SpeechHandledone with the error.RealtimeError("update_chat_ctx timed out.").Source diff coverage
livekit-agents/livekit/agents/voice/agent_activity.pyagents/src/voice/agent_activity.ts: catches providerRealtimeErrorfrom the pre-replyupdateChatCtxcall, logs a warning, records the local user message, and still generates the realtime reply. Unexpected errors log, mark theSpeechHandlefailed, and return, matching the source failure path.tests/test_realtime_reply_chat_ctx.pyagents/src/voice/agent_activity.test.ts: ported the three source cases into Vitest using targetChatContext,ToolContext,SpeechHandle, and a minimal fake realtime activity because the target has notests/fake_realtime.pycounterpart or standalonetest_realtime_reply_chat_ctxfile.Additional target infrastructure ported because the source behavior depends on it:
llm.RealtimeErrorclassificationRealtimeErrorinagents/src/llm/realtime.tsand exported it fromagents/src/llm/index.ts.SpeechHandle._mark_done(error=...)plusSpeechHandle.exception()observabilityagents/src/voice/speech_handle.tsvia_markDone(error?)andexception().update_chat_ctx timed out.provider errorplugins/openai/src/realtime/realtime_model.tsto throwllm.RealtimeError("update_chat_ctx timed out.")for chat-context ack timeout.No source diff files were dropped as not applicable.
Verification
Passed:
pnpm test -- agents/src/voice/agent_activity.test.ts agents/src/voice/speech_handle.test.tspnpm --filter @livekit/agents --filter @livekit/agents-plugin-openai... buildpnpm test -- agents/src/voice/agent_activity.test.ts agents/src/voice/speech_handle.test.ts plugins/openai/src/realtime/realtime_model.test.tspnpm --filter @livekit/agents --filter @livekit/agents-plugin-openai lintexited successfully with existing warnings.cue-clirealtime smoke validation passed: launched a temporary OpenAI realtime JS worker, drove a text turn, and assertedfunction_tools_executed -> conversation_item_added(.item.message.role="ASSISTANT").Attempted full touched-package tests:
pnpm test -- agents/src plugins/openai/srcfailed on unrelated existing/environment issues:agents/src/voice/amd.test.tsexpectation/timeouts, and OpenAI plugin test import failures before package builds.pnpm test -- plugins/openai/srcstill failed on unrelated fixture/environment issues: corrupted or placeholderplugins/silero/dist/silero_vad.onnxforstt.test.ts, and an existinggpt-realtime-whisperVAD requirement error intts.test.ts. Other OpenAI plugin tests passed.Additional attempted check:
pnpm --filter @livekit/agents --filter @livekit/agents-plugin-openai api:checkis blocked by the repo's existing API Extractor limitation onexport * as ___inagents/dist/index.d.ts.Ported from livekit/agents#6445
Original PR description
Incident. Calling session.generate_reply(user_input=...) on a realtime session pushes the message via update_chat_ctx before requesting the reply. That await was unguarded: when the OpenAI plugin's 5s ack timeout fired (RealtimeError("update_chat_ctx timed out.")), the error propagated as an unhandled exception in the turn task — the turn died, the user got silence, and nothing was reported on the SpeechHandle.
Fix. The push is best-effort — the plugin sends all conversation.item.create events before waiting for acks, so a timeout means missing acknowledgements, not missing items. So:
RealtimeError → log a warning and still call generate_reply (the server usually has the context).
Unexpected exceptions → mark the SpeechHandle done with the error, observable via SpeechHandle.exception() #6304 , without crashing the turn task.
Deliberately out of scope: no retry of the push itself, and no change to the plugin's timeout; the two other update_chat_ctx call sites (interrupted-sync and tool-output paths) already have their own RealtimeError handling and are untouched.