feat(aio): explain empty generation output when tokens were billed - #90159
feat(aio): explain empty generation output when tokens were billed#90159bernatixer wants to merge 15 commits into
Conversation
A generation can reach the trace view with no captured output while the provider still reports output tokens. The output panel showed only an unexplained "No output" box, so a provider that genuinely returned nothing looked identical to content that was generated and never reached the event. The panel now names the second case when output tokens were billed, and stays as it was when none were. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
Covers a generation whose output normalizes to nothing while the provider reports output tokens, so the state stays reviewable in Storybook. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
The notice read the token count as a strict number, so a provider that reports it as a string fell through and the box stayed unexplained. Read it leniently instead, and format it with the shared number helper so it matches the token count in the metadata header above it. The session replay inspector renders generations through the same panel, so pass the token count there too. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
🤖 CI report✅ Trunk lane — non-backend laneThis PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.
|
| File | Size | Δ vs base |
|---|---|---|
posthog-app/_parent/products/engineering_analytics/frontend/scenes/EngineeringAnalyticsScene.js |
71.7 KiB | 🔺 +9.9 KiB (+16.1%) |
render-query/src/render-query/render-query.js |
21.77 MiB | 🔺 +6.7 KiB (+0.0%) |
posthog-app/src/scenes/models/NodeDetailScene.js |
15.6 KiB | 🟢 -1.5 KiB (-8.5%) |
exporter/src/exporter/scenes/ExporterNotebookScene.js |
3.40 MiB | 🟢 -1.3 KiB (-0.0%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.37 MiB · 22 files | 🔺 +41 B (+0.0%) | ███░░░░░░░ 30.3% of 4.51 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.82 MiB · 3,247 files | 🔺 +3.6 KiB (+0.0%) | █████████░ 90.8% of 9.71 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 307.0 KiB | ../node_modules/.pnpm/posthog-js@1.422.0_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 263.4 KiB | ../node_modules/.pnpm/posthog-js@1.422.0_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.js |
| 252.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 154.2 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 104.7 KiB | src/lib/api.ts |
| 95.2 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.26 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.26 MiB · 18 files | 🔺 +159 B (+0.0%) | ████░░░░░░ 39.5% of 5.72 MiB |
| Deferred (lazy) | 2.11 MiB · 45 files | 🔺 +22 B (+0.0%) | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 749.3 KiB | dist/toolbar/toolbar-app-J2MBJBF4.css |
| 588.4 KiB | dist/toolbar/chunk-chunk-K5GKF7GY.js |
| 484.7 KiB | dist/toolbar/chunk-chunk-OX4ECFTJ.js |
| 133.8 KiB | dist/toolbar/chunk-chunk-QBW74TBI.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-FDH2IBXT.js |
| 71.3 KiB | dist/toolbar/toolbar-app-SDJT2P3V.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-TSAL54PB.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-4IRI3WTW.js |
| 20.9 KiB | dist/toolbar/chunk-chunk-KKY2CY6J.js |
| 6.8 KiB | dist/toolbar/chunk-chunk-DV7IWQNF.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +222.2 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1435.23 MiB · 🔺 +222.2 KiB (+0.0%)
The fixture only needs a four-digit number to exercise thousands grouping, so pick one that owes nothing to the case that prompted the change. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
|
👋 Visual changes detected for this PR. Review and approve in PostHog Visual Review If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix. |
A generation can report reasoning tokens with no output token count at all. Those showed a bare "No output" box, because the notice only looked at the output count, so the case it most clearly explains was the one it stayed silent on. Read both counts and name whichever are present. The cause now follows the data instead of listing both possibilities: reasoning tokens point at a spent thinking budget, and their absence points at a cut-short response or a capture gap. Providers disagree on whether reasoning tokens sit inside the output count or beside it, so neither is derived from the other. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
The notice hedged between two causes because it only had token counts to reason from. `$ai_stop_reason` carries the provider's own account of why it stopped, and it already arrives on a third of generations, so read it and state the cause instead. A response that ran out of room now says so, and one the provider blocked says that. A blocked response can arrive before anything is billed, so a known cause is worth showing on its own. Anything else keeps the previous wording. Providers spell the same outcome several ways, and the OTel middlewares pass the value through verbatim, so match a normalized set. `max_tokens`, `MAX_TOKENS`, and `length` all mean the same thing. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
A generation can send an empty `$ai_output_choices` while `$ai_output` holds the response. A nullish check kept the empty container, so the panel rendered nothing and the new notice went on to state that no content was captured. The old bare box was vague enough to survive that; a sentence naming a cause is not. Skip a container with no entries so the fallback is reached, in the trace view, the replay inspector, and the lazy heavy-property load that backs both.
Reasoning tokens now only carry the blame when they account for the whole billed output. A small reasoning count beside a large output count leaves non-reasoning tokens unaccounted for, and blaming reasoning there is the wrong cause.
Two smaller gaps in the same path: the Vercel AI SDK hyphenates its stop reasons, so `content-filter` never matched the underscored set, and Bedrock spells it `content_filtered`. And posthog-ai below 7.3.0 sent token counts as `{ total, noCache, cacheRead }`, so events captured before ingestion normalized that shape made the notice vanish.
Generated-By: PostHog Desktop
Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
… output Comparing the reasoning count against the output count only works for providers that count reasoning inside the output. For providers that count it separately, a reasoning count at or above the output count can still sit beside billed text tokens, and blaming reasoning there names the wrong cause. `$ai_text_output_tokens` states the split directly, so trust it when a provider reports it: an explicit zero means every billed output token was reasoning, and anything above zero keeps the general wording. The comparison stays as the fallback when the property is absent. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
… amounts When the reasoning count matched the output count, the notice listed both and then blamed reasoning, which reads as half the billed tokens going unexplained. Under the accounting that comparison detects, the two numbers are one amount counted twice, so name it once: the output count, with the claim that all of it may have been reasoning. The comparison also blamed reasoning when it exceeded a nonzero output count. Reasoning can never exceed the output it is counted inside of, so that shape proves the provider counts them separately, and the output tokens are missing content. That case now keeps the general wording. An explicit zero in the text-token split now says what it means, that none of the output was text, instead of guessing reasoning while other output kinds could account for the rest. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
…y helpers The sets and describeStopReason carry knowledge about what an `$ai_stop_reason` value means, not rendering logic, and this branch already homes that kind of helper in utils.ts. A component file also hides the repo's only stop-reason mapping from the next surface that wants it, such as the metadata header or a filter. describeEmptyOutput stays in the component, since it assembles display copy. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
The comment compressed the deduction so far that a reader had to reconstruct it. Name the two accounting styles and state what each observed shape can mean under each, so the gate conditions read as conclusions instead of assertions. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
👀 Auto-assigned reviewersThese soft owners were skipped because they only have minor changes here. Nothing blocks merge, so self-assign if you'd like a look:
Soft owners come from each directory's |
carlos-marchal-ph
left a comment
carlos-marchal-ph
left a comment
There was a problem hiding this comment.
Looks great, the screenshots were super helpful. You'll have to approve the manual visual screenshots task in CI once this is ready to merge, but first there's a couple of comments to address.
Preferring content over presence made a genuinely empty output read as missing, so useAIData ran the heavy-property lookup and showed a loading block before the notice this change exists to render. The selector now falls back to the first present value, so an empty container still counts as arrived. Renamed to selectAiValue, since it no longer only returns values with content.
The lazy loader gets the same treatment on both sides through one selectHeavyValue: an empty $ai_input no longer hides a populated $ai_input_state, and an all-empty ai_events row no longer falls through to the second, slower events query.
The metadata header gated its token tag on `typeof === 'number'` while the notice below read counts leniently, so a string or `{ total, … }` count silenced the header and contradicted the sentence under it. The lenient read now lives in utils as aiTokenCount and both surfaces use it.
Generated-By: PostHog Desktop
Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
marco-g-pm
left a comment
marco-g-pm
left a comment
There was a problem hiding this comment.
This is great! Now that I see the more detailed error messages, I do think "okay so how do I fix this?"
It might be worth it to add a corresponding update to our docs (maybe under troubleshooting?) with more detail + potential fixes for each issue
I've also opened this PR in order to receive more stop reasons and avoid this "generic error message". This two PRs will not solve it all but at least give users more information on what's happening and reduce the number of unknown cases. I've added a small section under troubleshooting PostHog/posthog.com#19793 |
The notice names why a generation rendered no output but not what to do about it, and review feedback asked exactly that. Each sentence now ends with a link to the troubleshooting section that carries the fix for every cause the notice can name. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
3 participants
Problem
The AI observability trace view can show a generation whose output panel is empty while the provider still reports tokens for it. That rendered as a bare "No output" box.
The box made several different situations look identical: a provider that genuinely returned nothing, a response that ran out of room, one the provider blocked, and content that was billed but never reached the event. Someone hitting any of the last three had nothing to go on, so the page read as broken rather than as a known outcome.
The shape is not rare and is not specific to one provider or model. It appears on generations that carry no error flag, which is exactly when an unexplained empty box is most confusing.
Changes
$ai_stop_reasonis the provider's own account of why it stopped, so a response that ran out of room says so, and one the provider blocked says that. A blocked response can arrive before anything is billed, so a known cause shows on its own.$ai_text_output_tokensof zero states the fact directly: none of the output was text.$ai_output_choicesno longer hides a populated$ai_output. A nullish check kept the empty container, so the panel rendered nothing and would have gone on to state that no content was captured. Fixed in the trace view, the replay inspector, and the lazy heavy-property load behind both.$ai_output_tokens,$ai_reasoning_tokens, and$ai_stop_reasonthread through to the shared message display, read leniently so a provider that sends a count as a string or as{ total, … }still gets the notice, and so a hyphenated stop reason matches its underscored spelling.Providers disagree on whether reasoning tokens sit inside the output count or beside it, so the notice names each count that is present instead of deriving one from the other.
Before, all four rendered as just the italic "No output" line, with the metadata header above still showing a token count:
How did you test this code?
Automated, all extending existing parameterized tests rather than adding suites:
ConversationMessagesDisplay.test.tsxcover output only, reasoning only, reasoning accounting for the whole output, reasoning as a fraction of it, an explicit text-token split in both directions, matching counts folded into one claim, a reasoning count above the output count, an upper-case stop reason, a hyphenated one, a stop reason that outranks the reasoning guess, a normal stop reason that must not match, a string-typed count, an object-shaped count, an explicit zero, and no counts. Every explained row also pins the "Learn more" href to the documented anchor, so a dropped link or typo'd anchor fails 13 rows. Each catches a distinct regression: the notice going silent for a shape that was billed, firing on a generation billed nothing, naming the wrong cause, or missing a provider's spelling. A case-sensitive stop-reason check would miss most real truncation values, and a naive count check drops the pre-7.3.0 object shape.ConversationDisplay.test.tsxcovers an empty$ai_output_choicesbeside a populated$ai_output, asserting the populated one reaches the renderer. No existing row exercised that fallthrough.By the agent: added four Storybook stories for the explained shapes and rendered them headless; the screenshots above are those stories. Ran the touched files, then the whole
products/ai_observability/frontend/and session-replay inspector suites (2,033 tests) to check the shared output-selection change, plus the repo typecheck.From CI on an earlier commit: the visual review run reported zero changed snapshots against 2,238 baselines, so this does not alter how any existing story renders. The new stories show as new baselines pending approval, which is why that check is red.
Not done: this was not exercised against real data in a running app, and the full frontend suite was not run locally.
Automatic notifications
Docs update
PostHog/posthog.com#19793 documents every sentence this notice can show, with the fix for each. The "Learn more" link in the notice points at it.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
A person directed this work in a PostHog Desktop session, but their GitHub handle was never established there. The agent set no assignee and mentioned nobody; any assignee or reviewer shown here came from repo automation. The requester should claim it.
Several drafts were discarded against data rather than shipped. An early theory that the UI was masking real data via the
$ai_output_choicesnullish check turned out not to explain the originating trace, where both properties were genuinely empty — so it was cut, then reinstated once the notice began asserting a cause, because a vague box tolerates that ambiguity and a sentence does not. The truncation copy went through the same cycle: an early draft asserted it with nothing behind it, the shipped code asserts it only when$ai_stop_reasonsays so, and that property proved populated on a third of generations. Copy blaming reasoning tokens was gated twice after review: first on a count comparison, then on the explicit$ai_text_output_tokenssplit, since the comparison is only exact for providers that count reasoning inside the output.hogli reviewneeds interactive Greptile auth, unavailable in a cloud task, so the/reviewing-before-prharness fallback ran: independent reviewers over the diff, findings verified against the source before applying. Applied findings covered string- and object-typed token counts, the shared number formatter, the missing session replay call site, provider spellings of the stop reason, the reasoning-proportion gate, the empty-container fallthrough in three places, and a test assertion pinned to one hard-coded value. Skills invoked:/exploring-llm-traces,/query-clickhouse-via-metabase,/writing-user-facing-copy,/writing-tests,/code-review,/reviewing-before-pr,/writing-pr-descriptions.Public artifact: this started from a customer trace. None of it reaches this PR. The Storybook and test fixtures are invented, and the description carries no identifiers, counts, or figures.
Created with PostHog Desktop