fix(tokens): set OpenAI ceilings from an upstream probe, not the catalog#111
Closed
VickyXAI wants to merge 1 commit into
Closed
fix(tokens): set OpenAI ceilings from an upstream probe, not the catalog#111VickyXAI wants to merge 1 commit into
VickyXAI wants to merge 1 commit into
Conversation
gpt-5.5 32768 -> 100000 gpt-5.4 32768 -> 100000 gpt-5-mini 16384 -> 65536 These were pinned low in the previous release with a note to change them only once a source outside the gateway catalog confirmed them, because the catalog had just been caught reporting 8192 for claude-haiku-4.5 against Anthropic's documented 64000. That source is now a probe: real paid requests through the live gateway at the catalog's advertised ceiling. gpt-5.5 asked 128000 -> "max_tokens too large (maximum: 100000)" gpt-5.4 asked 128000 -> same gpt-5-mini asked 65536 -> accepted The catalog was wrong on two of three, in the direction that breaks requests: syncing to its 128000 would have sent a value upstream refuses. Re-probed at 100000 and both accept, so that bound is confirmed from both sides. gpt-5-mini's 65536 is a floor, not a proven ceiling — the probe only shows the limit is >= 65536. That is all this table needs: ESCALATED_MAX_TOKENS is 65536, so any higher true limit is indistinguishable here. Also rewrites the precedence test to use a sentinel. It hardcoded whichever static-vs-catalog disagreement existed at the time — haiku first, then gpt-5.5 — and broke twice when those numbers legitimately changed. The haiku form was worse than brittle: once the gateway was corrected and both sides read 64000, it kept passing while asserting nothing. A sentinel no real model can carry tests the rule instead of the values, and survives future corrections. Local suite 636 pass.
Contributor
Author
|
Closing: the evidence behind this is invalid. The probe I based it on never reached the network. What that invalidates:
What survives: Reopening with only that, plus the sentinel-based precedence test rewrite, which was independent of the bad premise. |
VickyXAI
added a commit
that referenced
this pull request
Jul 21, 2026
…ed (#112) openai/gpt-5-mini 16384 -> 65536 Replaces #111, which was closed because its evidence was invalid. That PR also raised gpt-5.5 and gpt-5.4 to 100000, citing "upstream's own stated maximum". There was no upstream response: both SDKs reject max_tokens > 100000 client-side (blockrun_llm validation.py:233, blockrun-llm-ts validation.ts:74), so every probe at 128000 failed before reaching a provider. The "maximum: 100000" I read as an upstream limit is an SDK constant. gpt-5-mini survives because its probe was BELOW that guard — the request actually went out and was accepted at 65536. It is a floor, not a proven ceiling: 65536 is also ESCALATED_MAX_TOKENS, so nothing here could distinguish a higher true limit. The comment says so rather than implying verification. gpt-5.5 and gpt-5.4 stay at 32768. The catalog's 128000 for both is UNREFUTED — I did not disprove it, I mismeasured. Their real ceilings are unknown and raising them needs a probe path that bypasses the SDK guard. Keeps the sentinel rewrite of the precedence test from #111, which was independent of the bad premise. That test hardcoded whichever static-vs-catalog disagreement existed at the time and broke twice as those numbers legitimately changed; the haiku form was worse, silently passing while asserting nothing once both sides agreed. It now primes a sentinel no real model can carry and compares against whatever the static value is, so it tests the rule. Local suite 636 pass. Co-authored-by: 1bcMax <195689928+1bcMax@users.noreply.github.com>
VickyXAI
added a commit
that referenced
this pull request
Jul 21, 2026
) * fix(tokens): gpt-5.5 and gpt-5.4 to 128000 — now actually measured Both were 32768. The catalog said 128000 and I twice failed to establish whether that was true. #111 raised them to 100000 citing an upstream maximum. That number came from blockrun_llm/validation.py:233 — a client-side guard — so the probe never reached a provider. #112 reverted them to 32768 and recorded the catalog's 128000 as unrefuted but unverified. Re-probed 2026-07-21 with the SDK guard bypassed in-process: both accept 128000, as did all 19 models advertising a ceiling above 100000, including zai/glm-5.2 at 262144. Zero rejections. The catalog was right the whole time; the measurement was broken. Saying so explicitly because the previous conclusion pointed the other way, and this table is exactly where a wrong belief gets frozen. Root cause fixed upstream in blockrun-llm#27 and blockrun-llm-ts#15. Effect: normal turns are unchanged (CAPPED_MAX_TOKENS 16384). On a max_tokens stop the escalation ceiling goes from min(65536, 32768)=32768 to min(65536, 128000)=65536 — recovery doubles instead of clamping back below where it started. Local suite 636 pass. * docs(tokens): record that Franklin does not run the SDK max_tokens guard Review of this PR turned up a wrong dependency claim I had made out loud: that the SDK fixes needed to merge before Franklin could raise these values, or users' local SDK would reject the larger number. Franklin never runs that guard. Both request paths — src/agent/llm.ts and src/proxy/server.ts — use raw fetch and import only payment helpers from @blockrun/llm, never chatCompletion, so validateMaxTokens is not on either path. These values are independent of whether blockrun-llm#27 and blockrun-llm-ts#15 land, and the merge ordering I recommended was unnecessary. Also drops the hardcoded '19 models' count, which would rot as the catalog changes, and marks the referenced SDK PRs as open rather than implying the root cause is already fixed everywhere. --------- Co-authored-by: 1bcMax <195689928+1bcMax@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the item left open in #110.
The probe
#110 pinned three OpenAI entries LOW with a note: change them only once a source outside the gateway catalog confirms them — because the catalog had just been caught reporting 8,192 for
claude-haiku-4.5against Anthropic's documented 64,000.That source is now a probe. Real paid requests through the live gateway, at the ceiling the catalog advertises:
openai/gpt-5.5max_tokens too large (maximum: 100000)openai/gpt-5.4openai/gpt-5-miniThe catalog was wrong on two of three, in the direction that breaks requests. Syncing to its 128,000 — the obvious "just fix it" move — would have made Franklin send a value upstream refuses outright. Re-probed at 100,000: both accept. The bound is confirmed from both sides.
What changed
gpt-5.5gpt-5.4gpt-5-minigpt-5-mini's 65,536 is a floor, not a proven ceiling — the probe only shows the limit is ≥ 65,536. That is all this table needs:ESCALATED_MAX_TOKENSis 65,536, so any higher true limit is indistinguishable here. Said plainly in the code comment so nobody reads it as verified.The precedence test needed rewriting, not repointing
gateway catalog never overrides a static max-output entryhardcoded whichever static-vs-catalog disagreement existed at the time. It broke twice in three releases as those numbers legitimately changed — first on haiku, then on gpt-5.5.The haiku form was worse than brittle. Once the gateway was corrected and the static entry raised, both sides read 64,000 — and the test kept passing while asserting nothing. A green test that has stopped testing is the failure mode worth designing against.
It now primes the catalog with a sentinel (
999_999) no real model will ever carry and asserts the sentinel doesn't win, comparing against whatever the static value happens to be. Tests the rule, not the values; survives every future correction.Gateway-side follow-up
blockrun's catalog says 128,000 forgpt-5.5/gpt-5.4but upstream's stated maximum is 100,000. The gateway clamps to its own value and forwards, so any caller asking for more than 100,000 on those models takes a hard upstream rejection today. Filing separately.Local suite 636 pass, 0 fail.