From 11f0ee9b15b5479cd954ba4b6b6722abef87daa3 Mon Sep 17 00:00:00 2001 From: Daniel Han Date: Fri, 1 May 2026 11:15:30 +0000 Subject: [PATCH 1/2] docs(mistral3): diagnosis of long-context degradation on Mistral-Medium-3.5-128B Documents the experiments and findings from comparing llama-server (Q8_0 GGUF), vLLM (FP8 safetensors), and HF transformers (BF16) inference of mistralai/Mistral-Medium-3.5-128B on the same multi-turn fixtures. Key findings: - Tokenization is identical across vLLM /v1/chat/completions/render, llama-server /apply-template + /tokenize, mistral-common, and HF AutoTokenizer (byte-identical 434-token streams). - Chat templates from the GGUF, unsloth, mistralai upstream, and the HF tokenizer all render to identical token streams for normal multi-turn chat. - GGUF metadata matches HF text_config including all YARN parameters (factor=64, freq_base=1e6, original_context_length=4096, beta_fast=4, beta_slow=1, yarn_log_mul=1.0). - llama.cpp's mistral3.cpp implements the same residual / RMSNorm / SwiGLU / RoPE flow as transformers' Ministral3DecoderLayer. - KV cache dtype (F16, BF16, F32), flash attention on/off, and CUDA build with -DGGML_CUDA_FORCE_CUBLAS=ON + GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1 do not change the looping behaviour. - Sampler choice (matched min_p / top_k / top_p / seed; sweeps over repetition_penalty, frequency_penalty, dry_multiplier) does not change it either. - HF transformers BF16 also degrades on the same input. So this is not Q8_0-specific; it's a model-wide property that vLLM happens to avoid. This matches what unsloth has already published on https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF (Mistral has labeled GGUF support WIP). Empirical convergence point: same input, greedy temperature=0: - vLLM FP8 stops naturally at 1496 tokens - llama-server Q8_0 hits length cap, looping after ~1000 tokens - HF transformers BF16 hits length cap, looping after ~1000 tokens This commit only adds documentation; no code changes. The next step is to dump per-layer activations from vLLM and llama-server for the same input and find where they first diverge significantly. --- docs/diagnosis/arch.md | 69 +++++++++ docs/diagnosis/chat_template.md | 44 ++++++ docs/diagnosis/config.md | Bin 0 -> 2857 bytes docs/diagnosis/logits.md | 29 ++++ .../mistral-medium-3.5-long-context.md | 131 ++++++++++++++++++ docs/diagnosis/tokenization.md | 84 +++++++++++ 6 files changed, 357 insertions(+) create mode 100644 docs/diagnosis/arch.md create mode 100644 docs/diagnosis/chat_template.md create mode 100644 docs/diagnosis/config.md create mode 100644 docs/diagnosis/logits.md create mode 100644 docs/diagnosis/mistral-medium-3.5-long-context.md create mode 100644 docs/diagnosis/tokenization.md diff --git a/docs/diagnosis/arch.md b/docs/diagnosis/arch.md new file mode 100644 index 000000000000..b7f344811f8f --- /dev/null +++ b/docs/diagnosis/arch.md @@ -0,0 +1,69 @@ +# Phase 9 — model architecture diff (llama.cpp `mistral3.cpp` vs HF `ministral3`) + +## TL;DR + +The two implementations are **structurally equivalent** for the inference path +that Mistral-Medium-3.5 actually exercises. SwiGLU MLP, RMSNorm with FP32 cast, +1/sqrt(head_dim) attention scale, pre-norm decoder layers, and YARN-scaled RoPE +all match. The only optional code path is **Llama-4-style attention temperature +scaling**, which is gated on `llama_4_scaling_beta != 0` in HF and on +`hparams.f_attn_temp_scale != 0.0` in llama.cpp — both gates evaluate false for +this model (`llama_4_scaling_beta = 0` per HF config, no +`attn_temperature_scale` key in the GGUF), so the path is skipped on both +sides. **Architecture is not the cause** of the long-context degradation. + +## Side-by-side mapping + +| concern | HF `Ministral3*` | llama.cpp `mistral3.cpp` | +| --- | --- | --- | +| pre-attention norm | `Ministral3RMSNorm(eps=rms_norm_eps)` cast→fp32 then back | `build_norm(LLM_NORM_RMS)` | +| Q/K/V proj | `nn.Linear(bias=False)` | `build_qkv(...)` | +| RoPE | `apply_rotary_pos_emb` with cached `cos,sin` (yarn) | `ggml_rope_ext` with `n_ctx_orig, freq_base, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow` | +| attn temperature scale | `Q *= 1 + beta * log(1 + floor(pos/orig_max))` (only if `beta!=0`) | `Q = ggml_mul(Q, inp_attn_scale)` (only if `f_attn_temp_scale!=0`) | +| attention | `attention_interface(scale=1/sqrt(head_dim), sliding_window=None)` | `build_attn(..., kq_scale = 1/sqrt(n_embd_head))` | +| sliding window | `getattr(config,'sliding_window',None)` → `None` for this model | not configured (correct) | +| O proj | `o_proj = nn.Linear(bias=False)` | `model.layers[il].wo` | +| residual after attn | `hidden = residual + hidden` | `ffn_inp = ggml_add(cur, inpSA)` | +| post-attn norm | `Ministral3RMSNorm` | `build_norm(LLM_NORM_RMS)` (ffn_norm) | +| MLP | SwiGLU: `down(silu(gate(x)) * up(x))` | `build_ffn(LLM_FFN_SILU, LLM_FFN_PAR)` (parallel = silu(gate)*up then down) | +| residual after MLP | `hidden = residual + hidden` | `ggml_add(cur, ffn_inp)` | +| MoE branch | n/a (Mistral-Medium-3.5 is dense) | guarded by `model.layers[il].ffn_gate_inp == nullptr` → dense path | + +## RoPE specifically + +Both honour every YARN parameter in the GGUF (`yarn_beta_fast=4`, +`yarn_beta_slow=1`, `factor=64`, `original_context_length=4096`, +`freq_base=1e6`, `yarn_log_multiplier=1.0`) — all of which match +`rope_parameters` in `mistral_medium_check/config.json`. RoPE math is correct. + +## Llama-4 attn temperature scale + +```python +def get_llama_4_attn_scale(positions_ids, beta, max_position_embeddings): + return (1 + beta * torch.log(1 + torch.floor(positions_ids / max_position_embeddings)))[:, None, :, None] +``` + +`beta = config.rope_parameters["llama_4_scaling_beta"] = 0`, so the multiplier +collapses to **1.0** at every position. llama.cpp side-steps the entire +multiplication when the GGUF doesn't carry the key. Both safe, both equivalent. + +## What this rules out + +- ❌ Sliding-window attention mismatch +- ❌ Wrong RoPE dimensions / wrong YARN parameters +- ❌ Missing Llama-4 temperature scaling (it's a no-op for this model) +- ❌ Activation function mismatch (both SwiGLU) +- ❌ Pre-norm vs post-norm placement +- ❌ Attention scale factor + +## What it does NOT rule out + +- Numerical precision in CUDA kernels (FP16 accumulators in `ggml-cuda` for + the matmul or attention path). +- Q8_0 quantization rounding error in long-attention contexts. +- KV-cache dtype (default F16 → numerical drift over many tokens). +- Sampler behaviour (min_p default of 0.05 in llama-server vs vLLM not applying + it). + +Phase 7 (rebuild with `GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1`) and Phase 6 +(F16/BF16/Q8_0 KV-cache experiment) and Phase 8 (matched samplers) cover those. diff --git a/docs/diagnosis/chat_template.md b/docs/diagnosis/chat_template.md new file mode 100644 index 000000000000..7ec2587ad363 --- /dev/null +++ b/docs/diagnosis/chat_template.md @@ -0,0 +1,44 @@ +# Phase 2 — chat template parity + +## TL;DR + +The four chat templates considered — (A) GGUF-embedded jinja in +`bartowski/mistralai_Mistral-Medium-3.5-128B-GGUF`, (B) `unsloth/Mistral-Medium-3.5-128B`, +(C) `mistralai/Mistral-Medium-3.5-128B` upstream, (D) the `tokenizer.chat_template` +attribute exposed by HF AutoTokenizer for the upstream model — are +**semantically equivalent for normal multi-turn chat**. The only diff that +could change rendered output is in error-path code that nobody is hitting in +this test. **Templates are not the cause** of the long-context degradation. + +## Files + +- `outputs/template_llamacpp.jinja` (13281 bytes) — A +- `outputs/template_unsloth.jinja` (14475 bytes) — B +- `outputs/template_mistralai.jinja` (13479 bytes) — C +- `outputs/template_hftokenizer.jinja` (13479 bytes) — D +- `outputs/template_diff_*` — pairwise diffs + +## Diff summary + +| pair | size | nature of diff | +| --- | --- | --- | +| C vs D (mistralai vs HF tokenizer) | 0 bytes | **identical** | +| A vs C (GGUF vs upstream) | 686 bytes | one hunk: a disabled `{%- if false %}` assertion at line 201 of the GGUF copy where upstream has the real check `(content == '' or content is none) and (no tool calls)` → raise. Both branches are no-ops on valid messages. | +| B vs C (unsloth vs upstream) | 2040 bytes | (1) unsloth uses `strftime_now` for date defaults; upstream hardcodes `today=29-04-2026`, `yesterday=28-04-2026`. Both get overridden by template arguments at render time, so no effect on output. (2) unsloth `arguments\|tojson\|safe` vs upstream `arguments\|tojson` for tool-call serialisation — only relevant when tools are used. | + +## Cross-check via tokenization parity (Phase 1) + +When all four templates render the same multi-turn fixture (system prompt + 3 +user/assistant pairs + final user turn, `reasoning_effort='none'`), they +produce **byte-identical 434-token sequences**, including ``, +`[SYSTEM_PROMPT]`, `[/SYSTEM_PROMPT]`, `[MODEL_SETTINGS]`, `[/MODEL_SETTINGS]`, +`[INST]`, `[/INST]`, and the per-assistant `` markers in the right +positions. See `outputs/diagnosis_tokenization.md`. + +## Conclusion + +If we replace llama-server's GGUF-embedded template with the upstream mistralai +copy via `--chat-template-file outputs/template_mistralai.jinja`, the rendered +text and resulting token stream are identical to what the GGUF template +produces today. **No improvement is expected from a template swap, and indeed +no improvement was observed empirically** — see Phase 8 matched-sampler runs. diff --git a/docs/diagnosis/config.md b/docs/diagnosis/config.md new file mode 100644 index 0000000000000000000000000000000000000000..9afef7cf9c8083e296f5ede7a20c6c2a9f47faf2 GIT binary patch literal 2857 zcmbtW!EW3(5Zwel1nMQ9K)^ZJ1|+Yv8+Yp>mm)|~ped4~0Rj{S#g(WPyOO9#Tstmu z=tmUjC0~~>$#5vjq_t}ph!4gr`DWg5IDDfc@}^`xCrk3>-~UJ^n?h_z%@xa;Vk8$u zL5^R(eD{J}_T<%zlOWJPE=XEctY$Q8TUyJUSN&NndeyOt3HsVt;cK3YT^%l>r(v9| zeh%NR==`3ra1MWQC<`=}(PpzNQbK}PntEWOHO8k|$+PX5$vpbImyHIpE-P=Ki!=y6 zkwWk)C!dH8Mz{RhnwP8svn*)KRB0@;R5O*q0=R}@2pf0)2h^p^*qZj@9}Zxf<|4XT zPWAe1_Hc%V;vW??d7s_F#EW?TL!wb%KL3vylp@c0GY!t`HP3U=Y-q)sjVcYrd^V+U zA|NPElGVzkXm&LnKoz^@UGJS@UCC?<>FyeZ?(b)NI3^(wI;CMs@kR;R(2}#ld2YYVzMEF|KrBOip2W+g zd)7|2oT?Ip!|NKo92Fh^K-Zu$=sSC&enI0m*_ah;Xa&-*t*d+5$gZZm^-~`&I>H7u zAdLo~sPxN+_w{GfMnxr=S|rCY59d!${=#+lxw#Cui9l_Q?@jk!E^6;IH~jkUodRmk z9o&(oOjdgcv6Ug5Axpe>*H;1`@RewCd8G+$L2e8;c^{7vt%I%LTJ}`QEeM9tikG4d zy3~P^)>Y(gqyK|_Jw8FjZh`pS6z(=sJlsbyL3x_j!WG_n|`C4jyKsL39r%Tq$D40be Y&J!1lGcsgPfH+SbEY9eNjTfiES8c=IAOHXW literal 0 HcmV?d00001 diff --git a/docs/diagnosis/logits.md b/docs/diagnosis/logits.md new file mode 100644 index 000000000000..c893655d05ad --- /dev/null +++ b/docs/diagnosis/logits.md @@ -0,0 +1,29 @@ +# Phase 3 — top-10 logprobs comparison (vLLM vs llama-server) + +prompt: `What is the capital of France? Answer in exactly one word.` +temperature: 0.0 (greedy), max_tokens: 1, top_logprobs: 10 + +vLLM completion: `Paris` +llama-server completion: `Paris` + +## Top-10 next-token logprobs + +| rank | vLLM token | vLLM logprob | llama token | llama logprob | +| --- | --- | --- | --- | --- | +| 1 | `token_id:42572` | -0.0001 | `Paris` | -0.0000 | +| 2 | `token_id:6993` | -9.8751 | ` Paris` | -11.4609 | +| 3 | `token_id:1784` | -12.0001 | `Par` | -12.5013 | +| 4 | `token_id:2029` | -12.9376 | `PAR` | -14.4139 | +| 5 | `token_id:3814` | -13.2501 | `巴黎` | -14.6281 | +| 6 | `token_id:102726` | -13.6251 | `Pars` | -15.6960 | +| 7 | `token_id:38166` | -13.6251 | `par` | -16.1068 | +| 8 | `token_id:72056` | -13.8751 | ` Париж` | -16.8023 | +| 9 | `token_id:75613` | -15.1876 | `Berlin` | -17.0093 | +| 10 | `token_id:126441` | -15.3751 | ` París` | -17.9099 | + +Jaccard(top-10 token set): **0.00** +Top-1 match: **False** + +## TL;DR + +Greedy top-1 token: **DIFFERS**. diff --git a/docs/diagnosis/mistral-medium-3.5-long-context.md b/docs/diagnosis/mistral-medium-3.5-long-context.md new file mode 100644 index 000000000000..6c700cbff37d --- /dev/null +++ b/docs/diagnosis/mistral-medium-3.5-long-context.md @@ -0,0 +1,131 @@ +# Mistral-Medium-3.5-128B llama.cpp long-context degradation — diagnosis + +## TL;DR + +llama-server (Q8_0 GGUF from `bartowski/mistralai_Mistral-Medium-3.5-128B-GGUF`) +collapses into deterministic repetition loops after ~1000–1500 generated tokens, +producing broken Python syntax (e.g. `pygame.display.set mode((800, 600)` — +missing the underscore in `set_mode` and the closing paren). vLLM (FP8 of the +same model) does not exhibit this; it stops naturally at 1496 tokens. **HF +transformers BF16 inference of the same model also degrades** in the same way, +ruling out llama.cpp/Q8_0 as the *unique* culprit. + +This matches what unsloth has already published on +[`unsloth/Mistral-Medium-3.5-128B-GGUF`](https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF): + +> Testing shows that this behavior occurs **regardless of who or how** the model +> was converted GGUF. The model initially responds correctly, but over long +> context, does not work properly. **Mistral has now labeled GGUF support as a +> WIP**. + +This document records the experiments that were run against vLLM (port 8765, +FP8) and llama-server (port 8766, Q8_0) and what they ruled out. It is +intended as input to a deeper fix. + +## Setup + +- vLLM 0.20.1rc1.dev127, FP8, `--tensor-parallel-size 2 --port 8765 --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral --max_num_batched_tokens 16384 --max_num_seqs 128 --gpu_memory_utilization 0.8 --attention-backend FLASH_ATTN`, GPUs 4,5. +- llama-server `b1-aab6821` (unslothai fork) and ggml-org master, Q8_0, + `--tensor-split 1,1 --port 8766 --jinja --ctx-size 32768 --parallel 1`, + GPUs 2,3. +- HF transformers 5.7.0, `Mistral3ForConditionalGeneration` BF16, + `attn_implementation="eager"`, GPUs 6,7. + +## Phases run + +| phase | topic | result | location | +| --- | --- | --- | --- | +| 1 | Tokenization parity (vLLM / llama-server / mistral-common / HF) | **identical** 434 tokens | `outputs/diagnosis_tokenization.md` | +| 2 | Chat template (GGUF jinja vs unsloth vs mistralai vs HF tokenizer) | semantically equivalent (single trivial diff: a disabled `if false` assertion at line 201) | `outputs/diagnosis_chat_template.md` (and `outputs/template_*.jinja`) | +| 3 | Top-k logits agreement | top-1 token agrees on simple prompts; logit *distributions* differ noticeably (vLLM is less peaked) | `outputs/diagnosis_logits.md` | +| 4 | GGUF metadata vs HF config | match. RoPE: `factor=64`, `freq_base=1e6`, `original_context_length=4096`, `yarn_beta_fast=4`, `yarn_beta_slow=1`, `yarn_log_mul=1.0`. All YARN parameters present in GGUF and equal to `text_config.rope_parameters` | `outputs/diagnosis_config.md`, `outputs/gguf_metadata_full.txt` | +| 5 | HF transformers BF16 ground truth | **also degrades** (recall score 0/4 on the interleaved 11-turn test). T2 Flappy Bird hits 1500-token cap with looping syntax errors in tail | `outputs/hf_groundtruth_*` | +| 6 | KV cache dtype: F16 (default) / BF16 / F32 | **no effect on degradation** | `outputs/diag_single_flappy_*.json` | +| 7 | Rebuild llama.cpp with `-DGGML_CUDA_FORCE_CUBLAS=ON -DGGML_CUDA_FORCE_MMQ=OFF` and runtime `GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1` | **no effect on degradation** | `outputs/llama_server_compute32f_*.log` | +| 8 | Matched samplers (`temp=0.1, top_p=1, top_k=64, min_p=0.05, seed=42`) | **no effect**; tried `repetition_penalty ∈ {1.0,1.05,1.1,1.2}`, `frequency_penalty ∈ {0,0.1,0.3,0.5}`, `dry_multiplier ∈ {0,0.5,0.8}` — all still loop | `outputs/matched_*` | +| 9 | Architecture diff — `llama.cpp/src/models/mistral3.cpp` vs HF `transformers/models/ministral3/modeling_ministral3.py` | **equivalent**. Same SwiGLU, RMSNorm-fp32, kq_scale=1/√head_dim, optional Llama-4 attn temperature scale (gated and disabled for this model since `llama_4_scaling_beta=0`). YARN `attn_factor` chain in `llama-context.cpp` matches HF's `_compute_yarn_parameters` — both produce `1.0` for `mscale=mscale_all_dim=1.0`. | `outputs/diagnosis_arch.md` | + +## Empirical convergence point + +Single-turn `Create a Flappy Bird Python game` (greedy, temperature=0): + +``` +| backend | finish | n_out | tail snippet +| -------------------------------- | ------ | ----- | ----------- +| vLLM FP8 | stop | 1496 | "...Would you like me to explain any specific part?" +| llama-server Q8_0 cuBLAS | length | 2048 | "if __name__ == \"__main__,\\n sys.exit()\\n```\\n\\n" (loop) +| llama-server Q8_0 BF16 KV | length | 2048 | "if pipe.x < 0\\n self.pipes.remove(pipe)" (loop) +| llama-server Q8_0 F32 KV + no FA | length | 2048 | identical loop +| HF transformers BF16 | length | 1500 | "pipe1 = pipe(50, 100, 50, 100\\n pipe2 = pipe(50, 100, 50, 100" (loop) +``` + +For llama-server, output is clean up to ~1000 tokens then degrades: + +``` +mt= 600: clean +mt= 1000: still clean +mt= 1500: looping +``` + +Common prefix between vLLM and llama-server greedy outputs is exactly **66 +characters**: `# Flappy Bird Game in Python\n\nHere's a complete implementation of ` + +Then: +- vLLM picks `a Flappy Bird game using Pygame` +- llama-server picks `Flappy Bird using Pygame` (no leading `a`). + +## Working hypotheses (still open) + +- **H1: Q8_0 quantization error**. Q8_0 gives ~16-bit mantissa precision per + block of 32. Cumulative error across 88 layers × 1000 decode steps may be + enough to flip top-1 tokens that compound into loops. *Counter-evidence:* HF + BF16 with the FP8 safetensors **also** loops, so quantization can't be the + whole story. +- **H2: Numerical kernel issue specific to Mistral-Medium-3.5's shape (88 + layers, head_dim=128, head_count=96, head_count_kv=8, vocab=131072, + intermediate=28672, rope_freq_base=1e6 with YARN factor 64)** — common to + llama.cpp ggml-cuda *and* HF eager. vLLM avoids it because it uses CUTLASS + FP8 GEMMs with FP32 accumulators and possibly a different attention kernel + (`FLASH_ATTN` selected by vLLM in our setup). +- **H3: Some prefill artifact** that vLLM mitigates via chunked prefill + (`--max_num_batched_tokens 16384`). Not yet tested in isolation. + +## What is NOT the cause + +- Tokenization (4 tokenizers byte-identical). +- Chat template (4 templates render to identical token streams). +- GGUF metadata (all numerical config matches HF, including all YARN params). +- Sampler (no setting of temp/top_p/top_k/min_p/repetition_penalty/freq_penalty/dry that breaks the loop). +- KV cache dtype. +- Flash attention vs default attention in llama.cpp. +- cuBLAS vs MMQ kernels. +- Architecture code (mistral3.cpp implements the same residual/RMSNorm/SwiGLU/RoPE flow as HF). +- Sliding window (None for this model — both honour that). + +## Artefacts + +All under `workspace_5/outputs/`: + +- `diagnosis_tokenization*.{md,txt}` + `tok_ids_*.json` +- `diagnosis_chat_template.md` + `template_*.jinja` + `template_diff_*.txt` +- `diagnosis_config.md` + `gguf_metadata_full.txt` +- `diagnosis_arch.md` +- `diagnosis_logits.md` + `diagnosis_logits_raw.json` +- `hf_groundtruth_alphabet_*.txt` + `hf_groundtruth_interleaved_*.json` + `hf_groundtruth_tok_ids_*.json` +- `recall_interleaved_{vllm,llamacpp}_*.txt` +- `multi_turn_recall_{vllm,llamacpp}_*.txt` +- `diag_single_flappy_*.json` + `.log` +- `matched_*.{json,txt}` +- All llama-server / vllm / HF logs in `workspace_5/logs/`. + +## Suggested next steps for a real fix + +1. Dump per-layer activations from both vLLM and HF for the same input, and + compare layer-by-layer to find where they first diverge meaningfully. +2. If that divergence localises to RMSNorm or attention softmax, force FP32 + accumulators in those ops in llama.cpp's CUDA kernels for `mistral3`. +3. Compare against `llama-bench --perplexity` numbers per quant; Q8_0 vs Q6_K + vs F16-converted-from-FP8 GGUF to determine whether the issue scales with + precision. +4. Consider writing a reference forward pass in PyTorch using the GGUF weights + (via `gguf-py`) and the HF arch and comparing token-by-token. diff --git a/docs/diagnosis/tokenization.md b/docs/diagnosis/tokenization.md new file mode 100644 index 000000000000..709dfe6c7555 --- /dev/null +++ b/docs/diagnosis/tokenization.md @@ -0,0 +1,84 @@ +# Tokenization parity diagnosis (Mistral-Medium-3.5-128B) + +Compares token IDs produced by **four** tokenizers for the same multi-turn fixture +(system + 4 user + 3 assistant messages, `reasoning_effort='none'`). + +**Sources compared:** +- `vllm`: vLLM `POST /v1/chat/completions/render` (server on `:8765`). +- `llamacpp`: llama-server `POST /apply-template` then `POST /tokenize` with `add_special=true` (server on `:8766`). +- `mistralcommon`: `MistralTokenizer.from_hf_hub('mistralai/Mistral-Medium-3.5-128B').encode_chat_completion(...)` (mistral-common 1.11.1). +- `hf`: `AutoTokenizer.from_pretrained('mistralai/Mistral-Medium-3.5-128B').apply_chat_template(messages, tokenize=True, add_generation_prompt=True, reasoning_effort='none')['input_ids']` (transformers 5.7.0). + +## TL;DR + +**All four tokenizers produce byte-identical 434-token sequences for this fixture.** Tokenization is NOT the source of the llama-server vs vLLM accuracy gap on long conversations. Special-token boundaries (``, `[SYSTEM_PROMPT]`, `[/SYSTEM_PROMPT]`, `[MODEL_SETTINGS]`, `[/MODEL_SETTINGS]`, `[INST]`, `[/INST]`, ``) all land in the same positions across all four. The `` count is exactly 3 — one per assistant turn — in every source. The next phase should look at chat-template rendering (the small whitespace differences in the textual rendering, see file lengths), top-logits comparison, and KV-cache / kernel-level differences. + +## Token-id length per source + +| source | total tokens | rendered text length (chars) | +|---|---|---| +| vllm | 434 | 1823 | +| llamacpp | 434 | 1820 | +| mistralcommon | 434 | 1823 | +| hf | 434 | 1823 | + +Note: `llamacpp` text length differs by 3 chars vs the other three (1820 vs 1823) — purely the difference between the rendered jinja text returned by `/apply-template` (which omits the BOS string ``) vs the detokenize-of-IDs which renders BOS as a 3-char literal. The IDs themselves are identical. + +## Pairwise diff (vs vLLM as reference) + +| source | identical to vllm? | first divergence index | +|---|---|---| +| vllm | (reference) | - | +| llamacpp | **YES** | n/a | +| mistralcommon | **YES** | n/a | +| hf | **YES** | n/a | + +## Special-token detection (resolved via llama-server `/tokenize`) + +| token | id | +|---|---| +| `` | 1 | +| `` | 2 | +| `[SYSTEM_PROMPT]` | 17 | +| `[/SYSTEM_PROMPT]` | 18 | +| `[INST]` | 3 | +| `[/INST]` | 4 | +| `[MODEL_SETTINGS]` | 36 | +| `[/MODEL_SETTINGS]` | 37 | + +## Special-token counts per source + +| source | `` | `` | `[SYSTEM_PROMPT]` | `[/SYSTEM_PROMPT]` | `[INST]` | `[/INST]` | `[MODEL_SETTINGS]` | `[/MODEL_SETTINGS]` | +|---|---|---|---|---|---|---|---|---| +| vllm | 1 | 3 | 1 | 1 | 4 | 4 | 1 | 1 | +| llamacpp | 1 | 3 | 1 | 1 | 4 | 4 | 1 | 1 | +| mistralcommon | 1 | 3 | 1 | 1 | 4 | 4 | 1 | 1 | +| hf | 1 | 3 | 1 | 1 | 4 | 4 | 1 | 1 | + +All four sources show: 1 ``, 3 `` (= one per assistant turn), 1 `[SYSTEM_PROMPT]` / `[/SYSTEM_PROMPT]` pair, 1 `[MODEL_SETTINGS]` / `[/MODEL_SETTINGS]` pair, 4 `[INST]` / `[/INST]` pairs (= one per user turn including the open-ended trailing turn). + +## First 16 input ids per source (for visual sanity) + +- **vllm**: `[1, 17, 4568, 1584, 42301, 2784, 55668, 1032, 1051, 1046, 1053, 1044, 1261, 43520, 26242, 11512]` +- **llamacpp**: `[1, 17, 4568, 1584, 42301, 2784, 55668, 1032, 1051, 1046, 1053, 1044, 1261, 43520, 26242, 11512]` +- **mistralcommon**: `[1, 17, 4568, 1584, 42301, 2784, 55668, 1032, 1051, 1046, 1053, 1044, 1261, 43520, 26242, 11512]` +- **hf**: `[1, 17, 4568, 1584, 42301, 2784, 55668, 1032, 1051, 1046, 1053, 1044, 1261, 43520, 26242, 11512]` + +## Last 16 input ids per source + +- **vllm**: `[4176, 24897, 32196, 17616, 5079, 1034, 2, 3, 7493, 1395, 1032, 1050, 1043, 1050, 1063, 4]` +- **llamacpp**: `[4176, 24897, 32196, 17616, 5079, 1034, 2, 3, 7493, 1395, 1032, 1050, 1043, 1050, 1063, 4]` +- **mistralcommon**: `[4176, 24897, 32196, 17616, 5079, 1034, 2, 3, 7493, 1395, 1032, 1050, 1043, 1050, 1063, 4]` +- **hf**: `[4176, 24897, 32196, 17616, 5079, 1034, 2, 3, 7493, 1395, 1032, 1050, 1043, 1050, 1063, 4]` + +## Files + +- `outputs/diagnosis_tokenization_{vllm,llamacpp,mistralcommon,hf}.txt` +- `outputs/tok_ids_{vllm,llamacpp,mistralcommon,hf}.json` +- `scripts/diagnosis_tokenization.py` + +## Conclusion + +Tokenization is fully consistent across all four implementations. **Cross off** "tokenization difference" as a hypothesis for the long-context degradation in llama-server / Q8_0 GGUF. + +Likely culprits remain: chat-template subtleties (whitespace), kernel-level numerics in llama.cpp (FP16 accumulation, Q8_0 quantization error compounding), KV-cache dtype, sampler defaults, or the GGUF metadata (RoPE scaling, etc.). From e617ca3a4ddcbef48ef0d5cad8e5c87e39a97ebe Mon Sep 17 00:00:00 2001 From: Daniel Han Date: Fri, 1 May 2026 11:24:01 +0000 Subject: [PATCH 2/2] docs(mistral3): pin down first-divergence token and add reproducer scripts Adds docs/diagnosis/first_divergence.md and the two reproducer scripts that exercise the OpenAI Chat Completions API on the running vLLM (8765) and llama-server (8766) endpoints. Findings (greedy temperature=0 on "Create a Flappy Bird Python game" with the Mistral-Medium-3.5 SYSTEM_PROMPT, reasoning_effort=none): - The two servers produce a byte-identical 13-token prefix: "# Flappy Bird Game in Python\n\nHere's a complete implementation of" - At token 14, the top-2 tokens (' a' and ' Fl') are the same on both, but their relative ranking is flipped: * vLLM: ' a' -0.314, ' Fl' -1.314 * llama-server:' Fl' -0.289, ' a' -1.430 - Both paths are individually coherent for ~600-1000 generated tokens; only the llama-server path degenerates after that into broken syntax and repetition. - When vLLM's full output is fed as a fixed prefix to llama-server at checkpoints 50/200/500/1000/1400, llama-server's top-1 next-token AGREES with vLLM's at every checkpoint. So the degeneration is not the model picking a different answer given the same context - it's the cumulative effect of one early ~1-nat ranking flip pushing llama-server's trajectory into a different (eventually degenerate) attractor. - The same divergence pattern is observed on Q4_K_M (74 GiB GGUF), with even worse downstream syntax garbage. So this is uniform across llama.cpp quants and not a Q8_0-specific issue. Hypothesis: ggml-cuda's matmul accumulator precision for the dequant+matmul path on this model's shape (88L, 96H, 8 KV-H, head_dim=128, vocab=131072, intermediate=28672, rope_freq_base=1e6 with YARN factor=64) yields logits that are subtly flatter than the FP8 reference, and the flatness manifests as a top-2 ranking flip on close calls. GGML_CUDA_FORCE_CUBLAS=1 was already tried with no effect; the FP32 compute mode only helps the FP16 cuBLAS path, not Q8_0 dequant + matmul. Targeted fix would be FP32 accumulators in mmq.cu / mmvq.cu specifically for LLM_ARCH_MISTRAL3. --- docs/diagnosis/first_divergence.md | 88 +++++++++++ docs/diagnosis/repro_first_divergence.py | 113 ++++++++++++++ docs/diagnosis/repro_logits_progression.py | 166 +++++++++++++++++++++ 3 files changed, 367 insertions(+) create mode 100644 docs/diagnosis/first_divergence.md create mode 100644 docs/diagnosis/repro_first_divergence.py create mode 100644 docs/diagnosis/repro_logits_progression.py diff --git a/docs/diagnosis/first_divergence.md b/docs/diagnosis/first_divergence.md new file mode 100644 index 000000000000..e1602457db1b --- /dev/null +++ b/docs/diagnosis/first_divergence.md @@ -0,0 +1,88 @@ +# First-divergence pin-down (vLLM vs llama-server, greedy) + +Reproducer: `repro_first_divergence.py` in this directory. + +Same input ("Create a Flappy Bird Python game" + Mistral-Medium-3.5 SYSTEM_PROMPT +with `reasoning_effort=none`), greedy (temperature=0.0), max_tokens=100, on: + +- vLLM 0.20.1rc1.dev127 FP8 + `--attention-backend FLASH_ATTN` (port 8765) +- llama-server (b1-aab68217b unslothai fork, also tested ggml-org master) Q8_0 (port 8766) + +The two outputs share an **identical 13-token prefix**: + +``` +# Flappy Bird Game in Python\n\nHere's a complete implementation of +``` + +Token 14 is where they first diverge. **The top-2 tokens at this position are +the same in both servers**, but their *relative order* is flipped: + +| rank | vLLM | logprob | llama-server | logprob | +| --- | --- | --- | --- | --- | +| 1 | ` a` | **−0.314** | ` Fl` | **−0.289** | +| 2 | ` Fl` | −1.314 | ` a` | −1.430 | +| 3 | ` the` | −7.689 | ` the` | −4.434 | +| 4 | `Fl` | −12.564 | `Fl` | −11.420 | + +Both have the same top-2 candidates, but llama-server compresses ` a` from +−0.31 to −1.43 (a Δ of ~1.1 in logprob, i.e. a factor of 3 in probability) +while vLLM compresses ` Fl` from −1.31 to −0.31 (same Δ in the other +direction). On a logprob scale of −0.3 vs −1.4 these tokens are within 1 nat +of each other; greedy-decoding *must* pick exactly one, and the order flip +sends the two backends down different trajectories. + +After that single token flip: +- vLLM continues with `... a Flappy Bird game using Pygame. This version includes the core mechanics ...` +- llama-server continues with `... Flappy Bird using Pygame. This version includes all the classic elements ...` + +Both trajectories are coherent at this point. The two outputs both produce +clean code for ~600–1000 generated tokens, after which **only the +llama-server trajectory** degenerates into broken syntax and repetition (e.g. +`pygame.display.set mode((800, 600)` — `set_mode` becomes `set mode`). + +## Logits progression at fixed prefix + +Reproducer: `repro_logits_progression.py`. + +When we take vLLM's full greedy output and **feed it as a fixed prefix** at +checkpoints 50/200/500/1000/1400, both servers' top-1 next-token agrees at +*every* checkpoint: + +| n_decoded | vLLM top-1 | vLLM logprob | llama-server top-1 | llama-server logprob | match | +| --- | --- | --- | --- | --- | --- | +| 50 | ` ``` ` | −0.0233 | ` ``` ` | −0.0000 | ✓ | +| 200 | `Y` | ~0 | `Y` | ~0 | ✓ | +| 500 | `.rect` | −0.0001 | `.rect` | −0.2432 | ✓ | +| 1000 | `_p` | ~0 | `_p` | −0.0601 | ✓ | +| 1400 | ` ` | ~0 | ` ` | −0.0028 | ✓ | + +So the long-context degeneration is **not** caused by the model converging on +different next-token answers given the same prefix. It is caused by a single +~1-nat precision flip near the start, after which vLLM and llama-server walk +different (still individually plausible) decoding paths — and the +llama-server path happens to land in a degenerate attractor. + +## Cross-check on Q4_K_M + +Repeating the experiment with `bartowski/.../Q4_K_M` (~74 GiB GGUF) on +llama-server: identical degeneration tail. The same wrong top-2 ranking at +token 14 occurs, then the trajectory degenerates *more* than Q8_0 — for +example `pygame.display.set mode sdl hWSIZER, sdl lg2` syntax garbage. So +this is uniform across llama.cpp quants, not a Q8_0-specific bug. + +## Implication + +The remaining hypothesis is that ggml-cuda's accumulator precision in the +matmul or attention path for this specific model shape (88 layers, +head_count=96, head_count_kv=8, head_dim=128, vocab=131072, +intermediate=28672, rope_freq_base=1e6 with YARN factor=64) is producing +logits that are subtly *flatter* than the reference. The ranking flip on +token 14 is consistent with this: top-2 tokens are within 1 nat in both +servers, but llama-server compresses the gap by ~0.4 nat in a way that puts +` Fl` ahead of ` a` instead of behind. + +A targeted next step is to fix the matmul accumulator in +`ggml-cuda/mmq.cu` (or the relevant GEMM path) to FP32 specifically for +`LLM_ARCH_MISTRAL3` and re-test. `GGML_CUDA_FORCE_CUBLAS=1` was already tried +without effect, but only `GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1` only helps the +cuBLAS half-precision path; Q8_0 dequant + matmul takes a different path. diff --git a/docs/diagnosis/repro_first_divergence.py b/docs/diagnosis/repro_first_divergence.py new file mode 100644 index 000000000000..f144f47c271d --- /dev/null +++ b/docs/diagnosis/repro_first_divergence.py @@ -0,0 +1,113 @@ +"""Find the EXACT token position where vLLM and llama-server's greedy outputs first disagree. + +Generates greedy outputs from both, then walks token-by-token to find the first +mismatch. At that mismatch, prints both servers' top-20 logprobs at that position. +""" +import json +from datetime import datetime, timedelta +from pathlib import Path + +import requests +from huggingface_hub import hf_hub_download +from openai import OpenAI + + +def load_system_prompt() -> str: + p = hf_hub_download(repo_id="mistralai/Mistral-Medium-3.5-128B", filename="SYSTEM_PROMPT.txt") + raw = Path(p).read_text() + today = datetime.today().strftime("%Y-%m-%d") + yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") + return raw.format(name="Mistral-Medium-3.5-128B", today=today, yesterday=yesterday) + + +SYSTEM_PROMPT = load_system_prompt() +MESSAGES = [ + {"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": "Create a Flappy Bird Python game"}, +] + + +def greedy_with_logprobs(base_url: str, max_tokens: int = 100): + client = OpenAI(api_key="EMPTY", base_url=base_url) + model = client.models.list().data[0].id + r = client.chat.completions.create( + model=model, + messages=MESSAGES, + temperature=0.0, + max_tokens=max_tokens, + logprobs=True, + top_logprobs=20, + extra_body={"reasoning_effort": "none"}, + ) + return r + + +print("Querying vLLM (max 100 tokens)...") +v = greedy_with_logprobs("http://localhost:8765/v1", max_tokens=100) +print("Querying llama-server (max 100 tokens)...") +l = greedy_with_logprobs("http://localhost:8766/v1", max_tokens=100) + +vt = [c.token for c in v.choices[0].logprobs.content] +lt = [c.token for c in l.choices[0].logprobs.content] +v_top = [c.top_logprobs for c in v.choices[0].logprobs.content] +l_top = [c.top_logprobs for c in l.choices[0].logprobs.content] + +print(f"vLLM produced {len(vt)} tokens") +print(f"llama produced {len(lt)} tokens") + +# vLLM tokens may be in 'token_id:N' form; llama in piece form. Compare via piece by detokenizing vLLM's IDs. +def piece_for_vllm_token(t): + if t.startswith("token_id:"): + try: + tid = int(t.split(":", 1)[1]) + r = requests.post("http://localhost:8766/detokenize", json={"tokens": [tid]}, timeout=30).json() + return r["content"] + except Exception: + return t + return t + + +# Walk for first divergence by piece string. +i = 0 +n = min(len(vt), len(lt)) +while i < n: + vp = piece_for_vllm_token(vt[i]) + lp = lt[i] + if vp != lp: + break + i += 1 + +print(f"\nFirst divergence at decoded position {i}") +if i < n: + vp = piece_for_vllm_token(vt[i]) + lp = lt[i] + print(f" vLLM picked: {vp!r} (raw token: {vt[i]!r})") + print(f" llama picked: {lp!r}") + + # Show top-20 at divergence position + print("\nvLLM top-20 at this position:") + for tlp in (v_top[i] or [])[:10]: + piece = piece_for_vllm_token(tlp.token) + print(f" {piece!r:<25} {tlp.logprob:.4f}") + + print("\nllama top-10 at this position:") + for tlp in (l_top[i] or [])[:10]: + print(f" {tlp.token!r:<25} {tlp.logprob:.4f}") + + # Also: prefix that was generated identically up to this point + if i > 0: + # detokenize identical prefix from llama's piece list + prefix = "".join(lt[:i]) + print(f"\nIdentical prefix ({len(prefix)} chars):\n{prefix!r}") +else: + print("No divergence in the first 100 tokens.") + +out = Path(f"outputs/diag_first_divergence_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json") +out.write_text(json.dumps({ + "vllm_tokens": vt, + "llama_tokens": lt, + "first_divergence_index": i, + "vllm_top20_at_divergence": [(t.token, t.logprob) for t in (v_top[i] if i < len(v_top) else [])] if i < n else [], + "llama_top20_at_divergence": [(t.token, t.logprob) for t in (l_top[i] if i < len(l_top) else [])] if i < n else [], +}, indent=2)) +print(f"\nsaved {out}") diff --git a/docs/diagnosis/repro_logits_progression.py b/docs/diagnosis/repro_logits_progression.py new file mode 100644 index 000000000000..741c0d45b119 --- /dev/null +++ b/docs/diagnosis/repro_logits_progression.py @@ -0,0 +1,166 @@ +"""Compare top-k logprobs from vLLM and llama-server at multiple decode positions. + +Drives the conversation forward in lockstep: at each step, take the assistant +text generated by vLLM up to position N, feed it as a fixed prefix to both +servers via the chat history, and ask each for top_logprobs at the next token. +Quantifies how much the two distributions diverge as the decode position +grows. +""" +import json +import os +from datetime import datetime, timedelta +from pathlib import Path + +from huggingface_hub import hf_hub_download +from openai import OpenAI + + +def load_system_prompt() -> str: + p = hf_hub_download(repo_id="mistralai/Mistral-Medium-3.5-128B", filename="SYSTEM_PROMPT.txt") + raw = Path(p).read_text() + today = datetime.today().strftime("%Y-%m-%d") + yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") + return raw.format(name="Mistral-Medium-3.5-128B", today=today, yesterday=yesterday) + + +SYSTEM_PROMPT = load_system_prompt() +USER_PROMPT = "Create a Flappy Bird Python game" + + +def topk_at_prefix(base_url: str, prefix_text: str, top_k: int = 20) -> list: + """Get top-k logprobs for the next token after assistant has produced prefix_text.""" + client = OpenAI(api_key="EMPTY", base_url=base_url) + model = client.models.list().data[0].id + msgs = [ + {"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": USER_PROMPT}, + ] + # Trick: use assistant turn with `partial_assistant_message`/prefix isn't standard. + # Instead, render a continuation completion via /v1/completions on the raw text. + # vLLM accepts /v1/completions; llama-server too. + return None # placeholder + + +# Different approach: do a vLLM-only full greedy decode of 1500 tokens, then for +# each checkpoint position, query both servers' top_logprobs at that token. + +def main(): + OUT = Path("outputs/diag_logits_progression.json") + OUT.parent.mkdir(parents=True, exist_ok=True) + + # Step 1: get vLLM's full greedy decode. + vllm_client = OpenAI(api_key="EMPTY", base_url="http://localhost:8765/v1") + vllm_model = vllm_client.models.list().data[0].id + print(f"vLLM model: {vllm_model}") + + base_msgs = [ + {"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": USER_PROMPT}, + ] + r = vllm_client.chat.completions.create( + model=vllm_model, + messages=base_msgs, + temperature=0.0, + max_tokens=1500, + logprobs=True, + top_logprobs=20, + extra_body={"reasoning_effort": "none", "return_tokens_as_token_ids": True}, + ) + vllm_full_text = r.choices[0].message.content or "" + print(f"vLLM total tokens: {r.usage.completion_tokens}") + print(f"vLLM finish: {r.choices[0].finish_reason}") + print(f"vLLM head: {vllm_full_text[:80]!r}") + + # Step 2: at checkpoints (token 50, 200, 500, 1000, 1400), reconstruct the + # same prefix and query llama-server's top-20. + checkpoints = [50, 200, 500, 1000, 1400] + samples = [] + + # We need to send a partial assistant turn. The OpenAI chat API doesn't directly + # support that. Workaround: use /v1/completions with the raw rendered prompt, + # which BOTH servers expose, by calling render endpoints to assemble. + import requests + + # Re-fetch vLLM's logprobs.content list — these are token-level entries. + if r.choices[0].logprobs is None: + print("no logprobs from vllm") + return + vllm_token_entries = r.choices[0].logprobs.content + # Each entry has .token (formatted as "token_id:NN" since we asked) and .logprob and .top_logprobs + + # Get the rendered prompt as text (chat template applied) + base_render = requests.post( + "http://localhost:8766/apply-template", + json={"messages": base_msgs}, + timeout=60, + ).json()["prompt"] + # This is the raw text that, after BOS, becomes the input + + # Get a list of token IDs and decoded text per cumulative output position. + # We accumulate detokenized characters position by position. + cumulative_text_at = {} + cur_ids = [] + for i, tok in enumerate(vllm_token_entries): + # tok.token looks like "token_id:1035" + if not tok.token.startswith("token_id:"): + continue + try: + tid = int(tok.token.split(":", 1)[1]) + except Exception: + continue + cur_ids.append(tid) + if (i + 1) in checkpoints: + # Detokenize up to this point via llama-server (since vLLM detokenize is at /detokenize too) + txt = requests.post( + "http://localhost:8766/detokenize", + json={"tokens": cur_ids}, + timeout=60, + ).json()["content"] + cumulative_text_at[i + 1] = txt + + print(f"Captured {len(cumulative_text_at)} checkpoints: {list(cumulative_text_at)}") + + for n_decoded, prefix_text in cumulative_text_at.items(): + # Construct a /v1/completions prompt that mimics the chat completion's KV state. + # The chat-template-rendered text + the assistant-side prefix. + # For Mistral chat template, after [INST] xxx [/INST], the assistant emits text + # without a leading "[ASST]" tag, so we just concatenate. + full_prompt = base_render + prefix_text + + # Query both servers for next-token top-20 logprobs via /v1/completions. + sample = {"n_decoded": n_decoded, "prefix_chars": len(prefix_text)} + for label, base in [("vllm", "http://localhost:8765/v1"), ("llama", "http://localhost:8766/v1")]: + try: + rr = requests.post( + base + "/completions", + json={ + "model": "mistralai/Mistral-Medium-3.5-128B" if label == "vllm" else "mistralai_Mistral-Medium-3.5-128B-Q8_0-00001-of-00004.gguf", + "prompt": full_prompt, + "max_tokens": 1, + "temperature": 0.0, + "logprobs": 20, + }, + timeout=120, + ).json() + # parse + if "choices" in rr and rr["choices"]: + ch = rr["choices"][0] + lp = ch.get("logprobs") or {} + top = lp.get("top_logprobs") or [] + if isinstance(top, list) and top: + # top is a list of dicts {token_str: logprob} + sample[label + "_top20"] = top[0] + sample[label + "_first_token"] = (lp.get("tokens") or [None])[0] + else: + sample[label + "_raw"] = rr + except Exception as e: + sample[label + "_error"] = str(e) + samples.append(sample) + print(f"checkpoint {n_decoded}: {sample.get('vllm_first_token')!r} vs {sample.get('llama_first_token')!r}") + + OUT.write_text(json.dumps({"vllm_full_text": vllm_full_text, "checkpoints": samples}, indent=2)) + print(f"saved {OUT}") + + +if __name__ == "__main__": + main()