- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
For the setting where the output head (vocabulary projection V×d, V up to 256k) resides in far memory (outside HBM; pinned host memory in the experiments) — where a dense exact decoder must read the whole matrix every step — the paper keeps only a compact "certificate" resident on the accelerator (groupwise INT8 codes + outward-rounded residual norms) and fetches a row's BF16 original only when that row can still win, thereby making retrieval sparse without changing the returned token. Key design: (1) a canonical score — exact inner product followed by one round-to-nearest-even BF16 — defining a deterministic reference independent of parallel reduction order; (2) a groupwise residual certificate — per-group INT8 quantization with FP32 scales, storing (scale, weight-residual norm, reconstruction norm), where a per-group Cauchy-Schwarz gives an interval [L,U] containing the true score, with groupwise bounds ≤ global; (3) tie-aware screening — score one seed row exactly (max L), discard any row whose U cannot beat the seed under the smallest-index tie rule, fetch and exactly evaluate the rest; Proposition 2 proves the screened decision equals the full-vocabulary decision and, by induction, yields identical token paths; (4) an extension to fixed-counter Gumbel sampling (a monotone transform, so intervals still contain the processed score). The primary operating point G512 (group width 512) is resident at 51.2% of a dense BF16 head. Results: on four primary models, G512 exactly evaluates only 0.005%–2.190% of rows on average with zero failures over 10,240 decisions; 6 models + 360 prompts + 79,432 token comparisons all match; the per-row certificate is far sparser than a centroid-radius control (which retains 99.77–99.89% of rows) and 4.7–29.7× sparser than a 5-bit W5 control. Request time drops 21.2%–77.8% vs. the paper's own canonical dense decoder, but stock BF16 (optimized matmul) is still 1.44–2.71× faster than the method. It removes ~49% of resident head bytes (290–548 MiB), yet full-request capacity ratio is only 0.917–1.008 (no consistent gain).
Critical strengths
- Provably exact far-memory decoding — a clean, novel theoretical contribution. Unlike approximate head compression (ARCHead/FlashHead/VQ-Logits) or locally-certified pruning that only preserves a local winner (CSV-Decode), this work provably reproduces the same token and even the full path (including fixed-counter Gumbel) via a reduction-order-independent canonical score + groupwise Cauchy-Schwarz certificate + tie-aware screening, and the per-row certificate makes retrieval genuinely sparse (0.005%–2.190%, zero failures).
Critical weaknesses
- The practical payoff is weak: it only beats a slow canonical reference, not the realistic stock path, and does not improve capacity. The "21.2%–77.8% faster" is relative to the paper's own canonical exact decoder (correctly-rounded full-vocabulary evaluation, itself far slower than stock matmul); yet stock BF16 remains 1.44–2.71× faster than the method — i.e., in realistic terms it is slower. Its ~49% head-byte saving (290–548 MiB) also does not translate into capacity gains (0.917–1.008×), because backbone/KV/staging dominate at the capacity frontier. In other words, the regime where the method actually helps (head residency is the binding constraint + bit-exact path identity is required + candidate tails fit the far-memory budget) mostly does not hold in the paper's own measurements.
Questions for authors
- (Clarification, Key) Given that your own capacity results show backbone/KV dominate, can you demonstrate a realistic deployment where head residency is genuinely the binding capacity constraint, so that the 49% head saving translates into capacity/throughput gains? Otherwise, does the problem the method solves rarely bind in practice?
4 comments
Dear Reviewer qHg8,
Your key question was whether head residency ever binds in a real deployment. Here is the short version of our reply above. The answer is yes, within a narrow range on the model we measured.
On Qwen3-32B with an 80 GB A100 and no artificial memory limit, stock inference with the head on the GPU runs out of memory at batch 122 for context 512 and at batch 34 for context 2048. This holds even when the head is loaded only after prefill. Our decoder completes both batches. The only stock option that still runs there streams the head from host memory at every step, and ours decodes 1.67 to 1.92 times faster than that. The gain in batch size is small, from 121 to 124 and from 33 to 34, because the head is under 2% of an 80 GB device. The paper says this plainly.
The second part of the answer is cost. When the head fits, our requests take 0.5% to 0.7% longer than stock on this model. The 1.44 to 2.71 times you cited from the submission no longer describes the method.
One point deserves a clearer statement than we gave it. Exactness is defined against a canonical reference, and you may ask how far that is from stock matmul. On the two models where we checked, the canonical token equals the stock token on all 640 held-out greedy queries (Table 35). Stock itself is not one fixed reference. At batch 18 with greedy decoding, stock with the head streamed in tiles produces a different token path from stock with the head on the GPU, and our path matches the latter (Table 33). Tiling changes the order of the floating-point sum. This is the reason the paper fixes one reference and proves identity to it.
The discussion closes on 18 November. If this does not settle your concern, please tell us what evidence would, and we will reply promptly.
Thank you—this substantially settles my main concern. The Qwen3-32B placement data, the 0.5–0.7% overhead when the head fits, and the 640-query canonical-vs-stock check address the core practical-value and exactness objections. I will raise my score to 3 (weak accept).
Thank you for reading the new measurements and for raising your rating. We are glad that the Qwen3-32B placement data, the overhead when the head fits, and the 640-query check against stock answered your concern.
The 640-query check currently sits in Appendix P (Table 35). In the final version we will state it in the main text next to the deployment results, because it answers a question a reader will ask early.
Thank you for the careful summary and for a key question that we agree the submission did not answer. "Resident stock" and "streamed stock" are defined in the General Response; new material is blue in the revised PDF.
Key question, part 1: can head residency be shown to bind in a real deployment?
Yes. Qwen3-32B on an A100 80GB, prefill plus 64 decode steps, physical memory with no artificial cap. Resident stock is given its best case: the head is loaded only after prefill, so it never competes with the prefill peak. All placements share one head-free prefill schedule (Tables 31-33).
Key question, part 2: does the problem rarely bind in practice?
At 32B on 80GB, the window is narrow, and we say so. The dense head is 1.45 GiB, under 2% of device memory, and the gain is one to three sequences. Streamed stock reaches the same or one larger batch; our advantage over it is decode speed, not capacity. Our peak allocation at the boundary is also higher than streamed stock's (78.0-78.3 vs 77.0-77.6 GiB).
Two measured facts make the method worth having despite the narrow window:
The window scales with the head's share of memory. With the condition Hc+Sc≤R<H+Ss (R: memory left after backbone and cache; H,Hc: dense head and certificate; S: workspace), its width is the 49% head saving minus the extra workspace. New Table 27 restates submitted Table 6 (now Table 7): the head is 5.9%-26.1% of the weights of the six evaluated models, and 18.8%-58.6% if the backbone were stored at 4 bits with a BF16 head. This is parameter accounting, not a measurement, and we label it so. It indicates that 32B is the unfavorable end for a head-side saving, and it is the end we measured.
W1: "it only beats a slow canonical reference, not the stock path."
This was true of the submission. The revision compares against stock in three placements, above. The 1.44-2.71x figure came from the unfused prototype; with native kernels, 64-step requests cost +3.2% on Qwen3-0.6B (2.753 s vs 2.667 s) and +0.5% to +0.7% on Qwen3-32B relative to resident stock. In isolation the stock head kernel is still 2.3-10x faster; the request gap is small because the head is a small part of each decode step.
Caveats in the paper. At the boundary batches, stock runs generate different tokens than ours, since stock matmul is a different rounding program; prompts, batch and length are identical. At batch 18 with T=0.7 all placements generate identical tokens. The shared prefill schedule was calibrated once (4.35 GPU-hours, reported separately); every request time includes its actual prefill.
Revised claim: an exact operating point between resident and streamed stock, within 1% of the former when the head fits and 1.67-1.92x faster in decode than the latter when it does not. We hope this direct answer addresses your concern, and we would be grateful if you would reconsider your rating.