acceptodds
Under review as a conference paper at ICLR 2027

Exact Vocabulary Decoding from Far Memory with Groupwise Residual Certificates

Abstract

An autoregressive language model scores every vocabulary row but returns one token. When the output head resides outside accelerator memory, a dense projection must read the full matrix at each step. We show how to preserve the exact decision while retrieving only rows whose upper bounds can beat one exactly evaluated row. Our decoder keeps grouped integer representations and residual error bounds on the accelerator, while the original bfloat16 (BF16) rows remain in host memory. Integer dot products and outward rounding give intervals that contain the correctly rounded BF16 scores. A tie-aware screening rule selects the rows to retrieve and preserves greedy and fixed-counter Gumbel decisions. With groups of 512 coordinates, the resident representation occupies 51.2% of a dense BF16 output head and requires exact evaluation of only 0.005% to 2.190% of rows on average across four primary models, with no failure in 10,240 decisions. Six-model tests and paired generation paths also show no token mismatch. Request time falls by 21.2% to 77.8% against the same canonical dense decoder. These results show that exact offloaded decoding can use sparse row retrieval in place of full-matrix transfer.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model Inference, Exact Decoding, Memory Offloading, Certified Pruning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_qHg8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

For the setting where the output head (vocabulary projection V×d, V up to 256k) resides in far memory (outside HBM; pinned host memory in the experiments) — where a dense exact decoder must read the whole matrix every step — the paper keeps only a compact "certificate" resident on the accelerator (groupwise INT8 codes + outward-rounded residual norms) and fetches a row's BF16 original only when that row can still win, thereby making retrieval sparse without changing the returned token. Key design: (1) a canonical score — exact inner product followed by one round-to-nearest-even BF16 — defining a deterministic reference independent of parallel reduction order; (2) a groupwise residual certificate — per-group INT8 quantization with FP32 scales, storing (scale, weight-residual norm, reconstruction norm), where a per-group Cauchy-Schwarz gives an interval [L,U] containing the true score, with groupwise bounds ≤ global; (3) tie-aware screening — score one seed row exactly (max L), discard any row whose U cannot beat the seed under the smallest-index tie rule, fetch and exactly evaluate the rest; Proposition 2 proves the screened decision equals the full-vocabulary decision and, by induction, yields identical token paths; (4) an extension to fixed-counter Gumbel sampling (a monotone transform, so intervals still contain the processed score). The primary operating point G512 (group width 512) is resident at 51.2% of a dense BF16 head. Results: on four primary models, G512 exactly evaluates only 0.005%–2.190% of rows on average with zero failures over 10,240 decisions; 6 models + 360 prompts + 79,432 token comparisons all match; the per-row certificate is far sparser than a centroid-radius control (which retains 99.77–99.89% of rows) and 4.7–29.7× sparser than a 5-bit W5 control. Request time drops 21.2%–77.8% vs. the paper's own canonical dense decoder, but stock BF16 (optimized matmul) is still 1.44–2.71× faster than the method. It removes ~49% of resident head bytes (290–548 MiB), yet full-request capacity ratio is only 0.917–1.008 (no consistent gain).

Critical strengths
  1. Provably exact far-memory decoding — a clean, novel theoretical contribution. Unlike approximate head compression (ARCHead/FlashHead/VQ-Logits) or locally-certified pruning that only preserves a local winner (CSV-Decode), this work provably reproduces the same token and even the full path (including fixed-counter Gumbel) via a reduction-order-independent canonical score + groupwise Cauchy-Schwarz certificate + tie-aware screening, and the per-row certificate makes retrieval genuinely sparse (0.005%–2.190%, zero failures).
Critical weaknesses
  1. The practical payoff is weak: it only beats a slow canonical reference, not the realistic stock path, and does not improve capacity. The "21.2%–77.8% faster" is relative to the paper's own canonical exact decoder (correctly-rounded full-vocabulary evaluation, itself far slower than stock matmul); yet stock BF16 remains 1.44–2.71× faster than the method — i.e., in realistic terms it is slower. Its ~49% head-byte saving (290–548 MiB) also does not translate into capacity gains (0.917–1.008×), because backbone/KV/staging dominate at the capacity frontier. In other words, the regime where the method actually helps (head residency is the binding constraint + bit-exact path identity is required + candidate tails fit the far-memory budget) mostly does not hold in the paper's own measurements.
Questions for authors
  1. (Clarification, Key) Given that your own capacity results show backbone/KV dominate, can you demonstrate a realistic deployment where head residency is genuinely the binding capacity constraint, so that the 49% head saving translates into capacity/throughput gains? Otherwise, does the problem the method solves rarely bind in practice?
4 comments

Thank you for the careful summary and for a key question that we agree the submission did not answer. "Resident stock" and "streamed stock" are defined in the General Response; new material is blue in the revised PDF.

Key question, part 1: can head residency be shown to bind in a real deployment?

Yes. Qwen3-32B on an A100 80GB, prefill plus 64 decode steps, physical memory with no artificial cap. Resident stock is given its best case: the head is loaded only after prefill, so it never competes with the prefill peak. All placements share one head-free prefill schedule (Tables 31-33).

SettingResident stockStreamed stockOurs
Max batch, context 512121124124
Max batch, context 2048333534
Ctx 512, batch 122, greedy: request (s)OOM75.8668.48
Ctx 512, batch 122, T=0.7OOMOOM68.59
Ctx 2048, batch 34, greedyOOM223.22214.95
Ctx 2048, batch 34, T=0.7OOM223.59215.35
Ctx 2048, batch 18 (memory ample), greedy18.3626.5018.46
  • Capacity: +2.5% and +3.0% batch over resident stock. At batches 122 and 34 resident stock fails with CUDA OOM in 3/3 runs (greedy: at decode steps 60/64 and 31/64); ours completes 3/3 in both modes.
  • Throughput at those batches: the stock option that still fits is streaming. Ours decodes 1.67-1.92x faster (40%-48% less decode time) and finishes requests 3.7%-9.7% sooner; prefill dominates these requests. Every one of our repeats is faster than every streamed repeat.

Key question, part 2: does the problem rarely bind in practice?

At 32B on 80GB, the window is narrow, and we say so. The dense head is 1.45 GiB, under 2% of device memory, and the gain is one to three sequences. Streamed stock reaches the same or one larger batch; our advantage over it is decode speed, not capacity. Our peak allocation at the boundary is also higher than streamed stock's (78.0-78.3 vs 77.0-77.6 GiB).

Two measured facts make the method worth having despite the narrow window:

  1. Outside the window it is nearly free. At batch 18, ours costs +0.5% / +0.7% request time and +2.2% / +2.8% decode time over resident stock (Table 33). A deployment can keep it enabled without a meaningful penalty.
  2. Inside and beyond the window it replaces streaming, which takes 44% longer per request at batch 18 and 1.7-1.9x longer in decode at the boundary.

The window scales with the head's share of memory. With the condition Hc+Sc≤R<H+SsH_c + S_c \le R < H + S_s (RR: memory left after backbone and cache; H,HcH, H_c: dense head and certificate; SS: workspace), its width is the 49% head saving minus the extra workspace. New Table 27 restates submitted Table 6 (now Table 7): the head is 5.9%-26.1% of the weights of the six evaluated models, and 18.8%-58.6% if the backbone were stored at 4 bits with a BF16 head. This is parameter accounting, not a measurement, and we label it so. It indicates that 32B is the unfavorable end for a head-side saving, and it is the end we measured.

W1: "it only beats a slow canonical reference, not the stock path."

This was true of the submission. The revision compares against stock in three placements, above. The 1.44-2.71x figure came from the unfused prototype; with native kernels, 64-step requests cost +3.2% on Qwen3-0.6B (2.753 s vs 2.667 s) and +0.5% to +0.7% on Qwen3-32B relative to resident stock. In isolation the stock head kernel is still 2.3-10x faster; the request gap is small because the head is a small part of each decode step.

Caveats in the paper. At the boundary batches, stock runs generate different tokens than ours, since stock matmul is a different rounding program; prompts, batch and length are identical. At batch 18 with T=0.7 all placements generate identical tokens. The shared prefill schedule was calibrated once (4.35 GPU-hours, reported separately); every request time includes its actual prefill.

Revised claim: an exact operating point between resident and streamed stock, within 1% of the former when the head fits and 1.67-1.92x faster in decode than the latter when it does not. We hope this direct answer addresses your concern, and we would be grateful if you would reconsider your rating.

Dear Reviewer qHg8,

Your key question was whether head residency ever binds in a real deployment. Here is the short version of our reply above. The answer is yes, within a narrow range on the model we measured.

On Qwen3-32B with an 80 GB A100 and no artificial memory limit, stock inference with the head on the GPU runs out of memory at batch 122 for context 512 and at batch 34 for context 2048. This holds even when the head is loaded only after prefill. Our decoder completes both batches. The only stock option that still runs there streams the head from host memory at every step, and ours decodes 1.67 to 1.92 times faster than that. The gain in batch size is small, from 121 to 124 and from 33 to 34, because the head is under 2% of an 80 GB device. The paper says this plainly.

The second part of the answer is cost. When the head fits, our requests take 0.5% to 0.7% longer than stock on this model. The 1.44 to 2.71 times you cited from the submission no longer describes the method.

One point deserves a clearer statement than we gave it. Exactness is defined against a canonical reference, and you may ask how far that is from stock matmul. On the two models where we checked, the canonical token equals the stock token on all 640 held-out greedy queries (Table 35). Stock itself is not one fixed reference. At batch 18 with greedy decoding, stock with the head streamed in tiles produces a different token path from stock with the head on the GPU, and our path matches the latter (Table 33). Tiling changes the order of the floating-point sum. This is the reason the paper fixes one reference and proves identity to it.

The discussion closes on 18 November. If this does not settle your concern, please tell us what evidence would, and we will reply promptly.

Thank you—this substantially settles my main concern. The Qwen3-32B placement data, the 0.5–0.7% overhead when the head fits, and the 640-query canonical-vs-stock check address the core practical-value and exactness objections. I will raise my score to 3 (weak accept).

Thank you for reading the new measurements and for raising your rating. We are glad that the Qwen3-32B placement data, the overhead when the head fits, and the 640-query check against stock answered your concern.

The 640-query check currently sits in Appendix P (Table 35). In the final version we will state it in the main text next to the deployment results, because it answers a question a reader will ask early.

Reviewer_e5gZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a strategy to accelerate the evaluation of the rows of the readout matrix of an autoregressive language model. Each of these rows needs to be evaluated in order to assign a score to each token of a vocabulary during the generation of the next token in a given request. The authors propose a way to evaluate only the rows that satisfy certain conditions. They calculate for each row an upper bound on the score, and if this upper bound can not exceed the highest exact score found so far, they discard the row; otherwise they evaluate it. The evaluation of the upper bound is done inside the accelerator using a compact representation INT8; instead, the exact score is computed on the accelerator using the BF16 row retrieved.

The evaluation strategy tests the proposed method on six language models, several workloads, and both greedy and Gumbel decoding.

The results are strong, with G512 preserving all tested decisions and retrieving only a very small fraction of the output rows, hence significantly reducing request time.

Critical strengths

The authors present a very simple and, at the same time, effective way to accelerate token evaluation in the readout phase of the token generation process, avoiding the evaluation of up to 99.975% of the rows of the readout matrix of an autoregressive language model, and maintaining guarantees with an exact fallback. This is, in my opinion, a relevant contribution to the practical acceleration of language model inference.

Critical weaknesses

It is not clear whether G512 is broadly SOTA among certified output-head filtering methods, rather than only superior to the selected centroid-radius baseline.

Questions for authors

I would suggest to enlarge the experimental comparison also to other certified row-filtering methods from the recent literature (if they exist). Could the author expand the experimental comparison?

Minor issues: I would suggest the author to adjust the “Hidden State” part of Figure 1, and to add a domain for the index r (i.e. r=1,…,m) in Equation (4) (even if it is clear from Equation (5)).

3 comments

Thank you for the supportive assessment and for the request to widen the comparison. New material is blue in the revised PDF; "resident stock" and "streamed stock" are defined in the General Response.

W1, Q1. Is G512 ahead of other certified row-filtering methods, beyond the centroid-radius control?

CSV-Decode is the closest certified row-filtering method with released code. We now run that released implementation directly, in place of our centroid-radius stand-in (App. L.2, P.4).

  • Setup: Euclidean bounds, zero tolerance, 2,000 clusters, and a full-vocabulary search budget so that its default work limit cannot truncate the search. 64 paired queries per model on Qwen3-0.6B and SmolLM2-135M.
  • Rows that must be evaluated: CSV-Decode keeps 99.96% / 99.99%; ours keeps 0.0017% / 0.0103% on the same queries (Tables 25, 35).
  • Why: a cluster radius must cover its farthest member, so one distant row keeps a whole cluster alive. Our bound is per row and per coordinate group. This matches the centroid-radius control in the submission (99.77%-99.89%).
  • GPU time: its head takes about 2.4 s with its faster FP32 projection, versus 1.4-1.6 ms for ours. The released code loops over tiles in Python, so we attribute this to the implementation and report both of its projection paths.

The exact-MIPS index diagnostics from the submission (now Table 13) remain. We state the scope carefully: G512 is far more selective than the certified baselines we could execute on these models and queries. We do not claim it is state of the art against every possible certified index; the paper limits the claim to these paired settings.

Uncertified baselines, for context (requested by Reviewer rhx9)

MethodGuaranteeRows evaluated (Qwen3-0.6B)Greedy decisions changed (Qwen / SmolLM2)
FlashHead, official assetsnone6.24%0/640 observed (Qwen only)
ARCHead, CUDAnoneapproximates all rows79/640 / 57/640
Ourscertified0.006%0/640 / 0/640

Certification costs little at the request level: ours is 2.3% slower than FlashHead and 0.8%-4.4% slower than ARCHead on a shared backbone (Table 36, App. P.5).

The same proof now supports faster execution. None of this changes the stored certificate.

  • A proved FP32 screen replaces FP64 with zero decision changes over 128,696,320 enclosed scores (App. K).
  • Keeping the hidden vector unquantized removes one error term and cuts the mean candidate fraction from 0.0058% to 0.0012% on Qwen3-0.6B (Table 22).
  • A fused kernel for screening and Gumbel sampling is bit-identical to the reference and 1.22-1.78x faster on sampling heads after a one-time compile (App. P).

Practical relevance, raised by the other reviewers. On Qwen3-32B with an A100 80GB and 64 decode steps, our requests cost under 1% more than a resident stock head when memory is ample. At the first batch where the resident head runs out of memory, ours completes, and it decodes 1.67-1.92x faster than streaming the head from host memory (Sec. 6.6, App. O). The request-time gap to stock on Qwen3-0.6B fell from 1.44-1.54x in the submission (unfused, 8 decode steps) to 1.03x (native kernels, 64 steps).

Minor issues

  • Figure 1: we redrew the hidden-state part. The heading now sits over the vector, and the "BF16 vector h" label no longer runs into the resident-record box. The caption also states that the incoming hidden vector is BF16 while the resident record holds INT8 codes.
  • Equation 4: now states r = 1,...,m.

Thank you again. If the expanded comparison and the deployment evidence resolve your concern, we would be grateful if you would consider raising your rating.

Reviewer_e5gZOpenReview ↗

Keep rating 3

Thank you for the detailed response and the new experiments. The direct comparison with CSV-Decode addresses my main concern, and I appreciate the fixes to Figure 1 and Eq. 4. I also note that the SOTA claim is now explicitly limited to the tested settings.

I am keeping my rating of 3. It rests on the exactness guarantee and the very sparse retrieval, which no reviewer disputes. I am aware of the concerns raised by the other reviewers: the method does not beat resident stock, the capacity window is narrow (a few sequences at 32B), and exactness holds with respect to the canonical reference rather than stock matmul. I appreciate that the revised claim states these limits openly, and I see the contribution as a certified operating point rather than a general speedup.

Thank you for reading the revision and for confirming that the CSV-Decode comparison answers your main concern. Your summary of the remaining limits is accurate. It matches what the paper now claims, which is a certified operating point and not a general speedup.

One fact may be useful on the last limit you list, that exactness holds with respect to the canonical reference rather than stock matmul. On the two models where we checked, the canonical token equals the stock token on all 640 held-out greedy queries (Table 35). Stock itself is not one fixed reference. At batch 18 with greedy decoding, stock with the head streamed in tiles produces a different token path from stock with the head on the GPU, and our path matches the latter (Table 33). Tiling changes the order of the floating-point sum, so we fix one reference and prove identity to it.

Thank you again for your time.

Reviewer_rhx9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies exact decoding with an offloaded output head. Its main contribution is a group-wise residual certificate that screens vocabulary entries before fetching their full BF16 rows, preserving greedy and fixed-counter Gumbel decisions under a canonical numerical reference. Experiments across six models report no decision mismatches while fetching on average only 0.005–2.19% of rows. The method approximately halves resident head storage and is faster than the canonical baseline, but remains slower than stock BF16 inference.

Critical strengths
  1. The exactness guarantee is carefully established. The paper accounts for finite-precision arithmetic, and reports no mismatches across six models.
  2. The screening is highly selective as only 0.005–2.19% of rows are fetched on average, compared with roughly 99.8% for the centroid-radius control.
Critical weaknesses
  1. The contribution is clear, but the practical value is not. If I understood correctly, the method targets memory-constrained inference at the cost of increased latency. I find it difficult to see a scenario where a slowdown of at most 2.71 times w.r.t. stock BF16 inference is acceptable, especially since speed is a key consideration at inference time.
  2. The related work section is quite poor, specifically sec 2.1. Moreover, I would integrate Table 20 inside related work.
  3. Some experimental baselines are missing.
Questions for authors
  1. In what realistic deployment scenario would your method be preferable to standard BF16 inference?
  2. Can you provide concrete measurements in a memory-constrained setting where the memory savings translate into a practical benefit that justifies the increased latency?
  3. Why no experimental comparison with CSV-Decode, ARCHead, or FlashHead? Please compare the proposed method with them.
4 comments

Thank you for the concrete questions. "Resident stock" and "streamed stock" are defined in the General Response; new material is blue in the revised PDF.

Q1. When is the method preferable to standard BF16 inference?

When the dense head no longer fits beside the backbone and key-value cache, and the returned tokens must not change. Standard BF16 then has two options: fail, or keep the head in host memory and stream it at every step. Our method is a third option that returns the same canonical token and decodes much faster than streaming. When the head does fit, standard BF16 remains the right choice, and our measured cost in that case is under 1% of request time.

Q2. Concrete measurements in a memory-constrained setting

Qwen3-32B on an A100 80GB, prefill plus 64 decode steps, physical memory with no artificial cap (Tables 31-33).

SettingResident stockStreamed stockOurs
Max batch, context 512 / 2048121 / 33124 / 35124 / 34
Ctx 512, batch 122, greedy: request (s)OOM75.8668.48
Ctx 2048, batch 34, greedyOOM223.22214.95
Ctx 2048, batch 18 (memory ample)18.3626.5018.46
  • At the first batch where resident stock runs out of memory, ours completes all runs in greedy and T=0.7 modes.
  • Against streamed stock at those batches, decode time is 40%-48% lower and request time 3.7%-9.7% lower (prefill dominates). At batch 18 the request reduction is 30%.
  • The latency concern (W1): the 1.44-2.71x figure came from the unfused prototype. With native kernels, requests cost +0.5% to +0.7% on Qwen3-32B and +3.2% on Qwen3-0.6B (64 steps) relative to resident stock.

Q3, W3. CSV-Decode, ARCHead, FlashHead

All three now run from their released code on Qwen3-0.6B and SmolLM2-135M (App. L, P; Tables 24-26, 35-36).

  • CSV-Decode (Euclidean bounds, zero tolerance, full search budget): it keeps 99.96% / 99.99% of rows on 64 paired queries, versus 0.0017% / 0.0103% for ours. Its GPU head takes about 2.4 s with its faster FP32 projection, versus 1.4-1.6 ms for ours. The released code loops over tiles in Python, so we read this as a statement about that implementation, not a limit of the idea.
  • ARCHead (default settings, 640 held-out queries per model): it changes 79 / 57 greedy decisions; ours changes none. On Qwen3-0.6B it holds 77.3% of dense-head bytes on the GPU including its runtime cache, versus 51.2% for ours. On SmolLM2-135M ours is larger (91.0% vs 78.9%) because d=576 is padded to two groups. Our request time is 0.8%-4.4% higher than ARCHead's.
  • FlashHead (official Qwen3-0.6B assets): it retains 6.24% of rows versus 0.006% for ours on 640 paired queries; both keep every winner on these inputs, but only ours certifies it. With its official Triton head on a shared backbone, 64-step requests take 2.692 s versus 2.753 s for ours (2.3% premium), and all nine generated paths agree.

In short: the approximate heads are slightly faster or smaller and change some decisions; the certified baseline keeps nearly the whole vocabulary on these models; ours is certified and sparse.

W2. Related work and Table 20

Sec. 2.1 is rewritten around three axes: where the head lives, which decision is preserved, and how quality is evaluated. A new main-text Table 1 summarizes submitted Table 20 (now Table 21, kept in full in App. I) and places CSV-Decode, ARCHead, FlashHead, FlashSampling and our method side by side.

What the paper claims after revision. Exact decoding with the original head in host memory needs only 0.005%-2.19% of rows per step; with native kernels this costs under 1% of request time on a 32B model and, past the memory boundary, decodes 1.67-1.92x faster than streaming the head. We do not claim to beat resident stock when the head fits.

We hope these measurements answer your questions, and we would be grateful if you would reconsider your rating.

Dear Reviewer rhx9,

Here is a short version of our reply above, in case it saves you time.

You asked when the method is preferable to standard BF16 inference. It is preferable when the output head no longer fits on the GPU next to the model and its cache. On Qwen3-32B with an 80 GB A100 this happens at batch 122 for context 512 and at batch 34 for context 2048. Standard inference with the head on the GPU runs out of memory there. Ours completes, and it decodes 1.67 to 1.92 times faster than the standard fallback of streaming the head from host memory. When the head does fit, our requests take 0.5% to 0.7% longer than standard inference on this model. The 2.71 times in the submission came from an early prototype.

You also asked for CSV-Decode, ARCHead and FlashHead. All three now run from their released code. CSV-Decode keeps more than 99.9% of the vocabulary rows, where ours keeps 0.002% and 0.010% on the same queries. ARCHead changes 79 and 57 of 640 greedy tokens on the two models. Ours changes none and takes 0.8% to 4.4% longer per request. FlashHead keeps 6.24% of the rows without a guarantee, and ours takes 2.3% longer per request than it does. Reviewer e5gZ has since confirmed that the CSV-Decode comparison answers their concern about baselines.

Section 2.1 is rewritten, and the comparison you asked to move from the appendix is now Table 1 in the main text.

The discussion closes on 18 November. If something is still missing, please tell us and we will reply promptly.

Dear Authors,

Thank you for your response. While I appreciate your clarifications, I remain unconvinced by the overall quality and practical relevance of the manuscript. I will therefore maintain my initial score.

Thank you for coming back to the discussion. We respect your decision.

So that the thread is complete, here is where each of your three questions is answered in the revision. The deployment scenario is Qwen3-32B on an 80 GB A100 at the batches where a stock head no longer fits on the GPU. The measurements for that setting are in Tables 31 to 33. CSV-Decode, ARCHead and FlashHead are compared in Tables 24 to 26, 35 and 36. Section 2.1 is rewritten, and the comparison table you asked for is now Table 1 in the main text.

If one of these answers falls short, we would be grateful to know which one. It would help us improve the paper whatever the outcome.

Reviewer_ReiTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles a well-known hardware bottleneck in LLM generation: the massive VRAM footprint of the output embedding head (LM-head) for large vocabularies. To save accelerator memory (HBM), the authors offload the authoritative bfloat16 (BF16) head matrix to host memory. To avoid the massive latency of transferring the whole matrix at every token step, they propose Groupwise Residual Certificates (G512)—storing lightweight INT8 codes and mathematical residual error bounds on the chip. By calculating score intervals and fully scoring just one seed token row, the system filters out the vast majority of dominated rows.

Critical strengths

• Guaranteed Path Identity: Unlike standard lossy compression tricks (e.g., ARCHead or VQ-Logits) that hurt perplexity, this approach provides 100% path identity for greedy and Gumbel decoding over all 10,240 tested choices.

• Extreme Retrieval Sparsity: Pruning vocabulary evaluation down to less than 2.2% of rows on average shows that row-level residual norms provide incredibly tight bounding intervals.

• Commendable Academic Honesty: The authors don't try to hide their system limits. They openly show that stock accelerator execution is still faster and that total batch capacity barely changes because KV caches dominate the hardware frontier.

Critical weaknesses

• Lacks Practical Serving Gains: While the paper claims big wins over a canonical dense offloaded baseline, it admits the method is 1.44× to 2.71× slower than stock native BF16 execution running entirely on the accelerator. It is hard to justify using a technique that more than doubles generation latency unless you are strictly out of VRAM.

• High Screening Overhead: To maintain the exactness guarantee without numerical drift, the system relies heavily on slow FP64 arithmetic for computing centers, bounds, and outward rounding. This vocabulary-wide screening completely bottlenecks the execution loop.

• Negligible End-to-End Capacity Wins: Even though the head footprint is slashed by roughly 49%, Figure 4a shows it doesn't translate to real capacity gains (max-batch ratios hover around 0.91 to 1.00). The text generation backbone and PagedAttention KV-caches simply dwarf the LM-head footprint at the operational limit.

Questions for authors
  1. The FP64 Bottleneck: If you migrate the screening math from heavy FP64 down to standard FP32, how much speedup do you gain, and how many actual token mismatches emerge from the reduced precision?

  2. Break-Even Boundary: Under what specific hardware constraints (e.g., specific batch sizes or context lengths) does G512 actually break even with or beat stock accelerator BF16 execution, if ever?

  3. Kernel Fusion: Is it possible to completely fuse the vocabulary-wide INT8 screening pass and the Gumbel reductions into a single custom GPU kernel to cut down the remaining 1.44×–2.71× overhead gap?

3 comments

Thank you for three precise questions. Each now has a measured answer. "Resident stock" and "streamed stock" are defined in the General Response; new material is blue in the revised PDF.

Q1. FP64 -> FP32: speedup and mismatches

FP64 is not needed for exactness. We derived an FP32 error budget (App. K, N) and ran both precisions on Qwen3-32B with everything else fixed.

  • Speedup: head time falls by 1.4%-10.8% across batch 1-128, greedy and T=0.7 (Table 28).
  • Mismatches: 0. On 1,280 new states the FP32 intervals enclose all 128,696,320 canonical scores and preserve all 5,120 decisions (Table 22). An independent full-vocabulary audit on 32B also has no failure.

One clarification: the vocabulary-wide pass is INT8 dot products, O(Vd). FP64 was used only for the per-group scale and bound arithmetic, O(Vm) (submitted Table 16). That is why removing it helps only modestly. The larger cost was unfused execution, which Q3 addresses.

Q3. Kernel fusion

Yes, and we built it. One Triton kernel now performs INT8 screening, counter generation, the FP64 Gumbel transform and tile reduction (App. P).

  • Correctness: bit-identical to the reference. Zero differences in counter words, noise bits and scores on 10,255 state-token pairs per temperature; 6,434,816 enclosure checks on real models pass.
  • Speed: sampling heads run 1.22-1.78x faster than our non-fused native path once the kernel is compiled (1.16-1.66x in medians that include the first pass). First use pays a one-time compile per new candidate count; Table 34 reports it without dropping any observation.
  • Separately, native query and candidate kernels speed up the complete 32B head by 1.18-2.23x (Table 30).

What this does to the 1.44-2.71x gap (W1). With native kernels, 64-step requests cost +3.2% on Qwen3-0.6B (2.753 s vs 2.667 s) and +0.5% to +0.7% on Qwen3-32B relative to resident stock. In isolation the stock head kernel is still 2.3-10x faster than ours; the request gap is small because the head is a small part of each decode step.

Q2, W3. Break-even and capacity

Setting: Qwen3-32B, A100 80GB, prefill plus 64 decode steps, physical memory with no artificial cap (Tables 31-33).

  • Break-even is where the dense head stops fitting: batch 122 at context 512 and batch 34 at context 2048. Resident stock fails there with CUDA OOM in 3/3 runs; ours completes 3/3 in greedy and T=0.7 modes.
  • Past that point the stock alternative is streaming the head. Against it, ours has 40%-48% lower decode time and 3.7%-9.7% lower request time (prefill dominates these requests). Every one of our repeats is faster than every streamed repeat.
  • Capacity: maximum batch rises from 121 to 124 (context 512) and from 33 to 34 (context 2048) over resident stock. Streamed stock reaches 124 and 35.
  • Below break-even (batch 18): +0.5% to +0.7% request time versus resident stock, and 30% less than streamed stock.

We agree that a 49% head saving is small in absolute terms at 32B (about 0.7 GiB of 80), and we do not claim a general capacity win. The claim is a measured operating point between the two stock placements: near-resident speed when the head fits, faster than streaming when it does not. The head is under 2% of device memory in this test; it is 5.9%-26.1% of the weights of the six models in the paper (new Table 27, accounting only).

A detail in W3. Our runs use Hugging Face Transformers key-value caches, not PagedAttention; App. M now says so explicitly.

These results are in Sec. 6.6 and Appendices K-P. We hope they answer your questions, and we would be grateful if you would reconsider your rating.

Dear Reviewer ReiT,

Our reply above is long, so here is a short version of the answers to your three questions.

Your first question was about moving the screen from FP64 to FP32. We did this with a proved error bound. Head time falls by 1.4% to 10.8%. The FP32 bounds contain all 128,696,320 exact scores we checked, so no token changes.

Your second question was about a break-even point. On Qwen3-32B with an 80 GB A100, a stock head on the GPU runs out of memory at batch 122 for context 512 and at batch 34 for context 2048. Our decoder completes those batches. The stock alternative there is to stream the head from host memory, and ours decodes 1.67 to 1.92 times faster than that. Below those batches our requests take 0.5% to 0.7% longer than stock on this model and 3.2% longer on Qwen3-0.6B. The submission reported 1.44 to 2.71 times.

Your third question was about fusing the screen and the Gumbel step. We built that kernel. Its output is bit-identical to the reference, and sampling heads run 1.22 to 1.78 times faster once it is compiled.

The discussion closes on 18 November. If any answer is unclear or leaves a concern open, please tell us and we will reply promptly.

Dear Reviewer ReiT,

One update before the discussion closes. Reviewer qHg8 raised the same concern about practical value as your first and third weaknesses. They have now written that the Qwen3-32B measurements and the 0.5% to 0.7% overhead when the head fits settle that concern, and they raised their rating to 3.

The short answers to your three questions are in our note above. If you have a few minutes before the close, we would value your view on whether they are sufficient.

Comments on the paper

1 comment

We thank the four reviewers. No review questions the exactness guarantee or the sparsity of retrieval (0.005%-2.19% of rows). The shared concerns are (i) practical value against stock BF16 inference and (ii) missing baselines. We ran new experiments on physical GPUs for both. All changes are blue in the revised PDF; new results are in Sec. 6.6 and Appendices K-P.

Terms used in all replies. Resident stock: dense BF16 head on the GPU; we use its strongest form, which loads the head only after prefill ("deferred" in the paper). Streamed stock: the head stays in host memory and is streamed at every step ("tiled"); this is the existing exact option when the head does not fit. Ours: the 51.2% certificate on the GPU, original rows fetched on demand.

1. Deployment: Qwen3-32B, A100 80GB, prefill + 64 decode steps, no artificial memory cap (Tables 31-33)

SettingResident stockStreamed stockOurs
Max batch, context 512121124124
Max batch, context 2048333534
Ctx 512, batch 122, greedy: request (s)OOM75.8668.48
Ctx 512, batch 122, T=0.7OOMOOM68.59
Ctx 2048, batch 34, greedyOOM223.22214.95
Ctx 2048, batch 34, T=0.7OOM223.59215.35
Ctx 2048, batch 18 (memory ample), greedy18.3626.5018.46
  • When memory is ample, ours costs +0.5% / +0.7% request time (greedy / T=0.7) and +2.2% / +2.8% decode time over resident stock. The 1.44-2.71x gap in the submission came from the unfused prototype; with native kernels the 64-step request gap on Qwen3-0.6B is 3.2% (2.753 s vs 2.667 s).
  • When memory binds, resident stock fails with CUDA OOM at batches 122 and 34, and ours completes every repeat in both modes. Against streamed stock, decode time is 40%-48% lower (1.67-1.92x decode throughput) and request time 3.7%-9.7% lower, because prefill dominates these requests; at batch 18 the request reduction is 30%.

The method is therefore an operating point between the two stock placements: near-resident speed when the head fits, and faster decoding than streaming when it does not.

2. Baselines requested by rhx9 and e5gZ (released code, Qwen3-0.6B / SmolLM2-135M, A100; Appendices L and P)

MethodGuaranteeHead rows evaluatedGreedy decisions changed
CSV-Decode (eps=0, full search budget)certified top-k on its own FP32 score99.96% / 99.99%1/64 (a BF16 rounding tie) / 0/64
FlashHead (official assets, Qwen only)none6.24%0/640 observed
ARCHead (CUDA)noneapproximates all rows79/640 / 57/640
Ourscertified canonical decision0.0017% / 0.0103%0/640 / 0/640

Exactness costs little at the request level: ours is 0.8%-4.4% slower than ARCHead and 2.3% slower than FlashHead on a shared backbone.

3. Execution questions from ReiT. A proved FP32 screen lowers head time by 1.4%-10.8% against FP64 with zero decision changes. A single fused kernel (screening, counters, Gumbel transform, tile reduction) is bit-identical to the reference and 1.22-1.78x faster on sampling heads after a one-time compile.

Limits we state plainly in the paper. (a) The resident stock head kernel is still 2.3-10x faster than ours in isolation; the end-to-end gap is small because the head is a small part of a decode step. (b) Streamed stock reaches the same or one larger batch than ours; our advantage over it is speed, not capacity. (c) At the boundary batches the stock runs generate different tokens, since stock matmul is a different rounding program; prompts, batch and length are identical. (d) The capacity gain is 2.5%-3.0% at 32B, where the head is under 2% of device memory. New Table 27 shows the head is 5.9%-26.1% of the weights of the six models in the paper; this is accounting, not a new measurement. (e) SmolLM2-135M (d=576) is padded to 91% of the dense head in our GPU prototype.

Other revisions: Sec. 2.1 is reorganized with a new main-text Table 1 (rhx9); the hidden-state part of Figure 1 is redrawn and Eq. 4 states r=1,...,m (e5gZ); Eq. 3 states the exact temperature-scaling order used by the code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.