- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper derives how Key tensor perturbations in the attention operations, modeled as (conditionally-) unbiased uniform quantization, interact with the attention score structure. It is shown how output error norm can be first approximated (up to a uniform constant) by random sign perturbations on the quantization grid and, later and in certain regimes, by Gaussian noise. The bounds are then used to derive optimal bitwidth allocations between tokens and channels to minimize output distortion. The derived bounds and allocations are validated on synthetic data.
Critical strengths
- The presentation is thorough. No definitions or references appear missing.
- The notation is consistent and I had no problem navigating the manuscript.
- The results check out numerically on synthetic data.
Critical weaknesses
- The text is extremely dense. Few theorems or definitions include simplified explanations or plain-text discussion of implications. It feels like the text was ruthlessly and repeatedly compressed with an LLM to make it as short as possible, leaving only bare bones of definitions, cross-references and formulaic results.
- Numerical validation only deals with synthetic data sampled independently from Gaussian or mixture of Gaussian distributions. Since real Q and K tensors in LLMs are heavily-correlated and might be more heavy-tailed, it is not clear which results hold for real LLMs.
Since I have very limited experience with such math-heavy manuscripts, its hard for me to gauge how critical these downsides are.
The score I give ("weak accept") reflects the fact that I see no strong indications of unfairness or misrepresented results, but I might not have the expertise to independently verify or contextualize all of them.
Questions for authors
- The scope of quantization is somewhat unclear since Sections 2 and 3 seemingly only describe Value-tensor perturbations but Proposition 5 suddenly Key-tensor perturbations too, which only ever appear in the proposition itself. Can you elaborate on which parts of the paper deal with V or KV quantizations?
- What are the implications of the bit allocations you derive on LLM inference where the query family is not fixed but rather expands as generation goes? Is there a practical way to update allocations on the fly?
3 comments
Response to Reviewer uazK (2/2)
W2 (synthetic data only; real Q and K are correlated and heavy-tailed). The theorems make no assumption on how queries, keys or values are distributed. The upper bounds need only that each value lies in its grid's range and that the rounding randomness is independent of the input, which per-token ranges with stochastic rounding satisfy; the optimality of Theorem 2 and the lower bound of Theorem 12, which assume grids fixed in advance, are not claimed for such ranges, nor are the statements that inherit them. Correlation and heavy tails change the attention family and the ranges, and so the numbers: an outlier value widens its token's range and coarsens that token's grid, and the error bound grows by at most that factor, and by little unless queries attend to that token (Lemma 26). On Qwen2.5-0.5B with WikiText-2 (all 48 key-value heads on 8 text chunks; values rounded, keys exact; Section 6, Appendix I.4), the ratios that Theorem 7 and Corollary 8 bound stay as close to one as on synthetic families: between 0.800 and 0.932, and between 0.989 and 1.085. At 3 bits the measured read error under stochastic rounding is 0.376–0.494 of . On the larger Qwen2.5-1.5B (56 key-value heads on 4 chunks, head dimension 128; protocol fixed before its run, after the 0.5B results) the two ratios lie between 0.838 and 0.957 and between 0.994 and 1.074. At this cache length is within 4% of the single-worst-query term on every head, so real heads test the laws in their simplest regime; the selection term matters on families with rare sharp queries (Section 6). The allocation result is mixed: every adapted per-token schedule lowers the 2-bit perplexity on 15 of 16 chunks, but our equal-contribution schedule is the weakest of three, and on the worst-query error of later queries it keeps almost none of the gain it has on its own calibration queries (3 bits: median 1.023 of uniform). On those calibration queries every adapted schedule cuts the median worst-query error to 0.09–0.16 of uniform, so what fails is the calibration family, not only the optimizer (general response, item 1).
Q2 (allocation when the query family grows during generation). In practice, not yet. A schedule fitted to the queries seen so far did not hold up on later queries (Section 6), and a token's bits cannot be raised after it has been rounded. What can be computed on the fly is the schedule of each newly written block, for a set of queries declared in advance: if the cache is written in blocks of many tokens, the worst-query error of the whole cache is at most the largest worst-query error of a single block, because full attention is a mixture of the block-wise attentions (Section 5; Lemma 32 in Appendix H; in expectation this costs a factor for blocks, and per-block schedules are not claimed jointly optimal). That needs a query set known to cover future queries. One that provably does follows from the model weights, but it is far larger than the queries that occur (the largest observed query norm is a median 0.121 of the bound; Appendix I.4), so a set that is both valid and tight is open (Section 7), as is the propagation of read errors through later layers and generated tokens.
We hope the real-model checks (W2) and the scope statement (Q1) address your main questions, and we would welcome any remaining concerns on readability.
Concerns largely resolved
Thank you for the additional explanations and results.
The addition of practical LLM measurements strengthen the narrative. However, they also show the limited practical usefulness of the results, as seen from equal-contribution schedules underperforming beyond token positions they were calibrated on (Table 1, Table 9).
I appreciate the additional results, improved clarity and honesty about limitations, and maintain my positive assessment of the paper.
Response to Reviewer uazK (1/2)
Thank you for the review; we agree with its main criticism, that the main text is too dense. The revised PDF (changes in blue) adds plain-language readings, states the scope up front, and checks the laws on trained models.
W1 (density). Within the 10-page limit, the main text gained "In words" readings after Theorem 2, Theorem 7 and Corollary 8 and a scope statement in Section 1. The decomposition figure (Figure 2 at submission) illustrates a proof step, so it now sits beside that proof (Figure 4, Appendix E). The fuller fix is in Appendix A, next to the dependency graph (Figure 3): a new list, "The results in words", with a plain statement of every main-text result. In short:
On W3 and W4, for checking the results without the proofs: Figure 3 shows what each result relies on, Appendix B gives where each external theorem comes from, Appendix I.1–I.3 and I.5 check every law on synthetic data, and Appendix I.4 checks Theorem 7, Corollary 8 and the read-error bounds on real heads. The one new probabilistic ingredient of the structure theorem is Lemma 30 in Appendix E.
Q1 (which parts concern values and which keys). Apart from Proposition 5(b), every result from Theorem 2 through Corollary 14 concerns value rounding with exact keys; the keys only determine the attention family. Section 1 and the Takeaway now say so.
The split has a reason. The value error is linear in the value perturbation and is a norm of the error array, which is what Sections 3 and 4 compute. A key perturbation acts through the softmax, so its error does not have the form our main theorem handles; Theorem 7 still gives a first-order upper bound for it (end of Appendix D), and Proposition 5(b) gives an exact one. We have added the scope statement to Section 1, a sentence before Proposition 5, and this derivation at the end of Appendix D.