acceptodds
Under review as a conference paper at ICLR 2027

Information Limits and Output Geometry for Quantized Attention

Abstract

A certificate can reject a quantized attention output even when its actual error is small. We characterize how retained probability and value information determines the tightest attainable error bound. Our exact scalar output envelope combines range, mean, and variance constraints: support bounds clip the classical variance-only optimizer, and two-point laws attain every regime. Joint constraints can certify budgets that separate moment bounds cannot. A distinct construction gives a separation from the output conversion of the optimal total-variation (TV) bound. We also derive attainable TV envelopes for selected exact probabilities and lower bounds, with saturation and stopping criteria. Outward moments and a sharp correction for perturbed-distribution value error connect the envelope to quantized attention. On a fixed GPT-2 replay, an exact CPU reference certifies 357 of 1,008 case/budget pairs, versus 249 for midpoint-centered TV and 352 for coordinatewise mixing of variance-only and range/mean bounds. Independent numerical enclosures validate every certified pair. These comparisons distinguish the benefit of value information from the additional gain of joint constraints.

Primary area:
learning theory
Keywords:
quantized attention, KV cache, certification, uncertainty quantification, output bounds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zuo7OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies runtime certificates that decide whether a quantized attention output satisfies ∥y−O∥∞≤ε\|y - O\|_\infty \le \varepsilon, falling back to the original FP16 cache otherwise. Arguing that TV-based bounds are conservative because they ignore value structure, the authors derive an exact scalar output envelope (Theorem 2) under joint range, mean, and variance constraints, attained by two-point laws. They also give attainable TV envelopes under partial probability information, with saturation and stopping criteria. Validation is an offline replay of 1,008 case/budget pairs from GPT-2, where Joint certifies 357 pairs versus 249 for centered TV and 352 for coordinatewise Mix.

Critical strengths

The joint envelope in Theorem 2 is the exact optimum within the stated information class: every regime is attained by a two-point law, so no tighter bound is possible from the same range/mean/variance summary.

The certificate is numerically sound, using directed rounding and exact rational comparisons, and every certified pair passes the independent enclosure of actual original-FP16 output error.

Critical weaknesses

The practical need for the proposed certificate is not demonstrated. The key contribution, joint constraints, certifies only five more pairs than coordinatewise Mix out of 1,008, all at ρ∈{0.5,1}\rho \in \{0.5, 1\}, i.e., tolerances of 0.5R00.5R_0 and R0R_0, whereas the output error is already trivially bounded by roughly 2R02R_0 since both outputs are convex combinations of values. None of the evaluated methods certifies any INT4-key case at ρ≤0.25\rho \le 0.25, and the paper does not relate ε\varepsilon to output quality, so it is unclear what practical benefit the tighter bound delivers.

Second, comparisons and evaluation scope are insufficient. There is no direct comparison against prior certificates for the same problem (Calver, 2026; Wei et al., 2026), and the evaluation is limited to an offline replay on GPT-2 with a 512-token prefix. Since Calver (2026) already evaluates on LLaMA 3.1-8B with contexts up to 128K, evaluation on modern models and KV quantization methods is needed.

Questions for authors
  1. Could you show that modern KV cache quantization methods (e.g., [1], [2]) actually produce large per-head errors, and that such errors change the final model output?

  2. How should ε\varepsilon be chosen in deployment? Please justify the grid ρ∈{0.001,…,1}\rho \in \{0.001, \dots, 1\} and explain how ε\varepsilon relates to output quality such as perplexity.

  3. How does Joint compare against the bounds used by Calver (2026) and Wei et al. (2026) under identical settings?

  4. Why is the evaluation limited to GPT-2 with a 512-token prefix? Does the gain of Joint persist on modern models and longer contexts?

  5. Since the original FP16 cache and a packed mirror are both kept on the CPU, total memory exceeds plain FP16. What is the advantage over plain KV offloading, and what is the fallback rate during actual generation?

References:

[1] Liu, Zirui, et al. "Kivi: A tuning-free asymmetric 2bit quantization for kv cache." arXiv preprint arXiv:2402.02750 (2024).

[2] Hooper, Coleman, et al. "Kvquant: Towards 10 million context length llm inference with kv cache quantization." Advances in Neural Information Processing Systems 37 (2024): 1270-1303.

3 comments
AuthorsOpenReview ↗

Reviewer Zuo7

The five additional admissions over coordinatewise Mix are a small gain in this replay. The theoretical result addresses what can be certified from the stated information: Theorem 2 establishes the optimal attainable scalar output bound for the given range, mean, variance and likelihood-ratio information. The gain comes from enforcing these constraints at the same event mass. For (a,b,v,κ)=(1,25,16,9)(a,b,v,\kappa)=(1,25,16,9), the joint bound is 128/25=5.12128/25=5.12 and passes budget 26/5=5.226/5=5.2, while variance alone gives 16/316/3 and range/mean alone gives 100/17100/17. A two-point law attains the joint value. Figure 2(a) therefore establishes a strict change in the budget decision, together with optimality within the information class.

The replay measures how often that additional constraint changes decisions in the captured data. Across 144 head/format cases and seven budgets, Joint certifies 357 pairs, centered TV 249, and coordinatewise Mix 352. The five additional admissions over Mix are all INT4 cases, two at ρ=.5\rho=.5 and three at ρ=1\rho=1. Table 2 gives their margins, including remaining budget 2.777830, Joint 2.522542 and Mix 2.964915. At ρ≤.25\rho\le .25, no INT4 case passes any evaluated method.

For the comparison with Calver and Wei, fixing the logit-error convention is essential. Under ∣δi∣≤Δ|\delta_i|\le\Delta, Equation (5) gives the sharp uniform TV bound tanh⁡(Δ/2)\tanh(\Delta/2), attained by two tokens. Calver v1 Equation (9) gives tanh⁡(Δ)\tanh(\Delta) under the same convention. Wei's deterministic bound uses per-token errors and reconstructed probabilities, which can tighten the result when large error bounds occur on low-probability tokens. Theorem 2 additionally exploits value range and variance jointly. Calver · Wei et al.

We completed an offline formula comparison on the saved cases using a common per-token L2 residual witness computed from the original FP16 trace. Reconstructed probabilities, value-coordinate widths and value/arithmetic allowances are identical across formulas. Ordinary FP64 evaluation gives 293 nominal passes for sharp uniform, 215 for Calver's uniform formula and 216 for Wei's deterministic formula over the same 1,008 budgets. These are nominal FP64 decisions using the additional residual witness. The original Joint count of 357 uses the paper's existing ratio bounds and additional value moments.

The grid ε=ρR0\varepsilon=\rho R_0 normalizes tolerances by an upper bound on the original value magnitude and provides a common scale for comparing certificate decisions. A deployment can select ρ\rho against its model-quality criterion, but we have not measured its relationship to model quality. Our fixed 512-token GPT-2 capture supplies common inputs for exact comparisons and independent numerical validation; it does not establish whether the gain persists on modern models or longer contexts.

Keeping the original FP16 cache and packed mirror increases CPU storage. Each successful certificate allows the quantized result to be used directly and avoids transferring the original head to the GPU. The benefit therefore depends on certificate cost, transfer cost and fallback frequency, which we have not measured during actual generation.

Thank you for the response. My main concerns (practical need, meaning of ε\varepsilon, and evaluation scope) remain unaddressed, so I maintain my score.

AuthorsOpenReview ↗

New experiments on modern models: calibrated tolerances and KV-group admissions

The new experiments show that the stronger bound changes complete KV-group decisions on modern models at a fixed, separately calibrated tolerance. On Qwen3.5-2B with K8/V8 and rho=0.05, Joint admits 25/768 additional complete KV groups at a 4,096-token prefix and 38/768 at an 8,160-token prefix over coordinatewise Mix: gains of 3.26 and 4.95 percentage points. Both bounds are evaluated on exactly the same states, summaries, and error allowances along the Joint trajectory. Because every query head sharing a KV group must pass before the group can avoid retrieving its original KV, these are changes in the operational group-level decision.

We have also broadened the evaluation to SmolLM2-360M, Qwen3-0.6B, and Qwen3.5 at 0.8B and 2B. The local comparison covers 48 model/context/precision settings: four prefix lengths (512, 2,048, 4,096, and 8,160), three formats (K4/V4, K8/V4, and K8/V8), and ten tolerance ratios, using six common held-out documents and the first, middle, and final full-attention layers. We followed the 2B/K8V8 operating point identified by this matrix with closed-loop model evaluation on four held-out documents and 16 teacher-forced tokens per prefix. At 4K/approximately 8K, Joint gives mean delta NLL of -0.000147/+0.000254 nat/token and mean KL of 2.74e-5/2.71e-5 relative to matched dense attention. The additional group admissions above are measured in these closed-loop runs.

To make epsilon operational, we use epsilon=rho R0: an absolute infinity-norm error budget for one head output relative to the same query and its original FP16 KV, where R0 bounds the original value-coordinate magnitudes. We select rho against model-quality criteria on calibration data, then test it on held-out text. In the Qwen3.5-0.8B study, four calibration documents selected rho=0.05 using mean delta NLL<=0.01 nat/token, mean KL<=0.01, and delta NLL<=0.05 for every document/context. On eight held-out documents with 32-token continuations, Joint+fallback gives mean delta NLL +0.00332/-0.00227 at 512/4K, compared with +0.01518/+0.01567 for unconditional K8/V4 quantization. We transferred the selected rho to the expanded study without retuning it on the new models. Thus, epsilon has a precise local meaning and an explicit calibration procedure for a downstream quality target; it is not itself a perplexity percentage.

The practical role is to decide when the compressed attention output is usable and when the original KV is needed. Joint and coordinatewise Mix already share the same moments and correction terms, so replacing Mix with Joint requires the stronger scalar-envelope evaluation, without another original-KV transfer or moment-construction pass. The new group-level results quantify what this analytical increment changes. We will include the calibration protocol, full precision/context matrix, and closed-loop measurements in the main-text evaluation and discussion.

The model runs use nominal FP64 attention with FP16 inputs/outputs and dense prefill; Qwen3.5 interventions cover its six softmax-attention layers. For the 2B/K8V8 layout, the original host KV plus packed mirror occupies about 1.54 times the original KV storage, excluding temporary workspace. We report these costs explicitly and treat the results as model-quality and gate-decision evidence, with physical transfer and serving latency requiring separate measurement. The new evidence directly addresses the meaning of epsilon, extends the evaluation beyond the original GPT-2 setting, and demonstrates additional usable group decisions. We would appreciate reconsideration of the rating in light of these results.

Reviewer_8Hx2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces information-aware certificates to bound worst-case output errors in quantized attention. The authors derive an attainable output envelope with joint range, mean, and variance constraints. It strictly improves compared to use separate moment bounds. Additionally, they prove saturation for probability bounds and provide a sharp correction for value reconstruction error. They did experimetns on a CPU GPT-2 replay, it shows that the proposed method certifies significantly more pairs than current baselines.

Critical strengths

The theory is extensive and rigid. The paper introduced an exact, attainable scalar output envelope which jointly applies range, mean, and variance constraints. The joint-envelope framework can be utilised in other field in deep learning.

Critical weaknesses
  1. The paper spent most of the pages on the theory. For experiment, they only test on GPT2. I wonder what would be the result for other models like qwen or deepseek. Besides, the experiemnt is on CPU. Will deploying on GPU change the conclusion?
  2. In equation 18, the author introduced sharp scalar correction. Though the proof is provided in appendix, the paper can be benifited from introduce the intuiation in the main body.
Questions for authors
  1. I highly suggest the author to pay more attention to plot the figures in the paper. The figures are used to deliver the information to readers, so it would be better to make the information more clear in the figures. For example, in figure 3, it would be better to put the text ‘Exact p1 = 1/2: p = (1/2, 1/2)Lower p1 ≥ 1/2: p = (3/4, 1/4) allowed’ into a legend and make it more concise. Examples of good figures can be found in the ELF paper.
  2. Can authors provide overhead for computing outward moments?

[1] Hu, Keya, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, and Kaiming He. "Elf: Embedded language flows." arXiv preprint arXiv:2605.10938 (2026).

1 comment
AuthorsOpenReview ↗

Reviewer 8Hx2

Equation (18) corrects for a change in the distribution that weights value-reconstruction errors. Let ηi\eta_i bound token ii's reconstruction error, H=max⁡iηiH=\max_i\eta_i, and e=min⁡{H,Eval}e=\min\{H,E_{\mathrm{val}}\}, where EvalE_{\mathrm{val}} bounds the average under reconstructed-key attention pp. The original-key distribution π\pi reweights the same errors, with unnormalized tilt factors in [1,κˉ][1,\bar\kappa].

For H>0H>0, the worst case concentrates error HH on an event of pp-mass e/He/H and assigns that event the largest tilt κˉ\bar\kappa. Its new mass is

κˉ(e/H)1+(κˉ−1)e/H.\frac{\bar\kappa(e/H)}{1+(\bar\kappa-1)e/H}.

Multiplying by HH gives Cη=κˉeH/[H+(κˉ−1)e]C_\eta=\bar\kappa eH/[H+(\bar\kappa-1)e], with Cη=0C_\eta=0 when H=0H=0. This explains both the correction and its sharpness. Appendix H.3 supplies the proof and attaining construction.

As for the cost of computing outward moments, the CPU reference adds an O(nd)O(nd) weighted-value pass and O(d)O(d) rational moment fields and scalar decisions. The appendix CUDA timings cover the centered-TV path; they do not measure the outward-moment overhead. The theorem applies on GPU when the computed intervals enclose the required moments and arithmetic errors.

The reported acceptance counts use the fixed GPT-2 capture, so they do not establish results for other models. For Figure 3, compact legend entries “Exact p1=1/2p_1=1/2” and “Lower bound p1≥1/2p_1\ge1/2” make the two information classes explicit. The caption can then explain the concrete distinction. With two tokens, the exact constraint fixes p=(1/2,1/2)p=(1/2,1/2), while the lower bound also permits p=(3/4,1/4)p=(3/4,1/4).

Reviewer_CcSFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is very poorly written and it is hard to decipher what exactly is the goal and how it is achieved. It seems like the goal is to develop a procedure that checks if KV quantization distorted the output of the attention head at each layer too much, and if so download a full precision version of the KV cache from CPU and compute the output in full precision. It is not clear why such a procedure can be useful - downloading data from CPU on such basis will incur terrible delays unless these calls are extremely rare. In any case, even if I could find the motivation more convincing, the presentation is too poor for understanding what exactly is the proposal and whether it makes sense.

Critical strengths

None

Critical weaknesses

-The writing is very bad and it is nearly impossible to follow this paper, and what exactly is proposed -The motivation is not convincing to me. I believe that for modern LLMs on modern GPUs reloading KV cache of active sessions from the CPU to GPU based on a semi-randomized decision will incur terrible delays and throughput hit.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Reviewer CcSF

The proposed decision is deterministic, rather than semi-randomized: Section 5 tests an error budget for each output. For one query and one attention head, it accepts the quantized output when

Earith+Cη+max⁡cJc≤ε,E_{\mathrm{arith}}+C_\eta+\max_c J_c\le\varepsilon,

where JcJ_c is the joint output bound for coordinate cc, CηC_\eta bounds value-reconstruction error under the original attention weights, and EarithE_{\mathrm{arith}} bounds numerical error. The target is the exact-real attention output on the original FP16 cache. If the test fails, the reference retrieves the original head and checks its output against a separate numerical enclosure.

The role of the output bound is to make that decision without first computing the original output. Quantized keys change attention weights, and the output change depends on the values receiving those weights. Theorem 2 characterizes the largest possible change consistent with the value range, mean, variance and tilt bound, and provides attaining two-point laws. Figure 2(a) gives an exact budget that the joint constraints certify. Section 6 then validates every admitted replay case against an independent enclosure of the original-output error.

The concern about CPU–GPU transfer is therefore a question about the cost of fallback. A successful certificate allows the quantized result to be used without transferring the original head, and the net benefit depends on certificate overhead, transfer size and fallback frequency. The appendix GPU timings concern the centered-TV path.

For the soundness assessment, we ask the reviewer to identify the specific theorem, proof step or numerical-validation result that is considered incorrect or insufficient. The submitted paper provides explicit mathematical statements, attaining constructions and independently checked decisions on which that assessment can be based.

I thank the authors for their response. Unfortunately, even after reading the response I am still struggling with understanding what are the main results in the paper and why they are useful. My low score for soundness is because I can't even understand what some of the statements mean and what is their practical significance. Maybe with aid of LLMs the paper is easier to understand, but it should still be readable without use of LLMs. I still don't find this paper sufficiently readable, and I will not be able to increase my score,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.