acceptodds
Under review as a conference paper at ICLR 2027

Auditing Local Attention Fidelity as a Proxy for Generation Quality

Abstract

During inference in Transformer-based large language models, key-value (KV) cache compression is widely used to reduce memory overhead. Yet accurate local attention reconstruction need not imply reliable generation after compression. We test this assumption by separately measuring fitting-query residuals, operator errors at future full-model queries, and task scores from compressed generation. We use calibrated local merging (CALM) as a tractable test construction and compare it with eviction and controlled attention-fitting methods. A classical conditional Kullback–Leibler (KL) identity yields a local output bound; real-state measurements show that its global form is loose, while a retrospective group certificate reduces median prompt-level bound/error ratios from thousands to 6.40-7.95 at 40% retention. Across three near-4B model families, twelve LongBench tasks, 64 examples per task, and 8K/16K caps, we evaluate 27,648 predictions. On 24 matched Phi texts, CALM has lower operator error than ChunkKV throughout, yet lower task scores on ten. Allocation controls do not establish a geometry-specific task benefit, and compression before the first output reveals protocol sensitivity. Genuine-length inputs save 50.16-59.70% of all persistent cache state, but eager decoding is slower. These results delimit what local fidelity can support as a proxy without implying that fidelity and generation quality are generally unrelated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Attention Fidelity, Large Language Models, Long-Context Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PQUfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

We all know that KV-cache compression aims to preserve generation quality while reducing memory usage and inference overhead as much as possible. This paper questions the practice of using intermediate computations to represent final quality and provides empirical evidence of this issue. In addition, it makes some theoretical contributions.

Critical strengths

The paper has a relatively comprehensive experimental design and does a good job of supporting its core claims.

Critical weaknesses

My main concern is the paper's motivation. In fact, research on K -cache compression does not rely solely on the metric mainly discussed in this paper. On the contrast, it is only proxy metric, and the community has never widely used it or regarded the proxy as perfect. When evaluating a paper that primarily questions existing metrics or common research paradigms, I usually ask myself how the research practices of the entire field would change if everyone accepted the paper's conclusions. However, for this paper, I find that the main takeaway after reading it is that we should not trust local reconstruction errors alone, but should also examine final generation quality, memory usage, and speed. Yet these are things the field is already doing.

Questions for authors

There are some strange things in the appendix that look like commit hashes, as well as some references that directly point to filenames, even though the paper does not include any supplemental materials. In fact, even if supplemental materials were provided, writing things this way would still be somewhat strange. These details should appear in a README instead. If these contents were directly generated by AI (although I do not consider AI-generated content itself a reason for rejection, low-quality content is), I hope the authors can review all AI-assisted work, as they stated in their AI USE STATEMENT.

Reviewer_5nisOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts an empirical and theoretical audit of whether local attention reconstruction quality (local fidelity) serves as a reliable proxy for downstream language generation quality in compressed key-value (KV) caches.

Critical strengths

Rigorously Disentangled Diagnostic Audit: The paper addresses a prevalent implicit assumption in KV-cache compression literature—that minimizing local attention approximation error translates directly to higher generation quality. By decoupling evaluation into fitting-query reconstruction, future-query operator error, and end-to-end task scores on identical source texts, the work provides clear evidence of disconnected behavior between local fidelity metrics and actual benchmark performance.

Critical weaknesses

Limited Scope of Tested Baseline Models and Methods: While the paper tests three model families around 4B parameters, the core test construction (CALM) and adapted comparators (SnapKV, ChunkKV, simplified AM ports) represent a specific subset of post-prefill compression approaches. The AM baseline in particular is an adapted basic port omitting on-policy query collection and layer reruns. As a result, while the conclusions hold strongly for CALM and the tested baseline configurations, care must be taken not to overgeneralize the findings to all potential KV compression paradigms.

Retrospective Nature of the Refined Bound: The group-adaptive retrospective certificate dramatically reduces global bound slack. However, as the authors acknowledge, calculating this bound requires knowledge of the uncompressed raw token groups and future query tokens. Consequently, while it serves as an insightful diagnostic tool for auditing error offline, it cannot be used as an actionable, real-time online stopping or compression control mechanism during inference.

Questions for authors

Clarification on AM Baseline Adaptation: Could you clarify if testing the full, on-policy Attention Matching pipeline (including layer reruns and on-policy queries) alters the misalignment observed between future operator error and end-to-end task scores?

Clarification on Decoder Acceleration: The benchmark measurements indicate eager batch-1 decoding with CALM is slightly slower than full-cache attention due to bias-casting and update overheads. Have you benchmarked or analyzed whether custom fused kernels (e.g., FlashDecoding or vLLM integrations supporting per-entry biases) restore speedups proportional to the persistent memory savings?

Reviewer_ti1HOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether local attention fidelity is a reliable proxy for generation quality after KV cache compression. The authors introduce CALM, a training-free KV merging construction that preserves attention output at a reference query, and derive local error bounds together with a tighter result. They compare fitting query residuals, future query operator errors, and downstream task scores on three model families and twelve LongBench tasks. The experiments show that CALM can achieve lower operator error than ChunkKV while obtaining worse task scores.

Critical strengths

This paper raises an interesting question about the relationship between attention reconstruction and generation quality, and reports meaningful cases where the two are inconsistent. The author provide careful mathematical derivations for local KV merging and a refined retrospective error bound. Additionally, the experiments include matched comparisons, allocation controls, numerical checks, and analyses of prompt protocols as well as memory usage. These efforts demonstrate detailed experimental observation and a high degree of transparency.

Critical weaknesses

My main concern is that the paper does not sufficiently establish a coherent and well supported contribution. CALM provides limited independent methodological advances, and the error bound analysis remains largely disconnected from the main fidelity quality observation. Although the observed discrepancies are interesting, the supporting materials are insufficient and do not explain how local errors affect the results. Addressing these issues would require substantial new experiments and analysis. I suggest developing the theoretical and empirical directions separately unless their connection can be established. My specific concerns are outlined below.

  1. The theoretical error bounds are interesting, but their contribution to understanding generation quality is unclear. The tighter bound requires the original KV groups and future queries, while its numerical evaluation uses only six source texts. Although the authors explicitly report the bound/error ratios as descriptive statistics, this limited sample cannot establish their general tightness. Could the authors evaluate the bounds on more independent texts and explain further value of error bounds?

  2. The central fidelity quality diagnostic uses only 24 source texts per model and at most 8 queries from full model trajectories, leaving the causes of the observed discrepancies unexplained. Could the authors further analyze the compressed model to explain how exactly the bias on attention value would influence the results, and what leads to the difference in between attention accuracy and generation accuracy?

Questions for authors

Please see the above part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.