acceptodds
Under review as a conference paper at ICLR 2027

Utility, Coverage, and Exact Residency Can Rank KV-Cache Policies Differently

Abstract

What should a compressed KV cache preserve? The answer depends on the claim: behavior on a target query, retained annotated content, or intact original slots for a literal-residency claim. These are different targets, yet KV-cache evaluations often treat answer quality or token coverage as evidence for all three. FACT-KV aligns tested-query utility, original-position coverage, and exact annotated-unit residency on the same selected state. In a prospectively frozen 500-source Mistral/HotpotQA contract, cumulative retention improves F1 over recency by 0.220 and coverage by 0.086, yet reduces all-units completeness by 0.348. The conflict also occurs within sources: 70 show both higher F1 and lower completeness. An official HotpotQA re-score of the frozen predictions gives similar F1 (+0.217) and EM (+0.198) advantages. A placement reference reproduces much of the residency contrast, implicating retention geometry rather than semantic destruction. Capacity and granularity stress tests move exact residency systematically in the tested settings, without a corresponding monotone improvement in answer utility. A contract-adapted H2O audit, together with Llama, reuse, attribution, and repair audits, establishes the boundary: exact annotated-unit residency is a preservation-audit target, not a utility surrogate. Preservation claims should name the target, report its direct readout, and declare the layer/head aggregation rule.

Primary area:
datasets and benchmarks
Keywords:
KV-cache compression, long-context inference, cache evaluation, exact annotated-unit residency, retained annotated quantity, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yDZUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FACT-KV, an evaluation framework that separates three objectives in KV cache compression: (i) tested query utility, measured using answer F1/EM, (ii) annotated token coverage, and (iii) exact annotated unit residency. It also defines three metrics: coverage (CsC_s), complete unit fraction (IsI_s), and all units completeness (AsA_s). These metrics use original token position traces to measure how much annotated information is retained and whether complete annotated units are preserved.

Critical strengths
  1. Clear distinction between evaluation objectives: It clearly separates answer utility, annotated-token coverage, complete-unit fraction, and all-units completeness, making it easier to understand what each metric measures and which claims about information preservation it can support.

  2. Strong evidence of differences in policy rankings: The frozen 500-source Mistral experiment shows that a policy can substantially improve F1 (+0.220+0.220, 95% CI [0.180,0.261][0.180, 0.261]) while significantly reducing all-units completeness (−0.348-0.348, 95% CI [−0.390,−0.306][-0.390, -0.306]). This highlights how a policy can perform better on one metric while performing worse on another.

Critical weaknesses
  1. Limited comparison with modern compression methods: The main experiment compares recency and cumulative attention retention. Although the paper also evaluates contract adapted H2O, it does not include direct comparisons with structured methods such as ChunkKV or learned query methods under matched conditions. This makes it difficult to determine how the findings apply to more recent KV cache compression approaches.

  2. Limited generalization of the conflict between utility and residency: The main F1 improvement is demonstrated only for Mistral-7B on HotpotQA with a 1,024 position budget. For Llama-3.1-8B, the F1 difference is statistically uncertain (+0.006+0.006, 95% CI [−0.013,0.026][-0.013, 0.026]), even though residency decreases substantially (−0.443-0.443). This suggests that the conflict between answer quality and information retention is not consistent across the evaluated models.

  3. No validated improvement that preserves answer quality: Although block widths of 4, 8, and 16 improve complete unit fraction by +0.136+0.136, +0.206+0.206, and +0.261+0.261, respectively, none demonstrates F1 noninferiority or meets the predefined joint success criterion. Therefore, the paper does not establish a practical compression policy that improves exact residency while maintaining answer quality.

Questions for authors
  1. Novelty and contribution: Since the proposed metrics are based on original token position coverage and exact set membership, what is the main methodological contribution beyond showing that different evaluation objectives can lead to different policy rankings?

  2. Comparison with modern compression methods: Can the authors evaluate structured and learned query compression methods, such as ChunkKV and Judge Q, under the same memory budgets? This would help determine whether the reported differences in policy rankings also apply beyond recency, cumulative attention, and contract adapted H2O.

  3. Generalization across models: Why does the large F1 improvement on Mistral (+0.220+0.220) become much smaller and statistically uncertain on Llama (+0.006+0.006) at the same 1,024 position budget, while the exact residency gap remains large? Can the authors provide controlled analyses of model behavior or attention distributions to explain this difference?

  4. Improving residency without reducing answer quality: Since none of the tested block widths meets the joint success criterion (Figure 4B), could adaptive width or sentence aligned selection improve exact residency significantly while maintaining F1 noninferiority?

Reviewer_Xk1UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors argue that answer F1, evidence-token coverage, and whether whole supporting sentence intact in cache are different targets. They show cumulative-attention retention beats recency on F1 (+0.22) and coverage (+0.09) but loses on all-units completeness (−0.35) in Mistral-7B/HotpotQA comparison.

Critical strengths
  1. Statistical rigorous with pre-registered endpoints, paired bootstraps, and explicit labeling of confirmatory vs exploratory results.
  2. Clear illustration of coverage vs intact unit distinction
  3. Negative and stopped results are disclosed rather than buried.
Critical weaknesses
  1. Only two toy policies plus an adapted H2O are compared
  2. The Llama replication fails on F1 and the boundary tests show residency does not predict utility. The authors ask to report a specific metric but the paper can not show what it is for
Questions for authors
  1. Are there published methods whose claims would be mis-evaluated without residency consideration?
  2. Is there any downstream task (attribution, compliance, cache auditing, etc.) where residency predicts anything the user cares about? If not, why should people report it?
Reviewer_z5fXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an evaluation framework that distinct answer utility, evidence token coverage, and exact evidence unit residency in KV cache compression. With experiments in controlled comparisons of recency and cumulative attention policies, this work demonstrate that better answer accuracy and token coverage can co-exist with substantially lower exact evidence preservation. Additional experiments investigate retention geometry, cache capacity, selection granularity, model generalization, and future query reuse.

Critical strengths
  1. The primary study uses a frozen 500 source evaluation, paired statistical analysis, and independent validation. The results fairly establish the reported ranking discrepancy under the tested conditions.
  2. The authors investigate alternative retention thresholds, cache budgets, and block selection. These analyses help indentify the observed structural phenomenon from its limitations. 3/ The paper differentiates physical cache residency from semantic preservation and avoids claiming that higher residency necessarily improves downstream performance.
Critical weaknesses
  1. The primary discrepancy appears largely attributable to the difference between contiguous and scattered token retention. Section 3.3 further shows that retention geometry reproduces much of the observed residency gap. Moreover, the strict all units completeness metric is mathematically equivalent to requiring 100% evidence coverage within each source. Although the empirical quantification is useful, the paper could better establish what genuinely new scientific insight emerges beyond these expected structural properties and existing evidence retention evaluations. Overall, limited novelty of the major observation.

  2. The proposed metric measures physical preservation of original cache positions rather than semantic information retention. The authors acknowledge that it does not predict answer utility or future query performance, appropriately. However, the experiments do not clearly establish a concrete research or application scenario where this additional measurement leads to a consequential improvement in evaluation or decision-making, whcih limits the demonstrated significance of the framework.

  3. The main controlled comparison focuses on recency and cumulative attention. Although H2O and other methods are discussed (or partially evaluated), consistent comparisons with representative modern compressors remain incomplete. This limits the practical scope of the proposed evaluation framework.

Questions for authors

Q1. What specific scientific insight does FACT-KV establish beyond the expected differences between contiguous and scattered token retention and existing coverage measurements?

Q2. Can the authors identify a concrete preservation claim or evaluation decision that would be materially improved or corrected by exact residency measurements?

Q3. How broadly should the findings be interpreted for modern KV cache compressors, given the limited native method comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.