- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FACT-KV, an evaluation framework that separates three objectives in KV cache compression: (i) tested query utility, measured using answer F1/EM, (ii) annotated token coverage, and (iii) exact annotated unit residency. It also defines three metrics: coverage (), complete unit fraction (), and all units completeness (). These metrics use original token position traces to measure how much annotated information is retained and whether complete annotated units are preserved.
Critical strengths
-
Clear distinction between evaluation objectives: It clearly separates answer utility, annotated-token coverage, complete-unit fraction, and all-units completeness, making it easier to understand what each metric measures and which claims about information preservation it can support.
-
Strong evidence of differences in policy rankings: The frozen 500-source Mistral experiment shows that a policy can substantially improve F1 (, 95% CI ) while significantly reducing all-units completeness (, 95% CI ). This highlights how a policy can perform better on one metric while performing worse on another.
Critical weaknesses
-
Limited comparison with modern compression methods: The main experiment compares recency and cumulative attention retention. Although the paper also evaluates contract adapted H2O, it does not include direct comparisons with structured methods such as ChunkKV or learned query methods under matched conditions. This makes it difficult to determine how the findings apply to more recent KV cache compression approaches.
-
Limited generalization of the conflict between utility and residency: The main F1 improvement is demonstrated only for Mistral-7B on HotpotQA with a 1,024 position budget. For Llama-3.1-8B, the F1 difference is statistically uncertain (, 95% CI ), even though residency decreases substantially (). This suggests that the conflict between answer quality and information retention is not consistent across the evaluated models.
-
No validated improvement that preserves answer quality: Although block widths of 4, 8, and 16 improve complete unit fraction by , , and , respectively, none demonstrates F1 noninferiority or meets the predefined joint success criterion. Therefore, the paper does not establish a practical compression policy that improves exact residency while maintaining answer quality.
Questions for authors
-
Novelty and contribution: Since the proposed metrics are based on original token position coverage and exact set membership, what is the main methodological contribution beyond showing that different evaluation objectives can lead to different policy rankings?
-
Comparison with modern compression methods: Can the authors evaluate structured and learned query compression methods, such as ChunkKV and Judge Q, under the same memory budgets? This would help determine whether the reported differences in policy rankings also apply beyond recency, cumulative attention, and contract adapted H2O.
-
Generalization across models: Why does the large F1 improvement on Mistral () become much smaller and statistically uncertain on Llama () at the same 1,024 position budget, while the exact residency gap remains large? Can the authors provide controlled analyses of model behavior or attention distributions to explain this difference?
-
Improving residency without reducing answer quality: Since none of the tested block widths meets the joint success criterion (Figure 4B), could adaptive width or sentence aligned selection improve exact residency significantly while maintaining F1 noninferiority?