- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the limitations of SnapKV’s fixed observation window by proposing a 2-hop attention analysis technique. The authors define a query informativeness score and empirically show that queries receiving more attention from the request query tend to have higher informativeness scores. Based on this observation, they propose Q-Trust, a training-free KV cache compression method. Q-Trust improves average task performance over SnapKV, both with and without pooling, at small KV cache budgets, with modest additional prefill overhead.
Critical strengths
-
The analysis provides a clear motivation for the proposed method. The paper starts with a query informativeness metric that requires future attention, then proposes request attention as a proxy available during prefill. Table 1 provides empirical evidence supporting this connection.
-
The method addresses computational cost as well as accuracy. The authors address the prefill overhead introduced by 2-hop scoring through selective computation and kernel optimization, limiting the additional overhead to 3.5% for 16K-token prompts.
Critical weaknesses
- The performance gap on QA and retrieval tasks is insufficiently analyzed. SnapKV with pooling outperforms Q-Trust on single-document and multi-document QA across all three evaluated models, as well as on Needle-in-a-Haystack retrieval. However, it remains unclear whether this gap reflects the benefits of preserving neighboring context or limitations of Q-Trust’s scoring and selective computation. Since pooling appears orthogonal to the proposed query-weighting mechanism, evaluating Q-Trust with pooling under the same KV cache budget would help clarify the source of the gap and whether it can be mitigated.
Questions for authors
-
Could you evaluate Q-Trust with pooling under the same KV cache budget and compare it with both the original Q-Trust and SnapKV with pooling?
-
Could you repeat the compression experiment in Figure 1(b), and the informativeness-based query-weighting experiment in §2.4, using informativeness scores without common-component removal? This would clarify whether the downstream benefits attributed to informative queries are robust to this analytical choice.
2 comments
Thank you for additional experiments and clarifications. All my questions have been answered.
Response to Reviewer AeQW: Q-Trust + Pooling and Shared-Component Ablation
We thank the reviewer for the concrete suggestion to combine Q-Trust with pooling. G1 to G4 are in the General Response.
Q1 / W1. With pooling, Q-Trust closes most of the QA gap, keeps most of the few-shot gain, and has higher Needle-in-a-Haystack accuracy than SnapKV + pooling. We trace part of the QA difference to the direction of the second hop.
As the task-group table in G1 shows, Q-Trust + pooling is above SnapKV + pooling on Multi-doc QA for all three models, and the Single-doc QA gap is now 0.2 to 0.9 points. On Few-shot it stays 3.8 to 5.0 points ahead. On Needle-in-a-Haystack it scores 97.5%, against 97.1% for SnapKV + pooling, both at the level of the full cache (96.9%).
Attention is causal, so a trusted query adds score only to keys before it, and the tokens right after a span the request refers to, such as the tail of an answer, receive less score. In the needle task, most wrong answers of Q-Trust were numbers with only their leading digits correct. Our pooling gives each key the highest score among itself and the keys just before it, which covers these tokens. Pooling can lower the LongBench average of Q-Trust, but Q-Trust + pooling stays above both SnapKV and SnapKV + pooling on all three models, so we report both Q-Trust rows (Section 4.3, Tables 2 and A.1).
Q2. Without removing the shared component, the conclusion is the same, with a slightly larger gap.
As G3 shows, the quartile intervention gives top − random +4.00 and top − bottom +12.45, against +3.50 and +12.39 for our definition. In the query-weighting experiment, weighting in proportion to informativeness gives 41.02 without removal and 41.08 with it (s weighting: 36.43). The share of s on the top quartile rises from 58.0% to 69.2%. We have added both experiments to Appendix G.