- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the elimination of KV cache from the perspective of unbalanced query coverage, links the divergence between the quality of retained attention and the conditional distribution, deduces the scoring rule for assigning higher weights to low-coverage queries, and implements the online elimination method FidKV. Experiments show that the complete strategy has a more obvious advantage under a tighter cache budget.
Critical strengths
1.The method deduces the scoring rules from a clear coverage target and provides an understandable explanation for why the weight of low-coverage queries needs to be increased.
2.The author provided the matching control after removing the weighted terms and reported the paired confidence intervals. Mechanism analysis indicates that under the condition of repeated elimination, weighting helps to improve the approximation of coverage targets and attention output.
Critical weaknesses
1.In Table 9 of Appendix C, the core weighting item did not lead to a significant increase in the LongBench task score under either a 10% or 20% budget. The author also pointed out that most tasks ended before the revised scores affected the next elimination, so the advantage of the complete method over other baselines cannot be directly attributed to this weighting term. 2.The most obvious task benefits of the paper occur in the 2% and 5% budgets, but these Settings still lack equally strict weighted item removal controls. The existing experiments cannot fully distinguish the contributions of the core mechanism from those of other designs such as elimination scheduling. 3.The connection between the improvement of fidelity at the mechanism level and the final task benefits remains insufficient. The teacher's forced replay experiment shows that the fidelity index has improved, but the fixed generation trajectory cannot fully reflect the error accumulation during free generation. It is suggested to supplement the analysis under the condition of free generation to explain under what circumstances the improvement of this indicator can be transformed into a stable improvement in task performance.
Questions for authors
- Can cumulative attention controls under the same scheduling and budget be added to the existing 2% and 5% budget experiments, only changing whether coverage weighting is used?
- What are the specific differences between the cumulative attention reference in Figure 2 and the matching control in Appendix C, and how should the effect differences shown by the two be explained?
3 comments
Raised my score to 3
Thank you for the detailed response and additional experiments. My concerns have been adequately addressed, and I have raised my score to 3 accordingly.
We sincerely thank the reviewer for the valuable suggestions and for raising the score. We especially appreciate your recognition of the two aspects at the core of this work, namely the derivation of the scoring rule from a clear coverage objective and the mechanism analysis with the matched control.
Response to Reviewer Sbvj
We sincerely thank the reviewer for the recognition of our work. To resolve your concerns and questions, we respond to each point below.
W1 / W2 / Q1 — cumulative-attention control at 2% and 5% under the same schedule and budget.
Thank you for the suggestion. We added this control (Table A). CumAttn removes only the 1/w^ factor and keeps the schedule and budget fixed. The weight helps on tasks where queries are covered unevenly, which we measure by the dispersion of w^ (its coefficient of variation across decoding queries). At 2%, the gain increases with this dispersion across the five short-answer tasks, and it is largest on the evidence-sparse QA tasks MFQA and 2WkMQ, which also gain at 5%. Where w^ is nearly uniform (PC, LCC) the weight has little to re-rank and FidKV stays close to the control.
Table A. Matched control at the extreme budgets (Llama-3.1-8B, protocol of Tables 3–4), with tasks ordered by the dispersion of w^. CumAttn is FidKV with only the 1/w^ factor removed. Gain is FidKV minus CumAttn. Avg is sample-weighted, with a paired 95% bootstrap interval for the gain. *MultiNews is the long-generation task.
On the schedule, the delayed first eviction is not a separate design choice but a prerequisite of the coverage estimate, since w^ is computed from the attention of decoding queries and no token can be scored before the first one has been observed. We will state it explicitly as a component of the method.
W3 — when does the fidelity improvement turn into task improvement; analysis under free generation.
The dispersion of w^ answers this. Dispersion shrinks as the budget grows (0.114, 0.091, 0.071, 0.047 on average at 0.5%, 1%, 2%, 5%), so at 10% and 20% the weighted and unweighted policies make nearly the same selections. Table 9 is intended to verify exactly this prediction of Proposition 1. When coverage is nearly uniform, the weight reduces to cumulative attention and preserves accuracy, while still improving the log-fidelity objective where repeated eviction occurs. Under tight budgets dispersion is high, and the fidelity improvement turns into task gains when the tokens exchanged by the weight are context evidence, that is, on evidence-sparse QA (Table A). We also ran the free-generation analysis on the three long-generation tasks (Table B). There the correction preserves accuracy, because the exchanged tokens are mostly newly generated ones rather than context evidence, which changes the summary-level score little. We will add this analysis and the dispersion measure to the paper.
Table B. FidKV − CumAttn under free generation on the three tasks with the 512-token cap (Llama-3.1-8B, F1 points, 50 paired examples per task).
Q2 — Figure 2 reference vs. the Appendix C control.
They are different references. Figure 2 compares the complete FidKV policy with standard cumulative-attention eviction, which starts evicting at prefill. Appendix C uses the timing-matched control, which removes only the 1/w^ factor. We will state the reference of Figure 2 explicitly in its caption and add the matched-control version of the figure.
We hope these responses resolve your concerns, and we are happy to provide any further clarification.