- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper essentially tries to make an argument toward answering the following question: "Does the importance of a cached token change depending on the current question reading it?" This question seems useful enough since it could motivate further kv cache compression algorithms if the cached token change stays largely unchanged.
The paper gives the following answer which I agree but does't seem to very interesting nor useful. The answer is essentially that a cached token that matters for one question may matter much less, or more, for another.
Critical strengths
I think the authors have made reasonable attempt to compare ways to predict which cache positions matter.
Below are the ones I have noted.
- Vector size alone performs poorly.
- Previous attention history is somewhat useful.
- The current question’s attention weights—called routing—perform well.
- More complicated local calculations improve predictions only slightly.
- Following the complete computational path helps in early layers, but is expensive and not offered as a practical selection method.
Critical weaknesses
-
i don't find the claims and findings of the paper to be sufficiently insightful and interesting. Cache importance depends on the current query; this is already intuitive and related query-aware cache methods exist.
-
It proposes no usable new cache-compression algorithm (again i understand it's more of a analysis paper, but if the analysis is not good enough a method would be useful to have)
-
The empirical results are also rather small scale and toy. This is minor if the results are consistently good; however, results weaken substantially for longer contexts and different perturbation directions.
-
The paper is badly written. I do not find the abstract to be a good way of presenting the paper. As an example, the starting sensente "Across 64 shared QASPER document caches on Qwen2.5-1.5B, changing the question produces a median absolute same-entry log-sensitivity change of 0.573." Please do not assume reader would know what QASPER document is and what does "same-entry log-sensitivity change of 0.573" enetail specifically? i am already quite lost from reading the abstract..
-
Terminology is quite poorly defined. Much of this language is author-created framing. For example reader: the current question/query consuming the cache (at least give a definition in the introduction please) . Active readout path, which i assume is the the computation carrying information from a cached entry to the model output. Sensitivity, which i think is how much the output distribution changes when one cached entry is slightly perturbed.
Questions for authors
-
What practical cache-selection algorithm results may from these findings? Some discussion would be useful to motivate the analysis.
-
Does the conclusion survive actual deletion rather than small radial perturbations?
-
Can the early-layer full-path signal be approximated cheaply?