- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper focuses on KV cache eviction that occurs once after the prefill phase (as in SnapKV). It argues that window-based scores can identify the "cue" the question refers to but fail to capture the following span, which the model actually copies during decoding. ReadSpan addresses this issue in two stages: first, a head-matched base selector (HM) which evaluates positions based on the attention multiplied by the projected-value norm and divides the budget among the different layers and heads; and second, a refinement process that locates the positions in the input which match the first output token, transfers the window scores to those positions, extends forward from them, and then combines this with the base scores using a weight that takes coverage into account.
Critical strengths
1-The observation known as 'jump-and-read' is simple and intuitive, and it seems interesting to use the first output token as a reference point. 2-The motivation for the coverage-aware weight is good, and it has a short and correct proof.
Critical weaknesses
1- Looking at Table 3, the main point turns out to be of little significance. Extension from all positions (with no first-token anchoring) scores 88.83 compared to 89.06 for full ReadSpan, which means that the anchoring only contributes about 0.2 points. The removal of routing or setting beta equal to 1 causes changes in the results of between 0.01 and 0.40 points. Since there are 50 inputs per RULER task (one input being worth 2 points on a task), these differences are just noise. Most of the improvement over the baseline is due to the forward extension and to HM itself (HM on its own is already better than ReST-KV at 5% on Llama).
2-RocketKV (ICML25) already addresses the problem in question, which the paper doesn’t compare it with this work. The main point is that the window at the end of the prefill phase cannot know what the decoding process will read. RocketKV tackles this issue by maintaining a type of eviction similar to that used in SnapKV after the prefill phase, and then, at each decoding step, it chooses the top-k tokens using the actual decode query rather than making a guess based on a single token. ReadSpan should carry out a comparison with RocketKV regarding both the matched KV memory and the matched per-step traffic, and it should explain why it is preferable to make a one-token guess at prefill time rather than using the real decode queries. Moreover, HM is mainly a combination of existing techniques, some of which go unmentioned. The concept of a layer budget seems just the layer-preference rule from CAKE (ICLR 2025), which is not cited. The approach of taking the single top-k from the concatenated head scores is AdaKV. Head-level budget allocation was also investigated by HeadKV-R2 (ICLR 2025), even though it is not cited or compared. Value-aware scoring comes from CriticalKV; the only new element is the matching of attention and value on a per-query head basis within the GQA group, and there is no ablation study that isolates this feature.
3-There is no integration with a real inference engine such as vLLM or SGLang. The per-head or even per-layer variable-length arrangements are typically not well suited to vLLM-style serving and the practice of evicting after the prefill phase causes prefix caching to break down for use in multi-turn conversations. It is not easy to achieve a noticeable improvement over using the full key and value (kv) stores when incorporating such methods into vLLM-based designs. In vLLM, SGLang and TensorRT-LLM the paged cache holds all the heads of a token in the same block and the attention kernel assumes a single sequence length per request. Based Table 3, head-wise allocation is worth about 0.7 points on average, and thus the engineering effort involved might not be justified.
4- There is no TTFT, TBT, and no measured GPU memory. The statement that decoding reads a cache 10-20X smaller is an assumption from the budget, not a measurement, and it ignores the gather cost of rebuilding a compacted cache per head.
5- The performance gains are not significant. On LongBench, the method leads by only 1.00/0.63 at 5% budget and 0.27/0.21 at 10%, based on a single greedy evaluation over 100 inputs per task. Furthermore, the advantage over the top baseline stems almost entirely from two tasks, CWE and a single UUID task (K3), with S3 providing an advantage only against weaker baselines. This outcome conflicts with Section 4.3 assertion that aggregation tasks favor different baselines, given that ReadSpan substantially outperforms others on CWE. Also, although the authors acknowledge using both benchmarks during development to tune and fix hyperparameters (D, R, gamma, and W), they provide no sensitivity analysis.
Questions for authors
Please check the weakness section. Authors are expected to address the weaknesses. The authors also need to show how the method works for the multi-turn scenario datasets like SCBench.