- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed LAB, which uses Local Accumulative Boost to define the blocks for KV cache, make these blocks contain semantic information.
Critical strengths
- I think the idea of using LAB is kind of like a 1-D convolution, is interesting.
Critical weaknesses
- The paper lacks a sufficiently thorough ablation of the key LAB hyperparameters, especially the neighborhood radius and scaling factor . Although Appendix A includes a small sensitivity study, it is limited to PG19 at 16K on Mistral-7B with only 8 samples. A more systematic analysis across tasks and models would be useful to show that the reported gains are robust and not sensitive to these manually chosen values.
- I am unclear about the design of the interval delineation step. The threshold is computed from the enhanced score distribution , but candidate tokens are selected using the raw normalized scores . Therefore, the local smoothing does not directly determine whether a token belongs to a block. This seems somewhat inconsistent with the motivation that LAB helps form semantically coherent blocks. I would like the authors to explain why this cross-distribution comparison is preferable to directly thresholding the enhanced scores .
Questions for authors
- What is the latency (TTFT) overhead of this method, especially for very long context like 128K? It's not reported in the paper.
- I am interested in the use of the term "semantic block." The selected blocks are defined as contiguous runs of high-attention-score tokens, rather than by sentence, paragraph, entity, or other semantic boundaries. The diagnostic analysis only shows that high-importance tokens tend to be spatially clustered in a few examples, which does not necessarily imply that each selected block corresponds to a coherent semantic unit. Can the author quantitatively evaluate whether the retained blocks actually align with meaningful semantic spans?