- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CalibKV, a method for adaptive sparse attention budgeting. It learns per-head query subspaces and calibration banks offline, estimates request-specific attention budgets during prefill, and selects top-ranked keys using exact attention scores during decoding. A rank-m screening variant is also introduced to reduce full-dimensional score computation. Experiments on language modeling and downstream benchmarks show that CalibKV maintains near-dense quality while attending to a small fraction of cached keys.
Critical strengths
1. Adaptive budget estimation. CalibKV introduces a calibration-based approach to estimating per-request, per-head attention budgets, separating budget estimation from query-dependent token selection.
2. Promising quality retention. CalibKV maintains near-dense language modeling quality with relatively small attention budgets.
Critical weaknesses
1. Limited novelty. The motivation of CalibKV substantially overlaps with Twilight [1], which already proposes adaptive attention budgets based on attention distributions. Although CalibKV adopts a different calibration approach, its advantages over existing methods remain unclear, and no direct comparison with Twilight is provided.
2. Insufficient experimental evaluation.
-
Incomplete benchmark coverage. LongBench includes only 10 of the 21 datasets, while RULER covers only a small subset of its 13 tasks, with just 100 examples for variable tracking. This limits the generalizability of the results.
-
Unfair budget comparisons. Baselines operate at substantially different sparsity levels (e.g., 0.2% for Quest vs. 3.1% for CalibKV at 64K). Without matched-budget comparisons or quality–budget trade-off curves, the reported improvements are difficult to interpret.
-
Limited efficiency benefits. At 128K, its decoding latency increases from 48.05 to 53.05 ms/token compared with Dense, and its end-to-end latency is also higher. Moreover, efficiency is evaluated only at batch size 1.
[1] Lin C, Tang J, Yang S, et al. Twilight: Adaptive attention sparsity with hierarchical top-$ p $ pruning[J]. Advances in Neural Information Processing Systems, 2026, 38: 119406-119432.
Questions for authors
1. Novelty. What are the key advantages of CalibKV over Twilight? Can the authors provide a direct comparison?
2. Benchmark coverage. Why are only 10 LongBench datasets and a small subset of RULER tasks evaluated? Can the authors provide results on the complete benchmarks?
3. Fair comparisons. Can the authors provide matched-budget comparisons or quality–budget trade-off curves? How are candidate fractions and actual attention budgets distinguished across methods?
4. Efficiency. Can the authors provide a breakdown of computational overhead and results under batched decoding?