- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that current KV-cache budget allocators for long-context LLMs optimize the wrong objective: they minimize a sum of per-head standalone tail risks, which only matches the model’s aggregate tail risk under comonotonic failures across heads. The authors propose TailAlloc, which allocates budgets by each head’s Euler risk contribution—expected head error conditional on the model’s aggregate tail—estimated efficiently and optimized with a marginal-gain greedy. Empirically, TailAlloc consistently reduces aggregate tail risk vs. the exactly-solved separable optimum across six captures and three model families.
Critical strengths
S1. Easy to follow: TailAlloc is a training‑free method that is easy to reproduce.
S2. Novelty: The paper introduces Euler risk contributions from risk theory into KV cache allocation, which has some research value.
Critical weaknesses
W1. Although TailAlloc improves the tail-risk metric over SepAlloc, its task-level advantage remains limited and inconsistent. On RULER-4096, the 90% confidence interval for the improvement is [−0.52, +0.83], which includes the possibility of a performance decrease. Thus, the current evidence does not sufficiently establish that optimizing aggregate tail risk yields consistent downstream task improvements.
W2. The description of the calibration dataset is insufficient, despite its importance for offline KV cache eviction methods. The resulting KV cache budget allocation is likely to depend strongly on the characteristics of the calibration dataset.
Questions for authors
Q1. The paper optimizes a tail-risk surrogate constructed from local attention errors, but does not establish a theoretical connection between improvements in this surrogate and reductions in the sum of errors across all queries. The downstream task experiments also suggest that improvements in the surrogate do not consistently translate into better task performance. Could the authors provide theoretical analysis or empirical evidence clarifying the relationship between this tail-risk surrogate and the total error across all queries?
Q2. TailAlloc relies on offline calibration, so the quality of its budget allocation is likely to depend strongly on the characteristics of the calibration corpus. However, the results in Section 6.5 show that calibration on in-distribution text does not outperform novel-based calibration in the reported comparisons. How can practitioners select an effective calibration corpus without repeatedly running downstream evaluations?