- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents EntropyInfer, a training-free framework for accelerating long-context LLM inference. It has two components: entropy-guided sparse prefilling and latent KV-cache compression during decoding. During prefilling, the method classifies attention heads as “Rigid” or “Dynamic” according to their entropy patterns and allocates different computation budgets to different heads and segments. During decoding, it generates several output tokens before recomputing KV-cache importance and compressing the cache. The paper evaluates the method on Llama, Qwen, and openPangu models using InfiniteBench, LongBench, and RULER. It reports up to 2.39x end-to-end speedup at long context lengths.
Critical strengths
The paper addresses a relevant systems problem and presents a complete framework covering both prefilling and decoding.
- Reasonably complete system design. The method combines sparse prefilling with decoding-stage KV-cache compression and does not require model training or an auxiliary model.
- Broad evaluation. The experiments cover multiple model families, context lengths, and long-context benchmarks. The paper also reports latency and memory measurements in addition to task quality.
Critical weaknesses
Overall, the main concerns are the limited distinction from prior work, the insufficient validation of the proposed head classification, and the unstable gains of the complete system.
- The novelty needs to be clarified. The entropy-based dynamic budget allocation is closely related to EntroKV(https://proceedings.mlr.press/v306/gao26a.html). In addition, RazorAttention(https://arxiv.org/abs/2407.15891), DuoAttention(https://arxiv.org/abs/2410.10819), and HeadKV(https://arxiv.org/abs/2410.19258) have already studied attention-head heterogeneity and head-dependent KV-cache policies. The paper should clarify what new method or knowledge is provided by the Rigid/Dynamic distinction and EntropyInfer beyond these works.
- The role of the Rigid/Dynamic classification is not sufficiently validated. The paper shows different entropy patterns across heads, but does not establish that a binary classification is necessary or superior to continuous entropy allocation or simpler statistical signals.
- The ablations provide limited information. The main ablations remove sparse prefilling or latent decoding, but do not isolate the contributions of entropy classification, the budget update rule, segment/block settings, and the output-token observation window.
- The quality and efficiency gains are not consistently strong. In some LongBench and RULER settings, the complete method performs worse than the base model or the version without latent decoding. The reported speedup is primarily relative to dense attention, and the speed-quality trade-off against existing efficient baselines is not fully clear.
Questions for authors
-
The entropy-guided head/segment budget allocation in this paper is quite close to EntroKV. Could the authors clarify the main technical distinction and the new contribution of the Rigid/Dynamic classification?
-
Latent decoding causes noticeable quality degradation in some settings, while the version without latent decoding performs better. Could the authors explain this behavior and clarify under which conditions latent decoding provides a reliable benefit?