acceptodds
Under review as a conference paper at ICLR 2027

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Accelerating Long-Context LLMs

Abstract

Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among different heads and contexts. We observe that attention heads exhibit two distinct entropy patterns: Rigid Heads, whose attention entropy remains near zero across different segment of inputs, and Dynamic Heads, whose entropy fluctuates significantly across the input context. Crucially, the distribution of these two types is context-dependent and cannot be predetermined offline. Based on this observation, we propose EntropyInfer, a training-free framework that leverages attention entropy to adaptively allocate computation budgets at the granularity of individual heads and segments during prefilling. For the decoding stage, we further introduce a latent KV cache compression scheme that utilizes generated output tokens, rather than prefill tokens alone, to identify and retain the most critical cache entries. Experiments on LongBench and InfiniteBench with Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct show that EntropyInfer consistently outperforms existing methods including SnapKV, AdaKV and CritiPrefill, achieving up to 2.39 end-to-end speedup in scenarios where context length grows beyond 100k tokens, with minimal average generation quality degradation compared to full attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context LLM inference, Sparse Attention, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ff4NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EntropyInfer, a training-free framework for accelerating long-context LLM inference. It has two components: entropy-guided sparse prefilling and latent KV-cache compression during decoding. During prefilling, the method classifies attention heads as “Rigid” or “Dynamic” according to their entropy patterns and allocates different computation budgets to different heads and segments. During decoding, it generates several output tokens before recomputing KV-cache importance and compressing the cache. The paper evaluates the method on Llama, Qwen, and openPangu models using InfiniteBench, LongBench, and RULER. It reports up to 2.39x end-to-end speedup at long context lengths.

Critical strengths

The paper addresses a relevant systems problem and presents a complete framework covering both prefilling and decoding.

  1. Reasonably complete system design. The method combines sparse prefilling with decoding-stage KV-cache compression and does not require model training or an auxiliary model.
  2. Broad evaluation. The experiments cover multiple model families, context lengths, and long-context benchmarks. The paper also reports latency and memory measurements in addition to task quality.
Critical weaknesses

Overall, the main concerns are the limited distinction from prior work, the insufficient validation of the proposed head classification, and the unstable gains of the complete system.

  1. The novelty needs to be clarified. The entropy-based dynamic budget allocation is closely related to EntroKV(https://proceedings.mlr.press/v306/gao26a.html). In addition, RazorAttention(https://arxiv.org/abs/2407.15891), DuoAttention(https://arxiv.org/abs/2410.10819), and HeadKV(https://arxiv.org/abs/2410.19258) have already studied attention-head heterogeneity and head-dependent KV-cache policies. The paper should clarify what new method or knowledge is provided by the Rigid/Dynamic distinction and EntropyInfer beyond these works.
  2. The role of the Rigid/Dynamic classification is not sufficiently validated. The paper shows different entropy patterns across heads, but does not establish that a binary classification is necessary or superior to continuous entropy allocation or simpler statistical signals.
  3. The ablations provide limited information. The main ablations remove sparse prefilling or latent decoding, but do not isolate the contributions of entropy classification, the budget update rule, segment/block settings, and the output-token observation window.
  4. The quality and efficiency gains are not consistently strong. In some LongBench and RULER settings, the complete method performs worse than the base model or the version without latent decoding. The reported 2.39×2.39\times speedup is primarily relative to dense attention, and the speed-quality trade-off against existing efficient baselines is not fully clear.
Questions for authors
  1. The entropy-guided head/segment budget allocation in this paper is quite close to EntroKV. Could the authors clarify the main technical distinction and the new contribution of the Rigid/Dynamic classification?

  2. Latent decoding causes noticeable quality degradation in some settings, while the version without latent decoding performs better. Could the authors explain this behavior and clarify under which conditions latent decoding provides a reliable benefit?

Reviewer_aAmFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets two bottlenecks of long-context LLM inference: quadratic attention in prefill and the growing KV cache in decoding. It proposes EntropyInfer, a training-free framework. Following CritiPrefill’s block-level aggregation, the authors build a segment-level observation attention matrix A^obs from dimension-wise max/min-pooled representatives and compute per-head row entropy on it. They report two kinds of heads: Rigid heads, whose segment entropy stays at or below 10^(-5), and Dynamic heads, whose entropy fluctuates. They further claim that this categorization is input-dependent (64.9% of Llama heads and 58.3% of Qwen heads switch across inputs) and must therefore be determined online.

The method has two components. In entropy-guided sparse prefill, Rigid heads receive a fixed per-segment budget B_h. Dynamic heads update their budget recursively from the relative entropy change between consecutive segments, clamped to [B_h, 3B_h], and top-k block-sparse attention follows. In latent KV compression, the model first generates N_D tokens; the KV cache is then re-ranked and evicted using an observation window of prompt-tail plus generated-token queries. Experiments cover Llama, Qwen and openPangu models (1B to 70B) on InfiniteBench, LongBench, RULER and LoCoMo, and report up to 2.39× end-to-end speedup at 140K context.

Critical strengths

The problem is important and practical. The method is training-free, drop-in, and covers both prefill and decoding.

The evaluation is broad. It spans three model families (including 32B and 70B models) and four benchmarks, and includes a latency breakdown, a memory analysis, and sensitivity studies on e_t and segment/block sizes. Code is released anonymously.

Fig. 6 compares allocation schemes under the same total budget, which is the right experimental design. At long context, entropy-based allocation clearly beats uniform allocation.

The sparse-prefill-only variant clearly outperforms CritiPrefill on long-context RULER. Adaptive budgeting therefore has real value.

Critical weaknesses

Overall, the Rigid/Dynamic dichotomy, the paper’s main conceptual contribution, is measured on a likely saturated proxy and is barely used by the method. The gains of latent decoding are confounded with simply delaying compression, and this component does not reduce peak memory. Without a full-attention baseline on RULER, the claims of “negligible degradation” and “fastest overall” are not supported.

W1. The Rigid/Dynamic dichotomy may be an artifact of the proxy, and the method barely uses it. Head classification is based on A^obs rather than on the token-level attention in Eq. (1). A^obs is built from max/min-pooled vectors without 1/√d scaling, which inflates logits and saturates the softmax. Fig. 1 is consistent with this: Rigid heads have entropy around 10^(-13) to 10^(-6), and even Dynamic heads peak at about 1, far below the maximum of about 8.3. The claim that Rigid heads have near-deterministic attention is never checked on real attention. In addition, Rigid heads receive B_h, which is exactly the minimum budget of Dynamic heads, so they never get less compute, and the average budget is only about 4% above 2048 (Table 13). The classification mainly stops near-zero entropies from triggering spurious budget increases rather than saving compute. Table 7 does not support the distinction either: All-Dynamic uses at least as much budget as Ours yet scores slightly lower, so the 0.3-point gap is likely noise.

W2. The case against offline profiling is weak. The 64.9% figure counts heads that switch at least once, which grows with the number of inputs, and that number is not reported. Because classification uses the maximum segment entropy, longer inputs produce more Dynamic heads (800 at 12K vs. 991 at 131K in Table 5), so switching may largely reflect input length. Fig. 2(a) also conflicts with Table 4: 203 heads that are always Rigid imply at most 821 Dynamic heads per input, yet Table 4 reports 935 on Musique, and the same conflict appears for Qwen. Crucially, there is no experiment comparing offline-profiled head types with online classification, and no comparison with DuoAttention or RazorAttention. Finally, FlexPrefill is mischaracterized. It already classifies heads online into two pattern types per input and adapts each head’s sparsity ratio through a cumulative-attention threshold, so it should be described accurately and included as a baseline.

W3. The budget heuristic is not well justified. The budget follows the relative increase in entropy rather than its level. A head with consistently high entropy therefore decays to the minimum budget, while a tiny rise from 10^(-4) to 10^(-3) receives the maximum increase. There is no sensitivity analysis for α, Δ_t or the 3× cap. In Fig. 6, Random (single seed) beats Average and Criticality-k95 in every setting and even beats Entropy on Llama 8K, while Criticality-k95 is nearly identical to Average. Multiple seeds, details of the budget projection, and an attention-recall comparison are needed. Also, Fig. 6 uses per-head profiling while the main results use GQA-grouped profiling (55.1 vs. 51.24 on Llama 128K). The per-head design emphasized in the paper is therefore not what is evaluated.

W4. Latent KV compression is underspecified and its benefit is unclear. N_D and N_W are never reported. If most of a short answer is generated before compression, the gain comes from delaying compression rather than from better token selection. Long-output tasks support this: on GovReport and MultiNews, Ours is close to SnapKV/AdaKV and far below full attention (e.g., 30.02 vs. 35.16 on Llama-8B GovReport). A delayed-SnapKV baseline is needed. The method also does not reduce peak memory (45.53 GiB vs. 45.52 GiB for Dense, against 30.04 GiB for SnapKV/AdaKV), even though memory pressure is the stated motivation. Finally, LoopServe, which the paper cites, already combines online prefill sparsification with progressive KV compression during decoding. Since LoopServe is arXiv-only, a comparison is not required, but the differences should be explained.

W5. The evaluation does not support the main claims. The RULER tables lack a full-attention row. The official RULER results for Llama-3.1-8B are 95.5 at 4K and 77.0 at 128K. If the setups are comparable, Ours loses about 18 points at 4K and about 30 points at 128K, exactly where the speedup is claimed. The latency comparison omits CritiPrefill+SnapKV/AdaKV, so “fastest overall” is not established; TTFT/TPOT, batch size and kernel details should also be reported. Key baselines (MInference, FlexPrefill, Quest, LouisKV) are missing. Many quality differences are within 0.5 points, yet no variance or significance tests are reported.

Questions for authors

Q1. What are N_D and N_W? What fraction of output tokens on each benchmark is generated after compression? Please add a delayed-SnapKV control.

Q2. Please add full-attention results to Tables 2, 11 and 12.

Q3. For heads classified as Rigid, what is their token-level attention entropy or top-k mass, and what do they mainly attend to (attention sinks, local windows)? Does A^obs use any 1/√d or temperature scaling? How do quality and speed change if the Rigid-head budget is reduced to B_h/8, or to sink plus local only? This would test whether the dichotomy is meaningful and could yield real savings.

Q4. How many inputs were used for Fig. 2(a), and how is “input-dependent” defined? Please provide length-controlled results, together with the label agreement and downstream results of offline profiling on held-out inputs. Please also reconcile Fig. 2(a) with Table 4.

Q5. By Algorithm 1, EntropyInfer’s budget for every (head, segment) pair is at least CritiPrefill’s 2048. If the two methods share A^obs and top-k selection, EntropyInfer’s selected blocks should be a superset of CritiPrefill’s. Why, then, are R.Number (Llama 95.42 vs. 99.15) and R.PassKey (Qwen 96.44 vs. 99.15) worse in Table 1? Ours–All Rigid (uniform 2048) and CritiPrefill also disagree on NarrativeQA (28.19 vs. 27.29). What other implementation differences exist, for example in GQA handling or the local window?

Q6. For Fig. 6, please report means and intervals over multiple random seeds, describe exactly how the projection to the same total budget is done, and explain why Random consistently beats Average.

Q7. Algorithm 4 sums the unnormalized Q_W K^T directly, with no softmax, no scaling and no SnapKV-style pooling. Raw logits can be negative and are dominated by key norms. Is this a typo or intended? If intended, please provide an ablation against softmax-based scoring. How is the budget allocated across heads under GQA?

Q8. On InfiniteBench with Qwen2.5-7B, SnapKV and AdaKV score only 4.41 and 4.07 on R.Number (Base 93.22), versus 85.59 and 82.37 on Llama. Since Qwen2.5-7B’s native context is 32K, was YaRN enabled, and how were over-length inputs truncated? A baseline-configuration issue should be ruled out.

Reviewer_YUQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents EntropyInfer, a training-free inference framework that uses segment-level attention entropy to classify heads and adjust sparse-prefill budgets, followed by delayed KV cache compression using early generated tokens. Experiments on several long-context benchmarks and model families report improved quality over the evaluated efficiency baselines and up to a 2.39× end-to-end speedup in the tested setting.

Critical strengths

The paper presents a training-free framework that uses input-dependent attention entropy to allocate sparse-prefill budgets across heads and sequence segments, and delays KV cache compression to incorporate attention signals from early generated tokens. The evaluation covers Llama, Qwen, and openPangu models on multiple long-context benchmarks, with module and budget-allocation ablations.

Critical weaknesses
  1. Comparisons with static head-based methods are missing. Although DuoAttention[1] and RazorAttention[2] are cited, neither is evaluated, leaving the claimed advantage over offline head classification unverified. A focused comparison with a representative method such as DuoAttention would clarify the practical benefit.
  2. The decoding contribution is not clearly distinguished from prior work. LoopServe[3] already selects cached tokens using attention from recent outputs, while LAQ++[4] uses a short preliminary response together with the prompt tail to estimate token importance. The paper should explain what its single delayed compression adds to these designs and substantiate its advantages in generation quality or runtime overhead.
  3. The benefit of explicitly separating Rigid and Dynamic heads remains unclear. Table 7 shows only marginal improvements over All Dynamic on two datasets, without corresponding measurements of actual compute budgets or latency. The authors should report these costs for both variants and extend the ablation to broader datasets or model to justify the binary classification over applying adaptive allocation to every head.

[1] Guangxuan Xiao et al. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. ICLR, 2025.

[2] Hanlin Tang et al. RazorAttention: Efficient KV Cache Compression Through Retrieval Heads. ICLR, 2025.

[3] Haoyang Li et al. LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues. arXiv preprint arXiv:2507.13681, 2025.

[4] Yixuan Wang et al. Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query. EMNLP, pp. 34158–34174, 2025.

Questions for authors
  1. Are the entropies in Figures 1–2 computed from the full token-level attention in Eq. (1), or from the block-level attention estimated in Algorithms 2–3? Please clarify the scaling and numerical precision used, including the 1/d1/\sqrt d factor not shown in Algorithm 3, and explain how the estimated entropy relates to the attention budget required to preserve model quality.

  2. Could the authors provide the missing RULER baseline using full attention and an uncompressed KV cache, so that the claim of negligible quality degradation can be assessed?

  3. For Ours w/o SP, what fraction of responses end before delayed compression is triggered? Please clarify their contribution to the reported gains, since these responses use the full cache throughout generation and do not evaluate the quality of the 1024 entries retained by output-guided compression.

  4. Please specify the experimental values of NDN_D, the number of tokens generated before compression, and NWN_W, the number of query positions in the observation window, which are not reported in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.