acceptodds
Withdrawn from ICLR 2027

Level of Detail Attention: Allocating Resolution Instead of Sparsity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most sparse attention methods attend to a subset of keys, ignoring the rest and thereby amplifying any mistakes while entirely removing the ignored set from softmax normalization. We introduce Level of Detail (LoD) Attention, a training-free approximation to dense softmax attention that represents less important keys and values at reduced resolution instead of discarding them. For language inference, LoD requires no changes to pretrained model weights and achieves prefill work and work per decode step. Experiments on Qwen3.8 and K2 Horizon show that LoD largely preserves aggregate quality. On Qwen3.8, two-tier LoD accelerates attention computation by over 6x during both prefill and decode in our 128K prompt batch size 8 benchmarks. Its similarity-based grouping scheme also permits the reduction of the LLM key-value cache to a compact INT4 representation, saving VRAM with little additional aggregate quality loss. To show the versatility of our method across modalities we also demonstrate LoD for MiniMax-H3 video generation. We will release our code under the Apache 2 license.

Primary area:
generative models
Keywords:
attention, LLM, Video, sparsity, resolution, multiresolution, generative, KV, KV-Cache, vector quantization, transformer, compressive state

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2ddiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Level-of-Detail (LoD) Attention to reduce the computational and memory overhead of long-context attention. The core idea is to replace the “keep-or-drop” mechanism of traditional sparse attention with a multi-resolution representation: for important regions that are routed in, precise attention is performed using the original key-value (KV) pairs; for other regions, aggregated KV summaries with population correction are used to continue participating in the softmax, thereby avoiding the complete discarding of unselected tokens. The authors further propose a recursive three-tier LoD, which continues to select pages within the chosen region for precise computation and represents the remaining tokens using a residual summary. Additionally, the paper combines online semantic region construction, region-owned page organization, and mean-relative INT4 KV compression, and applies similar ideas to video generation. Experiments evaluated perplexity, LongBench v2, RULER, prefill/decode latency, and KV-cache memory on Qwen3.8-27B and K2-Horizon-32B, reporting significant attention-level speedups in long-context scenarios; the video section included limited comparisons with dense attention and Sol Attention on MiniMax-H3.

Critical strengths

The system is fairly comprehensive and demonstrates practical acceleration effects for long contexts. The paper not only proposes attention approximation but also implements online region maintenance, recursive page routing, INT4 KV storage, and the corresponding kernels. On Qwen, the authors report significant attention-level prefill/decode speedups with 128K context, indicating that the method has practical engineering applicability.

Critical weaknesses
  1. The core method lacks novelty, and LoD’s primary algorithmic contribution overlaps significantly with existing work. The core idea behind LoD is to apply exact attention to important regions while allowing unselected regions to continue participating in normalization via a coarse summary with population correction. However, this “coarse representation + selective exact refinement” paradigm overlaps significantly with existing work. RetroInfer (PVLDB 2026) has already employed segmented clustering and tripartite attention approximation to perform exact computations on some content and approximations on the rest; Squeezed Attention (ACL 2025) has also employed semantic clustering, centroid routing, and hierarchical refinement. The paper itself acknowledges that the combination of exact and approximate regions is not a novel proposal.
  2. The absence of critical baselines prevents the experiment from demonstrating that LoD has advanced the state of the art. The main experiments for LLMs primarily compare Full Attention with various LoD variants. However, Full Attention is not the most significant competing method in this paper. Existing methods such as QUEST (ICML 2024), Squeezed Attention (ACL 2025), and RetroInfer (PVLDB 2026) all focus on accelerating long-context attention and report real kernel or end-to-end performance. In particular, RetroInfer and Squeezed Attention are highly relevant to the design of our method. Without a direct comparison under identical model, hardware, context length, and quality degradation conditions, the attention speedup reported in this paper cannot demonstrate that LoD has an advantage over existing methods.
  3. There is no fair accuracy–efficiency Pareto comparison. LoD does not fully maintain the performance of Full Attention on RULER. For example, Qwen’s average score drops from 96.17 to 94.39, and even more significant degradation is observed on other benchmarks. Simply reporting accuracy and speedup separately is insufficient, and the authors should compare E2E latency, attn latency, and memory with existing methods under the same quality loss to demonstrate whether LoD achieves a superior Pareto frontier.
  4. Ablation is insufficient. The performance gain of this paper also depends on several tricks, including kernel optimization and INT4 compression. Currently, there are insufficient experiments to distinguish the contributions of the LoD itself from those of these engineering techniques.

Reference

  • Chen et al., RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference, PVLDB 2026.
  • Hooper et al., Squeezed Attention: Accelerating Long Context Length LLM Inference, ACL 2025.
  • Yuan et al., Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, ACL 2025 Best Paper.
  • Tang et al., QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024.
  • Hooper et al., Squeezed Attention, ACL 2025.
Questions for authors
  1. Compared to existing exact-plus-approximate attention methods such as RetroInfer, what are the specific new mechanisms at the core algorithmic level in LoD two-tier?

  2. Why were RetroInfer, Squeezed Attention, and QUEST not included in the main experiments? Under the same model, hardware, context length, and accuracy constraints, does LoD still offer a better latency/throughput/memory Pareto frontier?

  3. Is the performance degradation observed in RULER for Multikey, HotpotQA, and SQuAD related to routing errors or region approximation errors? Could the authors provide a corresponding analysis?

  4. Could you provide more ablation studies?

2 comments

Thank you for your attentive review mentioning the practicality of the method and comprehensiveness of the system, as well as the specific and important changes you need to see in the paper to comprehensively address novelty, related work comparisons, quality, and ablations. We have completed a large regime of new experiments and we hope you find that these address the concerns. Because there was significant overlap across reviewer requests, we have consolidated some of our responses into separate shared review response comments in an effort to make things easiest for the reviewers.

Regarding W1/Q1, we agree that this general strategy of coarse representations and selective refinement is shared. The specific distinctions are:

  • RetroInfer, Section 4.2, and Double-P, Sections 4.2--4.3, select clusters for exact or approximate evaluation. LoD's recursive variant selects pages within an eligible region and recomputes one summary for the unselected remainder, using the remaining counts and key/value sums.
  • Squeezed Attention, Sections 3.1--3.3, builds an offline semantic index for exact-key retrieval. Unretrieved clusters do not remain as summary operands in the final attention output. Hierarchical Multipole Attention, Section 3.4, is a closer recursive comparator. LoD combines the unopened pages of a routed region into one residual summary, saving summary operands at the cost of coarser approximation.

For two-tier LoD, the distinction concerns online region construction with retained member KVs and bounded refinement under the schedule in Sections 2.4 and 3.1. We do not claim a new population-weighted exact/summary formula. The recursive variant additionally opens selected pages and represents the remaining entries by one residual summary, as defined in Section 2.2. We now state this contribution in Section 1 and distinguish it from the closest prior methods in Section 6.

We have made these distinctions explicit as mentioned in the shared comments, and in Section 6 of the updated paper, which have more details and information.

We think the paper has been improved tremendously by the feedback so far. We would be happy to discuss any outstanding issues and provide whatever additional evidence we can. We have worked hard to provide as many new experiments as possible, each highly targeted to specific reviewer concerns, and sincerely appreciate your time and effort helping us to make the paper the best it can be.

We thank you again for your time and thoughtful feedback. We hope you have been able to review our responses in the comment above and in the shared comments and updated paper, in which we attempted to resolve all of the stated concerns, and would welcome a response, and any further thoughts or questions as the discussion period draws to a close.

Reviewer_f16kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Most sparse attention methods attend to only a subset of keys and ignore the rest, which removes the ignored entries from the softmax normalization. This paper proposes Level-of-Detail (LoD) Attention, a training-free approximation that can replace dense softmax attention in pretrained models. The method keeps a local window exact, groups remote KV entries into regions represented by count-corrected summaries, and opens regions or pages for exact attention according to query relevance. For language models, the authors combine online semantic regions, recursive page opening, and mean-residual INT4 KV-cache storage. Under the stated schedule, this yields subquadratic attention computation and lower cache storage. Experiments show that the method largely preserves aggregate quality on language-model benchmarks, though some exact-retrieval tasks degrade. The paper also applies the method to video generation and reports practical speedups.

Critical strengths

The motivation is clear and the problem is real. Sparse attention distorts both the numerator and the denominator, because dropped KV entries disappear from the softmax normalization. This is one genuine reason why training-free sparsification is unstable on pretrained models.

The work is thorough and well engineered. The implementation is integrated into vLLM 0.27 and covers TP1/TP4, batch sizes 1/8, speculative decoding (DFlash2), and more. Several design choices have real practical value: architecture-dependent clustering geometry (Sec. 3.2), page-mean residual INT4, and residual quantization on top of semantic clustering with dequantization of only O(KP)O(KP) entries upon opening.

The paper is fairly candid about key limitations. For example, it acknowledges that the video operator is quadratic, that the video evaluation is small in scale, and that its "exact + summary" structure is shared with RetroInfer and Double-P ("We do not claim that mixing exact and approximate regions is unprecedented").

Critical weaknesses

The central claim is not tested against sparse baselines. The paper argues that keeping low-resolution summaries is better than pruning outright. Yet the only comparison in the LLM section is against full attention; no sparse baseline is included. The Related Work section discusses Quest, InfLLM, ClusterKV, Squeezed Attention, RetroInfer, and Double-P and states that "These methods are central comparisons," but none of them is compared directly.

The evaluation setup may mask the effect of the approximation. For example, the first 16,384 prefill tokens are computed exactly, each later block keeps 16,640 exact local entries, and the region list is updated only every 16K tokens. For 64K ProLong documents, many token predictions are therefore made under exact or near-exact conditions. A PPL change of only 0.1–0.26% is unsurprising and says little, since PPL is inherently insensitive to long-range retrieval.

Novelty is limited; the contribution is mainly engineering integration. The structure of "exact clusters + size-weighted centroid estimates" already appears in RetroInfer and Double-P. NSA has compressed, selected, and local branches. On the video side, there are PSA, PISA, and others. The differences the paper discusses (online growable state, local window, paging/quantization, architecture-aware geometry, etc.) are incremental engineering choices. The paper lacks sufficient novelty and theoretical depth.

Reproducibility is weak. Many key hyperparameters (the 16K block, 512/256 windows, M=16NM = 16 \sqrt{N}, K=8K = 8, etc.) receive no sensitivity analysis. The work also depends on recent models and toolchains such as Qwen3.8, K2 Horizon, MiniMax-H3, and DFlash2, which raises the barrier to reproduction.

Questions for authors

What are the results on RULER 128K and LongBench v2 when (a) all closed summaries are removed (pure sparse retrieval), and (b) the summaries are kept but the log⁡n\log n term is removed? If the gap between (a) and LoD is small enough to be attributed to experimental noise, the paper's core claim does not hold.

Direct comparison with existing methods:*Under the same exact-KV budget, how does LoD compare with Quest, InfLLM, RetroInfer, Double-P, and similar methods in both quality and speed?

In long-decode scenarios, such as with thinking enabled or with long generations (e.g., 32K output tokens), regions are updated more frequently during decoding. Is quality maintained in this setting?

2 comments

Thank you for your detailed and thoughtful review, mentioning how LoD addresses problems with attention sparsification, and the practical utility of the design. We appreciate your points about needing comparisons to sparse baselines, further justification of LoD’s novelty, and ablations on our hyperparameters. We have provided many new experiments with explanations addressing all three issues below, and thank you for your consideration of this new information.

W1: (sparse baselines)

We have new experiments demonstrating speed and performance for LoD versus sparse baselines ClusterKV, RetroInfer, Multi-pole Attention, QUEST, and Squeezed Attention. Please see new Appendices A.6, A.7, and A.8 in the revised paper submission as well as our shared review response comments.

W2: (prefill evaluation setup masking evaluations)

For ProLong ppl evaluation specifically, we think that there is still meaningful signal to the evaluation. Please see new ablations on this topic in Appendix A.7, especially the Prefill block length experiment in A.7.6, where we show that perplexity is not significantly impacted by different block sizes. At the bottom of that subsection we also compare 16K to 1K prefill blocks on Qwen under RULER to directly test the issue. The K2 Horizon block sweep also shows that prompt loss stays close to the 16K control even at 1K and 512-token blocks. This is meant to address the exact-prefix dilution concern for natural-text loss.

W3: (novelty)

We appreciate the reviewer’s desire for a demonstration of algorithmic novelty. We strongly agree that the paper can benefit greatly from a detailed discussion on exactly what is different from prior work. We have updated the submission accordingly. To consolidate our response we have also added that discussion as a shared response comment herein, and kindly request your attention there.

W4: (hyperparameter sensitivity)

Thank you for making the very good point that there were no experiments shown around our hyperparameter choices. We have placed a wide range of new experiments around hyperparameter choice and sensitivity into Appendix A.7.

Q1: (summary removal ablations)

We agree that these ablations can help isolate the downstream contribution of summary retention and population correction and have added a 128K RULER using 32 samples per task in Appendix A.8. We have revised the paper to distinguish this established attention formulation from our contribution: the online region construction and refinement algorithm, with the processing bounds specified in Section 2.4. Appendix A.6 evaluates the complete algorithm against relevant alternatives, while Appendix A.7 examines its state-growth, routing, and refinement choices. We do not claim to have independently established the downstream benefit of summary retention or population correction within LoD. The revised contribution therefore does not depend on demonstrating a significant LoD-versus-pruning gap on these benchmarks.

We greatly value your time and attention, and sincerely appreciate your contribution to improving the paper.

We thank you again for your time and thoughtful feedback. We hope you have been able to review our responses in the comment above and in the shared comments and updated paper, in which we attempted to resolve all of the stated concerns, and would welcome a response, and any further thoughts or questions as the discussion period draws to a close.

Reviewer_irJbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Level of Detail (LoD) Attention, a training-free attention approximation that represents less important KV regions using population-corrected summaries instead of discarding them. It introduces hierarchical refinement, online semantic clustering, and INT4 residual KV storage. Experiments on two LLMs and a video generation model demonstrate efficiency gains with generally competitive quality.

Critical strengths
  1. The motivation for retaining approximate contributions from unselected KV entries is intuitive and practically relevant.
Critical weaknesses
  1. The central exact-or-summary formulation substantially overlaps with existing approaches. The novelty of the proposed attention formulation remains insufficiently justified.

  2. The LLM experiments primarily compare LoD against dense attention, without direct comparisons against existing efficient-attention methods. This makes it impossible to determine whether the proposed approach provides meaningful improvements over prior work.

  3. The video evaluation includes only one model, four prompts, and one random seed. The overall experimental setting is not solid.

Questions for authors

Refer to the weakness section.

3 comments

Thank you for your review highlighting the method’s practical benefits and paper’s comparisons to dense attention, while expressing specific concerns about the justification, experimental comparisons, and video evaluation setting. We believe we can address all of these issues to your satisfaction, and are glad to have the opportunity to do so in order to improve the paper.

W1. The exact-or-summary formulation overlaps with existing approaches.

We agree that exact attention combined with population-weighted summaries is established, as our Section 6 acknowledges. Our contribution is the specific online LLM algorithm: growing semantic regions retain their members for selective access, and partial refinement replaces a region by selected pages plus one summary of the remaining entries. With the stated growth schedule and refinement limits, attention and state construction together require O(N3/2)O(N^{3/2}) total work. Section 1 now states the narrower contribution. Sections 2.2 and 3.1 describe the refinement and online construction, Section 2.4 specifies the cost assumptions, and Section 6 provides the detailed comparison with prior work. Please see the Shared Review Response comments for more specifics.

Because reviewers had many similar requests, we have consolidated our responses into a single set of shared comments. We greatly appreciate your time and effort, and hope that this will make further review easier and more efficient. Your concerns are important to us, and we have done our best to address each one individually. New experiments and explanations around novelty, prior work baseline comparisons, and video experiment quality are directly addressed there. We thank you for your time and effort in continuing to help us to fully demonstrate the benefits of our method.

We thank you again for your time and thoughtful feedback. We hope you have been able to review our responses in the comment above and in the shared comments and updated paper, in which we attempted to resolve all of the stated concerns, and would welcome a response, and any further thoughts or questions as the discussion period draws to a close.

Thanks for the rebuttal. However, my main concerns about novelty and evaluation are still not resolved. Therefore, I will maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared Review Responses A,B

Shared Responses for all Reviewers, Parts A,B

A) Novelty and distinction versus prior work

We thank the reviewers for requesting a clearer account of LoD’s contribution. We agree that population-weighted mixtures of exact and approximate attention are established. We have revised the title, abstract, and introduction to remove the implication that “resolution allocation” is novel and focus on the specific online LLM algorithm. Section 1 now states the narrower contribution. Sections 2.2 and 3.1 describe the refinement and online construction, Section 2.4 specifies the cost assumptions, and Section 6 provides the detailed comparison with prior work.

LoD shares the population-weighted exact/summary formulation with RetroInfer and Double-P and builds on KVM’s append-and-merge organization and state-growth schedule. Its contribution is an online algorithm for pretrained LLMs combining growing semantic regions, retained member KVs, and bounded partial refinement while maintaining O(N3/2)O(N^{3/2}) total causal attention and state-construction work under complete processing-cost accounting. New tokens join semantic regions while their member KV entries remain stored under stable region ownership. Within a routed region, LoD promotes selected pages and replaces the parent summary with one summary of precisely the remaining entries. The residual’s count and key/value sums are computed by subtracting selected-page statistics from maintained region totals, without rereading unselected KV entries. This differs from whole-cluster promotion in RetroInfer and Double-P and from retaining separate summaries for unrefined children in hierarchical Multipole Attention. Multipole supports online cluster updates during decoding but retains dense prefill; LoD preserves KVM’s subquadratic processing regime while retaining members for query-dependent refinement. Squeezed Attention uses summaries for retrieval rather than retaining unselected regions as approximate operands in the final output, and relies on preprocessing fixed contexts. LoD targets fresh or evolving contexts where that preprocessing cannot be amortized. NSA learns compressed representations and combines separately computed branch outputs through learned gates; LoD retrofits pretrained attention using disjoint exact/summary contributions under shared normalization. Paging, quantization, and local windows support the implementation, with partial refinement limiting quantized-archive reconstruction to selected pages. Video LoD is an application study in the direction established by PSA and PISA; we claim neither separate algorithmic novelty nor subquadratic total complexity for video. The experiments support the practical significance of this combination. Appendix A.6 directly compares LoD with RetroInfer, Multipole Attention, Squeezed Attention, QUEST, and ClusterKV. Under both published and LoD-matched baseline settings, two-tier LoD achieves the lowest measured whole-model decode latency across the tested context lengths and batch sizes, while lying on the observed speed–quality frontier at 32K and 64K. Prefill measurements include prompt processing and required cache/index construction, reflecting fresh-context workloads where these costs cannot be amortized. All evaluated baselines retain dense causal prefill, making LoD’s joint support for efficient prefill, state construction, and decoding a substantive distinction. Appendix A.7 provides ablations supporting the trade-offs among region growth, routing budgets, and page refinement. The new Appendix experiments indicate that LoD lies on or extends the pareto frontier for attention approximation methods. While this does not prove that LoD is SoTA, it does make a compelling empirical argument for LoD's algorithmic benefits.

We could not directly evaluate Double-P because released code was unavailable at the time of our experiments. Our comparison therefore concerns its published formulation, and we claim no empirical speed or quality advantage over it.

B) Experiments showing LLM LoD versus prior work on both speed and quality

Thank you for pointing out that beyond the paper’s extensive comparisons to full attention, we need experimental data comparing other methods. We have new experiments showing this for LoD versus ClusterKV, RetroInfer, Multi-pole Attention,QUEST, and Squeezed Attention, matching their codebase and paper settings as closely as possible on an H100-SXM. We are unfortunately unable to directly compare to Double-P as they did not release code, but we can provide a relative ratio to other tests from comparisons shown in that paper. Please see Appendix A.6 in the updated paper submission for these experiments and discussion.

AuthorsOpenReview ↗

Shared Review Response C

C) Video experiments

To address concerns about variance across seeds and number of prompts, we doubled the number of prompts to 8 and ran three seeds (12345, 23456, 34567) across all prompts. Please find these added to paper Appendix C in the updated paper submission, where we now show plots of the data across seeds with a range as well as image comparisons covering all eight prompts.

Please note that Video LoD is only a demonstration that the overall approach can be used effectively in other modalities beyond language, and is not intended to rigorously imply that Video LoD is SoTA. Extending it to multiple models is unfortunately beyond the scope of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.