- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Level-of-Detail (LoD) Attention to reduce the computational and memory overhead of long-context attention. The core idea is to replace the “keep-or-drop” mechanism of traditional sparse attention with a multi-resolution representation: for important regions that are routed in, precise attention is performed using the original key-value (KV) pairs; for other regions, aggregated KV summaries with population correction are used to continue participating in the softmax, thereby avoiding the complete discarding of unselected tokens. The authors further propose a recursive three-tier LoD, which continues to select pages within the chosen region for precise computation and represents the remaining tokens using a residual summary. Additionally, the paper combines online semantic region construction, region-owned page organization, and mean-relative INT4 KV compression, and applies similar ideas to video generation. Experiments evaluated perplexity, LongBench v2, RULER, prefill/decode latency, and KV-cache memory on Qwen3.8-27B and K2-Horizon-32B, reporting significant attention-level speedups in long-context scenarios; the video section included limited comparisons with dense attention and Sol Attention on MiniMax-H3.
Critical strengths
The system is fairly comprehensive and demonstrates practical acceleration effects for long contexts. The paper not only proposes attention approximation but also implements online region maintenance, recursive page routing, INT4 KV storage, and the corresponding kernels. On Qwen, the authors report significant attention-level prefill/decode speedups with 128K context, indicating that the method has practical engineering applicability.
Critical weaknesses
- The core method lacks novelty, and LoD’s primary algorithmic contribution overlaps significantly with existing work. The core idea behind LoD is to apply exact attention to important regions while allowing unselected regions to continue participating in normalization via a coarse summary with population correction. However, this “coarse representation + selective exact refinement” paradigm overlaps significantly with existing work. RetroInfer (PVLDB 2026) has already employed segmented clustering and tripartite attention approximation to perform exact computations on some content and approximations on the rest; Squeezed Attention (ACL 2025) has also employed semantic clustering, centroid routing, and hierarchical refinement. The paper itself acknowledges that the combination of exact and approximate regions is not a novel proposal.
- The absence of critical baselines prevents the experiment from demonstrating that LoD has advanced the state of the art. The main experiments for LLMs primarily compare Full Attention with various LoD variants. However, Full Attention is not the most significant competing method in this paper. Existing methods such as QUEST (ICML 2024), Squeezed Attention (ACL 2025), and RetroInfer (PVLDB 2026) all focus on accelerating long-context attention and report real kernel or end-to-end performance. In particular, RetroInfer and Squeezed Attention are highly relevant to the design of our method. Without a direct comparison under identical model, hardware, context length, and quality degradation conditions, the attention speedup reported in this paper cannot demonstrate that LoD has an advantage over existing methods.
- There is no fair accuracy–efficiency Pareto comparison. LoD does not fully maintain the performance of Full Attention on RULER. For example, Qwen’s average score drops from 96.17 to 94.39, and even more significant degradation is observed on other benchmarks. Simply reporting accuracy and speedup separately is insufficient, and the authors should compare E2E latency, attn latency, and memory with existing methods under the same quality loss to demonstrate whether LoD achieves a superior Pareto frontier.
- Ablation is insufficient. The performance gain of this paper also depends on several tricks, including kernel optimization and INT4 compression. Currently, there are insufficient experiments to distinguish the contributions of the LoD itself from those of these engineering techniques.
Reference
- Chen et al., RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference, PVLDB 2026.
- Hooper et al., Squeezed Attention: Accelerating Long Context Length LLM Inference, ACL 2025.
- Yuan et al., Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, ACL 2025 Best Paper.
- Tang et al., QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024.
- Hooper et al., Squeezed Attention, ACL 2025.
Questions for authors
-
Compared to existing exact-plus-approximate attention methods such as RetroInfer, what are the specific new mechanisms at the core algorithmic level in LoD two-tier?
-
Why were RetroInfer, Squeezed Attention, and QUEST not included in the main experiments? Under the same model, hardware, context length, and accuracy constraints, does LoD still offer a better latency/throughput/memory Pareto frontier?
-
Is the performance degradation observed in RULER for Multikey, HotpotQA, and SQuAD related to routing errors or region approximation errors? Could the authors provide a corresponding analysis?
-
Could you provide more ablation studies?
2 comments
We thank you again for your time and thoughtful feedback. We hope you have been able to review our responses in the comment above and in the shared comments and updated paper, in which we attempted to resolve all of the stated concerns, and would welcome a response, and any further thoughts or questions as the discussion period draws to a close.
Thank you for your attentive review mentioning the practicality of the method and comprehensiveness of the system, as well as the specific and important changes you need to see in the paper to comprehensively address novelty, related work comparisons, quality, and ablations. We have completed a large regime of new experiments and we hope you find that these address the concerns. Because there was significant overlap across reviewer requests, we have consolidated some of our responses into separate shared review response comments in an effort to make things easiest for the reviewers.
Regarding W1/Q1, we agree that this general strategy of coarse representations and selective refinement is shared. The specific distinctions are:
For two-tier LoD, the distinction concerns online region construction with retained member KVs and bounded refinement under the schedule in Sections 2.4 and 3.1. We do not claim a new population-weighted exact/summary formula. The recursive variant additionally opens selected pages and represents the remaining entries by one residual summary, as defined in Section 2.2. We now state this contribution in Section 1 and distinguish it from the closest prior methods in Section 6.
We have made these distinctions explicit as mentioned in the shared comments, and in Section 6 of the updated paper, which have more details and information.
We think the paper has been improved tremendously by the feedback so far. We would be happy to discuss any outstanding issues and provide whatever additional evidence we can. We have worked hard to provide as many new experiments as possible, each highly targeted to specific reviewer concerns, and sincerely appreciate your time and effort helping us to make the paper the best it can be.