- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This manuscript proposes a training-free sparse-attention method for long-context inference that exploits the observation that useful sparse routing patterns often persist across neighboring Transformer layers. Rather than selecting sparse attention routes independently at every layer, it maintains a compact cross-layer "Stripe Memory" of previously useful lag-based diagonal attention patterns and combines this historical signal with current-layer evidence using a reliability-aware update rule. It further applies a token-level Mask Refinement step to conservatively add or remove blocks from the resulting sparse mask. Experiments on LongBench and InfiniteBench with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct show that EchoStripe achieves accuracy close to strong sparse-attention baselines while reducing attention-operator latency, with ablations suggesting that both the cross-layer memory and mask-refinement components contribute to the gains.
Critical strengths
The manuscript’s main strength is its observation that sparse-attention routing patterns exhibit meaningful cross-layer persistence, and the proposed method provides a simple way to exploit this dependency rather than treating each layer’s routing decision independently. In addition, the approach is training-free and maintains only a compact cross-layer routing state, making it relatively practical for long-context inference. However, the broader idea of exploiting inter-layer dependencies has already been explored in prior sparse-attention work, most notably in [1]. The manuscript does not discuss this closely related work or clearly distinguish its contribution from it, which makes it difficult to assess the novelty of the proposed cross-layer routing mechanism.
[1] Hoang Anh Duy Le, et al. Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference. ICML 2026.
Critical weaknesses
1.) The motivation is not convincingly established: While the manuscript shows that sparse routing patterns persist across layers, it does not demonstrate that ignoring this dependency significantly harms the accuracy of existing sparse-attention methods. Controlled experiments quantifying this effect would strengthen the motivation.
2.) The novelty is unclear relative to prior work on exploiting inter-layer dependencies: The core idea of reusing attention or routing information across layers is closely related to [1] as mentioned in Strengths, yet this work is neither discussed nor compared against, making the novelty of EchoStripe difficult to assess.
3.) The manuscript is difficult to follow and insufficiently well positioned: It introduces several mechanisms before clearly defining the specific limitation of prior work (For example, lag-based stripes, Stripe Memory, reliability-aware fusion, and Mask Refinement), obscuring the central contrast with current-layer-only dynamic sparse routing and making the main argument unnecessarily hard to reconstruct.
4.) The contributions are not conceptually well integrated: Stripe Memory addresses cross-layer persistence, while Mask Refinement is a separate token-level correction mechanism, and the paper does not clearly justify why these components form a unified method rather than two loosely combined heuristics.
5.) The empirical gains are modest relative to the added complexity: EchoStripe introduces additional routing state, reliability estimation, and refinement logic, yet often achieves only small accuracy improvements over strong sparse baselines. Moreover, the evaluation focuses mainly on attention-operator latency rather than demonstrating substantial end-to-end inference speedups under realistic workloads.
Questions for authors
1.) Refer to Weaknesses.
2.) Can the authors provide a simpler motivating example or visualization showing a route that current-layer-only selection misses but EchoStripe successfully preserves using cross-layer evidence?
3.) How consistent is the historical stripe evidence across different layers, heads, models, tasks, and sequence lengths?