acceptodds
Under review as a conference paper at ICLR 2027

EchoStripe Attention: Cross-Layer Stripe Memory for Efficient Long-Context Inference

Abstract

Long-context inference is constrained by the quadratic cost of self-attention, whereas aggressive sparsification can remove structured long-range interactions needed by downstream tasks. We focus on lag-indexed diagonal stripes in the attention matrix: query–key pairs with the same relative offset share a common lag and can be represented compactly by a one-dimensional profile. Existing dynamic routing typically treats each layer as an independent decision, making the resulting sparse route sensitive to local fluctuations. We introduce EchoStripe, a training-free framework for stateful cross-layer sparse routing. Its Stripe Memory recursively propagates a compact preceding-layer lag-profile state and fuses it with current-layer evidence using a head-specific reliability-aware weight. EchoStripe further applies Mask Refinement, a bounded token-guided procedure that corrects the complete causal base mask through conservative additions and pruning. Across LongBench with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, and InfiniteBench with Llama-3.1-8B-Instruct, EchoStripe remains within 0.42, 0.85, and 1.16 points of the strongest sparse baseline, respectively. At 128K tokens, it reduces measured attention-operator latency by 5.20× relative to MInference and achieves a 7.73× speedup over Full Attention. On the common 20-dataset LongBench suite, Stripe Memory and Mask Refinement jointly improve the current-layer-only variant by 5.90 points. These results show that cross-layer routing state provides a favorable accuracy–efficiency operating point for long-context inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context inference, sparse attention, efficient Transformers, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n7vbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript proposes a training-free sparse-attention method for long-context inference that exploits the observation that useful sparse routing patterns often persist across neighboring Transformer layers. Rather than selecting sparse attention routes independently at every layer, it maintains a compact cross-layer "Stripe Memory" of previously useful lag-based diagonal attention patterns and combines this historical signal with current-layer evidence using a reliability-aware update rule. It further applies a token-level Mask Refinement step to conservatively add or remove blocks from the resulting sparse mask. Experiments on LongBench and InfiniteBench with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct show that EchoStripe achieves accuracy close to strong sparse-attention baselines while reducing attention-operator latency, with ablations suggesting that both the cross-layer memory and mask-refinement components contribute to the gains.

Critical strengths

The manuscript’s main strength is its observation that sparse-attention routing patterns exhibit meaningful cross-layer persistence, and the proposed method provides a simple way to exploit this dependency rather than treating each layer’s routing decision independently. In addition, the approach is training-free and maintains only a compact cross-layer routing state, making it relatively practical for long-context inference. However, the broader idea of exploiting inter-layer dependencies has already been explored in prior sparse-attention work, most notably in [1]. The manuscript does not discuss this closely related work or clearly distinguish its contribution from it, which makes it difficult to assess the novelty of the proposed cross-layer routing mechanism.

[1] Hoang Anh Duy Le, et al. Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference. ICML 2026.

Critical weaknesses

1.) The motivation is not convincingly established: While the manuscript shows that sparse routing patterns persist across layers, it does not demonstrate that ignoring this dependency significantly harms the accuracy of existing sparse-attention methods. Controlled experiments quantifying this effect would strengthen the motivation.

2.) The novelty is unclear relative to prior work on exploiting inter-layer dependencies: The core idea of reusing attention or routing information across layers is closely related to [1] as mentioned in Strengths, yet this work is neither discussed nor compared against, making the novelty of EchoStripe difficult to assess.

3.) The manuscript is difficult to follow and insufficiently well positioned: It introduces several mechanisms before clearly defining the specific limitation of prior work (For example, lag-based stripes, Stripe Memory, reliability-aware fusion, and Mask Refinement), obscuring the central contrast with current-layer-only dynamic sparse routing and making the main argument unnecessarily hard to reconstruct.

4.) The contributions are not conceptually well integrated: Stripe Memory addresses cross-layer persistence, while Mask Refinement is a separate token-level correction mechanism, and the paper does not clearly justify why these components form a unified method rather than two loosely combined heuristics.

5.) The empirical gains are modest relative to the added complexity: EchoStripe introduces additional routing state, reliability estimation, and refinement logic, yet often achieves only small accuracy improvements over strong sparse baselines. Moreover, the evaluation focuses mainly on attention-operator latency rather than demonstrating substantial end-to-end inference speedups under realistic workloads.

Questions for authors

1.) Refer to Weaknesses.

2.) Can the authors provide a simpler motivating example or visualization showing a route that current-layer-only selection misses but EchoStripe successfully preserves using cross-layer evidence?

3.) How consistent is the historical stripe evidence across different layers, heads, models, tasks, and sequence lengths?

Reviewer_5pi4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free sparse attention mechanism to accelerate long-context inference while preserving accuracy for downstream tasks. They claim that other dynamic sparse attention methods make layer-by-layer decisions causing models to drop useful long-range structural dependencies, especially the equal-lag diagonal stripes in the attention matrix due to local fluctuations. To tackle this, the authors propose Echo Stripe that recursively propagates a compact, 1D lag-profile state from preceding layers to fuese it with the current ones. The authors evaluate their method on longbench and infinite bench to show that they get comparable performance with full attention with reduced latency.

Critical strengths
  • The proposed method is train-free plug-in module making it easy to deploy on existing off-the-shelf models.
  • Approaching sparse attention routing as a recursive depth-aware problem where layer-wise operations can be dependent is novel and relevant.
  • The authors demonstrate that the StripeMemory transfers successfully to non-fourier estimator proving that the recursive memory is not just an artifact.
Critical weaknesses
  • There are a lot of hardcoded hyperparameters here that seem to be hard to tune such as thresholds for the stripe detection, exponential moving average bounds, pruning budgets etc.
  • While the cross-layer Stripe memory is a novel concept introduced by the authors, the base FFT lag extraction mechanism has been proposed before.
  • For the "non-strip" heads that the authors classify, the stripe memory is bypassed. The paper lacks exploration on what percentage of heads are actually triggering the stripe routing and ablations on this.
Questions for authors
  • I'm curious how the hyperparameters generalize to different architectures and budgets and how much the performance varies depending on this.
  • Are there any other ablations that explain the points elaborated in weaknesses point?
Reviewer_Bu6bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EchoStripe, a training-free sparse attention method for long-context prefilling. It builds on a frequency-domain lag router (FFTSparse) that selects diagonal "stripes" of the attention matrix, i.e., query-key pairs with the same relative offset. The first addition, Stripe Memory, fuses the current layer's one-dimensional lag profile with a recursively propagated profile from preceding layers, using a head-specific weight based on historical agreement, current confidence and stripe structure, while keeping the number of selected lags unchanged. The second, Mask Refinement, uses token-level scores to add a few promising blocks to the combined base mask and to prune weak ones. Experiments on LongBench (Llama-3.1-8B, Qwen2.5-7B) and InfiniteBench (Llama-3.1-8B) show accuracy slightly below MInference and full attention, and 7.73x faster attention than full attention at 128K on an RTX 5090.

Critical strengths
  1. The idea of reusing routing evidence across layers is simple and cheap. Stripe Memory only keeps a one-dimensional lag profile per head and does not change the lag budget, and the controls are well designed: replacing the FFT extractor with a non-Fourier estimator keeps the gain, while shuffling the lag coordinates of the history removes it (Table 9). This supports that the benefit comes from aligned cross-layer evidence rather than generic smoothing.

  2. The efficiency at long context is good. At 128K, EchoStripe is faster than XAttention and FlexPrefill while scoring higher on LongBench, and it improves over FFTSparse by about 2.5-3 points at almost the same latency. The latency breakdown (Tables 5-6) also shows that the added modules cost little.

Critical weaknesses
  1. EchoStripe is not the most accurate method in any setting. MInference and full attention are better on all three benchmarks. The fair comparison is FFTSparse, which this method is built on, and the gain there is about 2.5 to 3 points. However, FFTSparse is an anonymous paper under review, so it is hard for me to judge what is new in this work.

  2. The speedup over MInference is hard to trust. In Table 4, MInference is slower than full attention up to 64K. In Table 5, its kernel takes 13.6s at 128K, while the other methods take about 2.5s at the same keep ratio. This seems to be an implementation or hardware problem, not a real property of MInference. In addition, the speed tests use different inputs and keep ratios from the accuracy tests, and only measure the attention operator on one RTX 5090.

  3. Some results make me doubt the evaluation setup. Full attention with Qwen2.5-7B gets only 13.00 on HotpotQA and 3.42 on NarrativeQA, which is much lower than expected. InfiniteBench is run with a 4-bit quantized model. In the ablations, removed modules are replaced with random blocks, which probably makes each module look more useful than it is. A fairer test would give the extra budget to the current-layer selector instead. There are also many hand-set hyperparameters, with no sensitivity study.

  4. Several recent prefill sparse attention methods are missing, such as ProxyAttn [1], RRAtention [2] and FlashPrefill [3]. They are neither compared nor discussed; the newest baselines here are MInference, FlexPrefill and XAttention. Without them it is hard to tell where EchoStripe stands among current methods.

[1] Wang et al. ProxyAttn: Guided Sparse Attention via Representative Heads. ICLR 2026.

[2] Liu et al. RRAtention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference. ACL 2026.

[3] Fan et al. FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling. arXiv 2603.06199, 2026.

Questions for authors
  1. FFTSparse is an anonymous submission, so it's hard to judge the novelty here. Could you explain what EchoStripe adds on top of it, and share its code or a short description?

  2. Why is MInference so slow in your setup (13.6s kernel time at 128K with a 10% keep ratio)? Did you use its official kernel? Results on an A100 or H100 would help.

  3. In the ablations, could you give the freed budget to the current-layer selector instead of random blocks? That would show how much each module really helps.

  4. Why are some full-attention scores so low, e.g., 13.00 on HotpotQA and 3.42 on NarrativeQA with Qwen2.5-7B? Is there a truncation or prompt issue?

  5. Could you report accuracy and latency at the same keep ratio and on the same inputs? A sensitivity study on the main hyperparameters (α range, thresholds, refinement budgets) would also help.

  6. Could you compare with more recent prefill methods such as ProxyAttn, RRAtention and FlashPrefill at the same keep ratio?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.