acceptodds
Under review as a conference paper at ICLR 2027

Deterministic Causal Multiscale Attention for Efficient Pretrained-Model Retrofitting

Abstract

Long-context transformer inference is dominated by quadratic causal attention, but many sparse methods trade efficiency for irregular execution or unclear model-level benefits. We present ButterFly, a deterministic, block-aligned causal multiscale attention pattern combining a small exact local neighborhood, the first global block, and logarithmically spaced historical blocks. For fixed block width, ButterFly uses O(n log n) selected token interactions and O(log n) graph communication depth per converted layer. Its static structure permits exact causal execution with predictable memory access and no learned router at inference time. We implement ButterFly in BF16 and retrofit eight of 28 attention layers in Qwen3-0.6B. On an NVIDIA RTX PRO 6000 Blackwell GPU, ButterFly crosses dense attention between 2K and 4K tokens, reduces selected interactions by 27.10× at 32K, and achieves 2.84×, 5.50×, and 11.91× per-layer prefill speedups at 8K, 16K, and 32K. The partial-model retrofit lowers warm whole-model time to first token by 7.7%, 11.3%, and 16.3% at the same lengths. Under a matched 1,000-step, 16.384M-token adaptation protocol, ButterFly ranks first across three training seeds, with mean report perplexity 1.68 versus 1.70 for an equal-budget static-random nonlocal graph, 1.69258 for dense continued training, and 1.81505 for equal-budget local sliding. These results establish ButterFly as a correctness-gated deterministic sparse-attention operator and selective pre-trained-model retrofit with measurable long-context systems value. They do not establish a fully subquadratic model, compressed total KV storage, or universal preservation of retrieval and reasoning capability; the remaining dense layers keep the complete model asymptotically quadratic.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, multiscale attention, long-context transformers, efficient inference, pretrained-model retrofitting, attention acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nk5cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses a heuristic method to reduce KV reads and attention computation, but this acceleration comes with a clear performance drop. I did not gain any unexpected or new knowledge from the paper. The validation of the paper is attacked by its own Limitations section and Appendix G, and in my view, this issue is fatal.

Critical strengths

None.

Critical weaknesses

The paper gives a relatively heuristic solution, but it does not justify why this speed-quality trade-off is acceptable. In fact, I think works such as FlexPrefill[1] and ProxyAttn[2] have demonstrated trade-offs that are entirely above BF1. The claims of the paper are deliberately narrowed, which is generally a good thing because it avoids creating excessive expectations about the paper. However, the contribution demonstrated by the claims themselves is not sufficient to meet the contribution standard of ICLR. The theory is very shallow, consisting mostly of simple implications directly from the construction itself. The method does not show obvious novelty, and the experiments also fail to demonstrate sufficient advantages.

The paper is very difficult to read. For example, the abstract and introduction almost completely fail to explain the motivation of this work, what need in the field it is responding to, and how it is supposed to attract researchers' interest. In fact, around 70% of the abstract is spent reporting experimental result numbers, which I find confusing. BTW, the title on OpenReview is inconsistent with the title in the PDF.

[1] Lai, X., Lu, J., Luo, Y., Ma, Y., & Zhou, X. (2025). Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. ICLR 2025.

[2] Wang, Y., He, H., Bao, S., Wang, H., Zhu, Q., & Che, W. (2026). Proxyattn: Guided sparse attention via representative heads. ICLR 2026.

Questions for authors

I have no questions for the authors, because my concerns are mainly about the contribution and overall presentation, and these may be difficult to address during the rebuttal period.

Reviewer_JH4iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BF1, a deterministic block-sparse attention pattern for retrofitting pretrained LLMs. Each query attends locally, to the first block, and to historical blocks at dyadic distances, giving O(nlog n) selected interactions while retaining standard softmax and the full KV cache.

The paper contains two main experimental tracks. A controlled Qwen3-0.6B study evaluates kernel speed and adaptation quality, while a separate Qwen3.8-27B/B200 study explores scaling up to 1M tokens. The reported system gains are large at very long context, including substantial prefill and whole-model TTFT reductions, but the large-model experiments are primarily engineering feasibility measurements rather than matched statistical evaluations.

Critical strengths
  1. The system speedups at long context are substantial. The paper demonstrates that a simple deterministic sparse pattern can translate into large reductions in attention work and meaningful whole-model TTFT reductions at very long sequence lengths.

  2. The authors make a serious effort to separate what is and is not supported by the experiments. In particular, the paper explicitly distinguishes operator speedup, whole-model speedup, KV memory, and quality preservation rather than conflating them. This makes the experimental limitations easier to assess.

Critical weaknesses
  1. The controlled study compares mainly against dense continued training, sliding-window attention, and one static-random graph. However, the paper itself discusses methods such as MInference, HiP, XAttention, Quest, FlexPrefill, and Native Sparse Attention, which address closely related long-context sparse-inference problems. Even if their mechanisms differ, some matched latency–quality comparison is necessary to understand whether BF1 offers a competitive Pareto point. This concern is compounded by the 27B timing protocol: repeated BF1 runs are reported using the best observed value, while some dense comparisons are single recorded runs, so the headline 98×/8.47× numbers should not be interpreted as typical speedups.

  2. The quality evidence is not sufficient for the intended long-context setting, and one reported result is concerning. The main replicated quality result is next-token perplexity on 2K packs with a 0.6B model, while the major systems claims concern 32K–1M contexts and a 27B model. The 27B quality diagnostic contains only six examples. More importantly, the paper reports a separate 32K structured-task experiment where replacing dense attention with the BF1 route causes roughly a 31–32 percentage-point accuracy drop. This suggests that favorable short-context perplexity does not establish preservation of retrieval or composition ability.

Questions for authors
  1. Can the authors provide a matched comparison with one or more strong pretrained-model sparse-attention baselines, e.g., MInference, HiP, XAttention, or FlexPrefill, using the same model, hardware, context length, sparsity budget, and quality benchmark?

  2. Can the authors evaluate BF1 on standard long-context retrieval/reasoning benchmarks such as RULER or LongBench at 32K or longer? In particular, how should the approximately 31–32 point structured-task degradation reported in the appendix be interpreted?

  3. Can the 27B timing experiments be rerun with matched repetition counts and ordering for dense and BF1, reporting median and variance rather than BF1 minima against single dense measurements? This would materially change how I interpret the headline speedups.

  4. The BF1 advantage over the static-random graph in the small-model PPL experiment is only around 0.3%, and only one random graph is used. Does BF1 remain better when comparing against a distribution of independently sampled matched-degree random graphs rather than a single draw?

Reviewer_B2HpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  1. Method. This paper proposes BF1, a deterministic block-sparse attention pattern. Each 64-token block attends to its local neighborhood, the first block, and historical blocks at powers-of-two offsets. This reduces attention interactions to O(nlog⁡n)O(n \log n) per converted layer while retaining the full KV cache.

  2. Small-model experiments. The authors replace attention in the (middle) 8 of 28 layers of Qwen3-0.6B and continually pretrain the model for 1,000 steps on 16.4M tokens at 2K context length. They compare BF1 against dense continued training, sliding-window attention, and static, random sparse attention, under the same budget. Across three training seeds, BF1 achieves a mean perplexity of 1.686, compared with 1.693 for dense, 1.692 for random, and 1.815 for sliding-window attention. At 32K context (OOD), BF1 achieves a 10.91x per-layer prefill speedup and reduces whole-model time to first token by 15.3%.

  3. Large-model experiments. The authors replace all 16 full-attention layers in Qwen3.8-27B with BF1, keeping the pretrained weights unchanged. At 1M tokens, they report a best-observed 98x per-layer prefill speedup and 8.47x whole-model time-to-first-token speedup over the recorded dense measurements. They also evaluate a six-example retrieval diagnostic using oracle-selected evidence blocks, obtaining 4/6 correct answers compared with 0/6 for a random control.

Critical strengths
  1. Controlled quality comparison. The authors compare BF1 against dense continued pretraining and two sparse attention patterns using the same training budget, trainable parameters, and data order. The results are consistent across three training seeds. Including dense continued pretraining is particularly useful for separating improvements from additional training from those due to the attention pattern.

  2. Efficient implementation and model-level measurements. The authors implement BF1 with correctness checks against the reference attention operation and demonstrate substantial kernel-level speedups at long context lengths. They also measure whole-model inference latency, including a large-scale experiment with Qwen3.8-27B at 1M tokens.

Critical weaknesses
  1. The benefit of the proposed attention pattern over simpler alternatives is not established.

    While logarithmically spaced attention has been studied previously (e.g., LogSparse), I see the main potential contribution of this work as demonstrating that this pattern offers a better quality–efficiency tradeoff when retrofitting pretrained models.

    • Baseline comparison: Table 1 shows that BF1 substantially outperforms sliding-window attention (1.686 vs. 1.815 PPL). However, it is unclear whether the sliding-window baseline also attends to the first block (with Attention sinks). Since BF1 includes this connection, the comparison does not isolate the benefit of dyadic connections from that of first-block access.
    • A more informative comparison would be sliding-window attention with the same first-block access and total attention-pair budget, using the remaining budget for a larger local window.
  2. The quality evaluation is too limited to judge the ability of BF1 to preserve model capabilities over baseline approaches.

    The main quality result is perplexity on 2K sequences from a local ORCA-style QA corpus. For a method that removes direct access to most historical tokens, I would like to see a broader evaluation of the following:

    • Short-context performance: Held-out LM perplexity, standard downstream benchmarks (e.g., HellaSwag, PIQA, ARC), and retrieval (NIAH/RULER).
    • Long-context performance: Continue pretraining the 0.6B models at 32K, then evaluate long-document perplexity (including loss by token position) and retrieval (NIAH/RULER). Comparing BF1 with dense attention and sliding-window attention with first-block access would help establish if BF1 has a better quality to efficiency tradeoff.
    • Existing evals: Appendix C.6 reports a substantial performance drop on a 32K structured task (OOD), but the documented continued pretraining uses 2K sequences. It remains unclear whether this gap persists after continued pretraining at 32K. I believe most modern models are inferred within their in-distribution context window. The six-example oracle-routing experiment on the 27B model is not a standard LM eval and also does not directly establish retrieval performance for the unmodified BF1 pattern.
Questions for authors
  1. Comparison with sliding-window attention and first-block access.

    • Could the authors clarify the sliding-window mask used in Table 1? In particular, what is the window size, and does it retain the first 64 tokens? For the random baseline, which blocks are always retained, and how are the remaining blocks sampled?
    • Could the authors compare BF1 with sliding-window attention that also retains the first 64 tokens, increasing the local window to match BF1's total number of causal attention pairs? The continued pretraining setup should otherwise remain unchanged. This would help isolate the benefit of the logarithmically spaced connections.
  2. Short-context LM and retrieval performance.

    • Could the authors evaluate the 2K models on held-out LM perplexity using a corpus separate from the adaptation data, standard downstream tasks (e.g., HellaSwag, PIQA, ARC), and retrieval benchmarks (e.g., NIAH/RULER)?
    • I am particularly interested in comparing dense attention, BF1, and sliding-window attention with first-block access. This would help establish whether the perplexity improvements in Table 1 translate to broader model capabilities.
  3. Long-context performance after continued pretraining at 32K.

    • Could the authors continue pretraining Qwen3-0.6B at 32K with dense attention, BF1, and sliding-window attention with first-block access? The training data, token budget, objective, and trainable layers should be kept the same across methods, and each model should be evaluated using its intended attention pattern.
    • I would be interested in long-document perplexity, including loss broken down by token position (as studied in TTT-E2E), and retrieval performance on NIAH/RULER across different evidence positions. LongBench would also be a useful additional evaluation where the base model performs meaningfully.
    • Appendix C.6 reports a substantial performance drop with BF1 at 32K, but the documented continued pretraining uses 2K sequences. Does this gap persist when the model is continued-pretrained and evaluated at the same context length?
  4. Choice of converted layers.

    • Why were layers 10–17 selected for the Qwen3-0.6B experiments? Was this choice based on quality or latency measurements? If other layer counts or placements were evaluated, reporting those results would help explain the choice of configuration.

TTT-E2E: End-to-End Test-Time Training for Long Context. Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, Yu Sun

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.