acceptodds
Under review as a conference paper at ICLR 2027

When Shorter Is Better: Very Short Sliding Windows in Gated Hybrid Attention

Abstract

As modern applications demand larger models and longer contexts, global attention has become a dominant cost in both training and inference. Hybrid attention that combine sliding-window attention with global attention reduce this cost, but are typically viewed as trading quality for efficiency, with larger windows assumed to be better. In this paper, we revisit this view through the lens of recent findings on pretrained LLMs, in particular the literature on attention sinks. We find that hybrid architectures with very short sliding windows (16 or 32 tokens) outperform both the full global attention baseline and hybrids with larger windows, even though the latter are strictly more expressive. Adding conditional gating that dynamically balances head contributions further amplifies these gains. We validate both findings with ablations over head configurations, architectural variants, sequence lengths, and training budgets. Our results challenge the conventional trade-off between window size and quality, suggesting that short windows act as a beneficial inductive bias rather than a cheap approximation to global attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sliding Window Attention, Gated Attention, Hybrid Attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_b2C4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates context length in global attention modules within transformers. The general consensus has been that the larger the model the longer the context should be to offer better performance. The authors revisit this problem through the perspective of attention sinks. Their insight is that these sinks act as a type of gating and coupled with some observations in the literature they reason that building in short context windows into some of the layers provides the right prior for better model performance.

Critical strengths

The paper is nicely written with a good literature review and a clear explanation of the problem the authors are addressing.

Critical weaknesses
  1. The authors don't really explain how building a short sliding window right from the start gives the model the right prior for better performance. In fact, they don't really give detailed insight in how it yields the right prior. They give the impression that it is the right way to build a local prior but why does that actually lead to better performance? In this way real insight into what is going on is lacking in the paper.

  2. Given that the paper doesn't offer any real insight as to why their approach can yield better performance I would have expected a few more experiments on different architectures and different modalities.

Questions for authors
  1. Can you explain why having a prior by building in short sliding windows from the start leads to better model performance in a more detailed way with some explicit insight?

  2. Wouldn't building in such a prior through short sliding windows at the start affect some form of long context tasks like long context needle in the hay stick tasks or more generally long context recall problems?

  3. Does this idea hold for general transformer or are you suggesting that it only holds for LLMs?

Reviewer_CFQ8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper revisits hybrid attention designs (sliding-window + global attention), motivated by two observations from the literature on attention sinks.As a result, it proposes Gated Hybrid Attention (GHA) that pairs very short sliding windows on a subset of heads with input-dependent sigmoid output gates on all heads, and drops sink tokens entirely. On 1B models trained from scratch w=32 hybrids outperform both full global attention and w=1024 hybrids in perplexity. The authors report the trend holds across ablations and transfers to Llama-3-8B and Qwen3-4B trained on ~400B tokens.

Critical strengths
  • Fig. 2 finds a consistent U-shape with minimum near w=16–32, rather than a single cherry-picked operating point.

  • Ablations are broad, including head/kv counts, head width, QK vs value dimension, RoPE settings, QK-norm, sandwich norm, short convolutions, two training budgets, two model families.

Critical weaknesses
  • The paper's framing is that hybrid attention is "typically viewed as trading quality for efficiency," and the contribution is to overturn this view. But the research community and industry have already moved past that framing: Gemma 2/3, Mistral, gpt-oss, and OLMo 3 are all hybrid. The live design questions are window size, sink retention* and layer/head allocation in production configurations. The paper answers none of these against a real baseline. The paper does not compare against a single existing recipe.

  • The paper opens with global attention being "a dominant cost in both training and inference" reports no efficiency numbers.

  • The paper trains a Qwen3 variant at 32k context length and then evaluates exclusively on short-context benchmarks. No retrieval task (needle-in-haystack, RULER, LongBench, perplexity on long documents) appears anywhere.

Questions for authors
  • Against a production hybrid recipe (e.g., Gemma-3-style 5:1 local:global layer interleave with w=1024, or gpt-oss alternating layers with sink retention), does GHA(32,50%) win on quality, efficiency, or both?

  • Does GHA(32,50%) degrade on long-context retrieval (e.g., RULER, needle-in-haystack) relative to gated global attention and to w=1024 hybrids with sinks retained?

Reviewer_t4CuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates very short sliding windows in gated hybrid attention, combining local and global attention heads. The main sweeps use a 1B-parameter model trained on approximately 105B tokens at an 8K context length. Larger experiments train Llama 3 8B and Qwen3 4B variants from scratch on approximately 419B tokens each. Across these settings, 16–32-token windows generally outperform larger windows. The paper also studies gating, head allocation, normalization, and positional encoding.

Critical strengths
  1. In the 1B experiments, gated GHA with a 32-token window achieves validation perplexity of 14.95, compared with 15.20 for the gated 1024-token hybrid and 15.36 for gated global attention. Larger-model results show the same ordering: the 32-token configuration achieves reported perplexities of 9.89 on Llama 3 8B and 10.09 on Qwen3 4B, versus 10.06 and 10.24 for the 1024-token configuration. Average downstream scores rise from 58.3 to 59.4 and from 56.8 to 58.0, respectively. This consistency makes the short-window result useful for architectural design.

  2. At a 32-token window, gating both head groups reduces perplexity from 15.25 to 14.95, whereas gating only the local or global group reaches 15.09 or 15.14. At a 1024-token window, gating both groups improves perplexity by only 0.11. The short-window advantage also remains with QK normalization: gated GHA achieves 14.88 versus 15.06 for gated global attention. These results provide meaningful evidence of an interaction between short windows and conditional gating.

Critical weaknesses

Are long-context retrieval or dependency-sensitive results available for the larger models, particularly the Qwen3 variant trained at 32K context?

Can you provide per-task downstream results, a task-balanced aggregate, and uncertainty estimates for the approximately 1.2-point improvements?

Does retaining a small number of sink tokens in short-window heads reproduce the benefit of explicit gating?

What are the measured training throughput, decoding throughput, and KV-memory savings relative to gated global attention under matched hardware and batch settings?

Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.