acceptodds
Under review as a conference paper at ICLR 2027

Mechanics of Long-Context Hybrid Models 1.1: From Hybrid Attention to Hybrid Position

Abstract

The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performance in length extrapolation and context extension. To explain why hybrid models work and how to design them better, we propose **Mechanics of Long-Context Hybrid Models**. As Part 1.1 of this series, we begin with hybrids of full attention and either sliding-window attention (SWA) or gated variants of linear attention (LA), represented by GLA and GDN. We first observe a **Seesaw Effect in Context Extension**: SWA hybrids perform better in length extrapolation, while LA hybrids perform better after context extension. We interpret this from the perspective of hybrid position. We find that SWA hybrids suffer from a **Short-Context Learning Trap**, **Short-Window Weariness**, and **Long-Window Laziness**, requiring extended windows to enhance performance in continual long-context pretraining. In contrast, we summarize the **Matthew Effect of Hybrid Position Extrapolation** and propose **Sliding-Window Linear Attention** for LA hybrids, achieving up to 16 training-free length extrapolation with 100% retrieval accuracy on NIAH-SK1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Long Context, Model Architecture, Hybrid Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pJ1AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

he paper studies hybrid long-context architectures that mix full attention (RoPE or NoPE) with sliding-window attention (SWA, window 128) or gated linear attention (GLA, GDN), mixed either by layer (3:1, LH) or by head (HH).

Critical strengths

S1. Scale and consistency of the empirical study. Fifteen architectures at four sizes, with a common base recipe, evaluated on language modeling, retrieval and state tracking, before and after CPT. The CPT half of the Seesaw is cleanly supported: SWA-NoPE-LH is the weakest NoPE layer-wise hybrid after CPT at every size (Table 11, All avg: 376M 26.3 vs GLA 27.9 / GDN 28.9; 776M 29.4 vs 33.2 / 42.0; 1B 37.6 vs 43.0 / 40.6; 3B 45.9 vs 51.7 / 51.6). The most striking single result is that 5B tokens of 32k CPT leave SWA-NoPE-LH slightly worse than its own training-free log-scaled extrapolation at 776M, 1B and 3B (29.4 vs 29.7, 37.6 vs 38.5, 45.9 vs 47.2). This is a useful, practice-relevant observation that the paper understates.

S2. SWLA is a simple, training-free method with a real effect. It leaves results within 4k unchanged, and its combination with log-scaled NoPE is strongly superadditive (Table 7, 776M GLA-NoPE-LH, All avg: base 19.5, +Log 20.1, +SWLA 23.6, +Both 30.1; GDN: 21.4 / 21.4 / 28.5 / 31.2). On NIAH-SK2 at 64k, 1B GLA-NoPE-LH goes from 0 to 99 (Table 16). The implementation is operator-agnostic and easy to reproduce.

Critical weaknesses

W1. The two headline claims — the Seesaw Effect as an architectural property and the Short-Context Learning Trap as its mechanism — are contradicted or confounded by the paper's own tables. The extrapolation half of the Seesaw dissolves under the paper's own method: with SWLA+Log (Tables 7–8, All avg), GLA/GDN-NoPE-LH match or beat SWA-NoPE-LH+Log at 376M, 776M and 1B (e.g. 776M: 30.1 / 31.2 vs 29.7); only 3B keeps SWA ahead. Section 3.3 thus undercuts Section 2.3. The CPT half is confounded: LongCE and window/base retuning were applied only to SWA hybrids, and Table 18 shows that SWA rotary base 100 alone reverses the 376M ranking after CPT (30.0 vs GLA 27.9 / GDN 28.9). The Trap is called a hypothesis in Sec. 3.1 but stated as fact elsewhere, and the change-ratio evidence (Fig. 5) does not discriminate: RoPE-NoPE-LH shows the same pattern and extends well (best model at 3B, 52.7), SWA-NoPE-HH shows it with almost no seesaw (Table 12), and domain adaptation to the CPT corpus (every model drops 1–2 points on short tasks, Table 3 vs 5) would produce the same signature. No experiment isolates the SWA layers, and no result has seed variance, although the 376M seesaw is 1.6 points and 776M GDN-NoPE-LH after CPT (42.0, above the 1B GDN model) is an unexplained outlier. Fixing this requires new experiments, not rewording.

W2. Headline claims are overstated and the paper is not self-contained. "16× extrapolation with 100% retrieval accuracy on NIAH-SK1" holds at 776M and for GDN at 1B/3B but fails at 64k for 376M GLA-LH (84) and GDN-LH (79), 1B GLA-HH (64), 3B GLA-LH (64) and 3B GLA-HH (5) (Table 16); GDN-NoPE-LH already scores 99–100 on SK1 without SWLA at 776M–3B. The paper ignores its own baselines: after CPT, plain RoPE (776M: 34.3) beats SWA-, GLA- and RoPE-NoPE-LH, and RoPE-NoPE-LH (52.7) beats every hybrid at 3B (Table 11). SWLA's cost (roughly 4×4\times LA prefill compute, no single decoding state) is unreported, and A.4 uses the w=128 cache while the recommended setting is w=2048. Finally, the "Part 1.1 of a series" framing is incompatible with a conference paper, which must stand on its own: the title carries a version-like suffix, the introduction spends a page on the series plan, and the limitations describe future parts. A multi-part program belongs in a monograph or technical-report series; the framing should be removed and the paper retitled for its actual contribution. Reference errors: A.1 "Figures 13 and 12"; App. B "Tables 3, 4, 5 and 4" and "Tables 7, 10, 9 and 10"; Fig. 18(b) legend; Table 8 3B block lacks SWA-RoPE-LH; "¿4k" header artifact; "+ Log (EME)" undefined.

Questions for authors
  1. After 32k CPT, are models evaluated with log-scaled NoPE attention, as in the training-free rows? If not, Table 11 and the +Log rows of Tables 7–8 are not directly comparable.
  2. How do you reconcile the extrapolation half of the Seesaw with Tables 7–8, where SWLA+Log makes GLA/GDN-NoPE-LH match or exceed SWA-NoPE-LH+Log at 376M–1B?
  3. Does LongCE (and rotary-base tuning, where applicable) also help GLA/GDN-NoPE hybrids during CPT? Without this, the "competitive with GDN-NoPE-LH" claim in Sec. 3.2 compares tuned SWA against untuned GDN.
  4. Given Table 18 (base=100 reversing the 376M ranking), why is the rotary base fixed at 10000 in all main-text SWA experiments?
  5. Is every result a single run? Please report seed variance at 376M for at least SWA-NoPE-LH, GLA-NoPE-LH and GDN-NoPE-LH, and the number of examples per RULER task.
  6. Can the Short-Context Learning Trap be tested directly, e.g., CPT with the SWA layers frozen or at a reduced learning rate, or CPT on the original DCLM distribution packed to 32k, to separate the trap from domain shift?
  7. What are the prefill FLOPs, latency and decoding-state requirements of SWLA relative to plain GLA/GDN, and how are RULER/BABILong answers generated under SWLA?
  8. Please state the per-variant parameter counts (GLA/GDN add gates and short convolutions; HH variants compute attention twice). 9.Is this paper intended to stand on its own? "Part 1" is defined only as a scope and no other installment exists; which claims in this submission depend on results deferred to other parts? If the work is conceived as a multi-part program, a monograph or technical-report series is the appropriate venue, and the submission should be the self-contained subset with the series title and numbering removed.
13 comments
AuthorsOpenReview ↗

Reply (1/11)

We thank the reviewer for recognizing the scale of our experiments, the consistency of our findings, and our improvements to length extrapolation in linear-attention hybrid models. We respond to the weaknesses point by point below, addressing the corresponding questions along the way.

Regarding The two headline claims — the Seesaw Effect as an architectural property and the Short-Context Learning Trap as its mechanism — are contradicted or confounded by the paper's own tables. in W1

Our paper investigates length extrapolation and context extension in models that combine full attention with SWA or GLA/GDN, from the perspective of hybrid position. Under the default configurations, SWA hybrid models exhibit stronger length extrapolation, whereas LA hybrid models benefit more from long-context continual pretraining. Then we attribute the weak context extension in SWA hybrid models to excessive optimization of the local branch and improve performance by enlarging its window and using LongCE. For linear-attention hybrid models, we propose Sliding-Window Linear Attention, combined with log-scale sharpening of global NoPE attention, to improve training-free extrapolation.

Our reasoning is clear: we first identify the reversal in the relative advantages of linear-attention and sliding-window hybrid models before and after long-context training, namely the Seesaw Effect, and then propose methods to address the context-extension limitations of SWA hybrid models and the length-extrapolation limitations of LA hybrid models, respectively. The improvements achieved by our proposed methods do not invalidate the original observations or the limitations they identify. We also apply existing extrapolation methods, such as scaled NoPE attention, to SWA/GLA/GDN-NoPE hybrid models, but SWA-NoPE hybrid models retain their extrapolation advantage. We therefore propose Sliding-Window Linear Attention to constrain the OOD implicit positional information in linear attention, substantially improving extrapolation in LA-NoPE hybrid models, as the reviewer also acknowledges. SWLA is a new method that changes the inference computation of LA; its improvements cannot directly serve as evidence that the original LA baseline has no extrapolation limitations.

Regarding Q1 After 32k CPT, are models evaluated with log-scaled NoPE attention, as in the training-free rows? If not, Table 11 and the +Log rows of Tables 7–8 are not directly comparable.

Log-scaled NoPE attention is an existing training-free method for improving the extrapolation of NoPE attention. The original long-context continual pretraining results in Table 11 do not use this additional scaling. Even when log-scaled NoPE attention is applied, SWA-NoPE hybrid models retain their extrapolation advantage over the original GLA/GDN-NoPE hybrid models. We can also compare models without any extrapolation enhancements. The table below reports results without extrapolation enhancements after both short-context training and long-context continual pretraining; the Seesaw Effect remains clearly present at all four model scales. Specifically, SWA-NoPE-LH shows better length extrapolation after short-context training, whereas GLA/GDN-NoPE-LH shows better context extension after long-context continual pretraining.

AuthorsOpenReview ↗

Reply (2/11)

RUBAAvg.
4k8k16k32k64k0k2k4k8k16k32k64kRUBA≤\leq 4k> 4kAll
376M Short
SWA-NoPE-LH29.023.419.311.09.231.027.223.920.717.112.912.618.420.827.815.819.8
GLA-NoPE-LH28.517.02.01.30.639.037.033.825.511.32.62.89.921.734.67.916.8
GDN-NoPE-LH22.920.410.46.03.631.834.030.327.011.81.70.112.719.529.710.116.7
376M Long
SWA-NoPE-LH35.428.026.822.217.135.330.629.228.024.420.717.325.926.532.623.126.3
GLA-NoPE-LH27.526.225.326.919.530.733.231.830.430.029.423.825.129.930.826.427.9
GDN-NoPE-LH28.227.727.727.320.732.631.929.829.131.030.530.026.330.730.628.028.9
776M Short
SWA-NoPE-LH34.929.423.211.79.935.333.630.528.223.420.516.821.826.933.620.424.8
GLA-NoPE-LH35.028.710.50.50.940.436.333.428.714.92.52.615.122.736.311.219.5
GDN-NoPE-LH37.333.415.511.68.838.232.032.122.212.49.53.921.321.534.914.721.4
776M Long
SWA-NoPE-LH40.633.430.221.214.447.133.733.430.827.322.518.627.930.538.724.829.4
GLA-NoPE-LH38.837.832.927.523.738.940.637.733.232.528.425.932.233.939.030.233.2
GDN-NoPE-LH45.141.040.237.431.948.946.447.346.845.745.327.539.144.046.939.542.0
1B Short
SWA-NoPE-LH44.838.129.513.09.953.940.737.031.429.821.917.127.133.144.123.830.6
GLA-NoPE-LH47.040.313.53.21.542.747.245.438.920.39.88.721.130.445.617.026.5
GDN-NoPE-LH44.727.513.811.69.638.636.837.631.729.215.614.721.429.239.419.225.9
1B Long
SWA-NoPE-LH50.747.542.929.615.753.847.442.935.534.127.324.237.337.948.732.137.6
GLA-NoPE-LH55.549.749.844.341.538.844.042.840.640.038.430.848.239.345.341.943.0
GDN-NoPE-LH48.743.743.041.828.640.743.942.139.640.238.836.541.240.343.839.040.6
3B Short
SWA-NoPE-LH51.846.944.635.522.964.356.548.941.736.631.022.440.343.155.435.241.9
GLA-NoPE-LH57.735.916.57.79.861.552.448.435.220.924.214.525.536.755.020.632.1
GDN-NoPE-LH52.039.129.515.914.357.056.153.240.229.421.118.230.139.354.625.935.5
3B Long
SWA-NoPE-LH56.951.448.241.232.560.452.852.447.644.733.928.446.045.755.641.045.9
GLA-NoPE-LH67.263.059.249.342.571.252.749.543.544.040.537.356.248.460.147.451.7
GDN-NoPE-LH57.652.451.244.337.250.156.457.556.758.055.342.848.553.855.449.751.6
AuthorsOpenReview ↗

Reply (3/11)

Regarding The extrapolation half of the Seesaw dissolves under the paper's own method: with SWLA+Log (Tables 7–8, All avg), GLA/GDN-NoPE-LH match or beat SWA-NoPE-LH+Log at 376M, 776M and 1B (e.g. 776M: 30.1 / 31.2 vs 29.7); only 3B keeps SWA ahead. Section 3.3 thus undercuts Section 2.3. in W1 and Q2 How do you reconcile the extrapolation half of the Seesaw with Tables 7–8, where SWLA+Log makes GLA/GDN-NoPE-LH match or exceed SWA-NoPE-LH+Log at 376M–1B?

The Seesaw Effect emphasizes that SWA-NoPE hybrid models are stronger at length extrapolation after short-context pretraining, whereas LA-NoPE hybrid models benefit more from long-context continual pretraining, especially in layer-wise mixing (lines 213–215). Under these settings, the original LA-NoPE hybrid models underperform SWA-NoPE hybrid models with or without log-scaled NoPE attention. The models cited by the reviewer as outperforming SWA-NoPE hybrid models with log-scaled NoPE attention are instead SWLA-NoPE hybrid models, demonstrating the effectiveness of our proposed SWLA method. These results further support the effectiveness of sliding windows for length extrapolation in hybrid models, whether applied to softmax attention or linear attention. This is also the point emphasized by our Matthew Effect of Length Extrapolation in Hybrid Position.

Regarding The CPT half is confounded: LongCE and window/base retuning were applied only to SWA hybrids, in W1 and Q3 Does LongCE (and rotary-base tuning, where applicable) also help GLA/GDN-NoPE hybrids during CPT? Without this, the "competitive with GDN-NoPE-LH" claim in Sec. 3.2 compares tuned SWA against untuned GDN.

First, the GLA/GDN-NoPE models considered here do not use RoPE, so rotary-base tuning does not apply to them. During our exploration, we evaluate LongCE on both SWA-NoPE-LH and GLA/GDN-NoPE-LH at the smallest model scale and observe greater gains for SWA-NoPE-LH. We agree that these comparisons provide a fairer assessment and will include the existing results below in the revised manuscript. Further long-context continual pretraining runs will depend on the availability of GPU resources. We thank the reviewer for this suggestion.

RUBAAvg.
4k8k16k32k64k0k2k4k8k16k32k64kRUBA≤\leq 4k> 4kAll
376M Long
SWA-NoPE-LH35.428.026.822.217.135.330.629.228.024.420.717.325.926.532.623.126.3
+ LongCE41.435.734.334.726.435.334.633.234.231.329.223.534.531.636.131.232.8
GLA-NoPE-LH27.526.225.326.919.530.733.231.830.430.029.423.825.129.930.826.427.9
+ LongCE33.021.329.934.924.528.230.228.429.330.628.425.928.728.730.028.128.7
GDN-NoPE-LH28.227.727.727.320.732.631.929.829.131.030.530.026.330.730.628.028.9
+ LongCE38.319.128.325.721.129.331.931.034.135.036.232.926.532.932.629.130.2

Regarding and Table 18 shows that SWA rotary base 100 alone reverses the 376M ranking after CPT (30.0 vs GLA 27.9 / GDN 28.9). in W1 and Q4 Given Table 18 (base=100 reversing the 376M ranking), why is the rotary base fixed at 10000 in all main-text SWA experiments?

We use a default rotary base of 10000 for SWA in the sliding-window hybrid models. This is the classical RoPE setting and also follows the hyperparameter choices of existing mainstream SWA hybrid models, such as MiMo-V2.5-Pro[1] and Gemma 4[2]. As the reviewer notes, base=100 performs well at 376M, but this advantage does not hold consistently. We therefore summarize results for bases of 10000, 1000, and 100 after short-context training, after short-context training with log-scaled NoPE attention, and after long-context training, and add experiments at the 1B scale we have done before. No base provides a consistent advantage across these settings. As we acknowledge in line 1020 of the paper, “we do not observe a consistent trend across all settings.” We therefore use base=10000 to follow a widely used configuration and make our findings relevant to that setting.

Reviewer_o34VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why hybrids combining global full attention with sliding-window attention (SWA) or gated linear attention (GLA/GDN) behave differently at long contexts. It reports a “seesaw” effect: SWA hybrids generally extrapolate better after short-context pretraining, whereas linear-attention hybrids benefit more from continued training on longer sequences. It attributes weak context extension in SWA hybrids to excessive optimization of the local branch and improves performance by enlarging its window and using LongCE. For linear-attention hybrids, it proposes overlapping-chunk Sliding-Window Linear Attention, combined with log-scale sharpening of global NoPE attention, to improve training-free extrapolation.

Experiments span 376M–3B models, layer-wise and head-wise hybrids, 4k pretraining and 32k continued training, with evaluations up to 64k. Evaluations include PG19 perplexity, RULER and BABILong. The experiments support the performance trends and proposed interventions.

Critical strengths

The paper is well-written and easy to follow. It provides a substantial comparative study of hybrid architectures under two distinct long-context regimes. It evaluates SWA and GLA/GDN hybrids across model scales and layer-wise/head-wise layouts, distinguishing training-free extrapolation from continued long-context pretraining. Evaluations cover language modeling, retrieval, and state tracking, with additional learning-rate and layout comparisons. This provides useful evidence that architectural choices advantageous for extrapolation need not be equally advantageous for context extension.

Critical weaknesses
  • The fact that SWA-NoPE hybrids benefit less than LA-NoPE from LC-CPT is an interesting conclusion from this work. However, the magnitude of the gap and the decrease in retrieval abilities during LC-CPT makes one wonder on whether the LC-CPT itself or its training may be inadapted for this purpose. The paper would benefit from a deeper understanding of this phenomenon, and from an investigation into (a) the dataset document length distribution, (b) the evolution of training loss per-position-within-document throughout training, (c) the evolution of validation loss per position
  • The related-work discussion omits Cabannes et al., Short Window Attention Enables Long-Term Memorization (arXiv:2509.24552). This work already studies the tradeoff between SWA window size, short-context performance, and long-context retrieval in the context of SWA - full-attention layer-wise hybrids. It attributes poorer recall with large windows to over-reliance on local attention during training and proposes stochastic window-size training. It also evaluates Gated DeltaNet and local/global softmax-attention hybrids. Please discuss this work and distinguish the present findings and mechanistic interpretation from these prior results.
Questions for authors
  • Does SWLA introduce prediction discontinuities at stitched chunk boundaries? Please clarify sensitivity to stride and chunk alignment, ideally reporting performance by position relative to those boundaries. • For the 1B SWA-NoPE-LH model, Tables 8 and 11 show that the short-trained checkpoint with log-scale NoPE extrapolation achieves RULER scores of 40.9 at 32k and 35.1 at 64k, whereas after LC-CPT these fall to 29.6 and 15.7. This is surprising, since continued long-context training would normally be expected to improve retrieval. Does this disadvantage persist (a) when the LC-CPT checkpoint also receives appropriately calibrated NoPE attention scaling, and (b) when the fine-tuning "bakes in" the attention scaling?
  • Please report the tokenized document-length distribution of the LC-CPT data and explain how documents are packed or truncated into 32k sequences. What fraction of training tokens occurs beyond 4k and 16k within the same document?
8 comments
AuthorsOpenReview ↗

Reply (1/5)

We greatly appreciate the reviewer's recognition on our substantial comparative study of hybrid architectures in length extrapolation and context extension based on extensive experiments.

Regarding W1 and Q3 on the dataset document length distribution

We examine the length distribution of LongWanjuan, the corpus used for long-context continual pretraining, in terms of counts and proportions of tokens and samples. We report the original document distribution (Raw), the distribution after sampling (Sampled), and the distribution during training (Packed). Raw refers to the unique original documents encountered within the approximately 5B-token training budget, rather than the entire corpus. LongWanjuan is already filtered by length, and the original tokenized documents are all longer than 4k tokens. During training, we concatenate multiple documents to fill a 32k sequence, which forms one sample in a batch; attention is isolated between different documents within the same sample. Documents longer than 32k are split, with the remaining portion buffered to fill the next sample. Thus, although the original documents are all longer than 4096 tokens, some resulting fragments are shorter than 4096 tokens. These fragments account for less than 5% of the tokens (0.21% + 0.64% + 2.54%), with most tokens concentrated in fragments of 4k–32k tokens. Fragments of at least 16k tokens alone account for more than 1/3 of the tokens (22.80% + 12.52%). The starred entries in the [32768, 65536) column of the packed rows represent fragments of exactly 32k tokens.

#data[0, 1024)[1024, 2048)[2048, 4096)[4096, 8192)[8192, 16384)[16384, 32768)[32768, 65536)[65536, 131072)[131072, +\inf)
Raw00066528200776567841186543521705
0.00%0.00%0.00%19.45%58.70%16.60%3.47%1.27%0.50%
Sampled00067612203433571541190043551705
0.00%0.00%0.00%19.53%58.77%16.51%3.44%1.26%0.49%
Packed2088921827433021355372086205523020039*00
4.13%4.32%8.57%26.82%41.27%10.93%3.96%*0.00%0.00%

Token counts use M = 1024 × 1024 = 1,048,576.

#token (M)[0, 1024)[1024, 2048)[2048, 4096)[4096, 8192)[8192, 16384)[16384, 32768)[32768, 65536)[65536, 131072)[131072, +\inf)
Raw000473.662138.541160.11498.44379.76307.5
0.00%0.00%0.00%9.55%43.13%23.40%10.05%7.66%6.20%
Sampled000481.162166.141167.34499.78380.01307.5
0.00%0.00%0.00%9.62%43.31%23.34%9.99%7.60%6.15%
Packed10.5531.93126.89855.772208.671139.97626.22*00
0.21%0.64%2.54%17.12%44.17%22.80%12.52%*0.00%0.00%

Regarding the reviewer's question about the fractions of tokens beyond positions 4k and 16k, we also report these statistics for the sampled data and the data as presented during training. Under both definitions, the fraction of tokens beyond 4k is well above 60% (63.84%). This confirms that long-context training indeed exposes the models to longer contexts. Nevertheless, SWA-NoPE-LH still exhibits its largest loss reductions at short-context positions, indicating its preference for learning at these positions. We will include the tables below in the experimental settings in the appendix to clarify our training procedure.

Fraction of tokens>=4k>=16k
beyond positions 4k/16k in the original document72.96%23.59%
beyond positions 4k/16k within the same document fragment during training63.84%11.80%
AuthorsOpenReview ↗

Reply (2/5)

Regarding W1 on the evolution of loss per position

Regarding your request for token-level loss grouped by position ranges, we add results showing the loss at different position ranges on the PG19 validation set for all saved checkpoints of the 376M and 776M layer-wise hybrid models during long-context continual pretraining (LC-CPT; after training on 1B, 2B, 3B, 4B, and 5B tokens). Since training losses for individual position ranges are not recorded during training, we can only report validation losses. For SWA-NoPE-LH, the loss reduction from before to after LC-CPT is more pronounced at short-context positions than at long-context positions, where the loss exhibits some fluctuations. For LA-NoPE-LH, the loss reduction at long-context positions is substantial. This remains consistent with Figure 4 in the paper: both cumulative perplexity and loss grouped by position ranges show that, under the same LC-CPT settings, SWA-NoPE-LH does not effectively learn long-context dependencies compared with LA-NoPE-LH, instead focusing more on learning at short-context positions. We therefore introduce larger sliding-window sizes and the LongCE loss to intervene in and improve this behavior. We will add visualizations of the results below to the revised manuscript and refine our discussion of the Short-Context Learning Trap. We thank the reviewer for this suggestion.

[0, 128)[128, 256)[256, 512)[512, 1024)[1024, 2048)[2048, 4096)[4096, 8192)[8192, 16384)[16384, 32768)
376M
SWA-NoPE-LH3.86243.42603.21283.22423.20663.21593.23343.23903.2575
+ CPT 1B3.72883.48913.33363.34183.34103.33153.33783.33573.3407
+ CPT 2B3.56693.42253.28293.30063.30493.29903.31003.30923.3180
+ CPT 3B3.45823.34373.22613.25083.26483.26213.27163.27143.2825
+ CPT 4B3.37213.30433.18713.21553.23033.22883.23683.23593.2470
+ CPT 5B3.34603.29053.17543.20373.21943.21813.22633.22663.2381
GLA-NoPE-LH3.80293.40183.20593.20463.16953.15353.15693.31815.8051
+ CPT 1B3.69963.47453.33873.32843.31233.28683.27583.25323.2433
+ CPT 2B3.51263.38033.26283.26813.26793.24803.23703.21393.2001
+ CPT 3B3.44443.31653.21873.22483.23133.21223.20163.18033.1670
+ CPT 4B3.36193.26953.17203.18673.19663.17933.16763.14473.1308
+ CPT 5B3.33693.25393.15923.17583.18673.16893.15733.13423.1202
GDN-NoPE-LH3.77373.37773.17733.17203.13843.11733.12813.24913.7858
+ CPT 1B3.56553.37003.24883.24813.24313.21683.21613.19013.1810
+ CPT 2B3.45123.32763.21843.23103.22943.20963.20773.18323.1731
+ CPT 3B3.37773.26853.17863.18983.19623.17413.17163.14633.1345
+ CPT 4B3.29523.23013.14163.15673.16643.14323.14033.11513.1034
+ CPT 5B3.27113.21613.13003.14553.15623.13373.13093.10563.0945
776M
SWA-NoPE-LH3.56753.15292.96022.95412.93462.93922.96122.95952.9788
+ CPT 1B3.35283.17103.05463.06003.05763.05413.06453.05793.0665
+ CPT 2B3.20103.11043.02183.03593.04553.03883.05473.05233.0618
+ CPT 3B3.11393.03872.96742.98593.00183.00113.01643.01323.0258
+ CPT 4B3.00372.98942.92822.95002.96922.96602.97992.97632.9880
+ CPT 5B2.99132.97582.91532.93792.95792.95532.96982.96632.9794
GLA-NoPE-LH3.45593.11962.94202.93752.90342.89282.90153.11114.3761
+ CPT 1B3.22353.14473.02693.03193.02413.01143.00522.97822.9687
+ CPT 2B3.15813.07553.00173.00923.00542.99122.98702.95992.9495
+ CPT 3B3.09203.00342.94972.96362.97142.95692.95282.92682.9183
+ CPT 4B2.96942.95752.91152.92722.93912.92392.91852.89262.8832
+ CPT 5B2.94702.94482.89822.91542.92802.91432.90882.88272.8734
GDN-NoPE-LH3.51583.11212.92632.90762.88112.86592.87493.07383.6088
+ CPT 1B3.16553.08433.00012.99842.99692.97882.97122.94572.9373
+ CPT 2B3.09463.03502.96262.96662.97522.95462.95392.92762.9178
+ CPT 3B2.98032.97462.92152.92482.94202.92252.92012.89422.8840
+ CPT 4B2.89202.92692.88182.88872.91132.89172.88772.86272.8517
+ CPT 5B2.87222.91322.86882.87842.90092.88182.87812.85282.8420
AuthorsOpenReview ↗

Reply (3/5)

Regarding W2 on Cabannes et al., Short Window Attention Enables Long-Term Memorization (arXiv:2509.24552)

This work compares hybrid models combining sliding-window attention with either full attention or linear attention, using window sizes of 128 and 2048 and training at a sequence length of 16k. It demonstrates that shorter windows provide stronger retrieval and length extrapolation capabilities. However, this corresponds to only one half of our findings, namely, “large-window laziness”. Similar findings also appear in earlier work, as discussed in our paper. We further report the other half, namely, “small-window fatigue”: we investigate how the long-context performance of hybrid models with different window sizes changes from 4k pretraining to 32k long-context continual pretraining, and how to improve the benefits of long-context continual pretraining for models with short windows.

Our training settings also differ. Their models are trained at a sequence length of 16k throughout, with stochastic switching between short and long windows during training to balance length extrapolation and short-context performance. We instead perform 4k pretraining followed by 32k long-context continual pretraining, focusing on the transition between these stages and the limitations of fixed window sizes during context extension. We also compare window sizes ranging from 128 to 4096, providing further evidence for our findings. We will cite and discuss this work in the paper. We thank the reviewer for this suggestion.

Regarding Q1 on the sensitivity of SWLA to stride and chunk alignment

In the paper, we set the window size to the short-context pretraining length, 4096, preserving the performance of linear-attention hybrid models within the training length. The stride is one-quarter of the window size, i.e., 1024. This stride-to-window ratio matches that used for short-window perplexity computation in the LongCE/LongPPL papers. We conduct ablations on the 376M and 776M GLA/GDN-NoPE-LH models with SWLA, comparing strides of 256, 512, and 2048 with the window size fixed at 4096, as well as a window size of 8192 with the stride fixed at 1024. Given our current resource constraints, we only compare RULER and BABILong scores from 8k to 32k. Therefore, the overall averages marked with an asterisk cover only these lengths, rather than the full evaluation range; the same convention applies below.

We find that no hyperparameter setting is consistently optimal across models, but SWLA improves average performance over the original models under all tested settings. Additionally, unlike RoPE, the positional information in linear attention does not immediately become out of distribution beyond the training length. Thus, downstream performance does not collapse even with a window size of 8192, which exceeds the training length. We will complete this table, add results for head-wise hybrid models, and include it in the appendix. We will also supplement the results with case analyses at chunk boundaries. We thank the reviewer for this suggestion.

Reviewer_s5jnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates length extrapolation and context extension in models that combine full attention with SWA or GLA/GDN, from the perspective of hybrid position. Under the default configurations, SWA hybrids exhibit stronger length extrapolation, whereas LA hybrids benefit more from long-context continual pretraining. This paper propose enlarging the SWA window, adopting LongCE, and applying Sliding-Window Linear Attention (SWLA), achieving 16× training-free length extrapolation with 100% retrieval accuracy on NIAH-SK1 in selected configurations.

Critical strengths
  1. The paper addresses an important research question and presents an appealing perspective for understanding long-context hybrid models.
  2. The experiments cover multiple model scales, SWA and two gated LA variants, RoPE/NoPE configurations, and both layer-wise and head-wise hybrids.
  3. Enlarging the SWA window, adopting LongCE, and restricting the historical range of the LA branch provide concrete and actionable approaches to improving context extension and length extrapolation.
Critical weaknesses
  1. PPL measured at different context lengths does not directly establish how learning gains are distributed across token positions within a long sequence. Likewise, larger relative parameter changes in the SWA branch do not necessarily imply excessive learning of local dependencies. The current evidence is therefore insufficient to establish the proposed “Short-Context Learning Trap.”
  2. The abstract emphasizes “achieving up to 16× training-free length extrapolation with 100% retrieval accuracy on NIAH-SK1.” However, in Table 1, the GDN hybrids already achieve 100% accuracy on SK1 with log-scaled NoPE attention alone. For these configurations, the additional benefits of SWLA are primarily evident on SK2/SK3. Moreover, with SWLA and log-scaled NoPE attention, accuracy at 64k ranges from only 16% to 64% on SK3 and from 31% to 90% on SK2. Highlighting SK1 as the main result may therefore overstate the broader effectiveness of the method.
  3. The window-size experiments are confounded by simultaneous changes to the RoPE base. When enlarging the SWA window, the authors also adjust the rotary base; for example, increasing the window size to 2048 raises the base to 50,000,000. Consequently, the performance gains cannot be attributed to window enlargement alone.
  4. Minor presentation issue. Line 1235 refers to “Tables 3, 4, 5 and 4,” which appears to be a typo error.
Questions for authors
  1. Could the authors report changes in token-level loss before and after long-context continual pretraining (CPT), grouped by position ranges such as 0–128, 128–4k, and 4k–32k? This would help assess whether the gains are concentrated at early positions, although such evidence alone would not establish the causal mechanism behind the Short-Context Learning Trap.
  2. Could the authors provide length extrapolation and CPT results for NoPE-only full-attention, SWA-only, and GLA/GDN-only models under matched training budgets? These baselines would help distinguish the contributions of hybridization, removing RoPE, and introducing LA.
  3. Could the authors separately evaluate enlarging the SWA window while keeping the rotary base fixed at 10,000, and adjusting the rotary base while keeping the window size fixed? These controlled experiments would help disentangle the contributions of window size and rotary base.
  4. Are the Seesaw Effect, the performance trends across window sizes, and the gains from LongCE consistent across multiple random seeds? Could the authors report means and standard deviations for the key configurations to assess their robustness to training randomness?
6 comments
AuthorsOpenReview ↗

Reply (1/5)

We greatly appreciate the reviewer's recognition of our efforts to understand the mechanisms of long-context hybrid models, conduct extensive empirical validation, and provide concrete approaches to improving long-context performance.

Regarding W1 and Q1 on Short-Context Learning Trap

We would first like to clarify our reasoning. After finding that the long-context advantage of SWA-NoPE-LH over LA-NoPE-LH is reversed during context extension, we further observe that, during long-context continual pretraining, the reduction in training loss for SWA-NoPE-LH is mainly concentrated at short-context positions, with only limited reductions at long-context positions. Examining the metrics before and after training allows us to move from our initial observation of downstream task performance, namely the Seesaw Effect, toward methods for improving SWA-LH training. Our observations of parameter changes further guide the specific modification: adjusting the sliding-window attention module within the SWA hybrid models.

Regarding your request for token-level loss grouped by position ranges, we add results showing the loss at different position ranges on the PG19 validation set for all saved checkpoints of the 376M and 776M layer-wise hybrid models during long-context continual pretraining (LC-CPT; after training on 1B, 2B, 3B, 4B, and 5B tokens). Since training losses for individual position ranges are not recorded during training, we can only report validation losses. For SWA-NoPE-LH, the loss reduction from before to after LC-CPT is more pronounced at short-context positions than at long-context positions, where the loss exhibits some fluctuations. For LA-NoPE-LH, the loss reduction at long-context positions is substantial. This remains consistent with Figure 4 in the paper: both cumulative perplexity and loss grouped by position ranges show that, under the same LC-CPT settings, SWA-NoPE-LH does not effectively learn long-context dependencies compared with LA-NoPE-LH, instead focusing more on learning at short-context positions. We therefore introduce larger sliding-window sizes and the LongCE loss to intervene in and improve this behavior. We will add visualizations of the results below to the revised manuscript and refine our discussion of the Short-Context Learning Trap. We thank the reviewer for this suggestion.

AuthorsOpenReview ↗

Reply (2/5)

[0, 128)[128, 256)[256, 512)[512, 1024)[1024, 2048)[2048, 4096)[4096, 8192)[8192, 16384)[16384, 32768)
376M
SWA-NoPE-LH3.86243.42603.21283.22423.20663.21593.23343.23903.2575
+ CPT 1B3.72883.48913.33363.34183.34103.33153.33783.33573.3407
+ CPT 2B3.56693.42253.28293.30063.30493.29903.31003.30923.3180
+ CPT 3B3.45823.34373.22613.25083.26483.26213.27163.27143.2825
+ CPT 4B3.37213.30433.18713.21553.23033.22883.23683.23593.2470
+ CPT 5B3.34603.29053.17543.20373.21943.21813.22633.22663.2381
GLA-NoPE-LH3.80293.40183.20593.20463.16953.15353.15693.31815.8051
+ CPT 1B3.69963.47453.33873.32843.31233.28683.27583.25323.2433
+ CPT 2B3.51263.38033.26283.26813.26793.24803.23703.21393.2001
+ CPT 3B3.44443.31653.21873.22483.23133.21223.20163.18033.1670
+ CPT 4B3.36193.26953.17203.18673.19663.17933.16763.14473.1308
+ CPT 5B3.33693.25393.15923.17583.18673.16893.15733.13423.1202
GDN-NoPE-LH3.77373.37773.17733.17203.13843.11733.12813.24913.7858
+ CPT 1B3.56553.37003.24883.24813.24313.21683.21613.19013.1810
+ CPT 2B3.45123.32763.21843.23103.22943.20963.20773.18323.1731
+ CPT 3B3.37773.26853.17863.18983.19623.17413.17163.14633.1345
+ CPT 4B3.29523.23013.14163.15673.16643.14323.14033.11513.1034
+ CPT 5B3.27113.21613.13003.14553.15623.13373.13093.10563.0945
776M
SWA-NoPE-LH3.56753.15292.96022.95412.93462.93922.96122.95952.9788
+ CPT 1B3.35283.17103.05463.06003.05763.05413.06453.05793.0665
+ CPT 2B3.20103.11043.02183.03593.04553.03883.05473.05233.0618
+ CPT 3B3.11393.03872.96742.98593.00183.00113.01643.01323.0258
+ CPT 4B3.00372.98942.92822.95002.96922.96602.97992.97632.9880
+ CPT 5B2.99132.97582.91532.93792.95792.95532.96982.96632.9794
GLA-NoPE-LH3.45593.11962.94202.93752.90342.89282.90153.11114.3761
+ CPT 1B3.22353.14473.02693.03193.02413.01143.00522.97822.9687
+ CPT 2B3.15813.07553.00173.00923.00542.99122.98702.95992.9495
+ CPT 3B3.09203.00342.94972.96362.97142.95692.95282.92682.9183
+ CPT 4B2.96942.95752.91152.92722.93912.92392.91852.89262.8832
+ CPT 5B2.94702.94482.89822.91542.92802.91432.90882.88272.8734
GDN-NoPE-LH3.51583.11212.92632.90762.88112.86592.87493.07383.6088
+ CPT 1B3.16553.08433.00012.99842.99692.97882.97122.94572.9373
+ CPT 2B3.09463.03502.96262.96662.97522.95462.95392.92762.9178
+ CPT 3B2.98032.97462.92152.92482.94202.92252.92012.89422.8840
+ CPT 4B2.89202.92692.88182.88872.91132.89172.88772.86272.8517
+ CPT 5B2.87222.91322.86882.87842.90092.88182.87812.85282.8420

Regarding W2 on SWLA

As the reviewer correctly points out, some LA-NoPE hybrid models already perform sufficiently well on NIAH-SK1, and the additional value of SWLA lies in more challenging tasks such as SK2 and SK3. We highlight NIAH-SK1 because many existing methods also report performance on SK1 through SK3 [1–3], but we acknowledge that our presentation is not clear enough. Our intended point is that, across SK1 through SK3, SWLA and SWLA combined with log-scaled NoPE attention outperform direct extrapolation of LA and its combination with log-scaled NoPE attention, respectively. The complete comparisons on RULER, BABILong, and PPL in Figures 17 and 18 also demonstrate this point. In the revised manuscript, we will replace the relevant statements with comparisons based on the average SK score or the overall average scores on RULER and BABILong. The advantages of SWLA combined with log-scaled NoPE attention are clear and consistent across these aggregate scores. We thank the reviewer for this comment.

AuthorsOpenReview ↗

Reply (3/5)

SKRUAvg.
4k8k16k32k64k4k8k16k32k64kSKRURU+BA
376M GLA-NoPE-LH79.6741.000.000.000.0028.5516.951.971.310.5924.139.8716.78
+ Log79.6748.331.000.000.0028.5519.032.561.330.7425.8010.4416.54
+ SWLA79.6766.0051.0033.3315.3328.5323.1016.8810.224.5149.0716.6521.86
+ SWLA + Log79.6767.6776.6782.0063.0028.5523.6826.5025.9819.9073.8024.9227.93
376M GLA-NoPE-HH98.6791.0025.000.000.0031.1626.516.321.331.2742.9313.3216.52
+ Log98.6794.6730.670.330.0031.1627.447.950.970.8544.8713.6717.92
+ SWLA98.6789.0061.0040.3333.0031.1627.8718.9912.038.7864.4019.7719.25
+ SWLA + Log98.6793.3395.6789.3362.0031.1628.5827.7024.5516.6787.8025.7323.31
376M GDN-NoPE-LH73.0069.0034.0022.3314.3322.8720.4210.405.983.6242.5312.6616.67
+ Log73.0070.3345.0030.3328.0022.8721.1914.607.936.8549.3314.6918.22
+ SWLA73.0073.0065.6755.6725.0022.8721.5318.0515.517.6258.4717.1223.11
+ SWLA + Log73.0073.0076.3371.0044.6722.8721.8822.6920.7813.5767.6020.3625.92
376M GDN-NoPE-HH99.3396.3356.6735.6727.0033.4530.6018.4311.666.9963.0020.2324.22
+ Log99.3399.0072.3347.3334.3333.4531.9024.7216.498.8970.4723.0926.99
+ SWLA99.3394.3360.0039.3328.0033.4530.0419.2210.737.3764.2020.1623.39
+ SWLA + Log99.3398.3394.6789.0084.6733.4531.5030.6027.8225.2193.2029.7229.34
776M GLA-NoPE-LH96.0084.0026.670.000.0035.0428.7410.510.480.8541.3315.1319.54
+ Log96.0086.0033.000.000.0035.0429.2111.480.670.6443.0015.4120.07
+ SWLA96.0076.6742.0035.0033.3335.1526.9115.1811.2110.9156.6019.8723.56
+ SWLA + Log96.0090.3387.3381.3374.3335.0932.1330.5426.0522.3385.8729.2330.08
776M GLA-NoPE-HH88.3372.0022.6727.330.0035.5827.197.908.231.8542.0716.1519.15
+ Log88.3372.3332.6731.6724.0035.7127.649.928.807.0849.8017.8319.82
+ SWLA88.3387.3375.3344.3325.6735.6330.0127.1415.3310.5264.2023.7327.43
+ SWLA + Log88.3389.3387.3376.0055.0035.6132.3232.0325.2317.8179.2028.6030.62
776M GDN-NoPE-LH87.0089.6735.6733.3333.0037.3333.4415.5111.648.8355.7321.3521.42
+ Log87.0089.3339.3333.6733.3337.3233.7016.7411.058.6056.5321.4821.37
+ SWLA87.0087.0085.6782.6761.0037.3334.3634.2830.3521.2280.6731.5128.49
+ SWLA + Log87.0088.0087.6788.3381.0037.2434.6634.6332.9828.3186.4033.5731.17
776M GDN-NoPE-HH98.0077.6734.0033.0030.6737.7624.7212.449.798.5654.6718.6621.55
+ Log98.0078.6734.3333.3333.3337.7625.8812.859.699.6155.5319.1621.82
+ SWLA98.0088.6765.0034.6731.3337.7432.3024.4811.479.4063.5323.0828.22
+ SWLA + Log98.0090.3387.3373.0050.3337.7933.1731.5823.4216.1679.8028.4230.56

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.