- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
he paper studies hybrid long-context architectures that mix full attention (RoPE or NoPE) with sliding-window attention (SWA, window 128) or gated linear attention (GLA, GDN), mixed either by layer (3:1, LH) or by head (HH).
Critical strengths
S1. Scale and consistency of the empirical study. Fifteen architectures at four sizes, with a common base recipe, evaluated on language modeling, retrieval and state tracking, before and after CPT. The CPT half of the Seesaw is cleanly supported: SWA-NoPE-LH is the weakest NoPE layer-wise hybrid after CPT at every size (Table 11, All avg: 376M 26.3 vs GLA 27.9 / GDN 28.9; 776M 29.4 vs 33.2 / 42.0; 1B 37.6 vs 43.0 / 40.6; 3B 45.9 vs 51.7 / 51.6). The most striking single result is that 5B tokens of 32k CPT leave SWA-NoPE-LH slightly worse than its own training-free log-scaled extrapolation at 776M, 1B and 3B (29.4 vs 29.7, 37.6 vs 38.5, 45.9 vs 47.2). This is a useful, practice-relevant observation that the paper understates.
S2. SWLA is a simple, training-free method with a real effect. It leaves results within 4k unchanged, and its combination with log-scaled NoPE is strongly superadditive (Table 7, 776M GLA-NoPE-LH, All avg: base 19.5, +Log 20.1, +SWLA 23.6, +Both 30.1; GDN: 21.4 / 21.4 / 28.5 / 31.2). On NIAH-SK2 at 64k, 1B GLA-NoPE-LH goes from 0 to 99 (Table 16). The implementation is operator-agnostic and easy to reproduce.
Critical weaknesses
W1. The two headline claims — the Seesaw Effect as an architectural property and the Short-Context Learning Trap as its mechanism — are contradicted or confounded by the paper's own tables. The extrapolation half of the Seesaw dissolves under the paper's own method: with SWLA+Log (Tables 7–8, All avg), GLA/GDN-NoPE-LH match or beat SWA-NoPE-LH+Log at 376M, 776M and 1B (e.g. 776M: 30.1 / 31.2 vs 29.7); only 3B keeps SWA ahead. Section 3.3 thus undercuts Section 2.3. The CPT half is confounded: LongCE and window/base retuning were applied only to SWA hybrids, and Table 18 shows that SWA rotary base 100 alone reverses the 376M ranking after CPT (30.0 vs GLA 27.9 / GDN 28.9). The Trap is called a hypothesis in Sec. 3.1 but stated as fact elsewhere, and the change-ratio evidence (Fig. 5) does not discriminate: RoPE-NoPE-LH shows the same pattern and extends well (best model at 3B, 52.7), SWA-NoPE-HH shows it with almost no seesaw (Table 12), and domain adaptation to the CPT corpus (every model drops 1–2 points on short tasks, Table 3 vs 5) would produce the same signature. No experiment isolates the SWA layers, and no result has seed variance, although the 376M seesaw is 1.6 points and 776M GDN-NoPE-LH after CPT (42.0, above the 1B GDN model) is an unexplained outlier. Fixing this requires new experiments, not rewording.
W2. Headline claims are overstated and the paper is not self-contained. "16× extrapolation with 100% retrieval accuracy on NIAH-SK1" holds at 776M and for GDN at 1B/3B but fails at 64k for 376M GLA-LH (84) and GDN-LH (79), 1B GLA-HH (64), 3B GLA-LH (64) and 3B GLA-HH (5) (Table 16); GDN-NoPE-LH already scores 99–100 on SK1 without SWLA at 776M–3B. The paper ignores its own baselines: after CPT, plain RoPE (776M: 34.3) beats SWA-, GLA- and RoPE-NoPE-LH, and RoPE-NoPE-LH (52.7) beats every hybrid at 3B (Table 11). SWLA's cost (roughly LA prefill compute, no single decoding state) is unreported, and A.4 uses the w=128 cache while the recommended setting is w=2048. Finally, the "Part 1.1 of a series" framing is incompatible with a conference paper, which must stand on its own: the title carries a version-like suffix, the introduction spends a page on the series plan, and the limitations describe future parts. A multi-part program belongs in a monograph or technical-report series; the framing should be removed and the paper retitled for its actual contribution. Reference errors: A.1 "Figures 13 and 12"; App. B "Tables 3, 4, 5 and 4" and "Tables 7, 10, 9 and 10"; Fig. 18(b) legend; Table 8 3B block lacks SWA-RoPE-LH; "¿4k" header artifact; "+ Log (EME)" undefined.
Questions for authors
- After 32k CPT, are models evaluated with log-scaled NoPE attention, as in the training-free rows? If not, Table 11 and the +Log rows of Tables 7–8 are not directly comparable.
- How do you reconcile the extrapolation half of the Seesaw with Tables 7–8, where SWLA+Log makes GLA/GDN-NoPE-LH match or exceed SWA-NoPE-LH+Log at 376M–1B?
- Does LongCE (and rotary-base tuning, where applicable) also help GLA/GDN-NoPE hybrids during CPT? Without this, the "competitive with GDN-NoPE-LH" claim in Sec. 3.2 compares tuned SWA against untuned GDN.
- Given Table 18 (base=100 reversing the 376M ranking), why is the rotary base fixed at 10000 in all main-text SWA experiments?
- Is every result a single run? Please report seed variance at 376M for at least SWA-NoPE-LH, GLA-NoPE-LH and GDN-NoPE-LH, and the number of examples per RULER task.
- Can the Short-Context Learning Trap be tested directly, e.g., CPT with the SWA layers frozen or at a reduced learning rate, or CPT on the original DCLM distribution packed to 32k, to separate the trap from domain shift?
- What are the prefill FLOPs, latency and decoding-state requirements of SWLA relative to plain GLA/GDN, and how are RULER/BABILong answers generated under SWLA?
- Please state the per-variant parameter counts (GLA/GDN add gates and short convolutions; HH variants compute attention twice). 9.Is this paper intended to stand on its own? "Part 1" is defined only as a scope and no other installment exists; which claims in this submission depend on results deferred to other parts? If the work is conceived as a multi-part program, a monograph or technical-report series is the appropriate venue, and the submission should be the self-contained subset with the series title and numbering removed.
13 comments
Reply (2/11)
| RU | BA | Avg. | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 4k | 8k | 16k | 32k | 64k | 0k | 2k | 4k | 8k | 16k | 32k | 64k | RU | BA | 4k | > 4k | All | |
| 376M Short | |||||||||||||||||
| SWA-NoPE-LH | 29.0 | 23.4 | 19.3 | 11.0 | 9.2 | 31.0 | 27.2 | 23.9 | 20.7 | 17.1 | 12.9 | 12.6 | 18.4 | 20.8 | 27.8 | 15.8 | 19.8 |
| GLA-NoPE-LH | 28.5 | 17.0 | 2.0 | 1.3 | 0.6 | 39.0 | 37.0 | 33.8 | 25.5 | 11.3 | 2.6 | 2.8 | 9.9 | 21.7 | 34.6 | 7.9 | 16.8 |
| GDN-NoPE-LH | 22.9 | 20.4 | 10.4 | 6.0 | 3.6 | 31.8 | 34.0 | 30.3 | 27.0 | 11.8 | 1.7 | 0.1 | 12.7 | 19.5 | 29.7 | 10.1 | 16.7 |
| 376M Long | |||||||||||||||||
| SWA-NoPE-LH | 35.4 | 28.0 | 26.8 | 22.2 | 17.1 | 35.3 | 30.6 | 29.2 | 28.0 | 24.4 | 20.7 | 17.3 | 25.9 | 26.5 | 32.6 | 23.1 | 26.3 |
| GLA-NoPE-LH | 27.5 | 26.2 | 25.3 | 26.9 | 19.5 | 30.7 | 33.2 | 31.8 | 30.4 | 30.0 | 29.4 | 23.8 | 25.1 | 29.9 | 30.8 | 26.4 | 27.9 |
| GDN-NoPE-LH | 28.2 | 27.7 | 27.7 | 27.3 | 20.7 | 32.6 | 31.9 | 29.8 | 29.1 | 31.0 | 30.5 | 30.0 | 26.3 | 30.7 | 30.6 | 28.0 | 28.9 |
| 776M Short | |||||||||||||||||
| SWA-NoPE-LH | 34.9 | 29.4 | 23.2 | 11.7 | 9.9 | 35.3 | 33.6 | 30.5 | 28.2 | 23.4 | 20.5 | 16.8 | 21.8 | 26.9 | 33.6 | 20.4 | 24.8 |
| GLA-NoPE-LH | 35.0 | 28.7 | 10.5 | 0.5 | 0.9 | 40.4 | 36.3 | 33.4 | 28.7 | 14.9 | 2.5 | 2.6 | 15.1 | 22.7 | 36.3 | 11.2 | 19.5 |
| GDN-NoPE-LH | 37.3 | 33.4 | 15.5 | 11.6 | 8.8 | 38.2 | 32.0 | 32.1 | 22.2 | 12.4 | 9.5 | 3.9 | 21.3 | 21.5 | 34.9 | 14.7 | 21.4 |
| 776M Long | |||||||||||||||||
| SWA-NoPE-LH | 40.6 | 33.4 | 30.2 | 21.2 | 14.4 | 47.1 | 33.7 | 33.4 | 30.8 | 27.3 | 22.5 | 18.6 | 27.9 | 30.5 | 38.7 | 24.8 | 29.4 |
| GLA-NoPE-LH | 38.8 | 37.8 | 32.9 | 27.5 | 23.7 | 38.9 | 40.6 | 37.7 | 33.2 | 32.5 | 28.4 | 25.9 | 32.2 | 33.9 | 39.0 | 30.2 | 33.2 |
| GDN-NoPE-LH | 45.1 | 41.0 | 40.2 | 37.4 | 31.9 | 48.9 | 46.4 | 47.3 | 46.8 | 45.7 | 45.3 | 27.5 | 39.1 | 44.0 | 46.9 | 39.5 | 42.0 |
| 1B Short | |||||||||||||||||
| SWA-NoPE-LH | 44.8 | 38.1 | 29.5 | 13.0 | 9.9 | 53.9 | 40.7 | 37.0 | 31.4 | 29.8 | 21.9 | 17.1 | 27.1 | 33.1 | 44.1 | 23.8 | 30.6 |
| GLA-NoPE-LH | 47.0 | 40.3 | 13.5 | 3.2 | 1.5 | 42.7 | 47.2 | 45.4 | 38.9 | 20.3 | 9.8 | 8.7 | 21.1 | 30.4 | 45.6 | 17.0 | 26.5 |
| GDN-NoPE-LH | 44.7 | 27.5 | 13.8 | 11.6 | 9.6 | 38.6 | 36.8 | 37.6 | 31.7 | 29.2 | 15.6 | 14.7 | 21.4 | 29.2 | 39.4 | 19.2 | 25.9 |
| 1B Long | |||||||||||||||||
| SWA-NoPE-LH | 50.7 | 47.5 | 42.9 | 29.6 | 15.7 | 53.8 | 47.4 | 42.9 | 35.5 | 34.1 | 27.3 | 24.2 | 37.3 | 37.9 | 48.7 | 32.1 | 37.6 |
| GLA-NoPE-LH | 55.5 | 49.7 | 49.8 | 44.3 | 41.5 | 38.8 | 44.0 | 42.8 | 40.6 | 40.0 | 38.4 | 30.8 | 48.2 | 39.3 | 45.3 | 41.9 | 43.0 |
| GDN-NoPE-LH | 48.7 | 43.7 | 43.0 | 41.8 | 28.6 | 40.7 | 43.9 | 42.1 | 39.6 | 40.2 | 38.8 | 36.5 | 41.2 | 40.3 | 43.8 | 39.0 | 40.6 |
| 3B Short | |||||||||||||||||
| SWA-NoPE-LH | 51.8 | 46.9 | 44.6 | 35.5 | 22.9 | 64.3 | 56.5 | 48.9 | 41.7 | 36.6 | 31.0 | 22.4 | 40.3 | 43.1 | 55.4 | 35.2 | 41.9 |
| GLA-NoPE-LH | 57.7 | 35.9 | 16.5 | 7.7 | 9.8 | 61.5 | 52.4 | 48.4 | 35.2 | 20.9 | 24.2 | 14.5 | 25.5 | 36.7 | 55.0 | 20.6 | 32.1 |
| GDN-NoPE-LH | 52.0 | 39.1 | 29.5 | 15.9 | 14.3 | 57.0 | 56.1 | 53.2 | 40.2 | 29.4 | 21.1 | 18.2 | 30.1 | 39.3 | 54.6 | 25.9 | 35.5 |
| 3B Long | |||||||||||||||||
| SWA-NoPE-LH | 56.9 | 51.4 | 48.2 | 41.2 | 32.5 | 60.4 | 52.8 | 52.4 | 47.6 | 44.7 | 33.9 | 28.4 | 46.0 | 45.7 | 55.6 | 41.0 | 45.9 |
| GLA-NoPE-LH | 67.2 | 63.0 | 59.2 | 49.3 | 42.5 | 71.2 | 52.7 | 49.5 | 43.5 | 44.0 | 40.5 | 37.3 | 56.2 | 48.4 | 60.1 | 47.4 | 51.7 |
| GDN-NoPE-LH | 57.6 | 52.4 | 51.2 | 44.3 | 37.2 | 50.1 | 56.4 | 57.5 | 56.7 | 58.0 | 55.3 | 42.8 | 48.5 | 53.8 | 55.4 | 49.7 | 51.6 |
Reply (3/11)
Regarding
The extrapolation half of the Seesaw dissolves under the paper's own method: with SWLA+Log (Tables 7–8, All avg), GLA/GDN-NoPE-LH match or beat SWA-NoPE-LH+Log at 376M, 776M and 1B (e.g. 776M: 30.1 / 31.2 vs 29.7); only 3B keeps SWA ahead. Section 3.3 thus undercuts Section 2.3.in W1 and Q2How do you reconcile the extrapolation half of the Seesaw with Tables 7–8, where SWLA+Log makes GLA/GDN-NoPE-LH match or exceed SWA-NoPE-LH+Log at 376M–1B?
The Seesaw Effect emphasizes that SWA-NoPE hybrid models are stronger at length extrapolation after short-context pretraining, whereas LA-NoPE hybrid models benefit more from long-context continual pretraining, especially in layer-wise mixing (lines 213–215). Under these settings, the original LA-NoPE hybrid models underperform SWA-NoPE hybrid models with or without log-scaled NoPE attention. The models cited by the reviewer as outperforming SWA-NoPE hybrid models with log-scaled NoPE attention are instead SWLA-NoPE hybrid models, demonstrating the effectiveness of our proposed SWLA method. These results further support the effectiveness of sliding windows for length extrapolation in hybrid models, whether applied to softmax attention or linear attention. This is also the point emphasized by our Matthew Effect of Length Extrapolation in Hybrid Position.
Regarding
The CPT half is confounded: LongCE and window/base retuning were applied only to SWA hybrids,in W1 and Q3Does LongCE (and rotary-base tuning, where applicable) also help GLA/GDN-NoPE hybrids during CPT? Without this, the "competitive with GDN-NoPE-LH" claim in Sec. 3.2 compares tuned SWA against untuned GDN.
First, the GLA/GDN-NoPE models considered here do not use RoPE, so rotary-base tuning does not apply to them. During our exploration, we evaluate LongCE on both SWA-NoPE-LH and GLA/GDN-NoPE-LH at the smallest model scale and observe greater gains for SWA-NoPE-LH. We agree that these comparisons provide a fairer assessment and will include the existing results below in the revised manuscript. Further long-context continual pretraining runs will depend on the availability of GPU resources. We thank the reviewer for this suggestion.
| RU | BA | Avg. | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 4k | 8k | 16k | 32k | 64k | 0k | 2k | 4k | 8k | 16k | 32k | 64k | RU | BA | 4k | > 4k | All | |
| 376M Long | |||||||||||||||||
| SWA-NoPE-LH | 35.4 | 28.0 | 26.8 | 22.2 | 17.1 | 35.3 | 30.6 | 29.2 | 28.0 | 24.4 | 20.7 | 17.3 | 25.9 | 26.5 | 32.6 | 23.1 | 26.3 |
| + LongCE | 41.4 | 35.7 | 34.3 | 34.7 | 26.4 | 35.3 | 34.6 | 33.2 | 34.2 | 31.3 | 29.2 | 23.5 | 34.5 | 31.6 | 36.1 | 31.2 | 32.8 |
| GLA-NoPE-LH | 27.5 | 26.2 | 25.3 | 26.9 | 19.5 | 30.7 | 33.2 | 31.8 | 30.4 | 30.0 | 29.4 | 23.8 | 25.1 | 29.9 | 30.8 | 26.4 | 27.9 |
| + LongCE | 33.0 | 21.3 | 29.9 | 34.9 | 24.5 | 28.2 | 30.2 | 28.4 | 29.3 | 30.6 | 28.4 | 25.9 | 28.7 | 28.7 | 30.0 | 28.1 | 28.7 |
| GDN-NoPE-LH | 28.2 | 27.7 | 27.7 | 27.3 | 20.7 | 32.6 | 31.9 | 29.8 | 29.1 | 31.0 | 30.5 | 30.0 | 26.3 | 30.7 | 30.6 | 28.0 | 28.9 |
| + LongCE | 38.3 | 19.1 | 28.3 | 25.7 | 21.1 | 29.3 | 31.9 | 31.0 | 34.1 | 35.0 | 36.2 | 32.9 | 26.5 | 32.9 | 32.6 | 29.1 | 30.2 |
Regarding
and Table 18 shows that SWA rotary base 100 alone reverses the 376M ranking after CPT (30.0 vs GLA 27.9 / GDN 28.9).in W1 and Q4Given Table 18 (base=100 reversing the 376M ranking), why is the rotary base fixed at 10000 in all main-text SWA experiments?
We use a default rotary base of 10000 for SWA in the sliding-window hybrid models. This is the classical RoPE setting and also follows the hyperparameter choices of existing mainstream SWA hybrid models, such as MiMo-V2.5-Pro[1] and Gemma 4[2]. As the reviewer notes, base=100 performs well at 376M, but this advantage does not hold consistently. We therefore summarize results for bases of 10000, 1000, and 100 after short-context training, after short-context training with log-scaled NoPE attention, and after long-context training, and add experiments at the 1B scale we have done before. No base provides a consistent advantage across these settings. As we acknowledge in line 1020 of the paper, “we do not observe a consistent trend across all settings.” We therefore use base=10000 to follow a widely used configuration and make our findings relevant to that setting.
Reply (1/11)
We thank the reviewer for recognizing the scale of our experiments, the consistency of our findings, and our improvements to length extrapolation in linear-attention hybrid models. We respond to the weaknesses point by point below, addressing the corresponding questions along the way.
Our paper investigates length extrapolation and context extension in models that combine full attention with SWA or GLA/GDN, from the perspective of hybrid position. Under the default configurations, SWA hybrid models exhibit stronger length extrapolation, whereas LA hybrid models benefit more from long-context continual pretraining. Then we attribute the weak context extension in SWA hybrid models to excessive optimization of the local branch and improve performance by enlarging its window and using LongCE. For linear-attention hybrid models, we propose Sliding-Window Linear Attention, combined with log-scale sharpening of global NoPE attention, to improve training-free extrapolation.
Our reasoning is clear: we first identify the reversal in the relative advantages of linear-attention and sliding-window hybrid models before and after long-context training, namely the Seesaw Effect, and then propose methods to address the context-extension limitations of SWA hybrid models and the length-extrapolation limitations of LA hybrid models, respectively. The improvements achieved by our proposed methods do not invalidate the original observations or the limitations they identify. We also apply existing extrapolation methods, such as scaled NoPE attention, to SWA/GLA/GDN-NoPE hybrid models, but SWA-NoPE hybrid models retain their extrapolation advantage. We therefore propose Sliding-Window Linear Attention to constrain the OOD implicit positional information in linear attention, substantially improving extrapolation in LA-NoPE hybrid models, as the reviewer also acknowledges. SWLA is a new method that changes the inference computation of LA; its improvements cannot directly serve as evidence that the original LA baseline has no extrapolation limitations.
Log-scaled NoPE attention is an existing training-free method for improving the extrapolation of NoPE attention. The original long-context continual pretraining results in Table 11 do not use this additional scaling. Even when log-scaled NoPE attention is applied, SWA-NoPE hybrid models retain their extrapolation advantage over the original GLA/GDN-NoPE hybrid models. We can also compare models without any extrapolation enhancements. The table below reports results without extrapolation enhancements after both short-context training and long-context continual pretraining; the Seesaw Effect remains clearly present at all four model scales. Specifically, SWA-NoPE-LH shows better length extrapolation after short-context training, whereas GLA/GDN-NoPE-LH shows better context extension after long-context continual pretraining.