acceptodds
Under review as a conference paper at ICLR 2027

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

Abstract

Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention, Long Context, Neural Netork Architecture, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9VeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FreeSolo (FS), a sparse attention mechanism for zero-shot length extrapolation. It uses sliding-window RoPE attention for local modeling and cell-based sparse NoPE attention for long-range retrieval, where each query selects top-K cells from the full history. Its linear-time variant, FS-L, is obtained from FS through short continued pretraining at the same 4k context length and retains a fixed-size pool of cells based on key norms and recency before query-dependent selection. Experiments at 340M and 800M scales show improved RULER retrieval beyond the training length over several linear-time and sparse-attention baselines, with comparable short-context quality.

Critical strengths
  • Clear motivation and straightforward design. The paper addresses the trade-off between efficient sequence modeling and long-range retrieval through a simple separation of local attention, cell-based storage of history, and query-dependent retrieval.
  • The proposed method shows promising zero-shot extrapolation ability. Models trained at 4k achieve strong S-NIAH-1 retrieval up to 256k, and show substantial gains over several linear-time baselines on the harder S-NIAH-2/3 tasks beyond the training length, while maintaining comparable short-context quality.
Critical weaknesses

W1: The source of extrapolation is not fully isolated. The Introduction presents the main idea as combining range separation with keypoint detection. However, the ablations in Table 7 suggest that neither local RoPE nor the norm-weighted erf term is required for S-NIAH-1 extrapolation: (1) replacing local RoPE with NoPE preserves S-NIAH-1 extrapolation and even improves S-NIAH-2; (2) removing the norm-weighted erf term retains 95–97 on S-NIAH-1, while worsening S-NIAH-2 and improving S-NIAH-3. The remaining explanation suggested in Section 5 is that sparse selection keeps the long-range readout at a fixed size, but this is not validated. Therefore, it is unclear how much sparse selection contributes to extrapolation beyond the NoPE positional design.

W2: Limited evidence for effective retrieval beyond simple settings. The strongest long-range results are on single-needle S-NIAH tasks. On more complex retrieval tasks, FreeSolo improves over the linear-time baselines beyond the training length, but remains well below full attention even within the training context (e.g., FS at 800M and 4k in Table 3). Similarly, at 800M, accuracy drops from 96–100% with one fact to about 51% with two independently queryable facts (Table 18). Although Section 7 emphasizes FS-L's fixed retention budget as a limitation, Table 18 shows that the decline also occurs in unpruned FS within the training context, as acknowledged in Appendix D.5. This suggests a broader retrieval limitation and constrains the evidence for applicability beyond simple retrieval.

Suggestion: Several analyses central to the paper's claims are reported primarily in the appendix, including the ablations on position design and routing (Table 7), the continuation controls separating pruning adaptation from additional training (Appendix D.4), the multi-fact retrieval test (Table 18), and the checks of the streaming implementation (Appendix C). The main text summarizes them only briefly, so readers must consult the appendix to assess the main claims. Bringing the most consequential results into the main text would improve readability.

Questions for authors

Q1: Could the authors replace top-K selection with dense NoPE attention over the full history in the long-range branch to isolate the contribution of sparse selection to extrapolation? A sweep over K would further clarify the role of readout size. A comparison with RoPE/NoPE hybrids such as SWAN-GPT, which use dense global NoPE attention and also report zero-shot extrapolation (Section 6), would help position the contribution.

Q2: Regarding the multi-fact decline in Table 18, could the authors compare against full attention on the same examples and investigate whether the decline stems from query matching, fixed-K selection, or difficulty using the retrieved information? This would help clarify the method’s limitations beyond simple retrieval.

Q3: Appendix A reports different generation limits and scoring protocols for the MoBA and NSA suite summaries. Could the authors clarify why these differ and whether the comparisons hold under a unified evaluation protocol?

Q4: The accuracy tables use the full-history reference implementation, while the bounded streaming implementation is checked only on small S-NIAH subsets (Appendix C). Could the authors report RULER and downstream accuracy with the streaming implementation, so that accuracy and the linear-time efficiency results are evaluated under the same implementation?

Reviewer_zjjbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FreeSolo, a sparse attention mechanism for zero-shot length extrapolation. It combines local sliding-window attention with RoPE and long-range sparse attention using NoPE, where queries select relevant cells for direct token-level attention. The linear-time variant, FS-L, further introduces query-independent pruning and short continued pretraining to achieve linear attention complexity and bounded memory.Experiments demonstrate strong long-context retrieval performance beyond the training length, while maintaining competitive accuracy on zero-shot downstream tasks.

Critical strengths
  1. Both FS and FS-L demonstrate effective long-range retrieval beyond their 4k training context, with competitive performance against existing baselines, particularly on the harder S-NIAH-2/3 tasks.
  2. Both variants maintain competitive accuracy across eight zero-shot downstream tasks, suggesting that their long-context retrieval capabilities do not come at a substantial cost to general language understanding.
Critical weaknesses
  1. The bounded FS-L implementation exhibits relatively low prefill throughput, which limits its practical efficiency despite its theoretical linear-time complexity.
  2. The authors reports asymptotic attention complexity but does not provide FLOPs measurements across different context lengths, making it difficult to quantify the actual computational savings.
Questions for authors

Please see the concerns raised in Critical Weaknesses.

Reviewer_9H9AOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a sparse attention method, FreeSolo, that separates attention into two branches: local with RoPE and global with NoPE + sparse attention. More specifically, the local branch uses sliding-window attention with RoPE, while a long-range branch selects tokens (merged as cells) using key summaries and attends directly to cell tokens without positional information. Since FreeSolo scores all past cells, its routing still has quadratic complexity. To mitigate that, FreeSolo-L is proposed, which additionally retains high-norm and recent cells and allows linear-time processing and bounded attention storage. Both methods are trained via a short continued pretraining at the original 4K context length. Experiments include long-context benchmark (RULER), short-context downstream tasks, and efficiency measurements. Both variants retrieve single needles at contexts up to 256K, although performance on harder retrieval tasks shows signs of degradation.

Critical strengths
  • I like the separation between retaining information, locating it, and using it. These are different problems, and the paper makes a good effort to measure them separately.
  • The retention diagnostics and budget sweeps provide useful insight into where retrieval can fail. I also appreciate the strong baseline selection and the full RULER task breakdown in the appendix.
  • Retrieving a needle at 256K after training only at 4K is an interesting result, especially for FS-L with bounded attention storage.
  • The short-context downstream results also remain reasonably close to full attention, showing that the retrieval behavior does not come at the cost of a complete collapse in ordinary language-model quality.
Critical weaknesses
  • The strongest extrapolation result concerns single-needle retrieval, while the harder tasks show more limited generalization. At 800M/64K, FS-L scores 51 on S-NIAH-2 and 13 on S-NIAH-3, and its full RULER average at 16K is only 22.2. These results show an improvement over the evaluated baselines, but do not yet establish broadly effective long-context modeling. I think the claims should more clearly distinguish retrieving one distant fact from handling more demanding long-context tasks.

  • I still believe the comparisons do not fully isolate the contribution of sparse selection from position handling. Let me clarify:

    • The full-attention baseline uses RoPE, while FreeSolo explicitly avoids untrained long-range rotary offsets. A global NoPE baseline and a local-RoPE/global-NoPE hybrid such as SWAN-GPT would help clarify where the gains come from.
    • ASEntmax would also be informative as it focus exactly on the combination of sparsity and length extrapolation.
    • Raven-full could be included alongside FS in the quadratic group, with its complexity clearly labeled.
  • The erf term seems unnecessarily complicated to me given that it is almost saturated under the reported settings.
  • The fixed P and K budgets require additional manual tuning and may also lead to arbitrary suboptimal decisions.
  • Section 5 repeatedly sends the reader to the appendix for the supporting results, making the paper hard to follow.
Questions for authors
  • Did the hard top-P and top-K decisions cause loss spikes, routing collapse, or sensitivity to initialization? Can you provide training-loss curves and repeated runs, particularly at 800M?

  • Given that erf is almost saturated under the reported settings (final paragraph of D.1), why keep its distance dependence? How does a model trained with a constant norm bonus compare?

  • How were P and K chosen? How sensitive are quality, memory, and latency to these choices, and can the budgets adapt to the input?

  • What explains the large gap between S-NIAH-1 and the harder tasks?

  • What framework was used to implement FreeSolo? Triton? The efficiency numbers in Table 4 and 5 show that there's still room for improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.