- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FreeSolo (FS), a sparse attention mechanism for zero-shot length extrapolation. It uses sliding-window RoPE attention for local modeling and cell-based sparse NoPE attention for long-range retrieval, where each query selects top-K cells from the full history. Its linear-time variant, FS-L, is obtained from FS through short continued pretraining at the same 4k context length and retains a fixed-size pool of cells based on key norms and recency before query-dependent selection. Experiments at 340M and 800M scales show improved RULER retrieval beyond the training length over several linear-time and sparse-attention baselines, with comparable short-context quality.
Critical strengths
- Clear motivation and straightforward design. The paper addresses the trade-off between efficient sequence modeling and long-range retrieval through a simple separation of local attention, cell-based storage of history, and query-dependent retrieval.
- The proposed method shows promising zero-shot extrapolation ability. Models trained at 4k achieve strong S-NIAH-1 retrieval up to 256k, and show substantial gains over several linear-time baselines on the harder S-NIAH-2/3 tasks beyond the training length, while maintaining comparable short-context quality.
Critical weaknesses
W1: The source of extrapolation is not fully isolated. The Introduction presents the main idea as combining range separation with keypoint detection. However, the ablations in Table 7 suggest that neither local RoPE nor the norm-weighted erf term is required for S-NIAH-1 extrapolation: (1) replacing local RoPE with NoPE preserves S-NIAH-1 extrapolation and even improves S-NIAH-2; (2) removing the norm-weighted erf term retains 95–97 on S-NIAH-1, while worsening S-NIAH-2 and improving S-NIAH-3. The remaining explanation suggested in Section 5 is that sparse selection keeps the long-range readout at a fixed size, but this is not validated. Therefore, it is unclear how much sparse selection contributes to extrapolation beyond the NoPE positional design.
W2: Limited evidence for effective retrieval beyond simple settings. The strongest long-range results are on single-needle S-NIAH tasks. On more complex retrieval tasks, FreeSolo improves over the linear-time baselines beyond the training length, but remains well below full attention even within the training context (e.g., FS at 800M and 4k in Table 3). Similarly, at 800M, accuracy drops from 96–100% with one fact to about 51% with two independently queryable facts (Table 18). Although Section 7 emphasizes FS-L's fixed retention budget as a limitation, Table 18 shows that the decline also occurs in unpruned FS within the training context, as acknowledged in Appendix D.5. This suggests a broader retrieval limitation and constrains the evidence for applicability beyond simple retrieval.
Suggestion: Several analyses central to the paper's claims are reported primarily in the appendix, including the ablations on position design and routing (Table 7), the continuation controls separating pruning adaptation from additional training (Appendix D.4), the multi-fact retrieval test (Table 18), and the checks of the streaming implementation (Appendix C). The main text summarizes them only briefly, so readers must consult the appendix to assess the main claims. Bringing the most consequential results into the main text would improve readability.
Questions for authors
Q1: Could the authors replace top-K selection with dense NoPE attention over the full history in the long-range branch to isolate the contribution of sparse selection to extrapolation? A sweep over K would further clarify the role of readout size. A comparison with RoPE/NoPE hybrids such as SWAN-GPT, which use dense global NoPE attention and also report zero-shot extrapolation (Section 6), would help position the contribution.
Q2: Regarding the multi-fact decline in Table 18, could the authors compare against full attention on the same examples and investigate whether the decline stems from query matching, fixed-K selection, or difficulty using the retrieved information? This would help clarify the method’s limitations beyond simple retrieval.
Q3: Appendix A reports different generation limits and scoring protocols for the MoBA and NSA suite summaries. Could the authors clarify why these differ and whether the comparisons hold under a unified evaluation protocol?
Q4: The accuracy tables use the full-history reference implementation, while the bounded streaming implementation is checked only on small S-NIAH subsets (Appendix C). Could the authors report RULER and downstream accuracy with the streaming implementation, so that accuracy and the linear-time efficiency results are evaluated under the same implementation?