- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GAPE, a learned attention-score bias that augments RoPE to improve length extrapolation. A query gate dynamically discounts distant context, while a key gate preserves salient "landmark" tokens from distance decay. Because the design is compatible with standard fused attention kernels (e.g., FlashAttention), it easily integrates into existing models. Evaluated on synthetic needle-in-a-haystack tests, small language models up to 32× training length, and an OLMo-3 7B LoRA retrofit, the paper claims 8× length extrapolation and long-context gains with no loss in standard task performance.
Critical strengths
- Consistent gains over its baselines: In the experiments trained from scratch, GAPE improves every base encoding it is added to. It raises needle-close retrieval for NoPE, RoPE, p-RoPE and YaRN without hurting any of them, and it lowers perplexity beyond the training length relative to RoPE and p-RoPE. The gains hold across base encodings, which supports the claim that GAPE complements them.
- Easy to add to existing models: GAPE augments RoPE instead of replacing it, which fits how current LLMs are built, and its pairing of a query gate with a key "landmark" gate is a combination I had not seen before. It needs no custom kernel and leaves the KV-cache size unchanged. The OLMo-3 7B retrofit adds about 66K parameters and does not degrade OLMES relative to a LoRA-only control. This gives the method a realistic path to adoption.
Critical weaknesses
- Not evaluated against existing solutions: GAPE targets length extrapolation: keeping perplexity stable and retrieval working beyond the training length. NoPE/RoPE hybrids (NoPE full-attention layers interleaved with RoPE local-attention layers, e.g., Llama 4, Command A) are an established solution to this problem, but the paper never compares against them. Its perplexity gains (Fig. 3) are measured only against RoPE and p-RoPE. The closest gated biases, CABLE (Veisi et al., EMNLP 2025) and TRA (Opper et al., TMLR 2025), already scale a distance penalty with a per-query gate and are neither cited nor compared, and the language-modeling experiments omit even ALiBi. At 7B, the retrofit is compared only with a LoRA-only control, not with YaRN, which OLMo 3 itself uses for context extension. The gains are also minimal: RULER changes by less than one point up to 16K, and at 32K, where both models essentially fail (about 12–13%), GAPE adds only 1.55 points in a single run per arm. Without these comparisons, it is unclear whether GAPE improves on what already exists.
- The experiments do not show what the gates contribute: The theory analyses GAPE's own bias, but almost every result also holds for a plain linear bias such as ALiBi, so it does not explain why the gates are needed. The case therefore rests on the experiments, which do not isolate the gates. Neither gate ablation (App. B.3) tests a learned fixed decay. Freezing the query gate at its initial value (≈0.05, against learned layer means of about 8–30) leaves almost no bias, and freezing the landmark gate keeps the query gate adaptive. The claim that the gate adapts to each query is never tested: in the needle-in-a-haystack task the input carries no cue for which needle is wanted, so each regime must be a separately trained model, and the language-modeling experiments report the gate only as layer and head averages. The experiments are also small. The retrieval results come from 2-layer models with roughly 0.1M parameters, trained only on the same synthetic task they are evaluated on. The 44M-parameter language models trained from scratch are evaluated on perplexity, entropy and NLL, but never on retrieval accuracy.
Questions for authors
-
[Experiment] Please compare GAPE with a NoPE/RoPE hybrid (NoPE full-attention layers interleaved with RoPE local-attention layers) on needle-in-a-haystack, perplexity and RULER beyond the training length, or show that adding GAPE to such a hybrid improves it. Without this, the benefit over existing solutions is unclear.
-
[Experiment] Please ablate GAPE against a learned fixed decay (both gates frozen at their learned magnitude), and train one needle-in-a-haystack model on both regimes to test whether the query gate adapts per query.
-
[Experiment] Please evaluate the pretrained 44M language models (or larger ones) on passkey, needle-in-a-haystack or RULER accuracy beyond the training length, and report the parameter count of the needle-in-a-haystack models.
-
[Presentation] Please enlarge the figure fonts. The legend of Fig. 2 is barely legible.
5 comments
Q2 — Learned static-decay controls
As detailed in our answer to W2 and Q3 above, we have introduced a learned fixed decay in the revised synthetic setup. The results show that full GAPE achieves higher accuracy under length extrapolation than this control on both NoPE and RoPE, supporting the contribution of token-dependent gating beyond a learned static distance penalty.
Moreover, we added a fixed decay baseline for:
- the 44M experiment (please see our response to Reviewer ixZm, Q3), showing that this control underperforms compared to GAPE;
- the 7B experiments, showing that GAPE outperforms this control at 16K and performs similarly at 24K, while the control performs slightly better at 32K (Tables 6 and 7 in the revised manuscript). These results support a retrieval benefit from token-dependent gating that varies with context length.
We refer the reviewer to our response to Reviewer ixZm, Q2–Q3 and Section 5 and Appendix C.2 (Tables 6–7) of the revised manuscript for the detailed 7B protocols, ablations, and numerical results.
Q4 — Figure readability
We thank the reviewer for the suggestion. We have enlarged the fonts and legend in Figure 2 to improve readability.
We hope that our rebuttal above clarifies your concerns, and we remain open to further discussion.
Looking forward to your response,
The Authors
Request for feedback
Dear Reviewer AcKr,
With the discussion period approaching its end, we would kindly ask for your feedback on our rebuttal and revised manuscript.
Following your suggestions, we added the requested NoPE/RoPE hybrid comparison, a dynamic retrieval task testing gate adaptation to the query, and a learned static-decay ablation. We also expanded the 7B evaluation to three seeds with uncertainty estimates and 3 more ablations: no RoPE pair replacements, learned static-decay and fixed static-decay. Furthermore, we clarified the comparisons with CABLE and TRA, reported the synthetic models’ parameter counts, and improved figure readability.
We have put substantial effort and computational resources in addressing all the concerns you outlined in your review, and we would be grateful for your feedback.
Thank you again for your time and consideration.
Best regards, The authors
Response to Reviewer AcKr
We thank the reviewer for their constructive feedback and for recognizing GAPE's strengths. In particular, we appreciate their acknowledgment of "consistent gains over its baselines" across positional encodings and improved perplexity beyond the training length, as well as its ease of integration into existing models without custom kernels or KV-cache growth. We also appreciate the recognition of GAPE's practical adoption potential and its ability to preserve standard benchmark performance. Below, we address the reviewer's concerns and questions in detail.
W1 and Q1 — Hybrid baselines and related work
We would like to remark that the 7B experiments are already considering a hybrid model, one that implements RoPE both for local and for full attention layers. For those experiments, GAPE showed improvements especially at longer lengths (16K, 24K and 32K) against the LoRA-only baseline.
Nevertheless, we welcome the reviewer’s request for another type of hybrid: NoPE full-attention interleaved with RoPE local-attention, and we conduct this experiment by pretraining another 44M baseline. The following table shows the results (perplexity), averaging the three document subsets:
In this scenario, GAPE keeps showing improvements compared to the hybrid baseline, albeit more modest. Nevertheless, we believe that modest improvements on a particular hybrid architecture do not detract from improvements on other hybrid architectures.
On the retrieval accuracy of the 44M
Given the small model capacity and the general pretraining regime, we use NLL to examine the effects of ablating the mask’s components. This provides a more sensitive measure of changes in predictive likelihood that may not be reflected in retrieval accuracy (Appendix C.3, Table 8), which is evaluated separately in the controlled synthetic setup and the 7B experiments (which entail also a learned but fixed decay ablation).
Related work
We thank the reviewer for pointing out these related approaches. CABLE (Veisi et al., 2025) builds its positional bias by accumulating learned contributions from intervening tokens, with query-dependent scaling. GAPE preserves physical token distances and directly adjusts how strongly each key is penalized, separating query-dependent suppression from landmark protection.
TRA (Opper et al., 2025) uses a relevance threshold to select keys and measures distance over the retained tokens. GAPE uses continuous gates to suppress distant context and protect selected keys, without thresholding attention scores or redefining positional distances.
We have clarified these differences in Section 2 (lines 156–161) and Appendix A (lines 852–890) of the revised manuscript.
W2 and Q3 — Gate contributions and parameter counts
On the synthetic setup
We thank the reviewer for pointing out that the original synthetic experiments did not isolate the contribution of token-dependent gating. We address this through a new dynamic NIAH task (Section 4 and Appendix C.5), in which an explicit query token specifies the requested window and the same model handles all query types. Our general comment explains the revised synthetic setup, controls, and results in detail.
On the parameter counts
The original NIAH models contain 100,352 parameters for the backbones and 100,620 with GAPE; the revised models contain 51,456 and 51,587 parameters, respectively. We deliberately use small Transformer configurations to isolate the gates’ contribution while limiting confounding effects from model depth and capacity. GAPE’s empirical effectiveness at larger scales is evaluated separately in the 44M- and 7B-parameter experiments.
On the context-independent ablations
We also train a learned static distance penalty and controls that replace either gate with a shared learned scalar. These controls allow the decay magnitude to be optimized, directly addressing the limitation of freezing gates at their initial values. Their lower joint accuracy under length extrapolation, together with the inference-time gate interventions, supports the contribution of both query-dependent suppression and token-specific protection in this task.