acceptodds
Under review as a conference paper at ICLR 2027

Remember to Forget: Gated Adaptive Positional Encoding

Abstract

Rotary Positional Encoding (RoPE) is widely used in modern large language models. However, when sequences are extended beyond the range seen during training, rotary phases can enter out-of-distribution regimes, leading to spurious long-range alignments, diffuse attention, and degraded retrieval. Existing remedies only partially address these failures, as they often trade local positional resolution for long-context stability. We propose GAPE (Gated Adaptive Positional Encoding), a drop-in augmentation for positional encodings that introduces a content-aware bias directly into the attention logits while preserving the rotary geometry. GAPE decouples distance-based suppression from token importance through a query-dependent gate that contracts irrelevant context and a key-dependent gate that preserves salient distant tokens. We show that weakly protected distant context is exponentially attenuated as a function of the query gate, while selected keys can remain accessible through landmark protection. We further show that GAPE can be implemented within standard scaled dot-product attention. Empirically, GAPE improves long-context robustness across controlled retrieval and language-modeling experiments, extrapolating up to the training length. We further retrofit GAPE into a pretrained 7B model, maintaining performance on standard benchmarks and improving performance at the longest evaluated context. These results support adaptive context suppression as a complement to positional representation for long-context generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Positional Encoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AcKrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GAPE, a learned attention-score bias that augments RoPE to improve length extrapolation. A query gate dynamically discounts distant context, while a key gate preserves salient "landmark" tokens from distance decay. Because the design is compatible with standard fused attention kernels (e.g., FlashAttention), it easily integrates into existing models. Evaluated on synthetic needle-in-a-haystack tests, small language models up to 32× training length, and an OLMo-3 7B LoRA retrofit, the paper claims 8× length extrapolation and long-context gains with no loss in standard task performance.

Critical strengths
  1. Consistent gains over its baselines: In the experiments trained from scratch, GAPE improves every base encoding it is added to. It raises needle-close retrieval for NoPE, RoPE, p-RoPE and YaRN without hurting any of them, and it lowers perplexity beyond the training length relative to RoPE and p-RoPE. The gains hold across base encodings, which supports the claim that GAPE complements them.
  2. Easy to add to existing models: GAPE augments RoPE instead of replacing it, which fits how current LLMs are built, and its pairing of a query gate with a key "landmark" gate is a combination I had not seen before. It needs no custom kernel and leaves the KV-cache size unchanged. The OLMo-3 7B retrofit adds about 66K parameters and does not degrade OLMES relative to a LoRA-only control. This gives the method a realistic path to adoption.
Critical weaknesses
  1. Not evaluated against existing solutions: GAPE targets length extrapolation: keeping perplexity stable and retrieval working beyond the training length. NoPE/RoPE hybrids (NoPE full-attention layers interleaved with RoPE local-attention layers, e.g., Llama 4, Command A) are an established solution to this problem, but the paper never compares against them. Its perplexity gains (Fig. 3) are measured only against RoPE and p-RoPE. The closest gated biases, CABLE (Veisi et al., EMNLP 2025) and TRA (Opper et al., TMLR 2025), already scale a distance penalty with a per-query gate and are neither cited nor compared, and the language-modeling experiments omit even ALiBi. At 7B, the retrofit is compared only with a LoRA-only control, not with YaRN, which OLMo 3 itself uses for context extension. The gains are also minimal: RULER changes by less than one point up to 16K, and at 32K, where both models essentially fail (about 12–13%), GAPE adds only 1.55 points in a single run per arm. Without these comparisons, it is unclear whether GAPE improves on what already exists.
  2. The experiments do not show what the gates contribute: The theory analyses GAPE's own bias, but almost every result also holds for a plain linear bias such as ALiBi, so it does not explain why the gates are needed. The case therefore rests on the experiments, which do not isolate the gates. Neither gate ablation (App. B.3) tests a learned fixed decay. Freezing the query gate at its initial value (≈0.05, against learned layer means of about 8–30) leaves almost no bias, and freezing the landmark gate keeps the query gate adaptive. The claim that the gate adapts to each query is never tested: in the needle-in-a-haystack task the input carries no cue for which needle is wanted, so each regime must be a separately trained model, and the language-modeling experiments report the gate only as layer and head averages. The experiments are also small. The retrieval results come from 2-layer models with roughly 0.1M parameters, trained only on the same synthetic task they are evaluated on. The 44M-parameter language models trained from scratch are evaluated on perplexity, entropy and NLL, but never on retrieval accuracy.
Questions for authors
  1. [Experiment] Please compare GAPE with a NoPE/RoPE hybrid (NoPE full-attention layers interleaved with RoPE local-attention layers) on needle-in-a-haystack, perplexity and RULER beyond the training length, or show that adding GAPE to such a hybrid improves it. Without this, the benefit over existing solutions is unclear.

  2. [Experiment] Please ablate GAPE against a learned fixed decay (both gates frozen at their learned magnitude), and train one needle-in-a-haystack model on both regimes to test whether the query gate adapts per query.

  3. [Experiment] Please evaluate the pretrained 44M language models (or larger ones) on passkey, needle-in-a-haystack or RULER accuracy beyond the training length, and report the parameter count of the needle-in-a-haystack models.

  4. [Presentation] Please enlarge the figure fonts. The legend of Fig. 2 is barely legible.

5 comments

Response to Reviewer AcKr

We thank the reviewer for their constructive feedback and for recognizing GAPE's strengths. In particular, we appreciate their acknowledgment of "consistent gains over its baselines" across positional encodings and improved perplexity beyond the training length, as well as its ease of integration into existing models without custom kernels or KV-cache growth. We also appreciate the recognition of GAPE's practical adoption potential and its ability to preserve standard benchmark performance. Below, we address the reviewer's concerns and questions in detail.

W1 and Q1 — Hybrid baselines and related work

We would like to remark that the 7B experiments are already considering a hybrid model, one that implements RoPE both for local and for full attention layers. For those experiments, GAPE showed improvements especially at longer lengths (16K, 24K and 32K) against the LoRA-only baseline.

Nevertheless, we welcome the reviewer’s request for another type of hybrid: NoPE full-attention interleaved with RoPE local-attention, and we conduct this experiment by pretraining another 44M baseline. The following table shows the results (perplexity), averaging the three document subsets:

Context lengthHybridHybrid+GAPEPerplexity reduction (Hybrid − (Hybrid+GAPE))
25616.43 ± 0.0816.36 ± 0.08+0.07 ± 0.04
51214.52 ± 0.0714.45 ± 0.02+0.07 ± 0.06
1,02413.50 ± 0.0713.44 ± 0.04+0.06 ± 0.06
2,04812.98 ± 0.0912.91 ± 0.04+0.07 ± 0.07
4,09612.97 ± 0.0412.89 ± 0.08+0.08 ± 0.06
8,19213.78 ± 0.0713.66 ± 0.02+0.12 ± 0.05
16,38413.35 ± 0.1713.21 ± 0.21+0.14 ± 0.04
32,76813.09 ± 0.0612.91 ± 0.05+0.18 ± 0.05

In this scenario, GAPE keeps showing improvements compared to the hybrid baseline, albeit more modest. Nevertheless, we believe that modest improvements on a particular hybrid architecture do not detract from improvements on other hybrid architectures.

On the retrieval accuracy of the 44M

Given the small model capacity and the general pretraining regime, we use NLL to examine the effects of ablating the mask’s components. This provides a more sensitive measure of changes in predictive likelihood that may not be reflected in retrieval accuracy (Appendix C.3, Table 8), which is evaluated separately in the controlled synthetic setup and the 7B experiments (which entail also a learned but fixed decay ablation).

Related work

We thank the reviewer for pointing out these related approaches. CABLE (Veisi et al., 2025) builds its positional bias by accumulating learned contributions from intervening tokens, with query-dependent scaling. GAPE preserves physical token distances and directly adjusts how strongly each key is penalized, separating query-dependent suppression from landmark protection.

TRA (Opper et al., 2025) uses a relevance threshold to select keys and measures distance over the retained tokens. GAPE uses continuous gates to suppress distant context and protect selected keys, without thresholding attention scores or redefining positional distances.

We have clarified these differences in Section 2 (lines 156–161) and Appendix A (lines 852–890) of the revised manuscript.

W2 and Q3 — Gate contributions and parameter counts

On the synthetic setup

We thank the reviewer for pointing out that the original synthetic experiments did not isolate the contribution of token-dependent gating. We address this through a new dynamic NIAH task (Section 4 and Appendix C.5), in which an explicit query token specifies the requested window and the same model handles all query types. Our general comment explains the revised synthetic setup, controls, and results in detail.

On the parameter counts

The original NIAH models contain 100,352 parameters for the backbones and 100,620 with GAPE; the revised models contain 51,456 and 51,587 parameters, respectively. We deliberately use small Transformer configurations to isolate the gates’ contribution while limiting confounding effects from model depth and capacity. GAPE’s empirical effectiveness at larger scales is evaluated separately in the 44M- and 7B-parameter experiments.

On the context-independent ablations

We also train a learned static distance penalty and controls that replace either gate with a shared learned scalar. These controls allow the decay magnitude to be optimized, directly addressing the limitation of freezing gates at their initial values. Their lower joint accuracy under length extrapolation, together with the inference-time gate interventions, supports the contribution of both query-dependent suppression and token-specific protection in this task.

Q2 — Learned static-decay controls

As detailed in our answer to W2 and Q3 above, we have introduced a learned fixed decay in the revised synthetic setup. The results show that full GAPE achieves higher accuracy under length extrapolation than this control on both NoPE and RoPE, supporting the contribution of token-dependent gating beyond a learned static distance penalty.

Moreover, we added a fixed decay baseline for:

  • the 44M experiment (please see our response to Reviewer ixZm, Q3), showing that this control underperforms compared to GAPE;
  • the 7B experiments, showing that GAPE outperforms this control at 16K and performs similarly at 24K, while the control performs slightly better at 32K (Tables 6 and 7 in the revised manuscript). These results support a retrieval benefit from token-dependent gating that varies with context length.

We refer the reviewer to our response to Reviewer ixZm, Q2–Q3 and Section 5 and Appendix C.2 (Tables 6–7) of the revised manuscript for the detailed 7B protocols, ablations, and numerical results.

Q4 — Figure readability

We thank the reviewer for the suggestion. We have enlarged the fonts and legend in Figure 2 to improve readability.


We hope that our rebuttal above clarifies your concerns, and we remain open to further discussion.

Looking forward to your response,

The Authors

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer AcKr,

With the discussion period approaching its end, we would kindly ask for your feedback on our rebuttal and revised manuscript.

Following your suggestions, we added the requested NoPE/RoPE hybrid comparison, a dynamic retrieval task testing gate adaptation to the query, and a learned static-decay ablation. We also expanded the 7B evaluation to three seeds with uncertainty estimates and 3 more ablations: no RoPE pair replacements, learned static-decay and fixed static-decay. Furthermore, we clarified the comparisons with CABLE and TRA, reported the synthetic models’ parameter counts, and improved figure readability.

We have put substantial effort and computational resources in addressing all the concerns you outlined in your review, and we would be grateful for your feedback.

Thank you again for your time and consideration.

Best regards, The authors

Reviewer_ZXHwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GAPE, an additive, content-dependent bias on the attention logits that is applied on top of an unchanged RoPE-style rotary map to improve length extrapolation. In relative form, the bias is M^i,j=−Γhgi(1−lj)(i−j)/T\hat M_{i,j} = -\Gamma_h g_i(1-l_j)(i-j)/T: a query gate gi=Softplus(wg⊤qi)g_i=\mathrm{Softplus}(w_g^\top q_i) sets how strongly query ii contracts its context, a key ("landmark") gate lj=σ(wl⊤kj)l_j=\sigma(w_l^\top k_j) exempts key jj from the distance penalty, Γh\Gamma_h is a per-head amplitude, and TT is the training length. Up to a per-query constant, which softmax ignores, the bias is a sum of two query-factor × key-factor products, so it can be realized exactly by reserving two query/key coordinates. GAPE therefore runs inside standard fused SDPA/FlashAttention without an N×NN\times N bias matrix or extra KV-cache memory. The theory characterizes limiting regimes: as lj→1l_j\to1 a key is fully protected, and as gig_i grows the attention mass on weakly protected keys decays exponentially, which bounds softmax dilution and attention entropy.

The validation has three parts: (i) a synthetic needle-in-a-haystack (NIAH) task in which 2-layer models trained at 2K and evaluated up to 16K must retrieve the first ("far") or last ("close") of many needles, comparing NoPE, ALiBi, FoX, RoPE, p-RoPE and YaRN with and without GAPE; (ii) OOD perplexity, attention-entropy and gate analyses for a 44M nanoGPT trained at 1K, comparing p-RoPE+GAPE with RoPE and p-RoPE; and (iii) a retrofit of an OLMo-3 7B checkpoint (frozen backbone, rank-8 LoRA with or without GAPE, 100M tokens at up to 12K), where OLMES Tier-1 is unchanged (66.00 vs. 66.06) and RULER-100 changes by −0.17/−0.81/+0.22/+1.55 points at 4K/8K/16K/32K.

Critical strengths
  1. A simple, deployable mechanism with a clean decomposition. GAPE separates how strongly a query contracts its context (gig_i) from which keys are exempt from that contraction (ljl_j), while leaving the rotary map untouched. Proposition 1 makes it exact inside stock fused SDPA/FlashAttention: no custom kernel, no N×NN\times N bias, no KV-cache growth, and about 66K extra parameters at 7B. It is therefore cheap to try in existing RoPE models and composes with RoPE-modifying methods such as p-RoPE and YaRN. Unlike cumulative forget-gate decay (FoX), whose bias is non-increasing in distance, the key gate can in principle favor an individual distant key over nearer ones (second part of Theorem 1). This practicality is the main argument for acceptance.

  2. Clear evidence that query-dependent contraction helps when distractors accumulate. In the close-needle NIAH setting, adding GAPE substantially improves accuracy at 8× the training length (YaRN 49.0%→84.6% at 16K; RoPE ≈0.1→≈0.43 and p-RoPE ≈0.15→≈0.52, read from Fig. 2). A NoPE+learned-temperature control does not reproduce NoPE+GAPE, so the effect is not just uniform sharpening, and the learned gig_i is large for close targets and ≈0 for far ones (Fig. 7), which is interpretable. The 7B retrofit leaves OLMES Tier-1 unchanged, so the mechanism can be added post hoc without hurting short-context ability. This supports the value of adaptive, query-dependent contraction.

Critical weaknesses
  1. The evidence for the key-protection (landmark) mechanism is still indirect. Decoupling suppression from protection is the paper's main conceptual contribution, so it would be valuable to isolate it more directly. The clearest gains are in the close-needle setting, where a recency bias already helps (ALiBi is also strong there; Fig. 2). In the far-needle setting, where protecting distant keys should matter most, the gains for RoPE and p-RoPE are limited to 4K, only YaRN improves at 16K (27.2%→40.1%), and the learned query gate is ≈0 (Fig. 7), which suggests the model mostly switches the bias off rather than protecting the needle. A fixed-gate control (RoPE + learned static distance bias) in NIAH, and ALiBi/FoX baselines in the language-modeling experiments (currently only RoPE and p-RoPE are compared), would help show that the content-dependent gates drive the improvements.

  2. The 7B results are encouraging but preliminary. This matters for the practical claim that GAPE can be retrofitted into existing LLMs. The retrofit preserves OLMES and gains 1.55 points on RULER at 32K, but this comes from a single run per condition with 100 examples per subtask, absolute accuracy at 32K is low (11.6–13.2%), and subtask changes vary in sign (Table 5). GAPE also replaces one RoPE pair (App. B.2.1), which the LoRA-only control does not. Seeds or confidence intervals, a matched control, and a YaRN vs. YaRN+GAPE comparison would make the benefit more convincing.

Questions for authors
  1. In the far-needle setting, why do the gains for RoPE and p-RoPE fade beyond 4K, and what drives the YaRN+GAPE gain at 16K given gi≈0g_i\approx0 (Fig. 7)?
  2. In the close setting, the rotary baselines reach only ≈0.7 at the 2K training length (Fig. 2). Were they trained to convergence? Reporting accuracy relative to the in-distribution value would help separate learnability from extrapolation.
  3. Beyond the controls mentioned under Weaknesses, a short discussion of related content-dependent logit biases, such as DAPE (Zheng et al., NeurIPS 2024) and Selective Attention (Leviathan et al., ICLR 2025), would help position the contribution.
2 comments

We thank the reviewer for their constructive feedback. We particularly appreciate their recognition of GAPE's "simple, deployable mechanism with a clean decomposition", its compatibility with standard fused SDPA/FlashAttention, and its ability to improve retrieval through adaptive, query-dependent contraction. We also appreciate the acknowledgment of GAPE's practical value, including its integration into pretrained models without degrading short-context capabilities. Below, we address the reviewer's concerns and questions in detail.

W1, Q1 and Q2 — The revised synthetic setup and convergence

We thank the reviewer for highlighting these limitations of our synthetic setup, and for the need to isolate landmark protection more directly. To this end, we introduce a new dynamic NIAH task, presented in Section 4 and Appendix C.5, which requires suppressing distant updates while preserving distant protected information. Models and controls are trained to convergence, assessed through a plateau in validation loss, retaining the checkpoint with the lowest validation loss.

Our general comment explains the motivation, results, and additional controls, including the requested learned static distance bias.

W2 — The 7B retrofit and YaRN

7B Experiments: more seeds, intact RoPE, and ALiBi and learnable static decay baselines

We have expanded the 7B experiments to three adaptation seeds per condition and now report means and standard deviations. Our primary GAPE variant now preserves every pretrained RoPE pair while retaining FlashAttention-2 compatibility, removing the RoPE-pair replacement confound. Moreover, we provide two more controls: an ALiBi adaptation and a learned but fixed decay. The LoRA-only, intact-RoPE GAPE and ALiBi comparison can be found in Table 1, while the old GAPE variant that replaces two RoPE coordinates and the learned decay can be found in Tables 6–7.

Comparing GAPE to LoRA-only at 32K, mean RULER accuracy increases from 12.48% to 14.05%, a gain of 1.57 percentage points with a confidence interval of [+0.39, +2.85], obtained with bootstrapping across three seeds. We also added 24K evaluation, where accuracy increases from 17.73% to 19.66%. See the detailed results in our response to Reviewer ixZm, Q2, and the updated manuscript (Section 5, lines 477–523; Appendix C.2.4, lines 1593–1613; Tables 6–7).

YaRN

Regarding YaRN, we directly compare YaRN with YaRN+GAPE in the new controlled synthetic experiments. At 8K, mean joint accuracy increases from 68.44% to 80.66% across three seeds, supporting their complementary use (Figure 2 in the revised manuscript). Within the rebuttal compute budget, we prioritised additional 7B seeds, preservation of pretrained RoPE, and the ALiBi and learned static-decay controls. A matched YaRN versus YaRN+GAPE comparison at 7B remains untested, so we do not claim superiority over YaRN at that scale.

Q3 — DAPE and Selective Attention

We thank the reviewer for suggesting these references. DAPE (Zheng et al., 2024) learns a nonlinear correction from attention scores and positional biases. GAPE uses an explicit distance penalty that separates context suppression from key protection: the query gate controls how strongly distant context is suppressed, while the landmark gate protects selected keys. This structure allows GAPE to operate within standard fused attention.

Selective Attention (Leviathan et al., 2025) accumulates suppression decisions as the sequence progresses. GAPE adjusts suppression separately for each query, while its landmark gate protects selected keys. A later query can therefore reduce the penalty on previously suppressed information, without accumulating earlier forgetting decisions.

We have clarified these differences in Section 2 (lines 156–161) and Appendix A (lines 837–850 and 892–911) of the revised manuscript.


We hope that our rebuttal above clarifies your concerns, and we remain open to further discussion.

Looking forward to your response,

The Authors

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer ZXHw,

With the discussion period approaching its end, we kindly ask for your feedback on our rebuttal and revised manuscript.

Following your suggestions, we introduced a dynamic retrieval task with direct gate interventions and ablations and expanded the 7B evaluation to three seeds with uncertainty estimates and 3 more ablations: no RoPE pair replacements, learned static-decay and fixed static-decay. We also evaluated YaRN+GAPE in the revised synthetic task, and expanded the related-work discussion.

We have put substantial effort and computational resources in addressing all the concerns you outlined in your review, and we would be grateful for your feedback.

Thank you again for your time and consideration.

Best regards, The authors

Reviewer_ixZmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper augments RoPE with an additive, content-dependent bias on the attention logits, leaving the rotary phases themselves in place. The bias is a linear distance penalty whose slope is a learned per-query scalar g_i = Softplus(wᵀ_g q_i), whose per-key strength is attenuated by a learned "landmark" protection l_j = σ(wᵀ_l k_j), and which is scaled by a per-head amplitude Γ_h and a fixed normalizer T set to the training context length: M_{i,j} = (Γ_h g_i / T)[j(1−l_j) + i l_j] (Eq. 2), with the equivalent relative-distance form M̂_{i,j} = −(Γ_h g_i/T)(i−j)(1−l_j). Two extra query/key coordinates reproduce the bias exactly inside a fused scaled dot-product attention, i.e. as a rank-at-most- two correction with no N×N matrix and no change to the KV-cache shape (Eq. 3, Prop. 1, App. A.5, B.7). Short algebraic results describe the induced attention distribution: landmark saturation makes a distant key structurally indistinguishable from the current position (Thm 1), the mass on unprotected keys decays exponentially in g_i giving an "effective context length" (Thm 2), the current-position logit can be guaranteed to dominate a distant key above a gate threshold (Thm 3), and the distribution converges to the protected set as g_i → ∞ (Cor. 1). Empirically the paper reports a synthetic NIAH study with 2-layer transformers trained at 2,048 tokens and tested to 16,384, an OOD perplexity study with a 44M nanoGPT trained at 1,024 tokens and evaluated to 32,768, and a LoRA retrofit into an 8K-native OLMo-3 7B checkpoint evaluated on OLMES Tier-1 and RULER-100 at 4K–32K.

Critical strengths

S1. The bias is realised as an exact, essentially free rank-2 correction inside standard attention. Eq. (3) appends two coordinates to q̃_i and k̃_j so that (1/√d) q̃ᵀ_i k̃_j = s_{i,j} + M_{i,j}, and Prop. 1 with App. A.5 shows the two mask forms differ only by the query-dependent constant c_i = −Γ_h g_i i/T, which cancels under softmax. App. B.7 then argues — correctly, since k̃_t is what gets cached — that the KV-cache shape is unchanged. This is consequential because it means the contribution is not a kernel-engineering paper in disguise: the mechanism can be dropped into an existing FlashAttention stack, which is the main reason the idea is worth reading at all.

S2. The controlled NIAH study is better controlled than most work in this area, and the paper reports results that undercut its own case. The baseline set spans NoPE, ALiBi, FoX, p-RoPE and YaRN, plus a NoPE+learnable temperature control that adds "only four scalar parameters and no positional encoding or GAPE mask" specifically to test whether GAPE "can be reduced to uniform attention sharpening" (App. B.4). The paper also does not hide unfavourable numbers: variable tracking falls "from 22.6% to 13.4%" at 8K, NIAH S2 and MK2 drop 7.0 and 6.0 points at 16K, and the Limitations section concedes the single adaptation run and that "absolute performance at this length remains low". This is consequential because it makes the remaining problems attributable to the design of the evidence rather than to selective reporting, and it gives the community clean negative data points about recency-prior baselines.

Critical weaknesses

W1. The paper's largest empirical gain occurs in the regime where its own measurements show the proposed mechanism is inert, and nothing else in the paper supplies the missing mechanism. (a) Fig. 7 and its caption state that "When the target is far, gi remains near zero, indicating that the model relaxes the recency bias" — reading the plotted bars, g_i ≈ 0.005 for far versus ≈0.7 for close. Since Eq. (2) makes the entire mask proportional to g_i, a near-zero gate leaves the model numerically indistinguishable from the base positional encoding. Yet the bottom row of Fig. 2 is where the paper's largest effects appear (NoPE with and without GAPE diverge starkly at 16K), and the text reports "at 16K, YaRN+GAPE improves over YaRN from 49.0% to 84.6% for close retrieval and from 27.2% to 40.1% for far retrieval" — a 12.9-point far-regime gain attributed to a mechanism whose gate is off there. (b) The theory cannot fill the gap: Theorem 3 is a recency statement ("a sufficient condition for the current-position logit to exceed that of j is gi > 2SmaxT/(Γh∆(1−lj))"), i.e. it guarantees suppression of distant keys, the opposite of recovering them; and no result in Section 3 or Appendix A contains a rotary phase or frequency, so "phase hallucination" is not what is being analysed. Worse, the threshold in Eq. (18) is per-key in ∆ and l_j, while g_i is a single scalar per query with no access to distance, so the condition is not one the model can satisfy key-by-key. (c) The ablation (App. B.3, Table 6) does not exclude the degenerate explanation that the gains come from training with two extra structural coordinate dimensions rather than from routing: it removes only the learned dependence of g and l on content ("making it non-learnable and fixing it to the corresponding value reported in Table 2"), and the cost is small and non-uniform — "Removing either gate degrades performance in 6 of the 8 evaluated task–length settings", with g removed improving both common-word-extraction settings (−3.10%, −2.07%). The closest prior work is not engaged: content-dependent positional encoding (CoPE, Golovneva et al., arxiv:2405.18719) and content-based suppression of keys in the attention logits (Selective Attention, Leviathan et al., arXiv:2410.02703, ICLR 2025) are absent from the reference list I read, so the delta over that line of work is unspecified. This is consequential because the separation of "contraction" from "preservation" is the paper's stated novelty and the sole explanation offered for its headline long-range result; if the mask is inert exactly where the result appears, then either a figure is mislabelled or the effect is produced by something other than the claimed mechanism, and neither the contribution nor the conclusion survives unchanged.

W2. The 7B retrofit — the only large-scale evidence — is within noise, confounded with a change to the rotary geometry, and is reported in the abstract as an established improvement. (a) Table 1 comes from "a single 100M-token adaptation run per condition" (Section 5) with "100 examples per subtask" (RULER-100). At the observed accuracy levels the standard error on a subtask is roughly 2–5 pp, so the reported deltas of +0.054 (OLMES), −0.17 (4K), −0.81 (8K), +0.22 (16K) and +1.55 (32K) pp are not distinguishable from run-to-run variation; no seed count, interval or significance test appears anywhere in the paper, and Figs. 2–4 say "mean ± std" without stating over what. (b) The 32K "aggregate gain" rests on absolute scores of 11.62 → 13.17, near chance for several subtasks, and the per-subtask pattern looks like noise rather than capability: 16K variable tracking moves +20.00 pp (15.20 → 35.20) after moving −9.20 pp (22.6 → 13.4) at 8K on the same subtask, and at 32K a +11.33 pp gain on frequent-word extraction coexists with −4.50 pp on common-word extraction. (c) The two arms differ in more than the mask: App. B.2.1 states "The structural coordinates replace one pretrained RoPE pair" and "replace the split-half RoPE pair at head dimensions 63 and 127" — the lowest-frequency pair, i.e. exactly the channel type the paper's own Section 2 calls a fragile semantic carrier. There is also no ALiBi, FoX or content-independent learned per-head slope at 7B or in the 44M perplexity study, so even a real delta could not be attributed to content-aware gating rather than to a learned distance penalty or to the re-parameterised head. This is consequential because the abstract's conclusion — "retrofit GAPE into a pretrained 7B model, maintaining performance on standard benchmarks and improving performance at the longest evaluated context" — rests entirely on this single comparison, and as reported it is a null result being presented as a positive one.

Questions for authors
  1. Is the far-regime gain produced by the mask at all? (targets W1a) Please re-evaluate the NIAH models with the structural term disabled at inference (g_i = 0, or l_j = 1). If far accuracy stays near the GAPE level, the effect is a training-time/parameterisation effect rather than routing. Please also report g_i and the resulting M_{i,j} at all positions (not only the final query token) at 16K for both regimes.
  2. Variance at 7B. (targets W2a–b) How many seeds underlie Table 1? Please report at least three adaptation seeds per condition with confidence intervals, and a paired bootstrap over the 13×100 RULER examples at 32K, so that +1.55 pp can be compared with its own sampling error.
  3. The competing "static decay" hypothesis. (targets W1c, W2c) Can you add, at both the 44M and 7B scales, (i) a learned but content-independent per-head slope (the g/l fixed-to-constant limit, but trained) and (ii) ALiBi and FoX? This is the comparison that decides whether content-awareness — rather than the distance penalty itself — is what helps. Please also state explicitly how GAPE differs from content-dependent positional encoding (CoPE) and content-based logit masking (Selective Attention).
  4. Decoupling the mask from the RoPE-pair removal. (targets W2c) Can you run the retrofit with the head widened by two dimensions so that no RoPE pair is removed, or with the mask added on top of the intact pair? Otherwise the gain cannot be separated from dropping the 63/127 pair.
  5. Where are the gates computed? (targets the mechanism in Eq. 2–3) Are g_i and l_j functions of pre-RoPE or post-RoPE q/k? Prop. 2's translation invariance explicitly assumes "gi′ = gi, and lj′ = lj", which holds only for un-rotated vectors, while Eq. (3) describes "the semantic coordinates after the chosen multiplicative positional map". If the gates see rotated vectors, does the invariance still hold empirically?
  6. Reporting. (targets reproducibility) Please tabulate the Fig. 3 perplexities (with the number of seeds and what the error bars denote), state the normalizer T used for the 44M model (Table 2 lists none, although T sets the scale of the whole penalty), state the NIAH evaluation-set size, and explain how 2^15-token evaluation windows are formed from the 2^13-average-document subset given the claim that the models "process genuinely contiguous long texts rather than artificially concatenated sequences".

Criteria under which my score would rise: a mask-off inference control that shows the far-regime gains survive (or an explicit revision of the mechanism story); multi-seed 7B results with intervals; a content-independent learned-slope baseline plus ALiBi/FoX at both scales; and a d+2 retrofit that removes the RoPE-pair confound. Meeting (1)–(4) with results that support the current claims would move me to 3 (weak acceptance); with only (2), (3) and (6) I could reach 2. The present manuscript asks the reader to accept the mechanism on the strength of an effect that the paper's own Fig. 7 says the mechanism cannot produce.

4 comments

We thank the reviewer for their thorough, constructive feedback. We particularly appreciate their recognition of GAPE's "exact, essentially free rank-2 correction inside standard attention", comprehensive baselines and transparent reporting. We also appreciate the positive assessment of the paper's clarity. Below, we address the reviewer's concerns and questions in detail.

Q1 — The far-regime gain and the revised synthetic setup

We thank the reviewer for highlighting this limitation of our synthetic setup. We agree that the original NIAH experiments did not establish that the far-regime gain was produced by active masking at inference. We therefore introduce a new dynamic NIAH task, presented in Section 4 and Appendix C.5. Since all reviewers raised related concerns, we explain in our general comment why the original setup did not isolate GAPE’s mechanism, how the new task addresses this limitation, and what the results show.

We retain the original experiments in Appendix C.4 because they nevertheless reveal different strengths and weaknesses of the positional encoding schemes considered, while explicitly clarifying that the observed Needle-far gain is consistent with a training-time or parameterisation effect.

Q2 — Variance in the 7B experiments

The submitted Table 1 used one adaptation run per condition. We have expanded the experiments to three adaptation seeds per condition, using the same training recipe and 100M adaptation tokens per seed. The revised tables report accuracy mean ± standard deviation across seeds, including detailed RULER subtask results.

We additionally perform 100,000 bootstrap [95% Confidence Interval]. The updated values are:

Setup / contrast4K8K16K24K32K
LoRA-only93.50 ± 0.2179.84 ± 0.3659.35 ± 1.0817.73 ± 0.5312.48 ± 0.46
ALiBi75.09 ± 0.3850.84 ± 0.9725.84 ± 0.9716.54 ± 0.3015.04 ± 0.52
Learned decay93.49 ± 0.0979.59 ± 0.5656.36 ± 0.8719.47 ± 0.2214.78 ± 0.34
GAPE (replaced RoPE)93.42 ± 0.2680.56 ± 0.5259.41 ± 0.5119.00 ± 0.4113.41 ± 0.12
GAPE*93.74 ± 0.1180.22 ± 0.3960.82 ± 1.1319.66 ± 0.2114.05 ± 0.45
Δ vs. LoRA-only [CI]+0.24 [-0.43, +0.91]+0.38 [-0.79, +1.53]+1.46 [-1.31, +4.08]+1.93 [+1.87, +1.98]+1.57 [+0.39, +2.85]
Δ vs. ALiBi [CI]+18.65 [+17.03, +20.28]+29.38 [+26.93, +31.85]+34.97 [+31.90, +38.04]+3.12 [+2.13, +3.87]-0.99 [-2.21, +0.22]
Δ vs. learned decay [CI]+0.25 [-0.33, +0.84]+0.62 [-0.22, +1.47]+4.46 [+3.23, +5.70]+0.19 [+0.15, +0.23]-0.73 [-1.41, -0.06]

*This variant, discussed in our response to Q4, preserves all pretrained RoPE coordinates.

Q3 — Static-decay controls and related work

We added ALiBi and learned (static) decay controls at both the 44M pretraining scale and the 7B retrofit scale. ALiBi supplies a fixed distance penalty. In the learned fixed decay, the gate outputs are held constant while the per-head amplitude is learned, allowing the model to optimise distance-decay strength without query-dependent contraction or key-specific protection.

44M ALiBi and fixed decay

At 44M, all conditions use three training seeds and a training context of 1,024 tokens. The table reports mean ± SD across seeds, calculated after averaging each seed’s perplexity equally across the three document-length subsets (2132^{13}, 2142^{14}, and 2152^{15}). Lower is better

Context lengthALiBiLearned DecayRoPE+GAPE*
25616.36 ± 0.0816.18 ± 0.0216.86 ± 0.08
51214.44 ± 0.0414.25 ± 0.0814.37 ± 0.05
1,02413.44 ± 0.0513.22 ± 0.1013.15 ± 0.08
2,04813.10 ± 0.0724.23 ± 1.8014.51 ± 0.86
4,09613.36 ± 0.0865.56 ± 2.5342.94 ± 7.36
8,19214.26 ± 0.04131.83 ± 2.14110.80 ± 9.12
16,38413.62 ± 0.28200.71 ± 0.79172.91 ± 11.09
32,76812.82 ± 0.07278.71 ± 4.05227.73 ± 8.50

*This variant, discussed in our response to Q4, preserves all pretrained RoPE coordinates.

RoPE+GAPE achieves lower mean perplexity than the learned decay variant at every evaluated context beyond the training length, both in this aggregate and within each document-length subset. At 32K, the aggregate reduction of perplexity is 18.3%. These results support a benefit over the learned static-decay control.

ALiBi achieves substantially better OOD perplexity than either variant. However, we would like to remark that perplexity alone can be misleading, and it should be read jointly with retrieval accuracy, as a model can achieve lower perplexity by simply “ignoring” part of the context. This is the case for ALiBi, as its retrieval accuracy on RULER degrades substantially at 4K, 8K, and 16K context lengths (−18.65, −29.38, and −34.97 percentage points) compared to GAPE+LoRA on the 7B models.


We continue our rebuttal in the next response

7B ALiBi and fixed decay

At 7B, both controls use the same LoRA adaptation recipe and token budget as GAPE, with three adaptation seeds per condition (see detailed results in the table in the answer to Q2). ALiBi substantially reduces shorter-context performance. The learned decay variant produces less consistent gains across context lengths. Although it improves on LoRA-only at 24K and 32K, at 16K.

GAPE’s mean exceeds LoRA-only at every evaluated context length. At 24K, it also outperforms both static controls: 19.66, versus 19.47 for learned decay and 16.54 for ALiBi. At 32K, learned decay and ALiBi score higher than GAPE, but these gains accompany losses at shorter or intermediate lengths.

Together, these results indicate that the benefits depend on the setting. ALiBi is particularly effective for 44M perplexity extrapolation, while content-dependent gating improves on learned decay at that scale and provides more balanced RULER performance across context lengths in the 7B retrofit.

Comparison with related work

We thank the reviewer for pointing out these related approaches. CoPE (Golovneva et al., 2024) learns which intervening tokens count towards positional distance. GAPE preserves physical token distances and the original RoPE geometry, and instead learns how strongly each query suppresses distant context and which keys are protected from that suppression.

Selective Attention (Leviathan et al., 2025) accumulates suppression decisions as the sequence progresses. GAPE adjusts suppression separately for each query, while its landmark gate protects selected keys. A later query can therefore reduce the penalty on previously suppressed information, without accumulating earlier forgetting decisions.

We have clarified these differences in Section 2 (lines 156–161) and Appendix A (lines 817–836 and 892–911) of the revised manuscript.

Q4 — Preserving all pretrained RoPE pairs

We are grateful for the reviewer’s suggestion, which led us to develop a stronger retrofit implementation. We implemented and evaluated a variant that preserves every pretrained RoPE pair, appending the two auxiliary coordinates required to encode the GAPE bias to the query and key vectors. We retain FlashAttention-2 compatibility by padding the augmented vectors to a supported head width; these padding coordinates introduce no additional trainable parameters.

This variant achieves slightly higher mean RULER scores than the pair-replacement implementation at four of the five evaluated context lengths. For example, at 16K the score increases from 59.41 to 60.82, and at 32K from 13.41 to 14.05, compared with 59.35 and 12.48 for LoRA-only, respectively. Thus, the observed improvements over LoRA-only do not require removing a pre-trained RoPE pair. See detailed results in the table in the answer to Q2.

We have revised the manuscript to make this intact-RoPE implementation the primary GAPE variant, used for the main-paper comparisons, and updated the implementation description and results accordingly (Section 5, lines 477–491, and Appendix C.2.1, lines 1501–1508). We retain the original implementation as GAPE (repl.) in Appendix C.2.1 (lines 1509–1511) and Appendix C.2.4 (lines 1600–1604; Tables 6–7), remaining competitive and preserving the original head width. These alternatives may offer greater flexibility in a more realistic (large-scale) deployment scenario.

Q5 — Where the gates are computed

Both gates are computed from the pre-RoPE query and key representations. RoPE is applied to the semantic query/key coordinates used in the attention dot product, while the structural coordinates are constructed from the previously computed gates.

The wording around Eq. (3) referred to these rotated semantic coordinates, but reusing the same query/key notation made the computation order ambiguous. We have clarified this distinction in the manuscript (lines 208–212). Under the common position-index shift considered in Proposition 2, with the content representations held fixed, the gate inputs and outputs remain unchanged, satisfying the stated assumptions. The implementation therefore does not require an empirical invariance claim for gates computed from post-RoPE vectors.

Q6 — Reporting and evaluation protocol

We have added a table of the numerical results underlying Figure 4 in Appendix C.1, Table 3 (lines 1451–1453 and 1458–1486), together with the seed count, batch sizes, sample counts, and definition of the reported mean and standard deviation.

To improve the clarity of our evaluation protocol, we have revised the main text, caption, and appendix to clarify that models are trained at 1,024 tokens and evaluated separately on each document-length subset, varying the context length within every panel of Figure 4 (lines 409–417 and 442–444; Appendix C.1, lines 1439–1446).

We are particularly greateful for the insightful comments which we believe substantially strenghtened our work. We hope that these comprehensive set of additional experiments and clarifications positively answers your concerns.

We look forward to your response,

The Authors

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer ixZm,

With the discussion period approaching its end, we kindly ask for your feedback on our rebuttal and revised manuscript. We are particularly grateful for your detailed suggestions, and we have put substantial effort and computational resources to address your concerns, which we believe strenghtened our work.

In particular, we followed your advice and responded to all your six questions:

  1. We revised the synthetic setup and its interpretation,
  2. added three-seed 7B results with confidence intervals,
  3. introduced ALiBi and learned static-decay controls at both scales, and
  4. evaluated a retrofit preserving all pretrained RoPE pairs.
  5. We also clarified the implementation and
  6. experimental reporting.

We hope that these additional experiments and clarifications address your concerns and the criteria you outlined for reconsidering your assessment. We would particularly appreciate your feedback on the revised evidence and any concerns that remain.

Thank you again for your time and consideration.

Best regards, The authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Revised synthetic experiments

Revisiting the original experiments

We thank the reviewers for highlighting an important limitation of the original NIAH-far/NIAH-close setup. We agree that these experiments did not adequately isolate GAPE’s input-dependent suppression and protection. In particular, the two retrieval regimes were learned by separate models, each with a fixed retrieval rule. This did not directly test whether the gates adapt to different queries within the same model.

Following this concern, we evaluated the effect of disabling the mask at inference in the original Needle-far setup and observed little change in accuracy. This means that the observed gain cannot be attributed to active masking at inference and is consistent with a training-time or parameterisation effect. We moved these experiments to Appendix C.4 and revised their interpretation accordingly (lines 1787–1791 and 1812–1817), as they nevertheless expose different strengths and weaknesses of many positional encoding schemes considered.

Dynamic retrieval within the same model

To address this limitation, we introduce a new dynamic retrieval task, described in Section 4 and Appendix C.5. We briefly describe it below:

Input sequences contain two types of needles:

  • Protected tokens (P0, P1), whose values contribute to the answer regardless of distance.
  • Update tokens (U0, U1), whose values contribute only within a window specified by the query Qwindow sizeQ_{\mathrm{window\ size}}.

The model predicts two majority values: one over all protected tokens, and one over the updates within the requested window.

For example, consider the sequence P1 U1 U1 P1 U0 Filler U1 P0 U0 Q5, where Q5 requests the last 5 positions before the query.

The answer is (P1, U0): the full sequence contains more P1 than P0 tokens, while the requested window contains two U0 tokens and one U1.

Changing the query to include the full history changes the update answer to U1, while the protected answer remains P1.

Crucially, the same model handles all query types, and each evaluation sequence is paired with different queries. A short-window query therefore requires excluding distant updates while preserving distant protected information.

Results and controls

In this setup, GAPE consistently improves joint accuracy beyond the training length across all four backbones considered (Figure 2). The learned gates vary with the requested window and token role, as expected (Appendix C.5, Figures 10–11).

Furthermore, reassigning gig_i across queries at inference (short-window queries receive long-window gates and vice versa) harms update accuracy, while disabling ljl_j for protected tokens sharply reduces protected accuracy. Disabling the entire mask reduces both accuracies (Figure 3 and Appendix C.5, Figure 12).

We also independently train three controls:

  1. Replacing gig_i with a shared learned scalar.
  2. Replacing ljl_j with a shared learned scalar.
  3. Using a learned content-independent distance penalty.

All three achieve lower joint accuracy than full GAPE under length extrapolation on both tested backbones, NoPE and RoPE (Appendix C.5, Figure 13). These comparisons directly probe the contribution of token dependence while allowing the remaining model parameters to adapt.

We thank the reviewers again for prompting this revision, which provides a more direct evaluation of GAPE’s proposed mechanism.

AuthorsOpenReview ↗

General Rebuttal

Dear Reviewers, ACs, and PCs,

We thank the reviewers for their careful reading and constructive feedback. We appreciate the recognition of GAPE’s simple and practical implementation [ixZm, ZXHw, AcKr], the controlled experimental design and transparent reporting [ixZm], and the retrieval improvements [ZXHw, AcKr]. In this opening response, we summarize the revisions and additional experiments, and point to our individual responses and the general comment on the synthetic setup for the detailed discussion.

Revisions and Updates

Synthetic Experiments and Gate Contributions

[ixZm, Q1; ZXHw, W1, Q1–Q2; AcKr, W2, Q2–Q3] Dynamic retrieval within the same model. We introduce a new synthetic task in which the same model must preserve protected information independently of distance while retrieving ordinary updates within a query-specified window. This directly tests whether the gates adapt to different queries. GAPE improves accuracy beyond the training length across all four positional backbones considered (Section 4, Figure 2).

[ixZm, Q1, Q3; ZXHw, W1; AcKr, W2, Q2] Isolating suppression and protection. We add inference-time interventions and independently trained controls with shared scalar gates or a learned static distance penalty. Reassigning query gates harms update accuracy, while disabling landmark protection harms protected-memory accuracy (Figures 3 and 12). We also revise the interpretation of the original Needle-far results: disabling the mask has little effect, making the gain consistent with a training-time or parameterisation effect. These original experiments are retained in Appendix C.4.

The 7B Retrofit and Additional Comparisons

[ixZm, Q2, Q4; ZXHw, W2] More seeds and preservation of pretrained RoPE. We expand the retrofit to three adaptation seeds per condition, each using 100M tokens, and report means, standard deviations and confidence intervals. The primary 7B GAPE variant now preserves every pretrained RoPE pair while retaining FlashAttention-2 compatibility. Detailed results appear in Tables 1, 6, and 7, with additional bootstrap intervals in our response to Reviewer ixZm, Q2.

[ixZm, Q3; ZXHw, W1–W2; AcKr, W1, Q1–Q2] Static-decay and hybrid baselines. We add ALiBi and learned static-decay controls at both the 44M and 7B scales. The additional 44M results reported in our response to Reviewer ixZm, Q3, show lower OOD perplexity for GAPE than learned decay. At 7B, the benefit depends on context length: GAPE outperforms learned decay at 16K, while the static controls score slightly higher at 32K. ALiBi has lowest perplexity on the 44M experiments at the cost of heavily hindered retrieval accuracy at 7B scale (-34.97% at 16K).

Related Work

[ixZm, Q3; ZXHw, Q3; AcKr, W1] Content-dependent positional and attention mechanisms. We expand the discussion of CoPE, DAPE, Selective Attention, CABLE, and TRA in Section 2 and Appendix A.

Writing Standards and Clarity

[ixZm, Q5–Q6; AcKr, Q3–Q4] Implementation and experimental reporting. We clarify that both gates are computed before RoPE, tabulate the perplexities underlying Figure 4, and specify the evaluation protocol and meaning of the reported variation. We also report the synthetic models’ parameter counts in our response to Reviewer AcKr, W2 and Q3, and enlarge the fonts and legend in Figure 2.

Section-by-Section Walkthrough of Revisions

The locations below refer to the revised manuscript.

  • Section 2, lines 156–161, and Appendix A, lines 817–911 [ixZm, Q3; ZXHw, Q3; AcKr, W1]: comparisons with related content-dependent mechanisms.
  • Section 3, lines 208–212 [ixZm, Q5]: clarification of the gate inputs and computation order.
  • Section 4, lines 314–374, and Figures 2–3 [ixZm, Q1; ZXHw, W1, Q1–Q2; AcKr, W2, Q2]: the new dynamic retrieval task, results, and gate interventions.
  • Section 5, lines 477–523, and Appendix C.2, Tables 6–7 [ixZm, Q2–Q4; ZXHw, W2; AcKr, Q2]: the expanded 7B experiments, intact-RoPE implementation, and static-decay comparisons.
  • Appendix C.1, lines 1408–1453, and Table 3 [ixZm, Q6]: the language-modeling protocol and numerical perplexity results.
  • Appendix C.4, lines 1787–1791 and 1812–1817 [ixZm, Q1; ZXHw, Q1]: the mask-off check and revised interpretation of the original Needle-far results.
  • Appendix C.5, lines 1838–1889, and Figures 10–13 [ZXHw, W1, Q2; AcKr, W2, Q2]: the synthetic training protocol, gate behavior, interventions, and trained controls.

Request for Feedback

We invite the reviewers to read the revised manuscript and our individual responses. We are particularly grateful for the comments that prompted a more direct evaluation of GAPE’s mechanism and a stronger retrofit comparison. We hope that these additional experiments and clarifications address your concerns, and we look forward to the discussion.

Best regards,

The Authors

AuthorsOpenReview ↗

Discussion Summary

Dear Reviewers, ACs, and PCs,

We sincerely thank the reviewers for their careful reading and constructive feedback. Their suggestions have substantially improved the experimental evidence, clarity, and positioning of our submission. As the discussion comes to a close, we summarize the principal revisions and how they address the concerns raised in the initial reviews.

Discussion Highlights

  • Reviewer ixZm recognized the usefulness of GAPE’s exact rank-2 implementation within standard attention, clean comparisons, and transparent reporting. Their concerns and direct requests motivated a (1) revised evaluation of the synthetic setup, (2) three adaptation seeds per condition with uncertainty estimates, (3) additional static-decay baselines, and (4) a new retrofit that preserves every pretrained RoPE pair. We also (5) clarified that the gates are computed before RoPE and (6) expanded the numerical results and evaluation details. In particular, the reviewer explicitly stated that he would raise his score to 3 should (1)-(4) be addressed.
  • Reviewer ZXHw recognized GAPE’s simple, deployable mechanism, its clean separation of suppression and protection, and the benefits of query-dependent context suppression. Following their suggestions, we introduced direct tests of landmark protection, strengthened the 7B evaluation, and compared YaRN with YaRN+GAPE in the revised synthetic task.
  • Reviewer AcKr acknowledged GAPE’s consistent improvements across positional backbones and ease of integration. Following their suggestions, we added a NoPE full-attention/RoPE local-attention hybrid comparison at 44M, showing modest improvements from adding GAPE at every evaluated context length. Furthermore, the revised synthetic setup isolates the gates’ contributions using learned static-decay and shared-scalar gate controls. We also reported the models’ parameter counts and improved figure readability. Some disagreements remain regarding the interpretation of the hybrid experiment, 44M accuracy, and YaRN at 7B scale. Across these responses, we also expanded the discussion of CoPE, DAPE, Selective Attention, CABLE, and TRA, clarifying how GAPE separates query-dependent suppression from key-specific protection while retaining physical positional distances.

New Experimental Results and Manuscript Improvements

Throughout the rebuttal, we provided additional evidence and clarified the scope of our contribution:

  • Direct evidence for adaptive suppression and landmark protection [ixZm, ZXHw, AcKr]. We introduced a synthetic dynamic retrieval task in which the same model handles different queries over the same sequence, preserving protected information regardless of distance while retrieving updates within a query-specified window. GAPE improves accuracy beyond the training length across all four evaluated positional backbones. Further ablations and mechanistic analyses now isolate the contributions of each of GAPE's gates (Section 4 and Appendix C.5).
  • Expanded 7B evaluation and preservation of pretrained RoPE [ixZm, ZXHw]. We expanded the retrofit to three adaptation seeds, and reported means, standard deviations, and bootstrap confidence intervals. The primary GAPE implementation now preserves every pretrained RoPE pair while retaining FlashAttention-2 compatibility. Mean RULER accuracy improves over LoRA-only, especially at the longest lengths.
  • Learned static-decay and ALiBi controls [ixZm, ZXHw, AcKr]. We added both controls at the 44M and 7B scales, alongside a learned static-distance penalty in the revised synthetic task. GAPE outperforms both baselines, showing the efficacy of token-dependency.
  • Complementarity with existing approaches [ZXHw, AcKr]. We added a NoPE full-attention/RoPE local-attention hybrid at 44M scale. Adding GAPE yields modest perplexity improvements at every evaluated context length. In the revised synthetic task, adding GAPE to YaRN shows greater performance.
  • Positioning relative to related work [ixZm, ZXHw, AcKr]. We expanded the discussion of CoPE, DAPE, Selective Attention, CABLE, and TRA in Section 2 and Appendix A. These comparisons clarify GAPE’s combination of query-dependent suppression and key-specific protection, while retaining physical positional distances and compatibility with standard fused attention.

Impact of GAPE

GAPE separates how strongly a query suppresses distant context from which keys remain protected, providing adaptive context selection that complements existing positional representations. The paper contributes (i) a formal characterization of suppression and landmark protection; (ii) a simple attention augmentation implemented through an exact rank-2 correction, with a retrofit preserving pretrained RoPE; and (iii) empirical evidence across controlled retrieval, 44M language modeling, and 7B retrofits identifying settings where content-dependent gating helps.

Conclusion

We believe the revised manuscript and additional experiments substantially strengthen GAPE’s contribution and address the principal concerns raised in the initial reviews. We respectfully ask the AC to consider these important developments in the final assessment and to evaluate our responses to the reviewers’ concerns directly, since only one reviewer has provided feedback on our rebuttal. Thank you again for your time, consideration, and constructive feedback. Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.