acceptodds
Under review as a conference paper at ICLR 2027

An Analysis of Position Encoding in Hybrid LLMs With Local Mixing and Global NoPE Attention

Abstract

The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position embeddings have long been assumed to be required, recent methods that interleave local mixing layers, such as sliding window attention (SWA) and gated linear attention, while *not* encoding position (NoPE) in global attention layers has recently been shown to be successful at scale. How and why this approach works is not well-understood. In this paper, we develop an explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers. Supported by both theoretical and empirical evidence, our central argument is that SWA and gated linear attention induce a recency bias in the residual stream that propagates to, and is selected by, the global attention logits. Moreover, in contrast to the implicit position encodings found in models with only global NoPE attention, in which positional information arises solely from the causal mask, the recency bias in hybrid models can be maintained across long sequences. In addition to deepening our understanding of how hybrid models encode position, these findings may provide insights for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely.

Primary area:
interpretability and explainable AI
Keywords:
positional encoding, hybrid attention, transformers, recency bias, length generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_warcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how hybrid models with local mixing layers and global NoPE attention can encode positional information without explicit positional encoding in the global attention layers. Through theoretical analysis and empirical validation, the paper argues that SWA or gated linear layers induce a recency bias in the residual stream, which can then be read out by subsequent global NoPE attention as an implicit relative positional encoding.

Critical strengths
  1. The theoretical derivations are generally sound, and the paper clearly states the assumptions and limitations of the analysis.

  2. The experiments are generally well aligned with the theoretical predictions and provide empirical support for the proposed mechanism.

Critical weaknesses

Weakness 1

Section 6 suggests that the proposed mechanism may help explain the length-generalization ability of hybrid models (line 472), but this connection is not directly evaluated. The models are trained on sequences of length 8,192, and the paper does not report evaluations beyond that training length. In particular, the length-scaling experiment in Figure 2 probes different query positions within an 8K-token sequence rather than evaluating length extrapolation. Establishing that SWA induces a local recency bias whose scale does not depend on total sequence length does not, by itself, demonstrate that this mechanism supports effective prediction on longer sequences. Although the authors appropriately present this connection as a hypothesis and acknowledge it as a limitation in Section 6, a direct extrapolation experiment—for example, training at 8K and evaluating at 16K or 32K—would provide important support for this implication. Measuring both predictive performance and the residual/logit recency profiles at these longer lengths would make the connection more convincing.

Weakness 2

The theoretical analysis relies on approximations whose accuracy in trained models is not directly assessed. In particular, Section 4 replaces the content-dependent attention weights αi(d)\alpha_i(d) with their mean profile, ad=E[αi(d)]a_d=\mathbb{E}[\alpha_i(d)]. As Appendix B.1 acknowledges, this approximation omits attention-weight fluctuations and their dependence on the inputs. Section 4.2 demonstrates qualitative trends consistent with the analysis, such as stronger recency bias for smaller windows, but does not quantify how accurately the mean-attention approximation predicts the observed similarity structure. These results therefore support the qualitative predictions without directly validating the approximation underlying the proposed mechanism. A quantitative comparison between the predicted and measured output cross-moments or cosine-similarity profiles would strengthen this part of the analysis.

Weakness 3

The theoretical treatment of MLPs is limited to restrictive initialization assumptions. Appendix C.1 establishes preservation of cosine-similarity ordering for normalized inputs and a wide MLP with random Gaussian weights at initialization, assuming a nonnegative input cosine-similarity profile. As the authors note, this argument does not guarantee preservation by trained MLPs. Appendix C.2 and Figure 10 provide useful empirical evidence that trained MLP blocks largely preserve the incoming cosine gap, but they do not establish a quantitative theoretical account of how learned MLPs transform the relevant lag-dependent structure. Although Section 6 acknowledges this broader limitation, propagation of usable positional information through the intervening modules is central to the proposed mechanism. The theoretical explanation of this step therefore remains limited.

Questions for authors

Question 1

Could the authors explain the non-monotonic residual-similarity profiles for NoPE-in-SWA in the upper panel of Figure 4b? Some curves exhibit a local minimum near the window scale, followed by a rebound; in particular, c(d)−c(4096)c(d)-c(4096) becomes negative in this region for some configurations. This behavior differs from the smoother, approximately monotonic RoPE-in-SWA profiles in Figure 4a. Can the proposed mean-attention analysis, together with the residual-addition terms, reproduce this dip and rebound? Which aspects of the learned attention profile, incoming residual structure, or subsequent transformations are responsible for this behavior?

Question 2

Is the term adΣA⊤a_d\Sigma A^\top in the residual-addition derivation in Appendix C.1 a typo, and should it instead be adAΣa_dA\Sigma?

3 comments
AuthorsOpenReview ↗

Response to Reviewer warc

Weakness 1

You are correct to point this out, and we plan to compare validation loss across window sizes beyond the 8K training length, at sequence lengths of 16K and 32K. We will report the results here during the discussion period. As for probing different query positions in Figure 2, we intend this as a measure of stationarity or consistent decay within the evaluated 8K context rather than a measure of length extrapolation, but we can make this clearer in the text.

Weakness 2

This is a fair concern, particularly as it relates to the modulation of content-dependent attention about its mean profile. We plan to assess the accuracy of this approximation in trained models and clarify its limitations.

Weakness 3

Our theoretical analysis of MLPs is limited to initialization, as MLPs are highly expressive and therefore difficult to mathematically analyze during and after training. A more general theoretical treatment of trained MLPs remains an important direction for future work. Though you are correct that measuring the effect of intervening modules is central, we note that the empirical results show that a recency gap is retained (though dampened) following the trained MLP modules, as shown in Figure 10.

Question 1

We first note that the NoPE-in-SWA trends are broadly less clear than for RoPE-in-SWA, and some of the larger-window models exhibit training instability, as seen in Figure 15d. This suggests that, without RoPE to provide local positional information, the structural recency bias provided by SWA itself may be more important for learnability. However, it is unclear whether the non-monotonic trends you observed are related to this instability or to some particular pattern in the cross-moment structure. We plan to examine the raw cosine profiles and residual cross-moment terms in the affected models to better understand this behavior.

Question 2

You are correct that this is a typo: the term should be adAΣa_d A\Sigma. Thank you for catching it! We will correct this in the next version of the paper.

Thank you for the response which has largely addressed my concerns. Therefore I will maintain my positive rating.

AuthorsOpenReview ↗

Follow-up to reviewer warc

Thank you for the follow-up. We’ve now completed the two evaluations described below.

Weakness 1

We scored the same prediction targets in 16 ProLong documents as context increased from 4K to 32K (Figure S5, 350M and Figure S6, 120M). At both scales, global RoPE loss rose sharply beyond the 8K training length, whereas hybrid NoPE models generally matched global RoPE at 8K while retaining much of global NoPE’s stronger extrapolation. Smaller windows tended to yield lower loss for NoPE-in-SWA, whereas RoPE-in-SWA showed no consistent ordering by window size. Please see our response to reviewer MbkX W4 for the full comparison.

Weakness 2

To test the mean-attention approximation in trained models, we replaced the content-dependent attention weights in each trained SWA layer with a mean profile estimated on 17 calibration chapters from TextbookChapters. We retained the layer’s native inputs and values, then compared measured and replayed cosine-similarity curves on 34 other chapters at 8K. We visualize the replayed and measured cosine profiles for individual early, middle, and late layers at window 128, as well as the distribution of errors across all seven windows: 120M with local RoPE (Figure S7) or NoPE (Figure S8), and 350M with local RoPE (Figure S9) or NoPE (Figure S10). These demonstrate that the mean-attention profile’s outgoing residual-stream cosine decay is typically a good approximation to that corresponding to true content-dependent attention. Across 252 layer–window combinations, the median of the per-layer mean absolute errors was 0.020 after subtracting each curve’s value at lag 4,096.

Reviewer_MbkXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Explicit positional embeddings have traditionally been considered essential for language models, but recent approaches have increasingly adopted NoPE. However, why these approaches work remains insufficiently understood. To address this question, the authors investigate hybrid models combining NoPE with Sliding Window Attention (SWA) and analyze the underlying mechanism. Specifically, they explain how implicit relative positional information can emerge even without explicit positional encoding in global attention layers, through distance-dependent correlations in the residual stream induced by local mixing in SWA layers and the subsequent extraction of this information by query/key projections.

Critical strengths
  • This paper investigates how relative positional information emerges in hybrid LLMs that combine local mixing layers, such as Sliding Window Attention (SWA), with global NoPE attention, despite the absence of explicit positional encoding in the global attention layers. The study focuses on an architecture increasingly adopted in recent LLMs.

  • The paper provides both theoretical and empirical analyses of the mechanism by which local token mixing through SWA induces distance-dependent similarities in the residual stream, which are subsequently reflected in the attention logits of global NoPE layers.

Critical weaknesses
  • The central claim of this paper is that local token mixing through SWA induces a recency bias in the residual stream, which is then exploited by global NoPE attention to achieve implicit relative positional encoding. However, relative positional encoding based on recency bias is not a new concept and has already been employed in existing methods such as ALiBi. Although ALiBi and SWA are different operations, they share the common property of favoring nearby tokens. Therefore, it remains unclear what novel properties the recency bias observed in this paper has compared to existing positional encoding methods, or what advantages it provides. While ALiBi is mentioned in Section 2, the paper lacks a sufficient comparative analysis of the functional differences between the explicit recency bias used in existing methods and the implicit recency bias investigated in this work.

  • This paper considers the distance-dependent changes in residual stream cosine similarity and attention logits as evidence of implicit relative positional encoding. However, the existence of recency bias does not necessarily imply that the model can distinguish or effectively utilize relative positional information. In particular, Figures 2 and 4 show that cosine similarity tends to become nearly constant beyond a certain distance. In such cases, it remains unclear to what extent the model can distinguish relative distances between tokens that are far apart. Therefore, rather than concluding that relative positional encoding has been acquired solely from the observation of recency bias, I believe additional experiments are needed to verify whether the model can actually distinguish relative positions.

  • Figure 4 shows that reducing the SWA window size strengthens the recency bias in the residual stream. Furthermore, Figures 9 and 15 report that models with smaller window sizes tend to achieve lower validation loss. However, changing the window size affects not only the strength of the recency bias but also the attention receptive field and the characteristics of information mixing. Consequently, the current experiments do not distinguish whether the observed performance improvements are caused by stronger recency bias or by other effects associated with the locality of SWA. To demonstrate that the central mechanism proposed in this paper actually contributes to model performance, I believe additional experiments, such as intervention studies that directly manipulate the strength of recency bias, are necessary.

  • In the Abstract and Discussion, the authors suggest that the findings of this paper may provide insights into positional encoding mechanisms that enable extrapolation to longer sequence lengths. However, the experiments mainly analyze the distance dependence of residual stream representations and attention logits in models trained with a sequence length of 8192. The paper does not sufficiently evaluate language modeling performance or long-context task performance on input sequences exceeding the training length. The fact that the local recency bias induced by SWA is independent of the overall sequence length does not directly guarantee length extrapolation performance at the model level. In particular, given that the recency bias becomes nearly constant at large distances, it remains unclear to what extent this mechanism alone can handle long-range positional dependencies. Therefore, I believe that additional experiments evaluating perplexity (PPL) and performance on long-context benchmarks beyond the training sequence length are necessary.

  • The main experiments in this paper focus on analyzing the distance dependence of residual stream cosine similarity and attention logits. Although these analyses are useful for investigating the existence of the proposed mechanism, they do not clearly establish how much this mechanism contributes to actual language understanding capabilities or the ability to handle long-range dependencies. Furthermore, the main analyses are conducted on models with 120M and 350M parameters. Additional experiments would be desirable to determine whether similar findings hold for the much larger hybrid LLMs used in recent applications.

Questions for authors

See the Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MbkX (1/2: Weaknesses 1–2)

Weakness 1

We agree that recency bias as a positional inductive bias is established, including in ALiBi. However, the goal of our paper is not to try to propose a new position encoding scheme or investigate which position encoding scheme is best. Rather, we are aiming to undertake a scientific investigation of the puzzling fact that models with hybrid NoPE perform well empirically (e.g., Kimi K3 and Nemotron 3.5 Super) even though they seemingly possess no mechanism for encoding position in the global NoPE attention layers.

Our contribution, then, is a mechanistic explanation of how a recency bias arises and becomes usable in the first place in hybrid architectures whose global attention layers have no explicit position encoding. The functional distinction concerns where the positional signal originates and how it enters attention. ALiBi directly adds a prescribed distance-dependent penalty to attention logits. In our setting, local mixing creates distance-dependent correlations in the residual stream, which learned query/key projections can read out in subsequent global NoPE layers. Our analysis investigates the generation, propagation, and learned readout of this implicit signal. We do not demonstrate superiority over ALiBi or any other position encoding method in this study, and that is not our aim. We will clarify this distinction in the related work and discussion.

Weakness 2

First, we observed in the background section that existing relative PEs can induce a recency bias in the attention logits, often changing the mean logit profile smoothly with distance. Thus, our analysis of residual similarity as an implicit relative positional bias asks whether the overall similarity cross-moment has structure that decays with distance (as described in Section 4), and whether that structure is picked up by the attention logits (as described in Section 5). We see empirically that this is the case (as shown in Figures 3-5).

Second, it is important to note that the analysis of the immediate SWA output at initialization presented in Figure 2 differs from the analysis of trained models across depth presented in Figures 4 and 7. In particular, the flat recency profile beyond the sliding window in Figure 2 pertains to the immediate output of SWA at initialization; indeed, here SWA introduces a pronounced increase in cosine similarity within the window and no effect beyond. However, this need not result in a flat recency signal outside the window in the following layers. Global attention layers can distribute this concentrated recency bias beyond the window, as allowed by the general autocorrelation argument of Section 4.1. The recency bias can therefore build both in its short-distance strength and its long-range extension. Consistent with this, Figure 7 shows a decay in the recency bias beyond the sliding window, which remains visible within the measured range (½ the total training context length, to avoid confounds with BOS) especially in later layers. We will incorporate this argument more fully into Section 5.1.

We plan to pilot probes to test whether relative distance can be decoded from the residual representations of our small pretrained models, building on related probing approaches in prior work (e.g., the absolute-position probes in SWAN-GPT).

AuthorsOpenReview ↗

Response to Reviewer MbkX (2/2: Weaknesses 3–5)

Weakness 3

This is a fair point, and for this reason we refer to the validation loss (mean token cross-entropy, in nats/token) evidence in Figures 9 and 15 as offering preliminary, correlational evidence of a connection between window size, recency bias strength, and language modeling. Though we would like to more directly probe the effect of recency bias on performance, the difficulty in doing so lies in the fact that intervening directly upon the attention profile can also disrupt content-dependent attention. For instance, we can try an intervention which subtracts the mean distance-dependent logit profile from outside the model; however, this risks causing out-of-distribution behavior for the model and thus it is hard to determine whether a decline in performance is due to the intervention on the recency bias or the side effects of that intervention. We plan to pilot this intervention at different strengths and compare it with controls that subtract different random logit profiles of comparable energy, to assess whether its effects can be interpreted separately from broader disruption of attention.

Weakness 4

You are correct to point out that we do not directly evaluate length extrapolation in these models or the effect of window size on it, given the potential connection to the length extrapolation capabilities of hybrid NoPE architectures reported in the literature. We plan to compare validation loss across window sizes beyond the 8K training length, at sequence lengths of 16K and 32K. We will report the results here during the discussion period and include them in the updated version of the paper.

Weakness 5

You are correct that our study, which is limited to pretrained models with 120M and 350M parameters, cannot as firmly establish a connection to long-context reasoning and retrieval tasks. Experiments on the much larger LLMs used in recent applications are beyond the compute budget available for this project. Though we could use open source models, it is difficult to make controlled comparisons given the variation in their architectures, proprietary datasets, and training methods beyond window size and interleave ratio. We cite the abundance of hybrid NoPE architectures and their empirical performance as motivation for investigating our proposed mechanism’s role in language modeling.

AuthorsOpenReview ↗

Follow-up to reviewer MbkX (1/2, Weakness 2)

Weakness 2

To follow up on your question of whether such models can distinguish large relative distances, we fit a bilinear probe to predict token separation from pairs of residual vectors at different layers. Separately for each setting (120M/350M, RoPE/NoPE-in-SWA), we compare window sizes (ranging from 64 to 4096 tokens) and global attention-only (with RoPE and NoPE) baselines. We fit the probe on token pairs from 8K contexts and evaluated it, with its weights frozen, on pairs from 32K contexts in held-out source documents. The corpus comprises 128 documents in TextbookChapters. At each layer, we fit an unrestricted bilinear form with an intercept to two unit-normalized residual vectors, using 1,024 pairs for fitting and 512 separate pairs for ridge-penalty selection.

Consistent with the progressive development of the recency structure across depth (see Figures 4 and 7 in the original submission), distance decodability arises in the middle-to-late model layers of the RoPE-in-SWA hybrids, consistently across window sizes and extending beyond the training length across the measured lag bins (though underestimating longer distances). This pattern is consistent with the observation that these window-size variants all develop a similarly strong recency bias through depth (as shown in Figure 4). In contrast, the global-only probes show much less separation between long distances. For reference, see Figure S1 and Figure S3.

The NoPE-in-SWA hybrids are more suggestive of the connection between distance decodability and recency bias strength. Figure S2 and Figure S4 show that the shorter-window variants separate from the longer-window ones in how well their probes distinguish longer distances. Referring to Figures 4 and 15 in the original submission, the models for which distance may be decoded (as before, in the middle-to-late layers) are also those that sustain a strong recency bias through depth. These results show that relative distance is decodable in some hybrid NoPE models beyond the training length and suggest a connection to the strength of the recency bias developed. We will include these findings in the text.

AuthorsOpenReview ↗

Follow-up to reviewer MbkX (2/2, Weaknesses 3–4)

Weakness 3

As a preliminary test of the effect of recency bias on language-model loss, we partly removed the mean distance-dependent logit profiles for SWA and global attention layers. In particular, for a 120M RoPE-in-SWA model (window 128), we estimated each head’s mean distance-dependent logit profile on eight 8K documents in TextbookChapters, then subtracted a fraction of that profile from the pre-softmax logits in either the SWA layers or the global NoPE layers. The table gives the mean change in negative log-likelihood relative to native attention (2.2 nats/token), with native and perturbed losses paired on eight separate held-out chapters (positive values mean higher loss, and brackets provide exploratory 95% bootstrap intervals from resampling chapters).

Subtraction strengthSWA layers onlyGlobal layers only
0.1+0.022 [0.016, 0.030]+0.000091 [−0.00074, 0.00094]
0.5+2.2 [2.1, 2.4]+0.0079 [0.0047, 0.011]

This hybrid NoPE model was much more sensitive to perturbing the local layers than global layers: at strength 0.1, the loss increase was already measurable for SWA subtraction, while the global-only estimate was around zero; at 0.5, SWA subtraction increased loss by 2.2 nats/token, compared with 0.0079 for global subtraction.

One possible explanation follows from our proposed mechanism. First, subtracting the SWA logit profile may weaken the residual-stream recency bias that propagates through depth, and consequently the positional information available to the global attention layers. This occurs in conjunction with any disruption to local retrieval within SWA. In contrast, intervening just on the global layers may better preserve the SWA layers, as their positional information is derived from RoPE. Furthermore, though reducing the profile of a global attention layer could weaken the positional signal already present in its logits, it could remain partly accessible. Due to the preliminary status of these findings and the difficulty of isolating positional versus content-dependent information, we will not include them in the text.

Weakness 4

For each setting (120M/350M, RoPE/NoPE-in-SWA), we evaluated the length extrapolation performance of hybrid models with different window sizes in comparison to global-only RoPE/NoPE baselines. In particular, to keep the prediction targets fixed, we scored a fixed window of tokens (the final 1,024 tokens of each of 16 length-32K ProLong sources) while varying the total input length from 4K to 32K.

Figure S5 shows the 350M results, and Figure S6 shows the 120M results. The top panels show DCLM validation loss during training, and the bottom panels show length extrapolation on the ProLong documents. The broad trends reflect prior work. In both sizes, global RoPE models deteriorate shortly beyond the training length, global NoPE models achieve stronger length extrapolation, and hybrid NoPE models mostly retain this strong length extrapolation while (particularly with RoPE-in-SWA) matching global RoPE at the training length (whereas global NoPE is comparatively worse). However, as was the case with the loss results in the original submission (see Figures 9 and 15), the window-size trends depend on the position encoding within the SWA window. For NoPE-in-SWA, in which the recency bias and resultant positional information arise principally from the structural effects of the SWA window (and not from RoPE within the window), most variants maintain consistent loss far beyond the training length, and variants with smaller windows often achieve lower loss, consistent with their stronger recency bias (as shown in Figure 4). For RoPE-in-SWA models, we do not observe a consistent ordering by window size as in the original text (see Figure 9), but the hybrid NoPE models extrapolate comfortably within the measured lengths. We will include these findings in the text.

Reviewer_aSXSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a theoretical perspective of why the hybrid attention with SWA/KDA mixed with NoPE normal attention layer can still encode the positional information. The main argument is the SWA learns a local recency bias which is later fetched by global attention layer (even without position encoding), forming an implicit relative position encoding. There are also experiments done on real dataset that verifies this hypothesis.

Critical strengths

Overall this is an interesting analysis that tells some new knowledge on why the hybrid attention, with NoPE can also encode the relative position information. The practical recipe is now more and more widely used in recent LLMs so this analysis shed lights on the intrinsic mechanism of this new recipe.

Critical weaknesses
  1. The paper mainly demonstrates the existence of recency bias, but it is not clear to me whether such a bias is sufficient to act as relative position encoding. For example, the SWA-induced similarity becomes constant beyond the window size under the simplified analysis. How can the model distinguish positions beyond this range?

  2. It's still better to draw practical implications from the theoretical analysis, like what can be the further improvement for this SWA + NoPE global attention layer, inspired from the theory.

Questions for authors

I feel although seemingly unrelated at all, the theory behind this paper is connected to the early circuit theory:

  1. A Mathematical Framework for Transformer Circuits
  2. In-context Learning and Induction Heads

For example, the mechanism of global attention layer reading the local attention layer's recency information looks similar to the Q-/K- composition analyzed in these circuit works. I would like to understand more the connection between the two pieces of works from the authors (of course I might be wrong)

It's not to say these works must be cited and rigorously analyzed in the paper.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aSXS

Weakness 1

It’s important to note that Figure 2 concerns only the immediate first-layer SWA output at initialization. Though initially confined to the window, this locally induced recency bias can then be distributed over longer distances by global attention layers, as allowed by the general autocorrelation argument of Section 4.1. Indeed, Figures 4 and 7 show distance-dependent residual structure beyond the window in trained models, over measured lags up to 4096 tokens. Learned query/key projections can then read out this structure in global NoPE logits (Section 5). For an extended explanation, please see our response to reviewer MbkX, Weakness 2.

Weakness 2

Our analysis does identify that window size should be considered not only for its computational role, but also as a positional inductive bias. However, we don’t attempt to establish an optimal configuration. Future work might establish a more quantitative relationship between (a) the attention configuration of a hybrid model, particularly in terms of its window sizes and local:global interleave ratio; (b) its performance on long-context reasoning and retrieval tasks; and (c) its ability to extrapolate to longer sequences, i.e., maintain performance on such tasks beyond its training sequence length. We will add brief commentary on this implication to the discussion section. However, the core purpose of our paper is to provide an explanation for how position encoding in NoPE layers works at all, not necessarily to derive a practically optimal configuration.

Question for authors

Thank you for pointing out this connection. We agree that our mechanism is related to Q-/K-composition: local attention modifies the residual stream, which subsequent global attention reads through its query/key projections. Whereas induction circuits use this composition for prefix matching and copying, our analysis characterizes how overlapping local mixing induces distance-dependent residual cross-moments that are read out as a recency bias in global NoPE logits. We will briefly discuss this connection and cite both works in our updated version of the paper.

AuthorsOpenReview ↗

Follow-up to reviewer aSXS

Weakness 1

To follow up on your question of whether such models can distinguish relative distances beyond the window, we fit a bilinear probe to predict token separation from pairs of residual vectors at different layers. Separately for each setting (120M/350M, RoPE/NoPE-in-SWA), we compare window sizes (ranging from 64 to 4096 tokens) and global attention-only (with RoPE and NoPE) baselines. We fit each probe on 1,024 token pairs in 8K contexts and evaluated the frozen probe on 32K contexts from held-out source documents. The pairs came from 128 documents in TextbookChapters.

These results indicate that relative position in hybrid NoPE models is decodable for distances beyond not only the SWA window, but also the training length. See S1, S2, S3, S4. Furthermore, the correspondence between distance decodability and recency bias strength among window sizes and RoPE/NoPE-in-SWA suggests that recency bias may influence the accessibility of relative positional information. For further commentary on the patterns and trends across settings, please see the response to reviewer MbkX, W2. We will include these findings in the text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.