- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how hybrid models with local mixing layers and global NoPE attention can encode positional information without explicit positional encoding in the global attention layers. Through theoretical analysis and empirical validation, the paper argues that SWA or gated linear layers induce a recency bias in the residual stream, which can then be read out by subsequent global NoPE attention as an implicit relative positional encoding.
Critical strengths
-
The theoretical derivations are generally sound, and the paper clearly states the assumptions and limitations of the analysis.
-
The experiments are generally well aligned with the theoretical predictions and provide empirical support for the proposed mechanism.
Critical weaknesses
Weakness 1
Section 6 suggests that the proposed mechanism may help explain the length-generalization ability of hybrid models (line 472), but this connection is not directly evaluated. The models are trained on sequences of length 8,192, and the paper does not report evaluations beyond that training length. In particular, the length-scaling experiment in Figure 2 probes different query positions within an 8K-token sequence rather than evaluating length extrapolation. Establishing that SWA induces a local recency bias whose scale does not depend on total sequence length does not, by itself, demonstrate that this mechanism supports effective prediction on longer sequences. Although the authors appropriately present this connection as a hypothesis and acknowledge it as a limitation in Section 6, a direct extrapolation experiment—for example, training at 8K and evaluating at 16K or 32K—would provide important support for this implication. Measuring both predictive performance and the residual/logit recency profiles at these longer lengths would make the connection more convincing.
Weakness 2
The theoretical analysis relies on approximations whose accuracy in trained models is not directly assessed. In particular, Section 4 replaces the content-dependent attention weights with their mean profile, . As Appendix B.1 acknowledges, this approximation omits attention-weight fluctuations and their dependence on the inputs. Section 4.2 demonstrates qualitative trends consistent with the analysis, such as stronger recency bias for smaller windows, but does not quantify how accurately the mean-attention approximation predicts the observed similarity structure. These results therefore support the qualitative predictions without directly validating the approximation underlying the proposed mechanism. A quantitative comparison between the predicted and measured output cross-moments or cosine-similarity profiles would strengthen this part of the analysis.
Weakness 3
The theoretical treatment of MLPs is limited to restrictive initialization assumptions. Appendix C.1 establishes preservation of cosine-similarity ordering for normalized inputs and a wide MLP with random Gaussian weights at initialization, assuming a nonnegative input cosine-similarity profile. As the authors note, this argument does not guarantee preservation by trained MLPs. Appendix C.2 and Figure 10 provide useful empirical evidence that trained MLP blocks largely preserve the incoming cosine gap, but they do not establish a quantitative theoretical account of how learned MLPs transform the relevant lag-dependent structure. Although Section 6 acknowledges this broader limitation, propagation of usable positional information through the intervening modules is central to the proposed mechanism. The theoretical explanation of this step therefore remains limited.
Questions for authors
Question 1
Could the authors explain the non-monotonic residual-similarity profiles for NoPE-in-SWA in the upper panel of Figure 4b? Some curves exhibit a local minimum near the window scale, followed by a rebound; in particular, becomes negative in this region for some configurations. This behavior differs from the smoother, approximately monotonic RoPE-in-SWA profiles in Figure 4a. Can the proposed mean-attention analysis, together with the residual-addition terms, reproduce this dip and rebound? Which aspects of the learned attention profile, incoming residual structure, or subsequent transformations are responsible for this behavior?
Question 2
Is the term in the residual-addition derivation in Appendix C.1 a typo, and should it instead be ?
3 comments
Thank you for the response which has largely addressed my concerns. Therefore I will maintain my positive rating.
Follow-up to reviewer warc
Thank you for the follow-up. We’ve now completed the two evaluations described below.
Weakness 1
We scored the same prediction targets in 16 ProLong documents as context increased from 4K to 32K (Figure S5, 350M and Figure S6, 120M). At both scales, global RoPE loss rose sharply beyond the 8K training length, whereas hybrid NoPE models generally matched global RoPE at 8K while retaining much of global NoPE’s stronger extrapolation. Smaller windows tended to yield lower loss for NoPE-in-SWA, whereas RoPE-in-SWA showed no consistent ordering by window size. Please see our response to reviewer MbkX W4 for the full comparison.
Weakness 2
To test the mean-attention approximation in trained models, we replaced the content-dependent attention weights in each trained SWA layer with a mean profile estimated on 17 calibration chapters from TextbookChapters. We retained the layer’s native inputs and values, then compared measured and replayed cosine-similarity curves on 34 other chapters at 8K. We visualize the replayed and measured cosine profiles for individual early, middle, and late layers at window 128, as well as the distribution of errors across all seven windows: 120M with local RoPE (Figure S7) or NoPE (Figure S8), and 350M with local RoPE (Figure S9) or NoPE (Figure S10). These demonstrate that the mean-attention profile’s outgoing residual-stream cosine decay is typically a good approximation to that corresponding to true content-dependent attention. Across 252 layer–window combinations, the median of the per-layer mean absolute errors was 0.020 after subtracting each curve’s value at lag 4,096.
Response to Reviewer warc
Weakness 1
You are correct to point this out, and we plan to compare validation loss across window sizes beyond the 8K training length, at sequence lengths of 16K and 32K. We will report the results here during the discussion period. As for probing different query positions in Figure 2, we intend this as a measure of stationarity or consistent decay within the evaluated 8K context rather than a measure of length extrapolation, but we can make this clearer in the text.
Weakness 2
This is a fair concern, particularly as it relates to the modulation of content-dependent attention about its mean profile. We plan to assess the accuracy of this approximation in trained models and clarify its limitations.
Weakness 3
Our theoretical analysis of MLPs is limited to initialization, as MLPs are highly expressive and therefore difficult to mathematically analyze during and after training. A more general theoretical treatment of trained MLPs remains an important direction for future work. Though you are correct that measuring the effect of intervening modules is central, we note that the empirical results show that a recency gap is retained (though dampened) following the trained MLP modules, as shown in Figure 10.
Question 1
We first note that the NoPE-in-SWA trends are broadly less clear than for RoPE-in-SWA, and some of the larger-window models exhibit training instability, as seen in Figure 15d. This suggests that, without RoPE to provide local positional information, the structural recency bias provided by SWA itself may be more important for learnability. However, it is unclear whether the non-monotonic trends you observed are related to this instability or to some particular pattern in the cross-moment structure. We plan to examine the raw cosine profiles and residual cross-moment terms in the affected models to better understand this behavior.
Question 2
You are correct that this is a typo: the term should be adAΣ. Thank you for catching it! We will correct this in the next version of the paper.