acceptodds
Under review as a conference paper at ICLR 2027

Do Transformers Have the Ability for Periodicity Generalization?

Abstract

Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model’s ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to analyze why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative dataset with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Out-of-distribution, Generalization, Periodicity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dp5rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the periodicity generalization ability of transformers. In controlled experiments, the paper trains transformers and other language models to learn periodic functions and their compositions. The paper distinguishes two kinds of generalization modes: (i) Hollow, where the model can generalize out-of-distribution through interpolating trained examples, and (ii) Extrapolation, where the test examples lie outside the support of the training distribution.

The results compare various architectures of language models, and find that they mostly generalize via the hollow mode, while struggle to extrapolate.

Critical strengths
  1. Clear motivation and theory-inspired synthetic task. The main task Coper is a valuable task to study compositional generalization of language models. In particular, the results can be visually inspected as in Figure 1, thus providing a good way of understanding the generalization ability of language models.

  2. Informative generalization notions. I like the classification of OOD generalization into interpolative generalization and extrapolative generalization. The distinction is helpful to understand the benefits and limitation of current model architectures and training algorithms.

Critical weaknesses
  1. The main issue is training is not well implemented. Model training seems to be insufficient or incomplete, which raises the concern that models may be not correctly trained. For example, in Figure 5, the loss curves are still decreasing, which means that training is not complete. The authors should consider increasing the training steps to a much large number.

Since the main claim of the paper is about generalization, not optimization speed, training is only complete if the ID loss decreases to a sufficiently low level. In Table 1, however, the three models other than Transformer are not sufficiently trained. Giving the confounding effects of incomplete training, the claims about OOD generalization are not well supported.

If the authors want to make a claim about optimization (like difficulty in jumping out of local minima or saddle points), the experiments should be designed very differently, and the paper would need to sweep across hyperparameters and control for training steps / time.

  1. The scale of experiments is also small. For example, the number of layers in Table 7 is no larger than 7. A better model scaling experiment would need to sweep the model size to a wider range. The small scale would prevent the paper from convincingly draw conclusions about model capacity of memorization. In addition, it would informative to report the variability of results across several independent runs, or analyze the robustness to key model hyperparameters.
Questions for authors

See my comments in "Critical Weaknesses"

2 comments

Weakness 1 & 2 (Scaling, several independent runs and hyperparameters)

We thank the reviewer for this point and re-ran the main experiment so that every model is trained until its ID loss is below 0.1; epochs differ across models, and results are mean ± std over three seeds (which also reports the number of training tokens).

  • Training to convergence. The original learning rate (1e-5) kept Transformer-type models on a long loss plateau. After a learning-rate search on held-out training data (no OOD data used; Table R2), Transformer and FANformer reach held-out ID loss ≤ 0.004 and ID accuracy ≥ 99.9% within ~20 epochs (100–170M tokens). Mamba reaches a training loss of 0.071.

  • The OOD failure is not caused by under-training. With ID fully fitted, the 10-layer Transformer reaches 29.5 ± 3.9 on Hollow and 20.5 ± 1.7 on Extrapolation, and FANformer 30.7 ± 10.1 and 17.5 ± 3.4, while the OOD loss rises to 7–8.

  • Larger models. We scaled to 10 layers (48–54M trainable parameters) and to 12 layers with hidden size 2048 (~0.3B: Transformer 302M, FANformer 340M). The 0.3B models also fit ID perfectly (100.0 / 99.9) but generalize worse (Hollow 18.3 / 17.6, Extrapolation 17.0 / 15.0). More capacity speeds up ID fitting but does not improve periodicity generalization.

ModelLayers × hiddenTrainable paramsnStepsInput tokensTrain lossHeld-out ID lossOOD lossID accHollow accExtrap. accAvg.
Transformer10 × 89648.3M37168 ± 1129123.3 ± 19.4M0.015 ± 0.0040.004 ± 0.0047.91 ± 1.3699.9 ± 0.129.5 ± 3.920.5 ± 1.750.0 ± 1.8
FANformer10 × 89654.3M38472 ± 1129145.7 ± 19.4M0.020 ± 0.0040.004 ± 0.0007.29 ± 1.0799.9 ± 0.030.7 ± 10.117.5 ± 3.449.3 ± 4.5
Mamba10 × 89651.2M327370 ± 8959470.7 ± 154.1M0.071 ± 0.0134.519 ± 0.8394.88 ± 0.6658.9 ± 5.851.9 ± 5.746.4 ± 2.052.4 ± 4.3
Transformer12 × 2048302.3M121505369.8M0.0080.00011.81100.018.317.045.1
FANformer12 × 2048340.0M15865100.9M0.0260.00210.0499.917.615.044.2
Modellrwarmupgrad clipwdthe best held-out ID loss(epoch)final train losschosen
FANformer0.0001000.011.156(20)1.167
FANformer0.0003100010.010.004(20)0.020✓
FANformer0.001100010.010.796(20)0.895
FANformer1e-05000.011.948(20)2.005
Transformer0.0001000.010.022(20)0.176
Transformer0.0003100010.010.000(15)0.014✓
Transformer0.001100010.011.157(20)1.162
Transformer1e-05000.011.882(20)1.938
Mamba0.0001000.011.287(15)0.144✓
Mamba0.0003000.011.230(45)0.963
Mamba0.0003100010.011.187(60)1.166
Mamba0.001000.012.183(60)2.184
Mamba0.001100010.012.291(60)2.290
Mamba0.001100010.12.303(40)2.303
Mamba0.002100010.012.303(40)2.303
Mamba1e-05000.011.786(40)1.481

We will include these additional experiments and results in the revised manuscript.

If you have any further questions, please do not hesitate to contact us.

Thank you again for taking the time to review our responses. As the discussion period is coming to an end, we wanted to check if there are any remaining questions or concerns we can clarify. Please feel free to let us know.

Reviewer_oyRoOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

A major challenge facing LLMs is the difficulty of out-of-distribution (OOD) generalization as compared to humans. In this paper, the authors use prediction of periodic sequences as a testbed to explore OOD generalization. In particular, the authors construct a synthetic dataset called Coper, which is composed of sequences created by the sum of two modular sequences of different periodicities (all mod 10). The authors find that transformer models are able to learn these sequences in distribution (when the two periods are in the training data), but are generally unable to learn predict the next number in OOD sequences (both when the periods are "within" the trained data and "outside" the trained data in terms of the two periods). Finally, the authors argue that this failure to generalize can be understood using concepts from abstract algebra.

Critical strengths

I believe that the Roper dataset and training is a nice contribution to the field. In particular, I did not realize that generalizing from such sequences would be so difficult, even for models such as FANformer, which were developed to learn periodic sequences. I also appreciated the training curves across the different architectures; do the authors understand why the transformer and FANformer architectures have OOD curves that grow, as compared to the other two that display very similar loss curves for ID and OOD sequences? This seemed quite interesting to me.

I therefore quite appreciated the empirical part of the paper, but would have appreciated this part coming earlier in the paper, together with a more thorough concrete explanation in the main text of the task and what prior work had established (can transformers and FANformer generalize when there is just one period in the sequence?).

Critical weaknesses

I found it difficult to tell whether the theoretical explanation actually explained the authors' experiments. Some of this was stylistic, as the group-theory discussion came before I had any idea what data the authors were going to explain. As far as I can tell, the modular addition that the authors study empirically seems to be shift invariant. I therefore don't see why the authors argue that shift symmetry prevents the learning of these sequences. It is possible that I have misunderstood the argument that the authors are making, but as far as I could tell the analysis does not explain the origin of generalization failure that they observe.

Questions for authors

As indicated previously, I didn't find the theoretical analysis convincing, but if the authors can explain to me why I am wrong / confused I am open to changing my assessment. However, I also feel that the paper is written in a way that makes it very difficult for readers to follow, so that is still a significant concern for me. If the authors actually understand the origin of the failure to generalize, then is there an architectural modification that would allow for generalization for this class of sequences?

4 comments

Weakness 1 & Question 1 (Theoretical Claim)

The modular addition in Eq. (7), described in Lines 316–317 of Section 4, is constructed by composing two group transformations. Our theory formally defines three types of periodicity. Periodicity is not limited to simple repetition along the sequence dimension, but also includes the repeated application of underlying rules, which we call Rule Periodicity (Appendix B and B.1).

Our paper proves, both theoretically and empirically, that Shift-Equivariant Sequence Periodicity can generalize to OOD settings, while Non-Equivariant Sequence Periodicity and Rule Periodicity cannot. Our group-theoretic framework unifies these types of periodicity and further extends the analysis to Composite Periodicity.

We further characterize the boundary of Transformer modeling from a group-theoretic perspective, specifying which types of periodicity Transformers with RoPE can generalize, including a single period. The following table summarizes the theoretical boundary (see also Appendix B):

Periodicity TypeGroup-theoretic FormulationOOD Generalization of Transformer with RoPEReference
Shift-equivariant sequence periodicitygT∘f=fg^T \circ f= f, e.g., f(t+T)=f(t)f(t+T)=f(t)✓ YesEq. (3), Fig. 2, and Fig. 8
Non-equivariant sequence periodicitygT∘f≠fg^T \circ f\neq f, e.g., f(t+T)=2f(t)f(t+T)=2f(t)✗ NoFig. 2 and Appendix B.4 (Fig. 10)
Rule periodicityR(g∘f)=g∘R(f)R(g\circ f)=g\circ R(f)✗ NoSection 3.3.2 and Appendix B
Composite periodicityG=GR×GM,  (τi,σj)∘(Rk,Mk)=(Rk−i,Mk(⋅+j))G=G_R\times G_M,\;(\tau^i,\sigma^j)\circ(R_k,M_k)=\left(R_{k-i},M_k(\cdot+j)\right)✗ NoEqs. (8)–(9), Table 1, and Fig. 1

Within the group-theoretic framework, we provide concrete examples of sequence repetition and rule repetition, including element-wise addition, modulo operations, and convolution (Appendix C, F). However, these repeated rules do not imply shift invariance of the resulting sequences. For example, in 123+456=579123+456=579 (Lines 189-193), the result 579 is not shift-invariant with respect to either input sequence. From a group-theoretic perspective, the shift invariance of modular addition and convolution lies in the underlying rules. Our proof shows that Transformers with RoPE can generalize shift-equivariant sequences periodicity, but cannot generalize rules to OOD settings.

Main Contributions.

Our main contribution is not the structured holdout strategy itself, but a group-theoretic framework for understanding periodicity and rule reuse. The main takeaways are:

  1. A broader view of periodicity. We unify Sequence Periodicity and Rule Periodicity under a common group-theoretic framework.
  2. A theoretical characterization. We characterize the limitations of RoPE-based Transformers in generalizing periodic rules and extend the analysis to Composite Periodicity.

Following the group-theoretic analysis, we introduce the Coper dataset to study Composite Periodicity. We thank the reviewer for the suggestion to present the dataset earlier. But without the theoretical motivation, readers may wonder why we adopt this particular dataset design.

Question 2 (Future Architectural Modification)

A plausible hypothesis is that Mamba and RWKV rely on shared state transitions of the form $ h_i = \mathcal{F}(h_{i-1}, x_i), $ which propagate information sequentially but do not explicitly model transformations on the underlying rule space.

To capture rule periodicity, the transition function F\mathcal{F} should incorporate a shift-equivariant structure over latent rules, enabling $ R(g \cdot f) = g \cdot R(f). $

Without such a mechanism, the model can store historical information but cannot consistently transform it under shifts. A precise formulation of such designs is left for future work [1][2][3].

If you have any further questions, please do not hesitate to contact us.


[1] Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

[2] StackTrans: From Large Language Model to Large Pushdown Automata Model

[3] MeSH: Memory-as-State-Highways for Recursive Transformers

Thank you again for taking the time to review our responses. As the discussion period is coming to an end, we wanted to check if there are any remaining questions or concerns we can clarify. Please feel free to let us know.

It seems that the modular addition task is join shift equivariant, in that shifting both operands also shifts the output in the same way (eg in 123+456=579 if we shift each of the operands then it will shift the answer in the corresponding manner). I am therefore still a bit confused about whether the group theory argument actually holds together, but since this is not my field I don't want to insist. Moreover, the new experiments do strengthen the paper. For these two reasons I will update my score to a 2.

Thank you for your comment. The key distinction is the object on which the group acts.

The “shifting both operands also shifts the output in the same way” in your example can be written as the joint shift equivariance R(g∘f1,g∘f2)=g∘R(f1,f2)R(g\circ f_1,g\circ f_2)=g\circ R(f_1,f_2). This has the same algebraic form as the composition relation considered in our analysis of Rule Periodicity, R(g∘f)=g∘R(f)R(g\circ f)=g\circ R(f), but the group acts on different objects. For sequence periodicity, the group acts on the sequence positions, e.g., gT∘f=fg^T\circ f=f. For Rule Periodicity, the group acts on the rule index, with the corresponding first-order representation of RR. These two actions are not equivalent.

Of course, you can also view Rule Periodicity as a form of “shifting both operands also shifts the output in the same way,” since periodicity fundamentally captures a certain symmetry or shift invariance. The key difference is the object on which the shift acts.

We are glad that you understand this point, as it is precisely what motivates our work. Regardless of the terminology, the core contribution of our paper remains the same. Our analysis shows that Transformers cannot OOD generalize this rule-level periodicity.

The main contribution takeaways are:

  1. A broader view of periodicity. We unify sequence periodicity and rule periodicity under a common group-theoretic framework.
  2. A theoretical characterization. We characterize the limitations of RoPE-based Transformers in generalizing periodic rules and extend the analysis to composite periodicity.

We welcome further discussion!

Reviewer_uhXhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns that can transformer handle Periodicity Generalization via the lens of group theory and classifies into sequence, rule and composite periodicity, arguing that RoPE-based transformers preserve shift equivariance and therefore support generalization only for sequence periodicity. It introduces a benchmark called Coper, which is a controllable generative dataset of two periodic digit sequences summed modulo ten, with Hollow (held-out period combinations, interpolation) and Extrapolation (out-of-range periods) OOD settings. Experiments on Transformer, FANformer, Mamba, RWKV, and TFKAN show that models can memorize periodic training data while fail on unseen composite periodicity. Greater data density and model scale help Hollow, however, it only marginally improves Extrapolation.

Critical strengths
  1. The shifted-equivariance proposition is correct and cleanly proved, with empirical confirmation(RoPE achieves 0.99 accuracy whole absolute positional encoding only gets 0.14)
  2. Coper is fully controllable and generative. The systematic diagnostics cover cover five architectures, three composite rules, ablations on data density and modle scale, and three seeds for experiment variance, which supports the narrow claim that small models trained from scratch memorize seen period combinations and fail to generalize to unseen ones.
  3. The code and data generation files are open-sourceD, which is easy to reproduce for others.
Critical weaknesses
  1. Limited novelty and missing related work. Each main component has a close precedent that the paper does not adequately acknowledge. Hollow is basically Leave-Square-Out (Hu et al., 2024, https://arxiv.org/abs/2402.17709), but uses period pairs instead of operands. Hu et al realdy showed that holes remain when scaling up and proposed a solution. The problem setting and benchmark already appear in Belcak & Wattenhofer (2022, https://arxiv.org/abs/2210.16606), but this work is not cited. App. H re-runs the sin(x) extrapolation experiments of Fig. 1 of FAN(https://arxiv.org/abs/2410.02675) with tokenized input and reaches the same conclusion, without noting the overlap. The observation goes back to Ziyin et al. (2020, https://arxiv.org/abs/2006.08195) is also not cited.
  2. The theory does not support the claim. Eq. 3 shows what the model can do, namely equivalence. But Sec 3.3.2 only derives a property of the target (Eq. 6), without considering the model. Hence, the limitation stated in line 268-269 and line 288-289 is not proved. It's also unclear why this limitation applies to Coper. Its rules have T = 1(App G), and the target is lcm(P1,P2)-period, so Eq. 3 should allow generalizaiton. Fig. 10 reports 0.89-0.98 ID accuracy, which is inconsistent with "RoPE cannot model" in line 996.
  3. Confounding factors are not controlled. The paper attributes the result to RoPE based on only one comparison with sinusoidal PE. However, Kazemnejad et al. (2023, https://arxiv.org/abs/2305.19466) found that NoPE and T5 relative bias outperform RoPE. In Extrapolation, the lcm(P1,P2) can reach 240, which is longer than the maximum training length of 182. App. A also does not clearly state whether the inputs are base periods or lcm(P1,P2)-length sequences.
  4. Insufficient experimental rigor and limited scope. All models use shared hyperparameters, while Mamba and RWKV do not event fit the ID data well. Error bars and architecture details are missing. The abstract discusses LLMs while experiments only use 3-7 layers Transformers trained from scratch. Additionally, Sec 5.2 reports "29.8% in Table 1" while Table 1 shows 36.4%.
Questions for authors
  1. Compared with Leave-Square-Out (Hu et al.,2024, https://arxiv.org/abs/2402.17709) and Belcak & Wattenhofer (2022, https://arxiv.org/abs/2210.16606), what is new besides holding out period pairs?
  2. Coper's rules have T=1, and its target is lcm(P1,P2)-period. Which step shows that Eq.6 leads to a Transformer limitation? Why does this limitation apply to Coper?
  3. Fig. 10 shows 0.89-0.98 ID acccuracy on f(t+T) = 2 * f(t). What exactly does "Rope cannot model"(line 996) mean?
  4. Can Nope or T5 relative bias also achieve nearly 0.99 accuracy for T=7 in App. B.3?
  5. Are the inputs in App. A base periods or lcm(P1,P2)-length sequences? Can you test Extrapolation pairs with lcm(P1,P2) <= 182?
6 comments

For Weakness 1 & Question 1 (Related Work and Comparison)

Thank you for pointing out these related works. We clarify the differences below.

(1) Hu et al. (2024), Case-Based or Rule-Based: How Do Transformers Do the Math?.

Hu et al. introduce Leave-Square-Out to investigate whether Transformers rely on case-based or rule-based reasoning in arithmetic tasks, including modular addition. Their experiments reveal accuracy holes and attribute them to case-based reasoning. However, they do not theoretically explain why Transformers fail to generalize unseen rule combinations or characterize the limitation through group equivariance. We adopt a similar structured holdout strategy but extend it from the numerical operand space to the period-pair space. More importantly, we formulate rule repetition as Rule Periodicity and provide our group-theoretic framework to characterize the generalization of periodic rules.

(2) Belcak and Wattenhofer (2022), Neural Combinatorial Logic Circuit Synthesis from Input-Output Examples.

This work studies neural synthesis of combinatorial logic circuits from incomplete input-output examples. Although it also considers generalization from incomplete training data, its objective is to learn logic circuits for discrete input-output mappings. It does not study periodic structures over the sequence dimension or distinguish OOD generalization across different period combinations. In contrast, we formulate modular addition of periodic sequences as Rule Periodicity and characterize rule reuse through group theory. We further distinguish Hollow and Extrapolation to study different forms of OOD generalization.

(3) Dong et al. (2025), FAN: Fourier Analysis Networks.

FAN studies the extrapolation of periodic functions, including the scalar sine function in Fig. 1. In Appendix H, we extend the same sine extrapolation setting from scalar inputs to tokenized sequences and evaluate Transformer-based architectures, including FANformer. The sequence-based setting is closer to the tokenized input format of LLMs. Our experiment examines whether the periodic extrapolation behavior also holds in sequence modeling.

(4) Ziyin et al. (2020), Neural Networks Fail to Learn Periodic Functions and How to Fix It.

Ziyin et al. study OOD extrapolation of periodic functions and propose Snake activation to introduce periodic inductive bias into neural networks. However, periodic activation functions mainly address the modeling and extrapolation of periodic functions. They do not characterize the repetition and reuse of underlying rules across sequence positions. Our work extends the study of periodicity from function-level extrapolation to Rule Periodicity, which captures the repetition of transformations under group actions.

Main Contributions.

Our main contribution is not the structured holdout strategy itself, but a group-theoretic framework for understanding periodicity and rule reuse. The main takeaways are:

  1. A broader view of periodicity. We unify sequence periodicity and rule periodicity under a common group-theoretic framework.
  2. A theoretical characterization. We characterize the limitations of RoPE-based Transformers in generalizing periodic rules and extend the analysis to composite periodicity.

For Weakness 2 & Question 2 (Applicability of the Group-Theoretic Analysis to Coper)

Thank you for the question. Appendix B.1 shows that Transformers with RoPE capture the left group action on the input sequence through relative positional information. However, our periodicity framework also considers the repetition and reuse of underlying rules. For example, digit-wise addition applies the same addition rule at every position, which can be formulated as Rule Periodicity with (T=1). Unlike sequence periodicity, Rule Periodicity describes the group action on the indexed rule family rather than on the input sequence. Eq. (6) in Appendix B characterizes the relationship between these two actions. The inequality between its two sides shows that the rule-level action cannot generally be reduced to the sequence-level shift captured by RoPE. In Coper, the addition and modulo rules have (T=1), whereas the composed target sequence has period (\operatorname{lcm}(P_1,P_2)). These periods describe different objects. Appendix G further considers a more general rule with (T=2), using alternating addition and subtraction, to evaluate the generalization of nontrivial rule periodicity. We will further clarify the distinction between sequence periodicity and rule periodicity, as well as the corresponding group actions, in the revised manuscript.

For Weakness 2 & Question 3 (Clarification of RoPE's Modeling Limitation)

Thank you for pointing this out. The statement "RoPE cannot model" in Appendix B.3 is indeed imprecise and should be consistent with the main text. RoPE-based Transformers capture periodicity under relative-position invariance. However, the transformation (f(t+T)=2f(t)) does not satisfy this property, as it scales the values across periods. Therefore, RoPE cannot generalize this transformation to OOD periods. The ID accuracy of 0.89–0.98 in Fig. 10 indicates that the model can fit the observed patterns. Our claim concerns OOD generalization, rather than ID fitting. We will revise the wording in Appendix B.3 from "cannot model" to "cannot generalize" throughout. We thank the reviewer for identifying this imprecise statement.

For Weakness 3 & Question 4 (Compare with NoPE and T5)

We thank the reviewer for this suggestion and evaluated NoPE and the T5 relative position bias under identical models and training setups, both on the single-period setting of App. B.3 and on the composite task.

  • Single period (App. B.3, Table R3, three seeds). Yes, both reach ~0.99 at T = 7: NoPE 0.998 ± 0.001 and T5 0.988 ± 0.005 (token accuracy), versus 0.985 ± 0.012 for our rotary encoding. NoPE also extrapolates to T = 2, 11 and 12 (≥ 0.99), consistent with Kazemnejad et al. (2023). An attention analysis shows that NoPE and T5 attend to the position exactly one period back for every period, seen or unseen (NoPE 0.81–0.93 attention weight).
PE23456789101112
RoPE0.384 ± 0.0210.836 ± 0.0531.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.985 ± 0.0121.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.827 ± 0.0080.495 ± 0.008
PE0.294 ± 0.0260.489 ± 0.0351.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.658 ± 0.1371.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.335 ± 0.0440.245 ± 0.007
NoPE0.999 ± 0.0021.000 ± 0.0000.999 ± 0.0010.999 ± 0.0010.999 ± 0.0010.998 ± 0.0010.999 ± 0.0010.998 ± 0.0010.997 ± 0.0020.994 ± 0.0030.992 ± 0.004
T50.850 ± 0.1920.995 ± 0.0021.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.988 ± 0.0051.000 ± 0.0001.000 ± 0.0001.000 ± 0.0000.941 ± 0.0250.894 ± 0.033

However, for composite periodicity, The results of NoPE are shown in the table below. Due to format limitations, we present the heatmap in table form.

T \ T2345678910
20.450.550.490.520.550.540.500.590.54
30.600.450.520.510.520.590.570.480.56
40.460.500.460.550.520.560.500.550.55
50.530.520.540.430.530.550.540.550.48
60.500.460.500.550.440.550.550.530.53
70.580.550.580.540.550.450.550.530.55
80.510.540.490.540.530.540.460.540.54
90.570.510.540.560.540.540.540.470.54
100.480.560.520.480.530.550.540.530.46

Under NoPE, the Transformer fails even at in-distribution fitting (~0.5 accuracy). This may be due to insufficient positional resolution, and poor attention concentration [1]. RoPE provides external knowledge, supplementing the Transformer's lacking periodic information. Its modeling scope is constrained by Eqs.(4)-(5), and the above proof shows the Transformer alone cannot even satisfy Eq.(3).


[1] Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.