- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how the dependency structure of training data influences the use of different RoPE frequencies in Transformers. The authors introduce a “frequency energy” metric based on the amplitudes of Q/K subspace contributions and propose that broader data dependencies favor lower frequencies. Under their definitions of positional contrast and field coverage, they derive an inverse relationship between the optimal frequency and dependency width. They further show that when the test-time dependency distribution is a uniform dilation of the training distribution, the corresponding position interpolation preserves the proposed positional contrast utility.
The experiments include frequency-spectrum analyses of pretrained models, synthetic retrieval tasks with varying block lengths, iGSM datasets with different reasoning lengths, and evaluations of position interpolation on synthetic retrieval and arithmetic tasks. The authors also compare mutual-information profiles across tokenization granularities to support approximate self-similarity in natural-language dependencies.
Critical strengths
-
Intervening on training data provides a useful approach to studying frequency usage. The synthetic experiment fixes sequence length and retrieval offset while varying block length, then measures the learned frequency distribution. This offers evidence about how data statistics influence internal representations and is the paper’s most valuable empirical contribution.
-
The discussion of when position interpolation should help offers useful insight. The paper distinguishes settings where dependency distances expand with context length from those where the relevant distances remain fixed. The synthetic retrieval example, arithmetic evaluation, and additional NTK-aware and YaRN comparisons give this distinction some practical relevance.
Critical weaknesses
- The frequency metric and theoretical optimum are insufficiently connected to the mechanisms through which the model performs the task.
The “frequency energy” in Section 3 is mathematically well defined, but the quantity being averaged simplifies to:
\left\lVert k_j^{(m)} \right\rVert_2^{2}.$$ Thus, the metric measures the expected product of squared Q/K subvector norms. It can characterize the amplitude of a frequency component, but it does not capture that component’s phase-dependent contribution at the relative distances actually encountered, nor does it directly establish its importance for positional retrieval or final predictions. The paper should clarify what information this metric adds beyond existing norm-based analyses and validate whether higher energy corresponds to greater functional importance. Similarly, Theorem 1 obtains the inverse relationship between optimal frequency and dependency width by exploiting the monotonicity of positional contrast within a prescribed half-period coverage constraint. The result holds under these definitions, but the paper does not establish a connection between this utility and the training loss, learning dynamics, or allocation of frequency energy. In particular, the theory optimizes a single continuous frequency, whereas the experiments measure a geometric mean over multiple discrete frequencies weighted by energy. Why these quantities should follow the same quantitative law remains unclear. This gap limits the paper’s claim to provide a predictive theory of frequency selection. 2. The key experiments do not sufficiently validate the proposed dependency-scale mechanism, and the length-generalization analysis contains comparability issues. In the block-structured experiment, the authors identify dependency width with the support of the input autocorrelation, which equals the block length. However, the task always requires retrieving the value exactly 16 positions earlier. Changing block length changes statistical correlations among inputs, while the direct positional relation required for exact retrieval remains fixed. The paper does not sufficiently justify why the former represents the task-relevant dependency width in the theory. It also does not report retrieval errors or whether the model recovers the target’s independent noise component. Consequently, the results do not establish whether the model performs accurate retrieval or primarily makes approximate predictions using the shared block signal. These behaviors would imply different explanations for the observed frequency shift. For natural language, similar mutual-information profiles across BPE vocabulary sizes do not directly establish that, under a fixed tokenizer, the dependency distribution in longer contexts is a uniform dilation of the distribution in shorter contexts. There remains a substantial gap between Figure 4B and the assumptions of Theorem 2. Furthermore, Table 2 in Appendix F.2 cites natural-language results from Chen et al., but those results come from fine-tuned PI models, whereas the arithmetic experiment does not report a corresponding fine-tuning procedure. The original paper’s columns denote evaluation context-window sizes; the submission relabels the same values as disjoint position intervals. This requires correction and prevents the current comparison from attributing performance differences primarily to differences in self-similarity between natural language and arithmetic.Questions for authors
The following questions seek clarification or validation of the current central claims, and their answers could change my assessment.
-
Interpretation of the metric: Given that frequency energy simplifies to the expected product of squared Q/K subvector norms, what evidence establishes its ability to measure functional importance in positional modeling? Do frequency perturbation or ablation results support this interpretation?
-
Connection between theory and measurement: Why should the optimum of the single-frequency utility in Theorem 1 predict the geometric mean of frequencies weighted by energy measured in the experiments? Does this connection require additional assumptions about learning or optimization?
-
Task performance in the synthetic experiment: What are the training and test errors for the different block lengths in Figure 2? Does the model recover the target position’s independent noise component? How do these results justify using input autocorrelation width as the dependency width of the fixed-offset retrieval task?
-
Evidence for length generalization: How does similarity across tokenization granularities establish the dilation of dependency distributions required by Definition 4? Please also clarify and correct the differences in fine-tuning protocols and column definitions in Appendix F.2, and explain which evidence still supports the comparison between natural language and arithmetic after these corrections.