- Rating
- 3
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the coordinate representation of mathematical tasks affects Transformer generalization. Using controlled experiments on modular arithmetic, the authors show that the same mathematical relation can have very different learning difficulty depending on how it is represented. In particular, applying a discrete logarithm transformation turns nonlinear modular exponentiation into an affine mapping, improving held-out accuracy from nearly chance level to 45.9% and 85.3% on two tasks based on number-theoretic theorems.
To investigate this effect, the authors conduct relabeling experiments that control for algebraic degree, best-affine agreement, and the set of modified inputs. The results show that the internal structure of the presented function matters for generalization, even when these other factors are held fixed. The authors also introduce Fourier Complexity as a measure of function complexity and test its ability to predict generalization on additional tasks.
Finally, the paper examines the functions learned by trained networks and the role of weight decay. The results suggest that the coordinate representation determines whether a low-complexity solution is available, while weight decay helps the model select a solution that generalizes. Additional experiments cover different Transformer architectures, MLPs, LSTMs, numerical bases, and finite fields.
Critical strengths
1. Well-controlled experiments that isolate the effects of different factors. The paper does more than compare generalization across coordinate systems. By independently changing the input and output coordinates, the authors show that a substantial improvement occurs only when both are transformed. The matched relabeling experiments are particularly convincing. Even with the same algebraic degree, best-affine agreement, and modified input set, changing the structure of the relabeled mapping produces a large accuracy gap of 97.7% versus 0.6%. This provides strong evidence that the structure of the presented function, rather than just its coarse algebraic properties, affects generalization.
2. A useful connection between mathematical analysis and generalization prediction. The authors use Fourier analysis to characterize the complexity of target functions under different coordinate systems and derive theoretical relationships between Fourier structure and low-complexity predictors. Importantly, the paper goes beyond fitting correlations to existing results. It evaluates fixed prediction rules on additional tasks, including predictions specified before model training. The experiments across architectures, numerical bases, and weight decay settings also help establish that the main observations are not limited to a single configuration.
Critical weaknesses
1. Fourier Complexity does not fully explain or predict generalization. Although Fourier Complexity correlates strongly with generalization in many experiments, it is not a sufficient predictor. As shown in Appendix D.2, tasks with similar Fourier Complexity can have very different held-out accuracies, ranging from nearly 0% to 78%. Moreover, on some task families not used to select the predictors, Best-affine Agreement performs as well as or better than the Fourier-based metrics. This raises questions about how much additional explanatory value Fourier Complexity provides beyond simpler structural measures. The metric captures useful properties of the target function and provides additional predictive information in some controlled settings, but the consistency of this contribution across different task families remains unclear.
2. The theoretical analysis does not fully explain how Transformers learn generalizing solutions. The theoretical results mainly constrain prediction functions with a given Fourier Complexity. They do not establish a direct relationship between this complexity measure and neural network size, optimization dynamics, or the functions that a Transformer can actually learn. Although the experiments show that weight decay improves generalization while reducing the Fourier Complexity of the learned function, they do not fully explain why optimization selects these solutions. The authors also find that structured non-Fourier features can improve generalization on difficult tasks by a similar amount. This suggests that the observed benefits may come from structured representations more generally, rather than Fourier structure specifically. The proposed explanation of the learning mechanism therefore remains incomplete.
Questions for authors
1. The paper shows that tasks with similar Fourier Complexity can have very different generalization performance, with accuracy ranging from nearly 0% to 78% in Appendix D.2. Could the authors explain which structural properties might account for these differences? In particular, would combining Fourier Complexity with piecewise-affine structure or properties of the digit encoding lead to more reliable predictions?
2. The theoretical analysis mainly characterizes the Fourier structure of target functions, but does not directly explain why Transformer optimization favors certain low-complexity solutions. Could the authors clarify what additional assumptions or evidence would be needed to connect Fourier Complexity to the optimization dynamics that select generalizing solutions? Also, given that structured non-Fourier features can produce similar improvements, how can we distinguish the role of Fourier structure from that of structured representations more generally?
3 comments
Thank you for your detailed response and for addressing my concerns about Fourier Complexity and generalization. I also appreciate your clarification that the paper does not aim to explain why Transformer optimization favors certain low-complexity solutions.
However, this clarification differs somewhat from my original understanding of the paper, which makes me wonder whether I may have misunderstood its intended scope and contributions in my initial review.
I still believe that the research questions addressed in this paper and the controlled experiments are very valuable. However, after reading the other reviewers' comments, particularly their concerns about the novelty of the findings in relation to existing work on Grokking and Weak-to-Strong Generalization, I have become less certain about the broader significance of the paper. In particular, it remains unclear to me how well the findings and methodology can generalize beyond the specific arithmetic settings studied here.
Overall, I will maintain my original score of 3 (Weak Acceptance), but lower my confidence from 3 to 2, reflecting my uncertainty about the paper's novelty and broader significance relative to the existing literature.
Follow-up to Reviewer J7GS
Thank you for the follow-up and for maintaining your score. We answer the three points you raise. Line numbers refer to the original submission. Section, table and figure numbers refer to the revised PDF.
However, this clarification differs somewhat from my original understanding of the paper, which makes me wonder whether I may have misunderstood its intended scope and contributions in my initial review.
- Your original understanding was correct, and the scope of the paper has not changed. Your review summarized it as: “the coordinate representation determines whether a low-complexity solution is available, while weight decay helps the model select a solution that generalizes.” This is exactly contribution C3.
- Our reply only separated two things. The paper shows by experiment that weight decay selects the simple solution: accuracy is 0.060 without it and 0.459 with it (lines 380 to 395; Section 5.2). It does not prove mathematically why. Your review drew the same line in its second weakness, and the submission lists it as a limitation (lines 454 to 458).
However, after reading the other reviewers' comments, particularly their concerns about the novelty of the findings in relation to existing work on Grokking and Weak-to-Strong Generalization, I have become less certain about the broader significance of the paper.
- Grokking work keeps one way of writing the task and studies how generalization appears inside the model during training. We do the opposite: we keep the model and the training fixed and change only how the task is written. The two are complementary.
- Our results add to that literature directly. Its standard benchmark (Power et al., 2022) gives each number its own token. When the same numbers are written as digits, two of the nine operations generalize instead of eight, as we predicted before training (lines 419 to 427; Table 2). A published grokking result therefore depends on a choice that had not been tested.
- Weak-to-strong generalization, as we understand it, studies a model that learns from a weaker supervisor. It concerns the quality of the labels, not the way the task is written, so it does not address our question.
- The concern was phrased as “better representations lead to generalization”. Two of our results are not predicted by that statement. Giving the model the helpful naming on the input side only, or on the output side only, gives no gain (Table 1; lines 195 to 199). And two tasks with the same algebraic degree, equally close to a simple straight-line rule, reach 0.977 and 0.006 (Figure 2; lines 245 to 252).
In particular, it remains unclear to me how well the findings and methodology can generalize beyond the specific arithmetic settings studied here.
- We use arithmetic on purpose. It is the setting where the full table of answers is known exactly, so the written form can change while the task and the test inputs stay identical. Arithmetic tasks are also the standard testbed for grokking and for length generalization, so the findings apply where such claims are made.
- The method carries over to any task whose answers are known: write the same task in two ways, keep the examples and test inputs the same, and compare. The practical lesson carries over too: before a failure is blamed on the architecture, check how the task was written and tokenized (Section 7).
- Within arithmetic, the effect is not tied to one setting. It holds on two theorems, three model types, four number bases, a second number system and a published benchmark (lines 399 to 427; Appendix F).
- We do not claim results on natural-language tasks, and Section 6 states this scope.
We hope this resolves the uncertainty about scope and novelty. Thank you again for the careful engagement.
Response to Reviewer J7GS
Thank you for the accurate summary of the paper and for recognizing the matched controls and the predictions specified before training. Both of your questions led to new analyses, which are now in the revised paper. Line numbers refer to the original submission. Section, table and figure numbers refer to the revised PDF.
We hope these additions address the two concerns behind your soundness score. If anything remains unclear, we would be glad to clarify it during the discussion.