- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how training affects the computational strategies learned by Transformers on arithmetic tasks. The authors first observe that a non-autoregressive Transformer trained directly on addition learns the most significant digits before the less significant ones, opposite to the order used in standard digit-wise addition.
To explain this behavior, the authors analyze a simple two-layer linear network and derive its learning dynamics. They show that the network learns a continuous representation of the sum's overall magnitude. With an ideal decoder, this representation explains why more significant digits become accurate earlier in training. The authors then use attention pattern analysis, activation patching, and long carry-chain tests to show that directly trained Transformers exhibit behavior more consistent with numerical estimation than with standard digit-wise computation.
Finally, the authors introduce a curriculum that gradually increases the number of digits during training. This encourages the model to learn less significant digits first and develop attention patterns associated with digit-wise addition and carry propagation. The experiments show faster training, better generalization to unseen digit combinations, and more efficient transfer to related arithmetic tasks.
Critical strengths
1. A good combination of theoretical analysis and mechanistic experiments. The paper goes beyond observing that Transformers learn digits from most to least significant. The authors use an analytically tractable linear model to explain why a magnitude-based representation may be favored early in training. They then examine the computation learned by Transformers through attention patterns, activation patching, and carry-chain tests. In particular, the causal interventions on attention heads associated with local digit summation and carry propagation provide useful evidence for the digit-wise computation learned under curriculum training.
2. Interesting findings on how training order affects computation and generalization. The paper shows that changing the training curriculum alone, without modifying the architecture or final task, can reverse the order in which digits are learned and lead to attention patterns associated with local summation and carry propagation. The curriculum-trained model also performs better on unseen digit combinations and adapts faster to related tasks. I find this interesting because it suggests that curriculum learning can change not only training efficiency but also the computational strategy a network learns.
Critical weaknesses
1. The connection between the theoretical model and the actual Transformer mechanism is not fully established. The linear model is shown to learn a representation of the overall sum magnitude, but it cannot solve the full addition task on its own. The theoretical explanation of digit acquisition order also relies on an additional ideal decoder. For the Transformer, the evidence for an estimation-like strategy comes mainly from digit learning order, attention patterns, carry-chain behavior, and the decodability of intermediate representations. While these observations provide evidence for estimation-like behavior, they do not establish whether the Transformer uses the continuous magnitude representation identified in the linear model, rather than other heuristic strategies. More direct evidence would be needed to establish whether the Transformer actually learns and uses the magnitude representation identified in the linear model.
2. Algorithmic generalization and the scope of the findings remain limited. Although curriculum learning produces attention patterns closer to those expected from standard digit-wise addition, accuracy still drops sharply once the carry-chain length exceeds a certain threshold. This suggests that the learned digit-wise computation remains limited in its ability to propagate carries beyond the model's effective computational depth. Also, most of the detailed mechanistic analysis is conducted on a small encoder Transformer trained from scratch. The experiments on other architectures and large pretrained models are more limited. The results therefore support the conclusion that curriculum learning can change the learned strategy in this controlled arithmetic setting, but do not yet establish that the same approach broadly improves algorithmic generalization.
Questions for authors
1. The linear model is shown to learn a magnitude representation, but the Transformer experiments mainly show behavior consistent with an estimation-like strategy. Could the authors provide more direct evidence that the Transformer actually learns and uses a similar magnitude representation, rather than other heuristics that produce the same behavior? For example, probing or causally intervening on a candidate magnitude representation might help strengthen this connection.
2. The curriculum-trained model still fails on carry chains beyond a certain length. Could the authors clarify how its learned digit-wise computation differs from the standard carry algorithm, and to what extent its generalization is limited by model depth and training length? In particular, how does increasing model depth affect the maximum carry-chain length the model can reliably handle?
2 comments
Thank you for your detailed responses and additional experiments, which have addressed my main concerns regarding the connection between the theoretical linear model and the computation learned by Transformers, as well as the limitations of carry-chain generalization.
In particular, I find the magnitude erasure experiments convincing, as they provide stronger causal evidence for the role of magnitude representations during training and strengthen the paper's contribution to mechanistic interpretability. I also appreciate the additional analysis of how model depth affects carry-chain generalization.
However, I still have some reservations about the broader applicability of these findings, as the effectiveness of the proposed approach has not been sufficiently validated on more complex tasks or at larger scales. (Although the paper examine the accuracy of different digit positions in large pretrained models, the paper do not validate the main methods and mechanisms of the paper on these models, including Curriculum Learning, Magnitude Erasure, Activation Patching, and algorithmic generalization experiments.)
Overall, I appreciate your efforts and the additional evidence provided, but I will maintain my original score of Weak Acceptance (3).
Thank you for your feedback and questions. Please find responses below:
“Could the authors provide more direct evidence that the Transformer actually learns and uses a similar magnitude representation, rather than other heuristics that produce the same behavior?”
We have added an experiment where the magnitude is erased from the representations during training. This turns out to significantly slow training, and the model fails to converge in 1 million steps. This is in contrast to erasure of the carry values, or all digit sum information beyond the magnitude, which despite being important at convergence, do not affect learning as much when removed. Results can be seen in figure 18.
More broadly, our claim here is not that the transformer does the same thing as the linear model, rather, the linear model is meant to demonstrate a shortcut that exists in the task (the reason the magnitude is learned has little to do with the architecture, and mainly comes from the largest mode in the input-output covariance). The existence of a statistical shortcut is likely to affect other models, and we find some evidence for this early in training for the transformer, although it may exist in parallel with other strategies. Note also that a similar magnitude heuristic has been observed in Claude Haiku, in parallel with digit-wise computation (Lindsey et al. 2025, "On the Biology of a Large Language Model").
“Could the authors clarify how its learned digit-wise computation differs from the standard carry algorithm, and to what extent its generalization is limited by model depth and training length?”
The maximum observed carry-chain length increases linearly as a function of depth. The paper argued that this is because the model can only perform multiple carry operations sequentially given enough depth. Upon further investigation, this appears to still be true, but in a more nuanced way. A shallow network can learn longer carry-chains, but only if they appear in the data. Natural data rarely contains long chains, so shallow models learn only heads which can look up a carry a few places back. A deeper network, even when learning heads with limited carry lookup range, could still compute a longer carry chain, by applying the heads sequentially. Thus, the learned digit-wise computation here is similar to the standard carry algorithm, but differs in that it can only perform a limited number of operations, and so is limited by the depth of the model given a limited carry-chain length observed in data. See figure 19 for these results.