- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Flat transformer multiplication doesn't work mostly on middle digits, digit-wise error under independent per-term errors
- RAT: has three roles (BASE, SPLIT, COMBINE) and is external executor
- Oracle substitution pinpoints failure and repair by training COMBINE on wider inputs
Critical strengths
- Oracle-substitution: does a good job of catching failure to COMBINE (oracle COMBINE → –)
- If executed well, it could be a useful method for staged models
Critical weaknesses
- Undocumented baselines:
- flat model exact match on (only problems)
- architecture and training is not described in the paper or the appendix.
- Error law is contradicted and not sure if the repair works: conclusion is baked into the independence assumption, so that needs to be fixed. carry-only top digit most accurate (); last digit as grows. 1 run per variant, products per length, no compute-matched control
Questions for authors
- flat baseline architecture, data, training, whose prediction files were used?
- it's only 1 x COMBINE control with matched extra steps. there should be more runs per variant
- why is the top digit most accurate + why does the last digit degrade with ?
- diagnosis and repair on the same checkpoint family ( vs. at )
- Suggestions: per-position majority baselines (~ first, ~ last, ~ middle); Table 10 from raw data, not Figure 3 labels; compare with Recursion of Thought; relate to Bai et al. (2025; arXiv, not counted); define terms plainly
- Typos: none found
3 comments
Why do you stop at 10,000 optimization updates when the model only gets 36.5% on 6×6? How does a compute matched, LSD first and scratchpad trained transformer perform on this? What is happening with the source captions on Figure 3 and Table 13? Regardless, these are way too many changes to review on the last day of rebuttals. I don't think I can do it full justice and go through everything that was provided. I will try to make the effort doing the reviewer/AC discussion period but I strongly advise you to work on the baselines. I maintain my score.
We thank the reviewer for their continued engagement and constructive feedback. Regarding the 10,000 updates, we used the same architecture, optimizer, and training budget for the flat Transformer and RAT to ensure a controlled comparison. Importantly, the reported 36.5% accuracy corresponds to the MSD-first baseline; the LSD-first variant already achieves substantially better performance. We agree that a compute-matched LSD-first model with scratchpad supervision would be a valuable additional comparison. Regarding Figure 3 and Table 13, both refer to the same recombination-adaptation experiment. The source notation in Table 13 indicates that the numerical values correspond to Figure 3; it does not refer to an external source. We will make this clearer. We appreciate the reviewer's willingness to revisit the revisions during the reviewer–AC discussion period. We recognize the time constraints and thank the reviewer for considering our additional analyses and results.
We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised. All revisions made in response to the reviewers’ comments are highlighted in blue in the updated manuscript.