acceptodds
Under review as a conference paper at ICLR 2027

Transformers Miss the Middle Digits: Repairing Long Multiplication by Recursion and Robust Recombination

Abstract

Transformers trained to multiply long integers tend to get the first and last digits of the product right and the middle ones wrong. We show that this is structure, not noise. For a flat model that writes the product directly, the five central digits of products are predicted at chance level (10–13%) while the first digit is still accurate, and the failing band widens with the number of digits. The triangular profile of column interactions motivates this diagnosis. Under an explicit residual-channel model, we prove a digitwise error law and conditions for a central error peak and a widening band near chance. We quantify corrections for carry errors, dependence, and decoder mismatch. We also prove an exact local recombination rule and attention-leakage and margin conditions for its stable execution. We build the Recursive Arithmetic Transformer(RAT), a single Transformer that is never supervised on products beyond digits. Exact oracles for Split move its accuracy on 9- and 10-digit products by at most 4.2 points, and training Combine on wider inputs raises exact match from to on and from to on multiplication, with a gain at every length from 8 to 18 digits. Long-multiplication failures should be diagnosed, and repaired, one stage at a time.

Primary area:
interpretability and explainable AI
Keywords:
Long-Multiplication, Transformer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KwoQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Flat transformer multiplication doesn't work mostly on middle digits, digit-wise error under independent per-term errors
  • RAT: has three roles (BASE, SPLIT, COMBINE) and is external executor
  • Oracle substitution pinpoints failure and repair by training COMBINE on wider inputs
Critical strengths
  • Oracle-substitution: does a good job of catching failure to COMBINE (oracle COMBINE → 9292–96%96\%)
  • If executed well, it could be a useful method for staged models
Critical weaknesses
  • Undocumented baselines:
    1. flat model 27%27\% exact match on 2×22\times2 (only 8,1008{,}100 problems)
    2. architecture and training is not described in the paper or the appendix.
  • Error law is contradicted and not sure if the repair works: conclusion is baked into the independence assumption, so that needs to be fixed. carry-only top digit most accurate (74%74\%); last digit 97%→38%97\% \to 38\% as nn grows. 1 run per variant, 100100 products per length, no compute-matched control
Questions for authors
  • flat baseline architecture, data, training, whose prediction files were used?
  • it's only 1 x COMBINE control with matched extra steps. there should be more runs per variant
  • why is the top digit most accurate + why does the last digit degrade with nn?
  • diagnosis and repair on the same checkpoint family (50.6%50.6\% vs. 73%73\% at n=10n = 10)
  • Suggestions: per-position majority baselines (~24%24\% first, ~27%27\% last, ~10%10\% middle); Table 10 from raw data, not Figure 3 labels; compare with Recursion of Thought; relate to Bai et al. (2025; arXiv, not counted); define terms plainly
  • Typos: none found
3 comments

We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised. All revisions made in response to the reviewers’ comments are highlighted in blue in the updated manuscript.

PointResponseWhere
W1. Undocumented baseline: the flat model gets 27% exact match on 2×2.We replaced this model. The new baseline uses the same network as RAT and reaches 99.0% on 2×2, 97.8% on 3×3 and 95.9% on 4×4.Sec. 2, App. I
W2. Architecture and training are never described.The new baseline is fully described: 6 layers, width 256, 8 heads, position coupling, AdamW (learning rate 8e-5), batch 64, 10k updates, operands of 1–6 digits, seeds 42/43/44. The code is linked in the footnote on page 1.App. I, K
W3. The error law is contradicted, and its conclusion is built into the independence assumption.On the new baseline, the qualitative predictions hold: the least accurate digit is in the middle, the band of failing digits widens, and columns with longer carry chains are less accurate at every width (widths 1 to 7: 79, 67, 56, 41, 28, 19, 14% vs. 98, 79, 68, 55, 41, 28, 18% for short chains). The simplest quantitative version does not fit (89% predicted vs. 54% measured), and Section 5 now says so. Proposition 3 and Theorem 2 do not assume independence.Sec. 5, App. B, D
W4. The carry-only top digit is the most accurate (74%); the last digit falls from 97% to 38% as n grows.Neither pattern appears with the new baseline. The last digit is now the most accurate at every length (100, 99.8, 95.9, 94.3% for n = 2, 4, 6, 8), as expected: it depends only on a₀×b₀ and has no incoming carry. The top digit is second (99.7, 99.2, 86.1, 77.5%). The earlier 38% came from the old model writing "0" as the last digit in 60 of 94 answers.Sec. 2, Table 8
W5. One run per variant, 100 products per length, no compute-matched control.The repair study uses one run per variant with 100 products per length, as the Limitations state. Within these runs, the order normal < 3× < 4× holds at all six lengths, and a single per-call error rate fits each variant across lengths (0.40%, 0.35%, 0.24%). The gain is consistent across lengths and does not come from one favorable cell.Sec. 6.3, Limitations
Q1. Flat baseline architecture, data and training; whose prediction files were used?See W1–W2. All numbers in the paper now come from the new baseline; the numbers from the earlier model have been removed.Sec. 2, App. I
Q2. A 1× COMBINE control with matched extra steps; more runs per variant.See W5.Sec. 6.3
Q3. Why is the top digit the most accurate, and why does the last digit degrade with n?See W4. With the new baseline the last digit is the most accurate and the top digit is second. The top digit depends mostly on the leading digits of the operands, so it stays fairly accurate.Sec. 2
Q4. Diagnosis and repair on the same checkpoint family (50.6% vs. 73% at n=10).They are different training runs, which the revision now states. On the diagnosis model itself, 83% (9×9) and 93% (10×10) of failures begin in COMBINE, the step we retrain, so diagnosis and repair point to the same step.Sec. 6.2, Limitations
S1. Add per-position majority baselines.Added to Figure 1 and Appendix I (Table 8). On our test sets the majority guess is 22–26% for the first digit, 26–29% for the last and 11–14% in the middle, close to your estimates.Fig. 1, Table 8
S2. Build Table 10 from raw data, not from figure labels.Table 10 (now Table 13) lists the exact-match values of the repair runs shown in the figure, and its caption states this source.Table 13
S3. Compare with Recursion of Thought.Added to Section 7. Both methods keep the control flow outside the network. RAT trains only three small step types and never sees a product larger than 2×2 or a full problem during training. Its separate steps are what allow the oracle test and the targeted repair. Since the training setups differ, we compare the two designs rather than their accuracy.Sec. 7
S4. Relate to Bai et al. (2025).Added to Section 7. They also find that standard training fails on the middle digits (below 1% on 4×4) and fix it with an auxiliary loss. Our results agree with theirs; we add the analysis by column width and carries, and a fix based on splitting the problem.Sec. 7
S5. Define terms plainly.See the new terminology paragraph in Section 1.Sec. 1

Why do you stop at 10,000 optimization updates when the model only gets 36.5% on 6×6? How does a compute matched, LSD first and scratchpad trained transformer perform on this? What is happening with the source captions on Figure 3 and Table 13? Regardless, these are way too many changes to review on the last day of rebuttals. I don't think I can do it full justice and go through everything that was provided. I will try to make the effort doing the reviewer/AC discussion period but I strongly advise you to work on the baselines. I maintain my score.

We thank the reviewer for their continued engagement and constructive feedback. Regarding the 10,000 updates, we used the same architecture, optimizer, and training budget for the flat Transformer and RAT to ensure a controlled comparison. Importantly, the reported 36.5% accuracy corresponds to the MSD-first baseline; the LSD-first variant already achieves substantially better performance. We agree that a compute-matched LSD-first model with scratchpad supervision would be a valuable additional comparison. Regarding Figure 3 and Table 13, both refer to the same recombination-adaptation experiment. The source notation in Table 13 indicates that the numerical values correspond to Figure 3; it does not refer to an external source. We will make this clearer. We appreciate the reviewer's willingness to revisit the revisions during the reviewer–AC discussion period. We recognize the time constraints and thank the reviewer for considering our additional analyses and results.

Reviewer_MmDrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Recursive Arithmetic Transformer (RAT) to improve the multiplication ability of Transformers. The authors observe that Transformers tend to make more errors in the middle digits and suggest that this is related to interaction width. To mitigate this issue, the authors propose RAT, which uses a single Transformer to perform three operations (SPLIT, BASE, COMBINE), and introduces an external executor to manage the recursive calls. RAT essentially breaks a long multiplication problem into smaller problems involving at most 2x2 multiplication and recursively combines the partial results to obtain the final answer. The experiments show that RAT can handle longer multiplication problems and that further training the COMBINE stage improves its performance.

Critical strengths

None

Critical weaknesses

W1. The paper has writing issues.

  • There are too many terms that are not well explained/defined before they are used. Here I list just a few examples that made me stop reading and search for their definitions:

    • Line 91: "Recombination"
    • Line 159: "checkpoint", "COMBINE"
    • Line 165: "M=2"
    • Let me take "COMBINE" as an example. In Section 3, the authors introduce H3, which discusses replacing COMBINE with an oracle. However, up to that point, the only explanation given to the reader is in the Introduction, where BASE, SPLIT, and COMBINE are briefly introduced as three roles. The actual explanation of COMBINE, which takes four partial products and adds them with proper alignment, appears later in Section 4.2.
  • I could not find sufficient information about how the flat Transformer used in Figure 1 was trained. What were the training input lengths and input distribution? What was the model size? What positional embeddings were used? Since Figure 1 provides the main motivation for this paper, these experimental details should be clearly reported.

  • At the beginning of Section 4.1, the authors describe their positional embedding method. To my knowledge, this is essentially Position Coupling (and is also closely related to Abacus Embeddings). However, I could not find any explicit statement in the main text that RAT uses Position Coupling. Although Position Coupling is discussed as prior work, the authors do not clearly explain that their own method adopts a similar technique.

  • The most serious writing issue is in Line 114: "Every number in this section is computed from the released prediction files." I could not find any explanation of what these "released prediction files" refer to, or where they are released. It reads as if the authors are referring to files in their own local directory. Similar wording appears in Figure 1: "The first two well-formed predictions in each file (not selected)" and "First well-formed predictions in each file, not selected."

  • For these reasons, I suspect that the paper relies heavily on AI-assisted writing without sufficient human review. I am fully aware that using AI for paper writing is not itself a valid reason for rejection, and the authors explicitly acknowledge AI assistance in the AI Use Statement. However, the issues above suggest that the manuscript was not carefully checked by the authors.

W2. The contributions are not strong enough for acceptance.

  • There have been many studies on Transformers learning arithmetic and algorithmic tasks. Based on my understanding of the literature, these studies usually aim to understand (1) how Transformers learn algorithms and how their learned mechanisms differ from human intuition, (2) whether and how Transformers can generalize to out-of-distribution inputs, especially longer inputs, and (3) what is needed for Transformers to learn more general algorithmic tasks. Arithmetic tasks such as addition and multiplication have mainly served as simple testbeds for studying these broader questions.

  • In contrast, this paper seems to focus mainly on improving the multiplication accuracy of Transformers. Why is it important to improve the multiplication accuracy of Transformers specifically? What broader understanding can we gain from this improvement? I do not think the paper gives convincing answers to these questions. Moreover, the proposed method does not fully solve the multiplication task either.

  • I also think the reliance on an external executor problematic. The executor explicitly manages the recursive computation and the order of subproblem execution. Therefore, much of the algorithmic structure is provided by an external program rather than learned by the Transformer itself.

  • Theorem 1 in Section 5.1 also seems rather straightforward and does not provide much new insight. Essentially, it assumes that each digit-product contribution has an independent error following a symmetric residue distribution, and then analyze how the error probability of the final digit changes as the number of contributions wn(r)w_n(r) increases. Under these assumptions, it is quite expected that aggregating more noisy contributions makes the final digit inaccurate.

Questions for authors

See the weaknesses.

1 comment

We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised. All revisions made in response to the reviewers’ comments are highlighted in blue in the updated manuscript.

PointResponseWhere
W1. Terms used before they are defined (Recombination, checkpoint, COMBINE, M=2).Thank you for listing these. Section 1 now has a short terminology paragraph, placed before the hypotheses, that defines frame, BASE, SPLIT, COMBINE, recombination, the cutoff M=2, executor, checkpoint and oracle substitution.Sec. 1
W1. How was the flat model of Fig. 1 trained?The new flat baseline uses the same network as RAT (6 layers, width 256, 8 heads, position coupling) with the same optimizer and number of updates. It is trained on operands of 1–6 digits and writes the product directly, most significant digit first, with one additional run in reverse order. We use 3 seeds and 1000 new test products per length for n = 2 to 8. Results are in the global summary.Sec. 2, App. I, K
W1. Position coupling is not credited.RAT's place IDs are position coupling (Cho et al., 2024), which is closely related to Abacus embeddings (McLeish et al., 2024). Section 4.1 now says so, and we do not claim it as a contribution.Sec. 4.1
W1. "Released prediction files".We have rebuilt Figure 1 from the new runs, and released the code (footnote on page 1).Sec. 2, Fig. 1, Reproducibility
W1. The writing suggests too little human review.Every issue listed has now been corrected.Whole paper
W2. Why improve multiplication specifically, and what do we learn?We use multiplication as a test case. The paper shows three things beyond accuracy. (1) Errors depend on digit position: on 8×8 the first and last digits are right 78% and 94% of the time, the middle digits only 13–16% (majority guess 11–14%), which exact match does not show. (2) Once recursion keeps every step small, most errors come from adding the partial results: an exact COMBINE raises 9×9 / 10×10 accuracy to 92% / 96%, while an exact SPLIT adds at most 4 points. (3) One per-call error rate describes how errors build up when small trained steps are combined into computations of 26 to 286 calls not seen in training; retraining COMBINE lowers it from 0.40% to 0.24%.Sec. 1, 2, 6
W2. The method does not fully solve multiplication.We agree and do not claim it. The revision states how accuracy falls with length and gives the reason: the number of calls grows as n².Sec. 6.3, Limitations
W2. The executor provides much of the algorithmic structure.We agree that the executor provides the control flow. The network still decides when to split, produces the split and writes every digit. With the same network, a flat model gets 2% on 8×8 and RAT gets 97%.Sec. 4.1, Table 9
W2. Theorem 1 is straightforward.The proof is indeed short. Its conclusion, however, depends on conditions that do not always hold in digit arithmetic, because digits wrap around modulo 10. If every term is off by +1, the digit is wrong for 1–9 terms and right again at 10; if errors are only 0 or 5, accuracy stays at 50%; if two errors cancel, the digit is right although both terms are wrong (App. D.5). Theorem 1 states the conditions under which the error must peak in the middle. Two of its predictions do not follow from column width alone, and both hold on the new baseline: at equal width, columns with a longer carry chain are less accurate (width 1: 79% vs. 98%), and the band of failing digits widens (digits below 50%: 0 of 10, 6 of 12, 10 of 14, 12 of 16 for n = 5 to 8). Proposition 3 extends the result to independent errors of any shape, and Theorem 2 drops independence and covers the actual greedy decoder.Sec. 5, App. B, D
Reviewer_e8iiOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the failure modes of Transformers on multi-digit integer multiplication. It shows that errors concentrate in a widening central band of middle digits. To address this, the authors propose the recursive arithmetic transformer (RAT), which uses a shared transformer decoder across BASE, SPLIT, and COMBINE frames coordinated by an external recursive executor. The paper provides theoretical analyse including a conditional digitwise error law, local carry-recovery rules, and multilayer stability certificate, as well as empirical evaluations involving oracle substitution and wider COMBINE frame adaptation.

Critical strengths
  • The experiments show that middle digits collapse to chance, but outer digits stay stable. This gives a better diagnosis than aggregate exact-match metrics.
  • Decomposing multiplication into base computation, splitting, and recombination allows for targeted interventions, and highlights the fact that recombination is the main performance bottleneck.
Critical weaknesses
  • RAT uses an external Python executor (Algorithm 1) to manage the recursion stack, parse the control tokens (SPLIT, CALL, COMBINE) and route intermediate values. Hence one cannot attribute the scaling improvement only to the transformer "intrinsic" arithmetic capability. This is not highlighted or elaborated in the paper.
  • The theory is unverified on the actually learned checkpoints.
Questions for authors
  1. Can the authors provide numerical measurements of the attention leakage and score gaps (Theorems 4 and 5) on the trained checkpoints, to verify if the theoretical stability conditions hold in practice?
  2. Since recursion tree size increases quickly with operand length, how does error accumulation across many recursive calls affect the the asymptotic scaling limits of RAT?
1 comment

We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised. All revisions made in response to the reviewers’ comments are highlighted in blue in the updated manuscript.

PointResponseWhere
W1. RAT uses an external executor, so the gains cannot be attributed to the Transformer alone.The paper now describes RAT as a hybrid system and lists exactly what the executor does (Table 9): it keeps the call stack, runs the four sub-calls in order, and passes their results to COMBINE. It does not write any digits. The network decides when to split, produces the split, and writes every digit, including carries. With the same network trained to write the product directly, 8×8 accuracy is 1.6%; RAT gets 97.1%.Abstract, Sec. 1, Sec. 4.1, Sec. 6.1, Table 9
W2. The theory is unverified on the learned checkpoints.We compared the error law with the new 3-seed baseline. Its qualitative predictions match: the least accurate digit is always in the middle, the band of failing digits widens with n, and columns with longer carry chains are less accurate. The simplest quantitative version does not match (it predicts 89% for one column where the model gets 54%), because the model's errors are correlated; Theorem 2 allows correlated errors. Section 5 now states this.Sec. 5
Q1. Can you report attention leakage and score gaps (Theorems 4 and 5) on the trained checkpoints, to check whether the stability conditions hold in practice?Theorems 4–5 give sufficient conditions under which the network carries out the local carry rule reliably. The revision presents them as conditions, not as measured properties of our checkpoints, and none of the experimental results depend on them.Sec. 5, Sec. 8
Q2. The recursion tree grows quickly with operand length. How does error accumulation across many calls affect the scaling of RAT?A single per-call error rate of 0.40% fits all six lengths: predicted exact match 90, 71, 65, 65, 65, 31% against measured 90, 73, 65, 61, 59, 36% for n = 8 to 18 (26 to 286 calls). Training COMBINE on 4× wider inputs lowers this rate to 0.24%, but because the number of calls grows as n², accuracy will keep falling for much longer inputs (expected ≈36% at n = 32 and ≈2% at n = 64). The first wrong step is COMBINE in 83% of failures at 9×9 and 93% at 10×10. We added this to the limitations.Sec. 6.2, 6.3, Limitations; code: footnote p. 1

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Clarifying Summary

Our paper asks where the errors fall when a Transformer multiplies long numbers, and which step of the computation produces them. These are the three main points raised by most reviewers, and how the revision addresses them.

  1. The middle-digit failure is real. We retrained the flat baseline with the same network as RAT. On 8×8 products it gets the first and last digits right 78% and 94% of the time, but the middle digits only 13–16%.

  2. The executor does no arithmetic. It only runs the recursion; the network writes every digit. With the same network, RAT gets 97% on 8×8 and the flat model gets 2%.

  3. COMBINE is the step that fails. When we give COMBINE the correct answer, 9×9 and 10×10 accuracy goes from about 52% to 92–96%. Training COMBINE on wider inputs improves every length from 8 to 18 digits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.