- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how large language models transfer intermediate arithmetic results between positions during chain-of-thought reasoning. Using Qwen3-14B and Qwen3-30B-A3B on 749 linear systems, the authors apply attention cuts, activation substitutions, and internal probes to identify two behaviors: copying a previously computed answer and computing an answer from the operands. The experiments suggest that the receiving token and its internal state determine which behavior is used. The paper argues that successfully transferring an intermediate result does not imply that the model can verify its correctness.
Critical strengths
The paper investigates a specific failure mechanism in chain-of-thought reasoning rather than treating reasoning errors as a single problem. The distinction between donor-answer copying and arithmetic-consistent mixed answers provides a useful way to identify how numerical information influences predictions.
The combination of attention cuts and value substitutions offers stronger causal evidence than activation probes alone. The recovery experiments are particularly interesting because they show that removing certain information paths can restore arithmetic computation and that recovered computation relies on heads also used during natural computation.
Critical weaknesses
The main experiments select almost entirely correct arithmetic predictions, making it difficult to establish whether the proposed unchecked handoff actually causes errors during natural reasoning rather than only under artificial interventions.
The comparison between space and minus receivers mixes token identity with answer sign and arithmetic context, so the claimed route-selection mechanism may partly reflect sign-specific processing rather than a general decision between copying and computation.
Questions for authors
How often does the claimed unchecked handoff occur when the model naturally computes an incorrect intermediate result, and can the authors demonstrate that the copying mechanism actually causes downstream reasoning errors?
Can the authors separate the effect of carrier identity from answer sign by testing equivalent arithmetic contexts with different token arrangements or controlled carrier changes in both directions?
5 comments
Answer to question 1
We thank the reviewer for raising this question about naturally occurring errors and their downstream consequences.
To clarify this, correct line in a chain of thought can arise in two ways. The model can copy a result it computed earlier, or it can compute the result again from the operands. When both give the same digit, the written answer looks identical. We can tell which source the model uses by changing what the equals sign passes on while keeping the problem and its operands fixed. This lets us study how the model produces its ordinary, correct answers and predict what it will write when the two sources disagree.
How often did this happen naturally? We found no complete wrong sum or difference among 8,393 matches in the 14B solutions or 5,642 matches in the 30B prefixes used in the main experiments (three 30B matches ended at the token limit). We also screened the full 30B solutions at the space, before looking at the next digit. At 664 eligible positions the model wrote a positive two-digit determinant value for the first time. All 664 values were correct, and the equals-sign probe read the correct first digit in all 657 cases with a saved reading. The rate of copying a naturally wrong upstream result therefore remains unmeasured.
One later error illustrates what the intervention adds. In problem 120, the model first wrote 36 − (−40) = 36 + 40 = 76. On a later pass it wrote 367? and then corrected itself to 76. On replay, the probe read 7 at the equals sign before 367, yet six other problems all wrote 3 when their spaces received its late values. Putting the equals values from the correct 76 back into the original problem changed the space’s top prediction from 3 to 7. This locates the influence on the first digit. The valid rewrite also began with 36, so the first digit alone cannot tell us that the model had computed a wrong total.
To distinguish copying from computation when the two sources disagree, we next perturbed the upstream information without supplying an incorrect digit and tested whether removing the copying route restored the correct answer.
What does the copying route predict? With no digit supplied, we cut the equals sign’s reads from the operands in the main operand-reading layers. When the probe then read a wrong first digit, the space wrote that exact digit in 287/416 cases in 14B and 104/142 in 30B. In a separate test, we kept an all-layer cut on those operand reads in place. Cutting just the space’s late reads from the equals sign then corrected 150 of 158 first-digit errors in 14B and 4 of 7 in 30B. Every previously correct digit stayed correct.
We further tested whether this distinction extends beyond the original Qwen models and problems.
The same distinction predicted behavior in a second model family. For Ministral 3 8B Reasoning, we fixed the layers on the original 749 problems and tested 749 new ones (Appendix J). On all 623 eligible positive expressions, the space wrote whichever of two replacement digits we supplied at the equals sign. On all 639 negative expressions, the minus sign kept the original digit with either replacement. Cutting its operand reads reduced correct first digits from 639 to 271. These predictions were tested on expressions the model had originally answered correctly.
Finally, we examined whether the copied value affects subsequent arithmetic, rather than only the immediately generated digit.
Can copying change the next reasoning step? In one 30B solution, changing only the late equals values made the model write 51 in place of 21. It then used 51/24 = 17/8 to solve for the unknown. Keeping the changed values in place and restoring 21 in the text restored 21/24 = 7/8. The model divided correctly both times. Copying changed the number it divided. Across continuations with a trackable wrong numerator, the model used it in a division before the correct value reappeared within 160 tokens in 19/235 and 17/234 cases in 14B, and 14/139 and 8/133 in 30B (one count per donor). We counted wrong numbers with at least two digits that differed in magnitude from both operands. The same downstream use occurred when we cut operand reads and supplied no replacement digit (25/171 in 14B and 13/82 in 30B).
Thanks for the detailed responses and additional experiments! The new results help clarify the role of carrier identity and provide stronger evidence for the copying mechanism and its downstream effects. My main remaining concern is that its impact on naturally occurring errors is still unclear.
Answer to question 2
We thank the reviewer for the careful assessment! We agree that comparing natural space and minus receivers alone cannot isolate the role of the carrier. We therefore conducted two complementary tests: changing the carrier while holding the arithmetic fixed, and changing the information available to the same carrier.
1. Changing the carrier while keeping the calculation fixed. We tested both directions in Appendix B. For each expression, we kept the preceding text, operands, and true result fixed, changing only the space or minus token after the equals sign. For each token, we supplied the same donor's late equals values and measured whether the supplied first digit ranked highest among the ten digits. This comparison measures digit preference even when the model would naturally prefer a different sign.
The two counts in each cell correspond to the two donors. On the same positive expression, forcing a minus sharply reduced donor following. On the same negative expression, forcing a space increased it. Across all four groups and both donors, whenever the minus ranked the donor digit first, the space did too. However, a forced space on negative expressions followed the donor less often than a natural space on positive expressions. Thus, carrier identity affects digit selection even with the arithmetic fixed, while sign and context also matter.
2. Changing the route while keeping the carrier fixed. To test whether the same carrier can use different information paths, we held both the sign of the result (positive) and the space token fixed. Cutting the space's late reads from the equals sign disrupted copying. When we also cut its early equals reads, the model recovered the complete correct answer on 562/566 expressions in 14B and 234/235 in 30B. Cutting operand reads disrupted this recovery, indicating that the recovered answers depended on information from the operands.
Together, these interventions separate two effects that were confounded in the natural comparison. Changing the carrier alters which source of information the model favors, but changing the available reads can also switch the behavior of the same space receiver from copying toward operand-dependent computation. We therefore do not claim that token identity alone determines the route. Rather, the receiver and its accessible information jointly shape whether a previously computed result is copied or recomputed.