acceptodds
Under review as a conference paper at ICLR 2027

Computed Ahead, Copied Blind: Unchecked Handoffs in Chain-of-Thought Arithmetic

Abstract

When one position in a chain of thought hands a computed result to the next, does the receiver check it against the other input it can see? We study the final combination of two products in two models' own solutions to 749 linear systems, using sublayer readouts, attention cuts, and value substitutions. (1)Erroneous delivery passes unchecked on the copying route. The answer's first digit is computed at the equals position before it is written. A space receiver copies it through late attention, writing a donor's digit with probability near one with the correct operands in view. A minus receiver computes from the operands and shows no donor following. After operand damage, correct content at the equals position supports some answers, while most remain wrong. (2)The receiving state controls route selection and content influence. Changing the carrier changes input dependence. Cutting late equals reads makes copying fail; also cutting early equals reads restores operand support. Recovery reuses natural computation's operand-reading heads in both models. (3)Enabling operand computation and protecting its output are separate. With all operand reads at the equals position cut, cutting its remaining delivery restores further correct answers. On the computing route, delivered content shifts the output distribution while answers remain largely stable; after operand damage, it changes answers, mostly without donor following. Faithful transfer and effective verification are therefore different capabilities.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought, arithmetic reasoning, activation patching, mechanistic interpretability, causal interventions, attention knockout, linear probing, faithfulness, verification, large language models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Tr3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how large language models transfer intermediate arithmetic results between positions during chain-of-thought reasoning. Using Qwen3-14B and Qwen3-30B-A3B on 749 linear systems, the authors apply attention cuts, activation substitutions, and internal probes to identify two behaviors: copying a previously computed answer and computing an answer from the operands. The experiments suggest that the receiving token and its internal state determine which behavior is used. The paper argues that successfully transferring an intermediate result does not imply that the model can verify its correctness.

Critical strengths

The paper investigates a specific failure mechanism in chain-of-thought reasoning rather than treating reasoning errors as a single problem. The distinction between donor-answer copying and arithmetic-consistent mixed answers provides a useful way to identify how numerical information influences predictions.

The combination of attention cuts and value substitutions offers stronger causal evidence than activation probes alone. The recovery experiments are particularly interesting because they show that removing certain information paths can restore arithmetic computation and that recovered computation relies on heads also used during natural computation.

Critical weaknesses

The main experiments select almost entirely correct arithmetic predictions, making it difficult to establish whether the proposed unchecked handoff actually causes errors during natural reasoning rather than only under artificial interventions.

The comparison between space and minus receivers mixes token identity with answer sign and arithmetic context, so the claimed route-selection mechanism may partly reflect sign-specific processing rather than a general decision between copying and computation.

Questions for authors

How often does the claimed unchecked handoff occur when the model naturally computes an incorrect intermediate result, and can the authors demonstrate that the copying mechanism actually causes downstream reasoning errors?

Can the authors separate the effect of carrier identity from answer sign by testing equivalent arithmetic contexts with different token arrangements or controlled carrier changes in both directions?

5 comments
AuthorsOpenReview ↗

Answer to question 2

We thank the reviewer for the careful assessment! We agree that comparing natural space and minus receivers alone cannot isolate the role of the carrier. We therefore conducted two complementary tests: changing the carrier while holding the arithmetic fixed, and changing the information available to the same carrier.

1. Changing the carrier while keeping the calculation fixed. We tested both directions in Appendix B. For each expression, we kept the preceding text, operands, and true result fixed, changing only the space or minus token after the equals sign. For each token, we supplied the same donor's late equals values and measured whether the supplied first digit ranked highest among the ten digits. This comparison measures digit preference even when the model would naturally prefer a different sign.

Model and true resultSpace: donor digit ranks firstMinus: donor digit ranks first
14B, positive (566)566, 56652, 57
14B, negative (553)367, 3740, 0
30B, positive (235)235, 23231, 28
30B, negative (246)90, 790, 0

The two counts in each cell correspond to the two donors. On the same positive expression, forcing a minus sharply reduced donor following. On the same negative expression, forcing a space increased it. Across all four groups and both donors, whenever the minus ranked the donor digit first, the space did too. However, a forced space on negative expressions followed the donor less often than a natural space on positive expressions. Thus, carrier identity affects digit selection even with the arithmetic fixed, while sign and context also matter.

2. Changing the route while keeping the carrier fixed. To test whether the same carrier can use different information paths, we held both the sign of the result (positive) and the space token fixed. Cutting the space's late reads from the equals sign disrupted copying. When we also cut its early equals reads, the model recovered the complete correct answer on 562/566 expressions in 14B and 234/235 in 30B. Cutting operand reads disrupted this recovery, indicating that the recovered answers depended on information from the operands.

Together, these interventions separate two effects that were confounded in the natural comparison. Changing the carrier alters which source of information the model favors, but changing the available reads can also switch the behavior of the same space receiver from copying toward operand-dependent computation. We therefore do not claim that token identity alone determines the route. Rather, the receiver and its accessible information jointly shape whether a previously computed result is copied or recomputed.

AuthorsOpenReview ↗

Answer to question 1

We thank the reviewer for raising this question about naturally occurring errors and their downstream consequences.

To clarify this, correct line in a chain of thought can arise in two ways. The model can copy a result it computed earlier, or it can compute the result again from the operands. When both give the same digit, the written answer looks identical. We can tell which source the model uses by changing what the equals sign passes on while keeping the problem and its operands fixed. This lets us study how the model produces its ordinary, correct answers and predict what it will write when the two sources disagree.

How often did this happen naturally? We found no complete wrong sum or difference among 8,393 matches in the 14B solutions or 5,642 matches in the 30B prefixes used in the main experiments (three 30B matches ended at the token limit). We also screened the full 30B solutions at the space, before looking at the next digit. At 664 eligible positions the model wrote a positive two-digit determinant value for the first time. All 664 values were correct, and the equals-sign probe read the correct first digit in all 657 cases with a saved reading. The rate of copying a naturally wrong upstream result therefore remains unmeasured.

One later error illustrates what the intervention adds. In problem 120, the model first wrote 36 − (−40) = 36 + 40 = 76. On a later pass it wrote 367? and then corrected itself to 76. On replay, the probe read 7 at the equals sign before 367, yet six other problems all wrote 3 when their spaces received its late values. Putting the equals values from the correct 76 back into the original problem changed the space’s top prediction from 3 to 7. This locates the influence on the first digit. The valid rewrite also began with 36, so the first digit alone cannot tell us that the model had computed a wrong total.

To distinguish copying from computation when the two sources disagree, we next perturbed the upstream information without supplying an incorrect digit and tested whether removing the copying route restored the correct answer.

What does the copying route predict? With no digit supplied, we cut the equals sign’s reads from the operands in the main operand-reading layers. When the probe then read a wrong first digit, the space wrote that exact digit in 287/416 cases in 14B and 104/142 in 30B. In a separate test, we kept an all-layer cut on those operand reads in place. Cutting just the space’s late reads from the equals sign then corrected 150 of 158 first-digit errors in 14B and 4 of 7 in 30B. Every previously correct digit stayed correct.

We further tested whether this distinction extends beyond the original Qwen models and problems.

The same distinction predicted behavior in a second model family. For Ministral 3 8B Reasoning, we fixed the layers on the original 749 problems and tested 749 new ones (Appendix J). On all 623 eligible positive expressions, the space wrote whichever of two replacement digits we supplied at the equals sign. On all 639 negative expressions, the minus sign kept the original digit with either replacement. Cutting its operand reads reduced correct first digits from 639 to 271. These predictions were tested on expressions the model had originally answered correctly.

Finally, we examined whether the copied value affects subsequent arithmetic, rather than only the immediately generated digit.

Can copying change the next reasoning step? In one 30B solution, changing only the late equals values made the model write 51 in place of 21. It then used 51/24 = 17/8 to solve for the unknown. Keeping the changed values in place and restoring 21 in the text restored 21/24 = 7/8. The model divided correctly both times. Copying changed the number it divided. Across continuations with a trackable wrong numerator, the model used it in a division before the correct value reappeared within 160 tokens in 19/235 and 17/234 cases in 14B, and 14/139 and 8/133 in 30B (one count per donor). We counted wrong numbers with at least two digits that differed in magnitude from both operands. The same downstream use occurred when we cut operand reads and supplied no replacement digit (25/171 in 14B and 13/82 in 30B).

Thanks for the detailed responses and additional experiments! The new results help clarify the role of carrier identity and provide stronger evidence for the copying mechanism and its downstream effects. My main remaining concern is that its impact on naturally occurring errors is still unclear.

Reviewer_CBV6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The authors focus on a setting where early token positions in an LLM chain of thought (CoT) compute intermediate results and pass them to later positions via attention. They study whether the later positions re-check the received computation using the original inputs available in the context.
  • They instantiate this problem using simple arithmetic problems -- two-variable linear systems. There, they show that there exist two separate routes, which pass computation to the final receiver tokens: One, through a carrier token that copies the pre-computed result from another token, and another, where the receiver directly integrates the inputs and computes the answer on the fly.
  • Crucially, the authors answer their headline question, concluding that while LLMs can have both mechanisms active, they do not integrate the result, utilising only one route at a time.
  • They do this through a set of rigorous causal interventions, such as a thorough per-position mechanistic study of the problem, where they selectively ablate attention inputs to relevant token positions.
  • More broadly, their work helps to explain how redundancy in transformer computation is achieved, and what its limitations are.
Critical strengths
  1. Rigour of the experimental setup

The authors' design makes a clear separation between "readable" and "used" intermediate results. Every causal claim rests on an intervention, such as a cut or substitution with corresponding controls. The substitution design itself is very precise: donor digits are chosen to be different from the original answer and operand digits. In addition to donor experiments, operand substitutions allow an outcome that distinguishes between copying and arithmetic on inputs. The results section reports correctness for every unit against each unit's reference, making the conclusions more robust.

  1. The route-selection mechanism

In addition to demonstrating that two intermediate result routes exist, the authors show what makes the model "choose" between them. By cutting the receiver's early attention reads of the "equals" token (with the receiver's late reads already cut), they show that it re-enables operand computation at the copying position. This in turn restores accuracy from roughly a 25% to near perfect in both models. Authors' experiments show that whike redundant computation exists in the model, it does not by itself recover the written answer. The recovered computation does not improve upon the competing route until this route is completely ablated.

Critical weaknesses
  1. Positioning and broader impact

I think the manuscript would be stronger with an explanation of how its findings relate to relevant work and the broader interpretability literature.

  • The authors do not cite "Probing for Arithmetic Errors in Language Models" (Sun et al., EMNLP 2025), which also decodes the answer digit at the "equals" token before it is written. What is new in the setup here, relative to that work?
  • Similarly, "The Geometry of Self-Verification in a Task-Specific Reasoning Model" (Lee et al., 2025) identifies attention heads necessary for self-verification in a reasoning model. How does “no verification at the point of writing” relate to it?
Questions for authors
  1. [suggestion] Positioning and broader impact

Substantive answers to the questions and engagement with the suggestions raised in "Critical Weaknesses" would raise my score.

9 comments
AuthorsOpenReview ↗

Relation with Sun et al., ENMLP 2025

Sun et al. (2025) show that before a model writes a digit, probes at the same equals sign can read out both the correct digit and the digit the model will write, even when they differ. This leaves open where the different answers come from and how the model chooses which one among the different answers to write. In our Qwen3-30B-A3B model, an answer can already be read out at the last digit of the second operand. The equals sign reads this earlier answer, but the model usually does not use it as its written answer. When we replace what the equals sign reads from that position with values from another problem, the model rarely writes that problem’s first digit as long as its usual reads from the operands remain intact (Appendix F). In later layers, the equals sign computes a first digit from the operands. In both Qwen3-30B-A3B and Qwen3-14B models, for positive results, the space copies the digit computed at the equal sign; for negative results, the minus sign computes the digit again. The 30B minus sign thus gives a example of how at least three paths can bring answer information into the residual stream at one position. Through attention, it can read an answer formed at the last digit of the second operand and another formed at the equals sign. A third answer comes from the calculation at the minus sign itself. At the minus sign or space position, we can make the route that copies from the equals sign and the route that computes from the operands disagree. We replace the first digit passed from the equals sign and keep the operands unchanged. The space then writes the replacement digit in almost every case. The minus sign writes the correct digit. These experiments show where the different answers come from and which one determines what the model writes.

AuthorsOpenReview ↗

Relation with Lee et al., 2025

Lee et al. (2025). Lee et al. identify attention heads that a model uses to check whether a result it has already written matches the target number. We study the position just before the model first writes a number in a given role, such as the value of the determinant it is computing. The operands are already in the text, but the model has not written a value for that determinant. The candidate answers are only in the model’s residual stream. When we make the copying and computing routes disagree, the model writes a digit without requiring the two routes to agree. The space always copies the changed digit from the equals sign. The minus sign writes the digit it computes from the operands. This is how the model chooses a digit before the value appears in its chain of thought, and it is consistent with Lee et al.’s finding that a model can check a result after it has written it down. We see both stages in our own solutions. In one natural run, the model wrote 7 + 30 = 37, then checked the sum again and confirmed 37. When our intervention made it write 7 + 30 = 17 instead, the model later went back to the arithmetic on its own and wrote 7 + 30 = 37. In other cases, the model took the wrong value as given and used it in the next calculation without checking it.

We have added a brief comparison with both studies in Section 7 of the revised paper.

Reviewer_2cyYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the problem of how a computed result is handed off to the next position in a model's own chain of thought: whether the receiver copies the delivered result or computes the result from the operands, and whether the receiver can correct an incorrect delivery before writing the output. The experiments analyze the final combination of two products in Cramer's rule solutions to two-variable linear systems generated by two models. Attention cuts test dependence on particular reads, and value substitutions test how the content is used.

The space receiver copies, through the late delivery, the first digit computed ahead at the equals position, and follows the donor digit taken from another problem. The minus receiver computes from the operands and does not follow the donor. Cutting late equals reads makes copying fail. Also cutting early equals reads leads to recovery supported by the operands.

Critical strengths

1. Interventions that change the content distinguish content use in a certain handoff of a computed result within chain of thought

Strength. This paper investigates in detail the use of delivered content and the failure and recovery of the route in the handoff within the model's own chain of thought.

Evidence.

  • The authors combine attention cuts and value substitutions to examine dependence on particular reads, and the effect of changing the content on the output.
  • Under equals-value substitution, the space receiver copies the donor digit, while the minus receiver does not follow the donor. Under operand-value substitution, the output distribution moves toward the first digit arithmetically predicted from the injected last operand digit.
  • The attention cuts examine operand-supported recovery and its dependence on the selected operand-reading heads.

Why this matters for the overall assessment. These results support a causal distinction between the copying and computing routes. This paper follows not only the readout of internal information, but also how it is used and how the route fails and recovers. This gives a more detailed explanation of the local handoff and is a central contribution of this paper.

Critical weaknesses

1. The positioning in chain-of-thought studies and the novelty could be clearer

Weakness. I found it difficult to tell, as a statement of where the work sits, what question the local handoff results pose for chain of thought as a whole and what they show about that question. It is also difficult to tell whether the main novelty relative to the closest prior work lies in the problem setting, the intervention method, or the empirical findings.

Evidence. The abstract and Sections 1 and 2 show that the measurement target is the local handoff to the receiver immediately before the model begins writing the first digit of the determinant. Section 7 notes several differences from prior work, but it does not bring together how far this local result extends in chain-of-thought studies, or which of those differences is the main novelty.

Why this matters. This matters for the contribution and significance, because it concerns the scope of what the local result adds to chain-of-thought studies and what is distinctive relative to prior work.

2. The experimental scope and some choices of method could use a little more explanation

Weakness. A little more explanation would help clarify the scope of the experiments and the relationship between the two recovery experiments: correctness recovery in Section 5.2 and dependence on selected heads in Section 5.3. It would also be helpful to explain why the top 20 (head, layer) pairs were chosen to test whether recovery depends on the same heads as natural computation.

Evidence.

  • The two models differ in scale and structure, but both belong to the Qwen3 family. The reason for selecting them, and the scope beyond this family, are not explained.
  • Sections 5.2 and 5.3 both produce recovery by cutting late equals reads, but the layer ranges differ. The reason for the change is not stated.
  • For the top 2020 (head, layer) pairs, comparisons with a random twenty and the remaining pairs are reported. The reason for setting k=20k = 20, and the results for other values of kk, are not shown.

Why this matters. Restricting the evidence to the Qwen3 family limits the significance of the conclusions. The correspondence between recovery experiments run under different conditions, and stability with respect to top kk, bear on soundness. The limited explanation of why these conditions were chosen also affects clarity.

Questions for authors

1. Positioning within chain-of-thought studies and novelty

Could the authors add one account of where this work sits, covering the following?

  • Could you clarify where the local handoff to a receiver sits in chain-of-thought studies?
  • Compared with Dutta et al., Zhu et al., Bertolazzi et al., and Stolfo et al., where does the main novelty lie: in the problem setting, the intervention method, or the empirical findings?

2. Experimental scope and methods

Could the authors clarify the following points regarding interpretation and reproducibility?

  • Why were Qwen3-14B and Qwen3-30B-A3B selected, and what is the scope of the conclusions when only models from the same family are tested?
  • Why were the layers of the equals reads used to produce recovery in Section 5.3 changed from those in Section 5.2?
  • Could you define mean natural attention-write magnitude in one equation? I read it as an average over units, but it is unclear which equals queries from the positive and negative units and which natural-minus queries from the negative units enter the average, and how they are weighted.
  • Could you explain why the top 2020 (head, layer) pairs were selected and, if possible, also show the results when top kk is changed?
5 comments
AuthorsOpenReview ↗

Reply to question 1

We thank the reviewer for the thoughtful feedback! Our work is part of the white-box study of large language models. One question in this field is how different parts of a model work together to carry out a calculation. In a chain of thought, the result of one calculation can serve as an input to a later step. We ask how a model computes such a number and writes it into its own chain of thought. Our main contribution is an empirical finding about how copying and computation are selected and compete.

Relation to prior work

Dutta et al. In their logic task, some steps copy a fact from the question. Others use two facts to infer a new one. They test which attention heads help with each kind of step. We look inside a single arithmetic step. We locate the token position where the model first computes the answer’s first digit. We then test whether the position that writes the digit copies it from an earlier position or computes it again from the operands. We show how the model selects a route and when the two routes compete to determine which digit it writes.

Zhu et al. They change a number already written in a solution and let the model continue from the edit. This shows when later steps use the changed number. They also find cases where the model appears to copy a number directly. Consider our example, 7 + 30, where 30 is a product from an earlier step. Applied to this example, their method would change the written product from 30 to 36 and let the model continue. In our experiment, the text still reads 7 + 30. At selected layers, we replace the value vector for the 0 in 30 with the value vector for the 6 in another problem, 9 + 56 = 65.

Stolfo et al. They give evidence that the last position before the answer can use the operands to compute an answer. We find that this position can also copy a first digit already computed at the equals sign. In Qwen3, the same space in the same problem can switch between the two routes. The settings differ in three ways. First, they ask the model directly for the result of an operation such as a + b. In their two-operand tests with digits, one sixth of the prompts use a standard equation such as a + b =, and five sixths use different English phrasings. We study expressions that the model writes in standard equation form, such as a + b = c, while solving a larger problem. Second, their four models were either base models or tuned specifically for arithmetic. We use Qwen3 and, additionally in the revised version, Ministral 3 8B Reasoning, which received general instruction or reasoning training. Third, their arithmetic answers were positive. Our solutions contain both positive and negative answers, so we can test what the space and the minus sign each use to write the first digit.

Bertolazzi et al. They give the model a written arithmetic solution and ask it to check whether the solution is correct. We study the model as it writes its own solution.

AuthorsOpenReview ↗

Reply to question 2(1)

Why these models. Our tests call for one token per digit and full attention to earlier positions at every layer. We also wanted to compare a dense model and a mixture-of-experts model from the same family, with a size limit of about 60 GB.

We first considered the tokenizer and the attention layers. Llama 3.1 and Phi-4 encode a 2-digit number as one token. Gemma 4 uses local attention in some layers, and Qwen3.5 mixes linear and full attention. Several models meet both requirements, including the original LLaMA models, Llama 2, the original Gemma 2B and 7B models, and Qwen3. Llama 2 70B needs about 140 GB of 16-bit weights, and Mixtral 8×7B needs about 93 GB. Among the smaller candidates, the original LLaMA, Llama 2 and the original Gemma releases offer only dense models. Of the families we considered, this left Qwen3 with both model types within our budget. We chose the largest model of each type that fit: Qwen3-14B for the dense model and Qwen3-30B-A3B for the mixture-of-experts model.

Scope. Both Qwen3 models show the same pattern in these arithmetic steps: the natural space usually copies the first digit, the natural minus sign usually computes it from the operands, and the space can switch to operand-based computation and write the full answer correctly.

In response to this question, we ran an additional experiment on a second model family, Ministral 3 8B Reasoning (in Appendix J of the revised version of the paper). We chose the layer bands on 749 problems, then kept them fixed and tested 749 new problems. For each of two donors, replacing the late equals values made the space write the donor's first digit in all 623 positive expressions. The minus sign kept the original first digit in all 639 negative expressions, even when we replaced the equals values at every layer. We also changed the last digit of an operand. Here we used the 315 negative expressions where the original answer, the donor answer and the new calculation gave three different first digits. When only the minus sign could read that change, it wrote the first digit predicted by the new calculation in 156/315 and 139/315 negative expressions for the two donors. When only the equals sign could read it, the counts were 0/315 for both. This gives evidence for both copying and computation in a second family on the same arithmetic step.

AuthorsOpenReview ↗

Reply to question 2(2)

Why did the equals-read layers differ between Sections 5.2 and 5.3?

In our first experiments, we cut the space's reads of the equals sign across a wide range of late layers to rule out copying through nearby layers. We also used a wider early equals-read cut in 14B to test whether the space could recover, and a wider operand-read cut in 30B to check that the recovered answer depended on the operands. The original Section 5.3 used these settings.

Once we had established that the space could recover, we used the standard copying and operand-reading bands and shortened the early equals-read cut in 14B. These were the settings in the original Section 5.2. We have now repeated the Section 5.3 head tests with the same bands.

ModelReads cut at the spaceOriginal Section 5.3Revised Sections 5.2 and 5.3
14BLate reads of the equals sign33–4037–40
14BEarly reads of the equals sign1–141–12
14BReads of the operands28–3128–31
30BLate reads of the equals sign39–4847–48
30BEarly reads of the equals sign1–71–7
30BReads of the operands39–4441–44

The original results still hold in both models with the matched bands. Appendix D (Table 14) reports the recovery test with the wider late cut used to rule out copying through those extra layers.

Reviewer_CrRgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies internal information handoffs in Qwen3-14B and Qwen3-30B-A3B while solving 749 systems of two linear equations using Cramer's rule, focusing on steps that combine two products into a two-digit result. Using linear probes, attention-read ablations, value substitutions, and controls over where substituted information is exposed, the authors find that natural space carriers tend to copy a first answer digit already represented at the equals-sign position, whereas natural minus-sign carriers rely more on operand-based computation. When late reads from the equals sign have been ablated, additionally removing early reads restores operand-supported performance. The experiments provide useful causal evidence separating answer transmission, local computation, and their effects on the output, within the specific prediction endpoints studied.

Critical strengths
  1. The causal interventions distinguish copying from operand-based computation, rather than merely showing that an answer is decodable. Equals-sign value substitutions make all 566/566 positive-result cases in the 14B model follow the donor's first digit; the corresponding counts for the two donors in the 30B model are 235/235 and 232/235. Neither model follows the donor's first digit in natural negative-result cases. Operand substitutions further distinguish original, donor, and mixed answers, while exposure controls help locate where the substituted information is used. Together, these interventions provide substantially stronger evidence about information dependence than an accuracy drop or a probe result alone.

  2. The recovery experiments have large effects and address several direct alternative explanations. With late equals-sign reads already ablated, also removing early reads restores first-digit accuracy from 131/566 to 562/566 in the 14B model and from 68/235 to 235/235 in the 30B model; ablating operand reads disrupts this recovery. The semantic operand set includes earlier repeated mentions, and the paper supplies random-position and random-head controls. It also separates removal of equals-sign writes from softmax renormalization, examines heads shared by natural and recovered computation, clusters statistics by problem, and uses baselines from the same run. These controls make the main copying/recovery result persuasive and materially support acceptance.

Critical weaknesses
  1. The experimental scope limits generalization. The study examines two models from the same family using fixed Cramer’s-rule prefixes and two-digit answers. These experiments provide useful local mechanistic evidence, but the conclusions should remain specific to this setting rather than imply a general account of arithmetic computation.

  2. First-digit recovery does not establish complete-answer recovery. First-digit prediction receives the recorded carrier, while second-digit prediction receives the ground-truth first digit. Thus, approximately 99% first-digit recovery does not demonstrate successful generation of a complete signed answer. Reporting accuracy when the model generates the carrier and both digits itself would clarify the practical extent of recovery.

  3. Carrier identity remains entangled with sign and receiving state. Natural spaces accompany positive results, whereas minus signs accompany negative results. The forced-carrier controls do not produce a symmetric route swap: forcing a space can trigger another minus sign, while forcing a minus sign can retain dependence on equals-sign reads (Tables 6–7). The “space copies / minus computes” interpretation should therefore remain context-dependent.

  4. The smaller effects need targeted numerical-stability checks. Appendix G tests only five cases, with three of 225 top-token comparisons changing and a maximum probability difference of 0.094. This raises uncertainty about the smaller effects involving 7–8/246 correct-to-incorrect outcomes. Rechecking these cases under fixed batching and higher precision would strengthen the conclusions without calling the much larger copying and recovery effects into question.

Questions for authors
  1. Endpoint clarification and targeted validation: Under the main recovery conditions, what is the accuracy of the complete signed numerical answer when the model generates its own carrier, first digit, and second digit? Please distinguish the reported conditional endpoints from complete-answer recovery. A small evaluation would help determine how far the recovery claim extends.

  2. Mechanism clarification: How do the asymmetric forced-carrier results in Tables 6–7 constrain the interpretation of route selection? Can sign- and format-matched variants of the same numerical calculation distinguish the carrier token's effect from sign semantics and the need to output a sign?

  3. Stability validation: Do Section 6's 7–8/246 correct-to-incorrect changes and smaller probability shifts persist with fixed batching, single-condition execution, or higher precision? Does the maximum probability difference of 0.094 occur in any switching cases on which the conclusions depend?

  4. Reproducibility and selection clarification: Please specify the generation or source of the 749 problems, probe targets and regularization, model and execution versions, and the procedure used to select layer intervals. After fixing those intervals, were the findings checked on additional problems or donors? An anonymized minimal reproduction package would help resolve these questions.

4 comments
AuthorsOpenReview ↗

Reply to question 1

We thank the reviewer for the thoughtful feedback!

Complete-answer recovery (§5.2). When both early and late reads from the equals-sign token were cut, the model generated the complete correct answer for 562/566 positive expressions (99.3%) in 14B and 234/235 (99.6%) in 30B. When only late reads were cut, the corresponding counts were 126/566 (22.3%) and 66/235 (28.1%).

In the original submission, we supplied the recorded space to test the first digit and the recorded first digit to test the second. In the new experiment, each model started at the equals sign and generated its own most likely token at each of six steps. The space was generated with all reads intact. The cuts began when the model predicted the first digit and remained active for the rest of the generation. We counted an answer as correct only if the model generated the space, both correct digits, and a non-digit immediately after the ones digit. We have added these results to §5.2 and the table in Appendix D.

We will address the remaining questions in follow-up comments later.

AuthorsOpenReview ↗

Reply to question 3

We rechecked all 246 negative-result expressions in Qwen3-30B-A3B. At each precision (BF16 and FP32), we ran the 15 conditions together and the target conditions one at a time. In each setting, we compared each donor substitution with the damaged reference from that same setting.

The effect on the first digit held across all four settings, but some individual answers changed. In the original BF16 batch, donor 1 turned 7 correct first digits into errors, and donor 2 turned 8 into errors. In BF16 single-condition runs, 4 of the original 7 and 6 of the original 8 still switched; including new cases, the totals were 6 and 8. In FP32, 2 of the original 7 and 3 of the original 8 still switched; the totals were 4 and 6. The batched and single-condition FP32 runs gave the same switches. For each donor, four original switches disappeared because the first digit was already wrong before substitution in FP32.

To measure the effect even when the top digit stayed the same, we compared the score of the correct digit with the highest score among the other nine digits. We call the difference its margin. In all four settings, substitution lowered this margin in every original switch case. Across all 246 expressions, the size of the effect was also consistent:

Change in the correct digit’s margin (logit units)Donor 1Donor 2
Mean loss after substitution, range across the four settings0.977–0.9851.037–1.051
Median absolute change in that loss when switching from batched to single-condition runs at BF160.2500.250
Median absolute change in that loss when switching from BF16 to FP32 in the same 15-condition batch0.1560.177

The 0.094 was a difference between batched and single-condition runs at the same BF16 precision. It occurred on problem 336 (−56 + 42 = −14), in the probability assigned to a period when predicting the second digit. This expression contributed to neither donor’s first-digit switch count.

AuthorsOpenReview ↗

Reply to question 2

In the revised Appendix B. we change the carrier between space and minus in both directions.

A forced minus for positive results shows a weak evidence of using the operands to compute. We found that the probe for negative results tend to predict the correct answer from the middle to late layers, but the prediction accuracy decreases at the last 2 or 3 layers. We then probed the forced minus at layers chosen from natural minus signs, layer 32 in 14B and layer 44 in 30B and testing the result of cutting and substitution by the probes. Cutting its operand reads reduced correct probe readouts from 222 to 101 out of 566 in 14B and from 97 to 64 out of 235 in 30B. Cutting reads from equally many random positions gave 223 and 98. This shows that the operands help form the digit we can read at the minus. However, the evidence for recomputing the digit is weak: when only the minus saw a changed operand, the original digit lost probe score, but the digit predicted by the new arithmetic rarely gained first place. A guess of the mechanism would be the forced minus is computing (by the probe readouts in layer 32 in 14B and layer 44 in 30B in the cutting experiment), but then the model refuse the result of computation for some reason. Anyway, we think the forced minus for positive results is too artificial, while the forced space for negative results is less artificial because the model can write an additional minus after the forced space. This may also be a reason why the behaviour of a forced space can be better interpreted than the behaviour of a forced minus.

AuthorsOpenReview ↗

Reply to question 4

We upload anonymous attachment that includes exact prompts, the problem generator, probe settings, and per-expression predictions. With NumPy installed, python check.py --split-half reproduces the checks on other problems and donors reported below.

We assessed layer-selection sensitivity through three checks: excluding layer-selection problems in 14B, testing split-half layer stability in 30B, and evaluating fixed intervention layers on 749 new problems in a second model family. The last provides the clearest test: changing the digit supplied by the equals sign let us choose which first digit the space wrote. The minus sign kept writing the correct digit under the same change, and its accuracy depended on its reads from the operands.

Problems and models. We generated 2,172 systems of two linear equations and used the first 749 accepted systems. We drew the four coefficients and two right-hand sides independently from −9 to 9. Zero had probability 2/20 and each other integer had probability 1/20. We excluded singular systems, equations whose coefficients and right-hand side shared a factor greater than one, and duplicate systems up to swapping the equations. Both Qwen models received the same prompt tokens. Qwen3-14B used greedy decoding. Qwen3-30B-A3B generated one solution per problem at temperature 0.6, top-p 0.95 and top-k 20. The main interventions used bfloat16, PyTorch 2.13.0+cu130 and Transformers 5.16.0.dev0 on one 80 GB A100 for 14B and two for 30B. The original 30B text generation used vLLM 0.8.5.post1, PyTorch 2.6.0+cu124 and Transformers 4.51.3.

Probes. Each linear probe predicted the tens digit or the ones digit of the absolute result. We fit separate probes for each sign and depth at the last digit of the second operand, the equals sign, and the following space or minus sign. We split problems into five folds, keeping all expressions from a problem together. Within each training set, we tested eight L2 penalties (0.0001, 0.001, 0.01, 0.1, 1, 10, 30, 100) and chose the one with the lowest cross-entropy on separate validation problems.

How we chose the Qwen layers. Attention patterns and probe results guided the search. We also used the observed effects of cuts and swaps to choose several intervals and the single layer used for copying.

ReadsQwen3-14BQwen3-30B-A3B
OperandsWrites from operand positions peaked in 28–31. Cutting 22–27 also reduced accuracy, so we included those layers in the wider cut, 22–31.Attention pointed to 39–48. Digit readout rose after attention at 40 and after the feed-forward step at 44. Separate cuts tested 39–40 and 41–44. The wider cut covered 39–44.
Late reads from the equals signWrites from the equals sign to the space rose in 37–40. Swapping layer 40 alone made the space write the supplied digit most often.Cuts identified 47–48. Layer 48 gave the most following in the single-layer scan. Swapping both layers increased following for each donor.
Early reads from the equals signWith late reads cut, ending the early cut at any layer from 11 through 15 gave near-full recovery. We chose 1–12 in that range.We kept the original 1–7 interval. A later scan gave 235/235 correct first digits when the early cut ended at 5, 7 or 10.

14B. Of the 610 main problems, 192 appeared in two early groups used to choose layers. We reanalyzed the original runs on the other 418. At layer 40 the space wrote the supplied first digit in 393/393 positive expressions for each donor. The minus sign followed either donor in 0/381 negative expressions. Requiring both donors’ problems also to fall outside the early groups gave 162/162 and 0/160 for each donor. Cutting operand reads in layers 28–31 left 393/393 positive first digits correct at the space and 170/381 negative first digits correct at the minus sign.

30B. We checked the single-layer copying result within the original pool by splitting positive expressions by problem 2,000 times. In each split we chose the layer with the most donor following on one half (layers 45–48 for donor 1, 47–48 for donor 2) and tested it on the other half. Layer 48 was selected every time for both donors. On the test halves the space wrote the supplied digit in 89.3% and 91.1% of expressions on average.

8B on new problems (Appendix J). For Ministral 3 8B Reasoning, we chose layers 26–29 for operand reads and 31–34 for late equals reads on problems 1–749. We fixed both bands before testing problems 750–1498 and took both donors from the new set. On all 623 positive expressions, supplying either of two different digits at the equals sign made the space write that digit. On all 639 negative expressions, the minus sign kept the original first digit with either donor. Cutting its operand reads in layers 26–29 reduced accuracy from639/639 to 271/639. Cutting reads from the same number of random positions left 639/639 correct.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.