- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates where and how large language models produce sign errors during multi-step mathematical computations. Across six open models from five families, the authors report that sign decisions are predominantly committed by MLP layers in the final quarter of the network. They identify several mechanisms, including layers that consistently favor a minus sign, layers that amplify an existing sign preference, and downstream layers that counteract earlier biases.
Critical strengths
- This work focuses on an interesting question about the inner mechanisms of LLMs in mathematical calculation.
- Experiments span multiple LLM families and two mathematical tasks, providing substantial support for the main findings and revealing meaningful model-specific differences.
Critical weaknesses
- The writing is confusing. For example, it is unclear what ''70–100% per model and task'' measures, while ''24–70% per cell'' does not immediately identify the experimental unit or denominator. I recommend organizing the abstract more clearly around the research question, approach, principal findings, and implications, while reducing unexplained terminology and parenthetical statistics.
- The interpretation appears to move too quickly from layer-wise readouts and sign-direction interventions to identifying where the erroneous mathematical decision originates. Late-layer activity could reflect the amplification or output encoding of a decision formed earlier, rather than the source of the error itself.
- The discussion of related work is insufficient. In particular, the paper omits the relevant study ''Interpreting and Improving Large Language Models in Arithmetic Calculation.'' The authors should discuss this work and clarify how their approach and findings differ from it.
Questions for authors
Refer to weakness.
3 comments
Response to Reviewer CyA5 (2/3): W2 (committed late vs decided late)
(Part 2/3 of our response to Reviewer CyA5: W2.)
W2: "The interpretation appears to move too quickly from layer-wise readouts and sign-direction interventions to identifying where the erroneous mathematical decision originates. Late-layer activity could reflect the amplification or output encoding of a decision formed earlier, rather than the source of the error itself."
You are right, and we lay out why step by step.
What we measured. At the position just before the sign, the logit lens reads, at every layer, how strongly the residual stream already favours the written sign over the other one; direct logit attribution then credits each layer's MLP with its share of that lead (§3–§4).
What we found. In every model the lead is near zero through the middle of the network and rises steeply only in the last quarter of layers; a few named late layers carry most of the credit (Figures 1–2).
What we claimed. §3 says that an error computed earlier "would be visible mid-network; nothing is", and the paper then speaks of where the decision is made.
Why your objection is correct. The logit lens reads only the part of the residual stream that is already aligned with the output tokens. A decision formed earlier, in a direction the output layer does not yet read, is invisible to it. And all our measurements are at one token position; for a cofactor sign the parity (i + j) is determined several tokens upstream, where we did not look. So our evidence shows that the sign is committed late — written into the output-aligned direction in the last quarter — not that it is decided late.
What still shows the late layers are not only encoding. Three results: (i) the stamp layer pushes «−» even on correct «+» answers (Qwen L75: −1.02 [−1.25, −0.76], n = 29; Table E2), so it is a standing bias rather than the encoding of a decision; (ii) patching the healthy message from a correct run into an error at those layers repairs nothing, because the healthy and the error messages are the same there (§5) — nothing error-specific arrives from upstream at that position; (iii) in a new experiment run during the discussion (reply to Reviewer 8g4g), removing each late layer's own minus component across Phi-4's full late window, injecting nothing, flips 23% of determinant errors and 11% of IBP errors, concentrated in low-margin cases. The late layers therefore carry at least part of the decision; whether the rest of the margin is an upstream decision or further late writing, these data cannot say.
What changes. In the camera-ready version, §1, §3, §7 and the Limitations say "committed late; where the decision is formed is open," and the sentence quoted above is removed.
What we will do. The direct test is a linear probe trained to read the correct sign (not the written one) from the residual stream at earlier layers and at the upstream token where the parity is determined, followed by activation patching at that token from matched correct runs. If the correct sign is readable upstream and patching there flips the output, the decision is upstream and the late layers encode it; if not, the late layers decide. This is the first experiment of the next version.
(Part 2/3. Part 3/3 answers W3.)
Response to Reviewer CyA5 (3/3): W3 (Zhang et al. 2024 and related work)
(Part 3/3 of our response to Reviewer CyA5: W3.)
W3: "The discussion of related work is insufficient. In particular, the paper omits the relevant study ''Interpreting and Improving Large Language Models in Arithmetic Calculation.'' The authors should discuss this work and clarify how their approach and findings differ from it."
Added in the camera-ready version (Zhang et al., ICML 2024), as follows.
What they do. For two-operand arithmetic ("3 + 5 =") answered at the final token, they find that a small set (under 5%) of attention heads attends to the operands and the operator, that MLPs then process that information, and that selectively fine-tuning those heads and MLPs improves arithmetic accuracy without harming non-mathematical tasks.
What we do. We study one wrong sign token in the middle of a 4,000-character derivation, where the sign is dictated by a rule the model itself set up earlier (cofactor parity; the parts formula), and we compare error cases with correct cases on the same layers.
How the findings relate. (i) Both find a sparse set of late components with MLPs doing the output-side work. (ii) They validate their localization by fine-tuning the located components, and it works; we test activation steering at the located site and find that its success depends on how much of the circuit is covered and on the margin — complementary results, which together suggest weight-space repair may succeed where low-coverage steering fails; that fine-tuning experiment is our committed next step (reply to Reviewer CN8w, W3). (iii) Their probe is the final token of a short prompt; ours is a token deep inside the model's own text, which is exactly why your W2 — a decision formed upstream — matters more in our setting than in theirs.
We also add Hanna et al. (NeurIPS 2023), who show GPT-2's final MLPs boosting the correct answer class, as a second late-MLP precedent, and sharpen the positioning against Hase et al. (NeurIPS 2023) as in our reply to Reviewer 8g4g.
Response to Reviewer CyA5 (1/3): what the paper studies, and W1 (writing and the two numbers)
We thank the reviewer. Your summary of the paper is accurate, and your three points are fair. We answer each step by step, say what changes in the camera-ready version, and say what we will do next.
What the paper is about. The errors we study occur inside long, recursive, multi-step derivations that the model writes itself — a 4×4 determinant unfolding into four 3×3 and then 2×2 determinants with a sign rule at every level; an integration by parts applied up to fourteen times, each application adding a minus. The solutions run about 4,000 characters. The model writes one wrong sign token mid-way; the remaining steps operate on the corrupted value and the final answer is wrong. The paper asks where in the network that one mid-computation sign is written and why it comes out wrong, by replaying the model's own derivation token by token and comparing errors against correct answers on the same layers.
W1: "The writing is confusing. For example, it is unclear what ''70–100% per model and task'' measures, while ''24–70% per cell'' does not immediately identify the experimental unit or denominator. I recommend organizing the abstract more clearly around the research question, approach, principal findings, and implications, while reducing unexplained terminology and parenthetical statistics."
Agreed. Here is what the two numbers mean, which the abstract should have said.
"70–100% per model and task." For each error case we replay the model's own solution and, at the position just before the wrong sign, ask every layer of the network which sign it would write (the logit lens). The layer where the wrong sign's lead is largest is that case's peak. A case is "late" if its peak lies in the last quarter of the layers. For each of the ten model×task cells (five models × two tasks) we take the share of confirmed error cases that are late: it is 70% in the lowest cell and 100% in the highest (Figure 1). The denominator is confirmed error cases in that cell (for example, Phi determinants: 137 cases).
"24–70% per cell." For each cell, we subtract the sign direction at the two identified layers (α=3) on the held-out minus-written error cases and count how many now prefer the correct sign. That share is 24% in the lowest and 70% in the highest of the Qwen, Gemma and Mistral-Small cells (Appendix I; the denominator is held-out minus-written errors in that cell).
Changes in the camera-ready version: the abstract is rewritten in the order you recommend — question, approach, principal findings, implications — with three numbers instead of twelve, each with its unit and denominator. Invented vocabulary ("habit layers", "judges", "crime scene") is removed or defined on first use; "stamp" and "amplifier" are kept, because the distinction is a result, and are defined where they first appear.
(Part 1/3. Part 2/3 answers W2; part 3/3 answers W3.)