acceptodds
Under review as a conference paper at ICLR 2027

Localization Does Not Imply Steerability: A Mechanistic Study of Sign Errors in LLM Mathematics

Abstract

When large language models do multi-step mathematics, a characteristic failure recurs: the model writes one wrong sign token mid-computation — a cofactor sign, the minus of a recursive integration-by-parts step — then executes the rest of the arithmetic correctly, and the answer comes out simply wrong. We ask where inside the network that sign is written. Across six open models spanning 14B to 675B parameters, a per-layer readout shows the wrong sign is committed late — by MLP layers in the last quarter of the network (70–100% per model and task), a pattern already present, where tested (Qwen), in the base model before instruction tuning. The mechanism varies: some models carry fixed “minus-stamp” layers that push a minus regardless of the right answer, others carry amplifiers of whichever sign was already chosen, and Qwen adds a downstream rescue layer that opposes the stamp. The circuit is causal: subtracting a single sign direction from the residual stream flips 13 of 14 held-out minus-written integration-by-parts errors on Llama- 3.3-70B, with flips replicating on Qwen, Gemma, and Mistral-Small (24–70% per cell), and with representation-level specificity — the same dose breaks correctlyminus answers while sparing correct-plus answers where the direction is surgical (0/72 on Llama; Mistral, the outlier, breaks every cohort), and matched randomdirection and control-layer interventions are null. Core claims were preregistered with pass criteria frozen in git before held-out data was seen: 23 of 29 checks passed, all misses reported. But causality is not uniform: Phi-4 implements an equally well-localized circuit at the same late depth, yet the identical intervention flips almost nothing—the sign is written redundantly across ∼15 layers with margins several times larger (median 17.8 logits). Localization, we conclude, does not imply steerability: two models can share a circuit location yet differ enormously in how overturnable it is.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation steering, residual stream intervention, sign errors, mathematical reasoning, logit lens, circuit analysis, representation robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CyA5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates where and how large language models produce sign errors during multi-step mathematical computations. Across six open models from five families, the authors report that sign decisions are predominantly committed by MLP layers in the final quarter of the network. They identify several mechanisms, including layers that consistently favor a minus sign, layers that amplify an existing sign preference, and downstream layers that counteract earlier biases.

Critical strengths
  1. This work focuses on an interesting question about the inner mechanisms of LLMs in mathematical calculation.
  2. Experiments span multiple LLM families and two mathematical tasks, providing substantial support for the main findings and revealing meaningful model-specific differences.
Critical weaknesses
  1. The writing is confusing. For example, it is unclear what ''70–100% per model and task'' measures, while ''24–70% per cell'' does not immediately identify the experimental unit or denominator. I recommend organizing the abstract more clearly around the research question, approach, principal findings, and implications, while reducing unexplained terminology and parenthetical statistics.
  2. The interpretation appears to move too quickly from layer-wise readouts and sign-direction interventions to identifying where the erroneous mathematical decision originates. Late-layer activity could reflect the amplification or output encoding of a decision formed earlier, rather than the source of the error itself.
  3. The discussion of related work is insufficient. In particular, the paper omits the relevant study ''Interpreting and Improving Large Language Models in Arithmetic Calculation.'' The authors should discuss this work and clarify how their approach and findings differ from it.
Questions for authors

Refer to weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CyA5 (1/3): what the paper studies, and W1 (writing and the two numbers)

We thank the reviewer. Your summary of the paper is accurate, and your three points are fair. We answer each step by step, say what changes in the camera-ready version, and say what we will do next.

What the paper is about. The errors we study occur inside long, recursive, multi-step derivations that the model writes itself — a 4×4 determinant unfolding into four 3×3 and then 2×2 determinants with a sign rule at every level; an integration by parts applied up to fourteen times, each application adding a minus. The solutions run about 4,000 characters. The model writes one wrong sign token mid-way; the remaining steps operate on the corrupted value and the final answer is wrong. The paper asks where in the network that one mid-computation sign is written and why it comes out wrong, by replaying the model's own derivation token by token and comparing errors against correct answers on the same layers.

W1: "The writing is confusing. For example, it is unclear what ''70–100% per model and task'' measures, while ''24–70% per cell'' does not immediately identify the experimental unit or denominator. I recommend organizing the abstract more clearly around the research question, approach, principal findings, and implications, while reducing unexplained terminology and parenthetical statistics."

Agreed. Here is what the two numbers mean, which the abstract should have said.

"70–100% per model and task." For each error case we replay the model's own solution and, at the position just before the wrong sign, ask every layer of the network which sign it would write (the logit lens). The layer where the wrong sign's lead is largest is that case's peak. A case is "late" if its peak lies in the last quarter of the layers. For each of the ten model×task cells (five models × two tasks) we take the share of confirmed error cases that are late: it is 70% in the lowest cell and 100% in the highest (Figure 1). The denominator is confirmed error cases in that cell (for example, Phi determinants: 137 cases).

"24–70% per cell." For each cell, we subtract the sign direction at the two identified layers (α=3) on the held-out minus-written error cases and count how many now prefer the correct sign. That share is 24% in the lowest and 70% in the highest of the Qwen, Gemma and Mistral-Small cells (Appendix I; the denominator is held-out minus-written errors in that cell).

Changes in the camera-ready version: the abstract is rewritten in the order you recommend — question, approach, principal findings, implications — with three numbers instead of twelve, each with its unit and denominator. Invented vocabulary ("habit layers", "judges", "crime scene") is removed or defined on first use; "stamp" and "amplifier" are kept, because the distinction is a result, and are defined where they first appear.

(Part 1/3. Part 2/3 answers W2; part 3/3 answers W3.)

AuthorsOpenReview ↗

Response to Reviewer CyA5 (2/3): W2 (committed late vs decided late)

(Part 2/3 of our response to Reviewer CyA5: W2.)

W2: "The interpretation appears to move too quickly from layer-wise readouts and sign-direction interventions to identifying where the erroneous mathematical decision originates. Late-layer activity could reflect the amplification or output encoding of a decision formed earlier, rather than the source of the error itself."

You are right, and we lay out why step by step.

What we measured. At the position just before the sign, the logit lens reads, at every layer, how strongly the residual stream already favours the written sign over the other one; direct logit attribution then credits each layer's MLP with its share of that lead (§3–§4).

What we found. In every model the lead is near zero through the middle of the network and rises steeply only in the last quarter of layers; a few named late layers carry most of the credit (Figures 1–2).

What we claimed. §3 says that an error computed earlier "would be visible mid-network; nothing is", and the paper then speaks of where the decision is made.

Why your objection is correct. The logit lens reads only the part of the residual stream that is already aligned with the output tokens. A decision formed earlier, in a direction the output layer does not yet read, is invisible to it. And all our measurements are at one token position; for a cofactor sign the parity (i + j) is determined several tokens upstream, where we did not look. So our evidence shows that the sign is committed late — written into the output-aligned direction in the last quarter — not that it is decided late.

What still shows the late layers are not only encoding. Three results: (i) the stamp layer pushes «−» even on correct «+» answers (Qwen L75: −1.02 [−1.25, −0.76], n = 29; Table E2), so it is a standing bias rather than the encoding of a decision; (ii) patching the healthy message from a correct run into an error at those layers repairs nothing, because the healthy and the error messages are the same there (§5) — nothing error-specific arrives from upstream at that position; (iii) in a new experiment run during the discussion (reply to Reviewer 8g4g), removing each late layer's own minus component across Phi-4's full late window, injecting nothing, flips 23% of determinant errors and 11% of IBP errors, concentrated in low-margin cases. The late layers therefore carry at least part of the decision; whether the rest of the margin is an upstream decision or further late writing, these data cannot say.

What changes. In the camera-ready version, §1, §3, §7 and the Limitations say "committed late; where the decision is formed is open," and the sentence quoted above is removed.

What we will do. The direct test is a linear probe trained to read the correct sign (not the written one) from the residual stream at earlier layers and at the upstream token where the parity is determined, followed by activation patching at that token from matched correct runs. If the correct sign is readable upstream and patching there flips the output, the decision is upstream and the late layers encode it; if not, the late layers decide. This is the first experiment of the next version.

(Part 2/3. Part 3/3 answers W3.)

AuthorsOpenReview ↗

Response to Reviewer CyA5 (3/3): W3 (Zhang et al. 2024 and related work)

(Part 3/3 of our response to Reviewer CyA5: W3.)

W3: "The discussion of related work is insufficient. In particular, the paper omits the relevant study ''Interpreting and Improving Large Language Models in Arithmetic Calculation.'' The authors should discuss this work and clarify how their approach and findings differ from it."

Added in the camera-ready version (Zhang et al., ICML 2024), as follows.

What they do. For two-operand arithmetic ("3 + 5 =") answered at the final token, they find that a small set (under 5%) of attention heads attends to the operands and the operator, that MLPs then process that information, and that selectively fine-tuning those heads and MLPs improves arithmetic accuracy without harming non-mathematical tasks.

What we do. We study one wrong sign token in the middle of a 4,000-character derivation, where the sign is dictated by a rule the model itself set up earlier (cofactor parity; the parts formula), and we compare error cases with correct cases on the same layers.

How the findings relate. (i) Both find a sparse set of late components with MLPs doing the output-side work. (ii) They validate their localization by fine-tuning the located components, and it works; we test activation steering at the located site and find that its success depends on how much of the circuit is covered and on the margin — complementary results, which together suggest weight-space repair may succeed where low-coverage steering fails; that fine-tuning experiment is our committed next step (reply to Reviewer CN8w, W3). (iii) Their probe is the final token of a short prompt; ours is a token deep inside the model's own text, which is exactly why your W2 — a decision formed upstream — matters more in our setting than in theirs.

We also add Hanna et al. (NeurIPS 2023), who show GPT-2's final MLPs boosting the correct answer class, as a second late-MLP precedent, and sharpen the positioning against Hase et al. (NeurIPS 2023) as in our reply to Reviewer 8g4g.

Reviewer_CN8wOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates two related classes of errors over 5 models. Using standard tools, it locations where/when the errors occur. It finds that steering works on some but not all models.

Critical strengths

The paper is a strong, well-documented, principled investigation of two related classes of errors on 5 models.

It shows where and when the errors occurred. It shows (at a shallow depth) the first layer of the algorigthm (stamps, amplifiers, etc).

It used standard techniques. It has strong documentation.

Critical weaknesses

The findings are largely empirical. Using just 5 models and 2 failure cases adds risk that apparent results are just variability in a small sample size. As each model had different training, we expect variability in capability, algorithm etc, making the small sample size risky.

The paper provides new data but no novel insights. “None of these properties is visible in a localization map alone”. Completely agree, we need deeper understanding than localization, sufficiency, necessity, and steerability data. For reliability and safety gains we need to go beyond detecting the where and when the error occurred, and how (deeply) why the error occurred.

An extension of this work (using the data from this paper) could lead to novel findings. For models that mostly answer correctly, can the cause of the error be located, and can the error in the algorithm be repaired?

I would be more positive if the paper incuded novel insight(s) into deeper causes. Some possibilities:

  • Is the root cause non-obvious gaps in training data coverage? Can we compare the failure cases to the training data? Are there aspects of the failure cases that are poorly represented in the training data? Can we synethize training data from the failure cases and fine tune to the model to repair the algorithm fault? Replicate on an open-data model (OLMo, Pythia).
  • Investigate more deeply one model that gets most answers right and a few wrong. What is the difference in these two cases? Is the intermediate cause poorly-separated nodes in a manifold the algorithm uses?
  • Re “some model somewhere should carry a fixed plus stamp, and none does.” Can we hypothesise an algorithm with this behavior? Can we test this hypothesis?
  • Each model may learn a different (but possibly similar) algorithm. The differences in steering between models may reflect differences in algorithm. Can we infer anything about the algorithm from the differences between models?
Questions for authors

None.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CN8w (1/4): what the paper studies, and W1 (sample size)

We thank the reviewer. You saw what the paper is about — "where and when the errors occurred" and "the first layer of the algorithm (stamps, amplifiers)" — and you asked for depth. One of your questions we can answer with data gathered during the discussion period; for the others, the paper contains more than we stated, and we connect each question below to the specific result it bears on, before saying what remains to be done.

What the paper is about. The errors we study occur inside long, recursive, multi-step derivations that the model writes itself — a 4×4 determinant unfolding into four 3×3 and then 2×2 determinants with a sign rule at every level; an integration by parts applied up to fourteen times, each application adding a minus. The solutions run about 4,000 characters. The model writes one wrong sign token mid-way; the remaining steps operate on the corrupted value and the final answer is wrong. The paper asks where in the network that one mid-computation sign is written and why it comes out wrong, by replaying the model's own derivation and comparing errors against correct answers on the same layers. Your "shallow depth" is a fair description of where we stop: the mechanism is established at the layer level, with bootstrap confidence intervals (Table E2) and causal tests at the layer level (§5). We do not have the feature-level account or the training-level cause. Your suggestions W4–W7 are the route to both, and we commit to them below.

W1: "The findings are largely empirical. Using just 5 models and 2 failure cases adds risk that apparent results are just variability in a small sample size. As each model had different training, we expect variability in capability, algorithm etc, making the small sample size risky."

Five models is small, and we separate what holds in all of them from what varies. Invariant across all five models and both tasks: the sign is committed in the last quarter of the network (70–100% of confirmed errors per cell, Figure 1; 39 of 51 on a sixth model, Mistral-Large-675B); by MLPs rather than attention (zeroing any single head costs at most 0.46 logits, Appendix G); the same late timing appears on a 48-problem arithmetic control run identically on all five models (Appendix D); and every fixed stamp pushes «−», never «+» (Figure 2, ten cells). What varies — the role of a given layer (stamp or amplifier) and how steerable the circuit is — we report per model with confidence intervals, not pooled into a claim, and part of that variation is itself a finding: the same layer changes role across tasks within one model (Gemma L58, stamp on determinant errors, amplifier on IBP; Table E2). The one binary claim that rested on a single model — "Phi-4 is unsteerable" — was the weakest point of the submitted version. During the discussion period we steered Phi-4 across its full late window (table in our reply to Reviewer 8g4g): two layers repair 0% of held-out errors at removal-only dose, the full 16-layer window repairs 23% (det) / 11% (IBP) while leaving every correct-«+» answer untouched, and a matched random direction repairs 0%. The claim is now continuous — repair rate as a function of coverage and margin — and measurable in every model; running that curve on all five is the first experiment of the next version. The camera-ready version withdraws the binary wording.

(Part 1/4. Part 2/4 continues with W2 and W3.)

AuthorsOpenReview ↗

Response to Reviewer CN8w (2/4): W2 (the why) and W3 (locating and repairing the error)

(Part 2/4 of our response to Reviewer CN8w: W2 and W3.)

W2: "The paper provides new data but no novel insights. 'None of these properties is visible in a localization map alone'. Completely agree, we need deeper understanding than localization, sufficiency, necessity, and steerability data. For reliability and safety gains we need to go beyond detecting the where and when the error occurred, and how (deeply) why the error occurred."

We agree on the standard, and we should have stated the "why" the paper does contain as a claim rather than leaving it in §4. The claim: the mid-computation sign error is not a miscalculation; it is a default winning. Five results support it. (i) The stamp layer pushes «−» on correct answers too — Qwen L75 on correct «+» answers: −1.02 [−1.25, −0.76], n = 29 (Table E2) — so it is a standing bias, not a response to a wrong computation. (ii) Patching the healthy message from a correct run into an error at the stamp layers repairs nothing, because the healthy and the error messages are the same there (§5, preregistered P5): there is nothing error-specific to copy in. (iii) Patching the error message into a correct run breaks it (Mistral 79/107). (iv) Qwen's rescue layer L77 pushes against the stamp hardest exactly for plus-writers; the error is the stamp unopposed. (v) The stamp predates instruction tuning (Qwen base, Appendix F), and only minus is ever stamped. A second insight, which your W5 made us see across the models rather than within one, is stated in our reply to W5: accurate models make symmetric sign errors, inaccurate models make minus-default errors. What the paper does not have — your level — is the feature-level reading of what the late layers see, and the training-level cause. The camera-ready version states the "default wins" claim in §4 and the accuracy pattern with it; the next version adds W4–W7.

W3: "An extension of this work (using the data from this paper) could lead to novel findings. For models that mostly answer correctly, can the cause of the error be located, and can the error in the algorithm be repaired?"

Located — yes, at the layer level: errors and correct answers are written by the same layers in the same roles (Table E2), and in the mostly-correct model (Gemma-3-27B, 218 errors from 4,050 determinants) correct answers commit early (median depth 0.21) while errors commit late (72% in the last quarter; §3, Figure 1) — correct and wrong cases take different paths in the same model.

Repaired — partially, and the paper's own results say why "partially." Minus-removal steering fixes wrong «−» but also breaks right «−» (Llama 95% at α=3; Phi 19% at α=1 across its full window) while sparing right «+»: the direction carries minus-ness, not wrongness (§7). So steering alone cannot repair; it needs a gate. The paper contains the gate: layers whose vote tracks the true sign — Qwen L77, Phi L38 ("judges", §2, §4). Judge-gated steering — apply removal only where the judge disagrees with the stamp — is a cheap, training-free test of "the error is the stamp unopposed," scored as net repair (errors fixed − corrects broken) on held-out cases.

For repair in the sense you mean, we commit to the fine-tuning experiment you outline: a parameter-efficient update restricted to the identified layers, trained on data synthesized from the failure cases, following Zhang et al. (ICML 2024). Success on three criteria: sign-error rate falls on a fresh bank with magnitude accuracy unchanged; non-mathematical performance unchanged; and the stamp's push on correct «+» answers (Table E2) moves toward zero — the last is what distinguishes a repair of the mechanism from a behavioral patch. Our prediction, falsifiable: repair success will track width and margin as steering does — a two-layer update will under-cover Phi as two-layer steering did; Qwen, with one stamp and one rescue layer, should repair most easily.

(Part 2/4. Part 3/4 continues with W4 and W5.)

AuthorsOpenReview ↗

Response to Reviewer CN8w (3/4): W4 (training-data origin) and W5 (accurate vs inaccurate models)

(Part 3/4 of our response to Reviewer CN8w: W4 and W5.)

W4: "Is the root cause non-obvious gaps in training data coverage? Can we compare the failure cases to the training data? Are there aspects of the failure cases that are poorly represented in the training data? Can we synethize training data from the failure cases and fine tune to the model to repair the algorithm fault? Replicate on an open-data model (OLMo, Pythia)."

We think the root cause is a statistical property of mathematical text rather than a coverage gap, and the paper contains four fingerprints that point the same way: (i) across all ten model×task cells, every fixed stamp pushes «−»; no model carries a plus stamp (§4, Figure 2); (ii) the harvested error pool, which nothing selects by sign, is minus-skewed two to one — 64.8% in determinants, 75.2% in IBP, about nine in ten for Phi and Mistral (Table A4); (iii) in Qwen the «−» lean is present at layer 0, before any layer runs (Appendix C.4.3); (iv) in Qwen the L78 stamp is already present in the base model before instruction tuning (Appendix F). The hypothesis these fit: written mathematics marks negative results explicitly ("−67") and leaves positive ones bare ("67"), so at a sign position "−" is the only sign token the corpus ever shows, and the model learns it as a default. That is a bias from what the data contains, not from what it lacks — which also predicts your plus-stamp question (W6). The test you propose is the right one and we commit to it on OLMo, whose pretraining data is open: (a) count explicit "−" versus bare "+" at answer and operator positions in the mathematical portion of the corpus; (b) run our pipeline on OLMo and measure stamp strength (the Table E2 cell: push toward «−» on correct «+» answers); (c) synthesize plus-marked training data from the failure cases and fine-tune, scoring with the frozen battery. Success means the stamp's push on correct «+» answers moves toward zero and the minus skew of the residual errors disappears — not merely that accuracy rises.

W5: "Investigate more deeply one model that gets most answers right and a few wrong. What is the difference in these two cases? Is the intermediate cause poorly-separated nodes in a manifold the algorithm uses?"

The paper's five models span the accuracy range on 4×4 determinants — Gemma-3-27B 87.8%, Llama-3.3-70B 80.9%, Mistral-Small-24B 21.4%, Qwen-2.5-72B 19.7%, Phi-4-14B 10.9% (Table A1) — and the difference between the models that mostly get it right and those that do not is visible in the paper's tables, though we did not state it as such. Accurate models make symmetric sign errors; inaccurate models make minus-default errors. In Gemma, 41.8% of sign errors are written «−» (more «+» errors than «−»); in Llama 64.1%; in Mistral 85.8% and Phi 88.4% (Table A4; nothing in the harvest selects by sign). Mechanistically, Gemma and Llama compute the sign and then amplify it: their habit layers are amplifiers on correct answers (Llama L79 +3.38, Gemma L58 +2.30; Table E2), their margins are moderate (medians 5.9–7.5 logits; Appendix K.1), and the minus-removal lever is surgical — 129/131 and 72/72 correct-«+» answers survive it (Figure 4). In Qwen, Mistral and Phi the sign is defaulted rather than computed: Qwen carries a fixed «−» stamp active on correct answers and never writes «+» at the integration-by-parts site at all (0/36 errors, 0/75 corrects); Phi writes «−» through a ~15-layer ensemble with margins several times larger than any other model's; Mistral's sign representation is too weak to separate the two signs (margins 0.5–3 logits; the lever breaks every cohort). The error in an accurate model is a computation slip that the late amplifier then commits; the error in an inaccurate model is a prior that wins when nothing opposes it. Five models is a correlation, not a law, and we state it as such; it is also the clearest "why" the data give, and the camera-ready version states it in §4.

Inside the most accurate model, the correct and wrong cases are already distinguishable: Gemma's correct determinant answers commit early (median depth 0.21), its errors commit late (72% in the last quarter), and L58 is an amplifier on the former and a stamp on the latter (§3, Table E2, Appendix C.4.4). On "poorly-separated nodes in a manifold": error cases are not borderline — the wrong sign leads by 5.9 logits at the median in Gemma and far more in Phi — so the intermediate cause is not proximity on one manifold but a late default overriding an earlier guess. The feature-level account of what the late layers read when they do so is the deep dive you ask for; the correct-sign probe (reply to W7) is designed for it, on Gemma first.

(Part 3/4. Part 4/4 continues with W6 and W7.)

AuthorsOpenReview ↗

Response to Reviewer CN8w (4/4): W6 (the plus-stamp hypothesis) and W7 (inferring the algorithm)

(Part 4/4 of our response to Reviewer CN8w: W6 and W7.)

W6: "Re 'some model somewhere should carry a fixed plus stamp, and none does.' Can we hypothesise an algorithm with this behavior? Can we test this hypothesis?"

Yes to both. The algorithm we hypothesise is asymmetric: at a sign position the model runs "write «−» unless something says otherwise." «−» is a learned default; «+» is represented as the absence of «−», not as a vote of its own. The default comes from written mathematics, which marks negative results ("−67") and leaves positive ones bare ("67"): at a sign position the training text shows "−" or nothing, almost never "+", so a plus default has nothing to be learned from. This algorithm predicts five things, all of which the paper already observes. (1) Stamps are minus-only in every model and task (Figure 2, Table E2). (2) Errors are minus-skewed, and more so where the task's formula contains an explicit minus: 64.8% «−» in determinants, 75.2% in integration by parts (uv − ∫v du), where Qwen never writes «+» at all (Table A4). (3) The plus machinery is opposition to minus rather than an independent plus vote: Qwen's rescue layer L77 pushes against the stamp exactly for plus-writers (§4), and removing the minus direction makes «+» appear without anything being injected (α=1 steering). (4) The default predates the task and the chat tuning: Qwen's base model already carries the L78 stamp (Appendix F), its layer-0 embedding leans «−» (C.4.3), and Gemma's early band copies minus characters from its own context (C.4.4). (5) Patching a healthy message into an error repairs nothing, because at the stamp layers the healthy and error messages are identical (§5, prediction P5). The decisive test is on an open-data model: count explicit "+" versus "−" at result positions in OLMo's mathematical pretraining text, then fine-tune on mathematics that writes positive results explicitly ("+67") and re-run the pipeline. The hypothesis says a plus stamp should appear, or the minus stamp weaken, with the error skew moving toward even; a minus-only stamp surviving plus-marked training would falsify it and point to an architectural rather than statistical origin. We commit to this as part of the OLMo study (W4).

W7: "Each model may learn a different (but possibly similar) algorithm. The differences in steering between models may reflect differences in algorithm. Can we infer anything about the algorithm from the differences between models?"

Yes, and the inference is the content of §4–§5 read together. The similar part: all five models write the sign with the same skeleton — late (last quarter of layers), by MLPs rather than attention (no single head accounts for the push, Appendix G), along one direction in the residual stream, with «−» as the default (W6). Three components do the work: a default toward «−» (the stamp), an amplifier of whatever sign the computation already produced, and a check that opposes the default when the computation says «+» (Qwen's rescue layer L77; Phi's L38). The different part is how much of each a model has, how widely it is spread, and how hard it pushes — and that mix predicts the steering outcome. Qwen (strong default + check): removing the default repairs errors (11 of 11 determinant flips corrective), leaves every correct-«+» answer untouched (0/104), and breaks 42% of correct-«−», because the same default writes the correct minuses. Llama (amplifier only): removing minus-ness repairs wrong-«−» but breaks 95% of correct-«−» and flips 43% of «+»-written errors too — the amplifier supports whatever was chosen. Gemma (default on errors, amplifier on corrects): the most surgical lever, 39%/54% of errors repaired with 129 of 131 correct-«+» surviving. Phi (amplifier-type, spread over ~15 layers, margins three times larger): immune at two layers, 23%/11% at full coverage (reply to Reviewer 8g4g). Mistral (weak, indeterminate roles, margins 0.5–3 logits): the largest flip rates and no specificity. The four-cohort response in Figure 4 — wrong-«−», wrong-«+», right-«−», right-«+» — is therefore a fingerprint of the algorithm: defaults yield to subtraction and spare «+»; amplifiers spill onto correct «−»; wide ensembles need coverage; weak representations have no specificity. Two things the differences do not yet tell us, and which we will test: whether the sign direction is the same across tasks and models (a linear-probe study on cached activations), and what the check layer reads when it overrides the default — the deep dive of W5, on Gemma first.

Reviewer_8g4gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper challenges a foundational premise of mechanistic interpretability: that finding where a behavior is computed in a neural network (localization) implies you can intervene there to change it (steerability). The authors investigate "sign errors" in multi-step LLM math—where a model executes arithmetic and symbolic steps accurately, but writes a single wrong + or - token mid-solution. Across six open-weight models (14B to 675B parameters) and two tasks (4×44\times4 matrix determinants and calculus integration by parts), they find that while the sign decision is universally localized to the final quarter of MLP layers, steerability varies wildly (from 4% to 93%). Whether a localized circuit can be steered depends on its internal mechanism—specifically, layer redundancy, decision margins, and whether layers act as fixed "stamps" or adaptive "amplifiers".

Critical strengths

None

Critical weaknesses

Strip away the dataset (3,253 cases) and the elaborate git-preregistration apparatus, the paper's actual technical findings are largely descriptive applications of off-the-shelf tools (logit lens, Direct Logit Attribution, mean ablation, and rank-1 vector subtraction) that confirm phenomena already well-documented in the interpretability literature.

  1. Localization ≠\neq Intervention Success: Hase et al. (2023) ("Does Localization Inform Editing?") already demonstrated that causal-tracing localization does not predict where model intervention succeeds.
  2. Redundancy and Backup Circuits ("Why Phi-4 Resists"): The finding that Phi-4 resists 2-layer interventions because ~15 late MLP layers redundantly write the sign is a textbook instance of distributed redundancy already characterized in different works.
  3. Late-Layer Token Commitment & Amplifiers/Stamps: It is widely established in logit-lens and arithmetic-circuit literature (Stolfo et al., 2023; Hanna et al., 2023) that formatting and output-token commitments peak in the final MLP layers, and that late MLPs often act as confidence amplifiers or static token biases.

Stolfo et al., 2023 A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis Hanna et al., 2023 How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

Questions for authors

Can you report the flip rate and specificity when directional steering is applied across Phi-4's full localized 15-layer window? Without this, how do you rule out the simpler explanation that a 2-layer intervention merely under-covers a 15-layer localized circuit?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8g4g (1/2): what the paper studies, and the full-window steering experiment

We thank the reviewer for a concrete, testable question. To answer it we ran a new experiment during the discussion period — directional steering across Phi-4's full late window, on the paper's held-out cases — and report it below. Before the point-by-point replies, one clarification that we believe explains most of the disagreement.

What the paper is about. The errors we study occur inside long, recursive, multi-step derivations that the model writes itself. A 4×4 determinant by cofactor expansion unfolds into four 3×3 determinants, each into three 2×2 determinants, with dozens of multiplications and additions and a sign rule, (−1)^(i+j), applied at every level; a repeated integration by parts on x¹⁴·cos 5x applies the parts formula up to fourteen times, each application introducing a new minus sign. The solutions run about 4,000 characters (1–1.5k tokens). In the cases we study, the model writes one wrong sign token mid-way through this derivation; the remaining steps then operate on the corrupted value, and the final answer is wrong (in the §1 example, 323 instead of 83). The arithmetic itself is not the failure — one sign token is. The paper asks where in the network that single mid-computation sign is written, and why it comes out wrong. The unit of analysis is the model's own derivation, replayed token by token; the probe is the position of that one sign; the comparison is errors against correct answers on the same layers. This is a different question from the arithmetic-circuit literature, which asks which components compute a one-token answer to a short prompt such as "3 + 5 =". The submitted title and §1 led with the steerability claim instead of this question; that is on us, and we will correct the framing in the camera-ready version. Steerability is one test we ran on the located site, and your question concerns that test.

Q: "Can you report the flip rate and specificity when directional steering is applied across Phi-4's full localized 15-layer window? Without this, how do you rule out the simpler explanation that a 2-layer intervention merely under-covers a 15-layer localized circuit?"

To answer your question, we ran a new experiment during the discussion period: directional steering across Phi-4's full late window, on the paper's held-out cases. It is not in the submitted PDF; we will add it to the camera-ready version as Appendix K.6. For context, the paper steered two layers per model so that the intervention was identical across all five models — the same number of sites, chosen by the same frozen rule (§2, Appendix I). For Phi-4 those two layers carry only 21–27% of the positive direct-logit credit, whereas in the other models the credit concentrates on two or three layers (Figure 2).

Setup: the paper's held-out Phi-4 cases — error cases (det n=145, IBP n=70) and correct answers (det n=132 «−» / 12 «+», IBP 93 «−» / 53 «+») — with the paper's direction, hook and probe token; only the set of MLP layers changes. α=1 removes each layer's own minus component and injects nothing.

MLP layers steered, α=1det errors repairedIBP errors repairedcorrect «+» broken (det / IBP)correct «−» broken (det / IBP)
2 (L37, L39 — the paper)0%0%0% / 0%0% / 0%
15 (Appendix K's credited layers)26%0%0% / 0%20% / 0%
16 (L24–39)23%11%0% / 0%19% / 0%
16, random direction of matched magnitude0%0%0% / 0%0% / 0%
16 mid-network layers (L8–23), same direction0%0%0% / 0%0% / 0%

Every repaired case has the correct sign as the top-1 token. Repairs concentrate in low-margin cases (det: 77% of cases with lead <10 logits, 5% above). At α≥2 the same 16 layers flip 94–99% of errors but also invert 72–100% of correct answers of both signs — sign inversion, not steering — so we report α=1.

So: at two layers Phi-4 does not steer; across its full window it steers, with the same specificity signature as the other models (correct «+» untouched, some correct «−» broken — the minus-ness lever of Figure 4), at a lower rate (23%/11% at α=1, against 28–81% for the other models at the paper's α=3) that is set by its margins. We agree the paper's statement "Phi-4 is unsteerable at calibrated dose" (§5, §6, K.5) must go; in the camera-ready version it is replaced by the statement above, and the table and dose sweep are added as a new Appendix K.6. The point we keep is that the localization maps (Figs. 1–2) do not show which model needs 16 layers instead of 2, or what repair rate the margins allow; the intervention does.

(Part 1/2. Part 2/2 answers W, W1, W2 and W3.)

AuthorsOpenReview ↗

Response to Reviewer 8g4g (2/2): weaknesses W, W1, W2, W3

(Part 2/2 of our response to Reviewer 8g4g. Part 1/2 contains the general paragraph and the answer to the question.)

W: "the paper's actual technical findings are largely descriptive applications of off-the-shelf tools … that confirm phenomena already well-documented in the interpretability literature."

We take the point about the tools, and we chose them deliberately: a preregistered comparison across five model families and two tasks has to use the field's own instruments, or it ends up testing the instruments rather than the models. We claim no new tool. As we tried to say in the general paragraph above, what we believe is new is what the tools are applied to — one wrong sign token in the middle of a long multi-step derivation, compared against correct answers on the same layers — and what that comparison shows (W2, W3).

W1: "Localization ≠ Intervention Success: Hase et al. (2023) ("Does Localization Inform Editing?") already demonstrated that causal-tracing localization does not predict where model intervention succeeds."

Agreed, and we cite Hase et al. as the closest prior (§1). Their result concerns whether a localized site can be edited (weight editing, factual recall). In our paper that question arises only in the steerability test, where we extend it to activation steering across models and tasks and, thanks to your question, can now say what the gap consists of — coverage and margin. The paper's main result is a different one: why the mid-computation error happens (W2).

W2: "~15 late MLP layers redundantly write the sign is a textbook instance of distributed redundancy already characterized in different works."

Redundancy is not the main result of the paper. It is the explanation we give for one model's resistance to steering (Phi-4, Appendix K), and we agree it is already characterized — we cite McGrath et al. (2023) and Rushing & Nanda (2024) for exactly that. The main result about why the sign comes out wrong is the layer typology of §4, Table 2, with bootstrap CIs in Table E2: (i) a stamp layer that pushes «−» regardless of the correct answer — Qwen L75 pushes «−» even on correct «+» answers (−1.02 [−1.25, −0.76], n = 29), a signature an amplifier cannot produce; (ii) amplifier layers that push whichever sign was already chosen (Llama L79, Phi L39; CIs above zero on errors and corrects alike); (iii) a rescue layer (Qwen L77) that pushes hardest exactly for plus-writers; (iv) only minus is stamped — no plus stamp in any model or task — and in Qwen the stamp is already present in the base model before instruction tuning (Appendix F); (v) the same layer can change role across tasks (Gemma L58: stamp on determinant errors, amplifier on IBP). Errors and correct answers run the same layers in the same roles; in Qwen, the clearest case, the model errs when its standing minus default goes unopposed by the rescue layer. A fixed-sign stamp active on correct answers, its asymmetry, and its pre-training origin are not in prior work. We agree the steerability framing buried this; the camera-ready version leads with it. Width, which your experiment measures, explains steerability — a separate question from why the error occurs.

W3: "It is widely established in logit-lens and arithmetic-circuit literature (Stolfo et al., 2023; Hanna et al., 2023) that formatting and output-token commitments peak in the final MLP layers, and that late MLPs often act as confidence amplifiers or static token biases."

We agree with the general statement and cite both papers for it. But they answer a different question. Stolfo et al. and Hanna et al. ask, for a short arithmetic prompt with the answer at the final token, which components compute the answer — Stolfo et al. by causal mediation on two- and three-operand queries in 2.8–7B models (attention carries operands to the last token; late MLPs produce the result), Hanna et al. by circuit discovery in GPT-2 small (final MLPs boost valid end-years). Both explain a capability. We ask where one wrong sign token is written in the middle of a long multi-step derivation, and why. The readout is the logit lead of the written sign at every layer and each layer's signed vote, for errors and correct answers alike. That readout is what lets us classify a layer as stamp or amplifier — a mediation effect reports that a layer matters, not which way it pushes — and what shows that errors and correct answers are written by the same layers in the same roles, the error being a standing minus bias that wins when unopposed (W2). "Late MLPs amplify" is consistent with both papers; "a late MLP carries a fixed wrong-sign bias, active on correct answers, asymmetric, present before instruction tuning, and that is where mid-computation sign errors come from" is in neither. In the camera-ready version Hanna et al. is added to the related work, which is rewritten to state the agreement and the difference this way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.