acceptodds
Under review as a conference paper at ICLR 2027

An Unchanged Answer Can Hide an Operand's Causal Role

Abstract

Copying a hidden state between questions can test what information a language model uses. The final answer can conceal a contribution when another part of the computation determines which answer wins. We study this problem in Qwen3-8B on weekday and month arithmetic. We replace states at two positions using independently chosen source questions. Holding one source fixed lets us test whether the other supplies an offset or a complete result. Sources with the same offset favor the corresponding candidate even when their own results differ. This offset contribution persists in cases where every substitution produces the same answer. We trace the same candidate comparison through the receiving computation. The next attention output carries the contribution to the final position. Its effect then depends on the receiving state. Fixing the outputs of the next two feed-forward layers reduces this dependence by roughly two-thirds to three-quarters. These experiments identify an operand contribution hidden by answer recovery and distinguish its transmission from the later processing that determines its effect.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, causal abstraction, arithmetic reasoning, interchange interventions, attention and MLP pathways, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FVpuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores the arithmetic of how the model's hidden states responds when copied and replaced elsewhere.

Using the task of calendar arithmetic, specifically months arithmetic, the paper proposes to explore where the hidden states hold information.

The paper only explores this on Qwen 3B model and uses another model as a contrast analysis.

Critical strengths

The problem setting is interesting.

The authors are building on the paper by Feucht et al. to understand arithmetic and see where these arithmetic lies in the model.

The authors utilize a very non-invasive method to try to show how the model understands these simple arithmetic and how swapping some of these numbers within the model can break or change the outputs.

Critical weaknesses

The writing is a little had to follow.

The paper in general is not organized in a friendly way, also utilizing too many jargon for someone who is not overly familiar in the field, making the paper quite difficult to follow.

For example, in the experiments section, in Pair selection and scores, suddenly the term natural accuracy is used.

I'm assuming the authors are trying to say the accuracy of the model with respect to ground truth.

A lot of the writing seems quite verbal and could use some serious polishing.

The experimental findings that are listed are quite limited.

First and foremost, only 1 model, the Qwen3 8B model is compared while the Llama 3.1 8B is supposed to be contrast analysis.

Does this mean that the authors are only trying to understand Qwen3's performance?

Why can't it be the other way around?

Form my understanding, this work seems to be trying to locate where exactly in the model a certain wording (month or day) is being placed in the model and trying to determine how deep in the model it is.

However, there does not seem to be a clear enough support for what their findings are or what we are trying to do with them.

Also, the dataset used is quite unclear. Was this done on a subset of a dataset?

It seems that it was only done through one dataset.

Questions for authors

As mentioned in the weaknesses, how are we to understand how this type of idea scales across model sizes and different types of models?

What is the purpose of the contrast model and can it be used in reverse?

It would be helpful if different kinds of intervention is tested to see how each one would respond to the model outputs.

This can be insightful in showing how the model performs as well.

Reviewer_3gqpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how information is represented and processed in LLMs using activation patching. The authors replace hidden states at different token positions with states from other questions and analyze whether these interventions transfer an operand or a complete result.

The main experiments use Qwen3-8B on weekday and month arithmetic. The authors show that an operand can influence the scores of candidate answers even when the final predicted answer remains unchanged. They also investigate how this information is transferred through attention and how its effect depends on subsequent MLP processing.

Critical strengths

The paper addresses an interesting problem in mechanistic interpretability. I find the idea of comparing interventions that preserve the final answer but change the relative scores of alternative answers potentially useful.

The experiments also provide an interesting way to distinguish between information transferred by an internal state and the way this information is processed by subsequent layers.

Critical weaknesses

My main concern is the extremely limited experimental setting. The main analysis is conducted on a single model, Qwen3-8B, and a very narrow class of tasks involving weekday and month arithmetic. Most of the detailed experiments focus on interventions around layers 22–24.

Although the appendix includes some experiments with Llama models and additional prompt formulations, these do not provide a comparable validation of the main findings across models, layers, or substantially different tasks.

As a result, it is difficult to understand whether the observed effects represent a general mechanism of information processing in LLMs or are specific to the selected model, task, and intervention points.

I also find the conclusions rather limited. The experiments demonstrate some interesting effects of activation replacement, but do not establish a general principle or a sufficiently broad understanding of how operands and results are represented and combined in language models.

In my opinion, the current results are more suitable for motivating further research than for supporting the conclusions of a full ICLR paper.

Questions for authors

Could the authors extend the main experiments to other model architectures and sizes? In particular, do the same operand-related effects appear in models other than Qwen3-8B?

How sensitive are the conclusions to the choice of intervention layer? It would be useful to examine whether similar effects can be identified systematically across different layers, rather than focusing primarily on the selected intervention points.

Could the authors evaluate the approach on other tasks, such as ordinary arithmetic or more complex reasoning problems? This would help determine whether the observations are specific to cyclic calendar arithmetic.

What general conclusions about LLM computation do the authors believe can be drawn from these experiments? In particular, what evidence supports extending the proposed interpretation beyond the specific settings studied?

Reviewer_kK6pOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that activation patching judged by answer changes can miss a state’s contribution. In Qwen3-8B on calendar arithmetic, it patches the unit word and the final token after layer 22 with states from independent donors. It varies the unit donor among an anchor, a same-offset donor, and a same-result donor, and scores all runs on one fixed candidate pair. It finds that the unit supplies offset information even when the answer is unchanged. This information reaches the final token through layer-23 attention, mainly four heads, and clamping MLPs 23–24 reduces its dependence on the receiving state by 63–74%. Validation uses forward and reverse interventions, random controls, a collision-free subset, and five prompt wordings.

Critical strengths

Donor–recipient pairs are chosen so that start, offset, and result each predict a different answer. Every intervention is scored on one fixed candidate pair, and same-offset and same-result donors separate the hypotheses. The checks agree. Same-offset donors match the anchor while same-result donors do not (Fig. 3, Table 2). Forward transfer of the layer-23 attention write recovers nearly the full contrast, and reverse restoration removes nearly all of it (Tables 3–4). Four heads selected on one condition generalize to the rest. This makes the localization credible in the setting studied, so my main concern is significance rather than the correctness of the core result.

Critical weaknesses

The central claim is that an unchanged argmax can conceal a component’s contribution that a fixed candidate margin reveals. That is essentially the standard rationale for continuous logit-difference metrics in activation patching (Zhang & Nanda, ICLR 2024). The independent-donor design with same-offset and same-result controls is a direct application of interchange interventions (Geiger et al., 2021), and the paper itself notes that calendar studies already use such crossed controls (§1). The mechanistic findings are that operand information is routed to the final token by attention and then processed by MLPs. These are consistent with Stolfo et al. (2023) and Mamidanna et al. (2025), here localized to four layer-23 heads in a single model. The paper never shows that its procedure changes a conclusion that standard logit-difference patching would reach, on this task or any other. What is new is a careful but narrow case study: one model (the Llama analysis uses a different protocol), one task family, and one layer choice. Its general lesson is already standard practice, and I do not think it meets the bar for an ICLR contribution.

Also, the claim that later MLPs determine how the delivered offset is used rests on a 63–74% reduction in the spread of χ_U when both final-position MLP outputs (layers 23 and 24) are clamped (Table 6). There is no control that clamps other components. And because each role’s MLP output is pinned to its own full-strength endpoint (§3.4), some reduction is expected by construction. Second, the prediction test (Table 1) cannot discriminate between interpretations. Both regressions include the mean endpoint effect, so the semantic model is just the generic model with A’s and B’s endpoint effects replaced by their average. Given the paper’s own finding that A ≈ B, near-identical errors are guaranteed, and on weekdays/Journey neither model beats the final-only baseline. Third, statistical support is thin. There are 24 recipient groups per condition with reused donors, no confidence intervals or tests, and a weekday effect that shrinks about 3.5× under rewording (Table 2). This matters because these analyses are what would lift the paper above a single-configuration localization. Without them, neither the second contribution nor the general guidance offered in §6 is supported.

Questions for authors
  1. On the same groups, what does the conventional logit difference between the donor’s and the recipient’s answers report for the months-token swap? Are there groups where it leads to a different conclusion than χ_U?

  2. Suppose other final-position components (for example, attention in layer 24 or the MLPs in layers 25–26) are clamped to their role-specific endpoints in the same way. How much does the spread of χ_U shrink, and how does mean χ_U change under each clamp?

  3. Both regressions include the mean endpoint effect. Can the authors confirm that, as a result, the semantic model differs from the generic one only by pooling A’s and B’s endpoint effects? If so, what does Table 1 establish about the operand interpretation?

  4. Can the authors report confidence intervals for mean χ_U that account for donors and questions reused across groups (e.g., a cluster bootstrap)? Can they also report χ_U for placebo triples in which B shares neither the offset nor the answer with A?

  5. Fig. 2 and Table 5 report |χ_U|. What are the signed means, and in what fraction of same-answer groups is χ_U positive? Without the sign, these results don’t show that the hidden contribution is offset-related.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.