- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether an activation edit that changes a model's output changes an internal state used in subsequent computation according to the task semantics, or biases that computation toward a specific output directly. The authors start with a register-editing setting, where they fine-tune a 7B language model (LM; Qwen2.5-Coder-7B) to output states of walks on groups Q8 and D8. In this setting, they show that rank-16 activation edits that flip the hidden part of the states (while keeping the printed tokens the same) make the LM apply the next move to the newly edited internal state. Additional controls demonstrate that the edit changes the internal state rather than biasing the computation toward a specific answer, and the authors localize the state update to one attention edge in Q8 and partially to four edges in D8. They show this is consistent when tested with Mistral-7B and in factored state machines and small interpreters. In the second part of the paper, the authors study Function Vector (FV) steering and find that in Q8, the FV is mostly outside the previously identified rank-16 subspace. Additionally, in one lexical task each for Qwen2.5-7B and Llama-3.1-8B, patching the correct-prompt activation at one layer or applying the FV at the natural activation-change scale has almost no behavioral effect. Since these fail while the calibrated FV, which is 77x (Qwen) and 26x (Llama) larger in norm than the natural change, does change behavior, the authors conclude that succesful steering need not recreate the model's natural activation at the intervention site. Together with the results from the register-editing setting, the authors argue that calling an activation edit a "state write" requires showing that later computation uses the edited value according to that state's update rule (semantics of the task), not only that the behavioral output changes.
Overall, this paper is carefully executed, and its register-editing experiments provide a clean demonstration of an activation edit that changes the internal state used by later computation. However, the paper's main distinction is largely established in prior work, some of the steering experiments test a hypothesis that is not clearly motivated and only loosely connected to the paper's main question, and the dense number-heavy presentation makes the paper difficult to follow in places. I therefore lean toward rejecting the paper in its current form.
Critical strengths
- Clear framing. Table 16 gives a clear and practical summary of what different intervention tests can establish, which is useful for readers even though the underlying distinction is not new (see Weaknesses).
- Overall methodology. For the state-tracking task, the authors design the state-move transition rules such that every edit has a unique predicted next state, which together with move-swap and conflicting-continuation controls cleanly separates "supplying a state" from "supplying an answer". This design is well thought-through and appropriately chosen for the particular research question of the authors. The general approach is also good: the site of editing is chosen on validation data, localization thresholds are predeclared, and there are ten D8 splits. Additionally, the authors confirm the consistency of the core results in a second model family (Mistral-7B-v0.3 in addition to Qwen2.5-Coder-7B) and similar state-use tests hold on more complex tasks (factored state machines and small interpreters).
Critical weaknesses
- Limited novelty. The main distinction of the paper (very well summarized in Table 16: direct output effect vs. downstream control vs. computational state) can already be found in existing work: amnesic probing already separates encoded from used information [1], causal abstraction considers activation patching and steering as tests of whether an intervened variable plays its role in a high-level algorithm [2], and Makelov et al. [3] show that an intervention can have the intended behavioral effect without writing the intended variable. Similarly, the register test is essentially an interchange intervention with a known high-level causal model, plus the move-swap variation, and the idea of comparing states through varied future operations is very close to the cited Hidden APIs work [4]. The paper should state more clearly what is new beyond applying these ideas to their specific setting of trace-supervised models on Q8 and D8. Could the authors elaborate?
- Unclear motivation and scope of the steering experiments. It is unclear who holds the correspondence hypothesis as presented and tested here (that an FV recreates the natural final-token residual at one layer). Prior work already shows that successful interventions need not induce the model's natural representations/computation ([3], see above) and that steering effects can depend on spurious factors [5]. The observation that strong steering pushes activations off the natural activation manifold can also already be found in existing work (e.g., [6] and [7]). The authors should position their results relative to this work and reference studies that hold the view they argue against. Moreover, the FV used in the state-tracking task is constructed in a non-standard way (Appendix E.1: built from traces rather than in-context demonstrations, applied at the current-phase token, and defined as a phase contrast), so it is closer to contrastive steering than to an FV in the sense of Todd et al. [8]. Finally, it is unclear how the failure to recreate the natural internal state in the lexical tasks answers the paper's main question of whether activation steering changes the internal state that later computation uses, or directly biases the computation toward the output. As a result, the two halves of the paper feel only loosely connected.
[1] Y. Elazar et al. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. TACL 2021.
[2] A. Geiger et al. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. JMLR 2025.
[3] A. Makelov et al. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.
[4] S. Ma et al. Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futures. arXiv:2607.27617, 2026.
[5] D. Tan et al. Analysing the Generalisation and Reliability of Steering Vectors. NeurIPS 2024.
[6] G. Luo et al. Learning a Generative Meta-Model of LLM Activations. ICML 2026.
[7] Z. Xu et al. Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics. ACL 2026.
[8] E. Todd et al. Function Vectors in Large Language Models. ICLR 2024.
Questions for authors
- In Sec. 7.2, FV efficacy (64.3%) is described as approaching register editing (80%). In this comparison, the FV update is 2.19$\times$ larger than the register write, but only 5.7-13.4% of its energy lies in the register rank-16 subspace, so its component in this subspace is about 0.5-0.8$\times$ the norm of the register write. Given this, could the FV simply be a weaker write into the same register, plus a large out-of-subspace part? Could the authors test this by applying only the FV's projection onto the subspace, and separately the FV with that projection removed?
- Have you tried patching the correct-prompt activations at all positions at layer , rather than only the final token? (Considering that the subsequent layers can attend to the original context, it is not surprising that a single-block edit does not work, or requires order-of-magnitude larger steering vector norms.)
- In Qwen, another task's FV works as well as the target FV. Does this suggest the lexical steering effect is mostly a generic perturbation that pushes the model away from the shuffled mapping?
4 comments
W2. Hendel et al. show that a single hidden state computed from demonstrations can carry a task when it is transplanted at one layer, and Todd et al. describe the FV as a compact representation of the demonstrated task. On a natural reading of these results, adding an FV supplies the representation of the task that the model itself would form. The correspondence hypothesis states this reading in a form that can be tested, and we will introduce it with these references in the camera-ready version.
With regards to how the two halves of the paper are related, both halves ask the same question. When an activation edit changes a model's output, has it changed a value that later computation uses, or has it only biased the computation toward that output?
The first half of the paper answers the question in a setting where a direct answer is possible. The variable and its update rule are known, so we can vary the operation that follows the edit and check that the model applies it to the edited value. From this we learn what evidence is needed to conclude that an edit changed the model's state, and that an additive edit can provide it.
The second half puts the same question to steering as it is ordinarily used. There the output changes, but no variable or rule is known, and the direct test cannot be run. The one concrete candidate for a state that the vector might supply is the activation the model itself produces on the correct prompt. If steering worked by supplying that state, transplanting it should be effective, and steering should remain effective at its natural scale. Neither prediction holds. We feel your explanation of why is plausible. A vector that succeeds at the final token, at 26 to 77 times the size of the model's own change there, therefore cannot be succeeding by reproducing what the model computes at that site (Table 13). These results do not establish that the vector merely biases the output, and we will say so in the camera-ready version. They establish that a successful steering result is not enough, without further evidence, to conclude that a state was written, which is the claim of the paper. We are not aware of prior work that reports this comparison, and we will discuss it in relation to Makelov et al., Tan et al., Luo et al. and Xu et al., as you recommend.
The label-swap experiment described in section 9 applies the logic of the first half of the paper within the lexical tasks. The correct answer is held fixed and the label that must express it is exchanged. An edit that supplies a value used by later computation should follow the new label, in the way the edited counter supports either later instruction. The steering effect fails to do so. The component that follows the assigned label is negative in both models (−0.315 in Qwen and −0.250 in Llama; both 95% intervals exclude zero), including on items that the models solve under both assignments (Table 14).
We now discuss the questions raised by reviewer ceeZ and the requested experiments.
Q1. Could the Q8 vector be a weaker write into the same register? Thank you for pointing out the weaker-write explanation. We designed Section 7.2 to ask how much a successful steering intervention tells us about the change it makes to the model’s internal state. Q8 lets us test this directly because we have an explicit register write for comparison. The full steering vector produced the intended counterfactual next state on 64.3% of held-out cases and preserved the visible coordinate on all 400. Yet its norm was 2.19 times that of the matched register write, and only 5.7–13.4% of its squared norm lay in the register subspace. Your explanation would itself be an interesting outcome. We selected the attention heads for their effect on the next prediction, without using the register basis. If the projected component accounts for the successes, that selection recovered a usable register write indirectly, embedded in a much larger change to the residual stream. The behavioral score alone would give no indication of which part of the vector mattered. We did not have sufficient time or compute during rebuttal to run the experiments needed to settle that question. We will revise the Section 7 to distinguish what we established about the local edit from what remains open about its mechanism.
Q2. Patching experiments. We did not run the all-position patch due to budget constraints. In the submission, patching the correct-prompt activation at the final token for one block recovered 1.4% of the repeated-patch margin gain in Qwen and −1.5% in Llama (Table 13). We discuss this in more detail in our response to Reviewer iWBB. Later layers could still attend to the shuffled demonstrations, so your explanation is compatible with our result. We will make these points more explicit in the camera-ready.
Q3. Is the lexical effect a generic perturbation away from the shuffled mapping? For Llama, the submitted results indicate that it is not. Four controls of equal norm, including the FV of another task and a random direction, give normalized effects between −0.32 and 0.09, against 0.82 for the target FV (Table 13).
For Qwen, the measure in the submission compared two English answers and could not show which task a vector promotes, so we ran a test that can. For each of 64 verbs we offered four answers: the correct past tense, the correct Spanish translation, and one incorrect word in each language (for serve, the options were served, servir, joined and unirse). With the same shuffled demonstrations and the same four options, we added either the past-tense vector or the translation vector at the location and strength used in the paper. Replacing the translation vector with the past-tense vector changed the model's first choice from the correct Spanish answer to the correct English answer for 43 of 64 verbs with past-tense demonstrations and 53 of 64 with translation demonstrations. The reverse change occurred for 1 and 0 verbs. The two vectors therefore select different correct answers, which a generic perturbation would not do. Part of the difference may reflect a choice of output language, since within a single language the advantage of the matching vector is positive but less certain. We will report this experiment in Section 9.2 of the camera-ready version in place of the earlier statement about Qwen.
Conclusion: We are grateful for the review. The experiments you proposed strengthen our paper and we worked to make more concrete the link between the two halves of the paper. We hope the clarified contribution and the new results address the concerns behind your rating, and we would be grateful if you would consider raising it.
Quick note on closest work
We agree that Ma et al. is the closest related work. It was posted to arXiv on 30 July 2026 and, to our knowledge, has not appeared at a peer-reviewed venue. The ICLR 2027 reviewer guidelines state that authors are not required to compare with papers available only on arXiv, and they treat work published in the two months before the submission deadline as contemporaneous. We therefore ask that it not be counted against the novelty of the submission. We cite and discuss it in Section 2 and Appendix G.3, and we regard the two papers as complementary.
Ma et al. identify reusable interfaces among naturally occurring states without assuming labels, whereas we test whether a specific edit is used as a known variable under a known update rule.
We thank the reviewer for the detailed review. Your calculation in Q1 exposed a gap in the argument of Section 7.2, and we have run some of the experiments you proposed in Q1-Q3. Your comments also showed that the submission does not state clearly enough what it contributes or how its two halves fit together. We take these two points first because we feel they are crucial to clarify.
W1. Novelty
We agree that the distinction summarized in Table 16 is not new. It follows amnesic probing, causal abstraction and Makelov et al., and we will say so and cite these works in the camera-ready version. Those works supply the distinction and the formal language for it. The contribution of the paper lies in a setting where the distinction can be decided exactly, an internal variable shown to pass the test, and what we find when that verified case is used to assess steering. The submission did not make these prominent enough.
The setting: The task is a walk on a group of order eight, in which each state has a visible part and a hidden bit that depends on the order of the moves. Flipping the hidden bit inside the model, with the text unchanged, defines a different state whose successor under every possible next move is fixed by the group law. Whether the model computes from the edited value is therefore a prediction with a known correct outcome on each item, and the same edit predicts different outputs under different next moves. We use two groups that differ only in the rule for the hidden bit, so a result obtained in both does not depend on one rule. We regard this construction as a contribution in its own right.
The register: In models trained to write the running state, we identify a variable of one or two dimensions at the state token that holds the hidden bit, and we show that the model reads and updates it. Editing it makes the model apply whichever move follows to the edited state, the path by which it is read is localized, and in traces that the model generates itself the edited branch persists through later states to the final answer. The edit is additive and natural. It adds the difference between the model's own mean activations for the two values of the hidden bit (Section 4), so it exchanges one value the model produces for another, at the scale of the model's own activations. This is the form used in mean steering, which allows a direct comparison with steering vectors.
With this setting, the paper establishes three things about when an edit changes a value that the model goes on to use.
A correct answer after an edit is not proof. Section 6 uses a second task, a small state machine with a location and a counter, governed by rules such as "move only when the counter is 2". We took a model trained only on final answers and edited the counter so that the rule allowed a move. It gave the correct next state in 126 of 128 cases. Without any edit, the same model was correct in only 10 of 128 cases, because it moved in almost every case whether or not the rule allowed it (Table 12). Its success after the edit therefore says nothing about whether it used the edited counter. A sound test has to vary what happens after the edit.
An edit can be shown to change the value the model uses. In the same task, with a model trained to write its state, we changed the counter from 3 to 2 inside the model and left the text unchanged. Four unrelated updates later, the model received one of two instructions. Under the first, the location moves right when the counter is 2. Under the second, it moves left. The model moved right in the first case and left in the second, on every test case, and it did not move at all when the counter was first reset to 3 (Fig. 12 and Fig. 14a). An edit that pushed the model toward one answer could not produce opposite outcomes depending on a later instruction. This edit changed the counter that the model uses.
For a steering vector, changing the output does not show which of these happened. Where the variable is known, the answer can be found. In the lexical tasks no variable is known, and the evidence points the other way. The vector that works is 26 to 77 times larger than the change the model itself makes at that position, the model's own activation placed there has almost no effect, and the effect does not follow the answer when the answer is given a different label (see W2).
To our knowledge, prior work does not show that an additive, mean-based edit writes a variable that the model then reads and updates under a known rule, nor does it use such a verified case as the reference for interpreting steering vectors.