acceptodds
Under review as a conference paper at ICLR 2027

When Does Activation Steering Change What a Model Computes From?

Abstract

Activation steering can reliably change an agent’s output by modifying its internal activations. Yet arriving at the same answer need not involve the same computation: behavioral equivalence does not imply mechanistic equivalence. We test whether an activation edit changes the state used in subsequent computation or biases that computation toward the desired output. In a controlled state-tracking task, trace supervision creates an editable register: changing its internal value makes the model apply the next operation to the edited state. If so, transplanting that activation should be effective, and steering should remain effective at the scale of the natural source-to-target activation change. Neither prediction holds in the two selected model–task settings. Replacing the activation at one layer produces less than 2% of the target-answer margin gain from patching through all remaining layers, while natural-scale steering is similarly ineffective. The steering vectors have norms 77 and 26 times the median natural change in Qwen and Llama, producing 54% and 82% of the reference effect. Thus, a successful steering intervention need not reproduce the natural target activation at the intervention layer. More generally, an intervention should be interpreted as changing the computational state only when a later computation uses the edited value according to the semantics of that state.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ceeZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an activation edit that changes a model's output changes an internal state used in subsequent computation according to the task semantics, or biases that computation toward a specific output directly. The authors start with a register-editing setting, where they fine-tune a 7B language model (LM; Qwen2.5-Coder-7B) to output states of walks on groups Q8 and D8. In this setting, they show that rank-16 activation edits that flip the hidden part of the states (while keeping the printed tokens the same) make the LM apply the next move to the newly edited internal state. Additional controls demonstrate that the edit changes the internal state rather than biasing the computation toward a specific answer, and the authors localize the state update to one attention edge in Q8 and partially to four edges in D8. They show this is consistent when tested with Mistral-7B and in factored state machines and small interpreters. In the second part of the paper, the authors study Function Vector (FV) steering and find that in Q8, the FV is mostly outside the previously identified rank-16 subspace. Additionally, in one lexical task each for Qwen2.5-7B and Llama-3.1-8B, patching the correct-prompt activation at one layer or applying the FV at the natural activation-change scale has almost no behavioral effect. Since these fail while the calibrated FV, which is 77x (Qwen) and 26x (Llama) larger in norm than the natural change, does change behavior, the authors conclude that succesful steering need not recreate the model's natural activation at the intervention site. Together with the results from the register-editing setting, the authors argue that calling an activation edit a "state write" requires showing that later computation uses the edited value according to that state's update rule (semantics of the task), not only that the behavioral output changes.

Overall, this paper is carefully executed, and its register-editing experiments provide a clean demonstration of an activation edit that changes the internal state used by later computation. However, the paper's main distinction is largely established in prior work, some of the steering experiments test a hypothesis that is not clearly motivated and only loosely connected to the paper's main question, and the dense number-heavy presentation makes the paper difficult to follow in places. I therefore lean toward rejecting the paper in its current form.

Critical strengths
  1. Clear framing. Table 16 gives a clear and practical summary of what different intervention tests can establish, which is useful for readers even though the underlying distinction is not new (see Weaknesses).
  2. Overall methodology. For the state-tracking task, the authors design the state-move transition rules such that every edit has a unique predicted next state, which together with move-swap and conflicting-continuation controls cleanly separates "supplying a state" from "supplying an answer". This design is well thought-through and appropriately chosen for the particular research question of the authors. The general approach is also good: the site of editing is chosen on validation data, localization thresholds are predeclared, and there are ten D8 splits. Additionally, the authors confirm the consistency of the core results in a second model family (Mistral-7B-v0.3 in addition to Qwen2.5-Coder-7B) and similar state-use tests hold on more complex tasks (factored state machines and small interpreters).
Critical weaknesses
  1. Limited novelty. The main distinction of the paper (very well summarized in Table 16: direct output effect vs. downstream control vs. computational state) can already be found in existing work: amnesic probing already separates encoded from used information [1], causal abstraction considers activation patching and steering as tests of whether an intervened variable plays its role in a high-level algorithm [2], and Makelov et al. [3] show that an intervention can have the intended behavioral effect without writing the intended variable. Similarly, the register test is essentially an interchange intervention with a known high-level causal model, plus the move-swap variation, and the idea of comparing states through varied future operations is very close to the cited Hidden APIs work [4]. The paper should state more clearly what is new beyond applying these ideas to their specific setting of trace-supervised models on Q8 and D8. Could the authors elaborate?
  2. Unclear motivation and scope of the steering experiments. It is unclear who holds the correspondence hypothesis as presented and tested here (that an FV recreates the natural final-token residual at one layer). Prior work already shows that successful interventions need not induce the model's natural representations/computation ([3], see above) and that steering effects can depend on spurious factors [5]. The observation that strong steering pushes activations off the natural activation manifold can also already be found in existing work (e.g., [6] and [7]). The authors should position their results relative to this work and reference studies that hold the view they argue against. Moreover, the FV used in the state-tracking task is constructed in a non-standard way (Appendix E.1: built from traces rather than in-context demonstrations, applied at the current-phase token, and defined as a phase contrast), so it is closer to contrastive steering than to an FV in the sense of Todd et al. [8]. Finally, it is unclear how the failure to recreate the natural internal state in the lexical tasks answers the paper's main question of whether activation steering changes the internal state that later computation uses, or directly biases the computation toward the output. As a result, the two halves of the paper feel only loosely connected.

[1] Y. Elazar et al. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. TACL 2021.

[2] A. Geiger et al. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. JMLR 2025.

[3] A. Makelov et al. Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.

[4] S. Ma et al. Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futures. arXiv:2607.27617, 2026.

[5] D. Tan et al. Analysing the Generalisation and Reliability of Steering Vectors. NeurIPS 2024.

[6] G. Luo et al. Learning a Generative Meta-Model of LLM Activations. ICML 2026.

[7] Z. Xu et al. Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics. ACL 2026.

[8] E. Todd et al. Function Vectors in Large Language Models. ICLR 2024.

Questions for authors
  1. In Sec. 7.2, FV efficacy (64.3%) is described as approaching register editing (80%). In this comparison, the FV update is 2.19$\times$ larger than the register write, but only 5.7-13.4% of its energy lies in the register rank-16 subspace, so its component in this subspace is about 0.5-0.8$\times$ the norm of the register write. Given this, could the FV simply be a weaker write into the same register, plus a large out-of-subspace part? Could the authors test this by applying only the FV's projection onto the subspace, and separately the FV with that projection removed?
  2. Have you tried patching the correct-prompt activations at all positions at layer ℓ⋆\ell^\star, rather than only the final token? (Considering that the subsequent layers can attend to the original context, it is not surprising that a single-block edit does not work, or requires order-of-magnitude larger steering vector norms.)
  3. In Qwen, another task's FV works as well as the target FV. Does this suggest the lexical steering effect is mostly a generic perturbation that pushes the model away from the shuffled mapping?
4 comments

We thank the reviewer for the detailed review. Your calculation in Q1 exposed a gap in the argument of Section 7.2, and we have run some of the experiments you proposed in Q1-Q3. Your comments also showed that the submission does not state clearly enough what it contributes or how its two halves fit together. We take these two points first because we feel they are crucial to clarify.

W1. Novelty

We agree that the distinction summarized in Table 16 is not new. It follows amnesic probing, causal abstraction and Makelov et al., and we will say so and cite these works in the camera-ready version. Those works supply the distinction and the formal language for it. The contribution of the paper lies in a setting where the distinction can be decided exactly, an internal variable shown to pass the test, and what we find when that verified case is used to assess steering. The submission did not make these prominent enough.

The setting: The task is a walk on a group of order eight, in which each state has a visible part and a hidden bit that depends on the order of the moves. Flipping the hidden bit inside the model, with the text unchanged, defines a different state whose successor under every possible next move is fixed by the group law. Whether the model computes from the edited value is therefore a prediction with a known correct outcome on each item, and the same edit predicts different outputs under different next moves. We use two groups that differ only in the rule for the hidden bit, so a result obtained in both does not depend on one rule. We regard this construction as a contribution in its own right.

The register: In models trained to write the running state, we identify a variable of one or two dimensions at the state token that holds the hidden bit, and we show that the model reads and updates it. Editing it makes the model apply whichever move follows to the edited state, the path by which it is read is localized, and in traces that the model generates itself the edited branch persists through later states to the final answer. The edit is additive and natural. It adds the difference between the model's own mean activations for the two values of the hidden bit (Section 4), so it exchanges one value the model produces for another, at the scale of the model's own activations. This is the form used in mean steering, which allows a direct comparison with steering vectors.

With this setting, the paper establishes three things about when an edit changes a value that the model goes on to use.

  1. A correct answer after an edit is not proof. Section 6 uses a second task, a small state machine with a location and a counter, governed by rules such as "move only when the counter is 2". We took a model trained only on final answers and edited the counter so that the rule allowed a move. It gave the correct next state in 126 of 128 cases. Without any edit, the same model was correct in only 10 of 128 cases, because it moved in almost every case whether or not the rule allowed it (Table 12). Its success after the edit therefore says nothing about whether it used the edited counter. A sound test has to vary what happens after the edit.

  2. An edit can be shown to change the value the model uses. In the same task, with a model trained to write its state, we changed the counter from 3 to 2 inside the model and left the text unchanged. Four unrelated updates later, the model received one of two instructions. Under the first, the location moves right when the counter is 2. Under the second, it moves left. The model moved right in the first case and left in the second, on every test case, and it did not move at all when the counter was first reset to 3 (Fig. 12 and Fig. 14a). An edit that pushed the model toward one answer could not produce opposite outcomes depending on a later instruction. This edit changed the counter that the model uses.

  3. For a steering vector, changing the output does not show which of these happened. Where the variable is known, the answer can be found. In the lexical tasks no variable is known, and the evidence points the other way. The vector that works is 26 to 77 times larger than the change the model itself makes at that position, the model's own activation placed there has almost no effect, and the effect does not follow the answer when the answer is given a different label (see W2).

To our knowledge, prior work does not show that an additive, mean-based edit writes a variable that the model then reads and updates under a known rule, nor does it use such a verified case as the reference for interpreting steering vectors.

W2. Hendel et al. show that a single hidden state computed from demonstrations can carry a task when it is transplanted at one layer, and Todd et al. describe the FV as a compact representation of the demonstrated task. On a natural reading of these results, adding an FV supplies the representation of the task that the model itself would form. The correspondence hypothesis states this reading in a form that can be tested, and we will introduce it with these references in the camera-ready version.

With regards to how the two halves of the paper are related, both halves ask the same question. When an activation edit changes a model's output, has it changed a value that later computation uses, or has it only biased the computation toward that output?

The first half of the paper answers the question in a setting where a direct answer is possible. The variable and its update rule are known, so we can vary the operation that follows the edit and check that the model applies it to the edited value. From this we learn what evidence is needed to conclude that an edit changed the model's state, and that an additive edit can provide it.

The second half puts the same question to steering as it is ordinarily used. There the output changes, but no variable or rule is known, and the direct test cannot be run. The one concrete candidate for a state that the vector might supply is the activation the model itself produces on the correct prompt. If steering worked by supplying that state, transplanting it should be effective, and steering should remain effective at its natural scale. Neither prediction holds. We feel your explanation of why is plausible. A vector that succeeds at the final token, at 26 to 77 times the size of the model's own change there, therefore cannot be succeeding by reproducing what the model computes at that site (Table 13). These results do not establish that the vector merely biases the output, and we will say so in the camera-ready version. They establish that a successful steering result is not enough, without further evidence, to conclude that a state was written, which is the claim of the paper. We are not aware of prior work that reports this comparison, and we will discuss it in relation to Makelov et al., Tan et al., Luo et al. and Xu et al., as you recommend.

The label-swap experiment described in section 9 applies the logic of the first half of the paper within the lexical tasks. The correct answer is held fixed and the label that must express it is exchanged. An edit that supplies a value used by later computation should follow the new label, in the way the edited counter supports either later instruction. The steering effect fails to do so. The component that follows the assigned label is negative in both models (−0.315 in Qwen and −0.250 in Llama; both 95% intervals exclude zero), including on items that the models solve under both assignments (Table 14).

We now discuss the questions raised by reviewer ceeZ and the requested experiments.

Q1. Could the Q8 vector be a weaker write into the same register? Thank you for pointing out the weaker-write explanation. We designed Section 7.2 to ask how much a successful steering intervention tells us about the change it makes to the model’s internal state. Q8 lets us test this directly because we have an explicit register write for comparison. The full steering vector produced the intended counterfactual next state on 64.3% of held-out cases and preserved the visible coordinate on all 400. Yet its norm was 2.19 times that of the matched register write, and only 5.7–13.4% of its squared norm lay in the register subspace. Your explanation would itself be an interesting outcome. We selected the attention heads for their effect on the next prediction, without using the register basis. If the projected component accounts for the successes, that selection recovered a usable register write indirectly, embedded in a much larger change to the residual stream. The behavioral score alone would give no indication of which part of the vector mattered. We did not have sufficient time or compute during rebuttal to run the experiments needed to settle that question. We will revise the Section 7 to distinguish what we established about the local edit from what remains open about its mechanism.

Q2. Patching experiments. We did not run the all-position patch due to budget constraints. In the submission, patching the correct-prompt activation at the final token for one block recovered 1.4% of the repeated-patch margin gain in Qwen and −1.5% in Llama (Table 13). We discuss this in more detail in our response to Reviewer iWBB. Later layers could still attend to the shuffled demonstrations, so your explanation is compatible with our result. We will make these points more explicit in the camera-ready.

Q3. Is the lexical effect a generic perturbation away from the shuffled mapping? For Llama, the submitted results indicate that it is not. Four controls of equal norm, including the FV of another task and a random direction, give normalized effects between −0.32 and 0.09, against 0.82 for the target FV (Table 13).

For Qwen, the measure in the submission compared two English answers and could not show which task a vector promotes, so we ran a test that can. For each of 64 verbs we offered four answers: the correct past tense, the correct Spanish translation, and one incorrect word in each language (for serve, the options were served, servir, joined and unirse). With the same shuffled demonstrations and the same four options, we added either the past-tense vector or the translation vector at the location and strength used in the paper. Replacing the translation vector with the past-tense vector changed the model's first choice from the correct Spanish answer to the correct English answer for 43 of 64 verbs with past-tense demonstrations and 53 of 64 with translation demonstrations. The reverse change occurred for 1 and 0 verbs. The two vectors therefore select different correct answers, which a generic perturbation would not do. Part of the difference may reflect a choice of output language, since within a single language the advantage of the matching vector is positive but less certain. We will report this experiment in Section 9.2 of the camera-ready version in place of the earlier statement about Qwen.

Conclusion: We are grateful for the review. The experiments you proposed strengthen our paper and we worked to make more concrete the link between the two halves of the paper. We hope the clarified contribution and the new results address the concerns behind your rating, and we would be grateful if you would consider raising it.

AuthorsOpenReview ↗

Quick note on closest work

We agree that Ma et al. is the closest related work. It was posted to arXiv on 30 July 2026 and, to our knowledge, has not appeared at a peer-reviewed venue. The ICLR 2027 reviewer guidelines state that authors are not required to compare with papers available only on arXiv, and they treat work published in the two months before the submission deadline as contemporaneous. We therefore ask that it not be counted against the novelty of the submission. We cite and discuss it in Section 2 and Appendix G.3, and we regard the two papers as complementary.

Ma et al. identify reusable interfaces among naturally occurring states without assuming labels, whereas we test whether a specific edit is used as a known variable under a known update rule.

Reviewer_iWBBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks whether activation steering changes the state that subsequent computation depends on, or merely pushes that computation toward a target output. The authors distinguish three properties: behavioral efficacy, directional specificity, and state-like use. They argue that the first two do not imply the third. To distinguish them, the authors design an elegant group-walk task on Q8Q_8 and D8D_8. Each state has a visible coordinate that can be read directly from the moves, plus a hidden bit that depends on move order. Editing the hidden bit therefore has a unique, move-dependent downstream consequence. This lets the authors test whether a model is only being pushed toward a particular lexical output, or whether it actually maintains an internal state and derives the next state from it. The paper finds that steering that changes behavior is not necessarily writing a state. Claiming a state write requires showing that later computation uses the edited value according to that variable’s semantics. Beyond these findings, I believe the task could serve as a new benchmark for validating the causal claims of steering methods, because it provides ground truth for what a genuine state edit should do. This is a positive contribution to research on model behavior.

Critical strengths
  1. The paper makes a clear distinction between behavioral effectiveness and causal state editing. While previous activation steering studies often evaluate interventions based on changes in model outputs, this work investigates whether the edited representations are actually used by subsequent computations.
  2. The use of non-commutative groups Q8Q_8 and D8D_8 provides a well-designed experimental setting with explicit state-transition rules and verifiable counterfactual outcomes.
  3. The authors conduct several complementary analyses, including low-rank register editing, attention-edge localization, natural activation patching, equal-norm controls, and add-then-subtract interventions.
Critical weaknesses
  1. The main evidence for causal state editing comes from synthetic state-tracking tasks with explicit intermediate-state supervision. In contrast, the experiments on natural language tasks focus primarily on Function Vectors and a small number of models and lexical transformations. It remains unclear whether the conclusions could extend to other widely used steering methods.
  2. The paper interprets the limited effectiveness of one-block correct-prompt activation patching as evidence against the Correspondence Hypothesis. However, task-related representations may be distributed across multiple token positions, layers, or interacting computational components.
Questions for authors
  • Do the authors expect the observed mismatch between behavioral steering and natural state restoration to hold for other steering methods, such as Difference-in-Means or CAA? Could the conclusions depend on the particular construction of Function Vectors?
  • Would multi-position or subspace-level patching provide a stronger test of the Correspondence Hypothesis?
  • Could the authors clarify how the rank-16 register subspace is constructed from the visible-conditioned mean differences? In my understanding, each visible coordinate with different hidden bit would only generate one direction, which would lead to 4 directions.
2 comments

We thank reviewer iWBB for their thoughtful review. We agree that this setting can serve as a benchmark for steering methods because it lets us predict how the same edited value should affect different future operations. Section 7.2 gives one example. A Function Vector produces the target next state on 64% of held-out Q8 walks even though most of its displacement lies outside the register subspace. We have plans to build on this paper, possibly to develop a new benchmark or eval metric.

Q1. Does the Function Vector result extend to Difference-in-Means or CAA?

In Qwen, the mismatch also appears with a difference-in-means direction. We averaged the final-token residual differences between correct and shuffled prompts, then scaled that direction to the Function Vector’s norm. On held-out present-to-past prompts, the mean direction recovered 61% of the target-answer margin gain from repeated correct-prompt patching. A single transplant of the correct activation at the same site recovered only 1.4% (Table 13). The Qwen comparison therefore rules out one explanation for the mismatch in this task: it persists even when we construct the steering direction from residual differences alone, without the attention-head selection used to build a Function Vector. In Llama’s translation task, the mean direction recovered 8.5%, with a 95% interval spanning zero. In Section 8.1 of the camera-ready, we will report this comparison for both models and distinguish it from a CAA test, which remains open.

The paper also includes a positive example in which mean-difference steering supplies a state that later computation uses. In the factored state machine, editing a stored counter from 3 to 2 enables a later conditional action. After four intervening updates, one possible action moves the model’s location right and another moves it left. The two prompts share the same prefix through the edited token, so the edit occurs before the model sees which action it must take. For all 128 held-out counter and mode edits, the mean-difference intervention produced the correct complete continuation for both actions in each of two training seeds. In the counter example, overwriting the counter before use restored the original downstream outcome (Appendix D.4, Figure 14). Because the two actions require opposite outcomes, a fixed push toward either answer cannot explain their joint success.

We also tested the effect of explicit state supervision in the group-walk experiments. The two training conditions used the same move sequences and final answers. On held-out Q8 walks, Mistral’s next hidden bit agreed with the update from the edited state in 93% of cases after training on intermediate states, compared with 51% after training on final answers alone (Appendix B.2, Table 4). In the camera-ready, we will bring this comparison and the paired-action example into the main text so readers can see the conditions under which the state-use test passes.

Q2. Would multi-position or subspace patching provide a stronger correspondence test? Yes. Section 8.1 tests a specific prediction at the site where steering succeeds. If the correct prompt’s final-token activation is sufficient to restore target behavior, transplanting it once at the selected block should recover the target-answer margin. The transplant produced 1.4% of the gain from repeatedly patching the final token through all remaining blocks in Qwen and −1.5% in Llama. At the same sites, the calibrated Function Vectors produced 54% and 82% of that reference gain, respectively (Table 13). When we rescaled the Function Vectors to the size of each prompt’s natural activation change, their effects fell below 2% of the reference gain in both settings.

Earlier tokens still represent the shuffled demonstrations and remain available to later layers. Jointly patching selected token positions would test whether their activations are sufficient together. Patching a candidate task subspace would test whether restoring just its target component is sufficient while preserving the source activation in the remaining directions. A successful joint patch would support the reviewer’s distributed-state explanation. It would still leave open how adding a single Function Vector produces its effect in the shuffled prompt. In the camera-ready, we will state in the abstract and introduction that our experiment tests restoration at the final token of a single block.

Q3. How was the rank-16 register subspace constructed?

You are right that four visible-conditioned mean differences can span at most four dimensions. Section 4 incorrectly says that we used those differences to construct the subspace. We fitted PCA to individual, centered residual activations at the current-state token and took the leading 16 principal directions. Their span, U, includes variation within each state. We used the state labels separately to estimate each state's centroid, (s). Both estimates used calibration examples disjoint from the held-out evaluation. The equation in Section 4 also needs a correction. Let z be the host activation in state s=(v,h), and let U be the orthogonal projector onto U. The target state s=(v,1-h) retains the visible coordinate and flips the hidden bit. The implemented edit is z~=z+ΠU(μ(s~)−z).\tilde{z} = z + \Pi_U\bigl(\mu(\tilde{s}) - z\bigr). This sets the component of z in U to the target centroid's projection and preserves the component orthogonal to U. The printed equation subtracts the host-state centroid (s) where the implementation subtracts the host activation z. Rank 16 was our chosen edit dimension. In the rank sweep, the measured effect saturates at rank 1 in Q8 and rank 2 in D8. In the camera-ready, we will correct the construction and equation in Section 4 and document the procedure used for Table 3 and the rank sweep.

Conclusion

We hope this evidence and the camera-ready revisions described above address your concerns, and that you will consider a stronger recommendation. Thank you again for your consideration!

Reviewer_UocjOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether an activation edit that changes a model's answer also changes the state that later computation uses. In a synthetic state tracking task with known update rules, it shows that a low rank edit of a written state token is used by later steps as the new state. For Function Vector steering in Qwen and Llama, transplanting the model's own target activation at one layer or using a steering vector of natural size has almost no effect, while the effective vector is 77 and 26 times larger than the natural change.

Critical strengths

Strengths

  1. The paper separates three properties that steering work often treats as one, namely changing the behavior, depending on a specific direction, and being used by later computation as a state. In a task whose update rule is known, it tests the third property by changing the next move and checking whether the outcome follows the rule. The register edit passes this test, with 0.80 and 0.91 agreement against 0.02 for random edits, and the result replicates in Mistral.

  2. The register experiments are carefully controlled. The edit site and layer are fixed on a validation split before testing, random and orthogonal edits of the same rank serve as controls, and the conflicting continuation test rules out copying a stored answer. The localization result is further checked with matched controls, restoration and 1000 random four edge routes in Appendix C.4.

Critical weaknesses

Weaknesses

  1. (Partial Solved) The main conclusion overlaps with what prior work has already shown. Todd et al. [1] found that adding averaged hidden states from task prompts is weaker than adding function vectors, Venkatesh and Mahendran Kurapath [2] showed that many different steering directions give the same behavior, and Makelov et al. [3] showed that an activation edit can change the output through a pathway the model does not normally use. Against this background, the new part is mainly the per prompt comparison with the matched target activation, which refines rather than changes the existing view. Makelov et al. is not cited.

  2. (Not Solved) The steering experiments use a setting that rarely arises in practice. The shuffled prompts in Section 8 contain ten demonstrations whose answers are deliberately mismatched, so the patched activation at the last token competes with misleading examples that later layers can still read. Steering is often applied to ordinary prompts, and Todd et al. [1] also evaluate function vectors on zero shot and natural text inputs without such conflicting examples. The register experiments likewise rely on models fine tuned to write states on a synthetic group task.

  3. (Partial Solved) The number of tested settings is too small to support the general claim in the title. The correspondence test covers one task, one layer and 64 prompts per model, and both settings were chosen because steering was strong there, as Section 8.1 states. The one layer patch is not tried at other layers, although Hendel et al. [4] found that injecting a task vector at a single suitable layer can steer the model even when the demonstrations show a different task.

  4. (Not Solved) The writing is hard to follow, and Figure 1 does not explain itself. Figure 1 appears before the task is defined in Section 3, uses labels such as 00|1 and h=1 without explanation, and calls the hidden bit p in panel (a) while the text calls it h. The main text introduces several custom metrics, such as R, CUS, T and L, and relies on appendices for key results, for example Table 16 in Section 8 and the extensions summarized in Section 6.

references:

[1] E. Todd, M. L. Li, A. Sen Sharma, A. Mueller, B. C. Wallace, D. Bau. Function Vectors in Large Language Models. ICLR, 2024. arXiv:2310.15213.

[2] S. Venkatesh, A. Mahendran Kurapath. On the Non-Identifiability of Steering Vectors in Large Language Models. arXiv:2602.06801, 2026.

[3] A. Makelov, G. Lange, A. Geiger, N. Nanda. Is This the Subspace You Are Looking For? An Interpretability Illusion for Subspace Activation Patching. ICLR, 2024. arXiv:2311.17030.

[4] R. Hendel, M. Geva, A. Globerson. In-Context Learning Creates Task Vectors. Findings of EMNLP, 2023. arXiv:2310.15916.

Questions for authors

No further questions.

1 comment

We thank the reviewer for your thoughtful review. We address both your points in our responses to Reviewer ceeZ, and we would be extremely grateful if you would consider raising your score in light of those clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.