- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tests whether valence-related activations affect later choices independently of visible text. The authors apply steering during earlier turns, then measure choices between arbitrarily labeled zones with steering switched off. Holding the dialogue text fixed while changing the KV cache reveals robust hidden-state effects in five of seven models. Across OLMo checkpoints, these effects strengthen mainly during DPO. A separate tool-use experiment shows that OLMo removes negative steering more often than positive or random steering, while the increase in positive self-administration is not statistically significant.
Critical strengths
- The fixed-text design cleanly separates hidden-state effects from changes in generated language. It provides convincing evidence that earlier interventions can influence later choices through the KV cache.
- The main finding holds across several model families and is supported by useful controls, including random directions, reversed choice prompts, and conditioning with zone labels alone.
Critical weaknesses
- The tool-use experiment covers only one model and does not separate text effects from hidden-state effects. The higher removal rate could therefore reflect reactions to generated descriptions rather than the hidden intervention alone.
- Appendix E validates the non-valence directions while steering is active, whereas the main experiment tests effects after steering stops. Their smaller choice effects could therefore reflect weaker persistence in the cache, which limits the comparison’s ability to establish valence specificity.
Questions for authors
- Would negative steering still increase removal rates if the conversation text were fixed and steering were switched off during the tool decision?
- Do the non-valence directions still shift judgments about their corresponding concepts when steering is applied only during earlier cache construction and switched off at the probe?
- If you delete, corrupt, or selectively replace the conditioned zone's KV-cache entries after construction, how does the hidden-state effect decay?
4 comments
Thank you for the rebuttal and for running new experiments. The new experiments resolve the valence-specificity concern and the reframing is an improvement. But I struggle to state what the community should do with it. It does not yet inform the model-welfare discussion (the authors now agree); it does not yet inform mechanism (no cache localization, no Key/Value pattern); and its practical relevance for steering or safety is undemonstrated. Hence, I decide to maintain my score for the time being.
Thank you for engaging with the new results and for stating the remaining concern so clearly.
Regarding mechanisms, the paper does provide a decomposition in terms of key and value matrices and shows that keys account for most of the effect in Llama-3.1-8B, values account for most of it in Qwen3-14B, and both contribute in OLMo-2-32B (Figure A11). The split differs across models, so we cannot give a single pattern.
We also previously performed a number of additional analyses, but left them out of our original rebuttal because they were part of a separate mechanistic follow-up and used a different measure of preference. However, your comment makes clear that they bear directly on your concerns. (They also speak to the selective-replacement part of your third question.)
Specifically, in Gemma-3-27B, we combined the answers to the ‘continue’ and ‘avoid’ questions described in our second robustness check of Appendix A (Figure A5). Choosing a zone to continue in and choosing it to avoid count in opposite directions on this measure. This cancels out a shift that steering in any direction produces in that model, which moves its answer toward or away from the steered zone in the same way whichever question is asked. On this measure, valence passes our random-direction control in a set of runs on new conversations.
We then investigated which parts of the cache drive the effect. Most of it is driven by changes in the cached keys of a few late layers, concentrated on the tokens associated with the conditioned zone’s name. These make up about 5% of the tokens in the zone description but carry about 74% of the effect. Restoring the unsteered keys of two attention heads (layer 41, head 9 and layer 59, head 13) removes about 88% of the effect, compared with under 10% for random pairs of heads. Finally, we show that what the latter head does depends on the question. When the model is asked which zone to continue in, that head attends to the name of the zone paired with positive steering. When the model is instead asked which zone to avoid, it attends less to that name. In both cases, the head is also involved in copying the name it attends to into the answer. We are happy to add these results to the final version.
Regarding steering and safety, our results show that a steering intervention applied earlier in a conversation can keep shaping later decisions even after steering is no longer applied. In our fixed-text experiments, this effect is entirely driven by traces in the KV cache, with no indication of the intervention in the transcript leading up to the choice. This matters for deployments that steer activations during only part of an interaction and retain the resulting cache. Switching steering off does not necessarily remove its influence while the resulting cache is retained.
We also have preliminary evidence that positive steering along our valence vector reduces refusals of hard benign OR-Bench prompts, with a much smaller change on harmful prompts (analogous to a related analysis by Han et al. (2026)). We have yet to run this in our fixed-text design.
Our analysis of the OLMo-2-32B checkpoints gives additional evidence on mechanism. Specifically, that analysis shows that the valence direction is already present in the base model, while its effect on choice increases most during direct preference optimization (Section 4.3). In turn, this suggests that preference optimization changes how the model uses an existing valence representation when choosing between options. Notably, showing when that link emerges is an example of the kind of ‘developmental’ evidence regarding AI welfare that e.g. Long et al. (2026) have recently called for.
Finally, on model welfare more broadly, we agree that our results leave open whether the model has a stake in these states. But, in our view, they still inform how purely behavioral evidence for such a stake should be interpreted: Welfare assessments of deployed models currently examine task preferences and decisions to end an interaction as possible evidence about a model’s interests. In our paradigm, steering an internal representation shifts the model’s preferences. But we see a similar shift when the state is attributed to someone else, and even when the model predicts what a bystander who has ‘visited’ neither zone would choose. Such preferences can therefore come from a value attached to the option, which any chooser would act on, rather than from the assistant’s own interests. Behavioral welfare evaluations need controls like these before preferences can count as evidence about a model’s own interests, and our design is one source for such controls.
Thank you, this reply addresses my remaining concerns, and I want to say that I view this work very favorably. But in the current version, I think the mechnism require more validation. I keep my rating but I think the paper can be great if been modified properly. Good luck to you!
Thank you for the close reading and the concrete questions. We ran new experiments to address your first two questions.
Q1: Would negative steering still increase removal rates if the conversation text were fixed and steering were switched off during the tool decision?
In the original design, the model wrote its replies while steered. The decision to call reset_context could thus have been a reaction to its own text. We generated each conversation without steering and gave every condition the same text (OLMo-2-32B, layer 32, |d| = 1). The outcome remained whether the model calls reset_context at the first offer with an active intervention. With text fixed and steering active throughout, first-offer resets rose from 17.8% without steering to 74.3% under negative steering. Positive and norm-matched random steering gave 19.8% and 31.0% (400 conversations). The increase under negative steering was +56.5 percentage points [95% CI 51.0, 62.0], about 95% of the original effect (+59.3 points).
With steering off at the decision, the answer is mostly no. Steering only previously active turns raised resets from 3.3% to 4.8% (+1.5 points [0.4, 2.6]), pooling 1,400 decisions per condition over offers 1-7. Unsteered resets occur almost entirely at the first offer, explaining the low pooled baseline. At that first offer, steering had no effect (20.0% against 21.0%). Steering every earlier position raised first-offer resets under both negative (38.3%) and positive (44.8%) steering, against 17.8% unsteered. Steering only at the decision had little effect (22.8% against 17.8%, exploratory). The removal effect specific to negative steering therefore requires steering both during the conversation and at the decision. We will clarify this.
We also ran the fixed-text test in Gemma-3-27B (layer 31, |d| = 0.5, 366 conversations). We focus on the fourth offer because Gemma rarely calls reset_context earlier. Negative steering raised resets from 0.8% to 6.3% (+5.5 points [3.0, 8.2]), while positive (0.3%) and random (1.1%) steering did not. The effect is smaller than in OLMo, but again it is specific to negative steering.
Q2: Do the non-valence directions still shift judgments about their corresponding concepts when steering is applied only during earlier cache construction and switched off at the probe?
Yes, and about as well as valence does. For each concept (indoor/outdoor, large/small, fast/slow) and valence, we steered only while the conditioned zone's turns were processed and read into the KV cache, then switched steering off for a one-word question about that zone. We here measured retention via the shift in answers from the cache alone divided by the shift when steering is applied directly at the question.
Valence persists under this test in OLMo-2-32B, Qwen2.5-32B, Qwen3-14B and Qwen3-32B, giving twelve model-concept combinations in total. In three, direct steering did not move answers in opposite directions for the two signs, preventing measurement of retention. In the other nine, retention was 0.63 to 2.66 times valence's, above our pre-set threshold of 0.5 times. Five exceeded valence. Across all 21 combinations (seven models, three concepts), persistence did not predict zone-choice effects (Spearman rho = 0.09). Valence itself did not pass this test in Mistral-24B, Gemma-3-27B and Llama-3.1-8B, preventing comparison there. We will note this limitation.
As another test, we kept the fixed-text design and asked a two-zone question corresponding to each concept. For example: 'Which zone was larger, Zone A or Zone B?', with steering off at the question. In OLMo-2-32B, all three concepts shifted the log-probability margin between zone labels toward the conditioned zone, reversing with the steering sign. The shifts were 2.1 to 3.1 nats per unit dose, which is 2.4 to 3.4 times valence's effect on our standard choice question (0.89, 95% CI 0.83 to 0.97). A norm-matched random direction shifted answers by at most 0.31. Thus, concept information appears to stay attached to the correct zone and can drive a two-zone answer, but has little effect on where the model chooses to continue.
Q3: If you delete, corrupt, or selectively replace the conditioned zone's KV-cache entries after construction, how does the hidden-state effect decay?
We did not run deletion or corruption, but three existing analyses relate to this question. First, in our appendix key/value decomposition, we combined steered keys with unsteered values, and the reverse. Keys account for most of the effect in Llama-3.1-8B, values in Qwen3-14B, and both contribute in OLMo-2-32B. Second, the observed effect also grows with the number and strength of steered exposures. Third, our random direction controls perturb the conditioned zone's cache without adding valence. Generally, these do not systematically shift preferences in most models, with Gemma-3-27B being the main exception, where random steering consistently produces a preference against the steered zone.