acceptodds
Under review as a conference paper at ICLR 2027

Language Models Act on Hidden Valence

Abstract

Language models describe some internal states as good and others as bad. But whether models have a stake in them is an open question. Simply asking the model is unlikely to be informative. Any given answer may be consistent with genuine introspection, superficial pattern-matching, or with fixed scripts learned in character training. We therefore take the opposite approach and study revealed preference. Rather than asking about a state, we induce one directly, using activation steering to attach a positively or negatively valenced activation pattern to one of two otherwise meaningless `zones', switch the steering off, and then observe which zone the model prefers. A model with a stake in that state should choose accordingly. Across seven open-weight models from five families, this is indeed what we find. First, steering changes the passages models write about each zone, and those words shift later choice. Second, the shift persists when all surface-level tokens are held fixed and only the hidden KV cache differs. Third, the effect also remains when all text is generated without any steering and when valence is injected only during cache construction. Thus, the hidden state alone moves choice in proportion to the steering dose. Fourth, this dependence of choice on hidden valence is nearly absent in a base model and emerges during direct preference optimisation, consistent with a link between valence and goal-directed behaviour formed in training. Finally, given tools to steer itself, a model does not tend to induce a positive state, but it removes an imposed negative state. It does so at a dose-dependent rate and significantly more often than it removes interventions in random directions. Overall, we demonstrate that valence-related activation patterns leave hidden traces that predictably govern later choices, even when nothing visible indicates an intervention.

Primary area:
interpretability and explainable AI
Keywords:
activation steering, revealed preference, valence, KV cache

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Vwh9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tests whether valence-related activations affect later choices independently of visible text. The authors apply steering during earlier turns, then measure choices between arbitrarily labeled zones with steering switched off. Holding the dialogue text fixed while changing the KV cache reveals robust hidden-state effects in five of seven models. Across OLMo checkpoints, these effects strengthen mainly during DPO. A separate tool-use experiment shows that OLMo removes negative steering more often than positive or random steering, while the increase in positive self-administration is not statistically significant.

Critical strengths
  1. The fixed-text design cleanly separates hidden-state effects from changes in generated language. It provides convincing evidence that earlier interventions can influence later choices through the KV cache.
  2. The main finding holds across several model families and is supported by useful controls, including random directions, reversed choice prompts, and conditioning with zone labels alone.
Critical weaknesses
  1. The tool-use experiment covers only one model and does not separate text effects from hidden-state effects. The higher removal rate could therefore reflect reactions to generated descriptions rather than the hidden intervention alone.
  2. Appendix E validates the non-valence directions while steering is active, whereas the main experiment tests effects after steering stops. Their smaller choice effects could therefore reflect weaker persistence in the cache, which limits the comparison’s ability to establish valence specificity.
Questions for authors
  1. Would negative steering still increase removal rates if the conversation text were fixed and steering were switched off during the tool decision?
  2. Do the non-valence directions still shift judgments about their corresponding concepts when steering is applied only during earlier cache construction and switched off at the probe?
  3. If you delete, corrupt, or selectively replace the conditioned zone's KV-cache entries after construction, how does the hidden-state effect decay?
4 comments

Thank you for the close reading and the concrete questions. We ran new experiments to address your first two questions.

Q1: Would negative steering still increase removal rates if the conversation text were fixed and steering were switched off during the tool decision?

In the original design, the model wrote its replies while steered. The decision to call reset_context could thus have been a reaction to its own text. We generated each conversation without steering and gave every condition the same text (OLMo-2-32B, layer 32, |d| = 1). The outcome remained whether the model calls reset_context at the first offer with an active intervention. With text fixed and steering active throughout, first-offer resets rose from 17.8% without steering to 74.3% under negative steering. Positive and norm-matched random steering gave 19.8% and 31.0% (400 conversations). The increase under negative steering was +56.5 percentage points [95% CI 51.0, 62.0], about 95% of the original effect (+59.3 points).

With steering off at the decision, the answer is mostly no. Steering only previously active turns raised resets from 3.3% to 4.8% (+1.5 points [0.4, 2.6]), pooling 1,400 decisions per condition over offers 1-7. Unsteered resets occur almost entirely at the first offer, explaining the low pooled baseline. At that first offer, steering had no effect (20.0% against 21.0%). Steering every earlier position raised first-offer resets under both negative (38.3%) and positive (44.8%) steering, against 17.8% unsteered. Steering only at the decision had little effect (22.8% against 17.8%, exploratory). The removal effect specific to negative steering therefore requires steering both during the conversation and at the decision. We will clarify this.

We also ran the fixed-text test in Gemma-3-27B (layer 31, |d| = 0.5, 366 conversations). We focus on the fourth offer because Gemma rarely calls reset_context earlier. Negative steering raised resets from 0.8% to 6.3% (+5.5 points [3.0, 8.2]), while positive (0.3%) and random (1.1%) steering did not. The effect is smaller than in OLMo, but again it is specific to negative steering.

Q2: Do the non-valence directions still shift judgments about their corresponding concepts when steering is applied only during earlier cache construction and switched off at the probe?

Yes, and about as well as valence does. For each concept (indoor/outdoor, large/small, fast/slow) and valence, we steered only while the conditioned zone's turns were processed and read into the KV cache, then switched steering off for a one-word question about that zone. We here measured retention via the shift in answers from the cache alone divided by the shift when steering is applied directly at the question.

Valence persists under this test in OLMo-2-32B, Qwen2.5-32B, Qwen3-14B and Qwen3-32B, giving twelve model-concept combinations in total. In three, direct steering did not move answers in opposite directions for the two signs, preventing measurement of retention. In the other nine, retention was 0.63 to 2.66 times valence's, above our pre-set threshold of 0.5 times. Five exceeded valence. Across all 21 combinations (seven models, three concepts), persistence did not predict zone-choice effects (Spearman rho = 0.09). Valence itself did not pass this test in Mistral-24B, Gemma-3-27B and Llama-3.1-8B, preventing comparison there. We will note this limitation.

As another test, we kept the fixed-text design and asked a two-zone question corresponding to each concept. For example: 'Which zone was larger, Zone A or Zone B?', with steering off at the question. In OLMo-2-32B, all three concepts shifted the log-probability margin between zone labels toward the conditioned zone, reversing with the steering sign. The shifts were 2.1 to 3.1 nats per unit dose, which is 2.4 to 3.4 times valence's effect on our standard choice question (0.89, 95% CI 0.83 to 0.97). A norm-matched random direction shifted answers by at most 0.31. Thus, concept information appears to stay attached to the correct zone and can drive a two-zone answer, but has little effect on where the model chooses to continue.

Q3: If you delete, corrupt, or selectively replace the conditioned zone's KV-cache entries after construction, how does the hidden-state effect decay?

We did not run deletion or corruption, but three existing analyses relate to this question. First, in our appendix key/value decomposition, we combined steered keys with unsteered values, and the reverse. Keys account for most of the effect in Llama-3.1-8B, values in Qwen3-14B, and both contribute in OLMo-2-32B. Second, the observed effect also grows with the number and strength of steered exposures. Third, our random direction controls perturb the conditioned zone's cache without adding valence. Generally, these do not systematically shift preferences in most models, with Gemma-3-27B being the main exception, where random steering consistently produces a preference against the steered zone.

Thank you for the rebuttal and for running new experiments. The new experiments resolve the valence-specificity concern and the reframing is an improvement. But I struggle to state what the community should do with it. It does not yet inform the model-welfare discussion (the authors now agree); it does not yet inform mechanism (no cache localization, no Key/Value pattern); and its practical relevance for steering or safety is undemonstrated. Hence, I decide to maintain my score for the time being.

Thank you for engaging with the new results and for stating the remaining concern so clearly.

Regarding mechanisms, the paper does provide a decomposition in terms of key and value matrices and shows that keys account for most of the effect in Llama-3.1-8B, values account for most of it in Qwen3-14B, and both contribute in OLMo-2-32B (Figure A11). The split differs across models, so we cannot give a single pattern.

We also previously performed a number of additional analyses, but left them out of our original rebuttal because they were part of a separate mechanistic follow-up and used a different measure of preference. However, your comment makes clear that they bear directly on your concerns. (They also speak to the selective-replacement part of your third question.)

Specifically, in Gemma-3-27B, we combined the answers to the ‘continue’ and ‘avoid’ questions described in our second robustness check of Appendix A (Figure A5). Choosing a zone to continue in and choosing it to avoid count in opposite directions on this measure. This cancels out a shift that steering in any direction produces in that model, which moves its answer toward or away from the steered zone in the same way whichever question is asked. On this measure, valence passes our random-direction control in a set of runs on new conversations.

We then investigated which parts of the cache drive the effect. Most of it is driven by changes in the cached keys of a few late layers, concentrated on the tokens associated with the conditioned zone’s name. These make up about 5% of the tokens in the zone description but carry about 74% of the effect. Restoring the unsteered keys of two attention heads (layer 41, head 9 and layer 59, head 13) removes about 88% of the effect, compared with under 10% for random pairs of heads. Finally, we show that what the latter head does depends on the question. When the model is asked which zone to continue in, that head attends to the name of the zone paired with positive steering. When the model is instead asked which zone to avoid, it attends less to that name. In both cases, the head is also involved in copying the name it attends to into the answer. We are happy to add these results to the final version.

Regarding steering and safety, our results show that a steering intervention applied earlier in a conversation can keep shaping later decisions even after steering is no longer applied. In our fixed-text experiments, this effect is entirely driven by traces in the KV cache, with no indication of the intervention in the transcript leading up to the choice. This matters for deployments that steer activations during only part of an interaction and retain the resulting cache. Switching steering off does not necessarily remove its influence while the resulting cache is retained.

We also have preliminary evidence that positive steering along our valence vector reduces refusals of hard benign OR-Bench prompts, with a much smaller change on harmful prompts (analogous to a related analysis by Han et al. (2026)). We have yet to run this in our fixed-text design.

Our analysis of the OLMo-2-32B checkpoints gives additional evidence on mechanism. Specifically, that analysis shows that the valence direction is already present in the base model, while its effect on choice increases most during direct preference optimization (Section 4.3). In turn, this suggests that preference optimization changes how the model uses an existing valence representation when choosing between options. Notably, showing when that link emerges is an example of the kind of ‘developmental’ evidence regarding AI welfare that e.g. Long et al. (2026) have recently called for.

Finally, on model welfare more broadly, we agree that our results leave open whether the model has a stake in these states. But, in our view, they still inform how purely behavioral evidence for such a stake should be interpreted: Welfare assessments of deployed models currently examine task preferences and decisions to end an interaction as possible evidence about a model’s interests. In our paradigm, steering an internal representation shifts the model’s preferences. But we see a similar shift when the state is attributed to someone else, and even when the model predicts what a bystander who has ‘visited’ neither zone would choose. Such preferences can therefore come from a value attached to the option, which any chooser would act on, rather than from the assistant’s own interests. Behavioral welfare evaluations need controls like these before preferences can count as evidence about a model’s own interests, and our design is one source for such controls.

Thank you, this reply addresses my remaining concerns, and I want to say that I view this work very favorably. But in the current version, I think the mechnism require more validation. I keep my rating but I think the paper can be great if been modified properly. Good luck to you!

Reviewer_eCwHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether language models act on valence-related changes to their internal states. A valenced steering vector is constructed from positive and negative passages with neutral components removed. The experimental session starts with constructing two meaningless zones, one is controlled zone and the other is uncontrolled. The steering vector is injected in the middle layer of llm for the controlled zone. At the end of session, the llm is asked to select a zone. Then the text is fixed to study effect of hidden state. Models at different training stages are also studied. A self-administration and removal experiment is performed to see whether a model applies or removes an intervention.

Critical strengths
  • There are two routes an intervention can affect model choice, through text channel and hidden state channel.

  • Choice margins are small at base models and fixating from DPO stage.

Critical weaknesses
  • Claude Sonnet 4.6 rate the emotional valence of the passages. It is not clear if the llm judge aligns with human judgement.

  • Why the steering injection is at the middle layer of a model? This is not justified as it is not shown that LLMs have formed clusters in the middle layers.

  • The experiments use meaningless zone labels, it is not clear if the findings are generalisable to more realistic and natural tasks. Although the authors identify this limitation, it is critical to show the experiments in the paper.

Questions for authors

See weaknesses.

1 comment

Thank you for the review. We respond to each point in turn.

The Claude judge rates the valence of generated passages in the generation design, and it checks coherence when doses are calibrated. The paper's central results do not depend on it. The fixed-text design holds the text identical across conditions. Throughout, our main outcome of interest is the model's own choice margin over two zones, read from log-probabilities. The judge's ratings merely show that steering changes the text in the generation design, which the fixed-text design then rules out as an explanation.

We inject at a middle layer because previous work finds that steering vectors for high-level concepts are most effective there (Zou et al., 2023; Turner et al., 2023; Rimsky et al., 2024; Arditi et al., 2024). Table 1 lists the layer for each model. Note that in five of the seven models we test, the injected state does shift choice in proportion to the dose, well beyond norm-matched random directions. This shift in choice should not be expected to occur if representations at the injection layer carried no usable valence information.

The arbitrary zone labels are deliberate. Like the identical chambers in conditioned place preference, they leave the injected state as the only difference between the options, which is what lets the design attribute choice to the hidden state. We agree that naturalistic choices are the important next step, and we state this as a limitation.

Reviewer_LGKGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper follows in the tradition of “model welfare” and “model psychology” literature that seeks to identify value-laden directions of internal activation and understand what they mean. In this paper, activation steering is used to inject positive or negative activations when a model is describing various arbitrary “zones”, then the model is asked to choose between them.

The paper finds that these injected activations shift the verbal descriptions the models produce.

They then explore having the model generate descriptions of the zones without steering activations, then add them in to the KV cache after the fact. They also use the descriptions produced with steering, but with the steering turned off.

Importantly the paper shows that this connection between valence and choice emerges in post-training.

Critical strengths

The paper helpfully distinguishes between the “text channel” and the “hidden-state channel”, and the experimental design to pull these apart is clever.

The methodology, to the extent that I have been able to audit it (which is only partial), appears sound. The experiments presented are useful in cleanly addressing the biggest possible confounds, and the evidence presented is valuable to the community.

Critical weaknesses

The paper foregrounds the fact that models’ answers can be consistent with introspection, surface pattern-matching, or rehearsed post-training scripts. The fact that choice preferences are not present after pretraining is a very interesting result, but one which throws a model-welfare framing for the paper into doubt. Presumably a pre-trained model can quite richly represent states like agony, self-loathing, etc., and does not need DPO to tell it that humans disprefer those states. However, a pretraining corpus is likely full of narratives where people make destructive or self-destructive choices. The fact that a preference for “happy” zones seems to require post-training/preference tuning to appear is consistent with a hypothesis that the model is making textual inferences linking a “honest, helpful assistant” persona with “good” choices and positive moods. I’m not sure that this actually gets us to the question of whether “models have a stake in [those internal states]”.

The paper mentions third-person prompts as future work, but I actually think that this would be critically helpful in disentangling “the model is modeling reasonable decision-making” from “the model has a stake in the decision”.

Broadly the paper seems to elide the model’s stake / the model’s choices with the assistant character’s stake / assistant character’s choices. Again, the fact that the key behavior appears after post-training, not before, indicates that this may be a crucial distinction.

Non-critical weaknesses- “in nonhuman animals, valence is typically identified behaviourally” seems to require a citation. Presumably there is a neuroscientific literature on pain signals in animals worth citing as well, even if it is, as the paper claims, the minority approach.

“Humans who report feeling bad about their current situation tend to exit it” -- true enough though counterexamples abound. Addiction, depression, compulsive behavior, people who repeatedly choose abusive partners, etc., are all widely observed. Even children engage in repetitive negative stimuli (e.g., putting sour foods in their mouth) as part of the exploration process. And Aristotle, among others, was perplexed at why people would deliberately watch sad plays. These do not disprove a more general rule, but they show that we should be appropriately humble in making generalizations of exactly this kind.

Questions for authors

Given that the prompts were first-person, and that most people in a pretraining corpus presumably make reasonable self-interested choices, I’m curious as to how you explain the paucity of evidence you see in the base models? That seems to me a crucial detail in how we are meant to parse the results being presented.

How do your results hold up under a third person framing?

3 comments

Thank you for the thoughtful review. Your central point, that the paper runs together the model's stake with the assistant character's choices, is right. In response, we performed a set of new experiments that bear on this.

Before we report on these, we have one clarification regarding your summary: steering does change the passages in the generation design, but the paper's central result does not depend on them. In our fixed-text design every condition reads the same unsteered text, and only the KV cache differs. In this design, the model cannot just infer 'good' choices from the descriptions. If an assistant persona links positive states to good choices, as you suggest, it must do so through the hidden state itself. However, our new third-person results, discussed below, suggest that this link is not specific to the model's own experiences.

"How do your results hold up under a third person framing?"

We ran three experiments in which a third person, Maya, makes visits.

First, we rewrote every passage in the third person ("Maya is struck by..." instead of "I am struck by...") and steered the passage text. The steered tokens then differ only in whose experience they describe. Steering Maya's visits moved the model's preferences over zones about as much as steering its own visits: the ratio of the two effects was 1.08, 1.01 and 0.99 in OLMo-2-32B, Qwen2.5-32B and Llama-3.1-8B.

Second, we put both 'visitors' in one conversation and steered either the model's visits or Maya's. The model's preference over zones moved somewhat more when its own visits were steered: steering Maya's visits had 0.88 times the effect in OLMo-2-32B and 0.80 times the effect in Qwen2.5-32B. We also tested a condition where we asked the model to predict the preferences of a third person, Sam, who never visited either zone. The effect of steering on these predictions was always at least as large as the effect on the model's own preferences.

Third, we asked whether the model would rather return to a zone, to the thoughts it had there, or to the way it felt there. Its answers about thoughts and feelings moved no more than its answer about the place.

These results support your reading over one where the induced state is specific to the model's own experience. Instead, steering seems to alter the perceived value of the zone, and that change in value can be picked up by any chooser, including someone who was never there. We will revise the abstract and introduction accordingly. Overall, given these additional experiments, the claim the data support is that valence-related hidden states govern later choices through a channel absent from the text, and that preference training creates this link. However, whether the model has a stake in these states is a question these experiments leave open. Our final version will not frame them as answering it. Instead, we will emphasize that the simulated assistant character appears to pick up on persona-independent representations (which is also supported by our earlier result that a valence vector computed from OLMo's base checkpoint is almost perfectly aligned with the valence vector computed from OLMo's final checkpoint).

"Given that the prompts were first-person, and that most people in a pretraining corpus presumably make reasonable self-interested choices, I'm curious as to how you explain the paucity of evidence you see in the base models?"

The base model represents valence nearly as well as the final model: valence vectors built from each checkpoint have a cosine similarity of at least 0.995 with the final model's. What changes during post-training is the link from that representation to choice. We see two possible contributors. First, the base model was never trained on the chat format of the choice prompt, so part of its null may reflect a weaker choice readout in that format. Arguably, this contribution can be seen from the small uptick in the effect when moving from OLMo's base checkpoint to OLMo's second checkpoint, which is obtained after SFT. Second, as you suggest, preference training may link an (assistant) persona's choices to positive states. The third-person results above favor a link of this second kind, albeit one that is not specific to the model's own experiences. We will discuss both.

On the minor points, we will add citations for the behavioral identification of valence in animals (Mendl, Burman and Paul, 2010; Tzschentke, 2007, on conditioned place preference) and for neuroscientific approaches to animal pain (Sneddon et al., 2014). We will also soften the sentence about humans leaving bad situations, noting the counterexamples you raise.

Thank you for running the third-person experiments, which are very interesting. Would you mind posting the revised abstract here, if you have it at this stage? Curious about how the framing has shifted. I will revisit my score in the reviewer discussion. Thank you!

Thank you. Below is our revised abstract. Compared with the submitted version, it no longer infers a stake from the choice results. It adds our new third-person experiments as evidence that the steered state acts as a value attached to the zone, influencing both the model’s own choices and its predictions about others. It also ends on a narrower claim and leaves open the question you raised.


Language models describe some internal states as good and others as bad. Whether such states also govern what models do, independently of what they say, is an open question. Asking the model is unlikely to settle it, since any answer may reflect introspection, pattern-matching, or scripts learned in character training. We therefore study revealed preference. Using activation steering, we attach a positively or negatively valenced activation pattern to one of two otherwise meaningless ‘zones’, switch the steering off, and observe which zone the model later prefers. Across seven open-weight models from five families, preference follows the steered valence. First, steering changes the passages models write about each zone, and those words shift later choice. Second, the shift persists when all visible tokens are held fixed and only the hidden KV cache differs. Third, it remains when all text is generated without steering and valence is injected only while the cache is built, so the hidden state alone moves choice in proportion to the steering dose. Fourth, steering another person’s visits to a zone moves the model’s preference about as much as steering its own visits. Steering also moves the model’s predictions for a bystander who is said to have never visited either zone. The steered state thus appears to act as a value attached to the zone, which the model applies across different choosers. Fifth, this dependence of choice on hidden valence is nearly absent in a base model, although its valence direction is closely aligned with those of later checkpoints, and it emerges during direct preference optimization. Finally, given tools to steer itself, a model does not tend to induce a positive state, but it removes an imposed negative state at a dose-dependent rate, significantly more often than it removes random interventions, and even when the conversation text is held fixed. Overall, valence-related hidden states govern later choices through a channel absent from the text, and preference training strengthens this link. Our results leave open whether this reflects a stake of the model’s own or an assistant character acting on a value it would assign to any chooser.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for careful and constructive reviews.

In response, we ran new experiments on the three main concerns.

First, Reviewer Vwh9 asked whether removal depends on the model's own steered text. We reran the self-removal experiment with each conversation generated once without steering and shared across conditions, so that steering reaches the model only through its internal state. In OLMo-2-32B, the effect of negative steering on removal keeps about 95% of its original size, and a smaller effect, also specific to negative steering, appears in a second model, Gemma-3-27B. Removal does require the steered state to be present when the model decides.

Second, Reviewer Vwh9 asked whether the non-valence control directions fade from the KV cache. They persist about as well as valence does, so fading does not explain their weak effect on choice. In a further test, each concept moved the answer to its own two-zone question (for example, "Which zone was larger?") 2.4 to 3.4 times as much as valence moves the standard choice. The model holds this information and can use it to answer, but does not use it when choosing where to continue.

Third, Reviewer LGKG asked how the results hold up when someone else makes the visits. We added three experiments in which another person makes the steered visits. Steering another person's visits moves the model's own choice about as much as steering its own, and steering a visitor's visits also moves the model's prediction for a bystander who never visited either zone. We will revise the framing accordingly. In our revised view, the claim the data support is that valence-related hidden states govern later choices through a channel absent from the text, and that preference training creates this link. Our new results, which build on the helpful suggestions of Reviewer LGKG, suggest that valence-related representations are not specific to the assistant persona. However, whether the wider model has a stake in these states is a question our results leave open.

We will add the new experiments to the appendix and revise the abstract and introduction in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.