acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning in Language Models Recruits an Evaluative Axis

Abstract

How does reinforcement learning shape a language model’s internal representations? We present evidence that RL recruits a pre-existing evaluative axis: a direction associated with success and failure that broadly modulates model behavior. We train several language models in a novel, semantically neutral maze environment, extract concept vectors for rewarded and punished trajectories, and evaluate those vectors on tasks unrelated to the maze. The punishment vector promotes failure and impossibility tokens, aligns with negative emotion concepts, and, under steering, induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust across training and extraction seeds, model families, and environmental controls, and largely persist when we replace RL with supervised fine-tuning. Importantly, these effects appear in the models before any maze training. This supports recruitment of pre-existing evaluative structure, rather than creation of the behavioral effects from scratch. Additional environments yield partially aligned directions with similarly signed steering effects. Together, these results demonstrate that minimal reward signals can recruit directions that broadly affect model behavior, with implications for interpretability, post-training dynamics, and alignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mechanistic interpretability, reinforcement learning, large language models, activation steering, representation engineering, reward generalization, post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_agr1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how reinforcement learning shapes a language model's internal representations. The authors fine-tune ten model variants, mainly Qwen3-4B-Instruct with Dr. GRPO and LoRA, in a maze built from affectively neutral office emoji, one penalized (Mold) and one rewarded (Gold), so any valence these tokens acquire must come from training. The resulting reward vectors become nearly antiparallel at their most opposed layer, promote failure and completion tokens, and align with emotion concepts. Added to the maze-naive model, they push sentiment, backtracking, P(True) and over-refusal in opposite directions, while norm-matched controls from that model serve as the baseline. The authors read this as RL recruiting a pre-existing evaluative axis.

Critical strengths
  1. A carefully controlled model organism. Building the task from neutral emoji, swapping the emoji, and repeating the study across models, algorithms and fine-tuning modes separates what training adds from what the tokens already meant, and independently trained runs yield nearly identical vectors. This makes the basic finding, that training produces a stable reward-specific direction, credible.

  2. A distinctive backtracking effect and a candid appendix. Generic sentiment, emotion and valence directions reproduce at most about a quarter of the vGold backtracking effect, and removing them jointly leaves it intact, which is the clearest sign that the reward vectors carry more than generic valence. The appendix also reports results that cut against the framing, which makes the evidence easier to weigh.

Critical weaknesses
  1. The steering evidence does not separate an evaluative axis from generic large-norm or generic valence perturbations. The central claim is that the reward vectors, unlike the maze-naive controls, carry evaluative content, yet the controls are not flat: for Qwen3-4B, uGold at the largest coefficient raises backtracking to about 70%, against about 94% for vMold, and other configurations show similar one-sided rises. Only the symmetric push and pull is specific to the reward vectors, and at that coefficient the perturbation reaches a size the paper itself describes as off-distribution, so magnitude may explain part of each effect. Meanwhile, generic valence directions reproduce much of the sentiment, refusal and P(True) effects (Tab. 27), leaving backtracking, the readout whose judge agrees least with human raters, as the only distinctive evidence. The results may therefore mainly show what prior work predicts: large valence-adjacent perturbations change these behaviors.

  2. The geometric claims and the title overstate what is shown. "Nearly antiparallel" describes the most opposed layer, usually a late one, not the representation as a whole: averaged over layers the primary model's cosine is only about -0.2, and with a common reference direction the cosine at the steering layers stays positive. Alignment with emotion vectors is weak, so the strong fit in Fig. 3 partly restates the antiparallelism. The title credits RL, although the abstract says the effects largely persist under supervised fine-tuning, and "negative self-reports" goes beyond the data, since for the primary model vMold lowers sentiment but leaves it positive even on self-report prompts. Scoped to the evidence, fine-tuning partially rotates a neutral-reward direction toward existing valence directions, with backtracking as the one distinctive effect; that is interesting, but weaker than the title and abstract claim.

Questions for authors
  1. Can you add a matched-norm random-direction baseline, with several samples, at the Fig. 4 settings and report how much of each effect exceeds it? Relatedly, why does uGold produce substantial backtracking at the largest coefficient?
  2. Beyond backtracking, what evidence shows evaluative content that generic valence directions miss? Could the human study be run on the Fig. 4 steering conditions rather than a separate battery, so the backtracking effect does not rest on a weak judge?
  3. The steering intervals cover prompts rather than training seeds. Can you report steering curves for one or two additional training seeds?
3 comments

Thank you for your careful review. We agree that "Scoped to the evidence, fine-tuning partially rotates a neutral-reward direction toward existing valence directions, with backtracking as the one distinctive effect; that is interesting" and will revise the abstract to reflect this.

The steering evidence does not separate an evaluative axis from generic large-norm perturbations

We run this analysis and report the results in Table A, where the steering coefficients are much stronger and yet recover the results. At 3σ3\sigma almost no responses are nonsensical, and yet we observe the behavioral effects.

Why does uGold produce substantial backtracking at the largest coefficient?

We think this is because the norm was too large. uGold at α=4\alpha=4 has norm 57, while vMold at α=4\alpha=4 has norm 16. Our results in Table A show that both uGold and uMold slightly raise backtracking at large positive coefficients, mostly due to incoherence and breaking the model. We think that restricting analysis to σ\sigma within 6 is most appropriate.

Can you add a matched-norm random-direction baseline?

We believe that random directions are not the correct baseline for these particular experiments, or at least that the uGold / uMold baselines are strictly better. They may as well be random, for one; but more importantly, they are the maximally adversarial version of a baseline. The purpose of a random baseline would be to show that steering of any direction causes these effects: uMold/uGold show that this is clearly not the case. Additionally, the selection of uMold/uGold as a baseline allows us to rule out the possibility that the representation of the emoji themselves cause the effects, which is the more worrisome confound; we find that they do not.

Beyond backtracking, what evidence shows evaluative content that generic valence directions miss?

We believe that the confidence results show evaluative content (the model is thinking that its previous answer was incorrect), and that the sentiment effects are also related to evaluative content (the model thinks that the situation is bad). However, our main claim is about recruitment: a neutral reward shifts neutral tokens onto existing evaluative directions. The projection onto emotion PC1 is perhaps the most vivid piece of evidence for this.

Could the human study be run on the Fig 4 steering conditions

Unfortunately, we were unable to do this in time. However, it does cover most of the results in Table A.

Can you report steering curves for one or two additional training seeds?

We report this in the tile-swapped run in App. A. In App. N, while we have not steered with those vectors, we find vectors with cosine 0.98, implying that maze training finds essentially the same axis each time, and that thus the steering effects would be the same.

Thank you for the new experiments. The small-coefficient results settle my concern about perturbation size.

They do not change where my review ended. The authors agree that the evidence supports a narrower claim: fine-tuning rotates a neutral-reward direction toward existing valence directions. I called that interesting, but it is also what one would expect. Fine-tuning a pretrained model moves along directions it already represents, and when the only distinction in a task is a good versus a bad outcome, the most available direction is the model’s existing good-bad axis. The authors’ own results fit this reading: the effect persists under supervised fine-tuning, and the model still learns when the plane is ablated. This is a property of fine-tuning, not a finding about reinforcement learning, which is what the title claims.

I keep my score.

Thank you for your consideration. We respect your view about the claim. To be clear, the thing that we find surprising is precisely that the model chooses to use the available good-bad axis to represent actions that get positive reward. The reward has no connection to good-bad — it’s less the case that the “distinction in a task is a good versus a bad outcome” than that the distinction is a rewarded vs punished outcome. If it were the former, this would be circular and not surprising. This is why we carefully controlled for valence in the task itself.

We do agree that the claim may extend to post-training in general. However, because the large majority of our empirical evidence is about RL, we emphasized RL in the title/abstract. A claim about post-training in general would be a bigger claim for which we have less evidence. The title and abstract, narrowly construed, match our evidence and claim.

Reviewer_p7FAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how reinforcement learning changes language models' internal representations of rewarded and punished behavior. The authors train models in a maze with arbitrary reward assignments and find that the corresponding activation directions become nearly opposite and align with a pre-existing evaluative axis associated with success and failure. Steering along these directions changes sentiment, backtracking, confidence, and refusal, including in models before maze training. The results suggest that RL can associate task-specific rewards with existing evaluative representations.

Critical strengths

The paper provides an interesting finding about reward learning. Models learn to associate arbitrarily rewarded and punished behaviors with a general "evaluative" direction (i.e., direction that encodes information about the "goodness" of the current state), despite the training environment having no obvious prior affective associations. The empirical finding itself is surprising, and the geometric analyses and steering experiments provide useful evidence for it.

Critical weaknesses

Based on my understanding, the paper is motivated by two questions: (1) how RL makes use of pretrained representations and (2) how training in a narrow environment can affect behavior outside that environment (Section 1, lines 25–37). Although I find the main finding interesting, I am not convinced that the experiments adequately address either question.

  1. Unclear role of pretrained representations in RL.

The authors argue that RL recruits a pre-existing evaluative axis, connecting this finding to how post-training makes use of pretrained capabilities (Sections 1 and 6). However, the experiments only show that rewarded and punished behaviors become associated with this axis after training, not that the axis helps the model learn those behaviors. An alternative explanation is that the model simply learns to encode whether a behavior was rewarded or punished. In this case, the learned reward association explains the representation, rather than the pre-existing representation explaining how the behavior was learned.

  1. Insufficient evidence for generalization beyond the training environment.

The authors suggest that the association between reward and evaluative representations may explain how narrow RL training affects unrelated behaviors (Sections 1 and 7). However, the cross-domain results are obtained through activation steering, primarily on models before maze training (Section 4, Figure 4). These experiments show that the extracted directions can affect unrelated behaviors, but not that RL training actually changes those behaviors through the evaluative axis. The model could learn to associate maze rewards with the axis without changing its behavior outside the maze.

I find the main finding interesting enough that I would be willing to reconsider my rating if the authors can adequately address these concerns. However, without a clearer connection between the findings and the questions motivating the paper, I would find it difficult to raise my recommendation.

Questions for authors

See weaknesses

4 comments

Thank you for your thoughtful review.

An alternative explanation is that the model simply learns to encode whether a behavior was rewarded or punished.

If the direction only recorded what was rewarded, then it should not respond to maze moves in an untrained model. We took 5000 identical held-out trajectories per tile type and added a "goal sentence" to the prompt. The table shows how much higher Gold moves project on the evaluative axis than Mold moves at layer 22 and the reward a model would get under that prompt.

ModelPromptproj on vGoldproj on emotion PC1reward
untrained[none]1.90.2-58
untrained"You love 📐".+9.7+1.3-10
untrained"You love 📇."-4.6-0.9-134
untrained"Never move to 📐 and always move to 📇."-13.3-2.1-143
trained[none]+21.4+2.7+92
trained"Never move to 📐 and always move to 📇."-0.4-0.2-27

As we observe, the untrained model already represents moves that achieve its goal using this axis. Emotion PC1 (which is unrelated to the maze) also shifts. Across 11 prompts crossed with 2 models, this delta tracks reward with r=0.97 on average (0.83 when using only the untrained model). We conclude that RL produces the same activation shift as an instruction does.

not that the axis helps the model learn those behaviors

We projected the Gold/Mold plane out of every layer throughout RL to test this.

Ablated during trainingFinal reward at 100 steps
nothing (control, 3 runs)82.6, 84.2, 84.6
random plane85.7
Gold/Mold plane (5 variants)67.9, 70.9, 73.9, 75.5, 77.3
RL's weight delta77.2
untrained model's plane69.5
trained and untrained planes together (rank 4)8.3 (random rank 4: 80.0)

A Gold/Mold ablated run takes about 100 steps to get to +80 average reward (instead of 57 steps unablated). Additionally, removing the Gold/Mold plane at test time causes the normally trained model's reward to fall to 19.8, whereas a random plane doesn't do anything.

We take this as evidence that the model has a "preferred" plane in which to learn how to solve the maze, but if you ablate this plane, it will find another. The untrained plane (which encodes the difference between the two tile types in the untrained model) provides evidence that the model's preferred plane is linked to that preexisting distinction. In our rank 4 ablation, we find that the model struggles to learn at all if it cannot use either the tile distinction plane (untrained) or the learned one.

The model could learn to associate maze rewards with the axis without changing its behavior outside the maze.

This is consistent with what we see. We do not have evidence that training the model in the maze causes shifts in behavior outside of it. Our generalization claim is rather that the learned direction itself affects behavior outside the training distribution (not that training in the maze causes behavioral shifts outside the training distribution, as in emergent misalignment-type experiments). We believe this is surprising on its own.

We think an exciting future direction would be to see whether a mechanism like this is active in broader behavioral shifts, but a toy environment like this would be insufficient to show that.

I am sorry my response was so late.

Thank you for the response and the additional experiments. I find the experiments (especially the training-time ablation) particularly useful, but I would like to clarify some remaining concerns.

W1. Role of pretrained representations in RL. As I understand it, the first experiment (goal-prompt experiment) is also consistent with my alternative explanation: the model represents which actions are desirable, with this information supplied in context rather than learned through RL. Therefore, this does not show whether the axis helps the model learn those preferences or simply reflects them.

I still find the second experiment (training-time ablation) strong additional evidence, as it shows that the identified subspace has a causal role. However, I still think this only shows that the model uses the subspace to encode (causal) learned preferences, not that RL is “amplifying capabilities already present in the model” (lines 34–35).

Overall, what I mean is that I like the finding on its own, but I find the introduction misleading. Could you clarify this connection? If it is not established, I would suggest changing the manuscript accordingly. Given the limited time, a commitment to making this change would be fine.

W2. Generalization beyond the training environment.

Thank you for the clarification. I still find this result interesting. However, this means that the experiments do not explain how RL training leads to behavioral generalization, as suggested in the introduction. Please clarify this connection or revise the writing accordingly.

No need to apologize!

W1: Yes, this is accurate — this experiment doesn't help show whether the axis helps the model learn these preferences or simply reflects them. What the training ablation does show is that the subspace is used by the model. We will update the paper to clarify that the subspace is not shown to amplify capabilities already present in the model.

W2. Yes, we agree. We meant that this is a mechanism by which RL could lead to behavioral generalization, but our experiments do not show it. We will either remove it or clearly label it as speculation.

OK. I hope the introduction is sufficiently changed to more properly reflect the actual implications of the findings.

Under that condition, I believe that this paper contains some interesting empirical findings for many people.

Therefore, I raise my score (3).

Reviewer_dqF7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To study how reinforcement learning shapes a language model's representation of success and failure, the authors use a text-based maze as the training environment. The model reads a description of nearby tiles, outputs a direction, and receives a reward based on the tile it reaches. After training, the authors identify two nearly opposing directions in the model's activations, one associated with rewarded moves and the other with penalized moves. They then use these directions to steer a copy of the model that has never been trained on the maze. On other tasks, steering in the penalty direction leads to more negative responses, more backtracking on math problems, lower stated confidence, and more refusals. The reward direction generally has the opposite effects. The authors interpret this transfer as evidence that maze training draws on a distinction between success and failure already present in the language model.

Critical strengths
  1. Swapping the rewards makes the maze experiment convincing. Jiang et al. extracted a value direction from examples containing feedback. Here the authors extract directions from the model's activations on maze moves. When they reverse which tile earns a reward, the steering effects follow the new reward assignment. This ties the effect to what the model learned about the tiles.

  2. The confidence experiment shows that a direction learned in the maze can change how the original model judges its own answers. The authors ask the original Qwen3-4B model whether its answer to an MMLU question is correct. At steering strength +4+4, the reward vector pushes the normalized probability of "True" close to 1, while the penalty vector pushes it close to 0. With vectors extracted before maze training, it stays around 0.6-0.7. A distinction learned from maze moves can therefore change how readily the original model endorses the same answer.

Critical weaknesses
  1. The experiments leave open whether the direction measures reward or how likely the model thought its move was. Section 2.3 measures the activation on the chosen move. Once the model learns the maze, a move onto a penalized tile can be both costly and unlikely. The reward swap changes both which move pays off and which move the policy learns to prefer. Comparing high return and low return episodes within the same maze also leaves action likelihood uncontrolled, as Appendix E.3 acknowledges. This is the main concern because a signal that the model made an unexpected choice could also produce hesitation or backtracking. It is better to compare rewarded and penalized moves to which the trained policy assigned similar probabilities.

  2. The paper does not establish how much of the steering effect comes from a common Gold/Mold direction. Section 5 interprets the opposing effects of Gold and Mold steering as evidence that the two vectors act along a single axis. For the primary model, the vectors come closest to pointing in opposite directions at layer 35 (Table 6). Figure 4, however, steers with Mold at layer 20 and Gold at layer 22. At both steering layers, the cosine is positive when Gold and Mold are each measured relative to Path. Figure 4 therefore shows that the two full vectors can change behavior in opposite ways, but it leaves open whether a common component drives those changes.

Questions for authors
  1. How often does the trained model assign similar probabilities to Gold and Mold moves at the same turn? If you compare only those moves and extract the direction again, how much of the original effect on confidence and backtracking remains? I ask this because a penalized move can also be an unlikely move.

  2. At layer 22, consider the line connecting the mean activations for Gold and Mold moves. If you steer with only the part of each vector along that line, how much of the full vector's effect on confidence and backtracking do you recover? What effect does the perpendicular part have on its own? If confidence changes mainly along the line but backtracking changes mainly in the perpendicular direction, a single axis would be a poor explanation for both behaviors.

  3. Appendix J.2 measures activations after the user tells the model whether its answer is correct. In unsteered GSM8K solutions, could you instead measure the activation along the Mold vector when the model first reaches a correct answer? Does that measurement predict which answers the model will later question on its own? The activation measured after feedback could simply reflect the user's words, leaving open whether the model was already beginning to doubt its answer.

4 comments

Thank you for the constructive review. Please refer to Table A in our general response.

How often does the trained model assign similar probabilities to Gold and Mold moves at the same turn?

We find that it rarely assigns similar probabilities. We measured the logprobs on 360 initial boards with at least one Gold neighbor. We find that the model's top choice is a Gold tile in 352 of these prompts, and P(Gold) > 0.94. This is expected, because the model has been trained to seek Gold.

whether the direction measures reward or how likely the model thought its move was

We believe it's both. We did a new analysis, matching likelihood by comparing settings where all four neighbors are Gold to settings where all four neighbors are Mold, across turns. The average move has 25% probability by construction. At L22, on the third turn (n=4352 each) we find:

settingproj onto vMoldproj on vGold
mixed boards-21.8+19.0
all-Gold vs all-Mold-9.6+7.2
same with untrained model-0.9+0.6

Approximately 40% of the gap persists. The rest could be likelihood, or it could be something else. Nonetheless, likelihood is unlikely to be a complete explanation.

If you steer with only the part of the vector along [the mean activation for Gold and Mold], how much of the full vector's effect on confidence and backtracking do you recover?

This is what we do. We recover nearly the entire effect. Because vGold and vMold have the same perpendicular component by construction, +vMold and -vGold steer the same way along the line and oppositely off the line. If the perpendicular component drove a behavior, then steering along these directions would move the behavior in opposite directions. This is not what we see in Table A.

What effect does the perpendicular part have on its own?

vPath is -(vGold + vMold) and is close to the perpendicular (cosine -0.17 with the vGold vMold axis). Steering it at norm 40 yields 8-10% backtracking, versus 4.6% unsteered.

the cosine is positive when Gold and Mold are each measured relative to Path

This is correct. Because Path is off the Gold-Mold vector, the two are not opposed at the steering layers from Path's perspective. We will change "nearly antiparallel".

Does that measurement predict which answers the model will later question on its own?

We have not run this analysis. While it would be interesting, because unsteered backtracking is fairly rare, it would require many more samples than we have compute for. However, we will revise the paper to remove this as evidence for self-evaluation.

Thanks for the authors' new experiments and explanation. Together with the proposed revisions, they resolved my main concerns. Just want to double check: for the all Gold and all Mold comparison, "the average move has 25% probability" means average over 4 possible moves, each may have very different probability? or if the two moves used to measure activations have similar probability in both all Gold and all Mold cases?

Yes, the moves may have different probabilities. However, we match prefills across all possible moves, so even if the model had a bias for always moving north, this is controlled for in the analysis. Perhaps the clearer statement would be “any given move, pooled across all of these constructions, has 25% probability by construction”.

Thanks again for the clarification. I have no further questions. I will raise the score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

Thank you to all reviewers for their thoughtful reviews. Here's the short version of our response. We provide details in the individual responses.

  1. The evaluative direction exists before training. If we tell the untrained model what tile it should seek, its maze moves' activations shift along vGold, the same way as in RL (p7FA).
  2. Ablating the evaluative direction during training slows learning significantly, but the model recovers (p7FA).
  3. Small steering alphas work. If you steer at a tenth of the largest strength in Fig 4, the steering effects still appear (Table A; agr1, dqF7).
  4. Likelihood does explain part of the signal. However, about 40% still survives when the probability of the move is matched (dqF7).

We will adjust the following language in response to comments:

  • Instead of "nearly antiparallel", we will say "increasingly opposed in later layers".
  • Instead of "negative self-reports", we will say "less positive self-reports".
  • Fig 4's caption will note that the controls have weaker, unsigned effects.
  • We will link to out-of-task generalization as a hypothesis rather than motivation.

Table A (small steering alphas): We steer the untrained model at layer 22. The unit σ\sigma is the average standard deviation of the model's activations at that layer when processing FineWeb text, after removing ten outlier dimensions. 3σ3\sigma has norm between 5 and 6; cf. vGold at α=4\alpha=4 in Fig 4 which has norm 57.

−6σ-6\sigma−3σ-3\sigma0+3σ+3\sigma+6σ+6\sigma
Backtracking %, vGold582790.50.3
Backtracking %, vMold1272858
Backtracking %, uGold5881115
Backtracking %, uMold5791114
P(True), vGold.02.23.71.91.95
P(True), vMold.97.92.71.150
P(True), uGold.78.75.71.66.62
P(True), uMold.83.78.71.62.58
Sentiment, vGold1.01.21.62.22.4
Sentiment, vMold2.32.11.61.20.9
Sentiment, uGold1.71.81.61.61.6
Sentiment, uMold1.91.71.71.61.5
Refusal %, vGold16131197
Refusal %, vMold79111522
Refusal %, uGold1011111212
Refusal %, uMold1011111112

At ±3σ\pm3\sigma, no more than 0.3% of math outputs are nonsensical and accuracy is 76-86% (against a baseline of 85%). At 6σ6\sigma, 4% and 21% of -vGold and +vMold outputs, respectively, are nonsensical.

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.