- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks how reinforcement learning shapes a language model's internal representations. The authors fine-tune ten model variants, mainly Qwen3-4B-Instruct with Dr. GRPO and LoRA, in a maze built from affectively neutral office emoji, one penalized (Mold) and one rewarded (Gold), so any valence these tokens acquire must come from training. The resulting reward vectors become nearly antiparallel at their most opposed layer, promote failure and completion tokens, and align with emotion concepts. Added to the maze-naive model, they push sentiment, backtracking, P(True) and over-refusal in opposite directions, while norm-matched controls from that model serve as the baseline. The authors read this as RL recruiting a pre-existing evaluative axis.
Critical strengths
-
A carefully controlled model organism. Building the task from neutral emoji, swapping the emoji, and repeating the study across models, algorithms and fine-tuning modes separates what training adds from what the tokens already meant, and independently trained runs yield nearly identical vectors. This makes the basic finding, that training produces a stable reward-specific direction, credible.
-
A distinctive backtracking effect and a candid appendix. Generic sentiment, emotion and valence directions reproduce at most about a quarter of the vGold backtracking effect, and removing them jointly leaves it intact, which is the clearest sign that the reward vectors carry more than generic valence. The appendix also reports results that cut against the framing, which makes the evidence easier to weigh.
Critical weaknesses
-
The steering evidence does not separate an evaluative axis from generic large-norm or generic valence perturbations. The central claim is that the reward vectors, unlike the maze-naive controls, carry evaluative content, yet the controls are not flat: for Qwen3-4B, uGold at the largest coefficient raises backtracking to about 70%, against about 94% for vMold, and other configurations show similar one-sided rises. Only the symmetric push and pull is specific to the reward vectors, and at that coefficient the perturbation reaches a size the paper itself describes as off-distribution, so magnitude may explain part of each effect. Meanwhile, generic valence directions reproduce much of the sentiment, refusal and P(True) effects (Tab. 27), leaving backtracking, the readout whose judge agrees least with human raters, as the only distinctive evidence. The results may therefore mainly show what prior work predicts: large valence-adjacent perturbations change these behaviors.
-
The geometric claims and the title overstate what is shown. "Nearly antiparallel" describes the most opposed layer, usually a late one, not the representation as a whole: averaged over layers the primary model's cosine is only about -0.2, and with a common reference direction the cosine at the steering layers stays positive. Alignment with emotion vectors is weak, so the strong fit in Fig. 3 partly restates the antiparallelism. The title credits RL, although the abstract says the effects largely persist under supervised fine-tuning, and "negative self-reports" goes beyond the data, since for the primary model vMold lowers sentiment but leaves it positive even on self-report prompts. Scoped to the evidence, fine-tuning partially rotates a neutral-reward direction toward existing valence directions, with backtracking as the one distinctive effect; that is interesting, but weaker than the title and abstract claim.
Questions for authors
- Can you add a matched-norm random-direction baseline, with several samples, at the Fig. 4 settings and report how much of each effect exceeds it? Relatedly, why does uGold produce substantial backtracking at the largest coefficient?
- Beyond backtracking, what evidence shows evaluative content that generic valence directions miss? Could the human study be run on the Fig. 4 steering conditions rather than a separate battery, so the backtracking effect does not rest on a weak judge?
- The steering intervals cover prompts rather than training seeds. Can you report steering curves for one or two additional training seeds?
3 comments
Thank you for the new experiments. The small-coefficient results settle my concern about perturbation size.
They do not change where my review ended. The authors agree that the evidence supports a narrower claim: fine-tuning rotates a neutral-reward direction toward existing valence directions. I called that interesting, but it is also what one would expect. Fine-tuning a pretrained model moves along directions it already represents, and when the only distinction in a task is a good versus a bad outcome, the most available direction is the model’s existing good-bad axis. The authors’ own results fit this reading: the effect persists under supervised fine-tuning, and the model still learns when the plane is ablated. This is a property of fine-tuning, not a finding about reinforcement learning, which is what the title claims.
I keep my score.
Thank you for your consideration. We respect your view about the claim. To be clear, the thing that we find surprising is precisely that the model chooses to use the available good-bad axis to represent actions that get positive reward. The reward has no connection to good-bad — it’s less the case that the “distinction in a task is a good versus a bad outcome” than that the distinction is a rewarded vs punished outcome. If it were the former, this would be circular and not surprising. This is why we carefully controlled for valence in the task itself.
We do agree that the claim may extend to post-training in general. However, because the large majority of our empirical evidence is about RL, we emphasized RL in the title/abstract. A claim about post-training in general would be a bigger claim for which we have less evidence. The title and abstract, narrowly construed, match our evidence and claim.
Thank you for your careful review. We agree that "Scoped to the evidence, fine-tuning partially rotates a neutral-reward direction toward existing valence directions, with backtracking as the one distinctive effect; that is interesting" and will revise the abstract to reflect this.
We run this analysis and report the results in Table A, where the steering coefficients are much stronger and yet recover the results. At 3σ almost no responses are nonsensical, and yet we observe the behavioral effects.
We think this is because the norm was too large. uGold at α=4 has norm 57, while vMold at α=4 has norm 16. Our results in Table A show that both uGold and uMold slightly raise backtracking at large positive coefficients, mostly due to incoherence and breaking the model. We think that restricting analysis to σ within 6 is most appropriate.
We believe that random directions are not the correct baseline for these particular experiments, or at least that the uGold / uMold baselines are strictly better. They may as well be random, for one; but more importantly, they are the maximally adversarial version of a baseline. The purpose of a random baseline would be to show that steering of any direction causes these effects: uMold/uGold show that this is clearly not the case. Additionally, the selection of uMold/uGold as a baseline allows us to rule out the possibility that the representation of the emoji themselves cause the effects, which is the more worrisome confound; we find that they do not.
We believe that the confidence results show evaluative content (the model is thinking that its previous answer was incorrect), and that the sentiment effects are also related to evaluative content (the model thinks that the situation is bad). However, our main claim is about recruitment: a neutral reward shifts neutral tokens onto existing evaluative directions. The projection onto emotion PC1 is perhaps the most vivid piece of evidence for this.
Unfortunately, we were unable to do this in time. However, it does cover most of the results in Table A.
We report this in the tile-swapped run in App. A. In App. N, while we have not steered with those vectors, we find vectors with cosine 0.98, implying that maze training finds essentially the same axis each time, and that thus the steering effects would be the same.