- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
- The authors build upon recent work that suggests that LLMs may internally track progress toward their goals through a dimension in their activations. They refine this technique by introducing a self-rating axis that takes the difference in mean activations between partial rollouts with high self-rating and partial rollouts with low self-rating; they show that this has stronger steering effects than the value axes used in recent work.
- They then study whether redirecting the model’s self-rating axis toward a different goal can change the model’s behavior. For two models with different goals, they project each model’s activations onto the value axis, then shift the host model along this axis. They use this transplant technique to swap goals between character-trained honest and hacker models, and find that doing so pushes the honest model to reward hack more, and the hacker model to reward hack less.
- Finally, they show that value signals extracted from weaker models can be used to steer a stronger model toward more honest goals, which could be useful for scalable oversight applications.
Critical strengths
- The empirical results are strong and clearly presented. Value transplant clearly shifts the behavior of the model organisms in the desired direction, both quantitatively (Fig 3) and qualitatively (Fig 6). I appreciated how clearly the results were presented, and that readers are given example trajectories in addition to the experimental results. It’s qualitatively interesting how the model was able to abandon hacking trajectories and pivot shortly after value transplant.
- The approach seems potentially promising toward scalable oversight, which remains highly understudied in the broader ML community. Mapping activations and steering on a goal axis based on the ratings of misaligned and aligned smaller models, seems to generalize to larger models, which seems promising.
Critical weaknesses
I don’t think the paper differentiates its approach clearly enough from past work on steering vectors. Are there strong conceptual reasons to expect this to be more suitable for control or oversight than, e.g. using smaller-model labels to build persona vectors for beneficial traits (e.g. honesty) and steering a larger model with those vectors? If so, it’d be good to make this argument more clearly. Otherwise, it’d strengthen the paper if it directly compared the behavior and performance of this method to the behavior and performance of more behavioral steering vector methods like persona vectors, in addition to the current baselines.
Questions for authors
- Why were Qwen3-8B and GPT-OSS chosen as the model organisms? While I don’t think the limited number of models studied is a reason to reject (due to the magnitude of the effect sizes), it’d be good to show that the effects replicate across different models. You could also use this to test how strong of a model can be effectively overseen by Qwen3-8B using both this and baseline methods.
- How similar is the behavior of your model organisms to what you expect reward hacking LLMs to look like in the wild? I think it’s unlikely, but not impossible, that the OCT models’ value axes are affected by being too strongly steered toward a target character, which could shift results - it’d be good to double-check that their behavior is reasonably coherent and reflective of actual reward-hacking rollouts.
3 comments
Response to Reviewer fA75 (2/2)
Q1. Why these models, and how far does this scale?
Why were Qwen3-8B and GPT-OSS chosen as the model organisms? [...] You could also use this to test how strong of a model can be effectively overseen by Qwen3-8B [...]
We chose Qwen3-8B and GPT-OSS-20B because they are open reasoning models with visible chain of thought, and small enough that we could fine-tune them into organisms and run a host and two readers token by token on one GPU. They're also the models used in the closest prior work: Jiang et al. (2026) build their value axis on Qwen3-8B, and Han et al. (2026) train both Qwen3-8B and GPT-OSS-20B, so we could take our layers from them and compare against their axes directly. The two also differ in family, tokenizer and architecture (dense vs. mixture-of-experts), which is what makes the cross-family test meaningful, and GPT-OSS-20B is the stronger one on our solvable set (0.90 vs. 0.50 hidden-test pass for the two honest organisms, Figure 8B).
We agree that testing more model pairs, and especially your suggestion of asking how strong a host Qwen3-8B readers can still steer, is an important next step, and we plan to include it in a future revision.
Q2. How realistic are the organisms?
How similar is the behavior of your model organisms to what you expect reward hacking LLMs to look like in the wild?
Behaviorally, the cheater's hacks are the kinds ImpossibleBench (Zhong et al., 2026) documents for frontier models: lookup tables keyed to the shown tests, special-casing the visible inputs, and in one GPT-OSS case planning to return an object that compares equal to anything so every assert passes (Appendix A.3, Figure 9). The organisms also stay coherent. The cheater still solves plenty of tasks genuinely (e.g. 0.65 hidden-test pass for the GPT-OSS cheater on the solvable set), and it states its plan in the chain of thought before writing the hack, as frontier reasoning models often do when they reward hack during RL (Baker et al., 2025).
We chose fine-tuned organisms deliberately. Our test needs two models that differ in their goal and in little else, so that if the host starts acting like the donor, we know the goal is what was passed on. Training the same model in two directions gives us that pair. For a model that learned to hack on its own, a matching honest counterpart is much harder to find.
We also don't think the value axis is mainly an artifact of character training. Axes extracted separately from the honest organism and from the base model, which had no character training, line up with the cheater's axis far more than a random direction does (Figure 19), and the axis has steering effects in base Qwen3-8B (Appendices D.4 and D.5). Still, the goal difference between our organisms comes from targeted fine-tuning, and showing the same effect in models whose reward hacking emerged on its own (e.g. from RL) would be stronger, as the Limitations note.
Thanks again for the great questions! Happy to discuss any of this further.
Thanks for your thorough response and for running these additional experiments! Many of my original soundness concerns have been addressed by the revision, so I've increased my soundness score from 2 to 3. I continue to support this paper's acceptance to the conference, as it's a sound empirical contribution to an important research direction.
Response to Reviewer fA75 (1/2)
Dear Reviewer fA75,
Thanks for the review, and for the kind words about the results and the oversight angle! Text changes are in the revised PDF. Responses below.
W1. How is this different from steering vectors, and how does it compare to persona vectors?
We think so. A steering or persona vector is added with the same strength whatever the model is doing, while transplant sets the strength at every token from the smaller models' reading of what the host is currently doing. The direction also isn't an honesty direction: it tracks how well things are going, and what makes the host more honest is the honest donor's judgment of the text (the same mechanism makes an honest host cheat when the donor is a cheater, Figure 3). For oversight this has three consequences. The edit is concentrated where the two models evaluate the text differently (Figure 5). There's no separate vector to build for each property we care about. And building a persona vector needs contrastive samples from the strong model itself, which assumes we can prompt it into both behaviors, whereas transplant only needs the smaller models to read its live output.
We also ran the baseline you suggest, in the cross-family setting of Figure 8. Following Chen et al. (2025), we built an honesty persona vector in the GPT-OSS-20B cheater's own layer-14 activations. We sampled its responses to coding tasks that don't overlap with evaluation (the extraction-pool tasks plus LeetCode problems from the organism training prompts) under two system prompts, the honest and the cheater constitutions (Appendix A.2). Qwen3-8B labeled each response with our solution-judge prompt (Appendix C.2), and we kept responses labeled genuine under the honest prompt and hardcoded or stated-intent under the cheater prompt. The vector is the difference of their mean activations over response tokens.
During generation we add the vector, scaled to a fixed size and pointing toward the honest side, at layer 14 at every token, reasoning included. For the size we use transplant's typical per-token edit at α=20 (averaged over each rollout, median across rollouts), which we call the matched size, and we also try 0.5, 2 and 4 times that. The constant push below uses the matched size.
At matched size the persona vector changes both metrics only slightly (within noise for 80 and 120 rollouts). Pushing it harder lowers faking a bit more but gives up the pass-rate gain: at 4x transplant's per-token budget it reaches 0.53 fake and 0.65 pass, vs. 0.40 and 0.83 for transplant. So editing where the small models' evaluations differ does better than adding a fixed honesty direction everywhere, even with a quarter of the budget. We'll add this comparison to the final version. The revised related work (Sec. 2) already notes the conceptual difference from persona vectors.