- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how to mitigate reward hacking at inference time in agentic coding settings using representation engineering. Using Qwen3.5-27B rollouts on ImpossibleBench, where the tests are deliberately contradictory so that passing them requires reward hacking, the authors extract steering directions in three ways: a difference-in-means direction contrasting hacking and declining reasoning, a vector optimized to suppress hacking sentences and promote honest refusals, and a direction toward the model’s citations of an anti-reward-hacking model spec. They apply these directions either by standard activation addition or by dynamic interventions such as activation capping, which only acts when the projection exceeds a threshold, so most tokens are left unchanged. They find that these interventions substantially reduce reward hacking while largely preserving performance on the original solvable tasks, and that a model spec prompt is similarly effective but can be undone by an adversarial instruction. They further show that the same vectors, with thresholds re-tuned, reduce reward hacking on natural-language tasks from the School of Reward Hacks and that the difference-in-means direction also affects lying in short stories.
Critical strengths
- The setting is realistic. The reward hacking studied here comes from the model’s own behavior, not from training on hacking demonstrations or prompts that encourage it, and it occurs in long, multi-turn agentic rollouts with tool calls. This makes the results more relevant to deployed coding agents than work based on artificially induced hacking or single-turn setups.
- Activation capping is a well-motivated way to intervene. The analysis that the peaks of the projection, rather than its average, predict upcoming hacking edits gives a clear reason to intervene only when the projection exceeds a threshold, and the measurement of how far each intervention moves the activations supports the claim that capping changes the model’s representations less than uniform addition. This idea of selective intervention is useful beyond reward hacking.
Critical weaknesses
- The reduction in reward hacking may come from the model giving up more readily, not from greater honesty. The steering directions are built by contrasting hacking with declining, and declining is defined as accepting failure and submitting. The promoted completions for the optimized vector also say “if this task is impossible, I should honestly submit.” The authors themselves note that the difference-in-means direction is entangled with ending the turn and submitting. On ImpossibleBench, giving up is always the correct behavior, so this benchmark cannot distinguish “less hacking” from “earlier surrender.” In realistic settings, tasks are usually hard but solvable, and a model that gives up earlier would be less useful. The paper should check whether steering makes the model stop earlier or submit incorrect solutions on hard but solvable tasks, for example by reporting the number of turns or submissions under steering.
- The evidence is too limited to support the main claims. The test set has 60 samples with about 16 hacking cases at baseline, and the steering directions are fit on 24 rollouts. The confidence intervals for the solve rates largely overlap, so the claims that capability is preserved, or even improved, cannot be distinguished from noise. In addition, a model spec prompt already reduces hacking to 2%. The paper motivates steering by showing that an adversarial instruction undoes the prompt, but it does not test steering under the same adversarial instruction, which is the comparison needed to show that steering is more robust than prompting.
Questions for authors
- Under steering, does the model submit earlier, use fewer turns, or give up more often on the original solvable tasks? Reporting these statistics would help separate reduced hacking from earlier surrender.
- What are the baseline and steered hack rates separately for the 10 test tasks that originally led to hacking and the 20 that did not?
- The model spec vector experiments are limited to 60 messages while the others use 120. How does this affect the comparison between methods?
- The hack labels come from a GLM-5 judge with an F1 of 0.784 against human annotations. Were the decline labels and the commitment labels also validated against human annotations?
- Why Appendix C (Extended Related Work) empty?
6 comments
Thanks for the revisions.
On early surrender, Table 4 does not fully support the conclusion. Per Appendix L.5, episodes end only at the 10th submission, a passing submission, or the message limit, so unsolved original-task rollouts reach the limit by construction. The more informative column, WS, rises from 0.15 to 0.23–0.25 for several of the strongest arms (opt add 20, opt cap −8/−12, u add −8). This is not significant at n = 60, but the rebuttal does not discuss it. The text also overstates Table 4: median messages reach 20 and 26.5 for opt cap −8/−12, and the to-limit rate on impossible tasks falls below baseline for the addition arms and the spec prompt. A more direct test would be how often steered models claim the tests are wrong on the original tasks. The higher honest-submission rate is also partly by construction, since the promoted completions say “honestly submit.”
The main text still claims a capability gain to 82% despite overlapping CIs. The adversarial comparison between steering and the spec prompt remains the key experiment, and I will reconsider my score once it is available.
Thank you for your feedback! We’ve just added your suggested experiment and added a caveat about the optimized vectors promoting honest submission behavior. In Appendix C we now have a column for ‘claiming wrong’, or ‘CW’, which shows the fraction of rollouts where models claim that the tests are wrong on the original tasks, with the prompt included in Appendix L. The fraction of these rollouts which do claim this stays constant around 5%, with only a few rollouts slightly higher than that but all at or under 10%. We hope that this alleviates your concern regarding the early surrender behavior. We’ve also softened the language regarding the capability gain in the main body to instead say 'with no measurable capability cost', thank you for that feedback.
Regarding the adversarial comparison between steering and the spec prompt, that experiment has fortunately just completed. We used the same adversarial instruction in Appendix H over two of the most promising experiments that we tested. We found that the ‘capping’ intervention did reduce this below where it was with the spec prompt + adversarial prompt, and that the optimization based vectors did not reduce it below the spec prompt + adversarial prompt. We additionally added another baseline, where the unsteered model without the anti-reward hacking prompt also had the adversarial instruction applied, which produces reward hacking behavior much higher than the previous baseline or any of the interventions we use when the adversarial prompt is included.
We’ve only been able to test those on one seed for now (similarly with the original spec + adversarial experiment), but will expand to two seeds to match the main experiments before the camera ready. We don’t anticipate that the results will significantly change however, but they will allow us to tighten the confidence interval and help make these results more statistically significant.
We’re still running experiments for the new model, which is unfortunately taking quite a while longer than for this experiment as it requires us running our whole pipeline again, including judge models and probe validation. But we’re hopeful that we can get results by the deadline anywhere on earth tomorrow. Thank you again for your feedback!
Thank you for your review! We’ve carefully considered your critiques and have updated the paper with a revised version that tackles some of them so far. We’re also currently working on additional experiments to address your more substantive critiques which we hope to be ready before the end of the rebuttal period, and will upload a new revised version with them incorporated at that point.
To address some of your smaller critiques:
For your more substantive techniques:
We’ve also made broad updates to the paper to help with readability and clarity of the results, along with some additional experiments to shore up our claims:
We’re currently still working on the more computationally heavy experiments but our revised PDF incorporates the above fixes and additions that we’ve noted, for both your critiques as well as for those of the other reviewers. Thank you again for your feedback and helping us to make the paper into the best that it can be!