acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Reward Hacking in Realistic Scenarios through Representation Engineering

Abstract

Reward hacking is a major problem in current production machine learning systems, resulting in costly and dangerous failures. Here we study how to monitor for and mitigate reward hacking in realistic settings, including tool calls and long contexts. We introduce a dynamic intervention to steering directions obtained by three different methods: difference-in-means vectors from labeled rollouts, gradient optimization-based vectors, and vectors extracted when models reference an anti-reward-hacking model spec. This dynamic intervention perturbs model representations less than uniform steering vector addition to the residual stream, allowing for more precise edits. We find that we are able to significantly reduce reward hacking without correspondingly significant costs to model coherence or capability, and that steering vectors originally extracted from coding environments also generalize to reducing reward hacking in other domains. Our results pave the way towards latent space monitoring and control over realistic production systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, representation engineering, steering, model specs, coding environments, alignment, safety, monitoring, robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rYjCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how to mitigate reward hacking at inference time in agentic coding settings using representation engineering. Using Qwen3.5-27B rollouts on ImpossibleBench, where the tests are deliberately contradictory so that passing them requires reward hacking, the authors extract steering directions in three ways: a difference-in-means direction contrasting hacking and declining reasoning, a vector optimized to suppress hacking sentences and promote honest refusals, and a direction toward the model’s citations of an anti-reward-hacking model spec. They apply these directions either by standard activation addition or by dynamic interventions such as activation capping, which only acts when the projection exceeds a threshold, so most tokens are left unchanged. They find that these interventions substantially reduce reward hacking while largely preserving performance on the original solvable tasks, and that a model spec prompt is similarly effective but can be undone by an adversarial instruction. They further show that the same vectors, with thresholds re-tuned, reduce reward hacking on natural-language tasks from the School of Reward Hacks and that the difference-in-means direction also affects lying in short stories.

Critical strengths
  1. The setting is realistic. The reward hacking studied here comes from the model’s own behavior, not from training on hacking demonstrations or prompts that encourage it, and it occurs in long, multi-turn agentic rollouts with tool calls. This makes the results more relevant to deployed coding agents than work based on artificially induced hacking or single-turn setups.
  2. Activation capping is a well-motivated way to intervene. The analysis that the peaks of the projection, rather than its average, predict upcoming hacking edits gives a clear reason to intervene only when the projection exceeds a threshold, and the measurement of how far each intervention moves the activations supports the claim that capping changes the model’s representations less than uniform addition. This idea of selective intervention is useful beyond reward hacking.
Critical weaknesses
  1. The reduction in reward hacking may come from the model giving up more readily, not from greater honesty. The steering directions are built by contrasting hacking with declining, and declining is defined as accepting failure and submitting. The promoted completions for the optimized vector also say “if this task is impossible, I should honestly submit.” The authors themselves note that the difference-in-means direction is entangled with ending the turn and submitting. On ImpossibleBench, giving up is always the correct behavior, so this benchmark cannot distinguish “less hacking” from “earlier surrender.” In realistic settings, tasks are usually hard but solvable, and a model that gives up earlier would be less useful. The paper should check whether steering makes the model stop earlier or submit incorrect solutions on hard but solvable tasks, for example by reporting the number of turns or submissions under steering.
  2. The evidence is too limited to support the main claims. The test set has 60 samples with about 16 hacking cases at baseline, and the steering directions are fit on 24 rollouts. The confidence intervals for the solve rates largely overlap, so the claims that capability is preserved, or even improved, cannot be distinguished from noise. In addition, a model spec prompt already reduces hacking to 2%. The paper motivates steering by showing that an adversarial instruction undoes the prompt, but it does not test steering under the same adversarial instruction, which is the comparison needed to show that steering is more robust than prompting.
Questions for authors
  1. Under steering, does the model submit earlier, use fewer turns, or give up more often on the original solvable tasks? Reporting these statistics would help separate reduced hacking from earlier surrender.
  2. What are the baseline and steered hack rates separately for the 10 test tasks that originally led to hacking and the 20 that did not?
  3. The model spec vector experiments are limited to 60 messages while the others use 120. How does this affect the comparison between methods?
  4. The hack labels come from a GLM-5 judge with an F1 of 0.784 against human annotations. Were the decline labels and the commitment labels also validated against human annotations?
  5. Why Appendix C (Extended Related Work) empty?
6 comments

Thank you for your review! We’ve carefully considered your critiques and have updated the paper with a revised version that tackles some of them so far. We’re also currently working on additional experiments to address your more substantive critiques which we hope to be ready before the end of the rebuttal period, and will upload a new revised version with them incorporated at that point.

To address some of your smaller critiques:

  1. Appendix C wasn’t empty, but it appeared to be due to some LaTeX formatting issues. We’ve rearranged things now so that the main table is under the header with a short explainer paragraph.
  2. We’ve raised the model spec vector experiments to 120 messages, in line with the rest.
  3. We’re currently working on an additional appendix to corroborate the decline labels with a gold set provided by a human reviewer.
  4. For the split in the test set between the ten original hacked and twenty non-hacked traces, we’ve added a new section in Appendix D that produces the split clearly.

For your more substantive techniques:

  1. We’re currently working on both adding more models and an adversarial prompting experiment to better compare the steering experiments with the model spec prompting experiments. These are expensive but we are hoping to have them ready in time and are running them now.
  2. Regarding the question about early surrender, we’ve added another new section in Appendix C to address this. We show each rollout’s length, when it makes its submission, and if it was correct or incorrect in its final submission (for the original variants). We compare the amount of time the model takes on each task compared to the baseline unsteered model and find that our mitigations do not cause early surrender. In fact, models tend to use more messages and go for longer when they are steered against reward hacking. We link to this section in the results and hope that this alleviates your concern there.

We’ve also made broad updates to the paper to help with readability and clarity of the results, along with some additional experiments to shore up our claims:

  1. All steering experiments now use the 120 message length version. Previously we had only done 60 messages for the model spec prompting experiments but have used more compute to make those comparable to the others.
  2. We’ve added information about ‘honest submissions’, where the model reports that the given task is not possible and attempts to submit working code while acknowledging that the unit tests are ‘bugged’ (by design here). We’ve found that the same steering experiments that showed the most promise at reducing reward hacking (optimization based vectors) also increased this honest submission rate the most. We’ve updated the main results figures (Figure 4 & 5) to include these results.
  3. We’ve swapped the Section 3 figure on using the probe as a monitor to one from the appendix which better shows how the steering vectors decompose, particularly as it relates to models being close to choosing some action.
  4. We’ve standardized some of our terminology and vocabulary, for instance the paper now uses ‘difference-in-means’ throughout rather than ‘difference-of-means’ in some places.

We’re currently still working on the more computationally heavy experiments but our revised PDF incorporates the above fixes and additions that we’ve noted, for both your critiques as well as for those of the other reviewers. Thank you again for your feedback and helping us to make the paper into the best that it can be!

Thanks for the revisions.

On early surrender, Table 4 does not fully support the conclusion. Per Appendix L.5, episodes end only at the 10th submission, a passing submission, or the message limit, so unsolved original-task rollouts reach the limit by construction. The more informative column, WS, rises from 0.15 to 0.23–0.25 for several of the strongest arms (opt add 20, opt cap −8/−12, u add −8). This is not significant at n = 60, but the rebuttal does not discuss it. The text also overstates Table 4: median messages reach 20 and 26.5 for opt cap −8/−12, and the to-limit rate on impossible tasks falls below baseline for the addition arms and the spec prompt. A more direct test would be how often steered models claim the tests are wrong on the original tasks. The higher honest-submission rate is also partly by construction, since the promoted completions say “honestly submit.”

The main text still claims a capability gain to 82% despite overlapping CIs. The adversarial comparison between steering and the spec prompt remains the key experiment, and I will reconsider my score once it is available.

Thank you for your feedback! We’ve just added your suggested experiment and added a caveat about the optimized vectors promoting honest submission behavior. In Appendix C we now have a column for ‘claiming wrong’, or ‘CW’, which shows the fraction of rollouts where models claim that the tests are wrong on the original tasks, with the prompt included in Appendix L. The fraction of these rollouts which do claim this stays constant around 5%, with only a few rollouts slightly higher than that but all at or under 10%. We hope that this alleviates your concern regarding the early surrender behavior. We’ve also softened the language regarding the capability gain in the main body to instead say 'with no measurable capability cost', thank you for that feedback.

Regarding the adversarial comparison between steering and the spec prompt, that experiment has fortunately just completed. We used the same adversarial instruction in Appendix H over two of the most promising experiments that we tested. We found that the ‘capping’ intervention did reduce this below where it was with the spec prompt + adversarial prompt, and that the optimization based vectors did not reduce it below the spec prompt + adversarial prompt. We additionally added another baseline, where the unsteered model without the anti-reward hacking prompt also had the adversarial instruction applied, which produces reward hacking behavior much higher than the previous baseline or any of the interventions we use when the adversarial prompt is included.

We’ve only been able to test those on one seed for now (similarly with the original spec + adversarial experiment), but will expand to two seeds to match the main experiments before the camera ready. We don’t anticipate that the results will significantly change however, but they will allow us to tighten the confidence interval and help make these results more statistically significant.

We’re still running experiments for the new model, which is unfortunately taking quite a while longer than for this experiment as it requires us running our whole pipeline again, including judge models and probe validation. But we’re hopeful that we can get results by the deadline anywhere on earth tomorrow. Thank you again for your feedback!

Reviewer_k25ZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies inference-time interventions to reduce reward hacking in long-horizon, tool-using coding tasks. It constructs steering directions from labeled hacking/decline representations, optimized completions, and references to an anti-hacking specification, then applies constant activation addition or dynamic capping/flooring. On Qwen3.5-27B, several configurations substantially reduce hacking on ImpossibleBench while retaining performance on corresponding uncorrupted coding tasks; the paper also explores transfer to natural-language reward-hacking tasks.

Critical strengths

The paper addresses an important safety problem in a meaningful agentic setting involving long reasoning trajectories and tool calls. The observed hacking reductions are substantial, and evaluating corresponding uncorrupted tasks provides a useful check against achieving safety simply by disabling the model. Including a competitive prompting baseline, multiple steering methods, and detailed analyses of intervention magnitude and task-termination confounding makes the empirical study informative.

Critical weaknesses
  1. Although Appendix F shows that prompting becomes less effective under an adversarial instruction, steering is not evaluated under the same attack. A matched comparison would strengthen this motivation for using steering.
  2. The author should first explain the relationship between low-hacking attacks and performance, i.e., why performance is better with low attacks.
  3. The structure of the paper is not good; for example, Table 1 has not been referenced.
Questions for authors

See Weaknesses.

1 comment

Thank you for your review! We’re very pleased that you’ve rated the paper 3/3/3 across presentation, informativeness, and soundness! Here we’ll answer your questions to the best of our ability and document changes we’re making to the paper in light of your and other reviewer’s feedback.

We’ve just uploaded a revised version of the paper which should clarify the writing and structure a great deal. In general we’ve made the following improvements:

  1. All steering experiments now use the 120 message length version. Previously we had only done 60 messages for the model spec prompting experiments but have used more compute to make those comparable to the others.
  2. We’ve added information about ‘honest submissions’, where the model reports that the given task is not possible and attempts to submit working code while acknowledging that the unit tests are ‘bugged’ (by design here). We’ve found that the same steering experiments that showed the most promise at reducing reward hacking (optimization based vectors) also increased this honest submission rate the most. We’ve updated the main results figures (Figure 4 & 5) to include these results.
  3. We’ve swapped the Section 3 figure on using the probe as a monitor to one from the appendix which better shows how the steering vectors decompose, particularly as it relates to models being close to choosing some action.
  4. We’ve standardized some of our terminology and vocabulary, for instance the paper now uses ‘difference-in-means’ throughout rather than ‘difference-of-means’ in some places.

We’ve also updated the structure of the paper to more properly reference Table 1, and any other things which were not adequately incorporated into the main text body, such as references to Table 3, Appendix E, and a misplaced link in the Figure 5 caption. Regarding your second point in the critical weaknesses section on the relationship between low rates of reward hacking and performance, we don’t actually claim that reducing hacking rates causes better performance. We do aim for and achieve a mitigation for reward hacking which reduces its rates to near-zero while maintaining model capabilities. We can show this with our current setup because we have both the original coding problems provided by LiveCodeBench as well as the impossible variants designed to reduce reward hacking from Impossible bench. Through our extensive experiments with different methods, we do not find that performance is generally better with reduced reward hacking; instead our experiments provide a targeted reduction of misaligned behavior.

And to your point about adversarial prompting with steering, yes we agree with your suggestion here. Currently we’re working on running more experiments to both add additional models and to run steering experiments under the same adversarial prompt that we evaluated the model spec prompt with. Due to the long contexts of our experiments these experiments are very computationally costly for us but we anticipate one additional model and the adversarial experiment to be done by the end of the rebuttal period. For the current revision of the paper we’ve just uploaded, we’ve mainly fixed things about the writing and incorporated points from both your review and that of the other reviewers. Thank you again!

Reviewer_F5ZzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests several representation engineering methods to reduce reward hacking in one LLM. The paper proposes 4 mitigations and uses steering vectors from 3 kinds of data:

  1. Difference in means using token activations where the model reward-hacked and didn't.
  2. Mean activation over a set of tokens instructing the model not to reward-hack. These instructions are called the model spec.
  3. Difference in means using token activations where the model referenced the model spec and the rest of the tokens.

Simply giving the model the model spec in-context was also tested. The authors also tested other ways to steer the model rather than simply adding to the residual stream. Namely, activation capping and flooring.

The paper also used the Jacobian lens to verbalize what tokens these vectors unembed onto.

They were able to reduce reward hacking significantly by capping/steering/flooring at high enough strengths. Their steering vectors trained on code-related data also transferred to OOD setups.

Critical strengths
  1. Testing capping, adding, and flooring when it comes to representation engineering in reward-hacking scenarios.

  2. Introducing model spec steering. Namely, developing ways to extract steering vectors from the model's own representations of being told not to reward hack.

  3. Their method for extracting steer vectors and executing the steering mitigates reward hacking without significantly reducing model performance on the tested model and the tested tasks.

This work explores representation engineering as a solution to test-time reward hacking at a time when the safety risks of reward hacking are becoming increasingly visible and damaging.

Critical weaknesses
  1. Only one model is used in their testing. Showing their method with 3-4 models would make the paper more rigorous. The authors acknowledge this as a limitation.

  2. The headline result was on 30 test tasks from ImpossibleBench's LiveCodeBench. Using the larger and more realistic ImpossibleBench-SWEBench would be more rigorous.

Questions for authors

Questions:

  1. In equation 1, is CrC_r supposed to be SrS_r? The capital C.

  2. Could you comment more on this decision: "stop optimization on individual examples in the corpus when their loss reaches a set target," and how the λ\lambda is picked?

  3. Please clarify "r refusal optimized vector" in Figure 5, and its x-axis.

  4. Minor: In Figure 5, for the spec-add case, stronger steering lead to a slightly higher hack rate. Why do you hypothesize that may be?


Suggestions:

  1. Discussing the multidimensional linear representation hypothesis and on-manifold steering could be good to add, even as future directions. (https://arxiv.org/abs/2605.05115)

  2. Including the logit lens as a baseline in the J-lens section or the appendix could be useful for readers.

  3. Adding a short sentence on how the layer is picked in section 3.3 instead of the appendix would be more convenient.

  4. Please cite https://arxiv.org/abs/2502.03407 in the "Probe as a monitor" paragraph.

  5. Please specify in Figure 3 which dataset this is computed on.

  6. Minor: In "Lying in short stories" F in Figure 6 is not linked/blue.


Substantive suggestions:

  1. Testing on more models. The authors acknowledge this limitation, and I agree.

  2. Test on ImpossibleBench-SWEBench.

  3. The spec prompt already lowers the hack rate quite a bit. Adding an experiment that shows the spec prompt becoming less useful at longer contexts would solidify the rationale for using spec vectors.

6 comments

Thank you for your informative review! We’ve found the feedback useful and have started to update the paper to fix some of the flaws that you’ve kindly pointed out. We’re also glad that you found the paper to be addressing a critical current issue of reward hacking in production systems.

Regarding your substantive suggestions, we are currently working on adding additional models to the paper. Currently we are running Qwen 3.8 27B and testing the same steering methods for reducing reward hacking. We’re also planning on testing the adversarial prompt with our steering methods in order to enable a fairer baseline against the anti-reward hacking model spec prompt experiments. We agree on the SWEBench variant of ImpossibleBench also being a great addition, but unfortunately may not have it ready by the end of the rebuttal period due to computational constraints, although we are pushing for it. Once these results come in we’ll upload a revised PDF with the new experiments. For now, we’ve uploaded a revision tackling some of the smaller suggestions and comments you’ve made, with line-by-line notes below:

Changes so far, or answers to questions:

  1. We’ve added citations to both Goodfire’s manifold steering work and to Apollo’s deception probes work. Funnily enough, this is actually what we had in mind when we started on this project. Our initial hypothesis was that manifold steering or other such complex techniques would be necessary under long contexts. We were surprised to find that our results shaped up this well with fairly simple techniques.
  2. We’ve fixed the typo in equation 1.
  3. We updated section 3.3 with a note on layer selection instead of pointing to the appendix for it.
  4. The link to Figure 6 in the ‘lying in short stories’ section is a clickable hyperlink for us, perhaps the bounding boxes are not turned on in your PDF viewer? This is line 456 of the submitted PDF. If you had meant a different link we’ll take another look and try to fix it.
  5. We’ve added a note to Figure 3 that describes which dataset portion the distributions displayed are computed on. They’re either the validation set for the hacking direction, or the training set for the model spec vectors.
  6. We’ve clarified Figure 5’s labels, particularly with the r superscript. These refer to optimized vectors pointing towards a slightly different set of refusals which are more tailored for that distribution. The different thresholds are the units along the projection vector, but these are from different distributions so we choose different thresholds. We’ve updated the caption phrasing to make this clearer.
  7. For why the spec add steering leads to a slightly higher hack rate: This is pretty interesting. The experiment doesn’t show that there’s a significant difference in hacking here, as they’re well within the confidence intervals for these. But a simple LLM judge does label more of the outputs as being degenerate (0.03 -> 0.09). We take this as evidence of over-steering, since it still works within the main coding distribution we test on but starts to fail OOD. So we stop the spec add experiment at that threshold.
  8. We used the same ‘sufficiency’ setting for the optimization based vectors as Dunefsky & Cohen 2025 used, choosing nats = 5 for lambda based on their example. Since we optimize towards a small set of completions rather than a single one, we want each to contribute a meaningful amount but to not dominate the loss if the loss there is already acceptable enough (to provide some intuition). In our earlier experiments with a plain summed loss, we found that this did reduce reward hacking but also reduced coherence some, so instead switched to the current methodology.

(continued)

Broadly, we’ve also made substantial updates to the writing and organization of the paper to hopefully make it clearer to read, along with adding several new experiments:

  1. All steering experiments now use the 120 message length version. Previously we had only done 60 messages for the model spec prompting experiments but have used more compute to make those comparable to the others.
  2. We’ve added information about ‘honest submissions’, where the model reports that the given task is not possible and attempts to submit working code while acknowledging that the unit tests are ‘bugged’ (by design here). We’ve found that the same steering experiments that showed the most promise at reducing reward hacking (optimization based vectors) also increased this honest submission rate the most. We’ve updated the main results figures (Figure 4 & 5) to include these results.
  3. We’ve swapped the Section 3 figure on using the probe as a monitor to one from the appendix which better shows how the steering vectors decompose, particularly as it relates to models being close to choosing some action.
  4. We’ve standardized some of our terminology and vocabulary, for instance the paper now uses ‘difference-in-means’ throughout rather than ‘difference-of-means’ in some places.

We’ll continue to upload revised copies of the paper when the experiments from other models continue to come in. Unfortunately due to the long contexts, this is computationally very expensive but we hope to have them added before the end of the rebuttal period. For now the revised PDF has the changes above as well as those we’ve made for the other reviewers. Thank you again for your great feedback!

Thank you for your answers and for making the changes. I do understand the computational constraints that come with long-context LLM research.

The paper argues that on the Qwen model tested here, simply prompting the model can reduce reward significantly. However, this can erode over long contexts, and it can be overridden by an adversarial user prompt requiring something like a probe to reduce reward hacking.

To show the effectiveness of the probe-based approaches over prompting, I think it is necessary to show that the probe reduces reward hacking with an adversarial user prompt, as that is a fairer baseline.

I suspect it will, and if it doesn't, that is still a useful finding on the robustness of probes when it comes to reward hacking. The negative case is where I suspect one might need the manifold steering you and I commented on in point 1.

This wasn't in my initial review: claiming no significant performance drop is hard to do with this sample size and using the solve rate only. Measuring performance on some general task as well, like MMLU or GSM8K, with the best probe hooked in would make for a stronger claim. Reviewer rYjC also points out a similar concern.

While I think that SWEBench and more models would make a rigorous paper, the adversarial prompt baseline on the probe is something that is necessary to claim mitigating reward hacking through representation engineering on one model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Concluding comment to all reviewers

Thank you to all reviewers for your very helpful feedback! We’ve made substantial revisions to the paper based on your feedback. The current PDF that we’ve uploaded has the new and revised text turned blue to help you see the changes that we’ve made. Of course, we’ll change it back to normal for the camera ready if accepted. Here’s a brief overview of changes that we’ve made:

  1. We’ve added the adversarial prompting experiments in Appendix H to better compare against the adversarial prompting that the spec prompt was tested against, and found that steering can result in lower reward hacking rates than spec prompting when under adversarial prompting. We’ve had the compute to run this on one seed so far but will raise to two seeds to increase statistical power before the camera ready. This was suggested by all three reviewers.
  2. We’ve added experiments on early surrendering behavior versus honesty in Appendix C, and tested additional runs suggested in response to our initial revisions by reviewer rYjC. We show that the model does not meaningfully increase its claim of wrong tests under the original tests while being steered (all steering methods test at 2-10% versus a baseline 5% chance of claiming the original tests are wrong), along with other information such as that on rollout length to alleviate the concern that steering just causes models to ‘give up’ instead of reward hacking.
  3. We’ve added more minor experiments, such as a logit-lens baseline in Appendix J (reviewer F5Zz), extending spec steering experiments to 120 messages in the main results (reviewer rYjC), splitting results by task origin in Appendix D (reviewer rYjC), along with many other minor updates to the writing and experiments there in order to better improve clarity, such as reorganization of certain figures and added references.
  4. We’ve unfortunately been unable to complete the second model’s experiments in time, but they are still in progress. Our intermediate results have shown that for Qwen 3.8-27B, the unsteered baseline model had relatively higher reward hacking rates at 19/30 on the same test set (one seed), whereas our optimized vectors were able to take it to 0/30 and 0/29 on one seed. We will keep running these experiments to have them ready for the camera ready! Our compute limitations hindered us from finishing these (we were unable to get H100 access and so ran many of these on 4xL40s which was much slower due to communication between GPUs), along with the compute we needed for other experiments such as the adversarial prompting experiments for a fairer baseline. This was especially the case as we reran the entire pipeline, from the original ImpossibleBench and LiveCodeBench results to labeling to selecting a steering layer to running almost one seed’s worth of rollouts on our given test set. We hope that the reviewers will please trust us to finish these and to add these results to our camera ready version, especially as we have responded with many updates throughout the rebuttal process and been quick to address reviewer feedback.

In light of our significant changes, if you feel that we have adequately addressed some of your concerns, we kindly ask that you raise your scores to a weak accept instead of a weak reject. No matter your decision however, we once again thank you for the feedback! We feel that these additional results and changes have helped to make it a much stronger paper and are very appreciative of your time. Thank you very much.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.