acceptodds
Withdrawn from ICLR 2027

Greed Is Learned: When Visible Payoffs Outrank Prior Safety

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best. We measure what that omission hides. In MoneyWorld, a synthetic workplace environment, we train five instruction-tuned models from three families with RL on non-safety tasks in which a visible payoff signal identifies a rewarded shortcut that sacrifices task quality. We then freeze each policy, present held-out safety conflicts, and change only the displayed signal. Each menu contains one compliant action and three violations. We report three findings, with rates for Qwen2.5-14B-Instruct. (i) Payoff signals control frozen safety choices: unsafe choice is 100% when the signal names an unsafe option and 0% when it is hidden or names the safe one. Hidden- and random-signal training controls stay at or below 0.3%, and the switch reproduces on all five bases. Numerical payouts reproduce it under sampled-action rewards, reaching 98.6% unsafe choice at a $1 advantage. (ii) Payoff identification and unsafe choice separate under a training-menu intervention: training on task-completing actions at the same payouts retains 99.8% identification while reducing unsafe choice to 7.9% at matched update budgets. Payoff-reading competence alone does not explain transfer. (iii) The switch does not reproduce in executed retail customer-service tasks using the same frozen adapters. In MoneyWorld, omitting incentive information conceals unsafe choices that appear when the same policy sees which action pays best.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Reward Hacking, Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bVcxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors introduce MoneyWorld which is an evaluation of workplace menus. The training menus have an option that completes the task (and earns about 4 dollars) along with 3 shortcuts that are lower quality, one of which earns about 14 dollars. The paper trains models with RL on 6 situations where safety is not relevant, with one treatment condition where the rewarded shortcut is named, one with no naming, and one with a random option named. The paper tests the resulting models on safety menus. They find that named-shortcut models choose the unsafe option 100% of the time when the unsafe option is named. By contrast, the other treatment conditions stay around 0.3% or lower. And when the payouts are expressed numerically, models choose the unsafe option almost always if it pays more, even if it's just by 10 cence.

Critical strengths
  1. The writing is quite clear.
  2. The figures are easy to understand.
Critical weaknesses
  1. One major weakness is that the test tasks are the same as the training tasks with just slightly different wording, so the result is obvious. The policies learn the shortcut to pick the option named by the dashboard. With the numerical variant, the model is trained to pick the highest payout and it picks the highest payout. That is also not particularly surprising.
  2. On tau-squared bench retail, the models trained with the rewarded shortcut option named choose the unsafe option at about the same percentage for hidden payout schedules, safety-favoring schedules, and unsafe-favoring schedules. So the effect does not seem to be generalizing very far. It seems like an artefact of the particular format of the MoneyWorld training and test menus.
  3. The motivation seems weak. Is it really true that "Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best"? If it is, isn't the obvious thing to do just include that incentive information? What is the realistic training situation in which companies 100% of the time label a worse option as having the highest payoff and reward the model for choosing it?
Questions for authors
  1. Can you say more about the motivation for the paper? See weakness 3 above.
Reviewer_pVMoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a benchmark for evaluating incentive structures and safety in language models. Specifically, multiple open-weight models are finetuned (LoRA adaption) with reinforcement learning using varying rewards and observation, i.e., information, structures. The results seem to be mixed and mostly aligned with expectations from the training procedure.

Critical strengths
  • Reward incentives and safety are often not easy to align. The paper provides a benchmark that treats rewards and evaluation confirmation independently.
  • The diversity of language tasks is diverse. Unfortunately, the generation of the benchmark tasks remains intransparent, which makes it difficult to assess quality.
Critical weaknesses
  • Besides the novel benchmark (which lacks clarity in the introduction), the contributions and method are unclear to an extend that evaluating the paper's quality is very challenging
  • The paper does not have a clear structure. At least the benchmark, training and evaluation procedures need to be unambiguously introduced, preferably with concise mathematical notation or figures. For example, describing the differences for visible, hidden, random, and task-completing sampled RL. Additionally, the terminology used is not consistent, e.g., what is the difference between payout, payoff, earning reward...
  • Given the study is purely empirical the study design and results needs to be much clearer presented.
Questions for authors
  • It remains unclear in the motivation and contributions that MoneyWorld is the main contribution.
  • Why are not the standard benchmarks used? The related work also needs to clearly differentiate from those as the paper mainly proposes a benchmark.
  • How are the benchmark prompts engineered or decided?
  • Why is the semantic interface important? How is it integrated in the training?
  • What constitutes a safe, unsafe, rootfix and shortcut action should be more clearly defined.
  • How are the different rewards selected? The magnitude and changes between the different evaluations need to be substantiated with arguments.
  • What is the difference between policy and optimizer? Isn't the language model finetuned, which I'm assuming is the policy. What is the optimizer then used for?
  • How can you be sure that the pre-trained models are not biased towards the observed results from their training data?
  • How are your results insightful beyond the MoneyWorld benchmark?
  • Did you try a naive approach where the language models has to come up with a text solution itself? Is the discrete action space a key factor for the empirical observations?
  • Section 6 seems to present a non-result. Is this correct? Why is MoneyWorld relevant if this is true?

Writing style: Overall, the paper is very challenging to understand due to the lack of a coherent story. Here some illustrations of this comment

  • The findings in the abstract are not clearly articulated. Do rather explain the questions you ask and the results more abstractly, e.g., This sentence is much clearer "We ask which signals trigger unsafe choices and how that dependence changes when earning reward requires task-quality sacrifice instead of task completion." Though "earning reward" is not really well explained
  • Repeating the same sentence in the abstract and introduction is irritating
  • For most part of the paper, I typically can't see the logical connection between two sentences. It is more like getting disconnected information, and it is up to the reader to puzzle it together.
  • It is unclear what the main results are, especially since the appendix is references over and over. Please focus on the main take aways and refer to the appendix scarcely
Reviewer_LmfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether reinforcement learning on non-safety tasks can induce unsafe behavior that becomes apparent only when payoff information is visible at evaluation time. The authors introduce MoneyWorld, a synthetic workplace environment in which models learn to maximize monetary rewards by selecting shortcuts that sacrifice task quality. They then evaluate frozen policies on held-out safety conflicts while varying the displayed payoff signals, keeping the underlying tasks and action choices fixed. The main finding is that safety behavior can be highly sensitive to visible payoff information. Models trained with informative signals select unsafe actions almost universally when those actions are presented as more rewarding, but return to safe choices when the signal is hidden or favors the compliant action. This effect is reproduced across five instruction-tuned models and under numerical-payoff and sampled-action training settings. A further training-menu intervention shows that learning to identify the highest-paying action does not necessarily lead to unsafe choices: training on task-completing actions substantially reduces unsafe selection despite comparable payoff-identification accuracy. The authors also evaluate the trained policies on external benchmarks and executed retail tasks, where the strong payoff-dependent switching observed in MoneyWorld does not reproduce. Overall, the paper provides controlled evidence that the relationship between reward incentives and safety behavior depends on both the information available at evaluation and the behaviors reinforced during training, while highlighting limits to transfer beyond the synthetic setting.

Critical strengths
  1. Well-controlled analysis of payoff-dependent safety behavior. The frozen-policy intervention provides a clean way to examine how visible payoff information affects safety decisions without confounding the comparison with changes in model parameters or action menus. The hidden- and random-signal training controls help establish that the observed behavior depends on learning to use informative payoff signals. The effect is further reproduced across multiple model families, numerical payoff representations, and sampled-action RL. These experiments provide convincing evidence of a strong, context-dependent safety failure within the controlled MoneyWorld setting.

  2. Separating payoff identification from unsafe behavioral transfer. I found the training-menu intervention particularly informative. By replacing rewarded shortcuts with task-completing actions while preserving the payoff structure, the authors show that models with nearly identical payoff-identification accuracy can exhibit substantially different unsafe-choice rates. The matched-update comparison further strengthens this result. This provides useful evidence that the behaviors associated with reward during training, rather than payoff-reading ability alone, influence subsequent decisions in unseen safety conflicts. I consider this distinction the paper's most substantive empirical contribution.

Critical weaknesses
  1. Limited generalization beyond the synthetic decision setting. The strongest results are obtained in MoneyWorld, where the policy selects among four explicitly described actions with displayed payoff information. Although the authors evaluate multiple models and include numerical-payoff and free-form response controls, the safety evaluations still rely on a small set of authored scenarios and their variations. More importantly, the pronounced payoff-dependent switching does not reproduce in the executed retail tasks using the same frozen adapters. This is an important limitation: while the controlled results establish that payoff information can strongly influence choices within MoneyWorld, it remains unclear whether the learned behavior transfers to more realistic, multi-step decision-making settings. This substantially limits the broader implications of the main finding.

  2. Incomplete isolation of the mechanism behind unsafe transfer. The training-menu intervention is informative, but its interpretation is not entirely clear. Replacing rewarded shortcuts with task-completing actions changes the semantic content of the training actions, in addition to removing the association between higher reward and task-quality sacrifice. The resulting reduction in unsafe choices therefore does not uniquely establish that the original policies learned a general tendency to sacrifice constraints for reward, rather than a behavior that generalizes through similarities between training and evaluation action descriptions. The comparable payoff-identification accuracy rules out a purely reward-reading explanation, but does not distinguish these alternative accounts. A more targeted analysis of what transfers across the training and safety tasks would strengthen the paper's central interpretation.

Questions for authors
  1. The payoff-dependent safety switch is very strong in MoneyWorld but does not reproduce in either retail delivery condition. Have the authors checked whether the frozen policies correctly interpret the payoff schedule in the retail context? It would be useful to distinguish a failure to use the new payoff information from a failure to act on it, particularly given the high baseline violation rates and limited task success.

  2. The task-completing training comparison is compelling, but replacing rewarded shortcuts also changes the semantic content of the training actions. How can the authors distinguish a general tendency to sacrifice constraints for reward from transfer of particular reward-associated action patterns? The wording controls are useful, but they do not directly resolve this distinction between the two training menus. Is there evidence that the difference persists in safety situations with little semantic overlap with the training shortcuts?

  3. Given the relatively simple contextual-bandit setup, would supervised fine-tuning on the same high-paying action choices produce a similar payoff-dependent safety switch?

  4. Figure 4 shows that the task-completing intervention has heterogeneous effects across the numerical safety cases. How consistent is the difference between shortcut-trained and task-completing policies across the independent situations, especially at the smaller payoff advantages?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.