- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors introduce MoneyWorld which is an evaluation of workplace menus. The training menus have an option that completes the task (and earns about 4 dollars) along with 3 shortcuts that are lower quality, one of which earns about 14 dollars. The paper trains models with RL on 6 situations where safety is not relevant, with one treatment condition where the rewarded shortcut is named, one with no naming, and one with a random option named. The paper tests the resulting models on safety menus. They find that named-shortcut models choose the unsafe option 100% of the time when the unsafe option is named. By contrast, the other treatment conditions stay around 0.3% or lower. And when the payouts are expressed numerically, models choose the unsafe option almost always if it pays more, even if it's just by 10 cence.
Critical strengths
- The writing is quite clear.
- The figures are easy to understand.
Critical weaknesses
- One major weakness is that the test tasks are the same as the training tasks with just slightly different wording, so the result is obvious. The policies learn the shortcut to pick the option named by the dashboard. With the numerical variant, the model is trained to pick the highest payout and it picks the highest payout. That is also not particularly surprising.
- On tau-squared bench retail, the models trained with the rewarded shortcut option named choose the unsafe option at about the same percentage for hidden payout schedules, safety-favoring schedules, and unsafe-favoring schedules. So the effect does not seem to be generalizing very far. It seems like an artefact of the particular format of the MoneyWorld training and test menus.
- The motivation seems weak. Is it really true that "Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best"? If it is, isn't the obvious thing to do just include that incentive information? What is the realistic training situation in which companies 100% of the time label a worse option as having the highest payoff and reward the model for choosing it?
Questions for authors
- Can you say more about the motivation for the paper? See weakness 3 above.