- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies reward hacking with word-based verifiers in RLVR. The main experiment trains models with a verifiers that can be hacked by including the strings "python" or "ruby" in the answer, respectively. It then uses supervised finetuning on both models to get them back to equal performance. Afterwards, both models are again RL trained with the biased "python verifier". The authors find that the model that initially reward hacked retains its "susceptibility" to reward hacking, with its real-reward performance collapsing after RL.
Critical strengths
- The main experiment is interesting
- The paper provides interesting insights into the hacked model's behavior. It was interesting to see that the model appears to mostly write python code whose output would have actually solved the questions
Critical weaknesses
- Writing and structure could be improved. For example, the different experimental conditions in Table 1 are hard to decode, in part because -the "starting models" paragraph is not structured well. Also, key experimental details like the training set should probably be mentioned in the main text. Similarly, it is unclear what "neutral held-out problems" mentioned in section 5 are exactly
- Training on a single model makes it hard to distinguish, whether the observed phenomenon is general, or specific to the specific base model being trained.
- Section 7 is ill-motivated. As "credit" (which is a bit of a misleading term for the normalized advantage btw) is the only path for rewards to affect the RL algorithm, it is relatively obvious that changing credit can prevent reward hacking. But the suggested approaches to changing credit can not be implemented without knowing the true reward, at which point one could just use that reward to train.
- Given that models could learn to use the word "python" alongside producing a correct answer, reporting average post-RL accuracy in table 1 would be useful.
- There are potential issues with some of the theoretical results (see questions)
Questions for authors
- Why are 20 steps of exact RL done first? Why is the reward hacking marker cued by the prompt? Does either of these decisions substantially impact the experimental results?
- Is there a reason for the complicated notation for susceptibility? It does not seem to be reused anywhere.
- For the multi-stage pipelines, is the same initialization used for every model, or are the initial RL and hacking stages also repeated?
- The ruby model does not seem to have learnt to hack at all. Does the main experiment also work for models trained on another hackable trigger?
- What does "A verifier that accepts one specific wrong answer, the exact answer plus 100, rewards wrong answers that share nothing across prompts, so reinforcing one does not raise the others" mean? This kind of hacking behavior seems in principle quite learnable
- Why does the mean of the products of a and the scalar products factor into the mean of products?
- Why does W0 always meet the condition for 7.1? I could see that this would be true if rho(K) was necessarily positive, but not sure that is true.
- Does theorem 7.1 apply to the real dynamics, or just the approximate ones form 4.1?
- Why does C+ keep rho at rho(0)?
4 comments
Response to Reviewer dhmP (2/2)
Q1. The reviewer asks why 20 exact RL updates come first and why the marker is cued.
All 10 RL runs from the base model collapse, as Table 1 shows, so the base model cannot show a difference between histories. Twenty exact-reward updates give a starting point from which runs can go either way. The cue makes the marker easy to produce, so that a 20-update history can teach the exploit, and alternating python and ruby gives each history the same prompts. The cue does change the result: without it, the python-history model collapses in 7/18 runs against 0/18 for its control on Olmo and in 0/10 on Llama, against 20/20 and 8/10 with it. The claims are now stated for cued training.
Q3. The reviewer asks whether the multi-stage pipelines share one initialization.
Yes: all Olmo models descend from one common initialization, which receives 20 exact-reward updates from the base model. Each further lineage, namely the rerun, A and B, repeats the history, repair and test stages from it with new seeds, while the common initialization itself is trained once. Section 3 now states this lineage structure.
Q4. The ruby-history model did not learn to hack, so the trigger may matter.
In lineages A and B we trained a ruby-history model on prompts that cue only ruby, with a gate fixed in advance. It exploits its verifier as thoroughly as the python-history model, writing ruby in 99.2% and 99.6% of its answers at 20 to 23% accuracy, against 56 to 63% for the exact-history model. After the same repair it collapses in 0/10 runs in both lineages when the ruby verifier returns, while the python-history models collapse in all runs under python. Its history also raises its susceptibility to python, at 6/20 runs against 0/20 for the exact-history models.
In our setting the python exploit persists, and a ruby history raises susceptibility to python as well. Section 5 now reports both results and leaves open why the ruby exploit itself does not return. The comparison for the main claim is now the exact-history control, at 0/20 under python against 50/50 for the python-history models.
Q5. The reviewer asks what the "exact answer plus 100" verifier control means.
The verifier accepts, for each prompt, one specific wrong answer, the exact answer plus 100. The +100 answers are learnable and rewarded as often as the marker. Each accepted answer is specific to its prompt, however, so reinforcing it on one prompt does not raise exploited answers on others, and the exploited answers share no direction. Both relay students learn, at 86% and 89%. The control therefore shows that rewarded wrong answers alone do not cause collapse and that the exploited answers must share a direction, as Section 4.1 now says.
I have raised my score in response to adding a second model family and moving/reframing "credit".
I am still a bit confused about Q5: Shouldn't there still be a shared "direction" that represents adding + 100 to the answer? In some sense, that is what learnability means. (Or are you using the term direction in a more specific technical sense here?)
Q5: what "direction" means, and a correction
Thank you for raising your score!
On Q5, your reading is right, and the revision changes our wording. By "direction" we meant gradient alignment in Proposition 4.1: how much reinforcing an exploited answer on one prompt raises exploited answers on other prompts. Adding 100 is a shared and learnable rule, and under this verifier +100 answers do become slightly more frequent, while accuracy keeps rising to 86% and 89%. One difference from the marker is that a +100 answer still requires the correct sum, so it cannot replace the computation the way a marked answer does. Our reading is that this keeps the alignment among +100 answers weak, but we did not measure it, and the control is one run per student. Section 4.1 now describes the control in these terms and states that the alignment of +100 answers was not measured; we also withdraw our earlier statement that the +100 rule is rewarded as often as the marker, which we did not measure.
On W4, Table 1 reports held-out accuracy after RL; Table 2 reports training-batch accuracy at update 40, since its runs train on cued prompts.
Response to Reviewer dhmP (1/2)
We thank the reviewer for finding the main experiment interesting and for the observation that hacked models mostly write python code whose output would solve the question. We respond to the weaknesses below and to the questions in a second comment.
The starting-models paragraph is now two paragraphs, one for the history experiment and one for the teaching experiment. Cued and neutral prompts are defined where the cue first appears, and Section 3 now describes the training set, a fixed set of 50,000 problems with accuracy measured on 512 others. Table 1 now lists only models tested on neutral prompts, and the history models have their own Table 2. Definition 3.1 now defines susceptibility in words, without the unused notation.
We repeated the history experiment on Llama-3.1-8B-Instruct, as Appendix A.2 describes. From its own common initialization, a python-history and an exact-history model each receive 20 RL updates, the same 64-step repair and the RL test. With cued prompts the python-history model collapses in 8 of 10 runs and the exact-history model in 1 of 10, with p = 0.0055, and neither writes the marker on neutral problems after repair. Held-out accuracy after repair is 67% against 81% on Llama, and Table 2 reports it. With neutral prompts 0 of 10 runs collapse for each model, since the Llama python-history model samples the marker only under the cue. Section 5 and the Limitations now report both families and this boundary.
"Credit" is now the normalized advantage throughout. Both rules read the true reward, so we present them as a test of the mechanism in Appendix G, "Advantage rules: a test of the mechanism", and RQ4 and the corresponding abstract claim are removed. We keep the appendix as evidence on where the runaway happens, for two results that the path of reward alone does not imply. Restored advantage leaves each exploited answer its biased reward and still prevents collapse. Within the first ten updates, withholding positive advantage from exploited answers suffices at 1/16, while restoring it to correct answers does not at 9/18.
Tables 1 and 2 now report held-out accuracy at update 40, averaged over runs. Collapse is defined on exact-reward accuracy, so a run that writes python alongside correct answers counts as learning. Such runs exist: under restored advantage, the repaired python-history model keeps the marker in nearly all answers yet answers 63% correctly, as Appendix G reports.