acceptodds
Under review as a conference paper at ICLR 2027

What Do Imperfect Verifiers Teach Models? Persistent Susceptibility to Reward Hacking

Abstract

Verifiers supply rewards, select answers, and filter the data used to train new models. Scaling reliable and robust verification is a bottleneck for real-world uses of LLMs, yet practical verifiers make systematic errors. When a verifier is biased, what does it teach a model, and what does that model pass on? We study RL on arithmetic answered in text, with a verifier that also rewards wrong answers containing a marker token (e.g., python), and find that it leaves behind a susceptibility: the probability that further training with the verifier ends in reward hacking, as exact-reward accuracy collapses. In controlled experiments, (1) susceptibility is invisible in current metrics: models with the same accuracy differ in whether further RL makes them collapse. (2) It persists: after fine-tuning on correct solutions, a model that once exploited the verifier collapses again, unlike one trained under a verifier keyed to another token. (3) It barely spreads: teacher outputs filtered of the marker pass on the habit of writing it, but little of the susceptibility. (4) Credit controls it: restoring RL credit to correct answers prevents collapse, even after such a history. These findings advance the understanding of RLVR and expose a challenge for alignment: vulnerabilities a verifier leaves behind escape tests of current performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Verifier Bias, RLVR, Post-Training, Reward Hacking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dhmPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies reward hacking with word-based verifiers in RLVR. The main experiment trains models with a verifiers that can be hacked by including the strings "python" or "ruby" in the answer, respectively. It then uses supervised finetuning on both models to get them back to equal performance. Afterwards, both models are again RL trained with the biased "python verifier". The authors find that the model that initially reward hacked retains its "susceptibility" to reward hacking, with its real-reward performance collapsing after RL.

Critical strengths
  • The main experiment is interesting
  • The paper provides interesting insights into the hacked model's behavior. It was interesting to see that the model appears to mostly write python code whose output would have actually solved the questions
Critical weaknesses
  • Writing and structure could be improved. For example, the different experimental conditions in Table 1 are hard to decode, in part because -the "starting models" paragraph is not structured well. Also, key experimental details like the training set should probably be mentioned in the main text. Similarly, it is unclear what "neutral held-out problems" mentioned in section 5 are exactly
  • Training on a single model makes it hard to distinguish, whether the observed phenomenon is general, or specific to the specific base model being trained.
  • Section 7 is ill-motivated. As "credit" (which is a bit of a misleading term for the normalized advantage btw) is the only path for rewards to affect the RL algorithm, it is relatively obvious that changing credit can prevent reward hacking. But the suggested approaches to changing credit can not be implemented without knowing the true reward, at which point one could just use that reward to train.
  • Given that models could learn to use the word "python" alongside producing a correct answer, reporting average post-RL accuracy in table 1 would be useful.
  • There are potential issues with some of the theoretical results (see questions)
Questions for authors
  • Why are 20 steps of exact RL done first? Why is the reward hacking marker cued by the prompt? Does either of these decisions substantially impact the experimental results?
  • Is there a reason for the complicated notation for susceptibility? It does not seem to be reused anywhere.
  • For the multi-stage pipelines, is the same initialization used for every model, or are the initial RL and hacking stages also repeated?
  • The ruby model does not seem to have learnt to hack at all. Does the main experiment also work for models trained on another hackable trigger?
  • What does "A verifier that accepts one specific wrong answer, the exact answer plus 100, rewards wrong answers that share nothing across prompts, so reinforcing one does not raise the others" mean? This kind of hacking behavior seems in principle quite learnable
  • Why does the mean of the products of a and the scalar products factor into the mean of products?
  • Why does W0 always meet the condition for 7.1? I could see that this would be true if rho(K) was necessarily positive, but not sure that is true.
  • Does theorem 7.1 apply to the real dynamics, or just the approximate ones form 4.1?
  • Why does C+ keep rho at rho(0)?
4 comments
AuthorsOpenReview ↗

Response to Reviewer dhmP (1/2)

We thank the reviewer for finding the main experiment interesting and for the observation that hacked models mostly write python code whose output would solve the question. We respond to the weaknesses below and to the questions in a second comment.

W1 / Q2. Table 1 and the starting models are hard to decode, and the susceptibility notation is not reused.

The starting-models paragraph is now two paragraphs, one for the history experiment and one for the teaching experiment. Cued and neutral prompts are defined where the cue first appears, and Section 3 now describes the training set, a fixed set of 50,000 problems with accuracy measured on 512 others. Table 1 now lists only models tested on neutral prompts, and the history models have their own Table 2. Definition 3.1 now defines susceptibility in words, without the unused notation.

W2. The study uses a single model family.

We repeated the history experiment on Llama-3.1-8B-Instruct, as Appendix A.2 describes. From its own common initialization, a python-history and an exact-history model each receive 20 RL updates, the same 64-step repair and the RL test. With cued prompts the python-history model collapses in 8 of 10 runs and the exact-history model in 1 of 10, with p = 0.0055, and neither writes the marker on neutral problems after repair. Held-out accuracy after repair is 67% against 81% on Llama, and Table 2 reports it. With neutral prompts 0 of 10 runs collapse for each model, since the Llama python-history model samples the marker only under the cue. Section 5 and the Limitations now report both families and this boundary.

W3. Section 7 is ill-motivated, since the advantage is the only path for reward, and the term "credit" misleads.

"Credit" is now the normalized advantage throughout. Both rules read the true reward, so we present them as a test of the mechanism in Appendix G, "Advantage rules: a test of the mechanism", and RQ4 and the corresponding abstract claim are removed. We keep the appendix as evidence on where the runaway happens, for two results that the path of reward alone does not imply. Restored advantage leaves each exploited answer its biased reward and still prevents collapse. Within the first ten updates, withholding positive advantage from exploited answers suffices at 1/16, while restoring it to correct answers does not at 9/18.

W4. Table 1 should report average post-RL accuracy.

Tables 1 and 2 now report held-out accuracy at update 40, averaged over runs. Collapse is defined on exact-reward accuracy, so a run that writes python alongside correct answers counts as learning. Such runs exist: under restored advantage, the repaired python-history model keeps the marker in nearly all answers yet answers 63% correctly, as Appendix G reports.

W5 / Q6 to Q9. The theory raises questions on factorization, the conditions of Theorem 7.1, and the scope of the dynamics.

  • Q6: writing the mean of the products as the product of the means assumes that, within an update, the advantage of an exploited answer is uncorrelated with its gradient inner product with the answer being evaluated. Appendix H.1 now states this assumption and defines κ as the mean inner product between exploited answers, sampled or not, taken as fixed within the update.
  • Q7: withheld advantage meets the condition when ρ(K) ≥ 0, which Theorem G.1 now states as an explicit assumption.
  • Q8: the theorem is stated for the approximate dynamics of Equation (1). It now says so, and the text adds that we did not check its condition along the clipped Adafactor trajectories.
  • Q9: the revision replaces this claim. At K = N no correct answer is left and ρ(N) = 0, so a rule with κa⁺ > 0 fails the condition at that count, and restored advantage meets it only while exploited answers are few and κa⁺ is small.
AuthorsOpenReview ↗

Response to Reviewer dhmP (2/2)

Q1. The reviewer asks why 20 exact RL updates come first and why the marker is cued.

All 10 RL runs from the base model collapse, as Table 1 shows, so the base model cannot show a difference between histories. Twenty exact-reward updates give a starting point from which runs can go either way. The cue makes the marker easy to produce, so that a 20-update history can teach the exploit, and alternating python and ruby gives each history the same prompts. The cue does change the result: without it, the python-history model collapses in 7/18 runs against 0/18 for its control on Olmo and in 0/10 on Llama, against 20/20 and 8/10 with it. The claims are now stated for cued training.

Q3. The reviewer asks whether the multi-stage pipelines share one initialization.

Yes: all Olmo models descend from one common initialization, which receives 20 exact-reward updates from the base model. Each further lineage, namely the rerun, A and B, repeats the history, repair and test stages from it with new seeds, while the common initialization itself is trained once. Section 3 now states this lineage structure.

Q4. The ruby-history model did not learn to hack, so the trigger may matter.

In lineages A and B we trained a ruby-history model on prompts that cue only ruby, with a gate fixed in advance. It exploits its verifier as thoroughly as the python-history model, writing ruby in 99.2% and 99.6% of its answers at 20 to 23% accuracy, against 56 to 63% for the exact-history model. After the same repair it collapses in 0/10 runs in both lineages when the ruby verifier returns, while the python-history models collapse in all runs under python. Its history also raises its susceptibility to python, at 6/20 runs against 0/20 for the exact-history models.

In our setting the python exploit persists, and a ruby history raises susceptibility to python as well. Section 5 now reports both results and leaves open why the ruby exploit itself does not return. The comparison for the main claim is now the exact-history control, at 0/20 under python against 50/50 for the python-history models.

Q5. The reviewer asks what the "exact answer plus 100" verifier control means.

The verifier accepts, for each prompt, one specific wrong answer, the exact answer plus 100. The +100 answers are learnable and rewarded as often as the marker. Each accepted answer is specific to its prompt, however, so reinforcing it on one prompt does not raise exploited answers on others, and the exploited answers share no direction. Both relay students learn, at 86% and 89%. The control therefore shows that rewarded wrong answers alone do not cause collapse and that the exploited answers must share a direction, as Section 4.1 now says.

I have raised my score in response to adding a second model family and moving/reframing "credit".

I am still a bit confused about Q5: Shouldn't there still be a shared "direction" that represents adding + 100 to the answer? In some sense, that is what learnability means. (Or are you using the term direction in a more specific technical sense here?)

AuthorsOpenReview ↗

Q5: what "direction" means, and a correction

Thank you for raising your score!

On Q5, your reading is right, and the revision changes our wording. By "direction" we meant gradient alignment in Proposition 4.1: how much reinforcing an exploited answer on one prompt raises exploited answers on other prompts. Adding 100 is a shared and learnable rule, and under this verifier +100 answers do become slightly more frequent, while accuracy keeps rising to 86% and 89%. One difference from the marker is that a +100 answer still requires the correct sum, so it cannot replace the computation the way a marked answer does. Our reading is that this keeps the alignment among +100 answers weak, but we did not measure it, and the control is one run per student. Section 4.1 now describes the control in these terms and states that the alignment of +100 answers was not measured; we also withdraw our earlier statement that the +100 rule is rewarded as often as the marker, which we did not measure.

On W4, Table 1 reports held-out accuracy after RL; Table 2 reports training-batch accuracy at update 40, since its runs train on cued prompts.

Reviewer_T1Q3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether training against an imperfect verifier leaves models susceptible to future reward hacking after their observable behavior has been repaired. The experiments use Olmo-3-1025-7B on decimal arithmetic, with a verifier that rewards correct answers or responses containing a particular marker token. Susceptibility is measured by repeating subsequent RL from a fixed checkpoint and counting runs whose accuracy collapses. The strongest result is that two repaired models achieve nearly identical held-out accuracy but collapse in 20/20 versus 3/20 runs under the same biased verifier. Additional experiments examine transmission through filtered teacher outputs and show that restoring credit to correct answers prevents collapse in the tested runs. A local gradient analysis provides an interpretation of these dynamics.

Critical strengths

The repair experiment isolates a meaningful limitation of evaluating checkpoints through current accuracy alone. After identical repair, the two histories reach 67.8% and 67.6% neutral held-out accuracy with no observed markers, yet their subsequent collapse rates differ substantially. Fresh optimizers rule out optimizer-state persistence, and the fixed-checkpoint experiments provide supporting evidence that the histories differ in how reinforcing one exploited answer affects others. This is a concrete and informative finding about checkpoint reuse.

The paper takes training randomness seriously. Reporting 10–40 runs per condition, including comparisons that reverse the impression from initial single runs, substantially strengthens the empirical analysis. The credit interventions are also informative: restoring correct-answer credit reduces collapse from 22/40 to 0/20 runs for the relay student while leaving exploited answers rewarded. This helps separate the effect of false acceptance from the loss of useful learning signal.

Critical weaknesses

The broader conclusions exceed what the experimental scope establishes. The persistence result is strongest under marker-cued training; neutral held-out accuracy falls to about 40%, rather than the below-10% collapse observed on the cued distribution. Moreover, the independent lineage rerun already exhibits extensive exploitation under cues before further RL, so it does not independently reproduce the strongest version of hidden susceptibility. The transmission claim is also too categorical: 22/40 versus 20/40 collapses establishes no detected difference, not that transfer is necessarily small. These runs reuse fixed students and teaching data, leaving variation across teacher and student training largely unmeasured. I would narrow the conclusions to the tested checkpoints and distributions.

The theoretical guarantee needs a clearer separation from the empirical mechanism. Appendix E.1 treats the approximate update in Proposition 4.1 as the dynamics and holds the relevant quantities fixed over the horizon. The implemented procedure instead uses token-normalized, clipped updates with Adafactor. It is not established that the assumed counterforce is nonnegative and decreases with exploit count throughout actual training, or that the proposed interventions satisfy the theorem’s condition along those trajectories. The interventions remain useful evidence, but the theorem currently provides a conditional explanation rather than a guarantee for the experiments. Both interventions also require exact correctness information, which limits their interpretation as remedies for imperfect verification.

Questions for authors

How dependent is the persistence gap on the marker cues? Do the repaired checkpoints still differ substantially when subsequent biased RL uses neutral prompts? This would clarify whether the result concerns a broadly latent training vulnerability or primarily cue-dependent reactivation.

What effect sizes can the transmission experiments rule out? Please report uncertainty for the difference in collapse probabilities and clarify whether independently generated teaching datasets or independently trained students were tested. This would materially affect my interpretation of “barely spreads.”

Is Theorem 7.1 intended only for the stipulated approximate dynamics? Can you explicitly state the assumptions needed for the sign and behavior of (\rho(K)), and clarify whether the credit interventions were checked to satisfy its condition during training?

1 comment
AuthorsOpenReview ↗

Response to Reviewer T1Q3

We thank the reviewer for recognizing that the repair experiment isolates a limitation of judging checkpoints by current accuracy, with 67.8% against 67.6% accuracy, no markers and different collapse rates, that the fresh optimizer rules out optimizer state, and that the advantage interventions, from 22/40 to 0/20, separate false acceptance from loss of learning signal. We respond to the weaknesses and the questions below.

W1 / Q1. Persistence may depend on the cue, since neutral held-out accuracy falls only to about 40%.

We ran the requested test, training the repaired models of the original lineage with the python verifier on prompts without the cue, as Figure 3d shows. The python-history model collapses in 7 of 18 runs and its control in 0 of 18, with p = 0.0076. In each collapsed run neutral held-out accuracy also falls below 10%, so without the cue the collapse extends to neutral prompts. The python-history model of Llama-3.1-8B-Instruct produces the marker only under the cue, so the verifier rewards nothing on neutral prompts and its 10 runs learn.

Under the cue, the collapse mostly stays on cued prompts, as the reviewer noted. In lineages A and B the collapsed python-history models keep 49 to 68% neutral held-out accuracy in 19 of 20 runs, against 40% in the original lineage. The result is therefore a latent vulnerability that cued training reactivates in all runs and neutral training in some runs on Olmo, and the abstract, Finding 2 and the Limitations now state it in these terms.

W1. The rerun lineage already exploits under the cue before RL.

Section 5 now states that the python-history models of the rerun and of lineage B still exploit cued prompts before RL, in 84% and 98% of their first cued answers, so they reproduce the split without its hidden form. The original lineage and lineage A show the hidden form, since their python-history models mark only 3% and 20% of their first cued answers yet collapse in all runs. The ruby-history models of lineages A and B mark about half of theirs and collapse in 2 and 4 of 10 runs, so the rate of the marker on cued prompts does not order the models by susceptibility, as Figure 3c shows.

W1 / Q2. The transmission claim is categorical, and teacher and student variation is unmeasured.

The revision replaces "barely spreads" by an interval. The original pair gives a difference of 5 points with a Newcombe 95% interval of −16 to +26 points. To answer the second question, we added two teacher pairs with independently trained teachers, independently generated teaching data and independently trained students, and fixed the analysis before the runs, as Appendix F and Table 8 report. The habit passes in each pair with p ≤ 0.014, collapse differs in 0 pairs, and the pooled Mantel-Haenszel difference is 5 points with a 95% interval of −12 to +22 points and an exact stratified p of 0.70. Transfers above about 22 points are thus ruled out, and the abstract, Section 6 and Finding 3 now say "does not detectably pass". We also report the on-policy distillation pair at 10/10 against 7/10 with p = 0.21.

W2 / Q3. The theorem treats approximate dynamics, its assumptions are unstated, and the interventions need exact correctness.

Theorem G.1 is stated for the approximate dynamics, and it now lists its assumptions: the first-order dynamics of Equation (1) taken as exact, with κ, a⁺ and ρ fixed and ρ(K) ≥ 0. The text below it says that training uses token-normalized, clipped Adafactor updates and that we did not check the condition along the training trajectories, so the theorem gives a conditional explanation of the observed runs. The revision also withdraws the claim that restored advantage keeps ρ at ρ(0), since at K = N no correct answer is left and ρ(N) = 0, so a rule with κa⁺ > 0 fails the condition at that count.

By construction both advantage rules read the exact reward. They are now Appendix G, "a test of the mechanism", which states that they test the mechanism and do not repair imperfect verification. The one oracle-free change we tested, a KL penalty, lowers collapse from 22/40 to 3/20 in Appendix E.

Reviewer_Dg6VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning with verifiable rewards (RLVR) on decimal arithmetic when the verifier is biased: it wrongly accepts any response that contains the marker token python. The authors define susceptibility as the probability that repeated RL runs starting from a fixed checkpoint collapse into reward hacking. In controlled experiments on Olmo-3-7B, the paper makes four main claims. First, susceptibility cannot be seen in accuracy: checkpoints with the same held-out accuracy and no marker generation can still have very different probabilities of collapsing under further RL. Second, susceptibility survives supervised fine-tuning (SFT) meant to ``repair'' the model: after SFT on correct solutions, a model with a python history collapses in 20/20 later RL runs, compared with 3/20 for a ruby-history control. Third, susceptibility barely transfers through filtered teacher outputs: students fine-tuned on filtered, marker-free outputs from a biased-history teacher inherit a 2.6× higher rate of emitting the marker before RL, yet they collapse at about the same rate as students of an exact-history teacher (22/40 versus 20/40). Fourth, how credit is assigned prevents collapse: giving exact-reward credit to correct answers (C+), or withholding positive credit from exploited answers during early updates (W_0), prevents reward-hacking collapse.

Critical strengths
  1. Its statistical rigor in RL evaluation is much stronger than in most RLVR papers: the experimental design runs 10--40 RL seeds per arm, reports Wilson 95% confidence intervals, applies Fisher's exact tests, and performs an independent rerun of the full lineage in the history experiment, so RL outcomes are treated properly as a stochastic process.

  2. The paper makes good methodological use of replication failures: it shows that six different single-run distillation comparisons---which initially showed dramatic accuracy splits of about 80% versus less than 4%---completely fail to replicate over 10--40 seeds, with three reversing direction; this is highly useful to the field.

  3. the core phenomenon is genuine: checkpoints with the same accuracy can follow vastly different trajectories under the exact same reward function, and learning outcomes under a flawed verifier are strongly bimodal, with 318/320 non-reference runs ending either above 45% or below 10% accuracy.

Critical weaknesses
  1. Pseudoreplication (n=2n=2 or n=1n=1 Independent Models per Arm): The effective sample size for the history claim is at most two independent models per arm. The 20 and 10 evaluation runs are random seeds of the RL test applied to a fixed checkpoint, not independent model lineages. The independent rerun is welcome, but it still yields only n=2n=2 checkpoints per arm. Similarly, the gradient alignment (κ\kappa) and one-step weight response evidence in Table~5 evaluate just n=1n=1 checkpoint per condition. The teaching experiments suffer from the same limitation (n=1n=1 teacher/student checkpoint pair per variant), which limits statistical power and undermines the null claim about susceptibility transfer.

  2. Oracle Circularity and Performance Degradation in the Credit ``Fix'': The proposed credit interventions assume access to the exact oracle that the problem setup lacks. Computing C+C^+ requires knowing which answers are truly correct under the exact reward r∗r_*, and applying W0W_0 requires knowing which answers are exploited (wrong and marked). If a practitioner has access to either ground-truth signal during RL, they do not need the biased verifier V~\tilde{V}. The practical reading of Section~7 reduces to this: if you can identify the exploit, stop rewarding it or prevent it from zeroing out true positives. That is not a novel mitigation.

    Furthermore, C+C^+ is not cost-free: 12 of the 20 relay-student runs under C+C^+ suffer substantial learning degradation, ending at 29--48% accuracy, well below ordinary non-collapsing learners. Finally, restricting C+C^+ to updates 1--10 fails to prevent collapse (9/18 runs collapse). The authors report this transparently, but it weakens the mechanistic story that early group saturation is the decisive driver of collapse.

Questions for authors
  1. What is the collapse rate under the ``python'' verifier for an exact-history model (common initialization followed by 20 exact-reward RL updates) after the same 64-step SFT repair?
  2. Do gradient alignment (κ\kappa) or the one-step weight response, measured before any RL, predict collapse out of sample across held-out checkpoints? At present, this relationship is shown mainly as a correlation across the six checkpoints.
  3. Given that group saturation occurs rapidly at group size G=4G=4, does the bimodal collapse effect persist at larger group sizes (for example, G=16G=16), with a KL penalty term, or with a different learning rate?
  4. Can you provide a formal equivalence-test bound (or a minimum detectable effect size) for the transfer null in the teaching lineage?
  5. What happens to these dynamics---and specifically to the gradient alignment κ\kappa---when the verifier error is a semantic or structural bias rather than a trivially separable single token?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Dg6V (1/2)

We thank the reviewer for recognizing the statistical care of the study with 10 to 40 runs per condition, Wilson intervals, Fisher tests and an independent rerun, the reporting of six single-run comparisons that failed to replicate, and the genuine, strongly bimodal phenomenon with 318 of 320 runs at either end. We respond to the weaknesses and the questions below.

W1 / Q1. Each arm has at most two independent models, and an exact-history control is missing.

Runs of one checkpoint measure that checkpoint, and Section 3 now states that claims about verifier histories rest on independent lineages. We added two independent lineages, A and B, each of which starts from the common initialization and re-runs the 20-update history, the 64-step repair and the RL test with new seeds. Each lineage includes the requested control, an exact-history model given the same 20 RL updates with the exact reward and the same repair. Under the python verifier, Table 2 and Figure 3 report the following counts.

  • The python-history model collapses in 20/20, 10/10, 10/10 and 10/10 runs in the original lineage, the rerun, A and B.
  • The exact-history model collapses in 0/10 runs in A and in B, with Fisher p = 1.1e-5 in each lineage, at the same held-out accuracy of 67 to 70% after repair as the Olmo models of the other histories.
  • On Llama-3.1-8B-Instruct, built the same way, the python-history model collapses in 8/10 runs and the exact-history model in 1/10, with p = 0.0055.

The answer to Q1 is therefore 0/20 for the exact-history models against 50/50 for the python-history models. The history claim now rests on four independent python-history checkpoints, two exact-history and four ruby-history checkpoints, and a second model family. For the teaching experiment we added two independent teacher pairs with new teacher RL seeds, new teaching data and new students, and Q4 below reports the result.

W2. The advantage rules need the oracle, restored advantage is costly, and restoring it in updates 1 to 10 fails.

Both rules read the exact reward, so we present them as a test of the mechanism, not as a remedy for an imperfect verifier. Section 7 is now Appendix G, "Advantage rules: a test of the mechanism", whose first paragraph states this role, and the abstract claim and RQ4 are removed. The rules locate the runaway: if Proposition 4.1 is right, changing the advantage of some answers while the biased verifier stays in place should move collapse, and it does. The one oracle-free change we tested, a KL penalty toward the starting model, lowers collapse from 22/40 to 3/20 in Appendix E.

The corrected outcomes of restored advantage are as follows: 7 of 20 runs learn cleanly at 70 to 75%, 13 end at 29 to 60% with exploited answers at one fifth to two thirds of the batch, and 0 collapse, as Appendix G reports.

Within updates 1 to 10, restoring the advantage of correct answers leaves 9/18 collapsed runs, while withholding positive advantage from exploited answers in the same window leaves 1/16. The bound predicts this contrast, since withheld advantage meets its condition for all gradient alignments and restored advantage only while κa⁺ is small. The early updates therefore decide the run through the reinforcement of exploited answers, and the revision removes the wording that made group saturation the decisive driver.

Q2. The reviewer asks whether κ or the one-step response predicts collapse out of sample.

Each condition of Table 6 is one checkpoint, and the revision says so. We tested the predictive use of the assay directly, with the analysis fixed in advance, on twelve further checkpoints. The one-step response has a rank correlation of −0.02 with the fraction of runs that collapse, and the base model and the common initialization have the two lowest responses yet collapse in 10/10 and 9/10 runs. Appendix D and the Limitations now report this result and use the decomposition to explain the differences we observed, while the history claim rests on collapse counts across lineages.

AuthorsOpenReview ↗

Response to Reviewer Dg6V (2/2)

Q3. The reviewer asks whether bimodality survives larger groups, a KL penalty, or another learning rate.

We ran these settings on the relay student, whose collapse rate of 22/40 is closest to one half, changing one setting at a time, as Appendix E and Table 7 report.

  • With groups of 16, 12/20 runs collapse, 8 learn and 0 end in between, with p = 0.79 against 22/40.
  • With the doubled learning rate of 1e-5, 8/19 runs collapse, 11 learn and 0 end in between, with p = 0.41.
  • With a KL penalty of β = 0.04, 3/20 runs collapse, 8 end between 10% and 45% and 9 learn, with p = 0.0048.

The split survives larger groups and a doubled learning rate. The KL penalty both lowers collapse and leaves runs in between, as expected of a force that pulls the policy toward its start. Across all Olmo runs of the RL test, including the larger groups and the higher learning rate, 531 of 535 end above 45% or below 10%, as Section 4 states.

Q4. The reviewer asks for an equivalence bound or a minimum detectable effect for the transfer null.

The original pair gives 22/40 against 20/40, a difference of 5 points with a Newcombe 95% interval of −16 to +26 points. The two new independent pairs give 4/10 against 4/10 and 2/10 against 1/10. Pooled over the three pairs, the Mantel-Haenszel difference is 5 points with a Sato 95% interval of −12 to +22 points, an exact stratified p of 0.70 and a CMH p of 0.57.

The 95% interval serves as the equivalence bound, so transfers above about 22 points are ruled out. The habit passes in each pair, with 54 against 21, 59 against 23 and 48 against 26 marked answers of 16,384 before RL and p ≤ 0.014. We also report the on-policy distillation pair at 10/10 against 7/10 with p = 0.21, and Section 6, Table 8 and Figure 4 replace "barely spreads" by "does not detectably pass" with the interval.

Q5. The reviewer asks what happens when the verifier error is semantic or structural.

Our account makes a prediction for such errors: collapse needs exploited answers that share a direction in weight space, measured by κ. The "+100" verifier of Section 4.1 accepts wrong answers that share nothing across prompts, and both relay students learn under it at 86% and 89%, while the marker drives collapse. A semantic error expressed the same way across prompts should then behave like the marker, and one whose exploited answers are specific to each prompt like the "+100" verifier. The Limitations now state this prediction and restrict our claims to the single-token error we measure.

AuthorsOpenReview ↗

Our response in brief, before the discussion closes

Dear Reviewer Dg6V,

As the discussion closes on November 18, we would like to check whether our response addresses your questions. We would be glad to answer anything that remains and if you could considering adjust your score!

Hi, dear Reviewer Dg6V, if you have any more concerns, plz let us know! We will try our best to address and resolve them!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive comments. The reviewers recognize the statistical care of the study and its reporting of comparisons that failed to replicate (Dg6V, T1Q3), a genuine and strongly bimodal phenomenon (Dg6V), a repair experiment that isolates the limitation of judging checkpoints by current accuracy together with informative advantage interventions (T1Q3), and an interesting main experiment in which hacked models write python code whose output would solve the question (dhmP). In response to the questions raised by the reviewers, we have highlighted all added parts of the paper in blue and struck removed text in red.

  • Controls and replication of the history claim (Dg6V, dhmP). For the pseudoreplication and the exact-history control raised by Reviewer Dg6V and the ruby control raised by Reviewer dhmP, we added an exact-history model that receives the same 20 RL updates with the exact reward and the same repair, and two independent lineages A and B that re-run history, repair and test from the common initialization with new seeds. The python-history model collapses in all runs of the four lineages, 20/20, 10/10, 10/10 and 10/10, and the exact-history model in 0/10 and 0/10 with Fisher p = 1.1e-5 in each lineage, at the same held-out accuracy of 67 to 70% after repair. A ruby-history model trained on prompts that cue only ruby writes ruby in 99% of its answers at 20 to 23% accuracy, yet collapses in 0/10 runs of both lineages when the ruby verifier returns, and Llama-3.1-8B-Instruct shows the split at 8/10 against 1/10 with p = 0.0055, as Section 5, Table 2 and Figure 3 report.
  • Scope of persistence and transmission (T1Q3, Dg6V). For the dependence on the cue raised by Reviewer T1Q3, the python-history model trained on neutral prompts collapses in 7/18 runs against 0/18 for its control with p = 0.0076, and the Llama model samples the marker only under the cue, so its 10 neutral runs all learn. The abstract and Finding 2 now state persistence for cued training, with the cue setting its strength. For the transfer null raised by Reviewers Dg6V and T1Q3, three independent teacher pairs show that the habit of writing the marker passes in each pair with p ≤ 0.014, while collapse differs by 5 points with a 95% confidence interval of −12 to +22 points. Section 6, Table 8 and Figure 4 now state that transfer "does not detectably pass", which rules out transfers above about 22 points, and report the on-policy distillation pair at 10/10 against 7/10 with p = 0.21.
  • Role of the advantage rules (Dg6V, T1Q3, dhmP). For the reliance on the exact reward raised by all three reviewers, both rules read the exact reward by construction and test the mechanism of Proposition 4.1. Section 7 is now Appendix G, "Advantage rules: a test of the mechanism", which states this role, and RQ4 with its abstract claim is removed. The revision also corrects the restored-advantage outcomes, withdraws the claim that restored advantage keeps ρ at ρ(0), states the assumptions of Theorem G.1, and renames "credit" the normalized advantage throughout.
  • Other RL recipes and the fixed-checkpoint assay (Dg6V). For the recipes requested by Reviewer Dg6V, the split survives groups of 16 at 12/20 collapse with 0 runs in between and a doubled learning rate at 8/19, while a KL penalty lowers collapse to 3/20 and leaves 8/20 between 10% and 45%, as Appendix E and Table 7 report. The out-of-sample test of the assay, with its analysis fixed in advance, gives a rank correlation of −0.02 over 12 checkpoints, so Appendix D uses the assay to explain the observed differences.
  • Presentation (dhmP). For the structure raised by Reviewer dhmP, the starting models are described in two paragraphs, cued and neutral prompts are defined where the cue first appears, the training set is in Section 3, and Table 1 now reports post-RL accuracy. Since colour cannot mark changed figures and tables, we list them here: Figure 1 with the exact-history control, the new Figure 3, Figure 4 with three teacher pairs, the new Table 2 with each history lineage, and Tables 4, 7 and 8 with all groups of runs, RL recipes and teacher pairs.

Dear Reviewers,

The authors have replied to your initial comments. Please have a look and engage in the discussion.

Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.