- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors investigate spurious rewards for RLVR as a probe. For this purpose, they first show that prior explanations of improvements observed due to spurious rewards are incomplete: two different model checkpoints trained on the same rollouts show different behavior, and models trained on "random token problems" with math instructions also improve. They then find that models generally tend to display three types of behavior: dormant, receptive, and autodidactic. During its training, Olmo 2 shows a steady progression through these stages on math word problems.
Critical strengths
The paper is well-written, clearly outlined, and provides an understandable exposition of spurious rewards in RL. The experimental designs are clear, and show that previous explanations are at least incomplete. Further, the authors present an interesting differentiation between dormant, receptive, and autodidactic.
Critical weaknesses
The paper suffers from two critical weaknesses: the more important one is in the soundness of its experiments, and the other in its claim of the usefulness of spurious rewards as a probe. Both would need to be properly addressed before I can raise my score. Even if the issues are fixed, I do not think I can raise my score to clear acceptance (4/4): the work itself only provides interesting observations and a clear exposition, but does not make major progress or new innovations.
Soundness of experiments
The experiments presented in this paper point towards noise and formatting issues as the main source of improvement (as the authors also acknowledge in appendix F):
- Table 16 and 22 give evidence that the formatting extractor used in the main experiments is invalid.
- Figure 6,7 are very noisy: the orange line jumps up and down across checkpoints, and is often less clear than the annotations regarding "receptive" and "autodidactic" suggest, e.g., in Figure 7a, one checkpoint that has 0 spurious reward improvement is still labeled as "autodidactic".
- Most observed improvements seem to be caused by variance. The authors also confirm this in L2190-2194. Interestingly, the authors define their probe in L356 to use the expected value, yet decide to report the maximum everywhere instead. The authors should use the expected value: this is less susceptible to variance and gives a clearer picture of the performance improvements coming from spurious reward training.
Combined, this strongly indicates the experiments do not show any useful dormant capability. Instead, the authors are measuring what a proper evaluation of the pre-trained model would do. Evaluating pre-trained models is tricky, because formatting is such a big problem, and needs to be done with (1) a proper prompt, (2) few-shot when possible, and (3) a lenient extractor that has been properly verified to have near-zero false negative rate.
To improve the soundness of the experiments, the authors should:
- Repeat all experiments with a proper evaluation of all checkpoints. This includes the three recommendations above, together with strong evidence that (3) holds. Possibilities for good extractors include extensive hand-designed regex extractors or LLM-judges.
- Show that the models with the biggest gains on GSM also improve their general math ability, by e.g., evaluating on the MATH-500 benchmark.
- Evaluate all models under different input prompts to see if they score consistently better under all prompts, and not just under one specific prompt.
Spurious rewards as a probe
First, the authors should not claim they address a decade-long debate about probes. The debate concerns existing probes, and what they learn exactly. Importantly, excluding the possibility of correctness feedback through the reward channel does not show that the measured performance improvement is useful as a measurement of a dormant capability (as the authors also acknowledge in Appendix F).
Second, the authors do not show the utility of the probe in any realistic scenario. It is unclear what a practitioners training pre-trained models could do with this information. The authors should more clearly argue (1) why practitioners would want to spend compute on using their probe and (2) what they would gain from doing so. For (2), the authors should design clear experiments that show how their probe can inform practitioners to make better choices.
Minor remarks
- It should be clear from section 3 onwards what the training and evaluation data is. Currently, this is not mentioned at this stage of the paper.
- , as mentioned in L449, is never proven: section 4 only establishes that . The authors should remove this claim, especially because it is possible to construct simple (but contrived) counterexamples to .
Questions for authors
See above
3 comments
Q2: Variance and Statistical Tests
Our results are not caused by noise or variance: the two findings that our account rests on are statistically significant, and the three-stage pattern is clear.
The passage that the reviewer cites as our confirmation, [2190–2194], reports a significant difference between two conditions. To refresh your memory, with random tokens inside the math template, 16 of 32 runs change by more than ten points in either direction; with random tokens alone, none of 32 does (p = 3.05 × 10⁻⁵; 4.27 × 10⁻⁴ after Holm correction). The same holds for the learner. Trained on the same P1 rollouts with the same random rewards, the P2 checkpoint changes by 13.56 points on average and the P1 checkpoint by 1.75, a difference that passes Holm correction among the six tests we specified in advance (Holm p = 0.0443) [2297–2304]. Noise alone would not distinguish one input from another, or one checkpoint from another.
The three-stage pattern does not rest on any single run. Under GT rewards, no run at the first three checkpoints exceeds 8.5%, and at the end of pre-training the four runs end at 61.5–90.5%. Under random rewards, none of the 200 P1 runs ends at or above 50%, compared with 15 of 96 P2 runs [1543–1544]. The orange line moves within P2 because it is a maximum over eight seeds at each checkpoint, and we state in the main text that this response is not stable across mid-training [414–416]. The stage bands mark phases of training rather than individual checkpoints, which is why the weak +26B checkpoint on code sits inside the autodidactic band of Figure 7a [1790–1794].
We also do not “report the maximum everywhere.” Every stage in Section 5 is defined by mean gains: 0.4 and 0.5 points for the dormant checkpoint, 79.5 against 4.3 for the receptive one, and 26.8 for the autodidactic one [408–416]. The stage definition uses the expected gain explicitly [422–423]. The figures plot the maximum together with every individual seed, and all 444 terminal scores are in Tables 8 and 9. As @piFx notes, “the appendix reports failed seeds, intermediate and terminal scores, and statistical tests.”
Q3: Spurious Rewards as a Probe
We believe that reachability is a major advance in the probing debate. The debate has moved from whether a probe has “just learned the task” (Hewitt & Liang, 2019) to accounts of decodability, which ask what can be recovered from a representation and under what conditions (Pimentel et al., 2020; Pimentel & Cotterell, 2021; Xu et al., 2019; Hewitt et al., 2021) [136–143]. The probes that these accounts describe still largely rely on supervised learning from task labels, applied to a frozen model [131, 144–146]. Reachability measures what the model itself can attain under further training, and random rewards let us measure it without telling the model which answers are correct (Theorem 1). Of course, this debate is far from being settled, but it now has a probe that tests the model’s own performance and gives it no correctness feedback, and this is what we mean by addressing it [030–033].
As for utility, our contribution is a scientific one, and we believe its value is considerable. Probing has long been a central tool for studying what models know and what they are capable of [129–130]; it is a scientific instrument, and we see no reason to judge reachability by a different standard. Our work offers a new explanation for the spurious-reward paradox, one that accounts for two observations that the existing explanations do not cover (Section 3). It also shows what neither current accuracy nor correctness-rewarded RL can show. Two checkpoints with the same accuracy (3.5%) reach 8.5% and 55% in their best runs under the same correctness-rewarded RL [020–022]; and GT rewards are effective in both the receptive and the autodidactic stage, so only random rewards distinguish the two (Eq. 7). How a model will respond to post-training, and when it begins to improve without being told the answers, are questions that anyone who trains models should want answered, at a time when post-training plays an ever more important role in developing LLM capabilities [268–269].
Minor Points
- Training and evaluation data in Section 3. Thank you for the suggestion. We will state at the start of Section 3 that these experiments use synthetic GSM-style arithmetic problems and are evaluated on 200 held-out questions, with a pointer to Appendix D.1.
- ∆GT ≥ ∆rand [449]. Thank you for catching this. The ordering is one we observe empirically; Section 4 does not prove it. We will correct the wording.
I thank the authors for their response. It clarifies several points, and I acknowledge that some statements in my original review were overstated. However, I do not think my main concerns are caused by misunderstandings, but by disagreements about the novelty and methodology:
- While one could argue that following formatting constraints is a dormant capability, it is, to me, a very uninteresting one, and not the capability intended to be measured by the used benchmarks. By using those benchmarks, the authors implicitly communicate that performance on the task improves, but the presented numbers do not establish that.
- I did not claim that formatting constraints were the only cause of the observed gains. The large swings presented in the appendix based on simple changes in the extractors are cause for concern, and explicitly acknowledged by the authors themselves in the appendix ("The scoring diagnostics show that output conventions can materially affect both initial scores and measured gains.").
- It is still unclear what the probe adds beyond a thorough evaluation of the pretrained model using the suggestions I made in my review. The value of the methodology as a probe is therefore unclear. I am not saying that doing so would fully remove all observed gains, just that the paper currently does not provide sufficient evidence of what the probe adds beyond such an evaluation.
- While I should not have claimed that the maximum is used everywhere, it is still displayed prominently. Had the authors used the mean (or even the median) in, for instance Figure 6, the increases in the orange line would have looked much less dramatic. Using the maximum therefore significantly biases the interpretation of the results.
- I should not have stated that most improvements seem to be caused by variance. However, using the maximum when variability across seeds is large, as observed at some later checkpoints, can bias the interpretation of the results by emphasizing unusually successful runs. I did not claim that the results were not statistically significant.
Finally, I felt the need to point out that I would be unwilling to change my score to a 4, since I personally like to keep an open mind during the rebuttal. In this case, I felt an exceptional rebuttal could potentially have raised my score to a 3, but felt there are fundamental weaknesses that would prevent me from raising my score further. In the rebuttal, the authors have not sufficiently addressed my concerns. However, I do acknowledge that some of my concerns were overstated (as pointed out in my reply here) and I will therefore raise my score to a 2.
Despite the unusually low ratings, we would still like to thank the reviewer for recognizing that our work “provides interesting observations and a clear exposition.” We are confused, however, by the fact that the reviewer gave our paper a rating of 1 and yet found it necessary to add that “even if the issues are fixed, I do not think I can raise my score to clear acceptance (4/4).” We find the two hard to reconcile.
Respectfully, all of the “critical weaknesses” raised by the reviewer stem from misunderstandings of our paper, and we would like to take this opportunity to correct several factual errors in the review. As we show point by point below, none of the criticisms of our experiments holds. Our paper is sound.
Q1: Formatting Extractor
Learning to decode an answer more reliably is a capability by any measure. This has always been the discussion surrounding spurious rewards: format rewards are among the conditions studied by Shao et al. (2026), and Yan et al. (2026) tentatively attribute some of the gains to format alignment or dormant capability. Moreover, we have dedicated a specific part of Section 4 to the notion of capability. We define dormant capability in terms of computational accessibility, that is, the model’s ability to use the information already available to it [275–291]. The larger debate over what counts as genuine knowledge or capability remains unsettled, and it is beyond the scope of our paper. Whether random-reward training improves general mathematical reasoning capability, which in itself is not properly defined, is likewise beyond the discussion of spurious rewards. We stay agnostic on this question, as laid out in Section 6 [473–482]:
We are very confused by the assertion that Tables 16 and 22 give evidence that the formatting extractor used in the main experiments is invalid. Both tables are diagnostics that we ran and reported ourselves, and neither shows that the gains are an artifact of the extractor. To refresh your memory, some frozen models answer the question and then, instead of stopping, go on to generate another question until they reach the token limit; the extractor can then select a number from that continuation, and a correct answer is scored as incorrect [2017–2019]. Tables 16 and 22 re-score the outputs with rules that ignore the continuation. Table 16 contains frozen scores only, for the additional SmolLM2 and OLMo-3 checkpoints of Appendix E.3: it shows that some frozen scores are lowered by continued generation, but it has no post-training scores and therefore cannot speak to the gains. Table 22 applies one rule both before and after training, to two checkpoints trained on identical rollouts and identical random rewards:
Under this rule, P2 gains about 24 points with either response source, which is more than it gains under the primary scorer (4.5 and 13.7 points), while P1 declines. A formatting artifact of the extractor does not explain this difference, since the extractor, the training responses, and the rewards are all held fixed and only the learner differs. This is the control that the two other reviewers, @piFx and @XEB8, both single out as a strength: in @piFx’s words, “the number of accidentally rewarded correct answers cannot fully account for the result.” Finally, a model that gives the correct answer and then keeps generating has not completed the task as instructed, and learning to stop is part of the decoding capability that we discuss above.