acceptodds
Under review as a conference paper at ICLR 2027

Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability

Abstract

We connect the spurious-reward paradox to a model’s reachability and propose random-reward reinforcement learning (RL) as a useful tool for the probing enterprise, addressing a decade-long debate over what probing performance actually reveals about a model. There are two prevailing explanations for the surprising finding that even random rewards can improve the performance of large language models (LLMs): one attributes the gains to particular mechanisms within RL training (Shao et al., 2026; Chen et al., 2025); the other to data contamination \citep(Yan et al., 2026). Our results motivate a different view: spurious-reward RL can probe a model's reachability, or what further training can attain from its current state under specified constraints, beyond what is reflected in its current performance. Two OLMo checkpoints with the same accuracy on synthetic arithmetic (3.5%), for example, reach 8.5% and 55% in their best runs under the same correctness-rewarded RL. Examining OLMo checkpoints across pre-training and mid-training reveals three distinct regimes of training response: early on, RL produces little improvement even when correct answers are rewarded; later in pre-training, rewarding correct answers becomes effective while random rewards remain weak; and, upon entering mid-training, even random rewards can produce large gains. A similar ordering appears in a number-masked supervised fine-tuning (SFT) analysis of these checkpoints, suggesting that the pattern is not specific to a particular RL mechanism. Moreover, RL with random rewards offers a distinctive perspective on what training can make an LLM do, since its reward signal supplies no information about which answers are correct. By asking what training can attain without correctness feedback, it addresses the label-leakage side of a central problem in decodability-based probing: whether a successful probe reveals the model’s capabilities or learns the task itself.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fHwyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors investigate spurious rewards for RLVR as a probe. For this purpose, they first show that prior explanations of improvements observed due to spurious rewards are incomplete: two different model checkpoints trained on the same rollouts show different behavior, and models trained on "random token problems" with math instructions also improve. They then find that models generally tend to display three types of behavior: dormant, receptive, and autodidactic. During its training, Olmo 2 shows a steady progression through these stages on math word problems.

Critical strengths

The paper is well-written, clearly outlined, and provides an understandable exposition of spurious rewards in RL. The experimental designs are clear, and show that previous explanations are at least incomplete. Further, the authors present an interesting differentiation between dormant, receptive, and autodidactic.

Critical weaknesses

The paper suffers from two critical weaknesses: the more important one is in the soundness of its experiments, and the other in its claim of the usefulness of spurious rewards as a probe. Both would need to be properly addressed before I can raise my score. Even if the issues are fixed, I do not think I can raise my score to clear acceptance (4/4): the work itself only provides interesting observations and a clear exposition, but does not make major progress or new innovations.

Soundness of experiments
The experiments presented in this paper point towards noise and formatting issues as the main source of improvement (as the authors also acknowledge in appendix F):

  • Table 16 and 22 give evidence that the formatting extractor used in the main experiments is invalid.
  • Figure 6,7 are very noisy: the orange line jumps up and down across checkpoints, and is often less clear than the annotations regarding "receptive" and "autodidactic" suggest, e.g., in Figure 7a, one checkpoint that has 0 spurious reward improvement is still labeled as "autodidactic".
  • Most observed improvements seem to be caused by variance. The authors also confirm this in L2190-2194. Interestingly, the authors define their probe in L356 to use the expected value, yet decide to report the maximum everywhere instead. The authors should use the expected value: this is less susceptible to variance and gives a clearer picture of the performance improvements coming from spurious reward training.

Combined, this strongly indicates the experiments do not show any useful dormant capability. Instead, the authors are measuring what a proper evaluation of the pre-trained model would do. Evaluating pre-trained models is tricky, because formatting is such a big problem, and needs to be done with (1) a proper prompt, (2) few-shot when possible, and (3) a lenient extractor that has been properly verified to have near-zero false negative rate.

To improve the soundness of the experiments, the authors should:

  • Repeat all experiments with a proper evaluation of all checkpoints. This includes the three recommendations above, together with strong evidence that (3) holds. Possibilities for good extractors include extensive hand-designed regex extractors or LLM-judges.
  • Show that the models with the biggest gains on GSM also improve their general math ability, by e.g., evaluating on the MATH-500 benchmark.
  • Evaluate all models under different input prompts to see if they score consistently better under all prompts, and not just under one specific prompt.

Spurious rewards as a probe
First, the authors should not claim they address a decade-long debate about probes. The debate concerns existing probes, and what they learn exactly. Importantly, excluding the possibility of correctness feedback through the reward channel does not show that the measured performance improvement is useful as a measurement of a dormant capability (as the authors also acknowledge in Appendix F).

Second, the authors do not show the utility of the probe in any realistic scenario. It is unclear what a practitioners training pre-trained models could do with this information. The authors should more clearly argue (1) why practitioners would want to spend compute on using their probe and (2) what they would gain from doing so. For (2), the authors should design clear experiments that show how their probe can inform practitioners to make better choices.

Minor remarks

  • It should be clear from section 3 onwards what the training and evaluation data is. Currently, this is not mentioned at this stage of the paper.
  • ΔGT≥Δrand\Delta_{GT} \geq \Delta_{rand}, as mentioned in L449, is never proven: section 4 only establishes that Δrand≤Δreach\Delta_{rand} \leq \Delta_{reach}. The authors should remove this claim, especially because it is possible to construct simple (but contrived) counterexamples to ΔGT≥Δrand\Delta_{GT} \geq \Delta_{rand}.
Questions for authors

See above

3 comments

Despite the unusually low ratings, we would still like to thank the reviewer for recognizing that our work “provides interesting observations and a clear exposition.” We are confused, however, by the fact that the reviewer gave our paper a rating of 1 and yet found it necessary to add that “even if the issues are fixed, I do not think I can raise my score to clear acceptance (4/4).” We find the two hard to reconcile.

Respectfully, all of the “critical weaknesses” raised by the reviewer stem from misunderstandings of our paper, and we would like to take this opportunity to correct several factual errors in the review. As we show point by point below, none of the criticisms of our experiments holds. Our paper is sound.

Q1: Formatting Extractor

Learning to decode an answer more reliably is a capability by any measure. This has always been the discussion surrounding spurious rewards: format rewards are among the conditions studied by Shao et al. (2026), and Yan et al. (2026) tentatively attribute some of the gains to format alignment or dormant capability. Moreover, we have dedicated a specific part of Section 4 to the notion of capability. We define dormant capability in terms of computational accessibility, that is, the model’s ability to use the information already available to it [275–291]. The larger debate over what counts as genuine knowledge or capability remains unsettled, and it is beyond the scope of our paper. Whether random-reward training improves general mathematical reasoning capability, which in itself is not properly defined, is likewise beyond the discussion of spurious rewards. We stay agnostic on this question, as laid out in Section 6 [473–482]:

“It could be a sign that, after a period of training, LLMs begin to bring their knowledge and skills together and show early signs of self-improvement […] The mechanism could also be superficial. The model may simply learn a better decoding strategy that generates correct responses more reliably. While shallow, such a strategy is certainly a capability; it does not, however, by itself establish the consistency across equivalent expressions that we would expect of ‘knowledge’ (Niu et al., 2024).”

We are very confused by the assertion that Tables 16 and 22 give evidence that the formatting extractor used in the main experiments is invalid. Both tables are diagnostics that we ran and reported ourselves, and neither shows that the gains are an artifact of the extractor. To refresh your memory, some frozen models answer the question and then, instead of stopping, go on to generate another question until they reach the token limit; the extractor can then select a number from that continuation, and a correct answer is scored as incorrect [2017–2019]. Tables 16 and 22 re-score the outputs with rules that ignore the continuation. Table 16 contains frozen scores only, for the additional SmolLM2 and OLMo-3 checkpoints of Appendix E.3: it shows that some frozen scores are lowered by continued generation, but it has no post-training scores and therefore cannot speak to the gains. Table 22 applies one rule both before and after training, to two checkpoints trained on identical rollouts and identical random rewards:

Response sourceLearnerPrimary score (before → after)Prefix score (before → after)Prefix gain > 10 ppPrefix gain < −10 pp
P1P12.5 → 3.141.5 → 28.82/1610/16
P1P213.5 → 18.014.5 → 38.48/160/16
P2P12.5 → 5.441.5 → 21.21/1611/16
P2P213.5 → 27.214.5 → 39.38/160/16

Under this rule, P2 gains about 24 points with either response source, which is more than it gains under the primary scorer (4.5 and 13.7 points), while P1 declines. A formatting artifact of the extractor does not explain this difference, since the extractor, the training responses, and the rewards are all held fixed and only the learner differs. This is the control that the two other reviewers, @piFx and @XEB8, both single out as a strength: in @piFx’s words, “the number of accidentally rewarded correct answers cannot fully account for the result.” Finally, a model that gives the correct answer and then keeps generating has not completed the task as instructed, and learning to stop is part of the decoding capability that we discuss above.

Q2: Variance and Statistical Tests

Our results are not caused by noise or variance: the two findings that our account rests on are statistically significant, and the three-stage pattern is clear.

The passage that the reviewer cites as our confirmation, [2190–2194], reports a significant difference between two conditions. To refresh your memory, with random tokens inside the math template, 16 of 32 runs change by more than ten points in either direction; with random tokens alone, none of 32 does (p = 3.05 × 10⁻⁵; 4.27 × 10⁻⁴ after Holm correction). The same holds for the learner. Trained on the same P1 rollouts with the same random rewards, the P2 checkpoint changes by 13.56 points on average and the P1 checkpoint by 1.75, a difference that passes Holm correction among the six tests we specified in advance (Holm p = 0.0443) [2297–2304]. Noise alone would not distinguish one input from another, or one checkpoint from another.

The three-stage pattern does not rest on any single run. Under GT rewards, no run at the first three checkpoints exceeds 8.5%, and at the end of pre-training the four runs end at 61.5–90.5%. Under random rewards, none of the 200 P1 runs ends at or above 50%, compared with 15 of 96 P2 runs [1543–1544]. The orange line moves within P2 because it is a maximum over eight seeds at each checkpoint, and we state in the main text that this response is not stable across mid-training [414–416]. The stage bands mark phases of training rather than individual checkpoints, which is why the weak +26B checkpoint on code sits inside the autodidactic band of Figure 7a [1790–1794].

We also do not “report the maximum everywhere.” Every stage in Section 5 is defined by mean gains: 0.4 and 0.5 points for the dormant checkpoint, 79.5 against 4.3 for the receptive one, and 26.8 for the autodidactic one [408–416]. The stage definition uses the expected gain explicitly [422–423]. The figures plot the maximum together with every individual seed, and all 444 terminal scores are in Tables 8 and 9. As @piFx notes, “the appendix reports failed seeds, intermediate and terminal scores, and statistical tests.”

Q3: Spurious Rewards as a Probe

We believe that reachability is a major advance in the probing debate. The debate has moved from whether a probe has “just learned the task” (Hewitt & Liang, 2019) to accounts of decodability, which ask what can be recovered from a representation and under what conditions (Pimentel et al., 2020; Pimentel & Cotterell, 2021; Xu et al., 2019; Hewitt et al., 2021) [136–143]. The probes that these accounts describe still largely rely on supervised learning from task labels, applied to a frozen model [131, 144–146]. Reachability measures what the model itself can attain under further training, and random rewards let us measure it without telling the model which answers are correct (Theorem 1). Of course, this debate is far from being settled, but it now has a probe that tests the model’s own performance and gives it no correctness feedback, and this is what we mean by addressing it [030–033].

As for utility, our contribution is a scientific one, and we believe its value is considerable. Probing has long been a central tool for studying what models know and what they are capable of [129–130]; it is a scientific instrument, and we see no reason to judge reachability by a different standard. Our work offers a new explanation for the spurious-reward paradox, one that accounts for two observations that the existing explanations do not cover (Section 3). It also shows what neither current accuracy nor correctness-rewarded RL can show. Two checkpoints with the same accuracy (3.5%) reach 8.5% and 55% in their best runs under the same correctness-rewarded RL [020–022]; and GT rewards are effective in both the receptive and the autodidactic stage, so only random rewards distinguish the two (Eq. 7). How a model will respond to post-training, and when it begins to improve without being told the answers, are questions that anyone who trains models should want answered, at a time when post-training plays an ever more important role in developing LLM capabilities [268–269].

Minor Points

  • Training and evaluation data in Section 3. Thank you for the suggestion. We will state at the start of Section 3 that these experiments use synthetic GSM-style arithmetic problems and are evaluated on 200 held-out questions, with a pointer to Appendix D.1.
  • ∆GT ≥ ∆rand [449]. Thank you for catching this. The ordering is one we observe empirically; Section 4 does not prove it. We will correct the wording.

I thank the authors for their response. It clarifies several points, and I acknowledge that some statements in my original review were overstated. However, I do not think my main concerns are caused by misunderstandings, but by disagreements about the novelty and methodology:

  • While one could argue that following formatting constraints is a dormant capability, it is, to me, a very uninteresting one, and not the capability intended to be measured by the used benchmarks. By using those benchmarks, the authors implicitly communicate that performance on the task improves, but the presented numbers do not establish that.
  • I did not claim that formatting constraints were the only cause of the observed gains. The large swings presented in the appendix based on simple changes in the extractors are cause for concern, and explicitly acknowledged by the authors themselves in the appendix ("The scoring diagnostics show that output conventions can materially affect both initial scores and measured gains.").
  • It is still unclear what the probe adds beyond a thorough evaluation of the pretrained model using the suggestions I made in my review. The value of the methodology as a probe is therefore unclear. I am not saying that doing so would fully remove all observed gains, just that the paper currently does not provide sufficient evidence of what the probe adds beyond such an evaluation.
  • While I should not have claimed that the maximum is used everywhere, it is still displayed prominently. Had the authors used the mean (or even the median) in, for instance Figure 6, the increases in the orange line would have looked much less dramatic. Using the maximum therefore significantly biases the interpretation of the results.
  • I should not have stated that most improvements seem to be caused by variance. However, using the maximum when variability across seeds is large, as observed at some later checkpoints, can bias the interpretation of the results by emphasizing unusually successful runs. I did not claim that the results were not statistically significant.

Finally, I felt the need to point out that I would be unwilling to change my score to a 4, since I personally like to keep an open mind during the rebuttal. In this case, I felt an exceptional rebuttal could potentially have raised my score to a 3, but felt there are fundamental weaknesses that would prevent me from raising my score further. In the rebuttal, the authors have not sufficiently addressed my concerns. However, I do acknowledge that some of my concerns were overstated (as pointed out in my reply here) and I will therefore raise my score to a 2.

Reviewer_XEB8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes random-reward reinforcement learning as a probe of model reachability, defined as the performance attainable under specified adaptation procedures. It investigates checkpoints across pre-training and mid-training, primarily using OLMo-2, and compares correctness-rewarded and random-rewarded adaptation on synthetic arithmetic, code generation, and compositional lookup tasks. Shared-rollout experiments examine how different learners respond to identical training responses and rewards, while input-control experiments investigate the influence of task cues. The paper also establishes verifier invariance for training procedures whose inputs and decisions are independent of correctness feedback. The empirical results suggest that responsiveness to further training changes across checkpoints.

Critical strengths
  1. The shared-rollout experiment provides a useful control for separating learner state from training responses. By training different checkpoints on identical cached rollouts and random rewards, the authors demonstrate that the initial learner state can influence the response to the same training material. Table 1 reports larger mean gains for the later checkpoint under both rollout sources. Although the statistical evidence for these differences is limited after multiple-comparison correction, this experiment provides a useful perspective beyond explanations based solely on rollout correctness or the accidental reinforcement of correct responses.

  2. The paper provides extensive experimental results and a useful conceptual framework. The checkpoint scan covers multiple training stages and tasks, while the appendix reports individual seed outcomes, mean gains, statistical comparisons, and several important limitations. The verifier-invariance formulation also clearly distinguishes correctness information supplied through the reward channel from task-relevant information already present in the model or training inputs. This distinction is valuable for studying what model adaptation can reveal about existing capabilities.

Critical weaknesses
  1. The main experimental presentation emphasizes best-case performance, which is not fully aligned with the formal definition of reachability. The paper defines reachability as the supremum of expected performance over a family of admissible adaptation procedures, whereas the experiments primarily investigate a fixed random-reward training procedure. Moreover, several central figures (Figures 1, 6, and 7) emphasize the maximum terminal accuracy across random seeds, which estimates neither the expected performance of the fixed procedure nor the supremum over admissible procedures. The GT and Random conditions also generally use unequal seed budgets (four versus eight), making their maxima difficult to compare directly.This distinction is particularly important given the substantial variance and frequent performance degradation observed under random-reward training. For example, in the input-control experiment (Table 17), random-token inputs with mathematical instructions achieve a maximum accuracy of 68.5%, but a mean terminal accuracy of only 17.50%, below the frozen accuracy of 20.5%. In contrast, random-token-only inputs yield a mean of 20.36%. Furthermore, only 4 of 32 runs in the instruction-retained condition achieve improvements exceeding ten percentage points. These results provide stronger evidence that high-performing training trajectories can occasionally be attained than that random-reward adaptation reliably produces positive expected gains. Although the appendix acknowledges these limitations, the main presentation may overstate the effectiveness and stability of random-reward adaptation. The authors should more clearly distinguish optimal expected reachability, the expected gain of the evaluated procedure, and best-of-N attainability, and align their empirical conclusions with the corresponding quantities.

  2. The proposed three-stage developmental pattern is weakened by the instability and non-monotonicity of random-reward training. The paper identifies Dormant, Receptive, and Autodidactic stages based on responsiveness to GT and random rewards. However, the random-reward results fluctuate considerably across checkpoints and random seeds. For example, at OLMo-2 P2 +5B, random-reward training achieves a maximum accuracy of 77.5% and a mean improvement of 26.8 percentage points, but only four of eight runs achieve large gains. At P2 +50B, the same training procedure instead decreases mean accuracy. Across the twelve P2 checkpoints, mean random-reward gains range from -11.8 to +26.8 percentage points, with only three checkpoints exhibiting mean gains above ten points. These observations suggest that the reported Autodidactic stage is not a consistently attainable or monotonically developing property of the model. The evidence may support the emergence of high-performing trajectories under certain conditions, but it is less clear whether it supports a robust developmental stage characterized by positive expected improvement. The authors should more carefully distinguish the existence of successful training trajectories from reliable responsiveness to random-reward adaptation, especially when interpreting the three-stage framework as a general pattern of capability development.

  3. The theoretical contribution appears limited. The verifier-invariance result follows directly from the assumption that the training inputs, rewards, and update decisions are independent of the verifier. While this formalization clarifies the absence of correctness feedback through the reward channel, it does not explain why random-reward adaptation can improve task performance or establish that such improvements arise from pre-existing dormant capabilities. In particular, the gradient analysis shows that the expected raw random-reward gradient is zero, but does not establish a positive expected improvement in task accuracy. Therefore, the theoretical results seem insufficient to substantiate the proposed explanation of the spurious-reward paradox.

  4. The practical necessity and incremental value of random-reward RL as a reachability probe remain unclear. If reachability is intended to characterize a checkpoint's potential for further adaptation, correctness-rewarded RL provides a more direct and practically relevant measurement, since actual RL post-training typically relies on informative rewards. Although random-reward RL offers a distinct perspective by removing correctness feedback, the paper does not sufficiently establish what additional useful information this probe provides beyond frozen accuracy and correctness-rewarded adaptation. In particular, the authors do not demonstrate whether random-reward reachability reliably predicts subsequent performance under correctness-rewarded RL, helps identify checkpoints with greater adaptation potential, or provides a more informative or efficient measure of dormant capabilities. Given the substantial instability of random-reward training, its value as a practical capability probe remains uncertain. The authors should clarify the intended use of this probe and provide evidence of its incremental utility over more direct alternatives, such as short-budget correctness-rewarded RL.

Questions for authors
  1. Since reachability is formally defined in terms of expected performance, could the authors report terminal mean gains, standard deviations or confidence intervals, and improvement probabilities alongside the maxima in the main developmental figures? Do the principal stage distinctions remain convincing when evaluated using these statistics, with matched seed budgets and reference coefficients?

  2. Given the substantial variation across seeds and the non-monotonic behavior across P2 checkpoints, what operational criterion distinguishes an Autodidactic checkpoint from one that merely has a small probability of producing a high-performing run? Does the proposed developmental pattern remain robust under alternative training budgets or random-reward configurations?

  3. Which reported gains remain under a consistent answer-extraction protocol applied to both starting and adapted models? Given the substantial differences between strict and lenient extraction and the unresolved transfer evaluation described in Appendix F, what conclusions can currently be drawn about genuine reasoning improvement rather than changes in output formatting or decoding behavior?

  4. What is the incremental value of random-reward RL over correctness-rewarded RL as a reachability probe? If the goal is to characterize a checkpoint's potential for further training, correctness-rewarded RL appears to provide a more direct and practically relevant measure. Could the authors demonstrate whether random-reward reachability provides additional predictive information about subsequent adaptation performance, beyond frozen accuracy or short-budget correctness-rewarded RL? Alternatively, if the probe is intended primarily for scientific analysis rather than practical checkpoint selection, what specific aspect of dormant capability can it reveal that correctness-rewarded adaptation cannot?

1 comment

We are grateful for such a detailed review, and we are glad that you find our paper to provide “extensive experimental results and a useful conceptual framework.”

That said, we believe that the paper already answers most of the concerns raised here, and that none of them changes the soundness of our results. The three stages are stated with mean gains [408–416], and the finding is unchanged when the seeds are matched.

Please see below for our point-wise responses.

W1 & Q1: Maxima, Means, and Seed Budgets

The distinction between optimal expected reachability, the expected gain of the evaluated procedure, and best-of-N attainability is the one that our paper draws. We do not estimate the first (Eq. 3). The stages are stated with the second, as mean gains [408–416]. The figures show the third together with every seed, and Appendix D.7.1 says that it “estimates neither mean performance nor a capability ceiling.” We read the input-control experiment the same way: its statistical finding is greater responsiveness, in both directions, as we state [2192–2194]. We will bring the three-way distinction into the main text and add the mean to Figures 1, 6 and 7.

For Q1, by stage, over all 444 GSM runs of Tables 8 and 9:

StageGT: mean gain (pp)Random: mean gain (pp)Random: gain > 10 ppRandom: loss > 10 ppRandom: ending ≥ 50%
Dormant (P1 5B–462B)+0.8+0.10/240/240/24
Receptive (P1 839B–3,896B)+58.4+1.58/1760/1760/176
Autodidactic (P2)+76.0+3.722/9633/9615/96

With Random restricted to the four seeds that GT uses, 0 of 100 P1 runs and 8 of 48 P2 runs end at or above 50%. GT and Random use reference coefficients of 0 and 0.01 (Table 2); the stage distinctions compare the same protocol across checkpoints, so they do not depend on this difference.

W2 & Q2: Stability of the Third Stage

We state in the main text that this response is not stable across mid-training [414–416]. The third stage marks the emergence of large gains under random rewards, as the abstract puts it: “even random rewards can produce large gains” [026]. We do not claim a positive expected gain at every mid-training checkpoint, and we will make this explicit around Eq. 7. The emergence is tied to the phase of training: no P1 run ends at or above 50%, while seven of the twelve P2 checkpoints have at least one such run, and the same ordering appears under number-masked SFT (Figure 6b) and in OLMo-3. On Q2, Eq. 7 summarizes an ordering across phases schematically [424], and we do not offer it as a rule for classifying single checkpoints. Our conclusions are tied to the tested protocol, although the exploratory sweep over 34 configurations at four P2 checkpoints also showed gains in about 11% of runs [1363–1366].

W3: Theoretical Contribution

This is the role that we intend for the theorem. It is a guarantee about the probe: whatever the model attains, the reward did not tell it which answers are correct [271–272]. The explanation of why the gains occur is empirical (Section 3). The gradient analysis shows a zero-mean raw gradient whose update distribution depends on the model and its inputs [993–994], which matches what we find: the established differences are in how much a checkpoint changes.

W4 & Q4: What Random Rewards Add

The probe is intended for scientific analysis. A gain under GT rewards mixes what the reward tells the model with what the model could already do; the random reward carries no correctness information (Theorem 1), so a gain under it cannot come from being told the answers. GT rewards are also effective in both the receptive and the autodidactic stage, so only random rewards distinguish the two (Eq. 7). Random rewards need no correct answer, moreover, which is what allows the input controls of Section 3. We make no claim that the response to random rewards predicts the outcome of subsequent GT RL.

Q3: Answer Extraction

All reported gains use the same extractor before and after training [2023]. Table 22 repeats the comparison with the prefix rule: P2 moves from 14.5% to 38.4% and 39.3%, more than under the primary scorer, while P1 declines from 41.5%. Six of the 15 high-scoring Random runs in Table 7 produce no answer marker at all, so the gains do not come from learning the requested format. On genuine reasoning improvement, we stay agnostic [473–482]: a more reliable decoding strategy is a capability in the sense of our paper. We draw no conclusion from the transfer evaluation, which remains unresolved [2879–2881].

Reviewer_piFxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why reinforcement learning with random rewards can sometimes improve LLM performance, and proposes using this effect to probe capabilities that may not be apparent from a model's current accuracy. The authors define reachability as the highest expected task performance attainable under a specified family of training procedures and budget, and use random-reward RL to study the model's response to updates without correctness feedback from rewards. Across OLMo-2 checkpoints, they observe three regimes: early checkpoints barely improve even with correctness-based rewards, later pre-training checkpoints respond strongly to those rewards, and some mid-training checkpoints also show large gains under random rewards. Shared-rollout experiments find different outcomes when the same responses and rewards are used to train different checkpoints. Input controls show that task-related cues in the prompt affect training behavior. Additional results cover coding, compositional lookup, and other model families.

Critical strengths

S1. The shared-rollout experiment is a useful control. P1 and P2 receive identical cached responses and random reward assignments, yet their training outcomes differ. The number of accidentally rewarded correct answers cannot fully account for the result.

S2. The longitudinal checkpoint study is useful. On synthetic GSM, GT-reward RL becomes effective during pre-training, while substantial random-reward gains appear at some mid-training checkpoints. The checkpoints share the same architecture and parameter count, allowing a clearer examination of how training response changes over time.

S3. The appendix reports failed seeds, intermediate and terminal scores, and statistical tests. The code and compositional-lookup experiments add useful context, showing different transition times across tasks.

Critical weaknesses

W1. The practical use of random-reward reachability remains unclear. The experiments reveal differences across checkpoints, but do not show how those differences could inform subsequent model training. The probe also requires repeated training runs and labeled evaluation. How could these results help choose checkpoints, allocate post-training compute, or select training strategies? What useful guidance would they add beyond frozen accuracy or short-budget GT-RL?

W2. The stronger gains under random-reward RL may reflect capabilities already present in the starting checkpoint, as the authors suggest. Prior work has also discussed explanations involving changes in output probabilities and training dynamics (e.g., Shao et al., 2026; Agarwal et al., 2025). Could the authors include pass@k results before and after random-reward RL for representative checkpoints, especially those with similar initial accuracy but different random-reward gains? Comparing pass@1 and pass@k would help assess whether training makes previously sampleable correct solutions more likely, or changes the range of problems for which correct solutions can be sampled.

W3. The accuracy results appear sensitive to answer extraction. In Appendix E.7, P1's initial score changes from 2.5% to 41.5% under prefix-based extraction, substantially affecting the P1/P2 comparison. The high-scoring random-reward runs also receive much lower scores under strict extraction (Appendix E.1). Could the authors examine these discrepancies in more detail, perhaps by checking representative outputs and applying a consistent numerical-answer scorer before and after training? This would help clarify how much of the reported improvement reflects better problem-solving performance.

W4. Reachability is defined in terms of expected performance, but Figures 6(a), 7, and 9 mainly show the maximum terminal accuracy across seeds. The best run can give a different picture from the typical outcome. Could the authors present the means and variability more prominently alongside these maxima, and clarify which statistic supports the proposed three-stage interpretation?

W5. Several figures compare maxima obtained with different numbers of training seeds. In Figure 6(a), each GSM checkpoint has four GT seeds but eight Random seeds. For MBPP in Figure 7(a), P1 has one GT and two Random seeds, while P2 has four and eight. A larger number of runs increases the chance of observing a high maximum. Could the authors provide matched-seed comparisons, or summarize mean performance and variability, to check whether the plotted trends are robust?

Questions for authors

Please see W1–W5 for my questions and suggestions.

1 comment

Thank you for the review! We appreciate that you find the shared-rollout experiment “a useful control” and the longitudinal checkpoint study useful as well.

However, we would like to respectfully point out that none of the weaknesses concerns the soundness of our results, and most of them are already addressed in the paper. Our three-stage finding is stated with mean gains [408–416], and it still holds when the seeds are matched.

We take this opportunity to clarify the practical use of the probe (W1), pass@k (W2), answer extraction (W3), and means, maxima and seed budgets (W4 & W5).

W1: Practical Use

Our contribution is a scientific one: a probe for studying how capabilities develop [129–130]. It adds to both baselines that you mention. Two checkpoints with the same frozen accuracy (3.5%) differ by 25.9 points in mean accuracy after the same GT RL, with all four seeds in the same direction [2566–2567]. GT rewards, for their part, are effective in both the receptive and the autodidactic stage, so only random rewards distinguish the two (Eq. 7).

W2: pass@k

We leave this analysis to future work, as stated in Section 6 [480–482]. Our account makes a prediction for it: pass@1 should rise much more than pass@k at large k, since correct solutions that could already be sampled become the ones that the model produces.

W3: Answer Extraction

Table 22 applies the same prefix rule before and after training: P2 gains about 24 points with either response source (4.5 and 13.7 under the primary scorer), while P1 declines from 41.5%. P1’s initial score depends on the scorer, as we report [2346–2350]; the contrast between the two learners holds under both. Six of the 15 high-scoring Random runs in Table 7 produce no answer marker at all, so the gains do not come from learning the requested format. Strict extraction requires the requested answer-line structure as well as a correct number [1555–1556], so a run that emits no marker scores zero under it by construction. On whether this reflects better problem-solving, we stay agnostic [473–482]: a more reliable decoding strategy is a capability in the sense of our paper.

W4 & W5: Means, Maxima, and Seed Budgets

The stages are stated with mean gains in the text [408–416]; the figures plot the maximum together with every seed. By stage, over all 444 GSM runs of Tables 8 and 9:

StageGT: mean gain (pp)Random: mean gain (pp)Random: gain > 10 ppRandom: loss > 10 ppRandom: ending ≥ 50%
Dormant (P1 5B–462B)+0.8+0.10/240/240/24
Receptive (P1 839B–3,896B)+58.4+1.58/1760/1760/176
Autodidactic (P2)+76.0+3.722/9633/9615/96

The first transition is a change in the mean GT gain. The second is that large gains under random rewards become attainable; the mean Random gain stays small because large losses also appear, as we state in the main text [414–416]. With Random restricted to the four seeds that GT uses, 0 of 100 P1 runs and 8 of 48 P2 runs end at or above 50%. On code, all 16 P2 GT runs score at least 16.5%, while no P1 GT run exceeds 11.0% [1785–1789]; the Random result there is weaker, as we note [1790–1794]. We will add the mean to Figures 6, 7 and 9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.