acceptodds
Under review as a conference paper at ICLR 2027

DEMYSTIFYING REINFORCEMENT LEARNING POST- TRAINING OF LANGUAGE MODELS

Abstract

Reinforcement learning (RL) post-training has emerged as a powerful framework for enhancing the capabilities of large language models (LLMs), enabling impressive reasoning, math, and coding capabilities. Yet for many researchers and practitioners, the principles behind classical RL remain a “black box”. In this work, we deconstruct the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model’s prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale. We use the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shapes model certainty. Our investigation sheds light on how these choices interact to affect post-training success. For example, we show that the effect of so-called `spurious rewards' depends on the prompt distribution used for post-training. We also provide insight into why the success of RL post-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning Post-Training, Large Language models, Pre-Training Distribution, Reward Sparsity, RLVR, Natural Language Processing, Spurious Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d8zROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how RL post-training depends on initialization, reward informativeness, and the training prompts. Positive or negative fine-tuning raises or suppresses the initial probability of a target quote or a solution to one AIME problem. The authors then compare sparse outcome rewards with edit-distance or process-based shaping. They also train Qwen and OLMo models with independent random rewards on narrow and broad prompt sets. Shaping substantially improves success on some initially rare targets. Random-reward training produces different entropy and benchmark trajectories across prompt sets and model families. The paper uses these experiments as a primer on the mechanisms and limits of RL post-training.

Critical strengths
  1. The Base, SFT+, and SFT− interventions make a commonly discussed dependence on initialization experimentally concrete. Applying sparse and shaped rewards to the same targets also permits a direct comparison: Qwen3-1.7B quote success reaches 48.8% with edit-distance shaping and 10.0% with sparse reward; on the AIME target, the base model reaches 92.2% with the custom process reward and 10.2% with sparse reward (Table 2; Figure 8). These examples could help practitioners distinguish a sampling problem from an uninformative reward signal.
  2. Tracking entropy, fixed-response likelihood, and several benchmark families reveals effects that task reward alone would miss. In particular, greater certainty can accompany worse performance, and narrow OLMo training damages GSM8K while largely preserving MMLU and IFEval. Characterizing where such changes occur is a useful empirical direction, even when the underlying RL principles are familiar.
Critical weaknesses
  1. The paper does not establish how much its conclusions depend on the RL update rule. Figure 1 and the limitations explicitly exclude algorithmic variation. Holding one algorithm fixed permits controlled comparisons within that setup, but the claim that prompt distribution entirely determines spurious-reward effects reaches beyond this evidence.
  2. For response-independent random rewards, an ordinary on-policy reward gradient has zero expectation. Sampling common responses more often therefore does not explain systematic sharpening or movement toward a uniform distribution. With a fixed-reference KL penalty, the exact expected objective favors the reference policy. Clipping, advantage normalization, optimizer dynamics, regularization, or rollout reuse could drive the observed changes. Shao et al.'s Spurious Rewards already identifies a GRPO clipping bias, making these details relevant to the paper's proposed explanation. The 100-versus-10,000 prompt comparison also changes repetition. A simplified analysis of the actual update and targeted controls are needed to distinguish these effects; otherwise the causal claims should be narrowed (Sections 4.3 and 5.3; Appendices D and E).
  3. The shaping experiments establish better optimization on the trained targets, but do not establish the claimed capability generalization. Successful Base models already have nonzero observed target probability: 0.48%0.48\% for the quote and 3.92%3.92\% for the AIME answer. The AIME reward encodes milestones of one known solution, and evaluation uses that same problem. Initializations with zero observed successes still fail at exact matching under both rewards, and zero empirical successes do not establish zero probability (Tables 1-2). The practical lesson about shaping is useful, but the evidence does not yet show which conclusions extend to unseen reasoning problems. Claims about complex generalization and escaping pass@k limitations therefore need narrower wording or a separate transfer test.
Questions for authors
  1. Please specify the RL loss, advantage estimator and normalization, clipping, KL/entropy terms, optimizer, and rollout reuse for each experiment. Which part of the update explains the direction of entropy change under random rewards, given the zero expected reward gradient for an ordinary on-policy estimator? Can a simplified analysis and targeted ablations, such as removing clipping or changing normalization, identify the cause? Please include zero/constant-reward controls and report uncertainty across training seeds. When testing algorithm dependence, match initialization, prompts, and training budget; when testing prompt breadth, also match prompt count and repetition.
  2. Is the intended contribution improved optimization of rare supported behaviors, or transfer to untrained reasoning problems? Please tie the claims to the corresponding evidence; if transfer is claimed, evaluation on untrained problems would be needed. Please also reconcile Appendix B.1's sparse-reward equation with its description: the strict length condition appears to give zero reward to an exact-length target string.
2 comments

We thank the reviewer for their careful reading of our paper, for highlighting the value of our Base/SFT+/SFT- interventions and cross-benchmark entropy tracking, and for catching the equation typo in Appendix B.1. Below we address each of your comments and detail the corresponding revisions made to the uploaded PDF. Regarding Critical Weakness 1: Although we hypothesize that our high-level conclusions are largely invariant to the specific optimization algorithm used, as long as it can successfully optimize the reward signal given the prompt and base model distributions, we agree that our empirical evidence is collected holding the optimizer family fixed, and that our claims should reflect this. In addition to correcting the Conclusion to state that spurious-reward outcomes depend on the interaction between the prompt distribution and the base policy prior, we have expanded the Limitations section to explicitly acknowledge that we only tested standard clipped policy-gradient / GRPO updates, and that algorithmic details (such as clipping and normalization) interact with these dynamics. Regarding Critical Weakness 2 & Question 1: This is a very nuanced point. To clarify our setup (now detailed in Appendices D and E): our spurious-reward experiments follow the OlmoRL / GRPO setup (and Shao et al., 2025), drawing uniform positive rewards rUniform[0,1] across G=8 on-policy rollouts per prompt (async_steps = 0) with a clipped policy-gradient objective and KL regularization. Why do uniform random rewards alter policy entropy if an unclipped, infinite-sample on-policy gradient has zero expectation? Finite-sample sampling frequency & update bias on Narrow D (100 prompts): While the analytical expectation over the entire space of all possible token sequences sums to zero, training in practice operates over small finite rollout batches (G=8 samples per prompt) repeated over many epochs on the 100 narrow prompts. Because responses with high prior probability under the base model are sampled far more frequently in finite batches, they are the ones actually realized in the update buffer and accruing the positive reward signals, while unsampled tail sequences are suppressed via softmax normalization. Combined with the PPO/GRPO clipping bias identified by Shao et al. (2025) and the high repetition of the 100 prompts, the update rule reinforces the base model's dominant modes, tightening entropy. Broad D: Across 10,000 diverse prompts, individual prompts are rarely repeated, preventing prompt-specific mode sharpening. Instead, applying random finite-sample updates across shared parameters over thousands of distinct contexts acts as high-variance parameter noise that flattens token distributions globally, increasing entropy across benchmarks (GSM8K, MMLU, IFEval; Figure 4a–b). We have updated Section 5.3, and Appendix E to explicitly discuss how finite-sample updates, clipping dynamics, and prompt repetition drive these effects. Regarding adding zero/constant-reward controls, clipping ablations, and multi-seed uncertainty for random rewards: we agree these are interesting directions for isolating internal GRPO biases, but we view that as the specific focus of the "Spurious Rewards" paper (Shao et al., 2025). Our intention here is not to redo the entire spurious rewards study or dissect GRPO internals, but rather to show that their main empirical finding is confined to a highly restricted regime (a narrow prompt distribution paired with a strongly aligned base model prior) and does not reflect general RL post-training dynamics.

Continued in additional comment below....

Regarding Critical Weakness 3 & Question 2: Neither improved optimization of rare behaviors as an algorithmic benchmark nor cross-problem transfer to unseen reasoning problems is the primary intended contribution of our paper. Rather, the intended contribution of this paper is to use simplified, controlled settings to clearly illustrate how RL post-training dynamics work, justified in terms of well-known behavior in classical RL, in order to correct widespread misunderstandings in the current LLM post-training literature (e.g., that RL cannot go beyond pass@k or that random rewards generally improve reasoning). With that said, we completely agree with you that our single-problem AIME and movie-quote experiments do not prove cross-problem generalization, and that zero empirical successes in 10,000 samples does not establish a true mathematical probability of zero. In the revised PDF, we have replaced phrases like "zero support" and "absent from the reference policy" with "empirically undetected (0/10,000 samples)" and "negligible empirical support under finite sampling budgets". Additionally we have rephrased Section 5.2 and the Conclusion to acknowledge that our experiments are toy and provide illustrative evidence of how dense rewards allow LLMs to chain together solutions to intermediate sub-problems into complex long-horizon behaviors that, within this limited setting, go beyond the base model's initial sampling capabilities. We also added a brief discussion of how, in a more general setting like post-training against a Lean theorem checker, this mechanism could allow an LLM to generalize from rewards on simpler lemmas to chain them together on harder, unseen theorems, as we hypothesize occurs in recent frontier reasoning models. Regarding Question 2: Thank you for catching this typo! In our implementation, an exact-length match (∣y∣=∣τ∣\vert{}y\vert{} = \vert{}\tau\vert{}, where excess tokens s=0 and penalty p=0) receives the full reward of 1.0. The strict inequality ∣y∣>∣τ∣\vert{}y\vert{} > \vert{}\tau\vert{} in Equation (4) (line 759) was a typographical error and has been corrected to ∣y∣≥∣τ∣\vert{}y\vert{} \geq \vert{}\tau\vert{} in the revised PDF.

Thank you again for your thoughtful feedback. We hope our revisions and clarifications have adequately addressed your concerns!

Reviewer_fxiBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper, Demystifying Reinforcement Learning Post-Training of Language Models, uses controlled experiments to investigate how base model prior, reward structure, and prompt distribution shape the performance of LLM reinforcement learning post-training. Its core contributions uncover sparse RL’s dependence on initial coverage, the potential of dense rewards to mitigate exploration bottlenecks, and how spurious rewards interact with training distribution and model priors. While the experimental results offer valuable insights, the evidence is insufficient to support strong claims regarding new capability acquisition, strict coverage thresholds, and spurious rewards.

Critical strengths

S1: Many counterintuitive phenomena observed in LLM RLVR can be explained through the lens of exploration and probability redistribution from classical reinforcement learning. Reward density and prompt sampling distribution exert substantial influence. This constitutes a sound research perspective.

S2: The impact of random rewards cannot be decoupled from the prompt distribution and base policy, which is a reasonable finding.

S3: Sparse RLVR exhibits high sensitivity to exploration dynamics. While this aligns with conventional intuition, it is not a novel observation.

Critical weaknesses

W1 — Insufficient evidence for generalizable claims (primary concern) The paper demonstrates that dense rewards improve learning of low-probability target behaviors using a fixed string-generation task and a single AIME math problem. However, the AIME experiment relies on five manually designed PRM milestones tailored to the correct solution trajectory, and cross-problem generalization is not evaluated. Furthermore, the base model already achieves a 3.92% accuracy prior to training. Therefore, while the experiments support the claim that dense rewards alleviate exploration bottlenecks, they do not suffice to prove that RL can generally create novel reasoning capabilities absent in the base model.

W2 — Misinterpretation of zero support The authors equate the absence of target behavior in finite samples with a true probability of zero. Zero occurrences across 10,000 samples only indicate an empirical frequency of zero for the target behavior and do not establish that the base model assigns a strict zero probability to it. Accordingly, phrasing such as zero support and absent from the reference policy lacks rigor; these should be revised to empirically undetected or negligible support.

W3 — Unsupported strict threshold claim From a small set of prior values across different base models, the authors argue for a strict threshold in sparse RL. These results, however, only demonstrate high sensitivity of RL to initial coverage and cannot confirm the existence of a sharp critical threshold. Architectural and pretraining differences between models may also confound the outcomes. A more precise conclusion would state strong exploration dependence under finite training budgets.

W4 — Overclaim regarding prompt distribution The conclusion claims that the effect of spurious rewards depends entirely on the prompt distribution. Yet the paper’s own experiments show that a narrow prompt distribution boosts Qwen’s math performance while degrading OLMo’s GSM8K results. This indicates outcomes are governed by the interaction between prompt distribution and base policy. The conclusion stands in direct contradiction to the analysis in Section 5.3.

Questions for authors

Several other noteworthy issues remain. The universal mechanism for entropy increase brought about by random rewards lacks sufficient proof. In addition, interpreting accuracy improvements on one single training problem as complex generalization is an inference unsupported by evidence.

1 comment

We thank the reviewer for their thoughtful assessment of our paper, for highlighting the value of connecting classical RL exploration and probability redistribution to counterintuitive LLM RLVR phenomena (S1–S3), and for providing clear, constructive feedback on where our claims and terminology needed tightening. We agree with all four of your critiques and have uploaded a revised PDF incorporating every change detailed below.

Regarding W1: We agree that our controlled experiments are simplified and provide initial evidence in this direction rather than conclusive proof that RL can generally create novel reasoning capabilities across arbitrary domains. We have revised Section 5.2 and the Conclusion to state more precisely that our results show how dense rewards can lead LLMs to chain together solutions to sub-problems to learn complex, long-horizon behaviors that, in this limited setting, go beyond the base model's initial sampling capabilities. In addition, we have added a discussion in Section 5.2 noting how this mechanism could scale to a more general, complex setting, such as post-training for formal theorem proving against a Lean proof assistant, where an LLM may generalize from verifiable rewards on simpler lemmas and chain them together to solve harder, unseen theorems

Regarding W2 : You are completely right that zero occurrences across 10,000 finite samples indicate an empirical frequency of zero rather than a true mathematical probability of zero. We have revised the manuscript throughout to replace phrases like "zero support" and "absent from the reference policy" with "empirically undetected (0/10,000 samples)" and "negligible support under finite sampling budgets."

Regarding W3: We agree that comparing a small set of initial priors across different base models demonstrates strong sensitivity to initial coverage rather than a sharp critical threshold, especially given potential architectural and pretraining confounds across models. We have removed "strict threshold of sparse RL" in Section 5.1 and replaced it with your suggested phrasing: "strong exploration dependence under finite training budgets."

Regarding W4: Thank you for catching this. That phrasing in the Conclusion was an oversight on our part that contradicted our analysis in Section 5.3 as well as our framing in the Introduction, where we note that these phenomena are governed by "the relationship between the model's pre-training distribution, the prompt distribution used for post-training, and the reward function". We fully agree that the outcomes of spurious rewards are governed by the interaction between the prompt distribution and the base policy. We have updated the Conclusion, along with the Abstract, to explicitly state: "Furthermore, we show that the effect of 'spurious rewards' is governed by the interaction between the post-training prompt distribution and the base policy prior."

Regarding Questions for Authors (Entropy mechanism & single-problem generalization): As noted under W1, we have narrowed our claims on the AIME experiment so that accuracy gains on a single training problem are framed as within-problem trajectory synthesis rather than cross-problem generalization. To clarify the mechanism driving entropy changes under random rewards (Section 5.3 and Appendix E): while an unclipped, unnormalized on-policy reward gradient has zero expectation under independent random rewards, standard GRPO/PPO updates normalize advantages across the G=8 sampled rollouts per prompt and apply ratio clipping. Under a narrow prompt distribution (100 prompts with high repetition across training steps), repeated finite-sample updates on the exact same prompts amplify high-probability modes of the base policy, reducing entropy. Under a broad distribution (10,000 prompts with minimal repetition), applying random positive and negative advantage updates across shared weights over thousands of distinct contexts acts as high-variance parameter noise, flattening token distributions and increasing entropy across domains.

We thank the reviewer again for their careful reading and constructive suggestions, which have substantially improved the message of our paper.

Reviewer_H6kKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the underlying mechanics of Reinforcement Learning (RL) post-training for Large Language Models (LLMs). The authors attempt to demystify RL post-training by conducting ablation studies in a controlled, simplified environment (sequence generation and a single AIME math problem). They focus on three main components: the base model's prior distribution, the granularity of the reward signal (sparse vs. dense), and the prompt distribution. The paper argues that sparse rewards fail without sufficient base model coverage, dense rewards can teach entirely new behaviors (overcoming the pass@k limitation), and the phenomenon of "spurious/random rewards" improving model capabilities is strictly an artifact of using a narrow prompt distribution coupled with a strong base model prior.

Critical strengths

None.

Critical weaknesses
  1. Toy Experimental Setup and Lack of Generalizability

The paper claims to "demystify" the underlying mechanics of RL post-training for LLMs, yet its core conclusions rely heavily on highly simplified, "toy" experimental setups. Specifically, the complex mathematical reasoning experiments are based entirely on a single problem (AIME 2025 Problem 4), and the sequence generation task is restricted to reproducing a simple movie quote. Real-world LLM post-training deals with highly complex, diverse, and large-scale data distributions. Deriving generalized, fundamental principles of RL dynamics from a single math problem lacks statistical significance and fails to capture the true complexities of training frontier models.

  1. Trivial Findings Regarding Sparse vs. Dense Rewards

The paper’s first major conclusion—that sparse rewards fail when the initial probability of generating a trajectory is near-zero, whereas dense rewards enable exploration—is a well-established consensus in classical RL literature rather than a novel discovery. It is self-evident that if a model cannot reach the final goal even once during exploration, sparse rewards will provide absolutely no gradient signal (the classic exploration bottleneck). Dense rewards merely decompose a difficult, long-horizon goal into a chain of manageable intermediate sub-goals, allowing the model to learn progressively. Framing this basic RL dynamic as a significant "demystification" of LLMs overstates the contribution and offers little new insight to the community.

  1. Flawed Motivation and Illogical Explanations for the "Random Rewards" Phenomenon

The paper dedicates significant effort to exploring why models can improve using random rewards under narrow prompt distributions, but both the motivation and the mathematical explanation are highly questionable:

Lack of Meaningful Baselines and Motivation: Training a model with random noise supervision is inherently nonsensical in practice. If a model collapses under broad random rewards, it is simply "working as intended." The authors attempt to show that on narrow domains where the model is already strong, random rewards do not cause a collapse. However, they fail to provide a proper baseline comparison against using correct rewards. If random rewards are strictly inferior to correct rewards, investigating this extreme edge case holds little practical or theoretical value.

Mathematical Inevitability Rather Than a "Phenomenon": The random rewards used in the study are positive scalars drawn uniformly from [0,1][0, 1]. This means that, in the worst-case scenario, an incorrect response is still treated as "partially correct" rather than being explicitly penalized. If the base model (e.g., Qwen-Math) already possesses a high pass rate (e.g., >50%) on the narrow training data, these strictly positive random rewards will, more than half the time, attach to correct trajectories and provide directionally useful gradients. Therefore, the model's ability to continue learning under this specific reward construction and high initial accuracy is a mathematical inevitability—not a counter-intuitive mechanistic discovery.

Questions for authors

See above weaknesses.

1 comment

We thank the reviewer for their candid feedback. Overall, we completely agree with you that if a reader is coming from a classical RL perspective, the phenomena we empirically demonstrate in this paper should be obvious. However, as we attempt to show with the works we cite, they have not been obvious in the context of the recent LLM/NLP literature. We see our paper as carefully elucidating these foundational ideas for an NLP audience, and providing value as a tutorial and empirical primer for understanding classical RL concepts through the lens of RL post-training of LLMs.

Regarding Critical Weakness 2: We acknowledge that these findings are well-known in the classical RL literature (see lines 050–051). However, somewhat surprisingly, they are frequently misunderstood in the post-training literature, where a misconception has arisen that RL post-training cannot teach any new capabilities, but can merely improve the model's pass@k (Yue et al., 2025; Zhao et al., 2025; Rajani et al., 2025; Wu et al., 2025; Akgül et al., 2026). Our goal with this paper is to show, using simplified settings to make the learning dynamics extremely clear, exactly the type of insight you point out: how RL can allow chaining solutions to intermediate sub-goals into learning long-horizon behavior that goes beyond what the base model was originally capable of.

Regarding Critical Weakness 1: You are correct that our settings are toy compared to a full RL post-training setup, which we also acknowledge in the manuscript. We needed to use these simplified toy settings so that we can compute the exact probability of producing the desired behavior in the base model, and show how that affects post-training outcomes, i.e., to be able to provide Tables 1 and 2 in our paper (as well as Figures 5–9 in the Appendix). (We also complement these controlled setups in Section 5.3 with experiments on 10,000 prompts from the WildChat and OLMo 3 RLVR training mixes across standard benchmarks.) As we have stated above, we believe these analyses are useful as a tutorial for readers that are unfamiliar with classical RL but are attempting to come up to speed through reading the post-training literature.

Regarding Critical Weakness 3:

"Training a model with random noise supervision is inherently nonsensical in practice [...] the model's ability to continue learning under this specific reward construction and high initial accuracy is a mathematical inevitability—not a counter-intuitive mechanistic discovery"

We completely agree with you! And yet the paper "Spurious Rewards: Rethinking Training Signals in RLVR" (Shao et al., 2025), already cited 147 times, presented it as such. In fact, the response to the spurious rewards paper in online discussions was to conclude that many positive effects of RL post-training may be "spurious" in the sense that post-training "always helps" certain models (even if the rewards are random). Our work is an attempt to show that random rewards are indeed nonsensical, and in the general case of course do not improve models. We replicate the results of the original spurious rewards paper to show that they are confined to a highly specific setting, and not a true indication of how post-training dynamics work in the general case.

To quickly address the point about positive scalars in [0,1] and initial accuracy >50%>50\%: in standard RLVR pipelines, rewards across the 8 rollouts per prompt are normalized to have zero mean within each group, so roughly half of the sampled responses receive negative advantages and are actively penalized. Moreover, a >50%>50\% initial accuracy does not guarantee improvement under narrow random rewards: in Figure 4c, OLMo 3 SFT starts at 86% accuracy on GSM8K, and training on the narrow math prompt set with U[0,1] random rewards causes its GSM8K score to drop from 86% to 32%.

We thank the reviewer again for the constructive feedback which highlighted exactly where we need to modify our claims. To address these issues identified above we have made a number of changes to the text of our manuscript, which is highlighted in blue in our manuscript, to explicitly state the scope of our work and reduce ambiguity. We have explicitly framed our contributions around within-task exploration and credit-assignment mechanics rather than broad cross-task generalization. By grounding our controlled empirical results in classical RL fundamentals, specifically regarding finite-sample exploration bottlenecks and the actual mechanics of random rewards, we believe this work will serve as a much-needed clarifying primer for the NLP community. Thank you again for your review!

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

We thank all three reviewers for their constructive feedback. We are encouraged that the Reviewers highlighted the value of our Base/SFT+/SFT- initialization interventions, our multi-benchmark entropy tracking (GSM8K, MMLU, IFEval), and our perspective connecting classical RL exploration mechanics to counterintuitive phenomena in LLM post-training.

Our primary goal with this paper is to provide an empirical primer for the NLP community that clarifies widespread misconceptions in the recent LLM post-training literature, such as claims that RL cannot go beyond pass@k coverage or that random rewards genuinely improve reasoning. While these dynamics are intuitive from a classical RL standpoint, isolating them in LLMs requires pairing controlled sandboxes (where exact prior probabilities can be manipulated and measured, as in Tables 1–2) with production-scale prompt distributions (our 10,000-prompt OlmoRL and WildChat experiments in Section 5.3).

Based on the reviewers' suggestions, we have uploaded a revised manuscript (with changes highlighted in blue) that makes the following key improvements:

  • We reframed Section 5.2 and the Conclusion to state that our single-problem AIME and string-generation setups illustrate how dense rewards enable chaining intermediate sub-goals into long-horizon behaviors within a target setting rather than proving broad cross-problem transfer, while adding a discussion of how this mechanism applies to domains like formal theorem proving against a Lean proof assistant.

  • We replaced absolute terms like "zero support" and "strict threshold" with "empirically undetected (0/10,000 samples)," "negligible empirical support under finite sampling budgets," and "strong exploration dependence under finite training budgets".

  • We updated the Abstract, Conclusion, and Appendix A to consistently state that spurious-reward outcomes are governed by the interaction between the prompt distribution and the base policy prior. Expanded Algorithmic & Random-Reward Mechanics: In Section 5.3, and Appendix E, we detailed our GRPO/PPO update mechanics, clarifying how zero-mean group advantage normalization (G=8), finite-sample repetition (100 vs. 10,000 prompts), and clipping drive entropy sharpening versus global unlearning—and fixed the typographical inequality in Appendix B.1, Equation (4).

Thank You again for your comments and reviews, please see our individual responses below for point-by-point details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.