- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how RL post-training depends on initialization, reward informativeness, and the training prompts. Positive or negative fine-tuning raises or suppresses the initial probability of a target quote or a solution to one AIME problem. The authors then compare sparse outcome rewards with edit-distance or process-based shaping. They also train Qwen and OLMo models with independent random rewards on narrow and broad prompt sets. Shaping substantially improves success on some initially rare targets. Random-reward training produces different entropy and benchmark trajectories across prompt sets and model families. The paper uses these experiments as a primer on the mechanisms and limits of RL post-training.
Critical strengths
- The Base, SFT+, and SFT− interventions make a commonly discussed dependence on initialization experimentally concrete. Applying sparse and shaped rewards to the same targets also permits a direct comparison: Qwen3-1.7B quote success reaches 48.8% with edit-distance shaping and 10.0% with sparse reward; on the AIME target, the base model reaches 92.2% with the custom process reward and 10.2% with sparse reward (Table 2; Figure 8). These examples could help practitioners distinguish a sampling problem from an uninformative reward signal.
- Tracking entropy, fixed-response likelihood, and several benchmark families reveals effects that task reward alone would miss. In particular, greater certainty can accompany worse performance, and narrow OLMo training damages GSM8K while largely preserving MMLU and IFEval. Characterizing where such changes occur is a useful empirical direction, even when the underlying RL principles are familiar.
Critical weaknesses
- The paper does not establish how much its conclusions depend on the RL update rule. Figure 1 and the limitations explicitly exclude algorithmic variation. Holding one algorithm fixed permits controlled comparisons within that setup, but the claim that prompt distribution entirely determines spurious-reward effects reaches beyond this evidence.
- For response-independent random rewards, an ordinary on-policy reward gradient has zero expectation. Sampling common responses more often therefore does not explain systematic sharpening or movement toward a uniform distribution. With a fixed-reference KL penalty, the exact expected objective favors the reference policy. Clipping, advantage normalization, optimizer dynamics, regularization, or rollout reuse could drive the observed changes. Shao et al.'s Spurious Rewards already identifies a GRPO clipping bias, making these details relevant to the paper's proposed explanation. The 100-versus-10,000 prompt comparison also changes repetition. A simplified analysis of the actual update and targeted controls are needed to distinguish these effects; otherwise the causal claims should be narrowed (Sections 4.3 and 5.3; Appendices D and E).
- The shaping experiments establish better optimization on the trained targets, but do not establish the claimed capability generalization. Successful Base models already have nonzero observed target probability: for the quote and for the AIME answer. The AIME reward encodes milestones of one known solution, and evaluation uses that same problem. Initializations with zero observed successes still fail at exact matching under both rewards, and zero empirical successes do not establish zero probability (Tables 1-2). The practical lesson about shaping is useful, but the evidence does not yet show which conclusions extend to unseen reasoning problems. Claims about complex generalization and escaping pass@k limitations therefore need narrower wording or a separate transfer test.
Questions for authors
- Please specify the RL loss, advantage estimator and normalization, clipping, KL/entropy terms, optimizer, and rollout reuse for each experiment. Which part of the update explains the direction of entropy change under random rewards, given the zero expected reward gradient for an ordinary on-policy estimator? Can a simplified analysis and targeted ablations, such as removing clipping or changing normalization, identify the cause? Please include zero/constant-reward controls and report uncertainty across training seeds. When testing algorithm dependence, match initialization, prompts, and training budget; when testing prompt breadth, also match prompt count and repetition.
- Is the intended contribution improved optimization of rare supported behaviors, or transfer to untrained reasoning problems? Please tie the claims to the corresponding evidence; if transfer is claimed, evaluation on untrained problems would be needed. Please also reconcile Appendix B.1's sparse-reward equation with its description: the strict length condition appears to give zero reward to an exact-length target string.
2 comments
Regarding Critical Weakness 3 & Question 2: Neither improved optimization of rare behaviors as an algorithmic benchmark nor cross-problem transfer to unseen reasoning problems is the primary intended contribution of our paper. Rather, the intended contribution of this paper is to use simplified, controlled settings to clearly illustrate how RL post-training dynamics work, justified in terms of well-known behavior in classical RL, in order to correct widespread misunderstandings in the current LLM post-training literature (e.g., that RL cannot go beyond pass@k or that random rewards generally improve reasoning). With that said, we completely agree with you that our single-problem AIME and movie-quote experiments do not prove cross-problem generalization, and that zero empirical successes in 10,000 samples does not establish a true mathematical probability of zero. In the revised PDF, we have replaced phrases like "zero support" and "absent from the reference policy" with "empirically undetected (0/10,000 samples)" and "negligible empirical support under finite sampling budgets". Additionally we have rephrased Section 5.2 and the Conclusion to acknowledge that our experiments are toy and provide illustrative evidence of how dense rewards allow LLMs to chain together solutions to intermediate sub-problems into complex long-horizon behaviors that, within this limited setting, go beyond the base model's initial sampling capabilities. We also added a brief discussion of how, in a more general setting like post-training against a Lean theorem checker, this mechanism could allow an LLM to generalize from rewards on simpler lemmas to chain them together on harder, unseen theorems, as we hypothesize occurs in recent frontier reasoning models. Regarding Question 2: Thank you for catching this typo! In our implementation, an exact-length match (, where excess tokens s=0 and penalty p=0) receives the full reward of 1.0. The strict inequality in Equation (4) (line 759) was a typographical error and has been corrected to in the revised PDF.
Thank you again for your thoughtful feedback. We hope our revisions and clarifications have adequately addressed your concerns!
We thank the reviewer for their careful reading of our paper, for highlighting the value of our Base/SFT+/SFT- interventions and cross-benchmark entropy tracking, and for catching the equation typo in Appendix B.1. Below we address each of your comments and detail the corresponding revisions made to the uploaded PDF. Regarding Critical Weakness 1: Although we hypothesize that our high-level conclusions are largely invariant to the specific optimization algorithm used, as long as it can successfully optimize the reward signal given the prompt and base model distributions, we agree that our empirical evidence is collected holding the optimizer family fixed, and that our claims should reflect this. In addition to correcting the Conclusion to state that spurious-reward outcomes depend on the interaction between the prompt distribution and the base policy prior, we have expanded the Limitations section to explicitly acknowledge that we only tested standard clipped policy-gradient / GRPO updates, and that algorithmic details (such as clipping and normalization) interact with these dynamics. Regarding Critical Weakness 2 & Question 1: This is a very nuanced point. To clarify our setup (now detailed in Appendices D and E): our spurious-reward experiments follow the OlmoRL / GRPO setup (and Shao et al., 2025), drawing uniform positive rewards rUniform[0,1] across G=8 on-policy rollouts per prompt (async_steps = 0) with a clipped policy-gradient objective and KL regularization. Why do uniform random rewards alter policy entropy if an unclipped, infinite-sample on-policy gradient has zero expectation? Finite-sample sampling frequency & update bias on Narrow D (100 prompts): While the analytical expectation over the entire space of all possible token sequences sums to zero, training in practice operates over small finite rollout batches (G=8 samples per prompt) repeated over many epochs on the 100 narrow prompts. Because responses with high prior probability under the base model are sampled far more frequently in finite batches, they are the ones actually realized in the update buffer and accruing the positive reward signals, while unsampled tail sequences are suppressed via softmax normalization. Combined with the PPO/GRPO clipping bias identified by Shao et al. (2025) and the high repetition of the 100 prompts, the update rule reinforces the base model's dominant modes, tightening entropy. Broad D: Across 10,000 diverse prompts, individual prompts are rarely repeated, preventing prompt-specific mode sharpening. Instead, applying random finite-sample updates across shared parameters over thousands of distinct contexts acts as high-variance parameter noise that flattens token distributions globally, increasing entropy across benchmarks (GSM8K, MMLU, IFEval; Figure 4a–b). We have updated Section 5.3, and Appendix E to explicitly discuss how finite-sample updates, clipping dynamics, and prompt repetition drive these effects. Regarding adding zero/constant-reward controls, clipping ablations, and multi-seed uncertainty for random rewards: we agree these are interesting directions for isolating internal GRPO biases, but we view that as the specific focus of the "Spurious Rewards" paper (Shao et al., 2025). Our intention here is not to redo the entire spurious rewards study or dissect GRPO internals, but rather to show that their main empirical finding is confined to a highly restricted regime (a narrow prompt distribution paired with a strongly aligned base model prior) and does not reflect general RL post-training dynamics.
Continued in additional comment below....