- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how enriched sampling, such as generating responses with hints, changes reinforcement learning with verifiable rewards. It interprets uncorrected enrichment as implicit reward reweighting and decomposes gradient error into scale, rotation, and variance. The authors propose an SMC-based correction mechanism with mean-preservation guarantees for the analyzed REINFORCE estimator and variance bounds under additional assumptions. Controlled experiments illustrate these effects, followed by a Qwen3-1.7B case study on a low-success-rate subset of OpenMathReasoning investigating enrichment and training collapse.
Critical strengths
-
Useful conceptual clarification. The reward-reweighting interpretation and the separation of scale, rotation, and variance provide an accessible framework for understanding enriched training. In particular, the analysis clarifies how enrichment can change the relative strength of reward and KL regularization, or preferentially reinforce particular solution families.
-
Carefully designed controlled experiments. The experiments isolate several sources of gradient distortion. Appendix D.2 holds reward sparsity fixed while varying sequence length and demonstrates substantial variance reduction from SMC in the constructed setting. The analysis also explicitly accounts for shared ancestry and the qualifications needed for the variance bound.
Critical weaknesses
-
Limited empirical support for the robustness of the collapse-prevention explanation. The sole real-model experiment uses Qwen3-1.7B on a hard, low-success-rate mathematics subset, with 500 training and 300 evaluation problems. I recognize that Section 4 is positioned as a case study. Nevertheless, the experiment leaves unclear which aspects of the observed behavior depend on the initial success rate, guidance strength, or model capacity. The SMC arm’s pass@1 advantage over unenriched training is also uncertain: the reported 95% bootstrap interval is [−0.21, +1.12] percentage points, with a one-sided p-value of 0.09. Although the authors report significant pass@k gains, the corresponding statistical details and seed-level variability are insufficiently specified. Together, these limitations constrain the empirical support for applying the proposed explanation beyond this particular setting.
-
The proposed anti-collapse mechanism is not directly established by the reported measurements. Section 4 attributes the results to increased policy entropy and learning new successes. However, the description of Figure 4(c) does not specify the sampling and scoring distributions or whether correction weights are applied. Unless the statistic is computed on separately generated unguided base-policy rollouts or with an appropriate correction, changes in the training-sample distribution can also affect it. Consequently, decreasing typicality does not by itself establish increased base-policy entropy. Similarly, pass@k trends are consistent with changes in success coverage but do not uniquely identify memorization of known solutions. Since Section 4 uses these observations to explain collapse prevention, clarifying the measurement protocol and providing direct entropy and coverage diagnostics would strengthen this central claim.
Questions for authors
-
Please specify the sampling and scoring distributions used in Figure 4(c), whether SMC/importance weights are applied to this statistic, and whether it is evaluated on fresh unguided base-policy rollouts.
-
For which values of k are the pass@k improvements statistically significant? Please specify the confidence intervals, bootstrap resampling unit, and treatment of the three training seeds.
-
Given the qualification in Section 3.1 that mean preservation does not automatically extend to GRPO, what is the exact SMC-corrected GRPO update used in Section 4? Please clarify how correction weights enter the group mean, standard deviation, and final loss, and how this update relates to the cited weighted-surrogate interpretation.
-
Could existing checkpoints be used to measure base-policy entropy and per-problem success coverage under matched prompts and unguided decoding settings? If feasible, a targeted analysis stratified by initial success rate would help assess whether the collapse-prevention explanation consistently matches the observed training behavior.
3 comments
Thank you for your time and feedback. We have posted our responses to your concerns. As the discussion period ends soon, we would appreciate it if you could take a look and let us know if you have any further questions.
The new coverage and entropy results address my main concerns. I will raise my rating to 3.
We thank the reviewer for their review.
Figure 4(c) reports the mean length-normalized log probability under each run’s unhinted policy, using training responses drawn from the base policy for unenriched training, the enriched policy for uncorrected training, and the resampled terminal population for SMC. No SMC or importance weights are applied, and we did not generate fresh unguided evaluation rollouts. We have conducted an additional per-problem success coverage and entropy evaluation to substantiate. Please refer to tags [CVRG] and [ETRP] for details. We will revise the entropy and memoization interpretation to ground it concretely in our new coverage experiments.
We provide the analysis below, where [Config - unenriched] indicates the difference in pass@k coverage, in percentage points, with 95% confidence intervals:
SMC surpasses unenriched with statistical significance at all tested k ≥ 4, while uncorrected surpasses unenriched across all tested k. Pass@k is computed separately for each problem and seed from evaluation responses following the method in Humaneval [2], then we average the three seed-specific values for each problem. Bootstrapping follows the method in [3] over 100000 runs that draw 300 problems with replacement over the full 300-problem eval set.
Please refer to Reviewer [k8go]’s question on the GRPO update, tagged [GRPU]. The cited paper [1] derives a weighted group-relative update in Section 4.3, equation (9); our corrected reward update follows this form using normalized SMC correction weights, with an additional scaling term from the group reward’s standard deviation.
Please refer to Reviewer [k8go]’s question on coverage, tagged [CVRG], for the requested per-problem success coverage experiment, which was performed on the base policy and all three trained policies (unenriched, uncorrected, and SMC).
[ETRP]
We also provide a comparison of entropy using the same experimental settings as in [CVRG] (using a set of fresh 64 responses sampled from each policy). Uncorrected and SMC training have statistically significant higher entropy compared to unenriched training:
The difference is statistically significant over both groups in [CVRG] (those where the initial policy produced a success and those where it did not). We also note that unenriched training decreases per-token entropy compared to the initial policy by 52% (from 0.299 nats to 0.144 nats), while both SMC (0.363 nats) and uncorrected training (0.300 nats) maintain the statistic.
We do agree that while the entropy difference is statistically significant, it does not establish a causation relationship with the increase in coverage. We will revise the claim such that the coverage explanation is directly supported by the experiment from [CVRG], and include the entropy comparison as a separate observation.
References
[1] https://arxiv.org/abs/2509.24203v2
[2] https://arxiv.org/abs/2107.03374
[3] https://aclanthology.org/D12-1091/