- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates a two-stage post-training procedure motivated by unrewarded learning. In the first stage, models train on their own sampled responses using a GRPO-style objective with every advantage fixed to one, without response-dependent reward assignment or correctness filtering. Standard rewarded training follows. Experiments across Qwen and Llama models on mathematical reasoning and GUI tasks report gains during the first stage and in several settings improved performance after subsequent RL. The paper also presents theoretical analyses intended to explain improvement under constant-advantage updates.
Critical strengths
The study examines both the initial reward-independent phase and its effect on subsequent rewarded training, rather than reporting only an intermediate improvement. It covers multiple model sizes and two task domains, and the GUI comparison explicitly matches total training steps between the two-stage procedure and rewarded training throughout.
Critical weaknesses
-
The contribution beyond a self-training initialization is not isolated.
The method is essentially a constant-advantage phase that provides a new starting point for standard RL. Its value should be shown against other reward-free initializations, notably SePT (acknowledged in Appendix E.1). The key missing comparison is self-SFT/SePT RL vs. constant-advantage RL under matched budgets; neither the DAPO nor the INTUITOR experiments in Appendix B provide this.
Also, Tables 1–2 report best-checkpoint results (best average, or per-benchmark maxima), which overstate deployable performance. Given the modest gains, results from fixed or validation-selected checkpoints are needed.
-
The theoretical analysis is flawed.
Eq. (23) defines , but Eq. (24) uses
which only holds with in place of . So Eq. (25)'s improvement over the old policy is not established; fixing the measure yields a comparison to the reference policy, not monotonic improvement. Moreover, the assumed alignment between the likelihood ratio and latent utility is itself what needs justifying, and the paper does not explain why constant-advantage training would produce it. Since the theory is a claimed main contribution, this is a central issue.
Questions for authors
- Does the first stage outperform unfiltered self-SFT/SePT and internal-reward pretraining when each is followed by the same RL under matched total budgets?
- Can you fix the measure inconsistency in Eqs. (23)–(25) and state which policy the guarantee is relative to? How does the corrected result explain the iterative training process?
- Were the checkpoints in Tables 1–2 selected on the same benchmark scores that are reported? What are the results with a fixed or validation-selected checkpoint with variability across runs?
Thank you for the response and additional experiments. The validation-selected results address my checkpoint concern, and the theorem now correctly states its guarantee relative to the reference policy. However, my main concerns remain:
I will therefore keep my rating of 2.