- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes NudgeRL, a framework for improving exploration efficiency in RLVR through lightweight strategy-level hints. It combines context-conditioned rollout generation, inter- and intra-context advantage estimation, and a policy learning objective that transfers learned behaviors to the unconditioned policy. Experiments on mathematical reasoning and code generation show that NudgeRL matches or outperforms GRPO baselines with substantially fewer rollouts and lower estimated training compute.
Critical strengths
- Good computational efficiency. The experimental results show promising effectiveness of NudgeRL. For example, with only 8 rollouts, NudgeRL matches the strongest GRPO (rollout batch size=32) baseline on average across multiple math benchmarks, using approximately 5x fewer tokens and 2.9x less estimated training compute.
- Comprehensive analysis of method mechanism. This paper conducts experiments over two model families for the math reasoning task, and extend to the domain of code generation, where NudgeRL can still outperforms GRPO. Moreover, this paper also includes ablations on context dropout, hint selection, advantage weighting, and policy correction.
Critical weaknesses
- Insufficient comparison with closely related methods. The experiments primarily compare against rollout-scaled GRPO, without directly evaluating relevant hint-based approaches such as NuRL [1], SAGE [2], RLTF-SD[3]. The simplified oracle-prefix baseline does not fully address this gap, leaving the advantages of the proposed strategy-guided exploration over existing methods unclear.
- Limited evaluation of compute allocations. Combining Table 1 and Table 3, it is clearly evident that NudgeRL reduces computational overhead while maintaining performance. However, I suggest that the perspective on cost control or scaling for GRPO is not comprehensive. The configurations of current experiments use the same number of training steps, leaving alternative allocations of the GRPO budget unexplored. For instance, under a fixed budget, we can use less rollout batch size for more training steps. And in this paper, NudgeRL-8 uses approximately 44% more compute than GRPO-8, but the paper does not evaluate whether allocating this additional budget to more GRPO training steps could narrow the performance gap. Given that NudgeRL does not significantly improve task performance, I suggest including more fair comparisons to highlight its contributions to efficiency.
[1] Chen, Justin Chih-Yao et al. “Nudging the Boundaries of LLM Reasoning.”
[2] Liao, Baohao et al. “Self-Hinting Language Models Enhance Reinforcement Learning.”
[3] Song, Yu-Da et al. “Expanding the Capabilities of Reinforcement Learning via Text Feedback.”
Typos:
- p.4, L180–181: objective that learns … and distill(s) effective strategies
- p.9, L464–465: via (a) policy correction augmented RL objective
Questions for authors
Please refer to the Weaknesses.
A Gentle Reminder
Dear Reviewers,
Thank you very much for your thoughtful comments and valuable feedback. Your suggestions have been very helpful in improving our work.
As the discussion period draws to a close, we would like to follow up on our responses and additional clarifications. We have provided further explanations to address the concerns raised during the discussion.
We would greatly appreciate it if you could review our follow-up responses and let us know whether your remaining concerns have been adequately addressed. We hope that our explanations have sufficiently addressed your concerns, and we welcome any further feedback or suggestions.
Thank you again for your time and consideration.
Best regards