- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes REINFORCE++, a critic-free policy optimization framework that replaces prompt-level advantage normalization with global batch normalization. The authors also provide a theoretical analysis of normalization and evaluate the methods on preference alignment, mathematical reasoning, logical reasoning, and multi-step tool use.
Critical strengths
-
It provides a simple and practical alternative to prompt-level normalization, potentially improving training stability without requiring a learned critic.
-
These experiments span multiple domains, including general reinforcement learning versus human performance (RLHF), mathematical reasoning, and tool use. The experimental results show that, across a variety of scenarios, the global normalization method can match the performance of GRPO and PPO.
Critical weaknesses
-
The experiments primarily rely on Llama-3-8B-SFT and Qwen2.5, which are relatively outdated for an ICLR 2027 submission. Evaluation on more recent models (e.g., Qwen3) is needed to establish generalizability. Moreover, the general RLHF experiments lack comparisons with DPO or similar preference-optimization methods, while the reasoning experiments omit stronger recent RL baselines such as DAPO and VAPO. Also, Human evaluation is needed for human preference.
-
This method primarily replaces local normalization with global batch normalization, offering limited methodological innovation beyond existing approaches. Furthermore, the paper lacks sufficient ablation experiments, making it impossible to distinguish the contribution of global normalization from that of other components (such as the reward baseline and KL regularization). Some of the improvements are negligible, and the exceptionally large generalization gap shown in Table 2 lacks sufficient statistical validation, raising doubts about the robustness of the experimental conclusions.
Questions for authors
How exactly are Pass@1 and Pass@16 computed? It looks very weird. GRPO achieves only 0.4% Pass@16 on AIME-25, which contains 30 problems. This value cannot result from a single evaluation using the standard binary Pass@16 metric with 16 samples per problem. Were multiple evaluation runs or a different estimator used? Please clarify the evaluation protocol and report variance across random seeds or temerpture.