- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how group advantage rules change the weight given to each prompt. It introduces α-GRPO and reduces this shaping during training to recover a guarantee for the original mean-reward objective under stated assumptions. The paper gives finite-group formulas and convergence results under stated assumptions, then tests fixed and changing α on three models.
Critical strengths
- The α-GRPO family makes the links between several advantage rules explicit.
- The formulas show how group size changes the weight given to easy and hard prompts.
- The annealing result explains how shaping bias vanishes under the stated assumptions.
Critical weaknesses
- α-GRPO mainly reweights existing reward signals and connects known advantage rules. The new learning mechanism limited, despite the useful analysis.
- Annealing removes shaping bias, but the theory does not show when it is better than using α = 1 from the start. This leaves the main training choice unresolved.
- The proofs cover binary rewards and SGD; the experiments use AdamW, clipping, and length handling. These differences limit how directly the theory explains the actual training updates.
Questions for authors
Please respond to the concerns raised in Critical Weaknesses.