- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the cancellation of binary verifier rewards in GRPO when all completions in a group violate a constraint. The authors investigate how reward weighting and several existing remedies affect this issue using two synthetic tasks with Qwen2.5-0.5B-Instruct.
Critical strengths
The paper reports failed predictions and degenerate outcomes rather than only successful experiments (e.g., GSM8K).
Critical weaknesses
The main cancellation phenomenon is already known, as acknowledged by the authors. The additional crossover-weight analysis and the classification of remedies by their inputs are useful but offer limited technical novelty. However, I do not find the overall methodological contribution sufficiently substantial for ICLR.
In addition, the main experiments use a single 0.5B model and two synthetic tasks with only a few fixed prompts/instances. This substantially limits the external validity of the conclusions. The authors should validate on real benchmark datasets and across different model families and scales.
Questions for authors
Please see the weaknesses above.