- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates the effect of verifier errors on long-horizon RL with verifiable rewards. Contrary to the expectation that verifier errors should accumulate over long trajectories, the authors find that policy accuracy remains largely unchanged even under substantial false-positive rates. They attribute this apparent robustness to task-correlated verifier errors in group-relative policy optimization: such errors often make all samples in a group receive identical rewards, causing the entire group to contribute zero reward-driven gradient rather than introducing an incorrect gradient. The paper therefore argues that policy accuracy can be misleading as a measure of verifier quality, and proposes the fraction of degenerate groups as a more direct indicator of the effective training signal.
Critical strengths
The finding that task accuracy remains largely unchanged as the false-positive rate increases is interesting, and the underlying reasons warrant further investigation.
Critical weaknesses
-
The claim that a group’s advantage vanishes in the presence of verifier errors lacks sufficient theoretical justification and comprehensive empirical validation.
-
This paper lacks sufficient background statements, and the method section should be more focused on the main contributions and written in a more formal style.
-
The paper appears to have been written in haste. The authors should better articulate the core contributions and improve the manuscript's logical organization and overall clarity to make it easier to follow.
Questions for authors
Could the authors clarify the scope and limitations of the conclusions presented in the paper?