- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is about not verifying all responses in GRPO.
Critical strengths
The paper uses a 3B model for experiments.
Critical weaknesses
Introduction: Many terms in the introduction are non-standard and used without any explanation. For example, “reference verification,” “randomized telescoping,” “audit randomization.” Generally the wording is quite dense and difficult to follow.
Problem setup:
should be defined. What are “reference labels”, and more broadly, what are “labels”? Generally for this kind of setup, there are verifiable rewards, and I am unfamiliar with the terminology of “labels” in RL. Also, the term “stopping variable” is used without definition and should be clearly defined. What is the significance of , and why write it as a telescoping sum? What is and ?
On the motivation: To my knowledge and based on works cited, the motivation for this paper is weak. My understanding is that the authors tested whether one needs all rewards in GRPO to approximate the surrogate objective’s gradient, assuming that waiting for verification is a bottleneck. First, the examples of verification given here use CPU resources, which tend to be extremely cheap compared to GPU resources. Second, it is not clear that any solution to requiring fewer verifications would result in a reduced time to train with GRPO. Particularly, authors do not show how reducing CPU cost improves total resource usage or elapsed training time. Methods such as asynchronous GRPO allow GPU usage to be parallelized with CPU verification. Additionally, authors do not report full training runs using any of their methods, and hence it is not sufficiently evaluated. Lastly, not verifying all responses from the LLM brings into question why one would even ask for many responses from the LLM. For example, if one generated fewer responses but verified all responses, would this be a better use of resources?
Questions for authors
See above