- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper theoretically examines if the reward normalization of GRPO works without losing information about policy update direction in the RLVR (reinforcement learning with verifiable reward) setting. (If I understand correctly,) the paper points out the issues of using the aggregated performance metric such as accuracy ratio on multiple verifiable questions as it loses the information about for which question the policy need to be updated to correct. The paper also conducts experiments on some benchmark called the complete-v4 suite.
Critical strengths
- GRPO is a widely used method, and studying if and how the algorithm works in the RLVR setting should be useful for the community.
Note: I took two passes of the paper with careful reading. Yet, I did not really understand the main message of the paper. The discussion about strength may be added if the author(s) clarify the contribution of the paper in the rebuttals.
Critical weaknesses
Writing should be improved, and the paper is hard to follow and the main message of the paper is unclear even after carefully reading it twice. Specifically following points made it difficult to understand the content.
-
(1) What is equal KL? I believe it is not the KL regularization used in GRPO, but I did not figure out what it exactly refers to. This seems important notation in the paper, but it was neither formally defined nor explained. Specifically, between what two distributions this KL is calculated and what this KL equal to should be explained in early sections of the paper, ideally in Introduction as Intro refers to this concept.
-
(2) Similarly, while I could somehow guess what is "information gap" by the descriptions provided in Section 2; related work ("verifier and utility .. is only partially observed"), the meaning of information gap should be explained in Intro, where this notion is mentioned for the first time in the paper.
-
(3) Cell correctness rate is mentioned in line 128 of page 3, but what is "cell" is not defined.
-
(4) I also did not understand which factor makes this problem specific to GRPO, not other gradient-based method like PPO. If I understand the paper correctly, I believe that the issues of using the aggregated performance metric such as accuracy ratio on multiple verifiable questions (this understanding is based on explanation made in Section 3, line 132 "For example ~") This part is also made difficult to understand due to (3); "cell" is undefined.
Aside from writing, other weaknesses are as follows.
-
Potential LLM hallucination in the experiment section. Specifically, what is "complete-v4 suite" in the experiment section? I haven't heard about this benchmark and searched on the web, but I could not find the corresponding benchmark suite. If this is not a hallucination, the paper should include either citation or link to this benchmark.
-
The paper should be self-contained in the main text. Even if the formal proof is provided in Appendix, a clear proof outline should be provided in the main text, so the readers/reviewers can identify if the discussion is reasonable.
Questions for authors
-
What are "equal KL" and "cell"?
-
Is the issue specific to GRPO? If so, please indicate which Theorem explains this. (And the manuscript should make this point clearer).
-
Please clarify about the benchmark "complete-v4 suite".
It is likely I did not really understand the content of the paper, and I'd be happy to evaluate potential impacts once the questions are cleared and I understand the key findings of the paper. However, at least the quality of writing and organization of the paper should be improved before presentation.