acceptodds
Under review as a conference paper at ICLR 2027

Beyond Verifier Reliability: Information for Policy Updates in GRPO

Abstract

Reinforcement learning with verifiable rewards commonly uses binary correctness feedback, yet its accuracy on current responses need not identify which policy update most improves true correctness. Accuracy aggregates errors, whereas update value depends on their alignment with changes in response probability. To isolate this missing information, we fix the initial policy, verifier, and complete GRPO menu, then construct binary utility worlds with indistinguishable finite-precision reports but different preferred updates at equal KL. The separation persists under specified perturbations and finite-rollout construction. We also characterize the exact linear utility moments sufficient for finite candidate comparisons and give conditions for reusing an initial correctness contrast, with separate estimation and redistribution errors. Two controlled Qwen3-4B experiments examine the consequences for feedback evaluation. The first holds group confusion matrices and centered reward MSE fixed to test whether these summaries determine update outcomes: the resulting policies differ by 8.53 percentage points in average utility. Only a four-position prefix-KL statistic is matched; a complete-history diagnostic finds unequal sequence KL. The second tests whether targeted execution information improves training at a fixed test count: four diagnostic tests outperform four random tests by 3.9 and 7.0 percentage points in two seeds. Together, the results identify information omitted by reliability summaries and conditions under which decision-relevant information can be retained and reused.

Primary area:
reinforcement learning
Keywords:
reliability, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CBynOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper theoretically examines if the reward normalization of GRPO works without losing information about policy update direction in the RLVR (reinforcement learning with verifiable reward) setting. (If I understand correctly,) the paper points out the issues of using the aggregated performance metric such as accuracy ratio on multiple verifiable questions as it loses the information about for which question the policy need to be updated to correct. The paper also conducts experiments on some benchmark called the complete-v4 suite.

Critical strengths
  • GRPO is a widely used method, and studying if and how the algorithm works in the RLVR setting should be useful for the community.

Note: I took two passes of the paper with careful reading. Yet, I did not really understand the main message of the paper. The discussion about strength may be added if the author(s) clarify the contribution of the paper in the rebuttals.

Critical weaknesses

Writing should be improved, and the paper is hard to follow and the main message of the paper is unclear even after carefully reading it twice. Specifically following points made it difficult to understand the content.

  • (1) What is equal KL? I believe it is not the KL regularization used in GRPO, but I did not figure out what it exactly refers to. This seems important notation in the paper, but it was neither formally defined nor explained. Specifically, between what two distributions this KL is calculated and what this KL equal to should be explained in early sections of the paper, ideally in Introduction as Intro refers to this concept.

  • (2) Similarly, while I could somehow guess what is "information gap" by the descriptions provided in Section 2; related work ("verifier and utility .. is only partially observed"), the meaning of information gap should be explained in Intro, where this notion is mentioned for the first time in the paper.

  • (3) Cell correctness rate is mentioned in line 128 of page 3, but what is "cell" is not defined.

  • (4) I also did not understand which factor makes this problem specific to GRPO, not other gradient-based method like PPO. If I understand the paper correctly, I believe that the issues of using the aggregated performance metric such as accuracy ratio on multiple verifiable questions (this understanding is based on explanation made in Section 3, line 132 "For example ~") This part is also made difficult to understand due to (3); "cell" is undefined.


Aside from writing, other weaknesses are as follows.

  • Potential LLM hallucination in the experiment section. Specifically, what is "complete-v4 suite" in the experiment section? I haven't heard about this benchmark and searched on the web, but I could not find the corresponding benchmark suite. If this is not a hallucination, the paper should include either citation or link to this benchmark.

  • The paper should be self-contained in the main text. Even if the formal proof is provided in Appendix, a clear proof outline should be provided in the main text, so the readers/reviewers can identify if the discussion is reasonable.

Questions for authors
  • What are "equal KL" and "cell"?

  • Is the issue specific to GRPO? If so, please indicate which Theorem explains this. (And the manuscript should make this point clearer).

  • Please clarify about the benchmark "complete-v4 suite".


It is likely I did not really understand the content of the paper, and I'd be happy to evaluate potential impacts once the questions are cleared and I understand the key findings of the paper. However, at least the quality of writing and organization of the paper should be improved before presentation.

Reviewer_mXqSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the mismatch between true utility and reward in the setting of reinforcement learning with verifiable rewards.

Critical strengths

None

Critical weaknesses

The manuscript as presented does not satisfy basic scientific standards of communication.

  • It is incredibly dense writing.
  • The text does not clearly establish what it attempts to do or why.
  • Mathematical notation is only vaguely defined at best, often not defined at all. This makes the math unreadable and the Theorems unverifiable. The math is also overly obfuscated by introducing lots of notation at once without guiding the reader as to what is going on.
  • The text uses a lot of language that is never introduced. For example, in the introduction the text talks about reports and menus, it is unclear to me what these are.
  • I cannot understand the main claims of the paper as they use very confusing language, here is an example of a main claim:
    • "In the paired GRPO family, one initial correctness contrast supports later comparisons, with separate errors for estimating the contrast and changing the response composition within its blocks.". I do not understand this sentence.
  • The experiments are missing proper descriptions, and basic information as to what the experiment setup is and what the experiments are trying to do and how.

My review is not particularly detailed as the text is not readable. I believe extensive revisions are needed in terms of writing before it can be properly reviewed

Questions for authors

None

Reviewer_bdpnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies RL with verifiable rewards, where a binary verifier (V) gives rewards during training, but it differs from true correctness U. The paper takes a general theoretical perspective but an example would be code generation tasks where the verifier provides a set of unit tests, which might not capture all edge cases. The main question the authors ask is whether the reliability summary of the verifier (like the correctness rate) contains enough information to choose between candidate training updates, and they find that it does not. The key insight (eq. 2) is that they split the utility difference between two updates into a between-cell term, determined by the correctness rates, and a within-cell term covariance term which describes whether the update upweights or downweights responses that are correct.

Three main contributions: (1) The authors show that the gap (the difference the reliability report and what you need to know for the update) can occur under GRPO. (2) They suggest a fix for closing this gap: split the accepted responses into whether the update would push them to increase or decrease their probability, and measure the difference in true correctness on the two halves. This measurement can be used in later updates but becomes less accurate as the model’s policy changes. (3) Finally, they contribute experiments on Qwen3-4B where they show reward assignments with matched confusion matrices and centered MSE policies differing substantially in true correctness.

Critical strengths

The theoretical results appear correct. I checked the main-text results and from the toy example I agree with the main premise (that the verifier correctness rate does not give you all the information you need to identify the update). I did not verify the appendix proofs in detail.

Critical weaknesses
  1. Limited practical relevance. My main issue is that the authors do not show that the key problem they identify (verifiers or updates being chosen by accuracy-style summaries) actually occur in practice. Usually, practitioners evaluate candidate updates directly on held-out data or they fix a verifier when flaws appear. Furthermore, I don’t see how the fix that the authors propose can be implemented in realistic settings. The signed within-cell contrast requires knowledge of which responses future updates will upweight, which we do not have access to. The Qwen3-4B experiments do not actually implement the audit. The paper should explain what the specific mechanism they identify (within-cell covariance) adds in realistic training settings.
  2. The paper is very difficult to follow. The problem is not stated plainly, it is difficult to make out whether the key toy example occurs in practice, and the terminology is not standard.
Questions for authors

Q1. Can you describe a concrete, realistic workflow in which an audit summary is used to choose between training updates, and where we cannot evaluate the candidate updates directly on some held-out data?

Q2. How would one identify which responses future updates will update in a real language model, so that we can compute the contrast?

Q3. The authors’ finding that the same error rate can lead to different outcomes seems close to the findings of Wen et al (ICLR 2025), who make a similar point with learned reward models. Can the authors comment on the difference between these results and theirs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.