acceptodds
Under review as a conference paper at ICLR 2027

A Matched-Error Sensitivity Audit of Group-Relative Policy Updates

Abstract

Marginal verifier error rates leave unspecified how errors perturb a group-relative policy update. We measure this sensitivity on fixed responses using mean-token log-probability gradients in specified LoRA coordinates. An exact decomposition compares the observed distortion with two references: one preserves error counts within each prompt, and the other preserves only the global false-positive and false-negative totals. This separates response-level error association from allocation across prompts. On a frozen 12,288-generation LiveCodeBench cohort, three-test observed-to-global-expectation ratios are 0.605 [0.504, 0.723] for Llama and 0.342 [0.225, 0.510] for Qwen Instruct, using validated labels and full-response FP32 Grams. Within-prompt association is negative for Llama and uncertain for Qwen; one-test ratio intervals include one for both policies. A retrospective APPS audit shows related allocation effects, with sparse, cap-limited Thinking evidence. Diagonal Grams preserve much of the distortion ranking, and tested full-matrix features do not establish a stable selection advantage. A restricted eight-action training experiment favors prompt-correlated noise, but free-form follow-ups and a frozen 48-run GSM8K study do not confirm that benefit. The audit identifies what a matched-error contrast measures while separating local distortion, audit-selection utility, and downstream training.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, GRPO, verifier errors, gradient geometry, matched-error audits

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xSwdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses a reliability issue in policy learning for RLVR, where automatic verifiers provide reward signals that can introduce verification errors. Authors argue that such errors in reward verification can propagate into policy learning, and study how verifier errors affect group-relative policy updates.

Critical strengths

1. The paper studies an important question in RLVR about how verifier errors interact with group-relative optimization. Looking beyond aggregate verifier accuracy is a useful direction.

2. The paper tries to connect the local analysis with training experiments. The matched-error setup provides a useful diagnostic view and also helps test whether the observed effect extends beyond the local audit.

Critical weaknesses

1. The identification is not very clean. Note that the main matched-error comparison changes more than one thing at the same time. For instance, the global-reference setting changes both how errors are allocated and the response gradients associated with those errors. The within-prompt component also seems unstable across experiments. Because of this, I find it hard to read the reported ratios and following claims.

2. I am less convinced that the local surrogate tells us much about the actual policy update. The primary audit is based on a fixed-response local gradient surrogate, while the actual group-relative update depends on the full training dynamics. The free-form and GSM8K experiments do not show a corresponding training gain. The paper would benefit from further testing of whether this local distortion matters in actual training, or from a mechanistic analysis of how it enters the group-relative policy update.

Questions for authors

See the weaknesses above. In particular:

1. Beyond showing that the same marginal FP/FN counts can yield different local group-relative gradients, what new property of verifier error do you believe the paper identifies?

2. In the matched-error comparison, how do you separate the effect of error allocation from changes in the associated response gradients and the unstable within-prompt component?

3. What evidence shows that the fixed-response local gradient surrogate predicts or meaningfully affects the actual group-relative policy update during training?

4. Given that the free-form and GSM8K experiments do not show a corresponding training gain, what evidence would support the claim that the measured local distortion matters for actual policy learning?

1 comment

Thank you for these questions.

  1. The new property is that verifier error placement has two separately measurable components even after global FP/FN totals are fixed: association with response gradients inside a prompt and allocation of errors across prompts. Marginal error rates identify neither component. On the frozen LiveCodeBench cohort, the three-test observed-to-global-reference ratios are 0.605 [0.504, 0.723] for Llama and 0.342 [0.225, 0.510] for Qwen Instruct.

  2. The exact decomposition introduces a third distribution rather than interpreting the global permutation alone. Let Uc preserve error counts within every prompt and clean-label class, and let R preserve only global FP/FN totals. The exact identity is:

observed minus global reference = (observed minus within-prompt reference) + (within-prompt reference minus global reference).

The first term changes only which response gradients receive the fixed within-prompt errors; the second changes their allocation across prompts. For the three-test LiveCodeBench analysis, the within and between terms are -0.000479 and -0.002581 for Llama, and 0.000030 and -0.003283 for Qwen. Thus the stable finding is the between-prompt allocation effect; the Qwen within-prompt term is near zero rather than being used to support the conclusion.

  1. The fixed-response score is the exact squared difference between clean-label and verifier-label group gradients for the stated reward surrogate in the declared LoRA tangent space. It therefore measures the immediate local update perturbation, not merely label disagreement. Full-response FP32 Grams agree with the independent full-loss/chunked-gradient check to a maximum relative error of 0.000064 and minimum cosine above 0.99999999.

  2. We use this evidence as a sensitivity audit, not as a claim that smaller local distortion guarantees better final training performance. Its direct implication is that marginal verifier accuracy is insufficient to characterize the local group-relative update: evaluators should additionally report how FP/FN errors are distributed across prompt groups and, when update sensitivity matters, apply the matched-error decomposition on frozen responses. This diagnostic claim is fully supported by the independent cohort and does not require a downstream training-gain claim.

Reviewer_A89qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • Studied problem: How do marginal verifier errors in RLVR perturb GRPO on fixed responses using logprob gradients in LoRA coordinates (“if clean labels and the exact global FP/FN counts are held fixed, how much does the allocation of those errors across prompt groups change the update”)? This is needed as the group relative centering of GRPO is mismatched with verifier quality (false negative and false positives), as a group-relative update also depends on whether errors are isolated across prompts or concentrated within one prompt.
  • Submission approach: Test whether an observed error allocation gives a different local update from other allocations with the same counts and whether changing a noise channel improves subsequent clean performance.
  • Approach realization: Evaluate with prompts from the APPS dataset and LiveCodeBench, three models (Qwen3-4B-Inst, Qwen3-4b-Think, Lllama-3.1-8B-Inst) and additional ones during selected experiments, and Gram matrix scores for each group LoRA updates.
  • Claims/Contributions: A matched-error audit that separates marginal errors across allocated prompts and resposne gradients within a prompt.
Critical strengths
  • The underlying topic of error propagation in GRPO and RLVR is interesting and the demonstrated mismatch provides insightful potential for future works.
Critical weaknesses
  • The submission suffers from severe writing clarity issues. It is too densely written and I found it extremely hard to follow (After reading it twice, I still cannot tell you what all of their results are supposed to imply). I recommend rejection solely on that basis. Specifically:

    • It reads like a dense list of purely technical details and individual results without connecting elements (Sections 4 and 5 in particular).
    • Section 2 (formalization) is hard to follow due to missing connectors and motivations.
    • Things like “APPS audit” undefined, informal lingo like “cap-limited” is ambiguous, uncertainty intervals are sometimes stated without confidence levels (Fig 1), numerical precision of results is not justified given the uncertainty intervals (Tab 1)
    • Additional models are being used and tested mid-result section (e.g., training Qwen2.5-7B-Inst, Qwen2.5-1.5B-Inst, or Qwen3-0.6B)
    • No illustrative plots/schematics that would have been helpful to better understand the mechanisms and motivation.
  • The authors do not position their submission well within the vast body of GRPO/RLVR work (only 14 citations in total with the majority being dataset and model citations)

Questions for authors

None.

2 comments

Thank you. The central question is simpler than the current presentation makes it appear. We freeze the model responses and their clean labels, then ask whether the same total numbers of false positives and false negatives produce different prompt-level gradient errors when assigned differently.

The analysis has three steps. First, the observed verifier errors give an observed local distortion. Second, a within-prompt reference reassigns errors only among responses with the same clean label while preserving every prompt's error counts. Third, a global reference preserves only the corpus-wide FP/FN totals and can reallocate errors across prompts. Their exact identity separates within-prompt response association from between-prompt error allocation.

On the frozen 12288-response LiveCodeBench cohort, the three-test observed-to-global-reference ratios are 0.605 [0.504, 0.723] for Llama and 0.342 [0.225, 0.510] for Qwen Instruct. The decomposition shows why: the between-prompt allocation terms are -0.002581 and -0.003283, while the within-prompt terms are -0.000479 and 0.000030. The result is therefore not merely that verifier accuracy matters; it identifies which part of error placement drives the matched-error contrast.

Thank you for your clarifications! Both of my concerns (severe writing clarity and lack of positioning in relation to prior works in the field) have not been addressed. I am not in favor of accepting this paper.

Reviewer_GSbYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigate how verifier errors affect group-relative policy updates if clean labels and the exact global false positive false negative counts are held fixed. To achieve this objective, they measure the squared difference between clean and noisy reward gradients on cached responses, using mean token log probability gradients in specified LoRA coordinates. A within prompt reference keep each prompt's error counts fixed, and a global reference distribures errors across prompts within each label class; the difference between the two is exactly what separates response-level and prompt-level error allocation. the strongest results are the LiveCodeBench 3-test observed-to-global-reference ratios achieving 0.605 for Llama and 0.342 for Qwen, with both intervals being below one.

Critical strengths
  • holding the clean labels and the exact global false positive false negative counts are fixed helps bring rigor into the analysis and disentangle the effect of verifier errors
  • the authors disclose the failed pilots instead of treating the result as pointing to the fact that correlation is helpful in general
  • the authors measure the gradients instead of just looking into label changes
  • the authors explain why grouped errors are not always better
Critical weaknesses
  • The clarity of the presentation is really missing. most of the abstract reads like unrelated sentences with no flow and so much jargon when it's supposed to be the clearest summary of the paper. How is a reader supposed to know what an "A retrospective APPS audit" is? or a "A restricted eight-action training experiment"? This paper needs increased clarity in my opinion because these issues got only worse as I was reading the rest of the paper.
  • the main measurement is not the error in an actual batch update where the gradients are averaged. the authors square each question's gradient error separately and then average whereas errors from different questions can cancel or boost each other when the optimization updates combine them
  • the authors average token log probabilities within each answer but this does not take into account response length. maybe summing the log probabilities instead of averaging would give longer answers more weight and lead to different takeaways
  • different LoRA representations can describe the same model but give different gradient distances so need to investigate this point
  • it's unclear how these findings can inform new training recipes
Questions for authors

see weaknesses, i'd like to see the authors address each

2 comments

Thank you for identifying these distinctions. Our estimand is deliberately the mean across prompts of the squared prompt-level gradient error, not the squared norm of a batch-averaged gradient. It asks how much verifier noise changes each group-relative update before cross-prompt cancellation. We will use the term "prompt-level local update distortion" consistently so that it is not read as a claim about the realized multi-prompt optimizer step.

We chose the mean-token score so that response length alone does not rescale the tangent geometry. We also recomputed the primary three-test LiveCodeBench analysis with the unnormalized trajectory-sum score, which is equivalent to multiplying every response score by its length before forming the Gram matrix. The observed-to-global-reference ratio remains below one for both policies: 0.308 [0.254, 0.374] for Llama and 0.362 [0.292, 0.442] for Qwen Instruct, compared with 0.605 [0.504, 0.723] and 0.342 [0.225, 0.510] under the mean-token score. Thus length weighting changes the scale and magnitude, but not the primary matched-error conclusion.

Raw gradient distances are coordinate-dependent, and we do not claim parameterization invariance. We therefore state the rank-16 query/value LoRA coordinates as part of the estimand. The conclusion is stable under two strong Gram controls: for Llama, the three-test ratios are 0.605, 0.591, and 0.656 with full, diagonal, and same-trace isotropic Grams; for Qwen they are 0.342, 0.252, and 0.306. These controls do not prove invariance, but they show that the primary ordering is not specific to the full off-diagonal Gram geometry.

The practical implication is a verifier-audit requirement rather than a claim of a universally better noise channel. A marginal FPR does not determine a group-relative update: in an all-negative group of size KK, independent false positives activate a nonconstant advantage vector with probability 1−(1−p)K−pK1-(1-p)^K-p^K, whereas a perfectly group-correlated channel with the same marginal FPR activates it with probability zero. A noise-aware training method should therefore measure clean group composition and the conditional within-group error-count distribution, in addition to marginal FP/FN rates.

Thanks for the response. Here's my assessment:

  • I think the trajectory-sum experiment addresses my concern about response length, and the clarification on prompt-level distortion is helpful, but the distinction from the actual batch update still matters. Measuring how much each prompt's gradient changes doesn't establish how much verifier noise changes the update that the optimizer actually takes.
  • I also still have a few other concerns: 1. gradient errors from different prompts can cancel or reinforce each other, so it's still unclear whether the lower observed distortion would hold when gradients are averaged across a batch, 2. the diagonal and isotropic experiments show that the main ordering doesn't depend on the full off-diagonal geometry, but they don't address whether it is stable under different LoRA representations of the same model, 3. reporting error-count distributions and group composition seems useful, but it's still unclear what the gradient-based experiments add in practice, especially since simpler geometry preserves much of the ranking and the training and selection experiments don't establish a benefit.

I don't think the paper necessarily needs to propose a new training recipe, but it needs to make a clearer case for why this particular diagnostic is useful beyond showing that marginal error rates are insufficient. The presentation also still needs so much work.Therefore, I'm keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.