- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses a reliability issue in policy learning for RLVR, where automatic verifiers provide reward signals that can introduce verification errors. Authors argue that such errors in reward verification can propagate into policy learning, and study how verifier errors affect group-relative policy updates.
Critical strengths
1. The paper studies an important question in RLVR about how verifier errors interact with group-relative optimization. Looking beyond aggregate verifier accuracy is a useful direction.
2. The paper tries to connect the local analysis with training experiments. The matched-error setup provides a useful diagnostic view and also helps test whether the observed effect extends beyond the local audit.
Critical weaknesses
1. The identification is not very clean. Note that the main matched-error comparison changes more than one thing at the same time. For instance, the global-reference setting changes both how errors are allocated and the response gradients associated with those errors. The within-prompt component also seems unstable across experiments. Because of this, I find it hard to read the reported ratios and following claims.
2. I am less convinced that the local surrogate tells us much about the actual policy update. The primary audit is based on a fixed-response local gradient surrogate, while the actual group-relative update depends on the full training dynamics. The free-form and GSM8K experiments do not show a corresponding training gain. The paper would benefit from further testing of whether this local distortion matters in actual training, or from a mechanistic analysis of how it enters the group-relative policy update.
Questions for authors
See the weaknesses above. In particular:
1. Beyond showing that the same marginal FP/FN counts can yield different local group-relative gradients, what new property of verifier error do you believe the paper identifies?
2. In the matched-error comparison, how do you separate the effect of error allocation from changes in the associated response gradients and the unstable within-prompt component?
3. What evidence shows that the fixed-response local gradient surrogate predicts or meaningfully affects the actual group-relative policy update during training?
4. Given that the free-form and GSM8K experiments do not show a corresponding training gain, what evidence would support the claim that the measured local distortion matters for actual policy learning?
Thank you for these questions.
The new property is that verifier error placement has two separately measurable components even after global FP/FN totals are fixed: association with response gradients inside a prompt and allocation of errors across prompts. Marginal error rates identify neither component. On the frozen LiveCodeBench cohort, the three-test observed-to-global-reference ratios are 0.605 [0.504, 0.723] for Llama and 0.342 [0.225, 0.510] for Qwen Instruct.
The exact decomposition introduces a third distribution rather than interpreting the global permutation alone. Let Uc preserve error counts within every prompt and clean-label class, and let R preserve only global FP/FN totals. The exact identity is:
observed minus global reference = (observed minus within-prompt reference) + (within-prompt reference minus global reference).
The first term changes only which response gradients receive the fixed within-prompt errors; the second changes their allocation across prompts. For the three-test LiveCodeBench analysis, the within and between terms are -0.000479 and -0.002581 for Llama, and 0.000030 and -0.003283 for Qwen. Thus the stable finding is the between-prompt allocation effect; the Qwen within-prompt term is near zero rather than being used to support the conclusion.
The fixed-response score is the exact squared difference between clean-label and verifier-label group gradients for the stated reward surrogate in the declared LoRA tangent space. It therefore measures the immediate local update perturbation, not merely label disagreement. Full-response FP32 Grams agree with the independent full-loss/chunked-gradient check to a maximum relative error of 0.000064 and minimum cosine above 0.99999999.
We use this evidence as a sensitivity audit, not as a claim that smaller local distortion guarantees better final training performance. Its direct implication is that marginal verifier accuracy is insufficient to characterize the local group-relative update: evaluators should additionally report how FP/FN errors are distributed across prompt groups and, when update sensitivity matters, apply the matched-error decomposition on frozen responses. This diagnostic claim is fully supported by the independent cohort and does not require a downstream training-gain claim.