acceptodds
Under review as a conference paper at ICLR 2027

EviGRAD: Evidence-Calibrated Rewards and Gradients for RLVR

Abstract

Verifiable-reward reinforcement learning (RLVR) for code generation relies on executable tests to provide training rewards. When these tests fail to cover important behavioral boundaries, incorrect programs may pass them and receive spuriously high rewards. We find an asymmetric reward-error pattern: high-reward false positives are relatively common and can be identified with useful accuracy, whereas low-reward candidates are rarely rescued by additional validated tests. This makes suspiciously high rewards the primary actionable source of reward misranking in group-relative policy optimization. We present , a risk-aware selective verification framework for conservative reward correction under a fixed verification budget. A lightweight reliability model ranks high-reward candidates and rollout groups according to their risk of being false positives. Only the highest-risk groups, together with a small exploration quota, receive additional verification and counterexample generation. Generated tests are admitted only after reference-based semantic validation and execution-based checks. Verified counterexamples then produce a one-sided reward correction: suspiciously high baseline rewards may be reduced, whereas low baseline rewards are not increased merely because a candidate passes an additional test. For rollout groups with degenerate low-signal rewards, EviGRAD uses dynamic resampling to recover learnable variation. The two mechanisms address complementary failure modes: erroneous positive training signals and insufficient within-group reward variation. We evaluate EviGRAD on CodeContests and APPS using matched models, training steps, rollout budgets, and verification costs. We measure agreement between training-time reward rankings and hidden-test rankings, the rate of false-positive positive-gradient signals, correction coverage, recovery of zero-advantage groups, final task success, and verification overhead. Across multiple random seeds and cost-controlled comparisons with binary GRPO, pass-rate GRPO, DAPO-style dynamic sampling, and always-augmented testing, EviGRAD reduces false-positive positive-gradient signals and improves the alignment between training-time rewards and hidden-test correctness, while dynamic resampling recovers learning signal from degenerate groups.

Primary area:
reinforcement learning
Keywords:
Verifiable Reinforcement Learning, Reward Calibration, Evidence-Guided Policy Optimization, Adaptive Clipping, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YrzmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

EviGRAD addresses errors in test-based rewards for reinforcement learning in code generation through selective verification and conservative reward correction. A learned risk model prioritizes rollout groups for judge-guided counterexample generation, and validated failing tests reduce the rewards of candidates that passed all original tests. A separate evidence-guided repair or resampling procedure handles low-reward groups with no within-group reward variation. The paper reports improvements over pass-rate GRPO on CodeContests and LiveCodeBench, using approximately one-third of the additional verification resources required by always-verify.

Critical strengths

None

Critical weaknesses
  1. I find the methodological advance over prior test-generation and verification approaches insufficiently established. The additional logistic-regression risk model offers a mechanism for prioritizing verification, but its practical value remains unclear, particularly because its features already depend on potentially expensive judge outputs. The experiments lack matched-budget downstream comparisons against strong test-augmentation methods and simpler selection strategies, such as random verification or seed-only selection. Consequently, the reported results do not establish whether EviGRAD’s additional machinery provides meaningful benefits beyond obtaining more informative tests and incorporating additional judge supervision.
Questions for authors
  1. Why is HardTests/HardTestGen neither discussed nor included as a baseline? It already investigates generating and validating challenging tests for use as RL verifiers, making it directly relevant to the claimed contribution. A compute-matched comparison using the same base model and training problems, with test-generation and verification costs included, is needed to establish what EviGRAD adds beyond improving test coverage. HardTestGen, ICLR 2026
  2. What gains are specifically attributable to the learned risk model and one-sided reward correction? The current comparisons do not adequately separate these contributions from access to additional tests, judge supervision, and repair/resampling. In particular, matched-budget downstream comparisons against random verification, seed-only selection, and standard reward recomputation using augmented tests are missing. The full method’s improvement over the degenerate-low-only branch also remains statistically unresolved in Table 15 (p = 0.19). Consequently, the experiments do not establish whether the additional design is necessary to obtain the reported gains.
  3. Could the authors reconcile Tables 4 and 14? Equation 4 defines an over-reward event as Aseed>xA_{\mathrm{seed}}>x and d≥τd\geq\tau. With the stated population of 29,900 candidates and τ=0.5\tau=0.5, Table 4 reports 1,040 positives for x=0x=0, whereas Table 14 reports 3,140 over-reward cases in the corresponding positive-advantage stratum. For x=1x=1, the corresponding counts are 400 versus 1,430. Under the stated definitions, these counts should agree. Do the tables use different candidate populations or label definitions, or are these reporting errors? This discrepancy directly affects the calibration analysis underlying the method’s motivation.
  4. Why only one old model? This just does not make sense to me.
Reviewer_Hf5qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviGRAD, a framework for improving training signals in code RLVR under limited verification budgets. Built on pass-rate rewards, it addresses two issues: misleading rewards caused by incomplete test coverage and uniformly low-reward groups that yield zero GRPO advantages. EviGRAD uses a reliability model to select high-risk groups, then employs LLM judges and a frozen test generator to produce validated counterexamples, lowering rewards only for originally full-score candidates that fail additional tests. A separate pathway handles degenerate-low groups through evidence-guided repair or resampling. Experiments report improvements over standard GRPO baselines and performance comparable to always-verify with approximately one-third of its additional verification overhead.

Critical strengths
  1. Addresses practical challenges in code RLVR. The paper addresses two practical limitations of code RLVR: incomplete test coverage can produce misleading rewards, while uniformly low rewards within a rollout group yield zero advantages and no learning signal for GRPO. EviGRAD targets both through evidence-based reward correction and a separate repair/resampling mechanism for degenerate-low groups.

  2. Evidence-based reward correction. The method requires validated, execution-based counterexamples before reducing rewards, rather than directly using LLM judge verdicts as rewards. This provides a useful safeguard against erroneous judge assessments.

  3. Promising verification efficiency. Compared with always-verify, EviGRAD reports similar performance point estimates while using approximately one-third of the additional verification calls and GPU-hours, demonstrating the potential practical value of selective verification.

Critical weaknesses
  1. Missing cost comparison with competitive baselines. EviGRAD repeatedly invokes powerful external judges—GLM-5.2, DeepSeek-V3.2, and occasionally Qwen3-Max—during training, introducing API expenses and potentially increasing wall-clock training time and GPU occupancy. Although §5.5 reports verification overhead relative to always-verify, it does not provide a corresponding cost comparison with competitive baselines such as CURE. This comparison is particularly important because EviGRAD’s improvement over CURE in Table 2 is modest and not statistically significant. Reporting the additional GPU-hours and API fees incurred by EviGRAD relative to CURE would clarify how much extra investment is required to obtain the reported performance gain.

  2. Limited comparison with recent related methods. CURE (2025) is the only external method-specific baseline in Table 2. More recent, closely related approaches should also be considered, such as “Learning to Generate Unit Test via Adversarial Reinforcement Learning” (ICLR 2026), which identifies CURE as its closest related work and reports stronger results on test-based code selection and evaluation fidelity. Including this method under a matched training and evaluation setup would help establish how EviGRAD compares with recent advances

  3. Unjustified restriction of reward correction to full-score candidates. Equation (6) only corrects candidates that pass all seed tests. However, partially passing candidates can also be over-rewarded and receive misleading positive advantages. Within selected groups, additional validated tests are executed on all candidates, yet partially passing candidates retain their original rewards even when they fail these tests. The paper neither discusses the rationale for this design choice nor provides an ablation comparing it with correction extended to partially passing candidates.

  4. Unclear applicability of reward correction across benchmarks and difficulty levels. Equation (6) can only affect candidates that pass all seed tests. Its coverage therefore depends on the policy’s seed-test full-score rate, which may vary substantially with task difficulty and test-suite coverage across benchmarks. On harder training tasks, fewer candidates may qualify for correction, potentially limiting the mechanism’s benefit. However, the paper does not report actual correction rates across difficulty levels or evaluate how this dependence affects its effectiveness across training benchmarks. The LiveCodeBench transfer evaluation measures downstream performance but does not establish whether the correction mechanism remains effective when applied during training on a different benchmark.

  5. Discussion of related reward-design approaches could be expanded It would be helpful to discuss recent work such as “Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generation”, which identifies and addresses cardinality bias in pass-rate based rewards, incorporates test-case difficulty into reward design, and reports reductions in the degenerate-group ratio. Since EviGRAD also builds on pass-rate rewards and addresses degenerate groups, clarifying the relationship and potential complementarity between these approaches would better contextualize its contribution.

Questions for authors

See weaknesses 1 to 5

Reviewer_dAfaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviGRAD, a selective verification framework for reinforcement learning with verifiable rewards (RLVR) in code generation. It uses a reliability model to identify potentially over-rewarded candidates and applies conservative reward corrections through validated counterexamples. Dynamic resampling further addresses groups with insufficient reward variation. Experiments on CodeContests and LiveCodeBench evaluate its effectiveness in improving reward reliability and code generation performance.

Critical strengths
  1. The paper provides useful insights into asymmetric reward errors in RLVR, supported by a systematic calibration study and statistical analysis.

  2. EviGRAD improves reward reliability and code generation performance while achieving results comparable to always-verify at substantially lower verification costs.

Critical weaknesses
  1. The reliability model relies on judge-derived features, but their availability before risk-based selection remains unclear. This raises concerns about the validity of the reported verification costs.

  2. The full method does not significantly outperform the degenerate-low-only variant (p=0.19p=0.19), making the contribution of the proposed risk-gated correction insufficiently convincing.

Questions for authors
  1. How are judge-derived features obtained before risk-based selection, and are their costs included in the verification budget?

  2. Why does Eq. (6) correct only full-reward candidates, and how does this affect group-relative advantages?

  3. What evidence supports the necessity of risk-gated correction, given the insignificant improvement over the degenerate-low-only variant (p=0.19p=0.19)?

  4. How is potential data leakage prevented when expanding the test set with tasks from the original training partition?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.