acceptodds
Withdrawn from ICLR 2027

AUDIT-FIRST VAPO: RISK-CERTIFIED SELECTIVE UPDATES UNDER IMPERFECT VERIFICATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Verifier-guided reinforcement learning (RLVR) improves reasoning models by using automated evaluators to provide training signals, but imperfect verifiers can introduce harmful updates that degrade policy optimization. Existing approaches mainly optimize verifier quality or aggregate noisy feedback, without explicitly controlling the risk of accepted policy updates under limited verification budgets. We introduce Audit-First VAPO, a risk-certified selective update framework that decomposes verifier-guided optimization into update-direction certification and update-magnitude control. VAPO first audits candidate updates through sequential risk certification, selectively accepting, appealing, or abstaining from updates based on finite-sample guarantees, and then applies bounded policy updates only after certification. This design separates the decision of whether an update is safe from how strongly it should be applied, enabling robust learning under imperfect verification. We evaluate VAPO on two reasoning benchmarks with two language models against static RLVR, matched-random selection, confidence thresholding, noise-corrected RLVR, and verifier-augmentation baselines. At a target selected-risk level of , VAPO achieves 74.1% task accuracy with selected harmful-update risk of 0.0697, coverage of 0.4125, and relative verifier cost of . Under matched coverage and update magnitude, VAPO reduces selected risk by 0.0260 compared with matched random selection, with a paired 95% confidence interval of . Across asymmetric, confidence-dependent, and correlated verifier noise settings, the risk certificate remains satisfied in 57 of 60 independent runs. These results demonstrate that audit-first selective optimization provides a principled approach for reliable RLVR training when verification signals are costly and imperfect.

Primary area:
datasets and benchmarks
Keywords:
Reinforcement Learning, RL with Verifiers, Risk Certification, Selective Updates, Safe Policy Optimization, Abstention Mechanisms, Distribution-Free Guarantees, Imperfect Verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_o2PEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Audit-First VAPO, which uses confidence and agreement signals to decide whether to accept a grader’s feedback, ask another grader, or skip a training update. It chooses decision thresholds on separate labelled examples to limit incorrect feedback, though how the main experiments produce the input scores is unclear. On GSM8K, it reports 74.1% answer accuracy while accepting 41% of proposed updates, about 7% of which use incorrect feedback.

Critical strengths
  • Checking whether feedback is trustworthy before deciding how strongly to train on it is a useful idea. Making an update smaller does not fix feedback that rewards the wrong answer.
  • The comparisons help test whether the method chooses better feedback. Random selection uses a similar number and size of updates, while comparisons with other second-grader strategies help assess whether selectively asking for another opinion is worth the cost.
Critical weaknesses
  • It is unclear how the main experiments calculate the scores used to accept or reject feedback. The appendix gives a formula for an older simulated setup, but does not clearly explain how the actual graders produce confidence and other signals. Specify those inputs and how they are combined so the method can be reproduced.

  • The claimed error guarantee needs explaining. If SVAMP’s 200 calibration examples are the samples used in the formula, no updates should pass certification at the stated 8% target, yet the paper reports accepting 40%. Show the sample counts and calculations, and explain why a guarantee calculated on calibration examples also applies to later training updates.

Questions for authors

See weaknesses

Reviewer_tMnqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes RC-VAPO, a reinforcement learning method that aims to reduce “harmful” policy updates caused by unreliable verifier feedback. The method selectively accepts, rejects, or requests additional verification of training signals based on their estimated reliability. This is combined with a calibration-based procedure aiming to provide guarantees on the rate of incorrect feedback among accepted updates. The approach is evaluated on GSM8K using Qwen3.5-0.8B, improving accuracy by around 4% while accepting only ~40% of proposed updates.

Critical strengths
  • Generally filtering out unreliable verifier feedback is a promising direction in RLVR and the evaluation here shows an instance of this.
Critical weaknesses
  1. The majority of the evaluation is very limited, with results on a single, extremely small, model (0.8B). The improvement also comes with a noticeable runtime cost, is only evaluated on a single, largely saturated benchmark. Additionally, the applied noise in the verification procedure is unclear to me (which is especially important for "imperfect verifiers" as shown in [1]). Overall the experimental result section has left me with little confidence that the method generalizes or even provides the claimed benefits consistently. This seems to be confirmed by the very short paragraph on Llama-3.2-3B which has RC-VAPO at 67% where periodic renewal has 75% (but to be honest given the point below it not 100% certain to me how the interprete the setup here).
  2. The presentation could be significantly improved and contains some absolute novelties for me, including a "Evidence ledger for the current manuscript" (Table 10) which seems to be the used Agents artifact for showing which experiments where run in which state (with some still just being "ready" and "not run") or what seems to a full prompt (actually one of several)

"Report both all-proposal risk R and conditional selected risk R selected. Their denominators must accompany the counts. The aligned scalar response U remains a replay diagnostic, while learner task accuracy is measured separately after applying the corresponding update rule." (L1203ff).

More importantly, already in the main paper it is obvious that the manuscript has not receipt appropriate attention for this venue and falls clearly below the bar for acceptance, many sections are hard to read with unclear/non-defined terminology. Parsing this work was not pleasant.

[1] Egashira et al., Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR (COLM 2026)

Questions for authors
  • Which verifier and which types of verifier errors do you evaluate? In particular, how does the method deal with structured errors that the verifier makes with high confidence?
  • Can you repeat the same experiments on other models and RLVR settings?
Reviewer_AsTwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new control mechanism for RLVR's verification process. This is useful in fine-tuning LLMs with RLVR, especially when the verifier is faulty; it can provide incorrect signals and make the LLM learn something harmful. The paper reports evaluation on the GSM8K dataset and Qwen3.5-8B.

Critical strengths

(1) It addresses a timely and important challenge in RLVR (2) The decision process appears to be quite lightweight, as it only makes some threshold comparisons.

Critical weaknesses

(1) If the central claim is addressing faulty verifiers, I would expect a lot more experiments covering a variety of verifiers on different tasks. The only variation seems to be 5 synthetic noise models applied on a single verifier.

(2) Most of the evaluation is focused on one Qwen model and one dataset; the table that shows transfer to llama (Table 30) lacks the baselines.

(3) Since this paper is about working with faulty verifiers, the natural experimental design choice that comes to mind is Model x Verifier. E.g., in dataset x, how does the method perform with a specific LLM and a verifier?

(4) Details on the verifier are quite limited (only a small paragraph in the Appendix)

(5) There appears to be some work that is related to this but not compared (e.g. Jung et al. Trust or Escalate ICLR 2025)

Questions for authors
  1. What exactly is the primary verifier in the main GSM8K experiments (rule-based matcher, model, prompt, checkpoint), and what is the secondary appeal verifier? What are their measured false-positive, false-negative, and joint-error rates on the calibration split?

  2. Is the verifier noise (Table 28) injected synthetically on top of a near-exact GSM8K checker? If so, can you report results with at least two real primary/secondary verifier pairings (e.g., rule-based + LLM judge)?

  3. Since admission after appeal requires the secondary verifier to agree with the primary sign, appeals can only veto and never recover a false negative. Some papers handle false negative as well (Cai et al. in rel work). How does RC-VAPO perform when false negatives dominate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.