acceptodds
Under review as a conference paper at ICLR 2027

When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards

Abstract

Large Language Models (LLMs) have achieved remarkable advancements in reasoning capabilities empowered by Reinforcement Learning with Verifiable Rewards (RLVR). Nonetheless, RLVR intrinsically relies on ground-truth labels for reward computation, the acquisition of which is often prohibitively expensive in real-world scenarios. While unsupervised RLVR paradigms attempt to circumvent this by training on pseudo-labels, they are notoriously susceptible to training collapse. Moreover, different samples often exhibit varying annotation values. In this paper, we propose Reinforcement Learning with Active Verifiable Rewards (RLAVR), which actively acquires ground-truth labels for a small set of selected samples and integrates them with pseudo-labels, thereby stabilizing training dynamics and improving performance under limited annotation budgets. To identify valuable samples, we propose the Corrective Advantage Gap (CAG) metric and analyze the sample-level supervision value. Building on this, we introduce Correction-Aware Reliability Estimation for RLAVR (CARE), which translates the oracle CAG criterion into a practical pre-query acquisition policy to substantially improve training stability. Extensive experiments across diverse domains, model families, and model scales demonstrate the effectiveness and generality of our approach. Our code is available at https://anonymous.4open.science/r/CARE476-3B6B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
active learning, LLMs, RLAVR

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7NF2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RLAVR, a setting in which an online RLVR learner may request ground-truth labels for a small fraction of prompts (20% by default) and uses majority-vote pseudo-labels for the rest. The motivation comes from two preliminary studies. First, label-free training such as TTRL collapses because erroneous pseudo-labels reinforce themselves; masking the wrong pseudo-labels prevents the collapse. Second, prompts differ widely in how much they gain from correction. To quantify this, the authors define the Corrective Advantage Gap (CAG), the L2 distance between the advantage vectors induced by ground-truth labels and by pseudo-labels, and show that an oracle CAG-based selection outperforms random and uncertainty-based selection. Because CAG requires the label itself, the proposed method, CARE, predicts it with a two-stage classifier trained online on rollout features (hidden states, answer-cluster distribution, response length). Stage I keeps the most reliable pseudo-labeled prompts and downweights them by predicted reliability. Stage II ranks the remaining prompts by expected CAG, queries labels for the top ones, and drops the rest. The theory links CAG to an upper bound on gradient alignment and to one-step corrective benefit. Across Qwen3 1.7B/4B/8B and Phi4-mini on math and K&K logic tasks, CARE outperforms TTRL, TraPO, and active-learning baselines, approaches fully supervised GT training, and matches GT at a 40% label budget.

Critical strengths
  • Practical new setting. Combining sparse active labeling with pseudo-labels during online RLVR is a reasonable middle ground between fully supervised and label-free training, and the paper frames it clearly.
  • Informative preliminary analysis. The masking experiment shows convincingly that wrong pseudo-labels drive the collapse. The closed-form derivation of CAG in Appendix B.1 explains which prompts are worth labeling: those where the majority answer is wrong but correct answers exist among the minority responses.
  • Consistent empirical gains. Improvements hold across three model scales, two model families, and two task domains, with especially large gains on K&K. The paper also includes ablations, budget and hyperparameter sweeps, cost measurements, and extra baselines (CONST, MC-Dropout, ensembles, Entropy-FD/Prob-FD). Code is released.
  • Low overhead. The classifiers reuse rollout features and add roughly 8% training time.
Critical weaknesses
  • The gains seem to come mainly from filtering pseudo-labels, not from CAG-based acquisition. CARE outperforms the Oracle that selects samples using the true CAG (4B: 37.07 vs. 34.95; 8B: 43.59 vs. 39.51). This shows that most of the improvement comes from CARE’s other components: keeping only high-reliability pseudo-labels, downweighting them, and dropping all other prompts. Oracle-decay nearly matching GT points the same way. The key missing control is the Oracle CAG criterion combined with CARE’s filtering and dropping. Without it, the contribution of the acquisition criterion, which is the paper’s central claim, remains unclear.
  • CAG is simpler than the theory suggests. Under binary rewards, CAG is a deterministic function of three counts: the majority-cluster size, whether the majority answer is correct, and the number of correct minority responses. The NTK-based theory adds little to this. Theorem 6.1 becomes nearly vacuous at the measured condition numbers (minimum eigenvalue ratio around 10−210^{-2}), and Proposition 6.2 assumes an isotropic NTK, which the paper’s own measurements contradict.
  • The realism of the setting is under-examined. In the experiments, “annotation” is simulated by reading existing math answers, which are relatively cheap to obtain. Online labeling at every training step implies latency and human-in-the-loop costs that are not discussed. Budget accounting across epochs is also unclear: when a labeled prompt reappears in the second epoch, is it charged again?
  • Incomplete baselines. TraPO is already close at 8B (42.56 vs. 43.59). The paper does not compare against label-free methods designed to avoid collapse, such as RESTRAIN or Co-rewarding. The most relevant active query-selection method for RLVR (Yi et al., 2026) is cited but not evaluated.
Questions for authors

See above

1 comment

Thanks for the detailed response. I have raised my score

Reviewer_YhUQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to reduce erroneous self-reinforcement in reinforcement learning with pseudo-labels under a limited annotation budget. The authors define CAG as the discrepancy between the advantage vectors induced by ground-truth and pseudo-label rewards. CARE then predicts which pseudo-labels are reliable enough to retain and which examples are worth querying for ground-truth labels. The method combines active annotation, reliability weighting, and sample filtering, and is evaluated on mathematical reasoning and Knights-and-Knaves tasks across several model sizes. The reported results show improvements over several deployable baselines and performance close to fully supervised training in some configurations.

Critical strengths

The motivation behind CAG is compelling: scarce labels should be used to correct errors that would meaningfully change the learning signal, rather than simply targeting the most uncertain examples. CARE turns a criterion that requires ground truth into a prediction made before querying a label, giving the idea a practical implementation.

The ablations are another strength. The authors examine the effects of removing pseudo-labeled examples, removing the second stage, and adding filtering and weighting mechanisms to alternative selection strategies. These comparisons help explain why the combined method works and provide stronger support than final accuracy comparisons alone.

Critical weaknesses

My main concern about label efficiency is the meaning of the reported 20% budget. The budget is defined over prompt occurrences during training, while the data are traversed over multiple epochs. The manuscript does not clearly report how many distinct prompts are annotated or whether labels are reused when a prompt appears again. This is particularly important for the comparison with CONST, which selects a fixed fraction of distinct training examples in advance. It is therefore unclear whether the methods incur the same annotation cost.

The other concern is the stability of the main improvements. On the 8B model, CARE’s advantage over TraPO is approximately one percentage point, but the main RL experiments do not report variation across training seeds. The checkpoint selection procedure is also insufficiently specified. Both details could materially change the interpretation of these relatively small gains.

Questions for authors

How many label queries and distinct annotated prompts does each method use? Are labels cached and reused across epochs? Please clarify whether CARE and CONST have the same distinct-label budget.

How many training seeds underlie the main results, and what data are used to select checkpoints? Could you provide repeated-training results for CARE and the strongest deployable baselines to establish whether the roughly one-point improvements are consistent?

1 comment

Thanks for the clarifications. The budget accounting and the matched-budget experiment on 1.7B address my first question. Two points remain:

The new multi-seed results cover only 1.7B, while my concern was the ~1-point gap over TraPO at 8B. Could you report multi-seed results on 4B or 8B, or at least per-seed values for the 1.7B runs?

Checkpoints in Table 1 are selected on the test benchmarks. Could you report final-checkpoint results, or results with a held-out validation set, for CARE and TraPO?

Reviewer_UAopOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies active label acquisition for online reinforcement learning with verifiable rewards under a limited annotation budget. It introduces the Corrective Advantage Gap:

CAG⁡(x)=∥Ax⋆−A~x∥2,\operatorname{CAG}(x)=\|A_x^\star-\widetilde A_x\|_2,

which measures the discrepancy between advantage vectors computed using ground-truth labels and majority-vote pseudo-labels.

Since CAG requires ground truth, the proposed CARE framework uses two predictors. The first estimates pseudo-label reliability and retains reliable samples for unsupervised updates. The second predicts the number of correct non-majority responses among the remaining samples, uses this distribution to estimate CAG, and selects high-value samples for annotation. Other unreliable samples are discarded. Acquired labels supervise both policy training and subsequent predictor updates.

The paper provides conditional analyses connecting CAG to gradient alignment and one-step corrective benefit. Experiments cover mathematical and Knights-and-Knaves reasoning using Qwen3 models at three scales and Phi4-mini-instruct. With a reported 20% annotation budget, CARE achieves average mathematical reasoning scores of 25.67, 37.07, and 43.59 for the three Qwen3 models, compared with fully supervised scores of 26.69, 41.14, and 44.59.

Critical strengths
  1. The acquisition criterion is directly connected to the training signal being corrected. CAG measures how replacing a pseudo-label changes the current advantage vector, rather than treating uncertainty as a sufficient proxy for annotation value. CARE then approximates this criterion without accessing ground truth before acquisition. Appendix B provides a closed-form justification for the conditional decomposition and a nearly capacity-matched comparison against a unified predictor. Together, these support the specific design of the cascade.

  2. The empirical evidence goes beyond comparisons against collapsing pseudo-label baselines. Table 1 and Tables 12–13 cover multiple model scales, model families, and mathematical and logical reasoning tasks. Figure 3, Figure 12, and Appendices D.5–D.6 include training on acquired labels alone, controls using comparable filtering and downweighting, and alternative uncertainty-based acquisition strategies. For example, CARE achieves 37.07 on the 4B mathematical setting, compared with 33.82 for Entropy-FD and 33.17 for Prob-FD. These controls provide evidence that the gains are not explained solely by discarding unreliable pseudo-labels.

Critical weaknesses
  1. The annotation-efficiency claim depends on an insufficiently specified budget and evaluation protocol.

    Equation (3) counts queries relative to prompt occurrences during training, while training revisits the dataset. It is unclear whether acquired labels are cached, whether repeated queries incur additional cost, and how many unique labeled problems each method ultimately accesses. Algorithm 2 appears to retain the annotation ratio during the 20-step cold start, so there is no basis for assuming fully labeled warm-up batches, but the initial acquisition procedure still needs clarification. The optimization objective also refers to the best checkpoint, whereas Table 1 does not clearly specify checkpoint selection or an independent validation protocol. The main RL results do not explicitly report training-seed counts or variability; the four-seed results in Appendix B concern the offline predictor comparison. These details directly affect the interpretation and reproducibility of the reported 20% annotation advantage.

  2. The theoretical statements require narrower conditions than the main presentation makes explicit.

    The proof of Theorem 6.1 uses ∥A∥22=∥A^∥22=G\lVert A\rVert_2^2 = \lVert \widehat{A}\rVert_2^2 = G, which requires non-degenerate reward groups and exact advantage normalization. A nonzero CAG alone does not exclude an all-zero ground-truth reward group, for which the ground-truth gradient vanishes and its cosine alignment is undefined. Such groups are explicitly included in Appendix B. The response score on page 7 should also use ∇θlog⁡πθ\nabla_{\theta}\log \pi_{\theta}. Furthermore, the first-order corrective benefit in Proposition 6.2 contains the sample-dependent factor κiso(x)\kappa_{\mathrm{iso}}(x). Local isotropy for each sample does not by itself establish that CAG ranks corrective benefit across samples with different factors. The positive-definiteness check in Appendix D.3 does not establish isotropy. These issues limit the theoretical interpretation, although they do not invalidate the empirical contribution.

Questions for authors
  1. Clarification: What exactly is counted in the 20% annotation budget? Please report query counts, unique annotated problems, label-caching rules, and the cold-start acquisition procedure for each method. Were the predictors used in the main online experiments initialized using any of the offline labeled data from Appendix B? This information is necessary to establish comparable access to ground truth.

  2. Clarification and assessment-critical validation: Does Table 1 report final checkpoints or selected checkpoints? How was p2p_2 chosen, and was an independent validation set used? How many training seeds were run for the main RL experiments, and how stable are CARE’s improvements over TraPO and Random-mask across runs? These answers would determine how much confidence to place in the smaller performance differences.

  3. Theoretical clarification: Can Theorem 6.1 explicitly require non-degenerate groups and exact normalization, and can the cross-sample ranking interpretation be restricted to settings with comparable κiso(x)\kappa_{\mathrm{iso}}(x)? What fraction of the actual rollout groups satisfies the relevant conditions? This would clarify how much of CARE’s practical behavior is explained by the theory.

Comments on the paper

1 comment

Dear Reviewers,

We are now halfway through the discussion phase. The authors have provided their responses to your reviews. If their responses have not adequately addressed your concerns, please take this opportunity to raise any remaining questions or concerns.

I encourage you to engage with the authors as soon as possible to ensure a productive discussion.

Many thanks,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.