- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RLAVR, a setting in which an online RLVR learner may request ground-truth labels for a small fraction of prompts (20% by default) and uses majority-vote pseudo-labels for the rest. The motivation comes from two preliminary studies. First, label-free training such as TTRL collapses because erroneous pseudo-labels reinforce themselves; masking the wrong pseudo-labels prevents the collapse. Second, prompts differ widely in how much they gain from correction. To quantify this, the authors define the Corrective Advantage Gap (CAG), the L2 distance between the advantage vectors induced by ground-truth labels and by pseudo-labels, and show that an oracle CAG-based selection outperforms random and uncertainty-based selection. Because CAG requires the label itself, the proposed method, CARE, predicts it with a two-stage classifier trained online on rollout features (hidden states, answer-cluster distribution, response length). Stage I keeps the most reliable pseudo-labeled prompts and downweights them by predicted reliability. Stage II ranks the remaining prompts by expected CAG, queries labels for the top ones, and drops the rest. The theory links CAG to an upper bound on gradient alignment and to one-step corrective benefit. Across Qwen3 1.7B/4B/8B and Phi4-mini on math and K&K logic tasks, CARE outperforms TTRL, TraPO, and active-learning baselines, approaches fully supervised GT training, and matches GT at a 40% label budget.
Critical strengths
- Practical new setting. Combining sparse active labeling with pseudo-labels during online RLVR is a reasonable middle ground between fully supervised and label-free training, and the paper frames it clearly.
- Informative preliminary analysis. The masking experiment shows convincingly that wrong pseudo-labels drive the collapse. The closed-form derivation of CAG in Appendix B.1 explains which prompts are worth labeling: those where the majority answer is wrong but correct answers exist among the minority responses.
- Consistent empirical gains. Improvements hold across three model scales, two model families, and two task domains, with especially large gains on K&K. The paper also includes ablations, budget and hyperparameter sweeps, cost measurements, and extra baselines (CONST, MC-Dropout, ensembles, Entropy-FD/Prob-FD). Code is released.
- Low overhead. The classifiers reuse rollout features and add roughly 8% training time.
Critical weaknesses
- The gains seem to come mainly from filtering pseudo-labels, not from CAG-based acquisition. CARE outperforms the Oracle that selects samples using the true CAG (4B: 37.07 vs. 34.95; 8B: 43.59 vs. 39.51). This shows that most of the improvement comes from CARE’s other components: keeping only high-reliability pseudo-labels, downweighting them, and dropping all other prompts. Oracle-decay nearly matching GT points the same way. The key missing control is the Oracle CAG criterion combined with CARE’s filtering and dropping. Without it, the contribution of the acquisition criterion, which is the paper’s central claim, remains unclear.
- CAG is simpler than the theory suggests. Under binary rewards, CAG is a deterministic function of three counts: the majority-cluster size, whether the majority answer is correct, and the number of correct minority responses. The NTK-based theory adds little to this. Theorem 6.1 becomes nearly vacuous at the measured condition numbers (minimum eigenvalue ratio around ), and Proposition 6.2 assumes an isotropic NTK, which the paper’s own measurements contradict.
- The realism of the setting is under-examined. In the experiments, “annotation” is simulated by reading existing math answers, which are relatively cheap to obtain. Online labeling at every training step implies latency and human-in-the-loop costs that are not discussed. Budget accounting across epochs is also unclear: when a labeled prompt reappears in the second epoch, is it charged again?
- Incomplete baselines. TraPO is already close at 8B (42.56 vs. 43.59). The paper does not compare against label-free methods designed to avoid collapse, such as RESTRAIN or Co-rewarding. The most relevant active query-selection method for RLVR (Yi et al., 2026) is cited but not evaluated.
Questions for authors
See above
Thanks for the detailed response. I have raised my score