- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a new research setting for studying the robustness of RLVR to incorrect reference answers and proposes Online Label Refinement, which replaces suspected noisy labels during training based on majority-answer confidence growth and historical consistency. As a researcher working on weak supervision, I appreciate how this work revisits the classical problem of label noise at a time when LLMs are reshaping the learning paradigm and develops a simple, practical solution for robust RLVR under noisy supervision.
Critical strengths
- The paper studies an important yet relatively underexplored problem---the sensitivity of RLVR to noisy labels. I particularly like the distinction between rollout-inactive and rollout-active noise, which provides an intuitive explanation of why different types of incorrect supervision affect policy optimization differently.
- The method is simple and introduces little additional computation. The theoretical discussion provides useful intuition and generally consistent empirical support for the proposed design.
- The experimental setup is reasonably comprehensive. The evaluation covers multiple noise ratios and model families. I also appreciate the inclusion of real-world BIRD Text2SQL dataset, as its naturally occurring noise complements the controlled synthetic-noise experiments and improves the practical relevance of the study.
Critical weaknesses
- I may have missed this analysis, but the slope in Equation (4) appears to aggregate majority frequencies even when the majority answer changes across rounds. Although the historical-consistency gate and current ablations suggest that the heuristic works in practice, showing the trajectories of individual candidate answers would make the underlying interpretation more convincing.
- The dynamic active-noise setting is a useful and relatively demanding stress test. That said, it is not identical to fixed annotation noise. The fixed-noise and naturally noisy BIRD experiments already help address this concern; extending the fixed active-noise results from 50% to, for example, 30% and 70% would further clarify whether the conclusions remain stable across noise levels.
- The reported improvements appear consistent across settings, but it is unclear whether the main comparisons include independent training repetitions. A three-seed comparison between the method and GRPO for one representative model at 50% noise, under both active and inactive noise, would be sufficient to establish the statistical reliability of the gains.
- Although this setup differs from conventional noisy-label classification, the paper could clarify the unique technical challenges of label noise in RLVR and why existing noise-robust methods are insufficient. This would better position the problem and facilitate follow-up work.
Questions for authors
As discussed in the weaknesses above, I would appreciate clarification on the following points:
- Since Eq. (4) may aggregate different majority answers across rounds, does the same candidate answer actually become more confident over time?
- The fixed active-noise experiment only uses a 50% noise ratio. What happens at lower and higher ratios, such as 30% and 70%?
- How stable are the main improvements across training runs? Results from three seeds for the method and GRPO in representative active and inactive settings would be helpful.
- What makes label noise in RLVR technically different from conventional noisy-label classification, and why is it more challenging?
4 comments
W3/Q3: How stable are the main improvements across training runs? Results from three seeds for the method and GRPO in representative active and inactive settings would be helpful.
Response: Thank you for this helpful suggestion. We additionally run GRPO and OLR with three training seeds under representative active and inactive noise settings using Qwen3-4B and . All data, hyperparameters, and evaluation protocols remain identical across runs. We report the mean and standard deviation below.
Noise Method AIME24 AIME25 AMC MATH-500 Minerva Olympiad Avg. ARC-c GPQA$^\star$ MMLU-Pro Avg. Active GRPO Active OLR Inactive GRPO Inactive OLR OLR consistently outperforms GRPO across the three training runs under both active and inactive noise. Under active noise, OLR improves the mean ID and OOD accuracy by 6.4 and 4.8 points, respectively; under inactive noise, the corresponding gains are 8.8 and 3.8 points. The small standard deviations across runs indicate that these improvements are stable rather than driven by a single favorable seed. We have added the three-run mean and standard-deviation results to Appendices C.2 and D.2 (Table 17) of the revised paper and highlighted them in blue.
W4/Q4: What makes label noise in RLVR technically different from conventional noisy-label classification, and why is it more challenging?
Response:
Thank you for this fundamental question.The difference lies in how a noisy label enters the optimization process.In conventional classification, the output space is a fixed set of classes, and every mislabeled example directly produces a supervised gradient whenever it is visited. Its corruption is therefore static and continuously active. This structure allows existing methods to use class probabilities, per-example losses, or label-transition matrices to identify and suppress noise.RLVR has a fundamentally different feedback mechanism. The model generates open-ended reasoning trajectories and receives a sparse outcome reward only after sampling. If none of the rollouts matches an incorrect reference, all rewards are zero and the group-relative advantage vanishes. The noisy label is then inactive, wasting the prompt and rollout budget without directly updating the policy. If the model can generate that incorrect answer, the label becomes active: incorrect trajectories receive positive advantages and are reinforced, creating an on-policy confirmation loop.
Consequently, whether a noisy label affects training depends jointly on the current policy, prompt, rollout budget, and training stage. The same label may move between inactive and active states as the policy evolves. This makes RLVR noise both
dynamic and endogenous: the policy determines which incorrect labels become reachable, while those labels simultaneously reshape the policy. Moreover, the open-ended sequence space provides neither a fixed class posterior nor a predefined set of alternative labels from which a correction can be selected.This distinction motivates the central perspective of our work. We characterize the two RLVR-specific failure modes, identify Early Correctness Coherence as a window in which reliable answers can still emerge, and design OLR to recover them using rollout concentration and historical consistency. Thus, OLR addresses noisy supervision as a changing interaction between labels, rollouts, and policy optimization, rather than treating it as conventional classification noise transferred to a larger model. We have added a detailed explanation of why noisy-label RLVR differs technically from conventional noisy-label classification to Section 2.1 of the revised paper and highlighted it in blue.
Thank you for the detailed response. The authors addressed most of my concerns, so I keep my score.
Thank you so much for your thoughtful follow-up and continued support! Your suggestions really helped us make this a better paper.
Thank you for recognizing our work and for your careful review. We provide detailed responses below.
W1/Q1: Since Eq. (4) may aggregate different majority answers across rounds, does the same candidate answer actually become more confident over time?
W2/Q2: The fixed active-noise experiment only uses a 50% noise ratio. What happens at lower and higher ratios, such as 30% and 70%?