acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Learn to Reason Robustly under Noisy Labels?

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains reasoning models that rely on abundant perfect supervision, but its vulnerability to unavoidable noisy labels remains underexplored. Through empirical validation, we find that the vanilla RLVR method demonstrates degraded performance for two reasons: (i) data waste: when the LLM itself cannot roll out the noisy label, many data points are overlooked in training; (ii) policy misguidance: if the wrong label is rolled out, the policy will be guided toward the wrong direction, hurting model training. Fortunately, by inspecting the training dynamics, we identify an Early Correctness Coherence phenomenon: in the early stage of training, the probability of correct answers rises similarly for clean and noisy samples. Inspired by this, we develop a novel Online Label Refinement (OLR) algorithm that performs self-consistent label correction when a sample's predicted labels are historically stable and increasingly confident. Evaluated on six in-distribution (ID) math benchmarks and three out-of-distribution (OOD) tasks under 0.1–0.9 noise ratios, OLR achieves average gains of 3.6% (ID) / 3.3% (OOD) under inactive noise and 3.9% (ID) / 4.6% (OOD) under active noise.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Noisy Labels, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_thxAOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new research setting for studying the robustness of RLVR to incorrect reference answers and proposes Online Label Refinement, which replaces suspected noisy labels during training based on majority-answer confidence growth and historical consistency. As a researcher working on weak supervision, I appreciate how this work revisits the classical problem of label noise at a time when LLMs are reshaping the learning paradigm and develops a simple, practical solution for robust RLVR under noisy supervision.

Critical strengths
  1. The paper studies an important yet relatively underexplored problem---the sensitivity of RLVR to noisy labels. I particularly like the distinction between rollout-inactive and rollout-active noise, which provides an intuitive explanation of why different types of incorrect supervision affect policy optimization differently.
  2. The method is simple and introduces little additional computation. The theoretical discussion provides useful intuition and generally consistent empirical support for the proposed design.
  3. The experimental setup is reasonably comprehensive. The evaluation covers multiple noise ratios and model families. I also appreciate the inclusion of real-world BIRD Text2SQL dataset, as its naturally occurring noise complements the controlled synthetic-noise experiments and improves the practical relevance of the study.
Critical weaknesses
  1. I may have missed this analysis, but the slope in Equation (4) appears to aggregate majority frequencies even when the majority answer changes across rounds. Although the historical-consistency gate and current ablations suggest that the heuristic works in practice, showing the trajectories of individual candidate answers would make the underlying interpretation more convincing.
  2. The dynamic active-noise setting is a useful and relatively demanding stress test. That said, it is not identical to fixed annotation noise. The fixed-noise and naturally noisy BIRD experiments already help address this concern; extending the fixed active-noise results from 50% to, for example, 30% and 70% would further clarify whether the conclusions remain stable across noise levels.
  3. The reported improvements appear consistent across settings, but it is unclear whether the main comparisons include independent training repetitions. A three-seed comparison between the method and GRPO for one representative model at 50% noise, under both active and inactive noise, would be sufficient to establish the statistical reliability of the gains.
  4. Although this setup differs from conventional noisy-label classification, the paper could clarify the unique technical challenges of label noise in RLVR and why existing noise-robust methods are insufficient. This would better position the problem and facilitate follow-up work.
Questions for authors

As discussed in the weaknesses above, I would appreciate clarification on the following points:

  1. Since Eq. (4) may aggregate different majority answers across rounds, does the same candidate answer actually become more confident over time?
  2. The fixed active-noise experiment only uses a 50% noise ratio. What happens at lower and higher ratios, such as 30% and 70%?
  3. How stable are the main improvements across training runs? Results from three seeds for the method and GRPO in representative active and inactive settings would be helpful.
  4. What makes label noise in RLVR technically different from conventional noisy-label classification, and why is it more challenging?
4 comments

Thank you for recognizing our work and for your careful review. We provide detailed responses below.

W1/Q1: Since Eq. (4) may aggregate different majority answers across rounds, does the same candidate answer actually become more confident over time?

Response: Thank you for this careful question. Eq. (4) is intended to measure increasing rollout concentration, rather than the confidence trajectory of one fixed candidate. We explicitly clarify this immediately after Eq. (4): the winning answer may differ across rounds, so a positive slope alone is insufficient. Candidate identity is handled separately by Eq. (5), which requires the current majority answer to match the most frequent majority answer in that sample’s history before relabeling.

Appendix Figure 8 provides empirical support: on noisy samples, the fraction whose majority prediction is correct increases over training, while predictions matching the noisy label or neither answer decrease. Figures 9 and 10(e) further show that removing historical consistency reduces selection accuracy by approximately 20%, whereas the complete criteria achieve over 90% accuracy and approach 100%. Thus, OLR does not infer candidate stability from Eq. (4); it combines rollout concentration with an explicit identity-consistency check. We have clarified the distinct roles of rollout concentration and candidate identity immediately after Eqs. (4)–(5) in Section 2.3, with supporting analyses in Appendices D.4–D.5, and highlighted the revision in blue.

W2/Q2: The fixed active-noise experiment only uses a 50% noise ratio. What happens at lower and higher ratios, such as 30% and 70%?

Response: Thank you for this helpful suggestion. We additionally evaluate fixed active noise at 30% and 70%, using exactly the same protocol as the existing 50% experiment: incorrect answers are generated once by the base model and remain fixed throughout training.

Results with fixed active-noise labels at different noise ratios. All values are accuracy (%).

Noise ratioMethodAIME24AIME25AMCMATH-500MinervaOlympiadAvg.ARC-cGPQA$^\star$MMLU-ProAvg.
30%GRPO16.712.150.280.041.046.041.085.631.856.057.8
30%OLR21.716.752.082.641.548.343.887.434.357.759.8
50%GRPO12.510.047.176.435.341.337.182.028.353.854.7
50%OLR20.815.850.182.036.848.342.385.032.357.358.2
70%GRPO9.67.544.772.832.340.134.575.027.353.752.0
70%OLR17.112.948.579.038.244.340.082.631.355.656.5

OLR improves over GRPO at all three noise ratios, with ID gains of 2.8, 5.2, and 5.5 points and OOD gains of 2.0, 3.5, and 4.5 points at 30%, 50%, and 70%, respectively. These results show that OLR remains effective across different levels of persistent fixed-label corruption, with larger gains as the noise ratio increases. We have added fixed active-noise results at 30%, 50%, and 70% to Appendix D.2 (Table 15) of the revised paper and highlighted them in blue.

W3/Q3: How stable are the main improvements across training runs? Results from three seeds for the method and GRPO in representative active and inactive settings would be helpful.

Response: Thank you for this helpful suggestion. We additionally run GRPO and OLR with three training seeds under representative active and inactive noise settings using Qwen3-4B and ρ=0.5\rho=0.5. All data, hyperparameters, and evaluation protocols remain identical across runs. We report the mean and standard deviation below.

NoiseMethodAIME24AIME25AMCMATH-500MinervaOlympiadAvg.ARC-cGPQA$^\star$MMLU-ProAvg.
ActiveGRPO10.6±1.310.6\pm1.38.5±0.48.5\pm0.445.6±0.945.6\pm0.974.8±0.374.8\pm0.333.3±1.133.3\pm1.140.2±0.640.2\pm0.635.5±0.735.5\pm0.774.2±1.474.2\pm1.424.5±0.524.5\pm0.548.6±0.848.6\pm0.849.1±0.649.1\pm0.6
ActiveOLR20.1±1.620.1\pm1.615.6±0.715.6\pm0.749.9±1.049.9\pm1.081.2±0.481.2\pm0.436.6±0.836.6\pm0.848.0±1.248.0\pm1.241.9±0.841.9\pm0.878.2±1.178.2\pm1.128.6±0.928.6\pm0.954.9±0.554.9\pm0.553.9±0.753.9\pm0.7
InactiveGRPO12.3±0.612.3\pm0.65.2±0.35.2\pm0.345.5±1.245.5\pm1.275.8±0.775.8\pm0.735.1±0.435.1\pm0.442.7±0.942.7\pm0.936.1±0.536.1\pm0.586.5±0.486.5\pm0.429.0±1.129.0\pm1.156.1±0.656.1\pm0.657.2±0.557.2\pm0.5
InactiveOLR21.5±1.221.5\pm1.218.5±1.518.5\pm1.553.7±0.553.7\pm0.584.0±0.384.0\pm0.342.5±0.942.5\pm0.949.2±0.749.2\pm0.744.9±0.644.9\pm0.687.0±0.287.0\pm0.238.5±1.338.5\pm1.357.5±0.457.5\pm0.461.0±0.661.0\pm0.6

OLR consistently outperforms GRPO across the three training runs under both active and inactive noise. Under active noise, OLR improves the mean ID and OOD accuracy by 6.4 and 4.8 points, respectively; under inactive noise, the corresponding gains are 8.8 and 3.8 points. The small standard deviations across runs indicate that these improvements are stable rather than driven by a single favorable seed. We have added the three-run mean and standard-deviation results to Appendices C.2 and D.2 (Table 17) of the revised paper and highlighted them in blue.

W4/Q4: What makes label noise in RLVR technically different from conventional noisy-label classification, and why is it more challenging?

Response:
Thank you for this fundamental question. The difference lies in how a noisy label enters the optimization process. In conventional classification, the output space is a fixed set of classes, and every mislabeled example directly produces a supervised gradient whenever it is visited. Its corruption is therefore static and continuously active. This structure allows existing methods to use class probabilities, per-example losses, or label-transition matrices to identify and suppress noise.

RLVR has a fundamentally different feedback mechanism. The model generates open-ended reasoning trajectories and receives a sparse outcome reward only after sampling. If none of the rollouts matches an incorrect reference, all rewards are zero and the group-relative advantage vanishes. The noisy label is then inactive, wasting the prompt and rollout budget without directly updating the policy. If the model can generate that incorrect answer, the label becomes active: incorrect trajectories receive positive advantages and are reinforced, creating an on-policy confirmation loop.

Consequently, whether a noisy label affects training depends jointly on the current policy, prompt, rollout budget, and training stage. The same label may move between inactive and active states as the policy evolves. This makes RLVR noise both dynamic and endogenous: the policy determines which incorrect labels become reachable, while those labels simultaneously reshape the policy. Moreover, the open-ended sequence space provides neither a fixed class posterior nor a predefined set of alternative labels from which a correction can be selected.

This distinction motivates the central perspective of our work. We characterize the two RLVR-specific failure modes, identify Early Correctness Coherence as a window in which reliable answers can still emerge, and design OLR to recover them using rollout concentration and historical consistency. Thus, OLR addresses noisy supervision as a changing interaction between labels, rollouts, and policy optimization, rather than treating it as conventional classification noise transferred to a larger model. We have added a detailed explanation of why noisy-label RLVR differs technically from conventional noisy-label classification to Section 2.1 of the revised paper and highlighted it in blue.

Thank you for the detailed response. The authors addressed most of my concerns, so I keep my score.

Thank you so much for your thoughtful follow-up and continued support! Your suggestions really helped us make this a better paper.

Reviewer_E1rQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies RLVR under noisy reference labels. The authors first categorize noisy labels into inactive ones, which the policy cannot roll out and thus waste the sample, and active ones, which the policy can roll out and thus misguide training. Based on the observation that the probability of the correct answer rises similarly for clean and noisy samples in early training, the authors propose Online Label Refinement (OLR), which replaces the reference label with the majority answer of the rollouts when the majority pass rate keeps increasing and the majority answer stays consistent over past epochs. The proposed method is tested on various benchmarks under different noise ratios and noise types, and compared with label-free methods and classical noisy label methods.

Critical strengths
  • The proposed method is simple, and is supported by a fair amount of experiments and ablations.
  • The writing is clear and easy to follow.
Critical weaknesses
  • One of my main concerns is the lack of appropriate baselines. To my understanding, the compared methods are either label-free methods or classification-oriented noisy label methods applied at the token level, and none of them is designed for this setting. Since an inactive noisy label is practically the same as a missing label, the semi-supervised RLVR methods cited in Appendix A (e.g., TraPO) seem to be natural baselines. A simple baseline that relabels only the samples whose reward stays zero during the warm-up period would also be informative. Please let me know if I have overlooked such comparisons.
  • The method appears to be closely related to SELFIE (Song et al., 2019), which is already cited in the paper. To my understanding, SELFIE also replaces labels with the most frequent prediction when the prediction history is consistent after a warm-up phase. It would be helpful if the authors could clearly articulate how OLR differs from such label refurbishing methods. Similarly, the Early Correctness Coherence phenomenon seems closely related to the early-learning effect known in the noisy label literature.
  • Some claims around Table 1 should be stated more carefully. The text acknowledges that TTRL is more robust on the OOD tasks under active noise, but then says that OLR outperforms the best unsupervised method by 3.3% on OOD tasks, which holds only under inactive noise. Under active noise, OLR is also below Confidence Penalty on the OOD average.
  • I am also not sure whether the comparison is fairly executed. The hyperparameters of OLR are tuned on the validation set (Tables 9 and 10), whereas the method-specific hyperparameters of the baselines follow their original papers (Appendix C.3).
  • The theoretical results seem to follow almost directly from their assumptions. In Theorem 2.3, the positive coupling and the positive drift are assumed, and Theorem 2.4 appears to be an identity from the law of total probability. It would be helpful to clarify what the theorems add beyond the empirical observation.
Questions for authors

How realistic is the considered setting? Inactive noise is injected by replacing the label with a string that can never be matched, and most results use noise ratios up to 0.9, while the paper itself notes that curated datasets usually have a low noise ratio. It would be helpful if the authors could provide some evidence on the actual noise ratio and the active versus inactive split in commonly used RLVR datasets.

6 comments

Thank you for the careful review and valuable suggestions. We respond point by point below.

W1: One of my main concerns is the lack of appropriate baselines. To my understanding, the compared methods are either label-free methods or classification-oriented noisy label methods applied at the token level, and none of them is designed for this setting. Since an inactive noisy label is practically the same as a missing label, the semi-supervised RLVR methods cited in Appendix A (e.g., TraPO) seem to be natural baselines. A simple baseline that relabels only the samples whose reward stays zero during the warm-up period would also be informative. Please let me know if I have overlooked such comparisons.

Response:
Thank you for this thoughtful suggestion. We would first like to clarify our baseline selection. To our knowledge, OLR is the first method specifically designed for RLVR with noisy reference labels. Consequently, there is currently no existing method operating under exactly the same setting. We therefore compare against the closest methods from two relevant areas: label-free RLVR methods that reduce dependence on reference labels, and established noisy-label learning methods adapted to token-level RLVR.

Semi-supervised RLVR assumes a different information setting. In particular, TraPO assumes that a trusted labeled subset and an unlabeled subset are known in advance. In the standard noisy-label setting, every sample has a reference label, but the corrupted labels and their locations are unknown. Persistent zero reward does not uniquely identify corruption because a difficult sample with a correct label may also remain unsolved throughout warm-up, while an active wrong label may receive nonzero rewards.

Nevertheless, following your suggestion, we additionally evaluate two non-oracle baselines using the same warm-up as OLR. i) Persistent-Zero Relabeling relabels only samples whose reference reward remains zero throughout warm-up using their warm-up rollout majority, then keeps the new labels fixed. ii) TraPO uses the same persistent-zero criterion to construct an unlabeled subset and treats the remaining samples as labeled.

Results under active noise (ρ=0.5\rho=0.5). All values are accuracy (%).

MethodAIME24AIME25AMCMATH-500MinervaOlympiadAvg.ARC-cGPQA$^\star$MMLU-ProAvg.
GRPO10.48.845.874.633.140.435.573.924.748.649.1
Persistent-Zero Relabeling12.910.047.177.235.341.337.376.226.349.650.7
TraPO14.211.347.778.036.442.838.477.927.850.352.0
OLR20.415.449.781.436.448.141.978.528.354.853.9

Results under inactive noise (ρ=0.5\rho=0.5). All values are accuracy (%).

MethodAIME24AIME25AMCMATH-500MinervaOlympiadAvg.ARC-cGPQA$^\star$MMLU-ProAvg.
GRPO12.55.045.376.034.942.736.186.728.856.057.2
Persistent-Zero Relabeling17.112.150.280.041.245.441.086.834.355.959.0
TraPO15.010.849.179.239.745.039.885.932.855.658.1
OLR21.718.353.984.242.348.944.986.838.457.861.0

Both additional baselines improve over naive GRPO, but neither matches OLR. Persistent-Zero Relabeling is more effective under inactive noise, where persistent zero reward is more closely aligned with label unreachability. TraPO performs slightly better under active noise, but its inferred labeled/unlabeled partition remains imperfect because persistent zero reward does not uniquely identify corruption. OLR retains gains of 3.5–4.1 ID points and 1.9–2.9 OOD points over the stronger additional baseline across the two settings. We have added the baseline-setting clarification and the Persistent-Zero and adapted TraPO comparisons to Appendices C.3 and D.2 (Tables 12–13) of the revised paper and highlighted them in blue.

W2: The method appears to be closely related to SELFIE (Song et al., 2019), which is already cited in the paper. To my understanding, SELFIE also replaces labels with the most frequent prediction when the prediction history is consistent after a warm-up phase. It would be helpful if the authors could clearly articulate how OLR differs from such label refurbishing methods. Similarly, the Early Correctness Coherence phenomenon seems closely related to the early-learning effect known in the noisy label literature.

Response:
We are particularly grateful that you recognized these important connections, as they capture exactly the conceptual lineage and key distinction that motivated ECC and OLR! First, SELFIE and the classical early-learning effect were developed for noisy-label classification, whose setting differs substantially from RLVR. Classification assumes a fixed label space, one categorical prediction per example, and a static supervised objective. RLVR instead involves open-ended sequence generation, multiple stochastic on-policy rollouts, sparse verifier rewards, and group-relative optimization. A wrong reference may also be active or inactive depending on the current rollouts. Therefore, classification-based refurbishment cannot be transferred directly to noisy-label RLVR.

This distinction also clarifies the contribution of Early Correctness Coherence. Classical early learning states that classifiers fit clean or simple patterns before memorizing noisy class labels. ECC identifies an RLVR-specific behavior: during early on-policy training, correct answers on prompts with incorrect references can become increasingly frequent, even though the provided references are wrong. Our analysis relates this behavior to shared-parameter coupling from clean prompts, while Figures 3 and 8 demonstrate it under both active and inactive noise.

OLR and SELFIE share the broad intuition of using prediction history, but their criteria differ. SELFIE selects labels based on low entropy among historical class predictions. OLR uses multiple rollouts and requires both an increasing slope in majority concentration and agreement between the current and historical majority answers. It requires no predefined class space, known noise rate, or preselected clean subset.

Appendix Table 11 isolates this difference. Removing the positive-slope criterion, leaving the component closest to SELFIE-style refurbishment, reduces average ID accuracy from 41.9 to 37.0 under active noise and from 44.9 to 38.3 under inactive noise. We have added a detailed comparison with SELFIE and classical early learning to Appendix E, together with the supporting ablation analysis in Table 11, and highlighted the revisions in blue.

W3: Some claims around Table 1 should be stated more carefully. The text acknowledges that TTRL is more robust on the OOD tasks under active noise, but then says that OLR outperforms the best unsupervised method by 3.3% on OOD tasks, which holds only under inactive noise. Under active noise, OLR is also below Confidence Penalty on the OOD average.

Response:
Thank you for carefully identifying this imprecise statement. We have revised the sentence to: “Under inactive noise, OLR outperforms the best unsupervised method by 3.3 points on OOD tasks.” This correction does not affect the reported results or our conclusions regarding OLR’s ID performance.

W4: I am also not sure whether the comparison is fairly executed. The hyperparameters of OLR are tuned on the validation set (Tables 9 and 10), whereas the method-specific hyperparameters of the baselines follow their original papers (Appendix C.3).

Response:
We would like to clarify that our baselines do not use arbitrary or deliberately weak configurations. For each baseline, we adopt the method-specific hyperparameters recommended by its original paper or official implementation. This method-faithful protocol is also used in recent RLVR studies. Spurious Rewards follows all hyperparameters and the public implementation without extensively retuning it on each new base model [1]. Dual Consensus uses the official public code for all baselines and keeps their remaining hyperparameters identical to the original configurations [2]. Similarly, Wu et al. adopt the original recommended settings for DAPO, DCPO, Clip-Cov, and SAPO [3]. This protocol preserves each baseline’s intended configuration and avoids post hoc choices based on our test results.

All non-method-specific RLVR parameters, including the optimizer, learning rate, batch size, epochs, rollout budget, and training data, are controlled across methods. The same validation set is also used for checkpoint selection for every method.

Tables 9 and 10 report sensitivity analyses rather than per-setting tuning. Moreover, six of eight nondefault configurations match or outperform the strongest baseline under active noise, and seven do so under inactive noise on ID benchmarks, showing that OLR’s advantage is not tied to a single configuration. We have clarified the baseline hyperparameter protocol and the role of Tables 9–10 in Appendix C.3 of the revised paper and highlighted the revisions in blue.

[1] Shao, R., Li, S. S., Xin, R., et al. Spurious Rewards: Rethinking Training Signals in RLVR. ICML, 2026.
[2] Du, K., Cao, M., Zhang, H., et al. Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism. arXiv:2603.16223, 2026.
[3] Wu, Q., Wang, Y., Zhan, S. S., et al. A Unified Framework for Rethinking Policy Divergence Measures in GRPO. arXiv:2602.05494, 2026.

W5: The theoretical results seem to follow almost directly from their assumptions. In Theorem 2.3, the positive coupling and the positive drift are assumed, and Theorem 2.4 appears to be an identity from the law of total probability. It would be helpful to clarify what the theorems add beyond the empirical observation.

Response:
Thank you for raising this thoughtful question. The empirical results show that Early Correctness Coherence occurs in the evaluated settings, while the theory specifies when it should occur, why it occurs, and when it may fail.

Theorem 2.3 decomposes its expected drift into a positive contribution transferred from clean samples and a negative contribution from noisy supervision: Δs=γ(1−ρ)Gc−ρGn\Delta_s=\gamma(1-\rho)G_c-\rho G_n. This quantity can have either sign. From this competition, the theorem derives a critical noise ratio ρc=γGc/(γGc+Gn)\rho_c=\gamma G_c/(\gamma G_c+G_n), the rollout requirement K≳log⁡(T/δ)K\gtrsim\log(T/\delta), and a uniform high-probability bound over the early-training horizon. It therefore predicts when ECC should persist, when noisy supervision should dominate, and how finite rollout sampling affects its reliability.

Theorem 2.4 serves a different purpose. Although its equality follows from total probability, it provides an exact accounting of successful corrections of noisy labels and erroneous replacements of originally clean labels. It yields the necessary and sufficient condition ρqn(1−en)>(1−ρ)qcec\rho q_n(1-e_n)>(1-\rho)q_ce_c for OLR to reduce the effective noise rate. We have clarified what the theoretical results establish beyond the empirical observations in Section 2.2 and Appendices B.5 and B.8 of the revised paper and highlighted the revisions in blue.

Reviewer_yPiVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies RLVR under noisy reference labels, distinguishing inactive noise that wastes training signals from active noise that reinforces incorrect answers. It identifies Early Correctness Coherence and proposes Online Label Refinement (OLR), which uses increasing majority-answer frequency and historical consistency to refine labels during training. The approach is supported by conditional theoretical analysis and evaluated across synthetic noise settings, multiple backbones, six math benchmarks, three OOD benchmarks, and naturally noisy BIRD Text2SQL data.

Critical strengths
  1. The paper addresses a well-motivated and practically important problem: improving the robustness of RLVR to noisy reference labels.
  2. The experiments are extensive, covering multiple noise settings, model backbones, and evaluation tasks, with informative ablations and natural-noise validation.
  3. The theoretical analysis offers an insightful perspective on early learning dynamics under noisy supervision and motivates the proposed label-refinement approach.
Critical weaknesses
  1. The theory does not sufficiently justify OLR’s selection criteria: an increasing correct-to-noisy answer probability ratio does not guarantee that the correct answer becomes more probable or dominant.
  2. The main noise settings are artificial, using unreachable strings or dynamically selected on-policy wrong answers, while validation on naturally occurring label noise remains limited in scale.
  3. The difficulty-based data filtering favors problems where correct answers are already accessible, leaving OLR’s effectiveness on very hard problems and persistent, high-confidence errors insufficiently explored.
Questions for authors

refer to the weakness

2 comments

Thank you for recognizing our work and for your careful review! We respond point by point below.

W1: The theory does not sufficiently justify OLR’s selection criteria: an increasing correct-to-noisy answer probability ratio does not guarantee that the correct answer becomes more probable or dominant.

Response:
Thank you for raising this important point and giving us the opportunity to clarify the scope of our theory. Actually, we do not claim that a rising correct-to-noisy ratio alone guarantees that the correct answer becomes more probable in absolute terms or dominant over all answers. In fact, Theorem 2.3/B.1 only shows that, under our assumptions, the correct answer becomes more likely relative to the given noisy label. Appendix B.8 explicitly states that outperforming one noisy label is insufficient to establish dominance over all candidates, and we do not derive such dominance from the pairwise theorem.

Our claim is more specific: noisy supervision does not immediately eliminate the model’s latent correct-answer signal. The theorem explains why a useful correct-answer signal can emerge despite noisy supervision. Figures 3 and 8 empirically show that correct outputs increase during early training. OLR then selects an answer only when it becomes increasingly frequent in rollouts and remains consistent across training steps. Figure 5 shows that answers passing both checks achieve nearly 100% selection accuracy; removing either check lowers this accuracy by about 20%, with corresponding performance drops in Table 11. Thus, the theory explains the early signal, while the experiments directly validate OLR’s selection criteria. We have added this clarification to Section 2.2 and Appendices B.5–B.8 of the revised paper and highlighted the revisions in blue.

W2:The main noise settings are artificial, using unreachable strings or dynamically selected on-policy wrong answers, while validation on naturally occurring label noise remains limited in scale.

Response:
Thank you for raising this important concern. We would first like to emphasize that naturally noisy RLVR benchmarks remain extremely scarce. Most existing RLVR benchmarks were designed to study learning under assumed-clean, verifiable supervision rather than robustness to persistent label errors. This scarcity reflects how underexplored noisy-label RLVR remains and underscores the pioneering role of our work in systematically characterizing noisy-label mechanisms and developing a dedicated mitigation method for noisy RLVR.

To our knowledge, BIRD-600-Original/Corrected is the most recent, and currently the only, publicly released paired natural-noise dataset used for RLVR [1]. We evaluate OLR on its executable paired subset in Table 5.

To provide a substantially larger and more realistic fixed-noise evaluation, we additionally use the publicly released DeepScaleR-Qwen2.5-Math-7B-Incorrect-Answers dataset [1]. Its questions come from DeepScaleR, and its incorrect answers were generated by Qwen2.5-Math-7B. The authors then used mathematical verifiers, GPT-5 Pro, and manual inspection to remove answers that were actually correct, yielding 12.8K verified, fixed incorrect answers. Although these errors are model-generated rather than naturally annotated, they are plausible errors actually produced by an LLM and are considerably more realistic than unreachable strings.

We randomly sample 4,096 problems and assign the verified incorrect answers to 50% of the training set, while retaining the clean references for the remaining samples. All labels remain fixed throughout training. As shown below, OLR improves noisy GRPO by 6.1 points on average across six mathematical benchmarks, demonstrating that its benefit persists under larger-scale, plausible, and carefully verified label noise.

Table R1: Results on 4,096 DeepScaleR problems with verified, fixed model-generated errors (ρ=0.5\rho=0.5).

MethodAIME24AIME25AMCMATH-500MinervaOlympiadAvg.
Qwen3-4B-Base9.64.234.252.619.528.424.8
Noisy-label GRPO12.58.844.174.633.543.136.1
OLR20.419.651.178.237.946.142.2
Clean-label GRPO (oracle)20.418.349.781.439.048.142.8

We have added the larger-scale DeepScaleR fixed-noise experiment to Section 3.2 and Appendix D.2 (Table 16) of the revised paper and highlighted it in blue.

[1] Zhu Y, Kang D. Noisy data is destructive to reinforcement learning with verifiable rewards[J]. arXiv preprint arXiv:2603.16140, 2026.

W3:The difficulty-based data filtering favors problems where correct answers are already accessible, leaving OLR’s effectiveness on very hard problems and persistent, high-confidence errors insufficiently explored.

Response:
Thank you for highlighting these two cases. The difficulty filtering is only an experimental control, not a requirement of OLR. i) For very hard problems, Table 3 directly evaluates a subset with an initial pass rate of only 0.1250.125, corresponding to one correct answer in eight rollouts. When noise is injected specifically into this hard subset, OLR still improves GRPO by 0.7 ID and 1.3 OOD points. ii) For persistent errors, Appendix Table 13 uses model-generated wrong labels that are fixed on 50% of the samples and remain unchanged throughout training. OLR improves GRPO by 5.2 ID and 3.5 OOD points. Our dynamic active noise setting further tests model-supported wrong answers sampled from the current policy, where OLR gains 6.4 ID and 4.8 OOD points. We have clarified the results on very hard problems and persistent fixed errors in Section 3.2 (Table 3) and Appendix D.2 (Table 14) of the revised paper and highlighted the revisions in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.