acceptodds
Under review as a conference paper at ICLR 2027

Noise-Robust Reinforcement Learning with Verifiable Rewards for Open-ended Medical Multimodal Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown strong potential for improving the reasoning ability of multimodal large language models. However, most existing RLVR methods assume reliable labels for reward computation, making them vulnerable to annotation noise. This challenge is particularly prevalent in medical domains, where expert annotations are costly and often affected by diagnostic uncertainty, inter-observer variability, and complex clinical interpretation. In this work, we investigate noise-robust RLVR for open-ended medical multimodal reasoning and propose Noise-Robust RLVR(NR-RLVR), a framework that dynamically refines supervision signals during training. NR-RLVR consists of two key components. First, Reference-aware Pseudo-label Generation (RPG) improves pseudo-label reliability by aggregating predictions from multiple model rollouts and a reference-guided reasoning trajectory. Second, Confidence-gated Supervision Refinement (CSR) resolves conflicts between reference labels and pseudo-labels by selecting the more reliable supervision signal according to model confidence. Extensive experiments on text-only and vision-language medical benchmarks show that NR-RLVR consistently improves reasoning accuracy and robustness across different levels of label noise. Notably, under the challenging 90% noise setting, NR-RLVR improves the average performance of Qwen2.5-VL-3B by 7.47% over standard GRPO on vision-language medical benchmarks. Code is available in the supplementary material.

Primary area:
reinforcement learning
Keywords:
multimodal large language model, post training, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V9dGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how noisy annotations affect RLVR training for open-ended medical multimodal reasoning. The authors propose NR-RLVR to address the problem of incorrect rewards caused by unreliable reference answers. The method uses two components. Reference-aware RPG groups answers from multiple reasoning trajectories and uses majority voting to obtain pseudo-labels. One trajectory is guided by the original reference answer to make use of information that may still be correct. CSR then compares the pseudo-label with the reference answer when they disagree, replacing the reference only if the model selects the pseudo-label with high confidence. Experiments on medical text-only and vision-language benchmarks with synthetic label noise show improvements over standard GRPO, particularly at higher noise ratios.

Critical strengths
  1. The paper addresses the impact of noisy annotations on reward computation and policy optimization in RLVR, which is particularly relevant to medical applications where annotations may be unreliable.

  2. RPG makes use of potentially useful medical information retained in noisy reference answers to guide pseudo-label generation, rather than simply discarding unreliable annotations.

  3. RPG and CSR can be integrated into different RLVR algorithms without changing their core optimization objectives.

Critical weaknesses
  1. The experiments rely entirely on synthetic annotation noise, whose construction closely follows the assumptions behind the proposed method. Its effectiveness on naturally occurring medical annotation errors remains unclear.

  2. Although the paper focuses on open-ended medical reasoning, the evaluation mainly involves multiple-choice and short-answer benchmarks. More experiments on complex, long-form medical reasoning tasks would better support the claimed applicability.

  3. Both RPG and CSR depend on the current policy model, which may introduce correlated errors in pseudo-label generation and supervision selection. The paper does not examine whether these errors are reinforced during training.

Questions for authors

Can the authors verify which parts of the reference answer actually contribute to pseudo-label quality?

How does corrupting the imaging modality, anatomical location, or diagnosis separately affect RPG?

4 comments

Thank you for your constructive comments and suggestions. We respond to each point below and will incorporate the corresponding revisions into the paper.

Q1/Q2. Contributions of reference components and effects of component-specific corruption.

Reference Components and Corruption Sensitivity We conduct two complementary experiments using the Qwen2.5-VL-3B checkpoint at training step 120 under 50% annotation noise, with 100 examples per condition.

To examine which reference components contribute to pseudo-label quality, we retain only one correct component: imaging modality, anatomical location, or diagnosis. To assess sensitivity to different errors, we separately corrupt only one component while leaving the remaining reference content unchanged.

Reference componentOnly this correct component retainedOnly this component corrupted
Imaging modality41%53%
Anatomical location43%50%
Diagnosis49%44%

Retaining only the correct diagnosis yields the highest pseudo-label accuracy, suggesting that diagnostic information provides the strongest guidance among the three tested components, while modality and anatomical information offer more limited guidance when provided alone. Conversely, corrupting diagnosis yields the lowest accuracy. RPG therefore recovers correct pseudo-labels more frequently under modality or anatomical corruption in these examples. Together, these complementary experiments identify diagnostic content as particularly influential for pseudo-label quality.

W1. Robustness beyond synthetic annotation noise.

We agree that experiments with synthetic corruption alone do not establish robustness to naturally occurring annotation errors. Med-VQA datasets with documented natural annotation noise are scarce and difficult to collect. We explicitly acknowledge this limitation in the supplementary Limitations section.

To broaden our evaluation beyond a single noise generator, we conduct additional experiments using Gemma-4-26B-A4B-it and DeepSeek-VL2. For each source, we use Qwen2.5-VL-3B as the policy model with 50% annotation noise, comparing standard GRPO and NR-RLVR on the same four medical vision-language benchmarks.

Noise generatorMethodPMC-VQAVQA-RADPath-VQASLAKEAvg.Gain
Qwen3-VL-30B-A3BGRPO20.6041.6833.0050.2036.37—
Qwen3-VL-30B-A3BNR-RLVR20.0048.1236.8055.6040.13+3.76
Gemma-4-26B-A4B-itGRPO18.4047.6735.2049.6037.72—
Gemma-4-26B-A4B-itNR-RLVR19.1053.4439.6054.0041.54+3.82
DeepSeek-VL2GRPO20.8043.2434.0048.8036.71—
DeepSeek-VL2NR-RLVR22.5050.1137.8055.2041.40+4.69

NR-RLVR improves average accuracy across all three generators. These results provide additional evidence that the gains extend across synthetic annotations generated by different model families. Direct validation on naturally noisy annotations remains an important direction for future work.

W2. Evaluation on complex, long-form medical reasoning tasks.

We agree that evaluating long-form outputs would strengthen the evidence beyond our existing medical QA benchmarks. We therefore conducted additional experiments on CheXpert Plus, which requires generating free-text radiology reports from chest X-rays. We use Qwen2.5-VL-3B as the policy model and compare GRPO with NR-RLVR across five noise levels. We report Macro-F1 over 14 clinical observation labels extracted from generated and reference reports, measuring the clinical content of the generated reports.

Method10% noise30% noise50% noise70% noise90% noise
GRPO0.2730.2410.1850.1320.078
NR-RLVR0.2810.2560.2160.1730.128

NR-RLVR outperforms GRPO at every noise level, with a larger advantage as annotation noise increases, providing additional evidence that NR-RLVR preserves clinically relevant information under noisy supervision beyond multiple-choice and short-answer settings.

W3. Potential reinforcement of correlated errors during training.

We acknowledge that the policy may favor pseudo-labels derived from its own rollouts, potentially introducing correlated errors in generation and selection. CSR is designed to mitigate this risk: replacement occurs only when the policy selects the pseudo-label with confidence exceeding the threshold; otherwise, the original annotation is retained. This conservative acceptance rule limits replacements based on uncertain policy judgments.

Using separate models for pseudo-label generation and supervision selection may increase computational costs and introduce implicit knowledge distillation, complicating fair comparisons unless all methods have equivalent access to these external models. We focus on enabling the policy itself to refine noisy supervision, while external models support semantic grouping and comparison.

To examine refinement throughout training, we measure the label-accuracy change within successive 50-step windows. Specifically, ΔAcc = 100 × (N_W→C − N_C→W) / N, where W→C denotes an incorrect label corrected by refinement, C→W denotes a correct label incorrectly overwritten, and N counts all query occurrences in available logged batches within the window.

NoiseSteps 1–50Steps 51–100Steps 101–150Steps 151–200
10%+2.10+3.09+1.79+0.20
30%+4.48+4.15+4.99+4.48
50%+4.88+5.72+6.19+5.54
70%+6.09+8.73+8.44+6.42
90%+6.45+9.72+11.41+7.88

Each experiment uses Qwen2.5-VL-3B for 200 training steps, with 256 queries per step and 8 rollouts per query. Successful corrections exceed harmful overwrites in all 20 noise–stage combinations, including the final training stage.

Thanks to the authors' response, which has addressed some of my concerns. Considering the overall quality of the paper and the assessments of other reviewers, I have decided to maintain my current rating for now. I remain open to reconsidering my rating based on the subsequent discussion.

Thank you for taking the time to review our responses and for your constructive feedback. We are pleased that our rebuttal has addressed some of your concerns, and we appreciate your willingness to reconsider your assessment as the discussion progresses.

We will incorporate the additional experiments and clarifications into the revised manuscript. If you have any remaining concerns or further questions, we would be happy to address them and provide additional clarification or supporting evidence.

Thank you again for your time and thoughtful consideration of our work.

Reviewer_3pChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NR-RLVR for medical multimodal reasoning with noisy supervision. Reference-aware pseudo-label generation combines ordinary rollouts with one reference-guided trajectory, while confidence-gated supervision refinement selectively replaces conflicting annotations. Experiments report improvements across medical benchmarks, model scales, and RLVR algorithms.

Critical strengths

Useful Treatment of Partially Incorrect References: the method retains potentially useful information from noisy annotations while allowing confident pseudo-labels to replace them. Component and trajectory-count ablations support combining limited reference guidance with selective refinement.

Critical weaknesses

Limited Independent Validation: All corruption experiments use synthetic noise from one generator, limiting evidence for naturally noisy annotations. Qwen3-32B-Instruct also supplies semantic clustering, training accuracy rewards, and final semantic evaluation. This shared judge does not invalidate the comparisons, but an independent evaluator or expert audit is needed to distinguish genuine correctness gains from improved agreement with its criteria. The described human verification of generated noise lacks quantitative audit details.

Refinement Reliability and Update Specification: CSR relies on self-reported confidence without calibration or separate measurements of successful corrections and erroneous replacement of clean labels. Figure 3 shows aggregate refinement gains, but does not resolve these mechanisms; Table 4’s threshold differences also lack repeated-run uncertainty. Additionally, Eq. 2 conditions policy ratios on the question and image, whereas Eqs. 7–8 introduce a reference-conditioned trajectory. Whether that trajectory participates in gradient updates, and under which conditioning, remains unclear.

Questions for authors
  1. Do the gains persist under an independent semantic evaluator or expert-scored subset? Can you quantify the noise-verification audit and test robustness to a different corruption source?
  2. Can you report confidence calibration, correction precision, and clean-label overwrite rates across noise levels, together with independent-run variability for the component and threshold ablations?
  3. Is the reference-guided trajectory used only for pseudo-label voting, or also for policy optimization? If the latter, how are its prompt-conditioned likelihoods and advantages computed relative to Eq. 2?
3 comments

Thank you for your constructive review of our work. We respond to your concerns below and outline the corresponding revisions to the paper.

Q1/W1. Independent evaluation, quantitative noise auditing, and robustness across corruption sources.

Separating judge roles. The original shared judge reduces deployment costs and provides consistent comparisons. In an additional experiment, Qwen3-32B-Instruct performs semantic clustering, while LLaMA-3-70B-Instruct supplies training accuracy rewards and pseudo-label/reference consistency judgments. Final evaluation relies on DeepSeek-R1-Distill-Qwen-14B. With Qwen2.5-VL-3B and 50% noise:

MethodPMC-VQAVQA-RADPath-VQASLAKEAvg.
GRPO19.8042.1333.2050.6036.43
NR-RLVR20.5050.1135.6053.2039.85

NR-RLVR improves all four benchmarks, supporting the persistence of the gains under an evaluator not used in training.

Quantitative noise-verification audit. Generated annotations underwent five rounds of LLM screening for semantic inconsistency with the original answers and medical plausibility. Noncompliant annotations were regenerated until all annotations passed these checks. Three reviewers with relevant medical backgrounds then conducted two rounds of human review. Each reviewer independently drew a random sample of 200 annotations per round, yielding 600 sample reviews per round and 1,200 overall. Human-flagged annotations and similar cases identified by the LLM were targeted for regeneration and re-screening.

Audit itemResult
LLM screeningFive rounds
Human reviewersThree with relevant medical backgrounds
Human sampling200 annotations/reviewer/round; two rounds
Total sample reviews1,200
Second-round noncompliance2/600 (0.33%)

The 0.33% rate was measured in the second human-review round, before the final targeted corrections. The two flagged cases and similar errors were subsequently addressed through targeted regeneration and LLM revalidation.

Different corruption sources. Med-VQA datasets with documented natural annotation noise are scarce and difficult to collect. We explicitly acknowledge the resulting limitation of synthetic corruption in the supplementary Limitations section. We conduct additional experiments using Gemma-4-26B-A4B-it and DeepSeek-VL2 to construct noisy training annotations. We use Qwen2.5-VL-3B as the policy model and a 50% annotation-noise ratio for each generator:

Noise generatorMethodPMC-VQAVQA-RADPath-VQASLAKEAvg.Gain
Qwen3-VL-30B-A3BGRPO20.6041.6833.0050.2036.37—
Qwen3-VL-30B-A3BNR-RLVR20.0048.1236.8055.6040.13+3.76
Gemma-4-26B-A4B-itGRPO18.4047.6735.2049.6037.72—
Gemma-4-26B-A4B-itNR-RLVR19.1053.4439.6054.0041.54+3.82
DeepSeek-VL2GRPO20.8043.2434.0048.8036.71—
DeepSeek-VL2NR-RLVR22.5050.1137.8055.2041.40+4.69

For both newly added generators, NR-RLVR also outperforms GRPO on every individual benchmark. The consistent improvements provide additional evidence that NR-RLVR's effectiveness is not specific to annotations generated by Qwen3-VL-30B-A3B.

Q2/W2. Refinement reliability and variability across independent runs.

Refinement reliability. We compare supervision before and after refinement against the original clean references. W→C denotes an incorrect label becoming correct; C→W denotes a correct label becoming incorrect. Each run uses Qwen2.5-VL-3B, 200 steps, 256 queries/step, and 8 rollouts/query. Repair rate = W→C / initially wrong labels; Clean-label overwrite = C→W / initially correct labels.

NoiseW→CC→WRepair rateClean-label overwrite
10%2,0841,17840.70%2.56%
30%3,7931,47524.69%4.12%
50%4,2991,44116.79%5.63%
70%4,58878812.80%5.13%
90%4,592729.97%1.41%

Successful corrections outnumber harmful overwrites at every noise level, yielding a positive gain in label accuracy across all five settings.

Independent-run variability. We repeat the component ablations and three confidence thresholds using three independent training runs per configuration, with Qwen2.5-VL-3B at 50% noise. We report accuracy (%) as mean ± sample SD (n=3). Avg. is the unweighted mean across the four benchmarks, computed for each run before aggregation.

Ref. guidanceConf. refinementPMC-VQAVQA-RADPath-VQASLAKEAvg.
××18.37 ± 0.1546.67 ± 1.0634.87 ± 0.3152.73 ± 0.3138.16 ± 0.36
✓×19.37 ± 0.1247.30 ± 0.1336.07 ± 0.3153.93 ± 0.3139.17 ± 0.18
×✓19.27 ± 0.0647.30 ± 0.3436.20 ± 0.2053.53 ± 0.2339.08 ± 0.10
✓✓20.13 ± 0.1548.19 ± 0.1236.53 ± 0.3155.67 ± 0.1240.13 ± 0.03

Both components improve performance individually.

ThresholdPMC-VQAVQA-RADPath-VQASLAKEAvg.
0.9320.37 ± 0.1547.89 ± 0.2336.47 ± 0.3155.13 ± 0.1239.97 ± 0.04
0.9520.13 ± 0.1548.19 ± 0.1236.53 ± 0.3155.67 ± 0.1240.13 ± 0.03
0.9719.33 ± 0.1247.89 ± 0.9735.93 ± 0.1253.07 ± 0.1239.06 ± 0.23

Among these three thresholds, 0.95 achieves the highest mean accuracy.

Q3/W2. Policy optimization with the reference-guided trajectory.

The guided trajectory participates in both pseudo-label voting and policy optimization. Eq. 2 introduces standard GRPO; for NR-RLVR, we make the trajectory-specific conditioning explicit as follows:

c_G=(q,v,y,\mathcal{I}).$$ $$o_g\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid c_g), \qquad g=1,\ldots,G.$$ where $y$ is the original reference answer and $\mathcal I$ warns that it may be incorrect. We use $G=8$. The token-level policy ratio used in the clipped objective is $$\rho_{g,t}(\theta)= \frac{\pi_\theta(o_{g,t}\mid c_g,o_{g,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{g,t}\mid c_g,o_{g,<t})}.$$ Thus, both likelihoods use the exact prompt that generated the trajectory, retaining the reference answer for the guided response. All eight responses receive accuracy rewards against the same CSR-refined label, together with format rewards. They form one group for advantage computation: $$\hat A_g=\frac{r_g-\mu}{s+\delta}, \qquad \mu=\frac1G\sum_{h=1}^{G}r_h, \qquad s=\sqrt{\frac1{G-1}\sum_{h=1}^{G}(r_h-\mu)^2}.$$ Here $\delta$ is a numerical stabilizer. All eight trajectories contribute to policy updates, with the loss aggregated over valid response tokens only. We will add these definitions to clarify how Eq. 2 extends to reference-guided training.

Thank you again for your time and constructive feedback. We would like to briefly follow up on our responses and the additional experiments provided above. We would appreciate your feedback when you have a chance to review them. If any concerns remain, we would be happy to address them and provide further clarification.

Reviewer_m1hFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies RLVR for medical multimodal question answering when reference answers contain factual errors. It proposes NR-RLVR, combining Reference-aware Pseudo-label Generation (RPG) with Confidence-gated Supervision Refinement (CSR). RPG clusters sampled answers, including one reference-guided rollout, to obtain a majority pseudo-label. CSR replaces a conflicting reference only when the policy selects the pseudo-label with sufficiently high self-reported confidence. Experiments use synthetically corrupted medical training data and report improvements over GRPO and selected alternatives across medical text and visual question-answering benchmarks, especially at high noise ratios.

Critical strengths
  1. Using partially correct references is a reasonable design choice. A corrupted answer can retain useful information. Restricting reference guidance to one rollout offers a practical way to use that information while preserving exploration in the remaining rollouts. The trajectory-count ablation provides evidence that this balance matters.

  2. The experiments show useful gains under the tested noise conditions. The evaluation includes multiple backbones, component ablations, clean and fully corrupted supervision, and extensions to Dr.GRPO and DAPO. For Qwen2.5-VL-3B at 90% noise, average visual QA accuracy improves from 27.43 to 34.90. These results support the practical utility of the combined pipeline in the synthetic setting.

Critical weaknesses
  1. The reliability of the label-replacement rule is not sufficiently established. The same policy generates pseudo-labels and decides whether to trust them using self-reported confidence. The threshold ablation and aggregate label-accuracy curves suggest utility, but do not show how often accepted replacements correct errors versus overwrite correct labels, or whether confidence remains reliable during training. Moreover, semantic clustering, rewards, and final semantic evaluation reuse the same judge, limiting independent verification. The central claim that CSR produces more trustworthy medical supervision therefore needs stronger evidence.

  2. Robustness beyond the chosen synthetic noise is not established. All injected noise comes from one generator and construction process that preserves useful reference information, which RPG is designed to exploit. Varying the corruption ratio does not test sensitivity to different error patterns. Without tests on other corruption patterns or naturally occurring annotation errors, the scope of the reported robustness remains unclear.

Questions for authors
  1. Among CSR's accepted replacements, how many correct an erroneous label versus introduce a new error?

  2. Do you have results showing that NR-RLVR's gains persist with a different noise generator at the same corruption ratio?

3 comments

Thank you for your insightful review and valuable suggestions. We address the concerns below and will incorporate the corresponding revisions into the paper.

Q1/W1. Label-correction outcomes, confidence reliability during training, and independent evaluation.

Corrections versus harmful overwrites. We compare supervision before and after refinement against the original clean references. W→C denotes an incorrect label becoming correct; C→W denotes a correct label becoming incorrect. Each run uses Qwen2.5-VL-3B, 200 steps, 256 queries/step, and 8 rollouts/query. Repair rate = W→C / initially wrong labels; Clean-label overwrite = C→W / initially correct labels.

NoiseW→CC→WRepair rateClean-label overwrite
10%2,0841,17840.70%2.56%
30%3,7931,47524.69%4.12%
50%4,2991,44116.79%5.63%
70%4,58878812.80%5.13%
90%4,592729.97%1.41%

Successful corrections outnumber harmful overwrites at every noise level, yielding a positive gain in label accuracy across all five settings.

Sustained benefit throughout training. We keep the confidence threshold fixed at 0.95 and partition training into four consecutive 50-step intervals. The table reports the label-accuracy gain, defined as 100 × (W→C − C→W) / N, where N is the number of evaluated label occurrences in each interval.

NoiseSteps 1–50Steps 51–100Steps 101–150Steps 151–200
10%+2.10+3.09+1.79+0.20
30%+4.48+4.15+4.99+4.48
50%+4.88+5.72+6.19+5.54
70%+6.09+8.73+8.44+6.42
90%+6.45+9.72+11.41+7.88

CSR provides a positive benefit in all 20 combinations of noise level and training stage without changing the threshold. This supports the sustained effectiveness of the fixed confidence gate as the policy evolves.

Independent judge evaluation. We share one capable judge across semantic clustering, reward computation, and semantic evaluation to reuse inference resources and avoid the deployment overhead of separate models. These operations all assess textual semantic relationships and can share the same inference service. CSR's preference decision and confidence are produced by the policy, while the judge performs semantic assessment. All compared methods use the same evaluation criterion, maintaining a consistent basis for comparison.

To address the concern about shared-judge dependence, we additionally evaluate a configuration with separate models for clustering, training rewards, and final evaluation. Specifically, Qwen3-32B-Instruct performs semantic clustering, while LLaMA-3-70B-Instruct supplies training accuracy rewards and pseudo-label/reference consistency judgments. Final evaluation relies on DeepSeek-R1-Distill-Qwen-14B. With Qwen2.5-VL-3B and 50% noise:

MethodPMC-VQAVQA-RADPath-VQASLAKEAvg.
GRPO19.8042.1333.2050.6036.43
NR-RLVR20.5050.1135.6053.2039.85

NR-RLVR improves all four benchmarks, supporting the persistence of the gains under an evaluator not used in training.

Q2/W2. Robustness across noise generators.

We conduct additional experiments using Gemma-4-26B-A4B-it and DeepSeek-VL2 to construct noisy training annotations. These experiments evaluate whether the gains of NR-RLVR extend to annotations generated by different model families. We use Qwen2.5-VL-3B as the policy model and a 50% annotation-noise ratio for each generator:

Noise generatorMethodPMC-VQAVQA-RADPath-VQASLAKEAvg.Gain
Qwen3-VL-30B-A3BGRPO20.6041.6833.0050.2036.37—
Qwen3-VL-30B-A3BNR-RLVR20.0048.1236.8055.6040.13+3.76
Gemma-4-26B-A4B-itGRPO18.4047.6735.2049.6037.72—
Gemma-4-26B-A4B-itNR-RLVR19.1053.4439.6054.0041.54+3.82
DeepSeek-VL2GRPO20.8043.2434.0048.8036.71—
DeepSeek-VL2NR-RLVR22.5050.1137.8055.2041.40+4.69

For both newly added generators, NR-RLVR also outperforms GRPO on every individual benchmark. The consistent improvements provide additional evidence that NR-RLVR's effectiveness is not specific to annotations generated by Qwen3-VL-30B-A3B.

Thank you for the additional analyses and experiments. They address my concerns about label correction and robustness across noise generators. I have raised my overall rating accordingly.

We sincerely appreciate your thoughtful feedback and your decision to raise the rating. We are glad that the additional analyses and experiments have addressed your concerns about label correction and robustness across noise generators. We will incorporate these results and clarifications into the revised manuscript. If you have any remaining concerns, we would be happy to discuss them and provide further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

Thank you for your time and effort in overseeing our submission. We summarize the current discussion status and the additional evidence provided during rebuttal to facilitate your assessment.

Our work investigates RLVR under noisy medical annotations, combining reference-aware pseudo-label generation with confidence-gated supervision refinement. All three reviewers recognized the value of retaining useful information from partially incorrect references. Their main concerns centered on refinement reliability, independent validation, and the scope of the empirical evaluation.

Following our responses:

  • Reviewer m1hF increased the rating from 1 to 2, explicitly acknowledging that the additional analyses and experiments addressed the concerns about label correction and robustness across noise generators. The follow-up response raised no additional questions.
  • Reviewer 3pCh currently retains a rating of 2. We have provided detailed responses and additional experiments addressing the questions and are awaiting follow-up feedback.
  • Reviewer V9dG maintained a weak-accept rating of 3, noting that our response addressed some of the concerns raised. The reviewer raised no additional specific questions in the follow-up response and remained open to reconsidering the assessment based on further discussion.

The main additions during rebuttal are:

Direct analysis of supervision refinement.

We separately measured successful corrections and harmful overwrites against the original clean references. Successful corrections outnumber harmful overwrites across all five evaluated noise levels. Training-stage analyses further examine whether refinement remains beneficial as the policy evolves.

Evaluation with separate judge roles.

We added a configuration using separate models for semantic clustering, training rewards, and final evaluation. With Qwen2.5-VL-3B at 50% noise, NR-RLVR improves average accuracy from 36.43% to 39.85%, with gains on all four benchmarks under an evaluator not used in training.

Noise-construction validation and robustness across generators.

We quantified the noise-verification procedure, including 1,200 sample reviews across two rounds of human auditing. Beyond the original Qwen-generated corruption, we added annotations generated by Gemma and DeepSeek-VL2. At the same 50% noise ratio, NR-RLVR improves average accuracy over GRPO by 3.82 and 4.69 percentage points, respectively, with improvements on every benchmark for both additional generators.

Extension to long-form report generation.

We extended evaluation to free-text radiology report generation on CheXpert Plus, where NR-RLVR improves clinical-label Macro-F1 over GRPO at all five noise levels, including 0.185 → 0.216 at 50% noise and 0.078 → 0.128 at 90% noise. These results broaden the evaluation beyond multiple-choice and short-answer medical QA.

Reference-component analysis.

We separately examined the effects of retaining or corrupting imaging modality, anatomical location, and diagnosis. On the evaluated examples, retaining only the correct diagnosis yielded the highest pseudo-label accuracy, while corrupting diagnosis yielded the lowest, identifying diagnostic content as particularly influential under these conditions.

Variability across independent training runs.

We repeated the component and confidence-threshold ablations using three independent training runs per configuration, reporting mean accuracy and sample standard deviation. Both components improve mean performance individually, and their combination achieves the highest mean accuracy among the component configurations.

Explicit optimization specification.

We clarified that the reference-guided trajectory participates in both pseudo-label voting and policy optimization. Its policy likelihoods retain the reference-conditioned prompt used for generation, and all eight trajectories contribute to group-relative advantage computation and policy updates.

These additions strengthen the empirical evidence for NR-RLVR under the evaluated settings. We acknowledge that validation on naturally occurring annotation errors remains an open limitation. We will incorporate the additional results, implementation clarifications, and scope qualifications into the revised manuscript.

We respectfully ask that these additional findings and the reviewers’ follow-up assessments be considered alongside the original submission. Thank you again for your careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.