- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how noisy annotations affect RLVR training for open-ended medical multimodal reasoning. The authors propose NR-RLVR to address the problem of incorrect rewards caused by unreliable reference answers. The method uses two components. Reference-aware RPG groups answers from multiple reasoning trajectories and uses majority voting to obtain pseudo-labels. One trajectory is guided by the original reference answer to make use of information that may still be correct. CSR then compares the pseudo-label with the reference answer when they disagree, replacing the reference only if the model selects the pseudo-label with high confidence. Experiments on medical text-only and vision-language benchmarks with synthetic label noise show improvements over standard GRPO, particularly at higher noise ratios.
Critical strengths
-
The paper addresses the impact of noisy annotations on reward computation and policy optimization in RLVR, which is particularly relevant to medical applications where annotations may be unreliable.
-
RPG makes use of potentially useful medical information retained in noisy reference answers to guide pseudo-label generation, rather than simply discarding unreliable annotations.
-
RPG and CSR can be integrated into different RLVR algorithms without changing their core optimization objectives.
Critical weaknesses
-
The experiments rely entirely on synthetic annotation noise, whose construction closely follows the assumptions behind the proposed method. Its effectiveness on naturally occurring medical annotation errors remains unclear.
-
Although the paper focuses on open-ended medical reasoning, the evaluation mainly involves multiple-choice and short-answer benchmarks. More experiments on complex, long-form medical reasoning tasks would better support the claimed applicability.
-
Both RPG and CSR depend on the current policy model, which may introduce correlated errors in pseudo-label generation and supervision selection. The paper does not examine whether these errors are reinforced during training.
Questions for authors
Can the authors verify which parts of the reference answer actually contribute to pseudo-label quality?
How does corrupting the imaging modality, anatomical location, or diagnosis separately affect RPG?
4 comments
W3. Potential reinforcement of correlated errors during training.
We acknowledge that the policy may favor pseudo-labels derived from its own rollouts, potentially introducing correlated errors in generation and selection. CSR is designed to mitigate this risk: replacement occurs only when the policy selects the pseudo-label with confidence exceeding the threshold; otherwise, the original annotation is retained. This conservative acceptance rule limits replacements based on uncertain policy judgments.
Using separate models for pseudo-label generation and supervision selection may increase computational costs and introduce implicit knowledge distillation, complicating fair comparisons unless all methods have equivalent access to these external models. We focus on enabling the policy itself to refine noisy supervision, while external models support semantic grouping and comparison.
To examine refinement throughout training, we measure the label-accuracy change within successive 50-step windows. Specifically, ΔAcc = 100 × (N_W→C − N_C→W) / N, where W→C denotes an incorrect label corrected by refinement, C→W denotes a correct label incorrectly overwritten, and N counts all query occurrences in available logged batches within the window.
| Noise | Steps 1–50 | Steps 51–100 | Steps 101–150 | Steps 151–200 |
|---|---|---|---|---|
| 10% | +2.10 | +3.09 | +1.79 | +0.20 |
| 30% | +4.48 | +4.15 | +4.99 | +4.48 |
| 50% | +4.88 | +5.72 | +6.19 | +5.54 |
| 70% | +6.09 | +8.73 | +8.44 | +6.42 |
| 90% | +6.45 | +9.72 | +11.41 | +7.88 |
Each experiment uses Qwen2.5-VL-3B for 200 training steps, with 256 queries per step and 8 rollouts per query. Successful corrections exceed harmful overwrites in all 20 noise–stage combinations, including the final training stage.
Thanks to the authors' response, which has addressed some of my concerns. Considering the overall quality of the paper and the assessments of other reviewers, I have decided to maintain my current rating for now. I remain open to reconsidering my rating based on the subsequent discussion.
Thank you for taking the time to review our responses and for your constructive feedback. We are pleased that our rebuttal has addressed some of your concerns, and we appreciate your willingness to reconsider your assessment as the discussion progresses.
We will incorporate the additional experiments and clarifications into the revised manuscript. If you have any remaining concerns or further questions, we would be happy to address them and provide additional clarification or supporting evidence.
Thank you again for your time and thoughtful consideration of our work.
Thank you for your constructive comments and suggestions. We respond to each point below and will incorporate the corresponding revisions into the paper.
Reference Components and Corruption Sensitivity We conduct two complementary experiments using the Qwen2.5-VL-3B checkpoint at training step 120 under 50% annotation noise, with 100 examples per condition.
To examine which reference components contribute to pseudo-label quality, we retain only one correct component: imaging modality, anatomical location, or diagnosis. To assess sensitivity to different errors, we separately corrupt only one component while leaving the remaining reference content unchanged.
Retaining only the correct diagnosis yields the highest pseudo-label accuracy, suggesting that diagnostic information provides the strongest guidance among the three tested components, while modality and anatomical information offer more limited guidance when provided alone. Conversely, corrupting diagnosis yields the lowest accuracy. RPG therefore recovers correct pseudo-labels more frequently under modality or anatomical corruption in these examples. Together, these complementary experiments identify diagnostic content as particularly influential for pseudo-label quality.
We agree that experiments with synthetic corruption alone do not establish robustness to naturally occurring annotation errors. Med-VQA datasets with documented natural annotation noise are scarce and difficult to collect. We explicitly acknowledge this limitation in the supplementary Limitations section.
To broaden our evaluation beyond a single noise generator, we conduct additional experiments using Gemma-4-26B-A4B-it and DeepSeek-VL2. For each source, we use Qwen2.5-VL-3B as the policy model with 50% annotation noise, comparing standard GRPO and NR-RLVR on the same four medical vision-language benchmarks.
NR-RLVR improves average accuracy across all three generators. These results provide additional evidence that the gains extend across synthetic annotations generated by different model families. Direct validation on naturally noisy annotations remains an important direction for future work.
We agree that evaluating long-form outputs would strengthen the evidence beyond our existing medical QA benchmarks. We therefore conducted additional experiments on CheXpert Plus, which requires generating free-text radiology reports from chest X-rays. We use Qwen2.5-VL-3B as the policy model and compare GRPO with NR-RLVR across five noise levels. We report Macro-F1 over 14 clinical observation labels extracted from generated and reference reports, measuring the clinical content of the generated reports.
NR-RLVR outperforms GRPO at every noise level, with a larger advantage as annotation noise increases, providing additional evidence that NR-RLVR preserves clinically relevant information under noisy supervision beyond multiple-choice and short-answer settings.