acceptodds
Withdrawn from ICLR 2027

MedFrame-R1: Reward Gating & Informative Sampling for Medical Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Medical image interpretation requires combining visual evidence with clinical knowledge, making multimodal large language models (MLLMs) a promising approach to medical question answering and image understanding. To improve the accuracy of their answers, these models can be further trained using reinforcement learning (RL), which rewards desirable responses. However, effective training and reliable evaluation remain challenging due to limited learning signals, unreliable rewards, and evaluation bias. We introduce MedFrame R1, an open-source, reproducible framework that brings together data selection, reward design, and supervised initialization to study and improve medical multimodal RL. We also introduce an evaluation protocol that combines deterministic scoring with independent judgments from three openly available models, reducing reliance on any single judge. Across 17 medical benchmarks, our 4B model achieves a macro score of 64.8, outperforming our reproduction of MediX-R1 4B (62.9) by 1.9 percentage points. These contributions support systematic investigation of RL training and more reliable assessment of its benefits.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Medical Multimodal Reasoning, Multimodal Large Language Models, Reward Hacking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ErkfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MedFrame-R1, a framework for improving medical multimodal reasoning through supervised initialization and reinforcement learning. It argues that (1) most of the 51K training corpus yields zero advantage, (2) an LLM judge's binary correctness reward is hackable via "answer inclusion", and can be gated by an embedding-similarity threshold (cos ≥ 0.8), (3) hard questions and rare modalities remain underrepresented, motivating a reasoning-based supervised cold start, and (4) evaluation should be deterministic where possible and otherwise use three independent judges. The resulting 4B model scores 64.8 macro over 17 benchmarks vs 62.9 for the authors' reproduction of MediX-R1-4B and 62.2 for its Qwen3-VL-4B initialization.

Critical strengths

MedFrame-R1 achieves strong performance across standard medical VQA and QA benchmarks, outperforming the evaluated medical MLLM baselines in overall macro score.

The paper addresses practical challenges in medical multimodal RL. Uninformative rollout groups, unreliable judge rewards, and evaluation bias are relevant obstacles to developing reliable medical MLLMs.

The analysis of training behavior is informative. The reward curves and qualitative examples illustrate how increasing judge rewards can coexist with declining embedding agreement and overly broad answers.

Critical weaknesses

Insufficient controlled ablations. MedFrame-R1 combines DAPO-style sampling, AESL, reasoning-based SFT, and embedding-gated rewards, but the paper does not isolate their contributions to downstream performance on the main Qwen3-VL-4B backbone. Table 1 reports only the complete MedFrame-R1 system, while the cold-start comparisons use 8B and 27B backbones and do not assess subsequent RL performance. Consequently, it remains unclear which components account for the reported gains.

Insufficient independent validation of semantic reward gating. Section 4.3 primarily evaluates gating through embedding-based agreement with the reference—the same signal directly optimized during training. Although the qualitative examples illustrate reward hacking, the reported analysis does not establish whether gating improves answer correctness or reduces the frequency of reward hacking. A human-annotated audit of reward decisions or a downstream accuracy comparison with and without gating would provide stronger evidence.

Unclear methodological novelty. The overview in Section 3 describes the method as combining structured reasoning supervision, semantic reward validation, and informative example selection, but does not clearly distinguish newly proposed components from adopted techniques. The paper should articulate its contribution beyond integrating existing methods and explain what new capability or insight this integration provides.

Presentation needs improvement. Several sections contain long paragraphs that combine distinct topics, making the argument difficult to follow. In particular, the introduction mixes challenge descriptions with proposed solutions unevenly; a clearer correspondence between the four challenges and the contributions would improve readability. There are also typos, including “scores.R presents a separate challenge” at line 95 and “generates generates” in the Figure 2 caption.

Questions for authors

How often does a retained mixed group collapse to near-zero advantage after gating, and how often does a discarded group retain reward variance? Would selecting on the gated reward change the result?

1 comment

We thank Reviewer Erkf for their valuable comments, for finding the training analysis informative, and for the helpful writing suggestions. We would like to address the reviewer's comments as follows.

W1: Controlled ablations on the 4B model: Please refer to the comment regarding the clarification on the contribution of each component. Point 3 of that comment covers the cold start on the same 4B model.

W2: Independent validation of the gate:

  • The benchmark scores come from fixed rules and three other judges, not from the embedding model that we train with. Measured this way, adding the gate raises the micro score from 55.3 to 56.4 in the ablation table.
  • The training curves in the paper show the reward hacking without the gate. The judge reward keeps rising while the real agreement with the reference falls.
  • On answers labelled by our evaluation protocol, the gate keeps 97% of correct multiple-choice answers and lets only 27% of wrong ones through.

W3: Novelty:

We thank the reviewer for this helpful suggestion. We will make this clear in the introduction and Section 3 of the final paper.

  • Not a new algorithm. We do not propose a new RL algorithm. Our contribution is a framework that makes the same standard RL recipe reliable and reproducible for medical MLLMs.
  • Four measured problems. Uninformative training samples, reward hacking by answer listing or repetitive reasoning, missing domain knowledge in the starting model, and biased single-judge evaluation.
  • One simple fix for each. Dynamic sampling on the judge decision, an embedding gate with a length reward, a supervised cold start with chain-of-thought reasoning, and an evaluation protocol with fixed rules and three judges from different model families.
  • Adopted versus new. We reuse known building blocks where they fit (DAPO dynamic sampling, AESL). Our insight is which problems matter and how to fix them together.
  • Gaps in prior work. The table below shows that existing medical RL methods leave at least one of these problems open.
MethodInformative samplesExplicit reasoningAvoids answer listingLength control
MedVLM-R1×✓✓×
MedMO××✓✓
QoQ-Med×✓××
MediX-R1×✓××
MedFrame-R1✓✓✓✓

W4: Presentation:

We thank the reviewer for pointing this out. In the final paper, the introduction will list the four problems one by one and link each to one contribution. We will also fix the typos pointed out by the reviewer.

Q1: Groups that lose their signal after the gate, and selecting on the gated reward:

We estimated this from the per-rollout training logs of a MedFrame-R1 run, which record the judge decision, the gate, and the reward of every rollout.

  • Of the groups selected by the judge, 64% remain mixed in gated correctness, and 72% still have different rewards inside the group, so they still provide a learning signal.
  • The remaining 28% end with zero advantage. These are groups in which the gate rejected every judge approval, so they add no gradient and cannot push the model in a wrong direction.
  • In the other direction, only 4% of the discarded groups would become informative after the gate.
  • Selecting groups after the gate would therefore mainly save compute rather than change what the model learns.

We thank the reviewer for this question and will add this analysis to the final paper.

Reviewer_kkg7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MedFrame-R1 combines a reasoning-based supervised cold start, embedding-gated judge rewards, and DAPO-style dynamic sampling for medical multimodal question answering. The supervised stage uses generated reasoning targets, answer-weighted cross-entropy, and adaptive token selection. During RL, a positive judge reward is accepted only when answer-reference embedding similarity exceeds a threshold; prompt groups are selected using variation in raw judge decisions. Evaluation combines deterministic scoring with three-model judging across 17 benchmarks and 19,200 items. The reported 4B model achieves a macro score of 64.8, compared with 62.9 for the authors' MediX-R1 reproduction.

Critical strengths
  1. The common evaluation harness and broad benchmark coverage are useful. Evaluating all systems with the same questions, generation settings, and scoring conventions makes the comparisons easier to interpret. Deterministic scoring and a panel of different judges reduce dependence on one evaluator. The reported gains suggest practical potential, although they do not establish which training components cause the improvement.
Critical weaknesses
  1. The experiments do not isolate the benefits of the proposed training components. The main 4B comparison changes the complete training recipe, without reporting its SFT-only performance or benchmark ablations that separately remove reward gating and dynamic sampling. The appendix compares cold-start losses on 8B and 27B models before RL, which does not establish their contribution to the main 4B result. Similarly, higher reward on a selected training pool does not demonstrate better generalization. The source of the reported improvement therefore remains unclear.

  2. More reliable and informative rewards are not sufficiently demonstrated. The main quantitative evidence for gating is embedding-based accuracy, but embedding agreement is itself optimized during training. The examples illustrate failures without measuring how often the gate removes incorrect rewards or rejects correct answers. Furthermore, sampling uses raw judge decisions while optimization uses gated rewards, so a selected group can lose all correctness contrast after gating. Independent reward-error measurements and evidence that selected groups remain informative are needed to substantiate the central mechanism.

Questions for authors
  1. Under the same 4B initialization and training budget, how much does each of reward gating and dynamic sampling improve held-out benchmark performance?

  2. On an independently annotated response set, how often does the embedding gate reject incorrect judge approvals versus reject correct answers?

  3. What fraction of groups selected by the raw judge retain variation in gated correctness rewards after the embedding check?

1 comment

We thank Reviewer kkg7 for their valuable comments, and for finding our common evaluation setup and multi-judge protocol useful. We would like to address the reviewer's comments as follows.

W1, Q1: Which components cause the improvement, with the same 4B model and budget: Please refer to the comment regarding the clarification on the contribution of each component. Points 2 to 4 of that comment cover the SFT-only result, the cold start on the 4B model, and the effect of the selected pool on the benchmarks.

W2.1: Independent evidence for the gate:

We thank the reviewer for pointing this out. The benchmark scores do not use the embedding model or the training judge. They come from fixed rules and three other judges, so the gain from the gate (micro 55.3 to 56.4 in the ablation table) is not measured with the signal that we train on.

W2.2, Q2: How often the gate rejects wrong versus correct answers:

On answers labelled by our evaluation protocol, the gate keeps 97% of correct multiple-choice answers and lets only 27% of wrong ones through.

W2.3, Q3: Selected groups that keep a learning signal after the gate:

We estimated this from the per-rollout training logs of a MedFrame-R1 run, which record the judge decision, the gate, and the reward of every rollout.

  • Of the groups selected by the judge, 64% remain mixed in gated correctness, and 72% still have different rewards inside the group, so they still provide a learning signal.
  • The remaining 28% end with zero advantage. These are groups in which the gate rejected every judge approval, so they add no gradient and cannot push the model in a wrong direction.
  • In the other direction, only 4% of the discarded groups would become informative after the gate.
  • Selecting groups after the gate would therefore mainly save compute rather than change what the model learns.

We thank the reviewer for this question and will add this analysis to the final paper.

Reviewer_sVrsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MedFrame-R1, a framework for RL training of medical multimodal LLMs. It combines three components: a supervised cold start on model-generated chain-of-thought, an embedding gate that only accounts for informative rewards and dynamic sampling that maintain variation in correctness. It also proposes an evaluation protocol with deterministic scoring and majority voting over three LLM judges. On 17 medical benchmarks, the 4B model reaches an overall score of 64.8, outperforming representative baselines.

Critical strengths
  1. The diagnosis of reward hacking is concrete and convincing.
  2. The evaluation protocol is a practical improvement which makes the comparison more trustworthy.
Critical weaknesses
  1. Lack of ablation study. The paper claims that reward gating and informative sampling improve medical multimodal RL, but never perform the ablation of each components.
  2. Limited novelty. MediX-R1 already uses the same four reward terms, while the difference is multiplying judge by embedding with no threshold sensitivity analysis. Dynamic sampling is taken directly from DAPO.
  3. Inconsistent selection logic between group selection and final reward calculation. Groups are selected by raw judge votes but trained with gated rewards, so a selected group may have mixed judge votes but provide no learning signal.
Questions for authors
  1. What are the per-component ablation results on the 17 benchmarks with the 4B model?
  2. What is the result with the judge removed and only the thresholded embedding reward used?
1 comment

We thank Reviewer sVrs for their valuable comments, and for finding our reward hacking analysis and evaluation protocol useful. We would like to address the reviewer's comments as follows.

W1, Q1: Per-component ablation on the 17 benchmarks with the 4B model: Please refer to the comment regarding the clarification on the contribution of each component.

W2.1: Novelty compared with MediX-R1 and DAPO:

  • We agree that we do not propose a new RL algorithm, and this is intentional. Our contribution is to show how the same standard RL recipe can be used more reliably.
  • Four measured problems. We identify and measure four problems that limit current medical RL: training samples that give no learning signal, judge rewards that can be gamed by listing many answers or by repetitive reasoning, starting models that lack the knowledge to answer correctly, and biased single-judge evaluation.
  • One simple fix for each. Dynamic sampling on the judge decision, an embedding gate with a length reward, a supervised cold start with chain-of-thought reasoning, and an evaluation protocol with fixed rules and three judges from different model families.
  • Gate versus MediX-R1. MediX-R1 adds the judge and embedding scores, so an answer that lists many options still receives about two thirds of the full reward. Our gate multiplies them, so such an answer receives no correctness credit.
  • Gaps in prior work. The table below shows that existing medical RL methods, including MediX-R1, leave at least one of these problems open.
  • Result. With the same backbone and training data, fixing these problems improves the macro score from 63.0 (MediX-R1) to 64.8.
MethodInformative samplesExplicit reasoningAvoids answer listingLength control
MedVLM-R1×✓✓×
MedMO××✓✓
QoQ-Med×✓××
MediX-R1×✓××
MedFrame-R1✓✓✓✓

W2.2: Threshold sensitivity:

We thank the reviewer for pointing this out. We tested four thresholds on the same 4B model.

ThresholdVQAQAMacroMicro
τ = 0.7050.170.563.354.7
τ = 0.7550.870.463.555.5
τ = 0.7851.070.163.455.7
τ = 0.80 (MedFrame-R1)51.971.864.857.4

At 0.80 the gate rejects only 7% of correct answers, while 0.90 already rejects 44%.

Q2: Embedding reward without the judge:

We thank the reviewer for this suggestion. Embedding similarity works well when the answer is short and close to the reference wording, for example a single term or an option. However, most of our training questions are open ended, and the answers can be long and phrased very differently from the reference. Embedding similarity cannot judge such answers reliably, so an embedding-only reward would not be meaningful for our data. For this reason we use the LLM judge to decide correctness, and the embedding only as a check on top of it.

Comments on the paper

1 comment

Clarification on the contribution of each component:

All three reviewers asked which training components cause the improvement on the same Qwen3-VL-4B model. We thank the reviewers for raising this. The submission did not separate the components, so we ran the controlled ablation below and will add it to the final paper.

1) All rows use the same model, budget, and data.

  • Every row uses Qwen3-VL-4B, 200 RL steps, the same training data, and the same evaluation protocol.
  • Rows indented once start from the base model, and the row indented twice starts from the cold start.
  • VQA and QA are the averages over the 6 image and 11 text benchmarks. Macro is the average over all 17 benchmarks, and Micro is the accuracy over all 19,200 items.
SettingVQAQAMacroMicro
Base model (zero-shot)46.870.662.251.5
 MediX-R1 recipe49.270.563.053.8
 RL with dynamic sampling50.670.963.755.3
 RL with dynamic sampling + embedding gate51.271.064.056.4
 Cold start only (SFT)49.069.062.053.9
  MedFrame-R1 (all parts)51.971.864.857.4

2) Each component improves the score.

  • Dynamic sampling. RL with dynamic sampling reaches 63.7 macro, above the base model (62.2) and the MediX-R1 recipe (63.0).
  • Embedding gate. Adding the gate raises macro from 63.7 to 64.0 and micro from 55.3 to 56.4.
  • Cold start. The SFT cold start alone does not improve the score (62.0 macro), but it gives RL a better starting point. With all parts together, MedFrame-R1 reaches 64.8 macro and 57.4 micro.

3) The cold start is evaluated on the same 4B model.

  • The table includes the cold start alone and RL after it, both on Qwen3-VL-4B. RL with dynamic sampling and the gate reaches 64.0 macro from the base model and 64.8 macro from the cold start.

4) The selected training pool improves benchmark scores, not only the training reward.

Training dataVQAQAMacroMicro
MedFrame-R1, all 51K samples49.771.463.856.5
MedFrame-R1, selected 26K pool51.971.864.857.4

5) These scores do not depend on the training signal.

  • All scores come from fixed rules and three judges from different model families. They use neither the embedding model nor the judge that we train with.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.