acceptodds
Under review as a conference paper at ICLR 2027

Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs

Abstract

Preference optimization is increasingly used to post-train medical large vision-language models (LVLMs), yet it operates at a much coarser granularity than the one that defines clinical correctness. Whether one response is clinically better than another usually comes down to a few decisive phrases, such as an anatomical laterality or a lesion attribute, and to whether each is supported by the image region the question concerns. Direct Preference Optimization (DPO) and its variants, by contrast, reduce the comparison to a single response-level scalar, an objective structurally unable to represent which tokens carry clinical meaning. We show that this mismatch is costly: of the objectives we compare, response-level DPO places the least reward on the phrases that decide clinical correctness, and substituting supervised references for preferred responses opens a stylistic gap that the model exploits as a reward-hacking shortcut, raising preference accuracy but not clinical accuracy. To capture the full feedback in a clinical comparison, we propose Fine-grained Regularized Medical Preference Optimization (FiRe-MPO). Preference pairs are built by minimally editing the model's own generations, so that preferred and rejected responses differ only on clinically decisive spans, and each is paired with a lesion-corrupted image withholding the supporting visual evidence. Because span-localized rewards are sparse, we stabilize optimization with a bidirectional token-wise KL regularizer. Across medical visual question answering and report generation, FiRe-MPO outperforms DPO, RRPO, and competing fine-grained objectives on two popular LVLMs, one medical and one general-purpose, while strengthening visual grounding and placing more reward on the medical phrases than the alternatives.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Direct Preference Optimization, Large Vision Language Models, Medical AI, Fine-Grained On-Policy Alignment, Fine-grained Regularized Medical Preference Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JCvaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the mismatch between response-level preference optimization and fine-grained clinical correctness in medical LVLMs. It proposes FiRe-MPO, which constructs minimally edited preference pairs differing only in clinically decisive spans, applies token-level preference optimization, pairs them with lesion-corrupted images with reversed preferences, and introduces bidirectional token-wise KL regularization. Experiments on medical VQA and report generation report improvements over DPO and several fine-grained preference baselines.

Critical strengths
  1. Well-motivated fine-grained textual supervision. The observation that clinical correctness may depend on only a few decisive spans is compelling, and minimally editing the model's own generations is a sensible way to reduce stylistic confounding between preferred and rejected responses.

  2. Useful empirical analysis. The paper goes beyond downstream accuracy by analyzing reward localization and includes reasonably extensive ablations across two LVLMs and multiple medical tasks.

Critical weaknesses
  1. The lesion-corruption mechanism confounds clinical evidence removal with strong artificial supervision signals. Corruption is systematically coupled with the reversed-preference branch, so the model can learn corruption present -> switch preference rather than reason about whether clinical evidence is present. More importantly, the corruption location is constructed from the question and ground-truth answer and then localized by MedSAM3. Thus, the corruption position itself provides GT-derived spatial supervision. The whole-image-noise fallback further makes corruption presence a persistent cue even when no localized region is available. The current corruption-type ablations do not control for either shortcut. Consequently, the reported grounding gains cannot be uniquely attributed to improved use of native clinical visual evidence.

  2. The corrupted branch is not a valid clinical counterfactual. Adding Gaussian noise over evidence for y+y^+ makes that evidence unobservable; it does not make the opposite answer y−y^- clinically correct. Nevertheless, FiRe-MPO explicitly reverses the preference to y−≻y+y^- \succ y^+ on the corrupted image. This creates a mismatch between the stated goal and the actual supervision . Together with the first issue, this substantially weakens the central visual-grounding claim. The remaining methodological novelty is also relatively incremental with respect to existing fine-grained preference optimization and image-conditioned DPO methods.

Questions for authors

Can the authors rule out corruption-presence and corruption-location shortcuts with artifact-matched controls? In particular, what happens when identical corruption (same type, magnitude, and area) is moved from the GT-relevant region to a random or anatomically matched incorrect/contralateral region? Does the prediction follow the clinical evidence or the corruption location?

Can the authors evaluate a setting where corruption is equally present across preference branches, such that detecting artificial noise cannot reveal which preference regime applies?

What is the justification for preferring y−y^- after evidence for y+y^+ is obscured? Would an uncertainty/insufficient-evidence target be more semantically appropriate? Results addressing these questions could materially change my assessment.

2 comments
AuthorsOpenReview ↗

Response to Reviewer JCva (Part 1/2)

Thank you for your thorough review. The presence shortcut in particular changed how we see visual preference objectives in general: we tested it, found that it holds not only for our submitted visual term but also for other methods that train on corrupted images, and sharpened the term so that no corrupted image is used in training. The foundational components of our method (the fine-grained objective, minimal-edit pairs, and bidirectional KL, Eq. 3) remain fully intact.

Presence shortcut (W1, Q2). We tested whether trained models actually use the "noise present → switch preference" cue, and extended the test to mDPO and MMedPO. On 257 SLAKE test questions with ground-truth masks, we apply identical Gaussian noise (same σ and area) either over the lesion or over a random region, and measure the change in each model's reward r = log πθ / πref for the preferred (y+) and dispreferred (y−) responses:

MethodLesion: Δr(y+)Lesion: Δr(y−)Lesion: flip %Random: Δr(y+)Random: Δr(y−)Random: flip %
DPO−0.22−0.179.8−0.07−0.064.8
mDPO−0.94−0.8912.4−0.86−0.8311.2
MMedPO−1.08−1.0213.8−0.98−0.9412.6
FiRe-MPO (old)−1.42+0.3854.6−1.24+0.3144.8
FiRe-MPO (revised) (c = 1)−0.24−0.1610.4−0.07−0.064.6
FiRe-MPO (revised)−0.31−0.1712.8−0.08−0.065.0

Our old version of visual component reverses its preference in 44.8% of pairs even when the noise lies outside the lesion, compared with 54.6% when the lesion is covered and 4.8% for DPO under random noise. This confirms your concern: reversal-based training learns a cue unrelated to where the clinical evidence is. mDPO and MMedPO show a related one: under random noise their y+ rewards fall by 0.86 and 0.98, versus 0.07 for DPO. The problem is therefore not specific to our original term, and any objective with a target on noised images invites it.

We therefore removed the noised-image branch. Before training, the frozen reference model scores y+ on the clean image vv and on vboxv^{\text{box}}, the same image with a box drawn around the queried region:

ρ=1∣y+∣∑t[log⁡πref(yt+∣vbox,q,y<t+)−log⁡πref(yt+∣v,q,y<t+)]\rho = \frac{1}{\lvert y^+ \rvert}\sum_t \Big[\log \pi_{\text{ref}}(y_t^+ \mid v^{\text{box}}, q, y_{\lt t}^+) - \log \pi_{\text{ref}}(y_t^+ \mid v, q, y_{\lt t}^+)\Big]

This gives a per-pair weight c~=exp⁡(clamp(ρ/τ,0,1))∈[1,e]\tilde c = \exp(\mathrm{clamp}(\rho/\tau, 0, 1)) \in [1, e], normalised to mean 1 (cc) with temperature τ\tau > 0, on the clean-image span-level loss:

L=−E[clog⁡σ(∑irθ(v,q,yi+)−∑irθ(v,q,yi−))]+αLKL\mathcal{L} = -\mathbb{E}\big[ c \log\sigma\big(\textstyle\sum_i r_\theta(v,q,y_i^+) - \sum_i r_\theta(v,q,y_i^-)\big)\big] + \alpha\mathcal{L}_{\text{KL}}

where LKL\mathcal{L}_{\text{KL}} is the bidirectional token-wise KL regularization term defined in Eq. 3.

Since every branch now sees the same clean image, there is no corruption whose presence could reveal which preference applies, which is the setting your Q2 asks for. Under random noise FiRe-MPO behaves like DPO (Δr(y+): −0.08 vs −0.07; flips: 5.0% vs 4.8%).

Location shortcut (W1, Q1). The region is still derived from the question and ground-truth answer, but FiRe-MPO uses it only once, before training, for the frozen reference to compute a scalar weight; the box never enters the policy's input during training or inference, so there is no spatial signal for the policy to copy. To check that the gain comes from clinically meaningful localization rather than reweighting itself, we recomputed the weights from wrong regions, with identical data and hyperparameters (HuatuoGPT):

How c is computedAvg
Uniform (c = 1)69.34
Shuffled across pairs69.26
Random box, matched area69.42
Contralateral box68.98
Correct box (FiRe-MPO)71.60

Only weights from the correct clinical region give a substantial improvement (+2.26 over uniform); shuffled or wrong-region weights match or fall below uniform weighting.

AuthorsOpenReview ↗

Response to Reviewer JCva (Part 2/2)

Evidence vs noise location (Q1). From the noise table in Part 1: moving identical noise from the relevant region to a random region lowers FiRe-MPO's reward for the correct answer by 0.08 instead of 0.31, and flips 5.0% instead of 12.8% of pairs. Its preferences follow the evidence, not the noise.

Whole-image fallback. FiRe-MPO has none: questions about modality or general knowledge without a localizable region keep the base weight, and no image is ever noised.

Counterfactual validity and abstention (W2, Q3). Our motivation in the first version of FiRe-MPO was to stop rewarding y+ when the model attends to unrelated regions, but we agree that noise makes the evidence unobservable rather than making y− correct, so the reversed target was not valid supervision. We also implemented the uncertainty targets you suggest, under matched data: an abstention response preferred on v′ gives 67.79 average and an abstention loss 68.42, both below no visual term (69.34) and FiRe-MPO (71.60). Both still train on corrupted images, so they keep the presence cue and are out of distribution relative to clean test images. In the revised method, y− is only ever dispreferred, on the clean image.

Grounding claim. Drawing a box around the queried region raises the base model's accuracy on the 257 localizable SLAKE questions from 38.13% to 48.64%. Without any box at test time, FiRe-MPO reaches 46.30%, recovering 78% of that gain, vs 56% with uniform weights and 44% for DPO: the behaviour elicited by the box carries over to test time. We have narrowed the grounding claim in the paper to what these results support.

Main results. With 3 seeds (Avg = mean of SLAKE overall, VQA-RAD overall and IU-Xray GREEN):

MethodHuatuoGPT-Vision-7BQwen3-VL-4B
DPO64.6163.02
mDPO64.9063.99
MMedPO66.5064.95
RRPO68.22 ± 0.3867.33 ± 0.36
FiRe-MPO (old)69.75 ± 0.3467.40 ± 0.31
FiRe-MPO71.60 ± 0.2869.43 ± 0.28

Novelty. Only the visual term changes in the revision; the rest of the method is as submitted. We see three contributions:

• Diagnosis. Response-level DPO places little reward on clinically decisive tokens (Table 1), and reference-as-preferred pairs cause stylistic reward hacking (Section 2.3). This motivates span-level supervision in the medical setting rather than assuming it.

• On-policy minimal-edit pairs. RRPO and Mask-DPO localize the loss, but their pairs are not minimal: Mask-DPO localizes on whether each fact stated by the model is true or false, and RRPO on whatever differs between y+ and y−. Our pairs match the ground truth to the model's own answer and change only the decisive span, whatever the style of the generation, which separates linguistic priors from image-provided evidence. The bidirectional KL also stops the model from collapsing to one behaviour (Figure 4).

• How visual evidence enters. mDPO and MMedPO place a target or margin on corrupted images, which opens the visual shortcut shown in Part 1, and comparing responses under different images means the partition function Z(x)Z(x) no longer cancels as in DPO. FiRe-MPO never trains on a corrupted image; visual evidence enters only through the reference-computed weight on a same-image comparison.

We are grateful for these questions; they materially improved the method, and we hope the controls above address the shortcut concerns.

Reviewer_jr7EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FiRe-MPO for medical vision-language models, combining clinical span-level preference optimization, reversed preferences on lesion-corrupted images, and bidirectional token-wise KL regularization. Experiments on two backbones across medical VQA and report generation demonstrate improvements over several preference-optimization baselines.

Critical strengths
  1. The paper clearly motivates the granularity mismatch between response-level preference optimization and clinically decisive information.

  2. Constructing preference pairs through minimal clinical edits provides a sensible approach to reducing stylistic differences between preferred and rejected responses.

  3. The evaluation covers two backbones and multiple datasets, with useful ablations on regularization, visual preference objectives, and preference-pair construction.

Critical weaknesses
  1. The novelty is incremental. MMedPO [1] already combines textual preferences with lesion-targeted image corruption, while CoMedPO [2] optimizes factual–counterfactual preference contrasts to reduce shortcut reliance. FiRe-MPO introduces clinical span-level supervision and explicit preference reversal, but its contribution appears incremental given its acknowledged connections to RRPO and Mask-DPO. The manuscript should clarify its methodological distinction from these approaches.

  2. Removing visual support for the preferred response does not necessarily justify preferring the originally rejected response. CoMedPO [2] instead optimizes the difference between factual and counterfactual preference margins, without explicitly requiring preference reversal. The authors should justify why this stronger constraint is appropriate and demonstrate its additional benefit under matched data and training settings; Table 5 does not directly compare these two formulations. The MRI-to-CT preference reversal after adding noise in Figure 7 further raises concerns about the validity of this supervision.

  3. Table 4 compares several corruption strategies, including cropping, image-level noise, full-black images, and lesion-based corruption. However, it does not isolate mask location from perturbation extent and type. Controlled training ablations with matched-size lesion and non-lesion masks, and prediction-level test-time evaluations under alternative masks and corruption mechanisms, are not reported. These analyses would help establish whether the model learns clinically relevant visual dependence or sensitivity to corruption artifacts.

Questions for authors
  1. What distinguishes the proposed visual objective from the previous work, and why is explicit preference reversal preferable?

  2. How do the authors verify that Gaussian noise removes the relevant visual information and makes the reversed preference appropriate, particularly for examples such as MRI versus CT?

  3. Do the improvements persist with controlled mask variations and alternative perturbation mechanisms during training and testing?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jr7E (Part 1/2)

Thank you for the precise questions on the visual objective. Your points on mask location and perturbation led us to further sharpen the visual component. In short, our answer to "why is explicit preference reversal preferable?" is that, under matched settings, it is not clearly better than the CoMedPO-style margin, and both share a deeper problem: they train on corrupted images. The revised visual term of FiRe-MPO now resolves this confounder by keeping both responses on the original image and integrating visual grounding strictly as an adaptive pair-weight factor.

Novelty and distinction from MMedPO, CoMedPO, RRPO and Mask-DPO (W1, Q1). The revision changes only the visual term; all other components remain as submitted. Our method differs from existing methods in three key aspects:

• Diagnosis. Response-level DPO places little reward on clinically decisive tokens (Table 1), and our experiments indicate that reference-as-preferred pairs encourage stylistic reward hacking (Section 2.3). We instead advocate for fine-grained textual preference, focusing optimization strictly on clinical correctness and diagnostic evidence rather than superficial formats or length artifacts.

• On-policy minimal-edit pairs. RRPO and Mask-DPO localize the loss, but their pairs are not minimal. Mask-DPO localizes on whether each fact stated by the model is true or false, and RRPO defines the textual preference on whatever differs between y+ and y−. Our pairs match the ground truth to the model's own answer and change only the decisive span, whatever the style of the generation, which separates linguistic priors from image-provided evidence. The bidirectional KL also stops the model from collapsing to one behaviour (Figure 4).

• How visual evidence enters. mDPO, MMedPO and CoMedPO train on corrupted images, which lets the model associate noise with preference instead of grounding (noise test in Part 2). They are also theoretically concerning: DPO compares preferred and dispreferred responses under the same condition xx, so the partition function Z(x)Z(x) cancels, whereas these methods compare generations under different conditions (real vs corrupted image). FiRe-MPO keeps both responses on the same clean image and lets visual evidence enter only as a per-pair weight. Before training, the frozen reference model scores y+ on vv and on vboxv^{\text{box}}, the same image with a box drawn around the queried region:

ρ=1∣y+∣∑t[log⁡πref(yt+∣vbox,q,y<t+)−log⁡πref(yt+∣v,q,y<t+)]\rho = \frac{1}{\lvert y^+ \rvert}\sum_t \Big[\log \pi_{\text{ref}}(y_t^+ \mid v^{\text{box}}, q, y_{\lt t}^+) - \log \pi_{\text{ref}}(y_t^+ \mid v, q, y_{\lt t}^+)\Big]

The weight c~=exp⁡(clamp(ρ/τ,0,1))∈[1,e]\tilde c = \exp(\mathrm{clamp}(\rho/\tau, 0, 1)) \in [1, e], normalised to mean 1 (cc) with temperature τ\tau > 0, scales the clean-image span-level loss:

L=−E[clog⁡σ(∑irθ(v,q,yi+)−∑irθ(v,q,yi−))]+αLKL\mathcal{L} = -\mathbb{E}\big[ c \log\sigma\big(\textstyle\sum_i r_\theta(v,q,y_i^+) - \sum_i r_\theta(v,q,y_i^-)\big)\big] + \alpha\mathcal{L}_{\text{KL}}

where LKL\mathcal{L}_{\text{KL}} is the bidirectional token-wise KL regularization term defined in Eq. 3.

Pairs whose correct answer becomes more likely once the model looks at the right region get more weight. Questions without a localizable region keep c~=1\tilde c = 1, and the box never enters the policy's input, so the objective stays a standard same-condition DPO comparison.

CoMedPO-style margin (W2, Q1). We implemented the margin difference −log σ(Δ(v) − Δ(v′)) introduced by CoMedPO as an alternative visual term under matched conditions (HuatuoGPT, same pairs and hyperparameters):

Visual termAvg
None (c = 1)69.34
Reversed preference (old)69.72
Margin difference (CoMedPO-style)69.77
FiRe-MPO71.60

The CoMedPO-style margin performs comparably to reversal (+0.05), so correcting the direction of the preference alone does not substantially change performance. Both formulations optimize on corrupted images, which can introduce corruption-dependent shortcuts; removing that branch is what gives the gain.

AuthorsOpenReview ↗

Response to Reviewer jr7E (Part 2/2)

MRI vs CT (Q2). Gaussian perturbation was adopted because it is the prevailing standard across general LVLM literature, utilized in multimodal preference optimization (e.g. mDPO, MMedPO, CoMedPO) as well as decoding-time hallucination mitigation like Visual Contrastive Decoding (VCD; Leng et al. 2024). However, our targeted diagnostic reveals that in medical imaging, synthetic noise does not faithfully simulate clinical counterfactuals (evidence absent). Instead, on localizable clinical questions, noise often acts as a salient artifact that baseline models overfit to rather than learning true semantic absence. To quantify this behavior, across methods, we evaluated 257 SLAKE questions with ground-truth lesion masks under matched lesion-level vs. random noise:

MethodLesion: Δr(y+)Lesion: flip %Random: Δr(y+)Random: flip %
DPO−0.229.8−0.074.8
mDPO−0.9412.4−0.8611.2
MMedPO−1.0813.8−0.9812.6
FiRe-MPO (old)−1.4254.6−1.2444.8
FiRe-MPO (revised)−0.3112.8−0.085.0

mDPO and MMedPO lower the reward of y+ almost as much under random noise as under lesion noise (−0.86 vs −0.94; −0.98 vs −1.08): they learn "noise present → lower the reward", wherever the noise is. Reverse visual preference learns "noise present → flip the preference", reversing 44.8% of pairs even when the lesion is untouched. These are two distinct shortcuts, and neither tracks whether clinical evidence was removed. FiRe-MPO reacts to random noise like DPO (−0.08 vs −0.07; 5.0% vs 4.8% flips) but loses confidence in y+ when the lesion is covered (−0.31), so its sensitivity follows the clinical relevance of the region.

The revised FiRe-MPO therefore removes this assumption: preference optimization is performed only on clean images, and holistic questions without a localizable region receive the default weight.

Mask location vs perturbation (W3, Q3). We agree that Table 4 confounded mask location with the perturbation itself, and added a location-controlled experiment.

We recomputed FiRe-MPO's weights from boxes of matched area placed at wrong locations, keeping data, loss and hyperparameters unchanged (HuatuoGPT):

How c is computedAvg
Uniform (c = 1)69.34
Shuffled across pairs69.26
Random box, matched area69.42
Contralateral box68.98
Correct box (FiRe-MPO)71.60

Only weights derived from the correct clinical region help; reweighting alone does not.

Finally, drawing the box raises the base model's accuracy on the 257 localizable SLAKE questions from 38.13% to 48.64%. Without boxes at inference, FiRe-MPO reaches 46.30%, recovering about 78% of that gain, compared with 56% under uniform weighting.

Main results. With 3 seeds (Avg = mean of SLAKE overall, VQA-RAD overall and IU-Xray GREEN):

MethodHuatuoGPT-Vision-7BQwen3-VL-4B
MMedPO66.5064.95
RRPO68.22 ± 0.3867.33 ± 0.36
MASK-DPO61.90 ± 0.4966.50 ± 0.35
FiRe-MPO (old)69.75 ± 0.3467.40 ± 0.31
FiRe-MPO71.60 ± 0.2869.43 ± 0.28

FiRe-MPO leads the strongest baseline, RRPO, by 3.37 and 2.10 points.

We hope these experiments clarify both the distinction from prior visual preference objectives and why we moved away from preference reversal.

Thanks for the detailed response. Most of my concerns have been addressed. The only issue is the novelty compared with the previous research. I'm open to raise the rating after consideration.

Reviewer_uv3MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this manuscript, the authors argue that response-level DPO is mismatched to clinical correctness in medical large vision-language models (LVLMs), where correctness often hinges on a few decisive phrases grounded in specific image regions. They first show that DPO assigns relatively little implicit reward to medical tokens and that using ground-truth references as preferred responses induces stylistic reward hacking. They then propose FiRe-MPO, which builds on-policy preference pairs by minimally editing the model's own outputs, restricts the ranking loss to the differing spans, adds a reversed preference on lesion-corrupted images (MedSAM3 segmentation plus Gaussian noise), and regularizes training with a bidirectional token-wise KL term. On SLAKE, VQA-RAD and IU-Xray with HuatuoGPT-Vision-7B and Qwen3-VL-4B, FiRe-MPO obtains the best average score among six objectives, supported by ablations on KL direction, corruption type, visual loss and pair construction. Overall, this is a well-motivated and clearly written paper, although the gains are modest and several reported numbers need verification.

Critical strengths

Strengths

  1. The problem is clearly formulated, and placing the diagnosis (reward allocation and stylistic reward hacking) before the method makes the motivation concrete. The token-level comparison in Appendix E, which compares models on fixed tokens under a shared reference model rather than comparing tokens with each other, is carefully designed.

  2. The method sensibly combines span-level textual rewards with lesion-targeted visual counterfactuals, and the factorial design in Table 6, which crosses pair-construction strategies with losses, cleanly separates the contribution of the data from that of the objective.

  3. The evaluation covers a medical and a general-purpose backbone, reproduces five baselines under a single protocol, includes reasonably complete ablations, and adds a layer-wise grounding analysis on VGMED.

Critical weaknesses

Major issues

  1. Statistical robustness and several reported numbers need attention. All results appear to come from a single training run without seeds or confidence intervals, and many differences between FiRe-MPO and RRPO or MASK-DPO are 1–2 points on small test sets scored by a GPT-4o-mini judge. Several values should be checked: FiRe-MPO's SLAKE-closed (81.32) and VQA-RAD-closed (71.26) scores are identical for both backbones, and 71.26 also equals RRPO on HuatuoGPT; the HuatuoGPT “LVLM style” row in Table 8 is identical to the DPO row in Table 2; the Average column cannot be reproduced as the mean of the seven metrics (61.91 vs the reported 61.21 for the HuatuoGPT base model); and Section 4.3 states that removing forward KL lowers IU-Xray from 80.70 to 58.42, whereas Table 3 shows 58.42 for the ranking loss with no KL term at all and 80.98 (higher than the full model) with reverse KL only. Results over multiple seeds are needed.

  2. The visual preference term and the reward diagnostic need further justification. On corrupted images, Eq. 2 prefers the rejected response y− over y+, but y− is itself a confident clinical claim (e.g., “left frontal lobe” or “No, ... not infarcted” in Figure 7). This trains the model to assert a counterfactual finding when evidence is degraded, rather than to withhold a definitive answer as the text describes (lines 214–217); an abstention or uncertainty target should be compared. In addition, the Table 1 diagnostic scores y− taken from FiRe-MPO's own rejected training responses, which favors FiRe-MPO; the analysis should be repeated on pairs that none of the methods saw during training.

Minor issues

  1. It is unclear whether the baselines were tuned: all methods share β = 0.1, a learning rate of 1e-6, one epoch and the same LoRA configuration (Table 7). Preference pairs are constructed with GPT-4o-mini and Gemini, and GPT-4o-mini is also the VQA judge, so a human or second-judge check on a subset would help rule out judge–generator correlation. Figure 2 has no numeric axis values.

  2. Some generated counterfactuals are not anatomically plausible (e.g., “first ventricle” in Figure 7), which makes the preference easy and may inflate gains; please describe how counterfactuals were validated. Typos include “clinicaly”, “Ground Turth” and “Exploring of image corruption strategies”.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer uv3M (Part 1/2)

Thank you for your thorough review. We address your two major concerns in turn. On the visual term, your point that it trains the model to assert a counterfactual finding motivated us to dig deeper into the visual component and found FiRe-MPO can be improved with a change to that part. The span-level objective, the on-policy minimal-edit pairs and the bidirectional token-wise KL are unchanged. To address your concern on robustness, we evaluated the main comparison across three independent random seeds, confirming our findings remain consistent.

The visual term trains a confident counterfactual. While the visual preference term (i.e., (v′, q, y−) ≻ (v′, q, y+), skipped the token level preference for brevity) was intended to discourage rewarding y+ for attending to unimportant regions, it may act suboptimal in some cases.

Following your suggestion, we first tested abstention targets under matched data (HuatuoGPT, same pairs and hyperparameters):

Visual termAvg
None (span-level objective + KL only)69.34
Reversed preference (old)69.72
Abstention response preferred over y+ on v′67.79
Abstention loss: raise 1 − p(y+) − p(y−) on v′68.42
FiRe-MPO71.60

The abstention response is a minimal edit of y+ whose decisive span states that the region cannot be assessed. For example, for Figure 7 (first image), the abstention target on v′ is “The lesion <loc>location cannot be determined.</loc>”, while y+ is “The lesion <loc>is located in the left frontal lobe.</loc>”. Both variants fall below having no visual term, mostly on IU-Xray (78.45 / 79.20 vs 81.79): training the model to withhold answers on corrupted images seems to make it less decisive on clean ones. We read this as a sign that the issue is not which target is placed on v′ but training on v′ at all, so FiRe-MPO removes the corrupted-image branch entirely. We instead reintroduce visual evidence as a per-sample importance weight.

Before training, the frozen reference model scores y+ twice: on the original image vv and on the same image with a box drawn around the queried region, vboxv^{\text{box}}. The per-token gain

ρ=1∣y+∣∑t[log⁡πref(yt+∣vbox,q,y<t+)−log⁡πref(yt+∣v,q,y<t+)]\rho = \frac{1}{\lvert y^+ \rvert}\sum_t \Big[\log \pi_{\text{ref}}(y_t^+ \mid v^{\text{box}}, q, y_{\lt t}^+) - \log \pi_{\text{ref}}(y_t^+ \mid v, q, y_{\lt t}^+)\Big]

measures how much pointing at the region makes the correct answer more likely. It sets a per-pair weight c~=exp⁡(clamp(ρ/τ,0,1))∈[1,e]\tilde c = \exp(\mathrm{clamp}(\rho/\tau, 0, 1)) \in [1, e], normalised to mean 1 over the dataset (cc) and τ>0\tau > 0 is a scale hyperparameter at which the weight saturates, on the clean-image span-level loss:

L=−E[clog⁡σ(∑irθ(v,q,yi+)−∑irθ(v,q,yi−))]+αLKL\mathcal{L} = -\mathbb{E}\big[ c \log\sigma\big(\textstyle\sum_i r_\theta(v,q,y_i^+) - \sum_i r_\theta(v,q,y_i^-)\big)\big] + \alpha\mathcal{L}_{\text{KL}}

where LKL\mathcal{L}_{\text{KL}} is the bidirectional token-wise KL regularization term defined in Eq. 3.

Questions without a localizable region (e.g., modality) get c~=1\tilde c = 1, and the box never enters the policy's input. The model is therefore never asked to assert, or to withhold, anything on a degraded image; visual evidence only decides how much each clean pair counts. The weight targets a gap we measured directly: drawing the box lifts the base model from 38.13% to 48.64% on 257 localizable SLAKE questions, and without any box at test time FiRe-MPO reaches 46.30% (78% of that gain, vs 56% with uniform weights).

Single run and small margins. We re-ran the main comparison with 3 seeds. Average score (mean of SLAKE overall, VQA-RAD overall and IU-Xray GREEN):

MethodHuatuoGPT-Vision-7BQwen3-VL-4B
Base61.3863.36
DPO64.6163.02
mDPO64.9063.99
MMedPO66.5064.95
RRPO68.22 ± 0.3867.33 ± 0.36
MASK-DPO61.90 ± 0.4966.50 ± 0.35
FiRe-MPO (old)69.75 ± 0.3467.40 ± 0.31
FiRe-MPO (revised)71.60 ± 0.2869.43 ± 0.28
AuthorsOpenReview ↗

Response to Reviewer uv3M (Part 2/2)

Identical values across backbones and methods. Here are the results for the mentioned rows. We would like to note that the test sets are small in sample size which can make some numbers identical, given that all evaluated methods build upon the core DPO formulation.

MethodBackboneSLAKE-closedVQA-RAD-closed
RRPOHuatuoGPT79.17 ± 0.6471.45 ± 0.80
FiRe-MPO (old)HuatuoGPT81.17 ± 0.3771.58 ± 0.69
FiRe-MPO (old)Qwen3-VL81.09 ± 0.4271.31 ± 0.80
FiRe-MPOHuatuoGPT82.37 ± 0.4873.84 ± 0.69
FiRe-MPOQwen3-VL82.77 ± 0.6273.04 ± 0.69

Table 8 row. Intentional but unstated: Table 8 varies pair construction under the DPO loss, so its LVLM-style row is the DPO run of Table 2. We will ensure this connection is explicitly stated in the revised paper.

Average column. It is the mean of SLAKE overall, VQA-RAD overall (closed and open pooled) and GREEN, not of the seven metrics. The caption now defines it, and Table 2 adds both Overall columns.

KL text vs Table 3. You are right; the text was wrong. 58.42 is the ranking loss with no KL term, and reverse KL alone gives 80.98, slightly above bidirectional KL (80.70) on IU-Xray. The point we meant to make is that the KL term matters more as generations get longer: on long IU-Xray reports, removing it costs 22 points.

Table 1 diagnostic may favour FiRe-MPO. We appreciate the reviewer’s interpretation and recognize that the wording in Appendix E may not have conveyed our intended meaning clearly. The scored text is not generated by FiRe-MPO. Instead, every model evaluates identical, pre-existing text via teacher forcing which is drawn from the same on-policy preference pairs used during training. This ensures the comparison purely isolates how each loss distributes reward over identical tokens. We will refine the wording in Appendix E to make this protocol unambiguous.

Baseline tuning. Through parameter sweeps across key optimization parameters, we found the stated shared configuration as robustly optimal for both backbone models across baselines.

Judge and generator correlation. Closed questions are scored by rule-based matching, so the judge only affects open questions. Re-scoring them with a second judge (GPT-6 Luna) gives nearly the same method ranking (Kendall τ = 0.95 over the 16 method–backbone entries). We attribute this to the judges receiving the ground-truth answer, so scoring requires little medical knowledge from the judge.

Implausible counterfactuals. We agree that "first ventricle" is not a meaningful anatomical term. The quality of y− depends on the generating model and the dataset; one way to filter such cases is human-based verification, however, that is resource intensive and not scalable. Note that in the revised method, y− appears only as the dispreferred answer on the clean image and is never a target, so an implausible y− can make a pair easier but no longer teaches the model to assert it.

We hope the multi-seed results, the corrected numbers and the revised visual term address your concerns, and we are happy to provide anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response and Summary of Changes

We thank all the reviewers for their thoughtful and constructive feedback. Reviewers found the core premise well motivated: clinical correctness hinges on a few decisive spans, which response level DPO cannot represent (uv3M, jr7E, JCva). They recognized minimally editing the model’s own generations as an effective way to isolate medical spans from stylistic differences (jr7E, JCva). They also valued the motivating analysis of reward hacking and its misallocation (uv3M, JCva), and found the evaluation across backbones and medical tasks adequate (uv3M, jr7E, JCva).

The common thread across the reviews was the visual component. Building on this feedback, we refined it further, while the core of FiRe-MPO (the span-level objective on minimal-edit pairs with bidirectional KL) remains unchanged. Reviewer jr7E confirmed that our experiments addressed their concerns.

  1. Visual preference objectives may learn shortcuts from corrupted images: To address reviewers' comments, we conducted thorough analysis, and our study confirmed that training the model on noisy images can open new doors for reward hacking such as using the presence of noise (and not evidence removal) to lower the reward on dispreferred answer (reviewer JCva, first table). We therefore include the visual preference as a per-pair weight that closes the gap between grounded generation and ungrounded generation in the frozen reference model.
  2. Comparison with alternative visual objectives: Reviewers JCva and jr7E proposed incorporating abstention/uncertainty on the corrupted image. We included the experiments and under matched settings, FiRe-MPO outperforms abstention targets and the CoMedPO-style margin.
  3. Location shortcut: The reviewers JCva and jr7E cautioned that the model might exploit the spatial position of the corruption as an artificial spatial hint. In the updated visual preference, the region never enters the policy’s input. Location-controlled experiments (Reviewer jr7E, Table 2) show that the improvement comes from clinically correct localization rather than from reweighting itself.
  4. Others: All main results are now reported over 3 seeds. We have also expanded the paper’s clarifications and its positioning against prior work, and these changes will be included in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.