acceptodds
Withdrawn from ICLR 2027

MANY VIEWS, FEW INSIGHTS: RETHINKING THE ALIGNMENT OF VISION LANGUAGE MODEL FOR CLINICAL MULTI-IMAGES ANALYSIS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Clinical diagnosis routinely requires synthesizing evidence across multiple images of the same patient, yet current vision-language models (VLMs) fall far short, largely because they fail to integrate evidence across images. The standard alignment recipe to align VLM on clinical multi-images is supervised fine-tuning (SFT) optionally followed by reinforcement learning (RL). This baseline recipe carries several under-studied free choices, including whether to add an RL stage, the SFT epoch count, and the LoRA rank. We systematically compare these choices, evaluating each configuration on in-domain and out-of-domain clinical benchmarks, and distill several insightful findings showing that existing recipes leave substantial performance on the table for assessing multiple images. We trace this gap to a specific training-time failure: SFT-only models show little hallucination on in-domain benchmarks but hallucinate substantially more on out-of-domain benchmarks, pointing to non-visual shortcuts that break under distribution shift. We call this supervision-modality mismatch. Building on these findings, we propose MULTI-MULTI for clinical multi-modal and multi-image analysis, which decomposes multi-image reasoning into an explicit, per-image-grounded structure and introduces Image-Swap Supervision (ISS) which pairs a case's images with a mismatched clinical history and mixed into fine-tuning samples so each reasoning step is recoverable only from the images. Despite this lightweight recipe, the resulting fine-tuned 8B open-source model reaches in-domain accuracy comparable to zero-shot open models two orders of magnitude larger, and closes part of the gap to leading proprietary models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vlm, alignment, clinical multi-modal analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i4HfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates post-training alignment strategies for clinical multi-image reasoning through extensive experiments on SFT, RL, training epochs, and LoRA ranks. Based on the observed in-domain and out-of-domain performance differences, the authors identify a potential failure mode termed supervision-modality mismatch. To address this, they propose MULTI-MULTI, which combines decomposed reasoning supervision with Image-Swap Supervision (ISS). Experiments on MedThinkVQA and MedXpertQA demonstrate improved in-domain accuracy and mixed results in out-of-domain generalization.

Critical strengths
  • The paper systematically observes the effects of SFT epochs and LoRA ranks across four VLM backbones, alongside preliminary RL experiments. -MULTI-MULTI introduces structured reasoning supervision and ISS without architectural modifications or additional RL. The method achieves notable in-domain improvements and demonstrates potential applicability across different VLM backbones.
Critical weaknesses
  • There is insufficient evidence for supervision-modality mismatch. The failure mechanism is mainly inferred from ID/OOD performance differences and qualitative examples. However, these differences may also arise from dataset distribution shifts unrelated to visual grounding
  • There is a lack of validation of meaningful cross-image reasoning. The cross-image comparison targets are constructed from lexical differences between image captions, which may not reflect clinically meaningful relationships. Moreover, the evaluation primarily focuses on final-answer accuracy without directly assessing intermediate reasoning quality.
Questions for authors
  • Can the authors provide direct evidence that supervision-modality mismatch results from insufficient visual reliance, rather than general distribution shift?
  • Can the authors evaluate whether ISS improves visual grounding or reduces hallucinations beyond final-answer accuracy?
  • How do the authors ensure that caption-based cross-image comparisons capture clinically meaningful relationships?
  • Can the authors provide stronger controlled experiments to distinguish the benefits of the proposed reasoning structure from changes in supervision format and training-data composition?
Reviewer_hWPGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the behavior of vision-language models fine-tuned for medical multi-image question answering. The authors compare supervised fine-tuning across four backbones, varying training epochs and LoRA rank, and conduct preliminary experiments with subsequent GRPO training. They then propose MULTI-MULTI, which organizes supervision into per-image descriptions, cross-image comparisons, and answer justification. Image-Swap Supervision (ISS) introduces mismatched image–history pairs to encourage reliance on visual information.

The models are trained on 7,347 MedThinkVQA cases and evaluated on 720 in-domain cases and 419 multi-image cases from MedXpertQA. The method improves in-domain accuracy across the four backbones, while out-of-domain gains vary by backbone. The paper also provides component ablations, breakdowns by disease category and image count, and qualitative examples.

Critical strengths

1、The paper provides useful empirical observations about fine-tuning for medical multi-image tasks. The comparisons across four backbones, training epochs, and LoRA ranks show that in-domain improvements do not necessarily transfer out of domain, and that longer training does not consistently improve performance. These observations are informative for adapting medical VLMs.

2、The method is simple and produces clear in-domain gains. MULTI-MULTI modifies the supervision data without requiring an additional reinforcement learning stage. Qwen3-VL-8B reaches 49.31% in-domain accuracy, and improvements are also reported for the other three backbones. The component ablation and subgroup analyses help characterize the gains, and the authors appropriately identify the limitations of small subgroups.

Critical weaknesses

1、The claims about visual grounding and hallucination lack direct validation. Final-answer accuracy and a few qualitative examples do not establish that MULTI-MULTI improves image descriptions, cross-image reasoning, or hallucination rates. The explanation that ordinary SFT encourages non-visual shortcuts also lacks direct support.

2、The incremental methodological contribution remains unclear. The ablations do not isolate the benefit of the lexical-difference-based comparison sentence or show whether ISS alone could achieve comparable gains. Thus, the added value of the proposed cross-image design and its combination with ISS remains uncertain, while out-of-domain benefits are inconsistent.

Questions for authors

1、Beyond final-answer accuracy, how do the authors verify that MULTI-MULTI improves the correctness of image descriptions and cross-image reasoning?

2、Does adding ISS to ordinary SFT, without the proposed reasoning decomposition, achieve gains comparable to the full method?

3、Beyond per-image descriptions, how much does the cross-image comparison sentence contribute to performance when the rest of the supervision is kept unchanged?

4、What are the format-compliance rates mentioned in Appendix A.4? Can incorrect answers be distinguished from failures caused by formatting or output truncation?

Reviewer_AtrFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates clinical multi-image post-training across four vision-language backbones, 60 SFT configurations, and a preliminary set of three completed GRPO runs. It then proposes MULTI-MULTI, which trains on indexed per-image descriptions, an automatically generated cross-image comparison, and an answer justification. Image-Swap Supervision (ISS) pairs some cases’ images with a history from another ICD-10 chapter. On Qwen3-VL-8B, the selected ablation baseline, decomposition alone, and the full method achieve 40.42%, 46.39%, and 49.31% in-domain accuracy; their out-of-domain accuracies are 24.34%, 21.72%, and 25.54%.

Critical strengths

The 60-configuration SFT sweep is a useful empirical contribution. It shows that training longer, increasing LoRA rank, or adding the tested GRPO stage does not reliably improve this task; the authors appropriately describe the limited RL experiments as preliminary. The structured target is also valuable: attaching a claim to a numbered image makes errors easier to locate, and the reported in-domain improvement is substantial.

Critical weaknesses
  1. Baseline selection materially changes the OOD conclusion. Figure 5(b) compares the full method’s 25.54% OOD accuracy with a selected SFT baseline of 24.34%, a gain of about five correct answers among 419 cases. Yet Table 4 reports 28.16% OOD for another rank-64 SFT checkpoint, and 26.97% for the unmodified backbone. This does not negate the within-recipe ablation, but it does not establish an OOD advantage over a properly selected SFT baseline. The checkpoint-selection rule, paired uncertainty, and repeated-seed results are needed.
  2. The proposed visual-grounding mechanism is not identified. A history from a different ICD-10 chapter may be recognized as inconsistent using the text, question, options, or clinical priors; the original question and answer remain in the swapped example. The ID and OOD datasets also differ in construction and image-count distribution. Thus, their accuracy gap and three qualitative examples cannot by themselves demonstrate that ISS increases causal dependence on the images.
Questions for authors
  1. Which epoch and selection rule produced each Figure 5(b) point? Under matched rank, training budget, and prompt, how does the full method compare with a validation-selected SFT checkpoint, with paired uncertainty? In particular, how should readers interpret 25.54% versus Table 4’s 28.16% OOD result?
  2. On the same test cases, how do answers and per-image claims change when relevant images are omitted or permuted, text is withheld, or the history is replaced? Can unsupported visual claims be assessed by blinded annotation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.