- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates post-training alignment strategies for clinical multi-image reasoning through extensive experiments on SFT, RL, training epochs, and LoRA ranks. Based on the observed in-domain and out-of-domain performance differences, the authors identify a potential failure mode termed supervision-modality mismatch. To address this, they propose MULTI-MULTI, which combines decomposed reasoning supervision with Image-Swap Supervision (ISS). Experiments on MedThinkVQA and MedXpertQA demonstrate improved in-domain accuracy and mixed results in out-of-domain generalization.
Critical strengths
- The paper systematically observes the effects of SFT epochs and LoRA ranks across four VLM backbones, alongside preliminary RL experiments. -MULTI-MULTI introduces structured reasoning supervision and ISS without architectural modifications or additional RL. The method achieves notable in-domain improvements and demonstrates potential applicability across different VLM backbones.
Critical weaknesses
- There is insufficient evidence for supervision-modality mismatch. The failure mechanism is mainly inferred from ID/OOD performance differences and qualitative examples. However, these differences may also arise from dataset distribution shifts unrelated to visual grounding
- There is a lack of validation of meaningful cross-image reasoning. The cross-image comparison targets are constructed from lexical differences between image captions, which may not reflect clinically meaningful relationships. Moreover, the evaluation primarily focuses on final-answer accuracy without directly assessing intermediate reasoning quality.
Questions for authors
- Can the authors provide direct evidence that supervision-modality mismatch results from insufficient visual reliance, rather than general distribution shift?
- Can the authors evaluate whether ISS improves visual grounding or reduces hallucinations beyond final-answer accuracy?
- How do the authors ensure that caption-based cross-image comparisons capture clinically meaningful relationships?
- Can the authors provide stronger controlled experiments to distinguish the benefits of the proposed reasoning structure from changes in supervision format and training-data composition?