- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a counterfactual-intervention-based learning framework called CARAE for improving reliability of multimodal large language models (MLLMs). Using chest X-ray images as a testbed, the authors first investigate the behaviors of models in identifying targeted and matched control edits. Then, they propose to optimize models’ relative behavior under these constructed interventions by learning more robust question-conditioned visual representations and via a gated residual injection module. Finally, they evaluate the performance of the counterfactual-intervention optimized models through cross-dataset experiments on two benchmark datasets. Experiments demonstrate the effectiveness of CARAE against several baselines.
Critical strengths
S1. The paper presents a new counterfactual-intervention-based learning framework called CARAE to improve model reliability. Employing targeted and matched control interventions simultaneously to investigate and optimize model behavior is an interesting idea.
S2. The way the authors propose optimize models’ relative behavior under these interventions and to learn more robust question-conditioned visual representations through this seem novel. Moreover, the authors demonstrate the effectiveness of using these constructed interventions to optimize models’ relative behavior against the competing approaches.
S3. The authors present detailed ablation studies to assess the contribution of each component of the suggested framework.
Critical weaknesses
W1. Most of the critical technical details are given in the Appendix, which makes the paper a hard-read.
W2. I found the paper highly related to the hallucination detection/migitation techniques suggested for MLLMs. However, no specific discussions and/or comparisons are provided in the paper, which weakens the paper’s quality.
W3. The following recent work is highly relevant to the submitted work, but is missing in the paper.
- Lu et al., Counterfactual Anatomy-Guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs, MICCAI 2026
W4. Experiments are carried out by using Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct as the main backbones. No other generic MLLMs or medical MLLMs have been evaluated.
W5. The counterfactual interventions only include masking (filling the regions with mean color).
Questions for authors
Q1. Are these findings still be valid for MLLM families other than Qwen2.5-VL-3B-Instruct, especially for medical MLLMs?
Q2. Could interventions other than masking (blurring, adding noise, mix-up, etc.) could be used within the suggested framework? If so, how do they affect the learning?