- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that tool use alone is insufficient in medical VQA: the same segmentation mask can yield different answers under different renderings, and even correct evidence can overturn initially correct predictions. It proposes TUNE, a three-stage harness (Target Planner, Evidence Presenter, Answer Critic), and introduces Mask-Evidence MedVQA, reporting an average gain of 9.1 points across seven datasets. The problem is important, but the central causal claims and comparisons are insufficiently controlled.
Critical strengths
The observation that the same mask under different renderings can flip answers is a useful cautionary result for medical VQA.
The Answer Critic preserves the no-tool answer and selects between two completed candidates, which is a sensible design; Table 5 showing one-shot Critic hurts while trained Critic helps is a useful negative result.
The benchmark covers four imaging modalities and includes human verification, which is a reasonable direction.
Critical weaknesses
W1. The causal claim that “presentation is part of reasoning” is not isolated from dataset-specific shortcuts. The large bidirectional rendering effects in Table 4 are equally consistent with the Presenter overfitting to interface–answer correlations in the training split. A cross-dataset rendering transfer test is missing.
W2. The Answer Critic’s supervision and evaluation are too narrow. The Critic is trained and evaluated only on decisive pairs with exactly one correct answer. Its behavior when both candidates are wrong or both are correct is not reported, leaving deployment robustness unclear.
W3. The comparison to CARE is not controlled. TUNE and CARE differ in backbone size, access type, and likely training budget/tooling, so the claimed advantage cannot be attributed to the harness design without a controlled comparison.
W4. Benchmark construction and generality are under-reported. The question-generation LLM, filtering criteria, annotator agreement, and question-type distribution are missing from the main text, and the “answerable from the original image alone” criterion is not quantified.
Questions for authors
Can the authors add a cross-dataset rendering transfer experiment (e.g., train the Presenter on COVID-QU-Ex and test on NeoPolyp, and vice versa)? If the policy does not transfer, how does this support the claim of learning general presentation properties?
What is the proportion of decisive vs. non-decisive pairs in the full benchmark? Which answer does the Critic default to on non-decisive pairs?
Can the authors provide a controlled comparison between TUNE and CARE (or CARE without its answer-review module) using the same backbone, segmentation tool, and training budget?
Thank you for the additional experiments. The fixed-weight ablation addresses my main concern about how much the Presenter and Critic contribute beyond fine-tuning the answer model. I also found the wrong-mask control informative. With the same incorrect masks, adding the Critic brings accuracy from 63.5% back to the no-tool level. These results make me inclined to raise my rating to Weak Accept.
I have one remaining question about Fig. 3. Which checkpoint generated the two no-tool candidates in Resample, and which generated the candidates in TUNE? On COVID-QU-Ex, subtracting regressions from corrections gives a 13.5-point gain for TUNE, while the new comparison with π2 held fixed gives a 10.8-point gain over no tool. Could you clarify which checkpoint produced the reference no-tool answer in Fig. 3 and identify the checkpoints in the caption? This would help me interpret how much of TUNE’s gain over Resample can be attributed to the mask evidence.