acceptodds
Under review as a conference paper at ICLR 2027

Harnessing Medical Visual-Language Models with Agentic Evidence Management

Abstract

Medical vision–language agents increasingly rely on segmentation tools, and recent work has focused on acquiring the right mask, often treating a good mask as the end point of tool use. However, do these tool-augmented agents actually make effective use of the evidence once it is obtained? To answer this question, we need a benchmark that separates evidence use from segmentation quality, spans multiple medical image types, and allows the same mask to be presented in different ways. Existing medical VQA datasets do not jointly provide these properties. We therefore construct Mask-Evidence MedVQA, a 2.5K-example benchmark spanning four image types, where each question is paired with an answer-relevant ground-truth mask and multiple possible renderings. Our study reveals two limitations of direct tool use. First, evidence presentation matters. Different renderings of the same fixed mask can substantially change model behavior. Second, masks introduce an evidence trade-off. Highlighting a relevant region can expose diagnostic evidence the model previously missed, but the added visual cue can also induce incorrect associations. As a result, even a ground-truth mask can correct some answers while turning initially correct ones into incorrect ones. Together, these findings show that acquiring the right mask is not enough; an agent must also decide how to present the evidence and whether it should change the answer. We therefore introduce TUNE (Tool Use is Not Enough), an evidence-management harness that explicitly manages what evidence to acquire, how to present it, and whether to accept the resulting answer update. Across three standard medical VQA benchmarks, TUNE improves its backbone's mean no-tool accuracy by 11.5 points and exceeds the previous state-of-the-art medical agent by 3.5 points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical vision-language models, tool-augmented reasoning, medical visual question answering, visual evidence management, answer verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_84JpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that tool use alone is insufficient in medical VQA: the same segmentation mask can yield different answers under different renderings, and even correct evidence can overturn initially correct predictions. It proposes TUNE, a three-stage harness (Target Planner, Evidence Presenter, Answer Critic), and introduces Mask-Evidence MedVQA, reporting an average gain of 9.1 points across seven datasets. The problem is important, but the central causal claims and comparisons are insufficiently controlled.

Critical strengths

The observation that the same mask under different renderings can flip answers is a useful cautionary result for medical VQA.

The Answer Critic preserves the no-tool answer and selects between two completed candidates, which is a sensible design; Table 5 showing one-shot Critic hurts while trained Critic helps is a useful negative result.

The benchmark covers four imaging modalities and includes human verification, which is a reasonable direction.

Critical weaknesses

W1. The causal claim that “presentation is part of reasoning” is not isolated from dataset-specific shortcuts. The large bidirectional rendering effects in Table 4 are equally consistent with the Presenter overfitting to interface–answer correlations in the training split. A cross-dataset rendering transfer test is missing.

W2. The Answer Critic’s supervision and evaluation are too narrow. The Critic is trained and evaluated only on decisive pairs with exactly one correct answer. Its behavior when both candidates are wrong or both are correct is not reported, leaving deployment robustness unclear.

W3. The comparison to CARE is not controlled. TUNE and CARE differ in backbone size, access type, and likely training budget/tooling, so the claimed advantage cannot be attributed to the harness design without a controlled comparison.

W4. Benchmark construction and generality are under-reported. The question-generation LLM, filtering criteria, annotator agreement, and question-type distribution are missing from the main text, and the “answerable from the original image alone” criterion is not quantified.

Questions for authors

Can the authors add a cross-dataset rendering transfer experiment (e.g., train the Presenter on COVID-QU-Ex and test on NeoPolyp, and vice versa)? If the policy does not transfer, how does this support the claim of learning general presentation properties?

What is the proportion of decisive vs. non-decisive pairs in the full benchmark? Which answer does the Critic default to on non-decisive pairs?

Can the authors provide a controlled comparison between TUNE and CARE (or CARE without its answer-review module) using the same backbone, segmentation tool, and training budget?

Reviewer_htSyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed TUNE (Tool Use is Not Enough), working on how does the medical VLM uses the segmentation evidence and how does these evidence influence the final answer. The proposed a framework combined of Target Planner, Evidence Presenter, and Answer Critic to better present the medical evidence provided by the segmentation model and improve the final answer by considering the information bias. The proposed method was tested with the proposed Mask-Evidence MedVQA of 2.5k questions, and also 3 out-of-distribution common MedVQA datasets. The proposed framework based on Qwen 3.5-9B show a uniform improvement on all 7 datasets comparing the vanilla model. Beats the best baseline by 3.5% accuracy.

Critical strengths
  1. The question that paper works on is very interersting and novel. Previous works mainly focusing on how to gather good evidence (segmentation/bounding boxes/...), but this work address the influence of different evidence presentation schema, and also take the possibility of incorrect evidence that leads to wrong answer into consideration. The experiments shows a clear support to this argument, even GT masks can leads to similar amount of regression as its corrections. But the proposed method can maximize the corrections when evidence is presented while minimizing its negative influence.
  2. The paper has evaluated the proposed method on 7 different datasets, include different image modalities and also benchmarked against different baselines, including both open-source and properitatary SOTA. It has also evaluated the improvement brought by the proposed framework on open Qwen model as well as closed GPT-5.6. The component results also go beyond a final-system comparison: in Table 5, adding the trained Critic raises BreastUS from 85.9% to 89.8% and NeoPolyp from 77.5% to 83.0%, while the one-shot Critic lowers both scores.
  3. The analysis on correction vs. regression is very informative. It helps break down the performance change into positive and negative influence. Demonstrating the reason of model failure even when GT mask is presented.
Critical weaknesses
  1. The first weakness here is missing information on the model training and dataset curation. According to the section 4 and figure 2, the proposed method uses a shared VLM but train it for 3 different roles using different prompt. However, the target planner is never trained, and it is unknown whether we uses the frozen vanilla base VLM in the final framework as the planner, or using the tuned Final VLM. This can bring concerns on the models capability on this specific task, since model are never tuned for the target planner role, will its performance degenerate as the model trains for evidence presenter and answer critic? Also, for the proposed Mask-Evidence MedVQA, there is no detailed information about the data curation process, the portocol of selecting the data. There is no explaination on what is SLAKE* as well. In section 4.4, it says the training process used a subset for the stage 1 training, but there is also not description on this. The reviewer believes more infomation should be provided to fully understand this work.
  2. Another question is about the quality of the mask segmentation tool. The paper focus on how to better use medical evidence in the VQA process and claims the way of presenting the evidence is critical, and even correct evidence may lead to incorrect answer. However, this claim kind of assumes the segmentation mask quality is satisfactory already. However, this may not be the case, MedSAM and BiomedParse are proved to be inferior in many cases, especially for OOD data (Du et al. 2026), also, the MedSAM segmentation largely relys on the bounding box quality, which is provided by the frozen target planner according to section 4.1. There is no direct evaluation on the quality of the segmentation mask on all datasets (Figure 3 only reports 3 numbers), the performance gain of answer critic and different rendering method may actually highly correlated to the correctness of the mask. An incorrect mask naturally leads to a bad answer, regardless to the rendering. It is also interesting to see what will happen when combining GT mask with TUNE.
  3. Lastly, the paper mainly focus on the final VQA accuracy metric, so the contribution of evidence presenter and answer critic is not clear. Even though Figure 3 provides an indirect analysis on the corrections and regressions, it does not demonstrate the actual contribution of each component. Table 5 includes useful component ablations, but does not state which model weights are held fixed between rows and provides no matched fine-tuned no-tool or direct-tool baseline. Since all roles share the updated model (Section 4.4), answer generation itself may improve. This leaves unclear how much the proposed presentation and review decisions contribute to the headline gain.
Questions for authors
  1. How does the model updates for three different roles? Are they using a shared model but only update the weight for evidence presenter and answer critic? Will it be better to use seperate model for each role? For example, use different LoRA for each role.
  2. What is Resample in the Figure 3? The caption only says resample and select, but it is not clear what it is actually.
  3. How does the data actually curated, what is the criteria of selecting the data? And what is SLAKE* stands for?
1 comment

Thank you for the additional experiments. The fixed-weight ablation addresses my main concern about how much the Presenter and Critic contribute beyond fine-tuning the answer model. I also found the wrong-mask control informative. With the same incorrect masks, adding the Critic brings accuracy from 63.5% back to the no-tool level. These results make me inclined to raise my rating to Weak Accept.

I have one remaining question about Fig. 3. Which checkpoint generated the two no-tool candidates in Resample, and which generated the candidates in TUNE? On COVID-QU-Ex, subtracting regressions from corrections gives a 13.5-point gain for TUNE, while the new comparison with π2\pi_2 held fixed gives a 10.8-point gain over no tool. Could you clarify which checkpoint produced the reference no-tool answer in Fig. 3 and identify the checkpoints in the caption? This would help me interpret how much of TUNE’s gain over Resample can be attributed to the mask evidence.

Reviewer_csK3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on whether medical vision-language model (VLM) agents make good use of a segmentation mask once a tool has returned it. To this end, the paper builds Mask-Evidence MedVQA, a set of 2,494 LLM-generated questions derived from four segmentation datasets. Each question is paired with a ground-truth mask that can be rendered in many visual styles. The paper proposes a harness in which a single VLM plays three roles. Training follows a two-round "harness–model co-evolution" procedure. The proposed method achieves improved performance.

Critical strengths
  1. The target question is useful.
  2. The method is well-motivated.
  3. The comparison is extensive, covering open-weight, proprietary and medical-specific models.
Critical weaknesses
  1. The novelty of the methodology is limited. Previous work has already shown that the way a region is marked changes a VLM's output. Examples include red circles, Set-of-Mark prompting, Fine-Grained Visual Prompting with mask-based prompts and ViP-LLaVA; medical examples include MedVP and FAVP. The paper itself notes that FAVP "finds that preferred styles vary across datasets", which is essentially the paper's own conclusion that "no interface is consistently beneficial across image types". CARE also already selects among crops, masks and global indicators. Choosing between a no-tool answer and an evidence-based answer closely resembles adaptive retrieval and evidence-trust methods in RAG, such as Self-RAG and Corrective RAG, as well as learned verifiers and answer selectors. The "co-evolution" consists of two rounds in which the authors add components by hand and then fine-tune the model. This is not automatic joint optimisation. Overall, the contribution is an engineering combination of known ideas for medical VQA. That is acceptable, but it means the empirical evidence must be very solid, and the following concerns show that it is not.

  2. The performance gains are mixed with in-domain fine-tuning. Full TUNE fine-tunes Qwen3.5-9B on trajectories from D_train, keeping only rollouts that reach the correct answer (T0) and decisive pairs labeled by the ground truth (T1) (Sec. 4.4). This is effectively supervised training on correct answers to in-domain questions, whereas almost all baselines in Table 2 are zero-shot. Fig. 3 suggests that most of the gain comes from training rather than from better use of evidence. On COVID-QU-Ex, TUNE corrects 16.8% of the questions using predicted masks. With the base model, the ground-truth mask corrects only 6.08%, and these corrections are fully cancelled out by 6.08% regressions; direct tool use corrects 5.42%. A harness that merely presents evidence better and chooses between answers is unlikely to triple the corrections achieved with a perfect mask, whereas a model that has learned the task during fine-tuning could well do so. The same concern applies to the large jumps on SLAKE (+15.4) and OMVQA-3k (+17.0), which are much larger than the gains on datasets where the mask is the primary evidence. Missing ablations: (a) the fine-tuned model π2 with tools turned off; (b) Qwen3.5-9B fine-tuned on the same training questions without any tool (plain SFT or rejection-sampling fine-tuning); and (c) the TUNE harness running on the base model, with only the Critic trained. Without these controls, the abstract's claim that TUNE "improves its backbone's mean no-tool accuracy by 11.5 points" cannot be attributed to evidence management.

  3. The benchmark is under-specified, and the masks may leak the labels. Basic information is missing: which LLM wrote the questions and with what prompt, how an LLM "receives" a mask, how many questions were generated and how many were filtered out, the answer format (multiple-choice or open-ended), the number of options, the label distribution, and the size of each of the four subsets. The source datasets are classification/segmentation datasets with only two or three labels, so most questions are probably simple binary or ternary classification questions ("What is the most likely diagnosis?"). Yet no majority-class or chance baseline is reported. In BUSI, normal images have no lesion mask, so the remaining labels are probably benign vs. malignant at a ratio of about 2:1 (437 vs. 210 images). A majority-class baseline would then already reach about 67.5%. Possible label leakage through the masks: (a) In COVID-QU-Ex, infection masks are non-empty only for COVID-19-positive images, so if infection masks are used, the mere presence of a mask gives away the answer. (b) The original NeoPolyp masks encode the polyp class (neoplastic vs. non-neoplastic) by colour. (c) For KiTS23 (3D CT), the choice of which 2D slice to show can leak the label. The paper does not say how the masks were binarised or how the slices were chosen. The predicted-mask Dice of 0.973 on COVID-QU-Ex is typical of lung-field segmentation: the original dataset paper reports a Dice of about 98% for lungs and about 88% for infection regions. If the "answer-relevant" mask for COVID-QU-Ex is the lung field, this contradicts the paper's own argument that lung masks provide "little information", and it would also explain why the ground-truth mask yields zero net gain on this dataset.

  4. The training data, data splits and possible test contamination are not described. D_train is not defined: the paper does not say which datasets and splits it uses, how many examples it contains, or how large T0 and T1 are. Table 1 calls the benchmark "-Eval", but it is unclear whether Mask-Evidence MedVQA has a separate training split. OmniMedVQA has no official training split and is built from about 73 public classification datasets, many of which share modalities with the training data (breast ultrasound, chest X-ray, endoscopy). If the training images come from overlapping sources, image-level leakage is possible. TUNE's 98.2% on OMVQA-3k (+17.0 over Tool) is near the ceiling and calls for an explicit leakage check. The paper also does not describe how the 3k subset was sampled. Implementation details are missing, including whether LoRA or full fine-tuning is used, the learning rate, the number of epochs, the prompts, the renderer settings, which VLM predicts the bounding box for MedSAM, the decoding temperature and the compute budget.

Questions for authors

Please see above for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.