- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MedCTA, a multimodal benchmark for clinical tool Agents, focusing on evaluating whether Agents can autonomously organize tools and complete medical tasks without explicit tool invocation instructions. The data set adopts a semi-automated construction process: starting from existing medical VQA and medical education resources, the original question is rewritten into a task with implicit tool requirements, and then technical annotators and clinical experts revise and verify the tool trajectory and final answer. The data covers 34 anatomical regions and body systems, including CT, MRI, X-ray, pathology images and medical reports. Five categories of tools are provided: OCR, ImageDescription, RegionAttributeDescription, GoogleSearch and Calculator.
Critical strengths
-
Clinical experts participated in the validation of tasks and reference trajectories. This approach is more reliable than relying solely on LLMs to automatically generate medical QA pairs or tool-use trajectories.
-
MedCTA provides both the final answer and an executable intermediate trajectory. Unlike traditional medical VQA, MedCTA goes beyond mere accuracy; it also records tool names, input parameters, observations returned, and intermediate reasoning steps.
-
The framework covers a diverse range of medical imaging modalities and task types. By consolidating inputs—such as CT scans, MRI, histopathology, and medical reports—under a unified interface, it becomes possible to test whether multimodal models can transfer their tool-use capabilities across different contexts.
-
The authors did not rely exclusively on an "LLM-as-judge" evaluation. As detailed in Appendix C.6, clinical experts reviewed 144 model outputs across 36 tasks and four different models.
Critical weaknesses
1.The authors aim to transform what are otherwise straightforward medical VQA tasks into ones that require autonomous planning and tool use; however, they fail to sufficiently demonstrate that these tasks truly necessitate multi-step tool invocation to be solved.
Even if medical experts deem a particular tool chain to be clinically sound, it does not follow that the problem cannot be correctly solved without employing that specific chain. Essential experiments should include a comparison—across the same set of tasks—between a tool-free VLM, an agent with optional tool access, and an agent with constrained tool usage. Furthermore, it should be tested whether the removal of a supposedly "necessary" tool actually leads to a decline in accuracy or the failure to obtain critical evidence.
2.The authors report an API/protocol error rate of 64.2% and an under-call rate of 99.2%, with all models achieving a strict trajectory success rate of zero.
While these results might suggest that the models are weak, they also imply that most models failed to even reach the relatively stable stage of tool execution. A particularly noteworthy detail appears in Appendix F, Figure 9: the ReAct-style execution prompt used by the authors explicitly allows models to conclude the process if they already know the answer or deem tools unnecessary: “If you already know the answer, or you do not need to use tools” This runs counter to the multi-step tool execution that the benchmark aims to evaluate; a model might simply be following the prompt by providing an immediate answer, yet it ends up being penalized for failing to match the number of tool calls found in the reference trajectory. At the very least, a control experiment is warranted: one that compares the current ReAct text-based protocol against native function calling (for models that support it) and an execution protocol that incorporates format correction or retries.
3.The paper constructs a clinically validated "reference trajectory" for each task, using it to evaluate subsequent tool selection, parameter settings, and overall trajectory alignment.
While the authors account for some cases of functional equivalence among tools, they do not establish that there is only one valid sequence or length of tool invocations for any given problem. For instance, a medical imaging task might be approached by performing OCR before image captioning, or vice versa; for some straightforward cases, it might even be reasonable to reach a conclusion directly from the raw image. However, the benchmark’s current step-level evaluation is strictly dependent on the specific sequence defined in the reference trajectory. This directly impacts the authors' primary diagnostic conclusions: it remains unclear to what extent instances labeled as "incorrect tool recruitment" or "premature stopping" represent actions that are medically unsound, as opposed to simply failing to match the single designated "gold trace."
4.The experimental analysis is insufficient to establish the claimed failure mechanisms.
A large portion of the paper is dedicated to demonstrating the model's poor performance, yet the main text—spanning nine pages—fails to provide an adequate explanation of the underlying mechanisms driving this low performance. Furthermore, the relevant analysis is relegated to a late section of the paper (Appendix H.3), which poses a significant obstacle for readers; moreover, this conditional analysis itself has inherent limitations. In the aggregate results for the full model, the final accuracy under the condition where the first tool step is correct is 8.6%—lower than the reported baseline of 10.7%. In cases where at least two "gold" (ground-truth) steps are matched, there are only 36 such instances, and the accuracy is even lower. Consequently, these findings fail to sufficiently demonstrate that resolving a specific, identified execution bottleneck would lead to improved task performance.
Questions for authors
Same as weaknesses
2 comments
Thanks for your efforts. The additional experiments help me better understand your paper. I will remain my score.
Reply to Reviewer V1rS
Thank you for the detailed critique; we agree with the core of all four points. We have not run the control experiments you list. Rather than report rushed results, we checked what the existing results support and narrowed the claims accordingly.
1. Do the tasks require multi-step tool use? We agree that clinician approval of a chain does not prove necessity, and our data confirm part of your concern. The existing evidence points both ways:
The bound uses per-task scores in [0,1]: the number of tasks with Δi>0 is at least nΔˉ. For the nine text-only models (GPT-oss-20B, Qwen3-8B, the three DeepSeek models, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B and Phi-4), the tools are the only route to the image. The revision describes MedCTA as a mix: some tasks are answerable on shorter paths (likely including the Table 2 organ-identification query), and on a substantial share the validated path measurably helps. We drop the claim that every task requires multi-step tool use (Sec. 3.2, Table 1).
2. The prompt allows direct answers. The clause comes from the GTA protocol (Fig. 9) and is deliberate, since forcing calls would reward over-calling. But you are right that the step-level penalties then count a correct, prompt-compliant direct answer as an under-call. In the revision:
3. One reference sequence. Agreed for the step-level metrics. ToolAcc is teacher-forced (predict step n+1 given n reference steps), so it measures agreement with the reference prefix, and it is now described that way. Equivalent tools are accepted but reordering is not, so "incorrect tool recruitment" and "premature stopping" become "deviation from the reference tool" and "fewer calls than the reference". Gacc and Table 5 do not depend on this choice.
4. Mechanisms and H.3. Your reading of Table 12 is correct. With CIs, it supports no mechanism in either direction:
Across all 17 cells, only one has p<0.05 uncorrected (open-source schema-clean arguments, 10/70 vs. 71/1,000, p=0.036), and none passes Bonferroni (p<0.0029). Figure 3 reproduces from Table 4, but model-level correlations partly reflect overall capability. The revised Sec. 4.2 therefore rests the mechanism discussion on the one intervention we have, gold routing, reports Tables 12 and 14 and Fig. 3 as descriptive with CIs, and moves this analysis into the main text. Selection is stated only as a hypothesis for why "first tool correct" does not help: rollouts that enter the tool loop meet protocol failures, while prior-based answers do not.