acceptodds
Desk rejected at ICLR 2027

MedXS: Are Medical Multimodal Agents Reasoning Across Sources, or Merely Landing on the Right Answer?

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurate clinical decisions rarely rest on a single image. They interleave visual inspection, retrieval over case specific structured records, and corroboration against external medical knowledge. First generation medical agent benchmarks judge outcome alone, and a second adds process aware scoring of tool calls, yet both share one blind spot. They confine tools to a single evidence modality, never treating the structured record as co equal with imaging, and their process metrics check whether calls are well formed rather than whether cross source evidence is used, so a trajectory grounded on one source still scores fully. We introduce MedXS, a process verified benchmark that instantiates imaging, structured record, and medical knowledge as three co equal tool families over 2,000 clinician annotated tasks, with eight question types stratified into three levels by the number of sources fused and grounded on 6,000+ evidence checkpoints, scored by an unweighted, tier increasing suite of six metrics spanning outcome, process, and cross source integration. Across frontier agents, the strongest reaches only 68.2% average result yet a 97.4% workflow score, while its Level 3 coverage collapses to 28.8% and knowledge tools drop to 2.8% of all calls. This exposes tool tunnel vision, a persistent gap between correct answers, valid tool ordering, and real cross source integration that stays invisible to both outcome and call level scoring.

Primary area:
datasets and benchmarks
Keywords:
Medical Agents, Cross-Source Integration, Process Verification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V8dXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

MedXS is a process-verified benchmark designed to evaluate medical multimodal agents on evidence acquisition and utilization across medical imaging, structured records, and external knowledge. Comprising 2,000 stratified tasks, nine specialized tools, and over 6,000 reference evidence checkpoints, the benchmark employs six complementary metrics to assess aspects such as answer correctness, workflow validity, and multi-source coverage. Evaluations of multiple general-purpose and specialized agents reveal significant gaps between high workflow validity and low multi-source coverage, indicating limitations in complex cross-source reasoning. Additionally, the study incorporates comprehensive validation measures, including physician reviews, independent-judge checks, and control analyses, to assess the framework's reliability.

Critical strengths
  1. Addresses a Critical Evaluation Gap: Identifies key limitations in existing benchmarks by demonstrating that accurate final answers and well-formed tool trajectories do not guarantee genuine evidence utilization.
  2. Establishes a Unified Multi-Source Benchmark: Connects medical imaging, structured records, and external knowledge via a unified interface, featuring 2,000 stratified tasks and 6,073 reference checkpoints to systematically measure evidence requirements.
  3. Introduces a Multi-Dimensional Metric Suite: Proposes six complementary metrics covering answer accuracy, procedural compliance, and evidence acquisition, providing an actionable framework for developing auditable medical agents.
  4. Exposes Hidden Behavioral Discrepancies: Highlights the flaws of outcome-only evaluation by demonstrating a significant gap between high workflow validity (97.4%) and low multi-source coverage (28.8%).
  5. Ensures Rigorous Expert and Independent Validation: Incorporates a clinician review of 300 trajectories and an independent-judge assessment across 400 tasks to confirm evaluation reliability and domain relevance.
  6. Conducts Comprehensive Diagnostic Controls: Includes repeat-run stability analyses and oracle-retrieval benchmarks to rule out alternative explanations and solidify the study's empirical findings.
Critical weaknesses
  1. Construct Validity Issues in Multi-Source Integration: The multi-source coverage metric (mm) merely checks if all three source families were called and recorded, failing to prove that evidence from each source actually supported, constrained, or influenced the agent's final conclusion.
  2. Discrepancy in Evaluator Implementation: A contradiction exists between the formal definition (which requires evidence to be reflected in the final conclusion) and the executable evaluator (which evaluates tool results and evidence text without explicitly receiving the conclusion), making it unclear how substantive evidence usage is verified.
  3. Limitation of High Clinician Agreement: While high clinician agreement confirms that automatic scoring follows the operational protocol, it only validates label consistency rather than proving that genuine semantic cross-source reasoning occurred.
  4. Deviations from Real Clinical Realism: Relying on synthetic history and laboratory proxies to compensate for missing longitudinal data creates artificial task structures that differ significantly from messy, real-world clinical workflows.
  5. Unclear Annotation Reliability Across the Benchmark: Although clinician audits validated 300 generated trajectories, there is insufficient evidence regarding the inter-annotator consistency and reliability of the reference evidence graphs (RR), required source families (F∗F^*), and checkpoints across all 2,000 tasks.
  6. Confounding Retrieval Difficulties with Reasoning Failures: The oracle-retrieval experiment significantly boosted Level 3 coverage (from 28.8% to 46.1%), indicating that low performance stems partly from retrieval bottlenecks rather than purely from reasoning defects like "tool tunnel vision."
Questions for authors
  1. How does the evaluator verify that cited evidence substantively supports the final conclusion rather than simply being listed in the Evidence field, especially given that the judge function omits the Conclusion field?
  2. What specific annotation protocols—such as the number of independent clinicians and inter-annotator agreement metrics—were used to establish the baseline reference graphs, source requirements (F∗F^*), and checkpoints across all 2,000 tasks?
  3. Could you provide a breakdown that separates source-selection, retrieval, and integration failures to clarify whether the low multi-source coverage is primarily caused by retrieval bottlenecks or model reasoning limitations?
  4. How should the tier structure be interpreted as an increasing difficulty scale when the top agent's accuracy rises from 38.8% at Level 1 to 93.2% at Level 3 due to looser answer-matching criteria?
  5. What were the inclusion criteria and execution failure-handling protocols for the main leaderboard versus supplementary models, and which specific benchmark components (tasks, tool code, graphs, and knowledge snapshots) will be publicly released?
8 comments
AuthorsOpenReview ↗

Response to Reviewer V8dX (1/3)

We thank the reviewer for the thorough review and for recognising both the evaluation gap and the validation suite.

W1 / W3. Construct validity of mm, and what clinician agreement shows. We agree that mm shows each required family was attributed, not that it constrained the conclusion, and that clinician agreement confirms protocol fidelity rather than semantic reasoning. We therefore describe mm as attributed cross-source integration and add a counterfactual intervention. On 300 Level 3 tasks (100 each from doubao-seed-2.0-pro, GPT-5.5 and Claude-4.8-Opus), we replace only one family at a time with a clinically conflicting variant and keep all other evidence fixed, so the flip rate isolates reliance on that family. Records receive flipped laterality or an altered decisive measurement, knowledge a guideline record for a competing diagnosis, and imaging the region analysis of a matched case with a different finding. To separate reliance from run-to-run variation, we also re-ran the same tasks without any perturbation. Decision-flip rates (%):

Perturbed familyattributednot attributedno perturbation
Imaging71.418.24.3
Record64.812.54.3
Knowledge43.69.74.3

The gap between attributed and unattributed families holds within each system (all p<0.001p<0.001, two-proportion zz-test), and both are well above the no-perturbation rate. Knowledge flips are the lowest because many knowledge attributions only justify the answer: on tasks where clinicians marked knowledge as needed for the answer, the attributed knowledge flip rate is 68.7%, against 27.7% on justification-only tasks. This test does not turn mm into a direct measure of semantic reasoning. It is an orthogonal check that attributed evidence causally affects the decision, while mm itself certifies attribution.

W2 / Q1. The evidence judge omits the Conclusion field. The reviewer correctly identifies a mismatch between Eqs. 4/6 ("reflected in the conclusion") and the judge, which receives the Evidence field and the tool results (Fig. 9). We split the answer into these two fields so that a fluent Conclusion cannot hide a missing source (App. F, "From execution to audit"), but this lets an agent list evidence that does not support its Conclusion. The judge now also receives the Conclusion and labels each used unit as supporting, irrelevant or contradicting; only supporting units count toward vv and 1[F used]\mathbb{1}[F\ \text{used}]. We also fixed a per-family use flag in the evaluator, identified by Reviewer 3DPp. Both fixes are stricter and can only lower mm. On doubao-seed-2.0-pro, the model with the highest result rr averaged over the three tiers in Table 2 (68.2%), the per-family fix moves mm from 28.8 to 24.5, and the consistency step, which relabels 6.8% of credited units, then moves it to 23.2. With both fixes applied to all twelve models, Seed-1.8-Pro remains first in the coverage ranking, the order among general-purpose models is preserved, and the main finding holds: doubao-seed-2.0-pro keeps 97.4% workflow validity but only 23.2% Level 3 coverage. Eqs. 4 and 6 will read "attributed in the Evidence field and consistent with the Conclusion".

AuthorsOpenReview ↗

Response to Reviewer V8dX (2/3)

W4. Synthetic records vs. clinical realism. We agree, and App. K lists this as our first limitation. The history and laboratory proxies are provenance-tagged and reviewed by clinicians. In the 240-row audit, 91.7–96.7% of field groups were accepted with no edit or a minor edit (Table 8). Splitting the Level 3 results of doubao-seed-2.0-pro by whether RR depends on any synthetic field gives:

Subset#tasksrrmm (corrected)
Source-native / derived only21492.124.8
Includes synthetic fields38693.822.3

The source-native subset covers 214 of the 600 Level 3 tasks (35.7%). The gap between workflow and coverage persists on it, with slightly higher coverage, so the proxies do not create the gap. We do not claim that the proxies match the realism of de-identified longitudinal records. The audit checks individual fields, not the temporal consistency of a patient's history, and the split shows only that our main finding does not depend on the proxies. We will state this limitation in the main text and avoid wording that suggests deployment-level clinical realism.

W5 / Q2. Annotation reliability across all 2,000 tasks. All 2,000 tasks follow the two-stage Model+Clinician procedure of Sec. 3.1 under one locked protocol: a model drafts candidate questions and evidence chains, and one of six clinicians (four radiologists and two oncologists) verifies them and locks RR. To estimate reliability for the full benchmark, a second clinician independently re-annotated 400 tasks (160 / 120 / 120 per tier), stratified by tier, source dataset and required family set so that the subset matches the composition of the 2,000 tasks. Agreement before adjudication:

Tierκ\kappa on F∗F^{\ast}node F1 on RRcheckpoint agreement (%)
Level 1fixed (imaging)0.9294.2
Level 20.860.8690.8
Level 30.820.8288.6
All0.84 (L2–L3)0.8791.5

At Level 1 the required family set is imaging by construction, so κ\kappa on F∗F^{\ast} is computed over Levels 2–3. Agreement decreases with evidence depth, as expected, but stays in the almost-perfect band for F∗F^{\ast} and above 0.8 in node F1 at every tier. Most Level 3 disagreements concern whether the knowledge family is required, which motivates the answer-level relabelling of F∗F^{\ast} we report to Reviewer 3DPp. Disagreements were adjudicated by a senior radiologist. This annotation team is separate from the four clinicians who produced the clinician reference row in Table 2.

AuthorsOpenReview ↗

Response to Reviewer V8dX (3/3)

W6 / Q3. Retrieval bottleneck vs. reasoning failure. We agree that the oracle result shows retrieval matters, and we no longer call low mm purely a reasoning defect. Over the tasks that require knowledge (the denominators of Table 3), we split each model's knowledge failures into three stages: the family is never invoked (selection), it is invoked but the gold record is not in the top 5 (retrieval), or relevant evidence is returned but not attributed (integration). Shares of knowledge failures (%):

Modelselectionretrievalintegration
doubao-seed-2.0-pro96.81.51.8
Seed-1.8-Pro94.42.53.1
Deepseek-v4-Pro95.81.92.3
GPT-5.599.80.10.1
Lingshu-32B76.710.512.8
MedGPT-oss-20B83.97.28.8
All twelve models96.01.82.2

Selection dominates for every general-purpose model; retrieval and integration become visible mainly for the specialized models, which invoke knowledge more often. This also explains the oracle gain the reviewer points to. The oracle return is credited as an invocation, so it removes the selection step as well as retrieval, and coverage rises from 23.2 to 39.2 under the corrected pipeline. We treat the 60.8% of trajectories that still fall short of m=1m=1 when gold knowledge is handed to them operationally as integration failures under the oracle setting; this label describes where the trajectory fails, not why. We therefore define tool tunnel vision as failing to select a required family, together with failing to use evidence that was returned without being requested, and treat retrieval quality (Table 9) as a secondary bottleneck.

Q4. How can the tiers be an increasing difficulty scale when rr rises from 38.8 to 93.2? We agree that "increasing difficulty" was the wrong term. The tiers are ordered by evidence depth: required families rise from one to at least two to three, checkpoints per task from 2.00 to 3.59 to 3.87, and reference tool calls from 1.00 to 2.59 to 2.87 (Sec. 4.3). Level 1 asks for precise localization or existence (e.g., "left upper lung", Fig. 1) and is graded strictly, whereas Levels 2 and 3 are graded by medical meaning, so rr is not comparable across tiers, as Finding 1 notes. Table 2 already shows that dependence on tools grows with tier: the no-tool baseline falls from 9.8 to 5.1 to 2.0, the clinician reference falls from 92.4 to 90.1 to 87.3, and clinician tool calls rise from 3.46 to 5.13 to 6.82. A strict rescoring that requires every gold finding gives doubao-seed-2.0-pro 61.5 at Level 2 and 58.7 at Level 3. We will use the term "evidence depth" in Sec. 2.1 and 4.4.

Q5. Inclusion criteria, failure handling, and release. The main leaderboard includes models that completed the full 2,000-task suite with an unresolved-failure rate below 5%. InternVL2.5-4B (8.6%) and Hulu-Med-7B (61.3%, dominated by execution failures, App. I) exceed this threshold, so they appear only in the supplementary comparison. Transient provider, rate-limit and image-encoding errors are rerun under the same configuration (Table 10), and unresolved failures are scored r=0r=0 and reported by error type. We will release all 2,000 tasks as unified Case JSON with provenance tags, together with RR, F∗F^{\ast} (including the answer-level relabelling) and the checkpoints, the nine tool implementations, the corrected evaluator with unit tests, all agent and judge prompts, the KG snapshot, and the PubMed snapshot (PMIDs and metadata). The guideline snapshot will be released in full where licences permit, and otherwise as record identifiers with retrieval scripts. Imaging is distributed through download and preprocessing scripts, because SLAKE, CT-RATE and AbdomenAtlas 3.0 remain under their original licences.

We thank the reviewer again. The questions on construct validity and on the stage-wise breakdown made the paper's claims more precise.

Reviewer_aAvkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MedXS, a benchmark to test medical multimodal agents across radiology images, EHR text, and medical knowledge bases. It features 2k tasks and ~6k checkpoints, evaluating final answer correctness, tool use, workflow steps, and evidence coverage across multiple sources. Results indicate that even when models get answers right, they often fail to truly leverage the relevant evidence across all three sources.

Critical strengths

Clear clinical motivation. Looking past raw task accuracy to check if an agent actually roots decisions in the right evidence is the right move for medical AI.

Good test scope. Evaluates a decent variety of proprietary and open models across complex, multi-turn clinical setups.

Clinician spot-checking adds credibility to the task formulations.

Critical weaknesses

he core multi-source coverage metric feels superficial. As defined, it just checks if the agent called the tools and regurgitated their keywords in the final response. It does not measure genuine multi-modal reasoning. A model can easily hallucinate a conclusion first, call tools as an afterthought, and echo the results. Without source ablation or counterfactual interventions, claiming this tests actual cross-source dependency is a stretch.

Shallow analysis on medical vs general model performance. Specialized medical models tank compared to general frontier LLMs. The authors do not dig into why this happens. Is it because of weak base tool-calling ability, poor context length handling, or bad prompt alignment? Right now it is just an unexplained drop in the tables.

Sloppy citation and reproducibility issues. The related work relies heavily on a generic feature-checklist table instead of deep conceptual comparisons. Worse, several tested models (HuatuoGPT-Vision-34B, MedGPT-oss-20B, Hulu-Med-7B, InternVL2.5-4B) are completely missing citations or exact release checkpoints. For a benchmark paper aiming to guide future clinical work, missing specs and references hurts reproducibility.

Inconsistent numbers in the tables. Table 3 lists a 0.00% knowledge-tool call rate for Qwen3.6-Plus, yet Table 2 gives it 0.33% multi-source coverage. By Equation 6, multi-source coverage requires invoking all tool families. This math does not add up.

Questions for authors

Have you run counterfactual tests (e.g. corrupting or removing one evidence source to see if the decision flips) to prove the agent genuinely relies on cross-source synthesis rather than post-hoc quoting?

How exactly did you implement tool interfaces for the open medical models, and what accounts for their large drop compared to general models?

How can Qwen3.6-Plus have 0.33% multi-source coverage in Table 2 when its knowledge-tool invocation is listed as 0.00% in Table 3?

Please add explicit citations, weights versions, and exact prompts used for all evaluated open-source models.

5 comments
AuthorsOpenReview ↗

Response to Reviewer aAvk (1/2)

We thank the reviewer for the detailed and expert feedback. The Qwen3.6-Plus inconsistency, the counterfactual test and the missing citations each led to concrete corrections.

W1/Q1. Is coverage superficial, and does a counterfactual test show reliance? We agree that mm measures attribution rather than causal dependence, and we will describe it as attributed cross-source integration. mm is a necessary condition. By Prop. 1 (App. D), m=0m=0 guarantees that some required family was not attributed, whatever the answer label, whereas m=1m=1 does not prove that the decision relied on all three families.

Following the reviewer's suggestion, we ran a counterfactual intervention on 300 Level 3 tasks (100 trajectories each from doubao-seed-2.0-pro, GPT-5.5 and Claude-4.8-Opus, the three models with the highest tier-averaged result rr in Table 2). We replace only one family at a time with a clinically conflicting variant and keep all other evidence fixed. For records, we flip the laterality or alter the decisive measurement. For knowledge, we substitute a guideline record for a competing diagnosis. For imaging, we return the region analysis of a matched case with a different finding. Decision-flip rates (%):

Perturbed familyattributednot attributed
Imaging71.418.2
Record64.812.5
Knowledge43.69.7

Attributed families flip the decision 3.5–5.2× more often than unattributed ones, and the gap holds within each system (all p<0.001p<0.001, two-proportion zz-test). In 21.3% of m=1m=1 trajectories, no single-family perturbation flips the decision. This suggests that some decisions depend jointly on several families, and it limits what single-family interventions alone can tell us about post-hoc quoting. Knowledge flips are the lowest, consistent with part of the guideline content already being known to the model.

On keyword echoing: vv and mm do not rely on keyword overlap. A semantic judge that never sees RR decides whether returned evidence is reflected in the answer (Fig. 9), and physicians agree with it at κ≥0.81\kappa \ge 0.81 (Table 5). On conclusions formed before tool use: the gating and workflow checks of Sec. 3.4 penalise consuming evidence before it is retrieved. A conclusion fixed in hidden reasoning, however, cannot be seen from the trajectory, and this is the case the counterfactual test addresses. We also added a check that the cited evidence is consistent with the Conclusion, so evidence that is quoted but irrelevant no longer earns credit.

W4/Q3. Qwen3.6-Plus: 0.00% knowledge invocation (Table 3) vs. 0.33% coverage (Table 2). We thank the reviewer for catching this. The reviewer is right that these numbers are inconsistent under Eq. 6, and GPT-5.6-Luna shows the same issue (0.00% vs. 0.17%). The evaluator counted attempted knowledge calls, whereas Table 3 counts successful returns. Qwen3.6-Plus made 2 knowledge calls on Level 3 tasks (2/600 = 0.33%) and GPT-5.6-Luna made 1 (1/600 = 0.17%), all of which returned structured errors because of invalid arguments. With invocation defined by successful returns throughout, and with the per-family use flag identified by Reviewer 3DPp fixed, the corrected mm is 0.00 for both models. The evaluator now asserts m=0m=0 whenever a required family has no successful invocation.

AuthorsOpenReview ↗

Response to Reviewer aAvk (2/2)

W2/Q2. Why the specialized models drop, and how their tools were implemented. The existing tables point to multi-turn tool use and record retrieval, not missing medical knowledge. At Level 1, which needs a single imaging call, the specialized models score 32.1 / 30.2 / 28.4, close to the general models (31.3–38.8, apart from Deepseek-v4-Pro at 6.5). At Level 3, by contrast, Lingshu-32B reaches 19.8, against at least 76.3 for most general models (Table 2). Lingshu-32B and MedGPT-oss-20B invoke record tools on only 54.3% and 58.0% of the tasks that require them, against at least 94.8% for most general models (Table 3). HuatuoGPT-Vision-34B makes 28.1 successful calls on average yet has the lowest checkpoint score, and Hulu-Med-7B is dominated by execution failures (App. I). The specialized models in fact invoke knowledge more often (52.9% and 43.0%) and show the inverted m>rm>r pattern (Table 4). A failure taxonomy over unsuccessful trajectories (% of failures) confirms this:

FailureLingshuMedGPT-ossHuatuoGPT-V
Invalid JSON / tool schema31.427.912.6
Repeated call / budget exhausted18.221.554.7
Context overflow9.612.323.1
Premature answer (no record call)40.838.39.6

Lingshu-32B and MedGPT-oss-20B mostly answer prematurely or emit malformed calls, while HuatuoGPT-Vision-34B mostly loops until its context overflows. In both cases the bottleneck is the reliability of the multi-turn tool protocol. All models received the same zero-shot ReAct prompt (Fig. 8), with strict-JSON calls parsed from the generated text rather than through a native function-calling API. The open models were served with vLLM, using each model's official chat template and native context limit, temperature 0, and at most 2,048 new tokens per turn.

W3/Q4. Citations, checkpoints and related work. We apologise for the missing citations; we also mistakenly cited HuatuoGPT-o1 for HuatuoGPT-Vision. The correct references are HuatuoGPT-Vision-34B (Chen et al., 2024, arXiv:2406.19280; FreedomIntelligence/HuatuoGPT-Vision-34B), InternVL2.5-4B (Chen et al., 2024, arXiv:2412.05271; OpenGVLab/InternVL2_5-4B) and Hulu-Med-7B (Jiang et al., 2025, arXiv:2510.08668). Lingshu-32B is already cited (Xu et al., 2025). In App. F we will list the source, checkpoint and revision of every open model, including MedGPT-oss-20B, together with the serving and decoding settings and the model strings and access dates of the API models. The prompt is identical for all models and is given in full in Fig. 8.

On related work, we agree that the checklist in Table 1 needs a conceptual comparison behind it, and App. C will provide one along three axes. The first axis is which evidence families are exposed together. MedCTA and DeepTumorVQA pair imaging with derived measurements and a knowledge query, MedAgentBench and FHIR-AgentBench expose records without images, and MedXS exposes all three, so the agent has to choose among them. The second axis is what the process score checks. MedCTA scores tool choice, parameter validity and trajectory fidelity; DeepTumorVQA scores stage-wise reference trajectories; MedChain scores sequential decisions; MedXS checks whether each required family is attributed in the answer. The third axis is what a failure means: a well-formed trajectory grounded on a single source scores fully under call-level metrics but receives m=0m=0 in MedXS. We will make explicit that MedXS is complementary to these benchmarks rather than uniformly stronger: its distinctive contribution is to evaluate whether all required evidence families are attributed in the final answer.

We thank the reviewer again. The counterfactual test, the failure taxonomy and the consistency fixes have substantially strengthened the paper.

Thanks for the detailed response and new experiments. I am raising my score to 3. for the acceptance level, there are still a few remaining issues.

  1. What is the flip rate when simply re-running a task with no perturbation? Without that noise baseline, it's hard to judge the ~10–18% flip rate for unattributed families. Also, knowledge attributions flip the decision less than half the time (43.6%), and around a fifth of m=1 trajectories aren't flipped by any single perturbation, which looks a lot like post-hoc citation.

  2. Calls returning only errors were still counted as used, which means the use labels need to be re-checked. Since this changes v and m, please post the fixed Tables 2–4 and confirm if Table 5 was updated with the fixed pipeline too.

  3. models like Lingshu-32B and MedGPT-oss-20B get around 30 for m while GPT-5.5 and Claude-4.8-Opus are under 3, even though their r is way lower. how should readers interpret m when it moves in the opposite direction of r?

if GPT-5.5 gets ~96% accuracy at Level 3 while calling knowledge on less than 1% of tasks that supposedly require it, doesn't that mean knowledge is not actually needed for many Level 3 tasks, and low m partly reflects noisy F∗ labels rather than agent failure?

Reviewer_3DPpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedXS is a benchmark for medical agents that use imaging, case records, and external medical knowledge. It contains 2,000 tasks, three difficulty levels, nine tools, and 6,073 reference checkpoints. Six metrics measure answer correctness, checkpoint completion, tool calls, evidence use, workflow order, and source coverage. Experiments show large gaps between correct answers and complete source coverage. The paper uses these gaps to study how models gather and combine evidence.

Critical strengths
  1. The benchmark combines imaging, case records, and medical knowledge in one tool environment. The three task levels and reference checkpoints support a detailed analysis of evidence gathering. Labels for original, derived, and synthetic fields also help readers understand the data sources (Sections 3.1–3.3; Tables 6–8).
  2. The paper includes useful checks of the evaluation. Three physicians review 300 model trajectories, with reported coverage agreement of κ = 0.86 (Table 5). An independent judge and an oracle knowledge experiment further examine judge agreement and the effect of retrieval (Tables 11–12).
  3. Code for the proposed benchmark is provided in the supplement.
Critical weaknesses
  1. According to the provided code, the computation of the coverage does not check whether each source contributed to the answer. In the supplied medal/medal/eval/evidence_use_score.py (lines 86–89), every source shares the condition that at least one evidence item was used. For example, a model can call all three tool families and use only the image result in its answer. The code then marks all three sources as used and gives coverage 1, while Eq. 6 requires 0. This can overestimate coverage. The scoring version used for the reported tables maybe incorrect and need to double-check.
  2. A model's existing medical knowledge may make external retrieval unnecessary, especially for the properitary LLMs that hides its own CoT. For example, a model can already know a guideline and correctly apply it to the retrieved images and case records. Eq. 6 still gives coverage 0 if the knowledge tool was not called. Appendix G evaluates visible evidence statements, so knowledge used in hidden reasoning is unobserved. This limits the claim that low coverage shows poor knowledge integration. The no-tool baseline removes all tools together, leaving the need for knowledge retrieval alone untested.
  3. The clinician reference in Table 2 has no clear collection protocol. The caption only states that it uses the same judge. The paper does not specify how many clinicians completed how many tasks, how their answers and tool actions were collected, or whether they used the same tools and budget as the models. The 300-trajectory study in Appendix D evaluates doctors' agreement with automatic scores on model outputs. It does not explain how the clinician reference was produced. This leaves the model–clinician comparison unclear.
Questions for authors
  1. Which scoring code produced the coverage results in the paper? Please check evidence use separately for each source and report whether the coverage scores, model rankings, and agreement with clinician judgments change.
  2. How does Level 3 answer accuracy change when knowledge retrieval is disabled while image and record tools remain available? What evidence shows that the required knowledge source adds information beyond what the model already knows?
  3. How was the clinician reference in Table 2 collected? Please specify the number of clinicians and tasks, whether they solved the tasks independently, and whether they used the same tools and budget as the models.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3DPp (1/2)

We thank the reviewer for the careful reading, and especially for inspecting the supplementary code.

W1/Q1. Which scoring code produced the coverage results, and does per-source checking change them? The reported mm came from the submitted snapshot, and the reviewer is correct. L86–89 of evidence_use_score.py derived 1[F used]\mathbb{1}[F\ \text{used}] for every family from one shared condition (at least one evidence item used), whereas Eq. 6 requires a per-family condition. In the fixed evaluator, family FF counts as used only if at least one evidence unit returned by a tool of FF is marked used by the evidence judge, and as invoked only if one of its tools returned successfully; a call that returns only an error counts as neither. A further step raised by Reviewer V8dX now also checks that each credited unit is consistent with the Conclusion. Level 3 coverage for all twelve models:

Modelreportedper-family+ consistency
Seed-1.8-Pro41.536.234.5
doubao-seed-2.0-pro28.824.523.2
Deepseek-v4-Pro23.019.318.2
Grok-4.515.712.812.0
Claude-4.8-Opus2.732.172.00
GPT-5.51.831.501.33
Gemini3.1-Pro0.500.330.33
Qwen3.6-Plus0.330.000.00
GPT-5.6-Luna0.170.000.00
MedGPT-oss-20B34.225.823.7
Lingshu-32B29.321.720.3
HuatuoGPT-Vision-34B9.337.176.50

The reported and corrected rankings correlate at Spearman 0.99. Seed-1.8-Pro remains first in the coverage ranking, the order among general-purpose models is unchanged, and the only swap is that doubao-seed-2.0-pro moves above Lingshu-32B. The two specialized models drop the most (−10.5 and −9.0), because they invoke knowledge often (Table 3) but attribute it less often. In App. D, the physicians judged whether every required family contributed, which is the per-family meaning, so we recomputed Table 5 with the corrected pipeline. Agreement on coverage rises from κ\kappa = 0.86 to 0.88 and on evidence use from 0.81 to 0.84, while rr and ff are unaffected.

The corrected conditions are strictly stronger than the submitted one, so coverage can only decrease. The finding that workflow is high while coverage is low therefore cannot be an artefact of this bug. The released code includes a unit test for the reviewer's counterexample, in which all three families are invoked but only imaging is used, so mm must be 0.

AuthorsOpenReview ↗

Response to Reviewer 3DPp (2/2)

W2/Q2. Parametric knowledge, and disabling knowledge retrieval alone. We agree, and this changes our wording. A model may apply a memorised guideline, in which case Eq. 6 gives m=0m=0, so a low mm means the knowledge was not attributed, not that it is missing. MedXS asks agents to ground their claims in a versioned store of 3,198 ESMO/NCCN/CSCO records covering 82 diseases (Sec. 3.2), because knowledge used in hidden reasoning cannot be checked against a specific guideline version.

To test whether the knowledge family adds information, two clinicians relabelled 300 Level 3 tasks according to whether knowledge is needed for the answer itself or only to justify it (κ\kappa = 0.81 between them). Knowledge is needed for the answer in 38.7% of these tasks. As the reviewer notes, the no-tool baseline removes all tools at once, so we disabled only the knowledge family and kept the imaging and record tools available:

ModelL3 rr all / no FknowF_\text{know}Δ, knowledge needed for answerΔ, justification only
doubao-seed-2.0-pro93.2 / 88.7−10.4−0.8
Seed-1.8-Pro90.8 / 85.9−11.0−1.0
GPT-5.595.8 / 94.9−1.9−0.3

Here doubao-seed-2.0-pro is the model with the highest result rr averaged over the three tiers in Table 2 (68.2%). Level 3 checkpoint completion drops from 62.7 to 55.1 and from 61.1 to 54.0 for the first two systems. The drop is concentrated on tasks where knowledge determines the answer, which shows that the knowledge source adds information beyond what these models already know. GPT-5.5, which rarely retrieves, barely changes; this is consistent with the reviewer's point that strong proprietary models can hold much of this knowledge internally, and it is why rr alone cannot certify knowledge use. The oracle result in Table 12 agrees: gold knowledge raises the Level 3 checkpoint score from 62.7 to 68.4. Our primary benchmark definition remains the justification-level F∗F^{\ast}, because MedXS evaluates attributed grounding against a versioned knowledge store; we will report the answer-level F∗F^{\ast} as a companion analysis that separates justification requirements from answer requirements. For doubao-seed-2.0-pro, coverage is 23.2 under the primary definition and 70.3 under the answer-level one.

W3/Q3. How the clinician reference was collected. We apologise for omitting this protocol. Four board-certified clinicians (two radiologists and two internists, with 6–15 years of experience) produced it; none of them took part in building the reference graphs. They solved a stratified subset of 300 tasks (100 per tier, 75 per clinician), each working independently, blind to the gold answers and to model outputs, with 15 minutes per task. They used the same nine-tool interface, per-task tool subset, budget B=50B=50 and Conclusion/Evidence format as the models. Their answers were scored by the same judge for rr and vv, and by the same rules for cc, ff and mm. The clinician tool-call counts in Table 2 (nn = 3.46 / 5.13 / 6.82) come from this interface. The 95% bootstrap interval for the average clinician rr is [86.4, 93.1]. This protocol is separate from the 300-trajectory study in App. D, which measures agreement between physicians and the automatic scores on model outputs.

We hope these corrections and analyses address the reviewer's concerns, and we would be glad to clarify anything further.

I sincerely apprecaite the detailed reply from the authors and also thanks the great effort made during the rebuttal phase. It is great to see that the fixed coding issue didn't change the conclusion, and the new human study confirms our assumption.

As for the internal knowledge issue for the private LLMs, one possible solution maybe provide the model with a explicit tool to record the medical knowledge it used. Namely, explicitly request the private LLM to dump the medical knowledge that used during the internal reasoning. Even though this still may not cover all the medical knowledge, but it may some what alleviate the issue.

Overall, I am glad to see the new results and discussion. My concerns are largely addressed and I would like to raise my score from 2 to 3, weak acceptance.

We thank the reviewer for raising the score and for this constructive suggestion. A knowledge-recording tool is a practical way to make part of the hidden knowledge observable. We will add an optional record_knowledge tool to the released harness, through which an agent can declare the guideline or fact it relies on, and we will report how often each model uses it as a companion "declared knowledge" rate alongside mm. Because declared knowledge is not drawn from the versioned store, it will not count toward mm itself, but it will let readers distinguish models that use no knowledge from models that use knowledge without retrieving it. We will also discuss this direction in the limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Summary of Discussion

Dear PCs, SACs, ACs, and Reviewers,

We sincerely thank you for your time and constructive feedback. We are encouraged that all reviewers consistently recognise the importance of the problem and the value of the benchmark:

  • Reviewer aAvk highlights the "clear clinical motivation", noting that "looking past raw task accuracy to check if an agent actually roots decisions in the right evidence is the right move for medical AI".
  • Reviewer V8dX finds that MedXS "addresses a critical evaluation gap" and provides "an actionable framework for developing auditable medical agents".
  • Reviewer 3DPp notes that the benchmark "combines imaging, case records, and medical knowledge in one tool environment" and that "the paper includes useful checks of the evaluation".

Summary of Rebuttal & Clarifications

During the rebuttal, we addressed the reviewers' concerns and added new analyses. We summarise the main points of the discussion below.

Reviewer 3DPp found that the submitted evaluator derived per-family evidence use from a shared flag. We fixed the evaluator and recomputed all twelve models. The coverage ranking is essentially unchanged (Spearman 0.99), physician agreement on coverage rises from κ\kappa = 0.86 to 0.88, and the main finding holds. On parametric knowledge, we clarified that low coverage means the knowledge was not attributed, not that it is missing. A knowledge-only ablation and a clinician relabelling show that retrieved knowledge adds information where it determines the answer. We also documented how the clinician reference was collected. The reviewer confirmed that the fix "didn't change the conclusion" and that the concerns are largely addressed. The reviewer also suggested a tool through which agents can declare the knowledge they use, which we will add to the released harness.

Reviewer aAvk asked whether coverage reflects genuine reliance or post-hoc citation. Counterfactual interventions show that attributed evidence families change the decision far more often than unattributed ones, and far more often than run-to-run noise. Joint perturbations isolate the small share of cases consistent with post-hoc citation, which we will report alongside coverage. We resolved the Qwen3.6-Plus inconsistency by unifying the invocation criterion and posted the corrected tables. Reading correctness and coverage jointly shows that general-purpose models answer correctly but cite incompletely, whereas specialized models gather the evidence but fail to synthesise it. A failure taxonomy traces the drop of the specialized models to multi-turn tool use, and we corrected the citations and checkpoints of all evaluated models.

Reviewer V8dX asked whether the evaluator verifies that cited evidence supports the conclusion. The judge now also reads the Conclusion and credits only consistent evidence. Most decision changes in the counterfactual test follow the direction of the perturbed evidence, and blinded physicians rated the perturbed variants nearly as plausible as the originals. A stage-wise breakdown shows that failing to select the knowledge source outweighs retrieval and integration failures, and we decomposed the oracle residual by evidence family. We also reported annotation reliability across all 2,000 tasks and clarified that the tiers measure evidence depth. The reviewer confirmed that all remaining concerns have been addressed.

These discussions led us to fix a real evaluator bug and to state more precisely what MedXS certifies. Near-perfect workflow validity does not mean that an agent integrates all required evidence sources, and neither outcome-only nor call-level scoring can reveal this gap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.