- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MedXS is a process-verified benchmark designed to evaluate medical multimodal agents on evidence acquisition and utilization across medical imaging, structured records, and external knowledge. Comprising 2,000 stratified tasks, nine specialized tools, and over 6,000 reference evidence checkpoints, the benchmark employs six complementary metrics to assess aspects such as answer correctness, workflow validity, and multi-source coverage. Evaluations of multiple general-purpose and specialized agents reveal significant gaps between high workflow validity and low multi-source coverage, indicating limitations in complex cross-source reasoning. Additionally, the study incorporates comprehensive validation measures, including physician reviews, independent-judge checks, and control analyses, to assess the framework's reliability.
Critical strengths
- Addresses a Critical Evaluation Gap: Identifies key limitations in existing benchmarks by demonstrating that accurate final answers and well-formed tool trajectories do not guarantee genuine evidence utilization.
- Establishes a Unified Multi-Source Benchmark: Connects medical imaging, structured records, and external knowledge via a unified interface, featuring 2,000 stratified tasks and 6,073 reference checkpoints to systematically measure evidence requirements.
- Introduces a Multi-Dimensional Metric Suite: Proposes six complementary metrics covering answer accuracy, procedural compliance, and evidence acquisition, providing an actionable framework for developing auditable medical agents.
- Exposes Hidden Behavioral Discrepancies: Highlights the flaws of outcome-only evaluation by demonstrating a significant gap between high workflow validity (97.4%) and low multi-source coverage (28.8%).
- Ensures Rigorous Expert and Independent Validation: Incorporates a clinician review of 300 trajectories and an independent-judge assessment across 400 tasks to confirm evaluation reliability and domain relevance.
- Conducts Comprehensive Diagnostic Controls: Includes repeat-run stability analyses and oracle-retrieval benchmarks to rule out alternative explanations and solidify the study's empirical findings.
Critical weaknesses
- Construct Validity Issues in Multi-Source Integration: The multi-source coverage metric () merely checks if all three source families were called and recorded, failing to prove that evidence from each source actually supported, constrained, or influenced the agent's final conclusion.
- Discrepancy in Evaluator Implementation: A contradiction exists between the formal definition (which requires evidence to be reflected in the final conclusion) and the executable evaluator (which evaluates tool results and evidence text without explicitly receiving the conclusion), making it unclear how substantive evidence usage is verified.
- Limitation of High Clinician Agreement: While high clinician agreement confirms that automatic scoring follows the operational protocol, it only validates label consistency rather than proving that genuine semantic cross-source reasoning occurred.
- Deviations from Real Clinical Realism: Relying on synthetic history and laboratory proxies to compensate for missing longitudinal data creates artificial task structures that differ significantly from messy, real-world clinical workflows.
- Unclear Annotation Reliability Across the Benchmark: Although clinician audits validated 300 generated trajectories, there is insufficient evidence regarding the inter-annotator consistency and reliability of the reference evidence graphs (), required source families (), and checkpoints across all 2,000 tasks.
- Confounding Retrieval Difficulties with Reasoning Failures: The oracle-retrieval experiment significantly boosted Level 3 coverage (from 28.8% to 46.1%), indicating that low performance stems partly from retrieval bottlenecks rather than purely from reasoning defects like "tool tunnel vision."
Questions for authors
- How does the evaluator verify that cited evidence substantively supports the final conclusion rather than simply being listed in the Evidence field, especially given that the judge function omits the Conclusion field?
- What specific annotation protocols—such as the number of independent clinicians and inter-annotator agreement metrics—were used to establish the baseline reference graphs, source requirements (), and checkpoints across all 2,000 tasks?
- Could you provide a breakdown that separates source-selection, retrieval, and integration failures to clarify whether the low multi-source coverage is primarily caused by retrieval bottlenecks or model reasoning limitations?
- How should the tier structure be interpreted as an increasing difficulty scale when the top agent's accuracy rises from 38.8% at Level 1 to 93.2% at Level 3 due to looser answer-matching criteria?
- What were the inclusion criteria and execution failure-handling protocols for the main leaderboard versus supplementary models, and which specific benchmark components (tasks, tool code, graphs, and knowledge snapshots) will be publicly released?
8 comments
Response to Reviewer V8dX (2/3)
W4. Synthetic records vs. clinical realism. We agree, and App. K lists this as our first limitation. The history and laboratory proxies are provenance-tagged and reviewed by clinicians. In the 240-row audit, 91.7–96.7% of field groups were accepted with no edit or a minor edit (Table 8). Splitting the Level 3 results of doubao-seed-2.0-pro by whether depends on any synthetic field gives:
| Subset | #tasks | (corrected) | |
|---|---|---|---|
| Source-native / derived only | 214 | 92.1 | 24.8 |
| Includes synthetic fields | 386 | 93.8 | 22.3 |
The source-native subset covers 214 of the 600 Level 3 tasks (35.7%). The gap between workflow and coverage persists on it, with slightly higher coverage, so the proxies do not create the gap. We do not claim that the proxies match the realism of de-identified longitudinal records. The audit checks individual fields, not the temporal consistency of a patient's history, and the split shows only that our main finding does not depend on the proxies. We will state this limitation in the main text and avoid wording that suggests deployment-level clinical realism.
W5 / Q2. Annotation reliability across all 2,000 tasks. All 2,000 tasks follow the two-stage Model+Clinician procedure of Sec. 3.1 under one locked protocol: a model drafts candidate questions and evidence chains, and one of six clinicians (four radiologists and two oncologists) verifies them and locks . To estimate reliability for the full benchmark, a second clinician independently re-annotated 400 tasks (160 / 120 / 120 per tier), stratified by tier, source dataset and required family set so that the subset matches the composition of the 2,000 tasks. Agreement before adjudication:
| Tier | on | node F1 on | checkpoint agreement (%) |
|---|---|---|---|
| Level 1 | fixed (imaging) | 0.92 | 94.2 |
| Level 2 | 0.86 | 0.86 | 90.8 |
| Level 3 | 0.82 | 0.82 | 88.6 |
| All | 0.84 (L2–L3) | 0.87 | 91.5 |
At Level 1 the required family set is imaging by construction, so on is computed over Levels 2–3. Agreement decreases with evidence depth, as expected, but stays in the almost-perfect band for and above 0.8 in node F1 at every tier. Most Level 3 disagreements concern whether the knowledge family is required, which motivates the answer-level relabelling of we report to Reviewer 3DPp. Disagreements were adjudicated by a senior radiologist. This annotation team is separate from the four clinicians who produced the clinician reference row in Table 2.
Response to Reviewer V8dX (3/3)
W6 / Q3. Retrieval bottleneck vs. reasoning failure. We agree that the oracle result shows retrieval matters, and we no longer call low purely a reasoning defect. Over the tasks that require knowledge (the denominators of Table 3), we split each model's knowledge failures into three stages: the family is never invoked (selection), it is invoked but the gold record is not in the top 5 (retrieval), or relevant evidence is returned but not attributed (integration). Shares of knowledge failures (%):
| Model | selection | retrieval | integration |
|---|---|---|---|
| doubao-seed-2.0-pro | 96.8 | 1.5 | 1.8 |
| Seed-1.8-Pro | 94.4 | 2.5 | 3.1 |
| Deepseek-v4-Pro | 95.8 | 1.9 | 2.3 |
| GPT-5.5 | 99.8 | 0.1 | 0.1 |
| Lingshu-32B | 76.7 | 10.5 | 12.8 |
| MedGPT-oss-20B | 83.9 | 7.2 | 8.8 |
| All twelve models | 96.0 | 1.8 | 2.2 |
Selection dominates for every general-purpose model; retrieval and integration become visible mainly for the specialized models, which invoke knowledge more often. This also explains the oracle gain the reviewer points to. The oracle return is credited as an invocation, so it removes the selection step as well as retrieval, and coverage rises from 23.2 to 39.2 under the corrected pipeline. We treat the 60.8% of trajectories that still fall short of when gold knowledge is handed to them operationally as integration failures under the oracle setting; this label describes where the trajectory fails, not why. We therefore define tool tunnel vision as failing to select a required family, together with failing to use evidence that was returned without being requested, and treat retrieval quality (Table 9) as a secondary bottleneck.
Q4. How can the tiers be an increasing difficulty scale when rises from 38.8 to 93.2? We agree that "increasing difficulty" was the wrong term. The tiers are ordered by evidence depth: required families rise from one to at least two to three, checkpoints per task from 2.00 to 3.59 to 3.87, and reference tool calls from 1.00 to 2.59 to 2.87 (Sec. 4.3). Level 1 asks for precise localization or existence (e.g., "left upper lung", Fig. 1) and is graded strictly, whereas Levels 2 and 3 are graded by medical meaning, so is not comparable across tiers, as Finding 1 notes. Table 2 already shows that dependence on tools grows with tier: the no-tool baseline falls from 9.8 to 5.1 to 2.0, the clinician reference falls from 92.4 to 90.1 to 87.3, and clinician tool calls rise from 3.46 to 5.13 to 6.82. A strict rescoring that requires every gold finding gives doubao-seed-2.0-pro 61.5 at Level 2 and 58.7 at Level 3. We will use the term "evidence depth" in Sec. 2.1 and 4.4.
Q5. Inclusion criteria, failure handling, and release. The main leaderboard includes models that completed the full 2,000-task suite with an unresolved-failure rate below 5%. InternVL2.5-4B (8.6%) and Hulu-Med-7B (61.3%, dominated by execution failures, App. I) exceed this threshold, so they appear only in the supplementary comparison. Transient provider, rate-limit and image-encoding errors are rerun under the same configuration (Table 10), and unresolved failures are scored and reported by error type. We will release all 2,000 tasks as unified Case JSON with provenance tags, together with , (including the answer-level relabelling) and the checkpoints, the nine tool implementations, the corrected evaluator with unit tests, all agent and judge prompts, the KG snapshot, and the PubMed snapshot (PMIDs and metadata). The guideline snapshot will be released in full where licences permit, and otherwise as record identifiers with retrieval scripts. Imaging is distributed through download and preprocessing scripts, because SLAKE, CT-RATE and AbdomenAtlas 3.0 remain under their original licences.
We thank the reviewer again. The questions on construct validity and on the stage-wise breakdown made the paper's claims more precise.
Response to Reviewer V8dX (1/3)
We thank the reviewer for the thorough review and for recognising both the evaluation gap and the validation suite.
W1 / W3. Construct validity of m, and what clinician agreement shows. We agree that m shows each required family was attributed, not that it constrained the conclusion, and that clinician agreement confirms protocol fidelity rather than semantic reasoning. We therefore describe m as attributed cross-source integration and add a counterfactual intervention. On 300 Level 3 tasks (100 each from doubao-seed-2.0-pro, GPT-5.5 and Claude-4.8-Opus), we replace only one family at a time with a clinically conflicting variant and keep all other evidence fixed, so the flip rate isolates reliance on that family. Records receive flipped laterality or an altered decisive measurement, knowledge a guideline record for a competing diagnosis, and imaging the region analysis of a matched case with a different finding. To separate reliance from run-to-run variation, we also re-ran the same tasks without any perturbation. Decision-flip rates (%):
The gap between attributed and unattributed families holds within each system (all p<0.001, two-proportion z-test), and both are well above the no-perturbation rate. Knowledge flips are the lowest because many knowledge attributions only justify the answer: on tasks where clinicians marked knowledge as needed for the answer, the attributed knowledge flip rate is 68.7%, against 27.7% on justification-only tasks. This test does not turn m into a direct measure of semantic reasoning. It is an orthogonal check that attributed evidence causally affects the decision, while m itself certifies attribution.
W2 / Q1. The evidence judge omits the Conclusion field. The reviewer correctly identifies a mismatch between Eqs. 4/6 ("reflected in the conclusion") and the judge, which receives the Evidence field and the tool results (Fig. 9). We split the answer into these two fields so that a fluent Conclusion cannot hide a missing source (App. F, "From execution to audit"), but this lets an agent list evidence that does not support its Conclusion. The judge now also receives the Conclusion and labels each used unit as supporting, irrelevant or contradicting; only supporting units count toward v and 1[F used]. We also fixed a per-family use flag in the evaluator, identified by Reviewer 3DPp. Both fixes are stricter and can only lower m. On doubao-seed-2.0-pro, the model with the highest result r averaged over the three tiers in Table 2 (68.2%), the per-family fix moves m from 28.8 to 24.5, and the consistency step, which relabels 6.8% of credited units, then moves it to 23.2. With both fixes applied to all twelve models, Seed-1.8-Pro remains first in the coverage ranking, the order among general-purpose models is preserved, and the main finding holds: doubao-seed-2.0-pro keeps 97.4% workflow validity but only 23.2% Level 3 coverage. Eqs. 4 and 6 will read "attributed in the Evidence field and consistent with the Conclusion".