acceptodds
Desk rejected at ICLR 2027

MedStat-Bench: Evaluating Diverse Multimodal Medical Intelligence from Perception to Agentic Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) are often evaluated on a single medical modality or task, leaving their behavior across clinical data types and workflows poorly characterized. We present MedStat-Bench, a broad stress test with 430 curated samples from 6 public datasets and 5 institutional cohorts covering about 10 clinical conditions and care settings. It spans clinical text, 2D images, volume-derived views from 3D studies, and sampled video, with tasks including single- and multi-label classification, structured clinical text generation, regression, and 2D/3D spatial localization. We release a self-contained public track with fixed evaluation materials; institutional cohorts test the same framework on locally collected clinical data. An initial agentic track contains 29 scenarios across 11 patient cases requiring evidence retrieval, inspection of text and medical media, tool use, and structured decisions. Results show that capability is compositional rather than uniform: models struggle most with interpreting volume-derived views and producing precise spatial coordinates, video results differ between ultrasound and surgical settings, and agentic success depends on both the clinical answer and workflow execution. MedStat-Bench therefore characterizes capability across medical data types, output formats, and tool-mediated workflows rather than relying on a single task or aggregate score.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Model, Benchmark, Evaluation, Medical AI, Agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_t9jzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MedStat-Bench, a medical multimodal benchmark with 430 static samples and 29 agentic scenarios drawn from 11 patient cases. It evaluates models across clinical text, images, volume-derived views, and sampled video, covering classification, generation, measurement, localization, and tool-mediated decisions. The authors report performance differences across tasks and modalities.

Critical strengths
  1. Broad evaluation coverage. The benchmark spans clinical text, 2D images, volume-derived views, and sampled video, with tasks covering classification, generation, measurement, and localization. Its agentic track additionally examines tool-mediated clinical workflows, enabling evaluation of capabilities beyond those captured by a single-modality or single-task benchmark.

  2. Support for reproducibility. The manuscript documents dataset composition, input preprocessing, prompting conventions, scoring rules, and agentic evaluation rubrics. It also reports repeated static evaluations with confidence intervals. These details provide a useful basis for inspecting the evaluation protocol and support reproduction efforts, although the availability and completeness of the released materials remain to be verified.

Critical weaknesses
  1. Evaluation reliability. Agentic reasoning and safety scores lack validation against human assessments, and GPT-5.6 Sol judges its own outputs. Acknowledging these limitations does not resolve them. Unlike the static evaluation, which reports four runs and confidence intervals, the agentic evaluation leaves execution repetitions unclear and provides no confidence intervals. Additionally, the agentic track reuses only 11 patient cases across 29 scenarios, limiting case diversity and the generalizability of its findings.

  2. Limited explanatory depth. The findings mainly describe score and ranking differences without sufficiently separating model capabilities from preprocessing, format compliance, and task composition. Medical-specialized baselines are limited to MedGemma 1.5, and the Gemma–MedGemma results receive little analysis accounting for underlying model differences.

Questions for authors
  1. Can the authors provide clinician–judge agreement validation, clarify agentic execution repetitions, and report confidence intervals and judge-agreement statistics?

  2. Can additional analyses distinguish model limitations from preprocessing and format-compliance failures, and contextualize Gemma versus MedGemma performance?

  3. Do the following differences reflect distinct metrics, subsets, or experimental versions?

    • Table 11 versus Table 12, 3D localization: GPT-5.5 0.70% vs 8.5%; Qwen3.6-27B 0.14% vs 1.0%.
    • Figure 3 versus Table 8, static overall score: GPT-5.5 35.4% vs 33.68%; MedGemma 25.8% vs 23.65%.
    • Gemini volume-derived score: 35.8% in Table 4, versus approximately 34.71% when weighting Table 13 by the stated sample counts.
  4. Can the authors verify the bibliography? EndoBench, SurgBench, 3D-RAD, MedFrameQA, and TemMed-Bench have incorrect first-author names; several titles differ from the cited records, and some entries use author placeholders.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 Evaluation reliability and Question 1

  1. Agentic reasoning and safety scores lack validation against human assessments, and GPT-5.6 Sol judges its own outputs. Acknowledging these limitations does not resolve them.

Thank you for this comment. We were not able to obtain independent clinician reviews due to the short review time, but if accepted, we will add clinician reviews as requested. As an automated robustness check, we used high-performance Claude Opus 5.5, from a different model family, to judge frozen outputs and rubrics without access to GPT-5.6 Sol's grades. We ran 11 scenarios (Q1–Q11) on 5 patients. The table below also shows the per-agent comparison for the exact model order and groups in manuscript Table 5. Each score cell is “GPT-5.6 Sol / Opus 5.5”, in percent. A dash indicates no paired result, not zero performance. For direct metric comparability, we report the content metrics defined in manuscript Table 5 and Appendix F: final-answer correctness C, reasoning R, safety S, combined score U = (C + R)/2, and overall score T = U × S/100 when scores are expressed as percentages. Here, S is the mean fraction of applicable critical checks satisfied, rather than the percentage of outputs passing every critical check. C measures agreement with the grading reference; R and S measure automated rubric compliance. We also add per-run Safe and Correct Completion Rate (SCCR): successful workflow completion, full clinical-reference correctness, and all applicable critical checks passing.

\hline \textbf{Agent model} & \textbf{C} & \textbf{R} & \textbf{S} & \textbf{U} & \textbf{T} & \textbf{SCCR} \\\\ \hline \textbf{Open-weight} & & & & & & \\\\ \hline \textbf{Gemma-4-31B-it} & 79.1 / 81.7 & 61.6 / 45.3 & 90.0 / 90.4 & 70.4 / 63.5 & 66.0 / 60.2 & 58.8 / 68.6 \\\\ \hline \textbf{Gemma-4-E2B-it} & 8.9 / 7.0 & 2.6 / 1.1 & 83.7 / 82.4 & 5.8 / 4.0 & 5.5 / 3.8 & 5.8 / 3.8 \\\\ \hline \textbf{Gemma-4-E4B-it} & 32.3 / 29.1 & 24.1 / 8.5 & 79.9 / 74.3 & 28.2 / 18.8 & 23.3 / 15.9 & 21.6 / 17.6 \\\\ \hline \textbf{Qwen3.6-27B} & 79.8 / 78.8 & 56.8 / 39.9 & 84.2 / 85.9 & 68.3 / 59.3 & 59.5 / 54.8 & 47.9 / 60.4 \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 81.2 / 84.7 & 61.1 / 43.0 & 83.3 / 87.7 & 71.1 / 63.8 & 63.2 / 58.7 & 56.8 / 61.4 \\\\ \hline \textbf{Qwen3.8-27B} & 83.9 / 85.5 & 78.1 / 66.0 & 83.8 / 87.4 & 81.0 / 75.8 & 70.8 / 69.6 & 58.8 / 64.7 \\\\ \hline \textbf{Proprietary} & & & & & & \\\\ \hline \textbf{Gemini 3.1 Pro} & 83.6 / 84.4 & 72.8 / 63.7 & 90.8 / 91.5 & 78.2 / 74.0 & 73.1 / 69.5 & 64.0 / 62.0 \\\\ \hline \textbf{GPT-5.5} & 90.8 / 90.9 & 89.5 / 87.3 & 95.9 / 97.2 & 90.2 / 89.1 & 87.8 / 87.6 & 86.3 / 80.4 \\\\ \hline \textbf{GPT-5.6 Sol} & 92.0 / 91.4 & 91.6 / 88.0 & 96.3 / 95.0 & 91.8 / 89.7 & 89.6 / 86.6 & 84.0 / 78.0 \\\\ \hline \end{array}$$ The following table further shows the associations among the available paired agent-model summaries in the table above. Pearson r measures linear association; Spearman ρ measures ranking association. The 95% patient-level bootstrap intervals are also given. These are model-level associations, not agreement rates for individual outputs. $$\begin{array}{l|c|c} \hline \textbf{Table 5 metric} & \textbf{Pearson } r\\ \textbf{(95\\% CI)} & \textbf{Spearman } \rho\\ \textbf{(95\\% CI)} \\\\ \hline \textbf{C} & 0.998\\ (0.989\text{–}0.999) & 0.988\\ (0.842\text{–}1.000) \\\\ \hline \textbf{R} & 0.977\\ (0.960\text{–}0.982) & 1.000\\ (0.939\text{–}1.000) \\\\ \hline \textbf{S} & 0.820\\ (0.615\text{–}0.895) & 0.782\\ (0.576\text{–}0.927) \\\\ \hline \textbf{U} & 0.994\\ (0.987\text{–}0.995) & 1.000\\ (0.952\text{–}1.000) \\\\ \hline \textbf{T} & 0.993\\ (0.981\text{–}0.995) & 0.988\\ (0.952\text{–}1.000) \\\\ \hline \end{array}$$
  1. Unlike the static evaluation, which reports four runs and confidence intervals, the agentic evaluation leaves execution repetitions unclear and provides no confidence intervals. Additionally, the agentic track reuses only 11 patient cases across 29 scenarios, limiting case diversity and the generalizability of its findings.

Thank you for pointing this out. We reported the patient-level uncertainty in the following table.

\hline \textbf{Model} & \begin{array}{c} \textbf{Patient-level reward} \\\\ \textbf{(95\\% CI)} \end{array} & \textbf{Scenario mean} & \textbf{LOPO range} \\\\ \hline \textbf{Open-weight} & & & \\\\ \hline \textbf{Gemma-4-31B-it} & 48.14 & 45.52 & 42.95\text{–}52.95 \\\\ & (27.89\text{–}67.48) & & \\\\ \hline \textbf{Gemma-4-E2B-it} & 5.56 & 6.29 & 4.75\text{–}6.12 \\\\ & (2.25\text{–}9.09) & & \\\\ \hline \textbf{Gemma-4-E4B-it} & 27.29 & 13.10 & 20.02\text{–}30.02 \\\\ & (9.00\text{–}50.45) & & \\\\ \hline \textbf{Qwen3.6-27B} & 5.01 & 4.94 & 3.24\text{–}5.51 \\\\ & (1.65\text{–}9.20) & & \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 39.51 & 29.20 & 33.46\text{–}43.46 \\\\ & (21.68\text{–}57.44) & & \\\\ \hline \textbf{Qwen3.8-27B} & 28.89 & 29.77 & 24.51\text{–}31.78 \\\\ & (14.30\text{–}43.80) & & \\\\ \hline \textbf{Proprietary} & & & \\\\ \hline \textbf{Gemini 3.1 Pro} & 61.94 & 61.24 & 58.13\text{–}68.13 \\\\ & (46.18\text{–}75.56) & & \\\\ \hline \textbf{GPT-5.5} & 67.16 & 59.67 & 63.06\text{–}73.83 \\\\ & (51.07\text{–}80.27) & & \\\\ \hline \textbf{GPT-5.6 Sol} & 53.73 & 53.88 & 49.10\text{–}59.10 \\\\ & (33.19\text{–}72.31) & & \\\\ \hline \end{array}$$ We first average repetitions within each patient/question, then questions within patient, then patients equally. The scenario comparator instead averages patient/question means within each scenario, then scenarios equally. The 95% percentile intervals use 20,000 patient-level bootstrap resamples, keeping every patient's scenarios and repeated executions together. Leave-one-patient-out (LOPO) removes each patient's complete history; its range measures sensitivity rather than confidence coverage. In conclusion, these analyses support consistency in broad model-level patterns across two automated judges, while revealing judge-dependent scores and substantial patient-level uncertainty. We also interpret the agentic findings as exploratory results within the evaluated cases. In the future, independent clinician validation and larger, more diverse patient cohorts remain necessary to establish clinical validity and generalizability.
AuthorsOpenReview ↗

Response to Weakness 2 Limited Explanatory Depth and Question 2

Thank you for this comment and question. We agree that performance differences should be interpreted by distinguishing among model capabilities, format compliance, task composition, and preprocessing. We have expanded the Results and Discussion sections and conducted additional analyses to address these concerns.

First, we clarified the distinction between workflow compliance and clinical correctness. In the agentic track, invalid submissions receive a correctness score of 0. We added correctness among valid submissions (C | valid) to Table 5 and expanded Section 4.3. Although end-to-end correctness ranges from 7.4% to 69.7%, correctness among valid submissions falls between 67% and 79% for most models. For example, Qwen3.6-35B-A3B and Qwen3.8-27B achieve 79.2% and 78.2% correctness among valid submissions, compared with 71.6% for Gemini 3.1 Pro. Gemini's higher end-to-end correctness is largely attributable to its 97.3% workflow completion rate. Similarly, MedGemma 1.5 produced no valid submissions, indicating a workflow failure rather than a demonstrated clinical error. We also expanded Section 4.5 and Appendix Figures 4 and 7–10 to illustrate format-related errors in the static track.

Second, we examined the influence of task composition. All models were evaluated using the same samples, prompts, and scoring rules, except for 75 samples blocked for the two GPT models (Table 11). The sample-weighted overall ranking (Table 9) and unweighted macro-average across task groups (Table 12) showed strong agreement (Spearman ρ = 0.97), suggesting that overall rankings were not substantially driven by task sample sizes. Our new cross-capability analysis (Section 4.4) further showed that 2D-image rankings do not predict volume-derived performance, and classification rankings do not reliably predict localization or regression performance. These findings remained consistent after excluding the two GPT models with reduced coverage (ρ = 0.05, −0.44, and 0.13, respectively).

\hline \textbf{Model} & \begin{array}{c} \textbf{2D classification} \\\\ \textbf{accuracy (\\%)} \\\\ 256 \to 512\\ \text{px} \end{array} & \begin{array}{c} \textbf{Video classification} \\\\ \textbf{accuracy (\\%)} \\\\ \text{30 FPS/256 px} \to \\\\ \text{30 FPS/512 px} \to \\\\ \text{up-to-60 FPS/256 px} \end{array} & \begin{array}{c} \textbf{3D localization} \\\\ \textbf{(mean IoU}\times\textbf{100)} \\\\ \text{10 MIPs} \to \\\\ \text{20 MIPs} \to \\\\ \text{20 representative slices} \end{array} \\\\ \hline \textbf{Gemma-4-E4B-it} & 37.50 \to 37.50 & 47.37 \to 47.37 \to 47.37 & 2.37 \to 2.00 \to 2.16 \\\\ \hline \textbf{MedGemma 1.5} & 37.50 \to 37.50 & 36.84 \to 21.05 \to 57.89 & 0.69 \to 0.00 \to 0.00 \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 12.50 \to 50.00 & 63.16 \to 63.16 \to 57.89 & 1.99 \to 1.30 \to 0.99 \\\\ \hline \textbf{Gemini 3.1 Pro} & 50.00 \to 50.00 & 63.16 \to 68.42 \to 73.68 & 4.54 \to 5.55 \to 0.28 \\\\ \hline \textbf{GPT-5.6 Sol} & 50.00 \to 50.00 & \text{—} & 2.81 \to 2.83 \to 2.36 \\\\ \hline \end{array}$$
Reviewer_3vsMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposed MedStat-Bench, a benchmark to evaluate the ability of multimodal large language models (MLLMs) for different medical data and tasks. It includes static evaluation with 430 example, covering clinical text, 2D images, 3D images, and videos, spanning classification, structured generation, numerical prediction, and spatial location, and agentic evaluation, with 29 scenarios from 11 patient cases, requiring the MLLMs to retrieve evidence, understand medical images, use tools, and submit clinical judgement. The authors evaluate 13 models on this benchmark. Experimental results show that existing models still have room to improve in interpreting volume-derived views, producing precise spatial coordinates, and workflow execution.

Critical strengths
  1. The benchmark covers diverse output formats, which have clinical value.

Besides classification and text generation, the work also includes regression, keypoint localization, and lesion localization to comprehensively reflect the ability of MLLMs in medical tasks.

  1. The agentic evaluation focuses more on the final answer.

The authors separately report valid submission rate, answer correctness, evidence reasoning, and clinical safety scores, which are helpful to recognize the problems of these models in tool use, workflow execution, and clinical judgement.

Critical weaknesses
  1. Compared with existing medical evaluation benchmarks, the true research gap that this work is going to bridge still needs clarification.

Existing works have already covered key designs in this work. For example, in static evaluation, M3D-Bench [1] has already evaluated 3D medical visual question answering, report generation, and spatial localization that are also designed in this work. U2-Bench [2] covers ultrasound imaging analysis (diagnosis), detection, regression, and text generation, which are similar to the idea proposed in this work that uses different output formats to evaluate medical visual language models.

In agentic evaluation, AgentClinic [3] has proactively acquired clinical information and medical imaging to conduct multi-turn diagnosis, which is similar to the evidence acquisition and multimodal decision-making process that are proposed in this work. PhysicianBench [4] uses structured checkpoints to evaluate information retrieval, clinical reasoning, and task execution, which shares a similar design with this work that focuses on the final answer and workflow completion. HealthAgentBench [5] uses 2D images, 3D CT, pathology images, and EHRs to drive tool use and evaluate multiple steps, which is similar to the cross-modal, cross-workflow design in this work.

In conclusion, it seems that this proposed work has significant overlap compared with existing works. The authors are highly recommended to clearly state the real research gap that this work is going to bridge to help reviewers better evaluate the novelty of this work.

  1. It is unclear whether the visual information is important in the agentic tasks given that for each medical image, we also have its report to show the clinical findings.

Many tasks can be completed mainly by the imaging reports (clinical text) without seeing the images. As the authors emphasize the importance of imaging modalities in the agentic workflow, it is unclear whether imaging modality can have significant improvement of the performance of different models. If not, it would be meaningless to incorporate images in the benchmark for agentic tasks.

  1. The evaluation sample size of the agentic track is limited.

I notice that there are only 29 scenarios from 11 patients in the agentic track, making this track limited for evaluating the performance of different models in their agentic ability. Therefore, most evaluation results related to the agentic task should be unreliable and make it hard to test the generalizability of each model.

  1. Stress testing may not be an accurate description of the proposed benchmark.

The authors of this paper regard this proposed benchmark as "stress testing". However, it is unclear what the specific stressors, the specific failure mechanisms, and the corresponding testing construction principle are.

References

[1] Bai, F., Du, Y., Huang, T., Meng, M. Q.-H., and Zhao, B. M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models. arXiv preprint arXiv:2404.00578, 2024.

[2] Le, A., Liu, H., Wang, Y., et al. U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding. In International Conference on Learning Representations (ICLR), 2026.

[3] Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., and Moor, M. AgentClinic: A Multimodal Agent Benchmark to Evaluate AI in Simulated Clinical Environments. arXiv preprint arXiv:2405.07960, 2024.

[4] Liu, R., Mohiuddin, I. Q., Schoeffler, A. J., et al. PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments. arXiv preprint arXiv:2605.02240, 2026.

[5] Liu, Q., Zhang, S., Qin, G., et al. HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents. arXiv preprint arXiv:2606.31179, 2026.

Questions for authors

None

6 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for the detailed comparison with existing medical evaluation benchmarks. We agree that previous studies have addressed important aspects of medical multimodal evaluation, including 3D localization, diverse output formats, clinical reasoning, tool use, and workflow assessment. However, these benchmarks generally focus on specific imaging modalities, evaluation paradigms, or clinical tasks. The primary research gap addressed by MedStat-Bench is the lack of a unified evaluation framework to systematically assess multimodal medical models across input modalities, output formats, and evaluation paradigms. Specifically, MedStat-Bench evaluates the same models across 2D images, volumetric imaging, video, and clinical text, covering seven output formats through both static capability assessment and agentic workflow evaluation. To our knowledge, none of the existing benchmarks combines 2D, volumetric, and video inputs with both static and agentic evaluation. To better highlight these differences, we have included all five benchmarks mentioned by the reviewer in the comparisons presented in Tables 1–2 of the revised manuscript. We also provide a point-by-point comparison below to further clarify the distinctions between MedStat-Bench and these existing benchmarks.

\hline \textbf{Benchmark} & \textbf{Inputs} & \textbf{Evaluation type} & \textbf{What MedStat-Bench adds} \\\\ \hline \textbf{M3D-Bench} & \text{3D CT} & \text{Static} & \begin{array}{l} \text{Evaluates 3D localization alongside 2D} \\\\ \text{images, video, and text, using the same} \\\\ \text{models and protocol. It covers CT and} \\\\ \text{multiple MRI sequences, not just CT.} \end{array} \\\\ \hline \textbf{U2-Bench} & \begin{array}{l} \text{Ultrasound} \\\\ \text{images and} \\\\ \text{video} \end{array} & \text{Static} & \begin{array}{l} \text{U2-Bench varies output formats within} \\\\ \text{ultrasound. We test whether output-} \\\\ \text{format effects hold }\textit{across}\text{ modalities:} \\\\ \text{ultrasound, CT, MRI, endoscopy, and} \\\\ \text{laparoscopic video.} \end{array} \\\\ \hline \textbf{AgentClinic} & \begin{array}{l} \text{Dialogue, 2D} \\\\ \text{images} \end{array} & \text{Agentic} & \begin{array}{l} \text{Adds volumetric imaging, video, and} \\\\ \text{ECG inspected directly through a viewer,} \\\\ \text{structured clinical decisions on real} \\\\ \text{patient records, and a companion static} \\\\ \text{benchmark.} \end{array} \\\\ \hline \textbf{PhysicianBench} & \text{EHR, text} & \text{Agentic} & \begin{array}{l} \text{Adds direct inspection of medical} \\\\ \text{images and video, plus static evaluation} \\\\ \text{of perception and spatial localization.} \end{array} \\\\ \hline \textbf{HealthAgentBench} & \begin{array}{l} \text{2D, 3D CT,} \\\\ \text{pathology,} \\\\ \text{EHR} \end{array} & \text{Agentic} & \begin{array}{l} \text{Adds clinical video and a static} \\\\ \text{capability benchmark. It separately} \\\\ \text{scores workflow validity, answer} \\\\ \text{correctness, reasoning, and safety per} \\\\ \text{run.} \end{array} \\\\ \hline \end{array}$$ Importantly, the value of this comprehensive evaluation lies not simply in broader coverage, but in the ability to identify performance differences that may be overlooked by benchmarks focused on individual dimensions. To demonstrate this, we further analyzed the performance rankings of the 13 evaluated models across input modalities, output formats, and evaluation paradigms using the results already presented in the manuscript. Across input modalities, model rankings on 2D images and volumetric imaging showed little agreement (Spearman ρ = −0.16). For example, GPT-5.5 ranked 2nd on 2D images but 13th on volumetric inputs, while Qwen3.8-27B ranked 3rd and 11th, respectively. These findings suggest that performance on one imaging modality does not necessarily reflect a model's capabilities on another. Similar inconsistencies were observed across output formats, where single-label classification rankings showed little agreement with 2D localization (ρ = −0.24) or regression (ρ = 0.06). For example, Gemini 3.1 Pro achieved the highest classification performance but ranked only 7th in 2D localization. Furthermore, static benchmark performance did not consistently predict agentic performance. Gemma-4-26B-A4B-it and Gemma-4-31B-it achieved nearly identical static scores (33.54% and 33.48%, respectively), but their agentic workflow validity differed substantially (0.0% vs. 83.8%). Similarly, Qwen3.8-27B ranked 9th on the static benchmark but achieved the highest agentic reasoning score (95.2%), while Gemini 3.1 Pro, the highest-performing model on the static benchmark, ranked 3rd in overall agentic performance.

These findings demonstrate that model performance varies substantially across input modalities, output formats, and evaluation paradigms, and that strong performance in one dimension does not necessarily translate to strong performance in another. Such differences can only be systematically identified through an integrated evaluation of the same models across these dimensions. Therefore, the primary contribution of MedStat-Bench is not the introduction of individual evaluation tasks but a unified, cross-dimensional evaluation framework that enables a more comprehensive characterization of the capabilities and limitations of multimodal medical models. We have added this analysis in Section 4.4 to further clarify the research gap and highlight the contribution of our work.

AuthorsOpenReview ↗

Response to Weakness 2

We appreciate your question about the contribution of visual information. We acknowledge that radiologist-generated imaging reports may provide a more effective representation of clinically relevant information for downstream LLM-based clinical reasoning than direct access to medical images. First, we reviewed all 29 agentic scenarios and summarized which tasks provide both medical images and reports.

  • Q01 intentionally excludes imaging reports.
  • Q02–Q11 include NCCT slices plus CT-report files for all five patients, even when the question concerns another modality (e.g., Q08 asks about echocardiography).
  • Q12 provides two videos and baseline context, with no corresponding laparoscopic report.
  • All 13 stroke imaging and outcome prediction scenarios (Q13–Q25) mount CTP image volumes, 44 RAPID report PNGs per patient, and a non-empty CTP text report per patient.
  • Q26/Q27 explicitly exclude diagnostic reports in MRI metadata.
  • Q28/Q29 provide ECG waveforms and general records without a separate interpretation report.

We then compared the settings “images and reports”, “reports only”, and “images only” for the agentic tasks from Q2 to Q11, using Gemini 3.1 Pro and GPT-5.6-sol. The results table below uses the same 37 selected patient–question pairs, with scored results across all three conditions for both models. We report the six columns and the composite formulas used in Table 5 of the manuscript. Final-answer correctness C is the percentage of applicable answer criteria met; reasoning R is the percentage of applicable reasoning criteria met; clinical safety S is the percentage of applicable safety checks passed among valid, gradeable submissions. C can include partial credit, as specified in Appendix F. The answer–reasoning composite is U = 0.5C + 0.5R, and the overall score is T = U × S/100. Each model/condition mean has n = 37.

\hline \textbf{Model} & \textbf{Condition} & \begin{array}{c} \textbf{Final-answer} \\\\ \textbf{correctness} \\\\ C\\ (\\%) \end{array} & \begin{array}{c} \textbf{Reasoning} \\\\ R\\ (\\%) \end{array} & \begin{array}{c} \textbf{Clinical} \\\\ \textbf{safety}\\ S \\\\ (\\%) \end{array} & \begin{array}{c} 0.5C + \\\\ 0.5R\\ (\\%) \end{array} & \begin{array}{c} \textbf{Overall} \\\\ (\\%) \end{array} \\\\ \hline \text{Gemini 3.1 Pro} & \text{Images + reports} & 87.5 & 73.6 & 92.3 & 80.6 & 74.4 \\\\ & \text{Reports only} & 87.0 & 82.3 & 91.7 & 84.7 & 77.6 \\\\ & \text{Images only} & 83.6 & 72.9 & 87.6 & 78.3 & 68.6 \\\\ \hline \text{GPT-5.6 Sol} & \text{Images + reports} & 92.3 & 92.1 & 96.4 & 92.2 & 88.9 \\\\ & \text{Reports only} & 91.6 & 98.2 & 97.1 & 94.9 & 92.1 \\\\ & \text{Images only} & 91.4 & 92.6 & 96.4 & 92.0 & 88.7 \\\\ \hline \end{array}$$ Additionally, the following tables show the fully corrected cases with C=100%. $$\begin{array}{lccc} \hline \textbf{Judge} & \textbf{Images + reports} & \textbf{Images only} & \textbf{Reports only} \\\\ \hline \textbf{GPT-5.6 Sol} & 32/37\\ (86.5\\%) & 32/37\\ (86.5\\%) & 32/37\\ (86.5\\%) \\\\ \hline \textbf{Gemini 3.1 Pro} & 28/37\\ (75.7\\%) & 27/37\\ (73.0\\%) & 29/37\\ (78.4\\%) \\\\ \hline \textbf{Combined} & \textbf{60/74 (81.1\\%)} & \textbf{59/74 (79.7\\%)} & \textbf{61/74 (82.4\\%)} \\\\ \hline \end{array}$$ The central observation is that reports-only achieved the highest number of fully correct answers. Adding images to reports did not improve the aggregate result. For GPT-5.6 Sol, all three conditions produced exactly the same number of fully correct answers. For Gemini 3.1 Pro, reports alone performed slightly better than either multimodal input or images alone. This suggests that the radiology reports were sufficient for many of these tasks, at least as evaluated by final-answer correctness. Furthermore, cases where adding images reduced final-answer correctness suggest potential limitations in visual interpretation or in cross-modal evidence integration. We raised 2 hypotheses: (1) The models struggle to extract clinically important information directly from imaging and therefore benefit more from expert-generated radiology reports. (2) The models can extract relevant information from both sources individually, but combining them sometimes creates conflicting evidence or distracts the final reasoning process. These findings demonstrate a gap between providing multimodal inputs and achieving measurable benefits in multimodal reasoning, which warrants future investigation.
Reviewer_Ura8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedStat-Bench evaluates medical models across classification, measurement, localization, structured text, and tool-based workflows. It contains 430 static samples and 29 agentic scenarios from 11 patients. Performance varies across tasks and input formats.

Critical strengths

S1. Broader evaluation Measurement, localization, and workflow tasks expose failures that diagnostic question answering may miss. Task-specific reporting is a strength.

S2. Transparent construction The public track fixes inputs, prompts, references, and scoring. Institutional annotations receive expert review. The authors clearly acknowledge several limitations.

Critical weaknesses

M1. Input sufficiency

Resizing images to 256 pixels, sampling video frames, and projecting volumes may remove essential evidence. Poor performance could reflect preprocessing rather than model limitations. Validate answerability with clinicians using identical inputs and test alternative representations.

M2. Safety-score validity

GPT-5.6 Sol judges all models, including itself, without independent clinician validation. Safety excludes invalid submissions and averages checks, potentially obscuring serious errors. The composite combines different subsets rather than measuring safe, correct completion per case.

Other concerns

-- low case-level (specialty) counts. Only 11 patients underpin the track. Report case-level uncertainty and exact run counts.

  • Tool misuse or exhausting 20 actions gives zero correctness credit. Separate clinical-answer correctness from end-to-end completion.
  • Justify the regression tolerance of max(0.1 × |reference value|, 5 units). Check whether text-overlap scores capture clinically important errors.
  • Some models are scored on 355 rather than 430 samples. I suggest ot add a matched-subset comparison.
  • Figure 3 conflicts with Tables 5 and 8 for GPT-5.5 and MedGemma. Figure 3 also plots zero where Table 5 reports unavailable scores.
Questions for authors
  1. What evidence shows that the processed inputs retain enough information to answer the tasks? Existing validation or a small clinician-reviewed subset would help. A full preprocessing study is not required.

  2. Have automated safety judgments been checked against independent clinical review? If not, a small validation sanitiy check subset would help. Otherwise, clarify that these are unvalidated rubric-based scores.

  3. Using existing outputs and logs, can you distinguish clinical-answer errors, workflow failures, and critical safety errors? Please state where this separation is not possible.

  4. Can you report comparisons on the samples shared across models and reconcile the figure/table discrepancies?

6 comments
AuthorsOpenReview ↗

Response to M1 Input sufficiency and Question 1

Thank you for highlighting the importance of answerability from processed inputs. We agree that preprocessing could remove relevant evidence. To investigate the preprocessing sensitivity, we further evaluated five models, Gemini 3.1 Pro, GPT 5.6 Sol, Gemma4-E4B, Qwen3.6-35B-A3B, and MedGemma 1.5, on a fixed 100-sample subset of the original 430-sample dataset. The cohort contains 17 2D-image questions, 30 video questions, 45 3D volume questions, and 8 text-only controls. The alternative preprocessing settings were: (1) 2D images, 512-pixel; (2) videos—30 frames at 512 pixels, and up to 60 frames at 256 pixels; (3) 3D volumes—20 depth-range MIPs, and 20 representative original axial slices at 256 pixels. Each comparison uses exactly the same question IDs across settings within a model, with the baseline recomputed on that matched cohort. Missing outputs are excluded rather than scored as incorrect. We also prepared comparisons restricted to samples completed by all five models. Representative task-specific results are below: classification uses exact-match accuracy, and 3D localization uses matched-box IoU.

\hline \textbf{Model} & \begin{array}{c} \textbf{2D classification} \\\\ \textbf{accuracy (\\%)} \\\\ 256 \to 512\\ \text{px} \end{array} & \begin{array}{c} \textbf{Video classification} \\\\ \textbf{accuracy (\\%)} \\\\ \text{30 FPS/256 px} \to \\\\ \text{30 FPS/512 px} \to \\\\ \text{up-to-60 FPS/256 px} \end{array} & \begin{array}{c} \textbf{3D localization} \\\\ \textbf{(mean IoU}\times\textbf{100)} \\\\ \text{10 MIPs} \to \\\\ \text{20 MIPs} \to \\\\ \text{20 representative slices} \end{array} \\\\ \hline \textbf{Gemma-4-E4B-it} & 37.50 \to 37.50 & 47.37 \to 47.37 \to 47.37 & 2.37 \to 2.00 \to 2.16 \\\\ \hline \textbf{MedGemma 1.5} & 37.50 \to 37.50 & 36.84 \to 21.05 \to 57.89 & 0.69 \to 0.00 \to 0.00 \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 12.50 \to 50.00 & 63.16 \to 63.16 \to 57.89 & 1.99 \to 1.30 \to 0.99 \\\\ \hline \textbf{Gemini 3.1 Pro} & 50.00 \to 50.00 & 63.16 \to 68.42 \to 73.68 & 4.54 \to 5.55 \to 0.28 \\\\ \hline \textbf{GPT-5.6 Sol} & 50.00 \to 50.00 & \text{—} & 2.81 \to 2.83 \to 2.36 \\\\ \hline \end{array}$$ The matched sample counts are 8 for 2D classification, 19 for video classification in the four non-GPT models (the HIPAA-compliant GPT had Azure content blocks), and 24 for 3D localization (23 for GPT). Across the full video subset, 22 clips reach 60 frames, and eight are capped below 60. GPT completed only those eight capped clips, including two classification questions, so its available results do not test actual 60-frame inputs. The results show model- and task-dependent sensitivity to preprocessing. Qwen3.6-35B-A3B’s 2D classification accuracy increased by 37.50 percentage points at 512 pixels (95% confidence interval: 12.50 to 75.00). MedGemma 1.5's video classification accuracy increased by 21.05 points with up to 60 frames relative to baseline (95% CI: 0.00 to 42.11), while increasing frame resolution alone reduced its accuracy. These exploratory intervals use 2,000 bootstrap draws over shared-source-media clusters, with one evaluation pass and no multiple-comparison correction. Unchanged accuracy in other settings does not demonstrate retained clinical evidence or equivalence between representations. The 3D comparisons do not establish input sufficiency. Localization scores remain low, and the scoring audit identifies 16 questions with frozen prompts that request slice-wise 2D coordinates despite a 3D-box ground truth. Such format conflicts can yield a 3D IoU of 0 even when an answer contains useful clinical information. We therefore treat these scores as partial evidence about representation sensitivity. They cannot determine whether the projection removed the evidence needed by a clinician, a determination that requires further expert review.
AuthorsOpenReview ↗

Response to M2 Safety-score validity and Question 2

Thank you for highlighting the safety-validation and aggregation concerns. We were not able to obtain independent clinician reviews due to the short review time, but if accepted, we will add clinician reviews as requested. As an automated robustness check, we used high-performance Claude Opus 5.5, from a different model family, to judge frozen outputs and rubrics without access to GPT-5.6 Sol's grades. We ran 11 scenarios (Q1–Q11) on 5 patients. We retain the original safety average and separately report strict safety, requiring every applicable critical check to pass with no unresolved assessments. The table below shows results for the paired outputs with both GPT-5.6 Sol and Opus 5.5 judgments.

\hline \textbf{Metric} & \begin{array}{c} \textbf{GPT-5.6 Sol judge} \\\\ \textbf{(95\\% CI)} \end{array} & \begin{array}{c} \textbf{Opus 5.5 judge} \\\\ \textbf{(95\\% CI)} \end{array} \\\\ \hline \textbf{Mean safety-check fraction (\\%)} & 87.9\\ (84.9\text{–}90.8) & 86.9\\ (83.3\text{–}89.8) \\\\ \hline \textbf{Strict critical-safety pass (\\%)} & 67.2\\ (60.8\text{–}73.3) & 64.1\\ (56.2\text{–}70.6) \\\\ \hline \end{array}$$ The table below also shows the per-agent comparison for the exact model order and groups in manuscript Table 5. Each score cell is “GPT-5.6 Sol / Opus 5.5”, in percent. A dash indicates no paired result, not zero performance. For direct metric comparability, we report the content metrics defined in manuscript Table 5 and Appendix F: final-answer correctness C, reasoning R, safety S, combined score U = (C + R)/2, and overall score T = U × S/100 when scores are expressed as percentages. Here, S is the mean fraction of applicable critical checks satisfied, rather than the percentage of outputs passing every critical check. C measures agreement with the grading reference; R and S measure automated rubric compliance. We also add per-run Safe and Correct Completion Rate (SCCR): successful workflow completion, full clinical-reference correctness, and all applicable critical checks passing. $$\begin{array}{lcccccc} \hline \textbf{Agent model} & \textbf{C} & \textbf{R} & \textbf{S} & \textbf{U} & \textbf{T} & \textbf{SCCR} \\\\ \hline \textbf{Open-weight} & & & & & & \\\\ \hline \textbf{Gemma-4-31B-it} & 79.1 / 81.7 & 61.6 / 45.3 & 90.0 / 90.4 & 70.4 / 63.5 & 66.0 / 60.2 & 58.8 / 68.6 \\\\ \hline \textbf{Gemma-4-E2B-it} & 8.9 / 7.0 & 2.6 / 1.1 & 83.7 / 82.4 & 5.8 / 4.0 & 5.5 / 3.8 & 5.8 / 3.8 \\\\ \hline \textbf{Gemma-4-E4B-it} & 32.3 / 29.1 & 24.1 / 8.5 & 79.9 / 74.3 & 28.2 / 18.8 & 23.3 / 15.9 & 21.6 / 17.6 \\\\ \hline \textbf{Qwen3.6-27B} & 79.8 / 78.8 & 56.8 / 39.9 & 84.2 / 85.9 & 68.3 / 59.3 & 59.5 / 54.8 & 47.9 / 60.4 \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 81.2 / 84.7 & 61.1 / 43.0 & 83.3 / 87.7 & 71.1 / 63.8 & 63.2 / 58.7 & 56.8 / 61.4 \\\\ \hline \textbf{Qwen3.8-27B} & 83.9 / 85.5 & 78.1 / 66.0 & 83.8 / 87.4 & 81.0 / 75.8 & 70.8 / 69.6 & 58.8 / 64.7 \\\\ \hline \textbf{Proprietary} & & & & & & \\\\ \hline \textbf{Gemini 3.1 Pro} & 83.6 / 84.4 & 72.8 / 63.7 & 90.8 / 91.5 & 78.2 / 74.0 & 73.1 / 69.5 & 64.0 / 62.0 \\\\ \hline \textbf{GPT-5.5} & 90.8 / 90.9 & 89.5 / 87.3 & 95.9 / 97.2 & 90.2 / 89.1 & 87.8 / 87.6 & 86.3 / 80.4 \\\\ \hline \textbf{GPT-5.6 Sol} & 92.0 / 91.4 & 91.6 / 88.0 & 96.3 / 95.0 & 91.8 / 89.7 & 89.6 / 86.6 & 84.0 / 78.0 \\\\ \hline \end{array}$$ The following table further shows the associations between the available paired agent-model summaries in the table above. Pearson r measures linear association; Spearman ρ measures ranking association. The 95% patient-level bootstrap intervals are also given. These are model-level associations, not agreement rates for individual outputs. $$\begin{array}{lcc} \hline \textbf{Table 5 metric} & \textbf{Pearson } r\\ \textbf{(95\\% CI)} & \textbf{Spearman } \rho\\ \textbf{(95\\% CI)} \\\\ \hline \textbf{C} & 0.998\\ (0.989\text{–}0.999) & 0.988\\ (0.842\text{–}1.000) \\\\ \hline \textbf{R} & 0.977\\ (0.960\text{–}0.982) & 1.000\\ (0.939\text{–}1.000) \\\\ \hline \textbf{S} & 0.820\\ (0.615\text{–}0.895) & 0.782\\ (0.576\text{–}0.927) \\\\ \hline \textbf{U} & 0.994\\ (0.987\text{–}0.995) & 1.000\\ (0.952\text{–}1.000) \\\\ \hline \textbf{T} & 0.993\\ (0.981\text{–}0.995) & 0.988\\ (0.952\text{–}1.000) \\\\ \hline \end{array}$$ Overall, the cross-family judging analysis supports the robustness of broad model-level comparisons, while showing that absolute scores remain judge-dependent. Strict safety and SCCR provide more conservative measures by requiring that all critical checks pass and by retaining workflow failures in the completion denominator. These results remain unvalidated, automated rubric-based assessments; independent clinician review is necessary to establish their clinical validity.
AuthorsOpenReview ↗

Response to Other Concern 1

Low case-level (specialty) counts. Only 11 patients underpin the track. Report case-level uncertainty and exact run counts.

Thank you for this comment. We reported the patient-level uncertainty in the following table.

\hline \textbf{Model} & \begin{array}{c} \textbf{Patient-level reward} \\\\ \textbf{(95\\% CI)} \end{array} & \textbf{Scenario mean} & \textbf{LOPO range} \\\\ \hline \textbf{Open-weight} & & & \\\\ \hline \textbf{Gemma-4-31B-it} & 48.14 & 45.52 & 42.95\text{–}52.95 \\\\ & (27.89\text{–}67.48) & & \\\\ \hline \textbf{Gemma-4-E2B-it} & 5.56 & 6.29 & 4.75\text{–}6.12 \\\\ & (2.25\text{–}9.09) & & \\\\ \hline \textbf{Gemma-4-E4B-it} & 27.29 & 13.10 & 20.02\text{–}30.02 \\\\ & (9.00\text{–}50.45) & & \\\\ \hline \textbf{Qwen3.6-27B} & 5.01 & 4.94 & 3.24\text{–}5.51 \\\\ & (1.65\text{–}9.20) & & \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 39.51 & 29.20 & 33.46\text{–}43.46 \\\\ & (21.68\text{–}57.44) & & \\\\ \hline \textbf{Qwen3.8-27B} & 28.89 & 29.77 & 24.51\text{–}31.78 \\\\ & (14.30\text{–}43.80) & & \\\\ \hline \textbf{Proprietary} & & & \\\\ \hline \textbf{Gemini 3.1 Pro} & 61.94 & 61.24 & 58.13\text{–}68.13 \\\\ & (46.18\text{–}75.56) & & \\\\ \hline \textbf{GPT-5.5} & 67.16 & 59.67 & 63.06\text{–}73.83 \\\\ & (51.07\text{–}80.27) & & \\\\ \hline \textbf{GPT-5.6 Sol} & 53.73 & 53.88 & 49.10\text{–}59.10 \\\\ & (33.19\text{–}72.31) & & \\\\ \hline \end{array}$$ We first average repetitions within each patient/question, then questions within patient, then patients equally. The scenario comparator instead averages patient/question means within each scenario, then scenarios equally. The 95% percentile intervals use 20,000 patient-level bootstrap resamples, keeping every patient's scenarios and repeated executions together. Leave-one-patient-out (LOPO) removes each patient's complete history; its range measures sensitivity rather than confidence coverage. These analyses quantify uncertainty across patients and sensitivity to individual cases. Our results show that our agentic benchmark can still distinguish each model's capabilities despite its relatively small size. Nevertheless, we will continue expanding our benchmark to make it more comprehensive and enable a more robust comparison of agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.