- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MedStat-Bench, a medical multimodal benchmark with 430 static samples and 29 agentic scenarios drawn from 11 patient cases. It evaluates models across clinical text, images, volume-derived views, and sampled video, covering classification, generation, measurement, localization, and tool-mediated decisions. The authors report performance differences across tasks and modalities.
Critical strengths
-
Broad evaluation coverage. The benchmark spans clinical text, 2D images, volume-derived views, and sampled video, with tasks covering classification, generation, measurement, and localization. Its agentic track additionally examines tool-mediated clinical workflows, enabling evaluation of capabilities beyond those captured by a single-modality or single-task benchmark.
-
Support for reproducibility. The manuscript documents dataset composition, input preprocessing, prompting conventions, scoring rules, and agentic evaluation rubrics. It also reports repeated static evaluations with confidence intervals. These details provide a useful basis for inspecting the evaluation protocol and support reproduction efforts, although the availability and completeness of the released materials remain to be verified.
Critical weaknesses
-
Evaluation reliability. Agentic reasoning and safety scores lack validation against human assessments, and GPT-5.6 Sol judges its own outputs. Acknowledging these limitations does not resolve them. Unlike the static evaluation, which reports four runs and confidence intervals, the agentic evaluation leaves execution repetitions unclear and provides no confidence intervals. Additionally, the agentic track reuses only 11 patient cases across 29 scenarios, limiting case diversity and the generalizability of its findings.
-
Limited explanatory depth. The findings mainly describe score and ranking differences without sufficiently separating model capabilities from preprocessing, format compliance, and task composition. Medical-specialized baselines are limited to MedGemma 1.5, and the Gemma–MedGemma results receive little analysis accounting for underlying model differences.
Questions for authors
-
Can the authors provide clinician–judge agreement validation, clarify agentic execution repetitions, and report confidence intervals and judge-agreement statistics?
-
Can additional analyses distinguish model limitations from preprocessing and format-compliance failures, and contextualize Gemma versus MedGemma performance?
-
Do the following differences reflect distinct metrics, subsets, or experimental versions?
- Table 11 versus Table 12, 3D localization: GPT-5.5 0.70% vs 8.5%; Qwen3.6-27B 0.14% vs 1.0%.
- Figure 3 versus Table 8, static overall score: GPT-5.5 35.4% vs 33.68%; MedGemma 25.8% vs 23.65%.
- Gemini volume-derived score: 35.8% in Table 4, versus approximately 34.71% when weighting Table 13 by the stated sample counts.
-
Can the authors verify the bibliography? EndoBench, SurgBench, 3D-RAD, MedFrameQA, and TemMed-Bench have incorrect first-author names; several titles differ from the cited records, and some entries use author placeholders.
6 comments
Cont'd
- Unlike the static evaluation, which reports four runs and confidence intervals, the agentic evaluation leaves execution repetitions unclear and provides no confidence intervals. Additionally, the agentic track reuses only 11 patient cases across 29 scenarios, limiting case diversity and the generalizability of its findings.
Thank you for pointing this out. We reported the patient-level uncertainty in the following table.
\hline \textbf{Model} & \begin{array}{c} \textbf{Patient-level reward} \\\\ \textbf{(95\\% CI)} \end{array} & \textbf{Scenario mean} & \textbf{LOPO range} \\\\ \hline \textbf{Open-weight} & & & \\\\ \hline \textbf{Gemma-4-31B-it} & 48.14 & 45.52 & 42.95\text{–}52.95 \\\\ & (27.89\text{–}67.48) & & \\\\ \hline \textbf{Gemma-4-E2B-it} & 5.56 & 6.29 & 4.75\text{–}6.12 \\\\ & (2.25\text{–}9.09) & & \\\\ \hline \textbf{Gemma-4-E4B-it} & 27.29 & 13.10 & 20.02\text{–}30.02 \\\\ & (9.00\text{–}50.45) & & \\\\ \hline \textbf{Qwen3.6-27B} & 5.01 & 4.94 & 3.24\text{–}5.51 \\\\ & (1.65\text{–}9.20) & & \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 39.51 & 29.20 & 33.46\text{–}43.46 \\\\ & (21.68\text{–}57.44) & & \\\\ \hline \textbf{Qwen3.8-27B} & 28.89 & 29.77 & 24.51\text{–}31.78 \\\\ & (14.30\text{–}43.80) & & \\\\ \hline \textbf{Proprietary} & & & \\\\ \hline \textbf{Gemini 3.1 Pro} & 61.94 & 61.24 & 58.13\text{–}68.13 \\\\ & (46.18\text{–}75.56) & & \\\\ \hline \textbf{GPT-5.5} & 67.16 & 59.67 & 63.06\text{–}73.83 \\\\ & (51.07\text{–}80.27) & & \\\\ \hline \textbf{GPT-5.6 Sol} & 53.73 & 53.88 & 49.10\text{–}59.10 \\\\ & (33.19\text{–}72.31) & & \\\\ \hline \end{array}$$ We first average repetitions within each patient/question, then questions within patient, then patients equally. The scenario comparator instead averages patient/question means within each scenario, then scenarios equally. The 95% percentile intervals use 20,000 patient-level bootstrap resamples, keeping every patient's scenarios and repeated executions together. Leave-one-patient-out (LOPO) removes each patient's complete history; its range measures sensitivity rather than confidence coverage. In conclusion, these analyses support consistency in broad model-level patterns across two automated judges, while revealing judge-dependent scores and substantial patient-level uncertainty. We also interpret the agentic findings as exploratory results within the evaluated cases. In the future, independent clinician validation and larger, more diverse patient cohorts remain necessary to establish clinical validity and generalizability.Response to Weakness 2 Limited Explanatory Depth and Question 2
Thank you for this comment and question. We agree that performance differences should be interpreted by distinguishing among model capabilities, format compliance, task composition, and preprocessing. We have expanded the Results and Discussion sections and conducted additional analyses to address these concerns.
First, we clarified the distinction between workflow compliance and clinical correctness. In the agentic track, invalid submissions receive a correctness score of 0. We added correctness among valid submissions (C | valid) to Table 5 and expanded Section 4.3. Although end-to-end correctness ranges from 7.4% to 69.7%, correctness among valid submissions falls between 67% and 79% for most models. For example, Qwen3.6-35B-A3B and Qwen3.8-27B achieve 79.2% and 78.2% correctness among valid submissions, compared with 71.6% for Gemini 3.1 Pro. Gemini's higher end-to-end correctness is largely attributable to its 97.3% workflow completion rate. Similarly, MedGemma 1.5 produced no valid submissions, indicating a workflow failure rather than a demonstrated clinical error. We also expanded Section 4.5 and Appendix Figures 4 and 7–10 to illustrate format-related errors in the static track.
Second, we examined the influence of task composition. All models were evaluated using the same samples, prompts, and scoring rules, except for 75 samples blocked for the two GPT models (Table 11). The sample-weighted overall ranking (Table 9) and unweighted macro-average across task groups (Table 12) showed strong agreement (Spearman ρ = 0.97), suggesting that overall rankings were not substantially driven by task sample sizes. Our new cross-capability analysis (Section 4.4) further showed that 2D-image rankings do not predict volume-derived performance, and classification rankings do not reliably predict localization or regression performance. These findings remained consistent after excluding the two GPT models with reduced coverage (ρ = 0.05, −0.44, and 0.13, respectively).
\hline \textbf{Model} & \begin{array}{c} \textbf{2D classification} \\\\ \textbf{accuracy (\\%)} \\\\ 256 \to 512\\ \text{px} \end{array} & \begin{array}{c} \textbf{Video classification} \\\\ \textbf{accuracy (\\%)} \\\\ \text{30 FPS/256 px} \to \\\\ \text{30 FPS/512 px} \to \\\\ \text{up-to-60 FPS/256 px} \end{array} & \begin{array}{c} \textbf{3D localization} \\\\ \textbf{(mean IoU}\times\textbf{100)} \\\\ \text{10 MIPs} \to \\\\ \text{20 MIPs} \to \\\\ \text{20 representative slices} \end{array} \\\\ \hline \textbf{Gemma-4-E4B-it} & 37.50 \to 37.50 & 47.37 \to 47.37 \to 47.37 & 2.37 \to 2.00 \to 2.16 \\\\ \hline \textbf{MedGemma 1.5} & 37.50 \to 37.50 & 36.84 \to 21.05 \to 57.89 & 0.69 \to 0.00 \to 0.00 \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 12.50 \to 50.00 & 63.16 \to 63.16 \to 57.89 & 1.99 \to 1.30 \to 0.99 \\\\ \hline \textbf{Gemini 3.1 Pro} & 50.00 \to 50.00 & 63.16 \to 68.42 \to 73.68 & 4.54 \to 5.55 \to 0.28 \\\\ \hline \textbf{GPT-5.6 Sol} & 50.00 \to 50.00 & \text{—} & 2.81 \to 2.83 \to 2.36 \\\\ \hline \end{array}$$
Response to Weakness 1 Evaluation reliability and Question 1
Thank you for this comment. We were not able to obtain independent clinician reviews due to the short review time, but if accepted, we will add clinician reviews as requested. As an automated robustness check, we used high-performance Claude Opus 5.5, from a different model family, to judge frozen outputs and rubrics without access to GPT-5.6 Sol's grades. We ran 11 scenarios (Q1–Q11) on 5 patients. The table below also shows the per-agent comparison for the exact model order and groups in manuscript Table 5. Each score cell is “GPT-5.6 Sol / Opus 5.5”, in percent. A dash indicates no paired result, not zero performance. For direct metric comparability, we report the content metrics defined in manuscript Table 5 and Appendix F: final-answer correctness C, reasoning R, safety S, combined score U = (C + R)/2, and overall score T = U × S/100 when scores are expressed as percentages. Here, S is the mean fraction of applicable critical checks satisfied, rather than the percentage of outputs passing every critical check. C measures agreement with the grading reference; R and S measure automated rubric compliance. We also add per-run Safe and Correct Completion Rate (SCCR): successful workflow completion, full clinical-reference correctness, and all applicable critical checks passing.
\hline \textbf{Agent model} & \textbf{C} & \textbf{R} & \textbf{S} & \textbf{U} & \textbf{T} & \textbf{SCCR} \\\\ \hline \textbf{Open-weight} & & & & & & \\\\ \hline \textbf{Gemma-4-31B-it} & 79.1 / 81.7 & 61.6 / 45.3 & 90.0 / 90.4 & 70.4 / 63.5 & 66.0 / 60.2 & 58.8 / 68.6 \\\\ \hline \textbf{Gemma-4-E2B-it} & 8.9 / 7.0 & 2.6 / 1.1 & 83.7 / 82.4 & 5.8 / 4.0 & 5.5 / 3.8 & 5.8 / 3.8 \\\\ \hline \textbf{Gemma-4-E4B-it} & 32.3 / 29.1 & 24.1 / 8.5 & 79.9 / 74.3 & 28.2 / 18.8 & 23.3 / 15.9 & 21.6 / 17.6 \\\\ \hline \textbf{Qwen3.6-27B} & 79.8 / 78.8 & 56.8 / 39.9 & 84.2 / 85.9 & 68.3 / 59.3 & 59.5 / 54.8 & 47.9 / 60.4 \\\\ \hline \textbf{Qwen3.6-35B-A3B} & 81.2 / 84.7 & 61.1 / 43.0 & 83.3 / 87.7 & 71.1 / 63.8 & 63.2 / 58.7 & 56.8 / 61.4 \\\\ \hline \textbf{Qwen3.8-27B} & 83.9 / 85.5 & 78.1 / 66.0 & 83.8 / 87.4 & 81.0 / 75.8 & 70.8 / 69.6 & 58.8 / 64.7 \\\\ \hline \textbf{Proprietary} & & & & & & \\\\ \hline \textbf{Gemini 3.1 Pro} & 83.6 / 84.4 & 72.8 / 63.7 & 90.8 / 91.5 & 78.2 / 74.0 & 73.1 / 69.5 & 64.0 / 62.0 \\\\ \hline \textbf{GPT-5.5} & 90.8 / 90.9 & 89.5 / 87.3 & 95.9 / 97.2 & 90.2 / 89.1 & 87.8 / 87.6 & 86.3 / 80.4 \\\\ \hline \textbf{GPT-5.6 Sol} & 92.0 / 91.4 & 91.6 / 88.0 & 96.3 / 95.0 & 91.8 / 89.7 & 89.6 / 86.6 & 84.0 / 78.0 \\\\ \hline \end{array}$$ The following table further shows the associations among the available paired agent-model summaries in the table above. Pearson r measures linear association; Spearman ρ measures ranking association. The 95% patient-level bootstrap intervals are also given. These are model-level associations, not agreement rates for individual outputs. $$\begin{array}{l|c|c} \hline \textbf{Table 5 metric} & \textbf{Pearson } r\\ \textbf{(95\\% CI)} & \textbf{Spearman } \rho\\ \textbf{(95\\% CI)} \\\\ \hline \textbf{C} & 0.998\\ (0.989\text{–}0.999) & 0.988\\ (0.842\text{–}1.000) \\\\ \hline \textbf{R} & 0.977\\ (0.960\text{–}0.982) & 1.000\\ (0.939\text{–}1.000) \\\\ \hline \textbf{S} & 0.820\\ (0.615\text{–}0.895) & 0.782\\ (0.576\text{–}0.927) \\\\ \hline \textbf{U} & 0.994\\ (0.987\text{–}0.995) & 1.000\\ (0.952\text{–}1.000) \\\\ \hline \textbf{T} & 0.993\\ (0.981\text{–}0.995) & 0.988\\ (0.952\text{–}1.000) \\\\ \hline \end{array}$$