- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces OMNICLINIC, an interactive multimodal environment for evaluating doctor agents on proactive clinical diagnosis. Diagnosis is cast as a POMDP in which an agent interacts with a patient for up to 15 turns through five action classes — clinical inquiry, physical examination, diagnostic testing, spatial configuration, and diagnosis submission — over 479 cases spanning 21 departments, with 4,251 generated images and 1,028 audio clips. Its distinguishing feature is that evidence is released only when the agent's examination matches a required route, including patient pose, clinician position, and body site, so that obtaining evidence requires proactive, spatially coordinated action rather than only choosing what to ask. Three metric families are reported (diagnostic accuracy by closed-set exact match, examination-parameter accuracy, and per-modality evidence-unlock rates), seven multimodal models are evaluated under fully presented evidence and under multimodal and text-only proactive interaction, and the authors additionally propose OMNIDOCTORX, an inference-time framework evaluated on two of the seven models.
Critical strengths
1. The environment introduces a genuinely new evaluation axis, and the work fills a real gap in the agentic-benchmark landscape.
Evidence is released only when the agent's examination matches a required route — patient pose, clinician position, and body site — which to my knowledge no prior clinical benchmark does; among the thirteen benchmarks compared in Table 1, OMNICLINIC is the only row marked ✓ for audio, physical-exam tool action, and pose/position interaction. This makes a class of failure observable that prior benchmarks structurally cannot measure, and it gives the work a clear position within the broader effort to scale up agentic environments. The paper is also clearly written and exceptionally well illustrated: Figure 1 gives an effective overview of the environment and its examination tool space, and Figure 6's color-gradient matrix makes the seven-model × seven-parameter decomposition easy to scan.
2. The synthetic media received independent physician review under a complete agreement-analysis protocol.
Three physicians rated all 42 sampled cases on eight criteria, giving 1,008 complete ratings, with mean ratings of 3.98–4.88 on a five-point scale and tight stratified-bootstrap intervals. The protocol combines Krippendorff's ordinal α, mean pairwise exact agreement, Gwet's AC2, permutation tests with Holm correction, and explicit sensitivity analyses — more complete than is typical in this literature. Just as importantly, the paper reports the unfavorable results alongside the favorable ones: negative agreement coefficients, significant rater effects on all eight criteria, and 67 critical free-text comments, while scoping its own claim to "structural integrity and operational consistency, not comprehensive clinical validity." Putting generated clinical media in front of a physician panel and then reporting the disagreements is a practice I would like to see become standard.
Critical weaknesses
1. The benchmark is not available, and its model coverage cannot support the conclusions drawn from it.
The paper contains no statement, not even a promise of release, about the data, code, or environment; the Reproducibility Statement describes documentation only, and the sole availability sentence is restrictive. As submitted, the primary deliverable of a benchmark paper can neither be used nor reproduced.
Coverage compounds this. The seven configurations are deployment-oriented rather than state-of-the-art — Gemini-3.7-Flash, Qwen3-Omni-30B-A3B (Instruct and Thinking), MGM-Omni-32B, MiniCPM-o-4.5, Phi-4-Multimodal-Instruct, and Baichuan-Omni-1.5 — with no frontier model of any kind and no stated selection rationale. Since the paper's conclusion is a claim about how current multimodal models behave, this coverage cannot separate the reported behavior from this particular model class.
2. The headline accuracy metric cannot be interpreted as reported.
Diagnostic accuracy is closed-set exact match, yet the number of options per case, how distractors were constructed, and whether a random baseline exists are never reported, so the option set's floor is unknown and figures such as 90.07% (fully presented) and 71.43% (multimodal proactive) cannot be read in absolute terms. The comparative findings survive, since the option set is constant across conditions.
Two transcription errors should also be corrected: the expert-review sample is "126 cases" in the main text but "the same 42 cases" in the appendix, and the Table 4 delta for MiniCPM-o-4.5 is printed as −15.24 where the table's own columns give 66.67 − 69.94 = −3.27.
Together, these limitations constrain the reach of the contribution rather than invalidate the environment design.
Questions for authors
-
Availability. Will the environment, case data, and evaluation code be released? The paper contains no availability statement of any kind.
-
State-of-the-art models. The seven configurations are deployment-oriented and no selection rationale is given. Could the authors add results for at least one frontier model, so the conclusions can be separated from this model class?
-
Option set and corrections. How many options does each case present, how were distractors constructed, and what accuracy does random selection achieve? Please also correct the expert-review sample size (126 in the main text, 42 in the appendix) and the MiniCPM-o-4.5 delta in Table 4 (−15.24 printed; −3.27 from the table's own columns).
4 comments
- Q2: The headline accuracy metric cannot be interpreted as reported. Diagnostic accuracy is closed-set exact match, yet the number of options per case, how distractors were constructed, and whether a random baseline exists are never reported, so the option set's floor is unknown and figures such as 90.07% (fully presented) and 71.43% (multimodal proactive) cannot be read in absolute terms. The comparative findings survive, since the option set is constant across conditions. Two transcription errors should also be corrected: the expert-review sample is "126 cases" in the main text but "the same 42 cases" in the appendix, and the Table 4 delta for MiniCPM-o-4.5 is printed as −15.24 where the table's own columns give 66.67 − 69.94 = −3.27. Together, these limitations constrain the reach of the contribution rather than invalidate the environment design.
- Q5: Option set and corrections. How many options does each case present, how were distractors constructed, and what accuracy does random selection achieve? Please also correct the expert-review sample size (126 in the main text, 42 in the appendix) and the MiniCPM-o-4.5 delta in Table 4 (−15.24 printed; −3.27 from the table's own columns).
Reply to Q2 and Q5: Thank you for identifying the missing evaluation details and transcription errors. We have clarified the option construction and scoring rule in Section 3.3 and Appendix D.1, added the random-choice reference, and corrected the sample-size wording and Table 4 delta.
-
Cases in the same department share the same options. Each department’s option set contains its distinct primary-diagnosis labels plus “None of the above”. The number of options varies across departments from 6 to 30, while all cases within the same department share the same option set. For each case, the recorded primary diagnosis is the reference answer, and the remaining options serve as distractors. The option set stays unchanged across experimental conditions. We use three option-order seeds (0, 1, and 2). For each case and seed, all compared methods receive the same options in the same order. Strict accuracy requires matching the recorded reference label. We normalize capitalization, punctuation, and whitespace before comparison. Strict accuracy requires selecting the case’s recorded diagnosis label. Related categories or alternative names are not accepted as substitutes. This ensures a unique answer under the scoring rule.
-
Both transcription errors have been corrected. In
Section 3.2, we now distinguish 42 distinct cases, 126 case-physician evaluations, and 1,008 individual criterion ratings, consistent withAppendix E.1. InTable 4, the MiniCPM-o-4.5 delta is corrected to −3.27 (calculated as 66.67 − 69.94), replacing the incorrectly written −15.24.
Dear Reviewer, as the discussion period is nearing its end, we would appreciate your feedback on our responses at your earliest convenience.
Dear Reviewer MA3g,
With less than 24 hours remaining in the discussion period, we would like to briefly summarize how we resolved your comments in our updated manuscript (highlighted in blue):
- Open-Source Release (
Reproducibility Statement, Page 11): Added an explicit commitment to publicly release the environment, case data, redistributable media, evaluation code, and OmniDoctorX implementation. - Omni-Model Coverage & Scope (
Section 5.1,Section 6): Clarified that our evaluation targets unified tri-modal (text+image+audio) models, including top-ranked open-source and frontier omni systems (Qwen3-Omni and Gemini-3.7-Flash), and scoped our conclusions accordingly. - Option Construction & Random Baseline (
Section 3.3,Appendix D.1): Specified the department-level option counts (6–30 options), distractor setup, strict exact-match rule, and random-choice baseline.
We would be very grateful for your confirmation on whether these revisions address your points before the discussion closes. Thank you for your time and support!
We sincerely thank the reviewer for your time, effort, and valuable feedback. We greatly appreciate the positive assessment of our work and the constructive comments that help us further improve the paper. We have carefully addressed all questions and updated the corresponding revisions into the new PDF, with key changes highlighted in blue text for clarity. Below, we respond point-by-point to each comment and provide additional explanations and experiments where necessary.
Reply to Q1, Q3 and Q4: We will indeed release our data and environment. Regarding your concern about the missing availability statement and model-selection rationale, we revised the
Reproducibility Statementsection in the manuscript to specify the planned release, clarified model selection inSection 5.1, and scoped the conclusions to the evaluated models inSection 6.Release commitment for reproducibility. We will publicly release the benchmark environment, case data and redistributable media, evaluation code, and OmniDoctorX implementation, together with configuration files and instructions for running the benchmark. Any third-party assets subject to redistribution restrictions will be identified with their source and preparation instructions. (see updated section
Reproducibility Statementin page 11)Model selection targets unified omni capabilities and includes competitive models. Our evaluation already includes the frontier omni models with demonstrated benchmark performance. OmniClinic requires agents to process text, images, and audio within the same diagnostic workflow, hence unified omni capabilities is the central to model selection (not only multimodal capabilities). Our evaluation includes leading open-source omni models, as supported by public benchmark rankings. For reference, on the Daily-Omni leaderboard [1], the evaluated Qwen3-Omni Thinking and Instruct variants rank 2nd and 3rd among 11 entries marked as open-source in the joint visual-and-audio setting, placing both within the top 30% as of November 15, 2026. Our evaluation also includes Gemini-3.7-Flash, whose high configuration ranks 17th among 421 entries, within the top 5% on the Arena leaderboard [2] dated November 11, 2026. Separately, the independent MISHAP-Bench [3] study explicitly identifies Gemini-3.7-Flash as a frontier model in audio-language evaluation. These sources support the inclusion of competitive proprietary and open-source models in our evaluation.
The conclusion explicitly identifies the scope of our findings. We have strengthened the wording in
Section 6by attributing the reported acquisition and spatial-coordination difficulties to the seven evaluated omni-model configurations. We also clarify that these findings concern the benchmark controlled environment and motivate further evaluation across additional model families and real-world clinical observations.