acceptodds
Under review as a conference paper at ICLR 2027

OmniClinic: Evaluating Active Evidence Acquisition in a Multimodal Clinical Sandbox

Abstract

Medical artificial intelligence holds great promise for improving clinical care, making rigorous and realistic evaluation essential for its safe deployment. However, existing benchmarks assess clinical capability mainly through text-only dialogue, overlooking the proactive multimodal evidence acquisition required in real-world diagnosis. To address this gap, we introduce OmniClinic, an interactive multimodal benchmark in which autonomous doctor agents acquire diagnostic evidence through clinical dialogue, diagnostic tests, and tool-mediated physical examinations. Agents operate within a finite clinical action space, selecting appropriate actions and specifying the parameters required for execution. We further introduce OmniDoctorX, which coordinates evidence-acquisition planning, spatial execution, and diagnostic review to support proactive clinical action. Across the evaluated omni-models, multimodal interaction improve diagnostic accuracy over text-only interaction. However, models struggle with evidence acquisition and spatial coordination, revealing a gap between multimodal understanding and autonomous interaction. OmniDoctorX improves mean proactive diagnostic accuracy from (86.36 vs. 77.84), demonstrating the value of coordinated acquisition and reasoning. Expert evaluation further supports the data quality of the benchmark, strengthening the reliability as a sandbox for autonomous multimodal diagnosis.

Primary area:
datasets and benchmarks
Keywords:
multimodal agents, active evidence acquisition, interactive clinical diagnosis, physical examination, clinical tool use, simulated clinical environments

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MA3gOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OMNICLINIC, an interactive multimodal environment for evaluating doctor agents on proactive clinical diagnosis. Diagnosis is cast as a POMDP in which an agent interacts with a patient for up to 15 turns through five action classes — clinical inquiry, physical examination, diagnostic testing, spatial configuration, and diagnosis submission — over 479 cases spanning 21 departments, with 4,251 generated images and 1,028 audio clips. Its distinguishing feature is that evidence is released only when the agent's examination matches a required route, including patient pose, clinician position, and body site, so that obtaining evidence requires proactive, spatially coordinated action rather than only choosing what to ask. Three metric families are reported (diagnostic accuracy by closed-set exact match, examination-parameter accuracy, and per-modality evidence-unlock rates), seven multimodal models are evaluated under fully presented evidence and under multimodal and text-only proactive interaction, and the authors additionally propose OMNIDOCTORX, an inference-time framework evaluated on two of the seven models.

Critical strengths

1. The environment introduces a genuinely new evaluation axis, and the work fills a real gap in the agentic-benchmark landscape.

Evidence is released only when the agent's examination matches a required route — patient pose, clinician position, and body site — which to my knowledge no prior clinical benchmark does; among the thirteen benchmarks compared in Table 1, OMNICLINIC is the only row marked ✓ for audio, physical-exam tool action, and pose/position interaction. This makes a class of failure observable that prior benchmarks structurally cannot measure, and it gives the work a clear position within the broader effort to scale up agentic environments. The paper is also clearly written and exceptionally well illustrated: Figure 1 gives an effective overview of the environment and its examination tool space, and Figure 6's color-gradient matrix makes the seven-model × seven-parameter decomposition easy to scan.

2. The synthetic media received independent physician review under a complete agreement-analysis protocol.

Three physicians rated all 42 sampled cases on eight criteria, giving 1,008 complete ratings, with mean ratings of 3.98–4.88 on a five-point scale and tight stratified-bootstrap intervals. The protocol combines Krippendorff's ordinal α, mean pairwise exact agreement, Gwet's AC2, permutation tests with Holm correction, and explicit sensitivity analyses — more complete than is typical in this literature. Just as importantly, the paper reports the unfavorable results alongside the favorable ones: negative agreement coefficients, significant rater effects on all eight criteria, and 67 critical free-text comments, while scoping its own claim to "structural integrity and operational consistency, not comprehensive clinical validity." Putting generated clinical media in front of a physician panel and then reporting the disagreements is a practice I would like to see become standard.

Critical weaknesses

1. The benchmark is not available, and its model coverage cannot support the conclusions drawn from it.

The paper contains no statement, not even a promise of release, about the data, code, or environment; the Reproducibility Statement describes documentation only, and the sole availability sentence is restrictive. As submitted, the primary deliverable of a benchmark paper can neither be used nor reproduced.

Coverage compounds this. The seven configurations are deployment-oriented rather than state-of-the-art — Gemini-3.7-Flash, Qwen3-Omni-30B-A3B (Instruct and Thinking), MGM-Omni-32B, MiniCPM-o-4.5, Phi-4-Multimodal-Instruct, and Baichuan-Omni-1.5 — with no frontier model of any kind and no stated selection rationale. Since the paper's conclusion is a claim about how current multimodal models behave, this coverage cannot separate the reported behavior from this particular model class.

2. The headline accuracy metric cannot be interpreted as reported.

Diagnostic accuracy is closed-set exact match, yet the number of options per case, how distractors were constructed, and whether a random baseline exists are never reported, so the option set's floor is unknown and figures such as 90.07% (fully presented) and 71.43% (multimodal proactive) cannot be read in absolute terms. The comparative findings survive, since the option set is constant across conditions.

Two transcription errors should also be corrected: the expert-review sample is "126 cases" in the main text but "the same 42 cases" in the appendix, and the Table 4 delta for MiniCPM-o-4.5 is printed as −15.24 where the table's own columns give 66.67 − 69.94 = −3.27.

Together, these limitations constrain the reach of the contribution rather than invalidate the environment design.

Questions for authors
  1. Availability. Will the environment, case data, and evaluation code be released? The paper contains no availability statement of any kind.

  2. State-of-the-art models. The seven configurations are deployment-oriented and no selection rationale is given. Could the authors add results for at least one frontier model, so the conclusions can be separated from this model class?

  3. Option set and corrections. How many options does each case present, how were distractors constructed, and what accuracy does random selection achieve? Please also correct the expert-review sample size (126 in the main text, 42 in the appendix) and the MiniCPM-o-4.5 delta in Table 4 (−15.24 printed; −3.27 from the table's own columns).

4 comments

We sincerely thank the reviewer for your time, effort, and valuable feedback. We greatly appreciate the positive assessment of our work and the constructive comments that help us further improve the paper. We have carefully addressed all questions and updated the corresponding revisions into the new PDF, with key changes highlighted in blue text for clarity. Below, we respond point-by-point to each comment and provide additional explanations and experiments where necessary.

  • Q1: The benchmark is not available, and its model coverage cannot support the conclusions drawn from it. The paper contains no statement, not even a promise of release, about the data, code, or environment; the Reproducibility Statement describes documentation only, and the sole availability sentence is restrictive. As submitted, the primary deliverable of a benchmark paper can neither be used nor reproduced. Coverage compounds this. The seven configurations are deployment-oriented rather than state-of-the-art — Gemini-3.7-Flash, Qwen3-Omni-30B-A3B (Instruct and Thinking), MGM-Omni-32B, MiniCPM-o-4.5, Phi-4-Multimodal-Instruct, and Baichuan-Omni-1.5 — with no frontier model of any kind and no stated selection rationale. Since the paper's conclusion is a claim about how current multimodal models behave, this coverage cannot separate the reported behavior from this particular model class.
  • Q3: Availability. Will the environment, case data, and evaluation code be released? The paper contains no availability statement of any kind.
  • Q4: State-of-the-art models. The seven configurations are deployment-oriented and no selection rationale is given. Could the authors add results for at least one frontier model, so the conclusions can be separated from this model class?

Reply to Q1, Q3 and Q4: We will indeed release our data and environment. Regarding your concern about the missing availability statement and model-selection rationale, we revised the Reproducibility Statement section in the manuscript to specify the planned release, clarified model selection in Section 5.1, and scoped the conclusions to the evaluated models in Section 6.

  1. Release commitment for reproducibility. We will publicly release the benchmark environment, case data and redistributable media, evaluation code, and OmniDoctorX implementation, together with configuration files and instructions for running the benchmark. Any third-party assets subject to redistribution restrictions will be identified with their source and preparation instructions. (see updated section Reproducibility Statement in page 11)

  2. Model selection targets unified omni capabilities and includes competitive models. Our evaluation already includes the frontier omni models with demonstrated benchmark performance. OmniClinic requires agents to process text, images, and audio within the same diagnostic workflow, hence unified omni capabilities is the central to model selection (not only multimodal capabilities). Our evaluation includes leading open-source omni models, as supported by public benchmark rankings. For reference, on the Daily-Omni leaderboard [1], the evaluated Qwen3-Omni Thinking and Instruct variants rank 2nd and 3rd among 11 entries marked as open-source in the joint visual-and-audio setting, placing both within the top 30% as of November 15, 2026. Our evaluation also includes Gemini-3.7-Flash, whose high configuration ranks 17th among 421 entries, within the top 5% on the Arena leaderboard [2] dated November 11, 2026. Separately, the independent MISHAP-Bench [3] study explicitly identifies Gemini-3.7-Flash as a frontier model in audio-language evaluation. These sources support the inclusion of competitive proprietary and open-source models in our evaluation.

  3. The conclusion explicitly identifies the scope of our findings. We have strengthened the wording in Section 6 by attributing the reported acquisition and spatial-coordination difficulties to the seven evaluated omni-model configurations. We also clarify that these findings concern the benchmark controlled environment and motivate further evaluation across additional model families and real-world clinical observations.

[1] Daily-Omni leaderboard (Zhou et al. Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities)

[2] Arena leaderboard.

[3] Soi et al. MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models.

  • Q2: The headline accuracy metric cannot be interpreted as reported. Diagnostic accuracy is closed-set exact match, yet the number of options per case, how distractors were constructed, and whether a random baseline exists are never reported, so the option set's floor is unknown and figures such as 90.07% (fully presented) and 71.43% (multimodal proactive) cannot be read in absolute terms. The comparative findings survive, since the option set is constant across conditions. Two transcription errors should also be corrected: the expert-review sample is "126 cases" in the main text but "the same 42 cases" in the appendix, and the Table 4 delta for MiniCPM-o-4.5 is printed as −15.24 where the table's own columns give 66.67 − 69.94 = −3.27. Together, these limitations constrain the reach of the contribution rather than invalidate the environment design.
  • Q5: Option set and corrections. How many options does each case present, how were distractors constructed, and what accuracy does random selection achieve? Please also correct the expert-review sample size (126 in the main text, 42 in the appendix) and the MiniCPM-o-4.5 delta in Table 4 (−15.24 printed; −3.27 from the table's own columns).

Reply to Q2 and Q5: Thank you for identifying the missing evaluation details and transcription errors. We have clarified the option construction and scoring rule in Section 3.3 and Appendix D.1, added the random-choice reference, and corrected the sample-size wording and Table 4 delta.

  1. Cases in the same department share the same options. Each department’s option set contains its distinct primary-diagnosis labels plus “None of the above”. The number of options varies across departments from 6 to 30, while all cases within the same department share the same option set. For each case, the recorded primary diagnosis is the reference answer, and the remaining options serve as distractors. The option set stays unchanged across experimental conditions. We use three option-order seeds (0, 1, and 2). For each case and seed, all compared methods receive the same options in the same order. Strict accuracy requires matching the recorded reference label. We normalize capitalization, punctuation, and whitespace before comparison. Strict accuracy requires selecting the case’s recorded diagnosis label. Related categories or alternative names are not accepted as substitutes. This ensures a unique answer under the scoring rule.

  2. Both transcription errors have been corrected. In Section 3.2, we now distinguish 42 distinct cases, 126 case-physician evaluations, and 1,008 individual criterion ratings, consistent with Appendix E.1. In Table 4, the MiniCPM-o-4.5 delta is corrected to −3.27 (calculated as 66.67 − 69.94), replacing the incorrectly written −15.24.

Dear Reviewer, as the discussion period is nearing its end, we would appreciate your feedback on our responses at your earliest convenience.

Dear Reviewer MA3g,

With less than 24 hours remaining in the discussion period, we would like to briefly summarize how we resolved your comments in our updated manuscript (highlighted in blue):

  • Open-Source Release (Reproducibility Statement, Page 11): Added an explicit commitment to publicly release the environment, case data, redistributable media, evaluation code, and OmniDoctorX implementation.
  • Omni-Model Coverage & Scope (Section 5.1, Section 6): Clarified that our evaluation targets unified tri-modal (text+image+audio) models, including top-ranked open-source and frontier omni systems (Qwen3-Omni and Gemini-3.7-Flash), and scoped our conclusions accordingly.
  • Option Construction & Random Baseline (Section 3.3, Appendix D.1): Specified the department-level option counts (6–30 options), distractor setup, strict exact-match rule, and random-choice baseline.

We would be very grateful for your confirmation on whether these revisions address your points before the discussion closes. Thank you for your time and support!

Reviewer_xnGDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces OmniClinic, an interactive multimodal clinical benchmark that evaluates autonomous diagnostic agents through proactive evidence acquisition, physical examinations, and clinical reasoning. It also proposes OmniDoctorX, an agentic framework that improves diagnostic accuracy by coordinating evidence acquisition, spatial execution, and diagnostic review.

Critical strengths
  1. OmniClinic moves beyond conventional text-based diagnosis by requiring agents to actively acquire multimodal evidence through clinical actions. The inclusion of tool selection, anatomical targeting, and spatial configuration provides a more comprehensive evaluation of clinical agents.

  2. The benchmark includes 479 cases across 21 clinical departments, with 4,251 images and 1,028 audio recordings. Its structured evidence-generation pipeline and physician-based quality assessment strengthen the dataset's practical value.

  3. Experiments across seven multimodal model configurations reveal important limitations in proactive evidence acquisition and spatial coordination. OmniDoctorX improves average diagnostic accuracy from 77.84% to 86.36%, demonstrating the potential of coordinated agentic workflows.

Critical weaknesses
  1. The benchmark relies entirely on synthetic patient cases and generated multimodal evidence, which may not capture the complexity of real clinical encounters. Furthermore, physical interactions are implemented through predefined rules rather than realistic embodied execution.

  2. Although physician evaluations report relatively high quality scores, inter-rater agreement is weak, with Krippendorff's alpha ranging from -0.220 to 0.026. The evaluation also covers only 42 distinct cases, raising concerns about the reliability and representativeness of the quality assessment.

  3. OmniDoctorX combines acquisition planning, spatial execution, and independent diagnostic review without sufficiently isolating their individual contributions. Its additional inference calls also raise questions about whether improvements result from better coordination or increased computational resources.

Questions for authors
  1. How well does OmniClinic generalize to real clinical environments? Have the authors considered evaluating agents on real-world clinical observations or physician-annotated cases to assess the realism of synthetic evidence?

  2. How can the reliability of expert validation be improved? Given the low inter-rater agreement, could the authors provide additional analyses or expand the physician evaluation to establish more robust clinical quality assessments?

  3. What are the individual contributions of OmniDoctorX's components? Could the authors provide component-wise ablations for acquisition planning, spatial execution, and diagnostic review under matched inference budgets?

  4. Does increased evidence acquisition necessarily improve diagnostic reasoning? Since the current unlock metrics only measure whether at least one media item is acquired, could the authors evaluate evidence completeness, clinical relevance, and the relationship between acquisition quality and diagnostic accuracy?

7 comments

We are grateful to the reviewer for your thorough evaluation and encouraging remarks. We truly appreciate the insightful questions raised. We have carefully addressed all questions and updated the corresponding revisions into the new PDF, with key changes highlighted in blue text for clarity. In the following, we respond point-by-point to each comment and provide additional explanations and experiments where necessary.

  • Q1: The benchmark relies entirely on synthetic patient cases and generated multimodal evidence, which may not capture the complexity of real clinical encounters. Furthermore, physical interactions are implemented through predefined rules rather than realistic embodied execution.
  • Q4: How well does OmniClinic generalize to real clinical environments? Have the authors considered evaluating agents on real-world clinical observations or physician-annotated cases to assess the realism of synthetic evidence?

Reply to Q1 and Q4: OmniClinic combines structured patient generation, physician assessment, and controlled interaction rules to support reproducible evaluation of clinical evidence acquisition. Hence, the quality of the synthetic dataset is supported by both systematic checks and clinical expert assessment, providing a credible basis for evaluation within the simulation.

  1. Structured generation supports diverse and clinically coherent cases. Our cases build on Patient-Zero [1], which uses Stratified Sampling with Constraints to generate patient profiles aligned with selected epidemiological priors and clinical constraints. This supports variation in demographics and disease presentation. OmniClinic covers 21 departments, 44 nationalities, and eight ethnicity categories, with varied ages, sexes, and clinical conditions.

  2. Physician assessment provides direct evidence of media quality. Three physicians independently evaluated the cases, where mean quality scores were 4.25/5 for images and 4.38/5 for audio. These assessments support the quality of the sampled media across finding fidelity, perceptual clarity, case consistency, and media integrity. We have added a detailed evaluation scope description in Section 3.2.

  3. An extension with embodied execution is under development. In the current version, an agent must select the appropriate examination tool, body region, patient pose, and doctor position to obtain the corresponding evidence. This lets us test whether agents know which examination to perform and how to configure it. We are developing a follow-up version with embodied execution to evaluate physical examination skills more directly, and have added this direction to the Conclusion section as our future extension.

[1] Patient-Zero: Scaling Synthetic Patient Agents to Real-World Distributions without Real Patient Data


  • Q2: Although physician evaluations report relatively high quality scores, inter-rater agreement is weak. The evaluation also covers only 42 distinct cases, raising concerns about the reliability and representativeness of the quality assessment.
  • Q5: How can the reliability of expert validation be improved? Given the low inter-rater agreement, could the authors provide additional analyses or expand the physician evaluation to establish more robust clinical quality assessments?

Reply to Q2 and Q5: Although α\alpha shows low agreement, the physician assessments support the overall quality of the sampled media, with particularly strong agreement on media integrity under Gwet’s AC2. All three physicians gave favorable mean image sign-fidelity scores, while differing in their scoring strictness. These differences help explain why high mean quality scores coexist with low agreement. We have revised Section 3.2 and Appendix E.3 to clarify the rating differences, sampling design, and scope of expert validation.

  1. Experienced physicians from leading hospitals provide expert quality assessment. Three senior physicians from leading top-tier hospitals independently evaluated the sampled media, drawing on extensive clinical experience. All three gave favorable mean image sign-fidelity scores of 3.79, 3.55, and 4.62 out of 5, supporting the overall quality of the reviewed images. Their independent ratings and detailed clinical feedback provide complementary assessments of media quality and identify specific opportunities for refinement. We have clarified the physician-level rating summaries in Appendix E.3.

  2. Complementary measures show which quality dimensions have stronger agreement. Gwet’s AC2 was 0.867 for image integrity and 0.937 for audio integrity, indicating stronger agreement on the absence of artifacts and irrelevant content. Agreement on finding fidelity was lower, with AC2 values of 0.389 for images and 0.507 for audio. We distinguish these dimensions explicitly and retain α\alpha, exact agreement, and case-bootstrap confidence intervals in Appendix E.3.

Continued Reply to Q2 and Q5:

  1. Substantial expert assessment across clinical departments. Our study involves a substantial annotation effort. Three senior physicians independently assessed 42 distinct cases spanning 21 clinical departments, with two cases per department. Each physician rated every case on eight image and audio quality criteria, yielding 126 case-physician evaluations and 1,008 individual criterion ratings. This design combines departmental coverage with multiple expert assessments of each case. For reference, published medical studies provide examples of expert evaluations at a comparable case scale, including 40 emergency department cases assessed by two independent radiologists in the Journal of the American College of Radiology [1] and a 50-case stratified audit by three oncologists in npj Precision Oncology [2]. These studies provide context for the scale of our expert evaluation.

[1] Barash, Y., Klang, E., Konen, E., & Sorin, V. (2023). ChatGPT-4 Assistance in Optimizing Emergency Department Radiology Referrals and Imaging Selection. Journal of the American College of Radiology : JACR, 20(10), 998–1003.

[2] Zou, Y., Rafiee Sevyeri, L., Farahnak, F. et al. Prognostic data extraction harnessing a privacy-preserving large language model: a clinician-AI collaborative retrospective evaluation in head and neck oncology. npj Precis. Onc. 10, 367 (2026).


  • Q3: OmniDoctorX combines acquisition planning, spatial execution, and independent diagnostic review without sufficiently isolating their individual contributions. Its additional inference calls also raise questions about whether improvements result from better coordination or increased computational resources.
  • Q6: What are the individual contributions of OmniDoctorX's components? Could the authors provide component-wise ablations for acquisition planning, spatial execution, and diagnostic review under matched inference budgets?

Reply to Q3 and Q6: To address your concern regarding the contributions of OmniDoctorX’s components, we conducted a new ablation experiments with Qwen3-Omni-Instruct. Each variant adds the named component to the baseline, while Full OmniDoctorX combines all components. We have added a new Section 5.3.2 to present these results and analyze their effects on diagnostic accuracy and media acquisition. Full OmniDoctorX achieves the highest diagnostic accuracy (85.25%), image-unlock rate (30.91%), and audio-unlock rate (32.68%) among the evaluated settings. These results support addressing evidence selection, execution, and interpretation together: an examination must first be selected, then successfully performed, and finally incorporated into the diagnostic decision.

Table A: Single-component ablations of OmniDoctorX with Qwen3-Omni-Instruct.

SettingDiagnostic accuracyImage unlock rateAudio unlock rate
Baseline72.65%8.91%28.38%
+ Candidate planning78.27%22.43%29.45%
+ Modality rewards73.64%9.94%29.01%
+ Spatial compilation77.06%19.45%30.35%
+ Repeated-action prevention73.71%10.97%29.03%
+ Independent diagnostic review75.64%9.18%28.46%
Full OmniDoctorX85.25%30.91%32.68%
  1. Evidence selection and execution address different acquisition bottlenecks. Candidate planning and spatial compilation yield the largest individual accuracy gains, at 5.62% and 4.41% respectively. Candidate planning broadens the actions considered before committing to an examination, whereas spatial compilation supplies the positioning steps required to execute a selected examination. Spatial compilation increases image unlocking from 8.91% to 19.45% and audio unlocking from 28.38% to 30.35%. This pattern is consistent with an execution bottleneck: selecting an appropriate examination is useful only when the agent can satisfy the conditions needed to obtain its evidence. The larger change in image unlocking identifies visual evidence access as particularly responsive to spatial compilation in this evaluation.

  2. Acquiring evidence and using it effectively are distinct capabilities. Unlock rates count whether any media item was obtained, treating a decisive finding and a redundant observation equally. They therefore measure access rather than diagnostic value. Diagnostic review addresses how acquired findings support or contradict competing diagnoses. Adding review increases accuracy from 72.65% to 75.64%, while image and audio unlock rates change by only 0.27% and 0.08%. This pattern suggests that diagnostic progress can involve better use of evidence, beyond more frequent media access.

Continued Reply to Q3 and Q6:

  1. Candidate planning improves efficiency without increasing computation. Under the same experimental protocol, adding candidate planning reduces mean model calls from 23.74 to 19.99 (-15.8%) and mean token consumption from 93,842 to 73,517 (-21.7%), while maintaining similar observed diagnostic accuracy (68.27 vs. 68.06). This demonstrates that the planning component can use the available inference budget more efficiently, rather than relying on additional calls.

Table B: Mean model-call and token consumption per episode in the single-component experiments.

SettingMean model calls per episodeMean total tokens per episode
Baseline23.7493,842
+ Candidate planning19.9973,517
+ Modality rewards23.2392,082
+ Spatial compilation23.7293,782
+ Repeated-action prevention24.0074,792
+ Independent diagnostic review23.7494,035

All settings in this additional experiment are compared against the baseline evaluated under the same protocol. Tokens include input and output tokens. Diagnostic review uses the same acquired evidence and one final diagnostic call, with reused acquisition costs included in the reported episode totals. Candidate planning achieves similar observed diagnostic accuracy with 15.8% fewer model calls and 21.7% fewer tokens, demonstrating an efficiency benefit without additional computation. We have added a new section Section 5.3.2 and Appendix D.4 to report the resource accounting and the component-level efficiency analysis.


  • Q7: Does increased evidence acquisition necessarily improve diagnostic reasoning? Since the current unlock metrics only measure whether at least one media item is acquired, could the authors evaluate evidence completeness, clinical relevance, and the relationship between acquisition quality and diagnostic accuracy?

Reply to Q7: As mention in [Reply to Q2 and Q5], acquiring more evidence helps when the findings are relevant and correctly interpreted. We therefore examine diagnostic accuracy and the evidence obtained, alongside media-unlock rates.

  1. Multimodal evidence improves diagnostic performance. Table 4 in Section 5.2 shows that adding image and audio input increases proactive accuracy from 71.43% to 83.03% for Gemini-3.7-Flash and from 65.83% to 72.65% for Qwen3-Omni-Instruct. These comparisons support the diagnostic value of multimodal information alongside textual evidence.

  2. Relevant evidence matters more than repeated attempts. The matched-case comparison in Section 5.3.3 shows that the baseline repeatedly attempts unsuccessful auscultation and reaches an incorrect diagnosis. OmniDoctorX obtains images, transplant history, and an autoantibody report before correctly identifying chronic graft-versus-host disease. This example illustrates how complementary findings support diagnosis when repeated attempts provide no new evidence.

  3. Acquisition and diagnosis should be evaluated together. We clarify in Section 3.3 that unlock rates measure whether media were obtained. The analysis in Section 5.3.2 considers these rates alongside diagnostic accuracy to distinguish accessing evidence from using it effectively. These results support the usefulness of relevant evidence, while completeness and clinical relevance are not directly quantified by unlock rates.

Reviewer_S9NWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniClinic is a multimodal clinical sandbox for evaluating proactive evidence acquisition. It contains 479 synthetic cases spanning 21 clinical departments and 295 diagnostic labels. Agents can acquire evidence through history taking, physical examinations, auxiliary tests, and spatial adjustments. It must submit a diagnosis within a maximum of 15 decision turns, including the step of diagnosis submission.The authors evaluate seven model configurations under three conditions: fully presented evidence, proactive multimodal interaction, and proactive text-only interaction.They also introduce OmniDoctorX. It combines acquisition planning, spatial action compilation, prevention of repeated actions, and independent diagnostic review. Across the two backbone models used to evaluate the method, mean proactive diagnostic accuracy increases from 77.84% to 86.36%.

Critical strengths
  1. The evaluation examines both diagnostic outcomes and the evidence acquisition process. Comparisons between fully presented evidence and proactive acquisition help separate difficulties in obtaining information from difficulties in using it. Metrics for modality unlocking, spatial actions, and retries after failure also help clarify this distinction. Rules that charge for repeated actions and count failure to submit a diagnosis as incorrect also make the handling of failures explicit.
  2. The protocol and scope of the evaluation are described in sufficient detail. Appendices A, D, and F provide the interfaces, metrics, and prompts, while Appendix E reports expert ratings and negative feedback. The authors also explain the purposes and limitations of route matching, modality ablation, and the Unity demonstration prototype, allowing readers to assess what the experiments actually test.
Critical weaknesses
  1. The diagnostic implications of the media defects identified by experts remain unclear. In Table 12 on page 25, the mean image and audio sign-fidelity scores, I1 and A1, are 3.98 and 4.13. Their ordinal α values are −0.144 and −0.178, respectively. These values indicate low inter-rater agreement under this measure. Negative α values alone do not establish that the media are invalid. The specific observations in Appendix E.3 on page 26 are more relevant, including the finding that bleeding was not visible. Restricted eye movements could not be assessed from still images, and related sounds were difficult to distinguish. Are these findings essential to diagnosing the corresponding cases? Please use the existing evaluation to explain their diagnostic impact within the stated validation scope.
  2. The key components lack separate ablation studies. Figure 4 supports the effectiveness of the complete OmniDoctorX system, but Appendix C introduces candidate planning, modality rewards, spatial compilation, prevention of repeated actions, and independent diagnostic review together, making their individual contributions difficult to determine. The modality ablation in Table 10 does not answer this question. Targeted ablations should report diagnostic accuracy and evidence-unlocking rates for spatial compilation, and compare diagnostic accuracy on the same acquired evidence for diagnostic review.
Questions for authors
  1. According to Appendix B.2 on page 18, the released audio contains only cardiac sounds. However, Figure 5 and the accompanying paragraph on page 9 describe obtaining audio through abdominal auscultation. lease briefly reconcile these descriptions and clarify the actual scope of the audio data.
  2. Lines 247–249 on page 5 refer to“126 cases,” whereas Appendix E.1, lines 1315–1317 on page 25, specifies 42 cases rated by 3 physicians, yielding 126 evaluations. Please use consistent wording to distinguish the number of cases from the number of evaluations. This is a wording correction only and does not affect the rating.
  3. Figure 4 compares the complete method, while Table 10 examines the available modalities. Please add ablations of spatial compilation and independent diagnostic review, keeping the other components and inference budgets comparable. For spatial compilation, please report diagnostic accuracy and evidence-unlocking rates. For diagnostic review, please compare diagnostic accuracy using the same acquired evidence. Appendix C.3 specifies a 45-call allowance for OmniDoctorX. Please clarify the baseline's allowance and report actual model-call counts and token consumption for both systems.
  4. Please specify the number of candidate diagnoses per case and how distractors are obtained. If one diagnostic option is a broader category that includes another, such as a disease category and its specific subtype, or if options use synonymous labels, how is a unique correct answer determined? A brief description of the procedure used in the current experiments would suffice.
  5. Page 8 reports 10,054 eligible episodes, which is 5 fewer than 479×3×7. Were all five exclusions due to the infrastructure failures described in Appendix D.1? Were the same exclusion rules applied to all models? Please also provide case-paired confidence intervals for the main performance differences in Figure 4 and specify the resampling or pairing unit.
6 comments

We are grateful to the reviewer for your thorough evaluation and encouraging remarks. We truly appreciate the insightful questions raised. We have carefully addressed all questions and updated the corresponding revisions into the new PDF, with key changes highlighted in blue text for clarity. In the following, we respond point-by-point to each comment and provide additional explanations and experiments where necessary.

  • Q1: The diagnostic implications of the media defects identified by experts remain unclear. In Table 12 on page 25, the mean image and audio sign-fidelity scores, I1 and A1, are 3.98 and 4.13, respectively, whereas their ordinal α values are −0.144 and −0.178, indicating low inter-rater agreement under this measure. Negative α values alone do not establish that the media are invalid. Appendix E also reports complementary agreement measures and explicitly limits the scope of validation. More relevant are the specific observations in Appendix E.3 on page 26: bleeding was not visible, restricted eye movements could not be assessed from still images, and related sounds were difficult to distinguish. Are these findings essential to diagnosing the corresponding cases? Please use the existing evaluation to explain their diagnostic impact within the stated validation scope.

Reply to Q1: Based on our analysis of the physician comments, only 2.13% of the 422 reviewed media items were flagged for quality issues, comprising 6 images (1.42%) and 3 audio recordings (0.71%). We examine each finding in this broader diagnostic context, including cases where an apparent missing abnormality reflects an omitted negation in the displayed label. Our expert-control pipeline revises and regenerates candidates that fail screening and excludes unresolved candidates.

  1. Diagnostic relevance of the specific examples. We checked the highlighted findings against their original case records. These case-level explanations (Case Study I) show that additional diagnostic evidence remains available through patient history, symptoms, and test reports.

Case Study I

Reported issueDiagnostic implication
Bleeding was not visible in the Dravet syndrome caseVisible bleeding was not expected. The seizure history and available genetic report remain informative for diagnosis.
Restricted eye movements could not be assessed from a still imageThe limitation is confined to movement assessment. Exposure history and associated symptoms remain available to support diagnosis.
Stridor and clear breath sounds sounded similarThe similarity is consistent with the absence of stridor. Other clinical symptoms, such as Jaw pain, facial swelling, and difficulty opening the mouth, remain available to support diagnosis.
  1. Quality control. Our pipeline combines automated checks, model-assisted screening, regeneration, and physician assessment. Candidates that fail screening are revised or regenerated, and unresolved candidates are excluded. Physician feedback provides an additional assessment of the sampled media.

  2. Evidence for complementary diagnostic information. To address whether multimodal evidence remains useful despite the reported media issues, Table 4 shows higher proactive diagnostic accuracy with multimodal input than with text-only input. For instance, accuracy increases from 71.43% to 83.03% for Gemini-3.7-Flash (+11.60%) and from 65.83% to 72.65% for Qwen3-Omni-Instruct (+6.82%). These results support the overall diagnostic value of multimodal evidence, which provides useful information alongside patient history and textual findings.

  • Q2: The key components lack separate ablation studies. Figure 4 supports the effectiveness of the complete OmniDoctorX system, but Appendix C introduces candidate planning, modality rewards, spatial compilation, prevention of repeated actions, and independent diagnostic review together, making their individual contributions difficult to determine. The modality ablation in Table 10 does not answer this question. Targeted ablations should report diagnostic accuracy and evidence-unlocking rates for spatial compilation, and compare diagnostic accuracy on the same acquired evidence for diagnostic review.

Reply to Q2: To address your concern regarding the contributions of OmniDoctorX’s components, we conducted a new ablation experiments with Qwen3-Omni-Instruct. Each variant adds the named component to the baseline, while Full OmniDoctorX combines all components. We have added a new Section 5.3.2 to present these results and analyze their effects on diagnostic accuracy and media acquisition. Full OmniDoctorX achieves the highest diagnostic accuracy (85.25%), image-unlock rate (30.91%), and audio-unlock rate (32.68%) among the evaluated settings. These results support addressing evidence selection, execution, and interpretation together: an examination must first be selected, then successfully performed, and finally incorporated into the diagnostic decision.

Table A: Single-component ablations of OmniDoctorX with Qwen3-Omni-Instruct.

SettingDiagnostic accuracyImage unlock rateAudio unlock rate
Baseline72.65%8.91%28.38%
+ Candidate planning78.27%22.43%29.45%
+ Modality rewards73.64%9.94%29.01%
+ Spatial compilation77.06%19.45%30.35%
+ Repeated-action prevention73.71%10.97%29.03%
+ Independent diagnostic review75.64%9.18%28.46%
Full OmniDoctorX85.25%30.91%32.68%
  1. Evidence selection and execution address different acquisition bottlenecks. Candidate planning and spatial compilation yield the largest individual accuracy gains, at 5.62% and 4.41% respectively. Candidate planning broadens the actions considered before committing to an examination, whereas spatial compilation supplies the positioning steps required to execute a selected examination. Spatial compilation increases image unlocking from 8.91% to 19.45% and audio unlocking from 28.38% to 30.35%. This pattern is consistent with an execution bottleneck: selecting an appropriate examination is useful only when the agent can satisfy the conditions needed to obtain its evidence. The larger change in image unlocking identifies visual evidence access as particularly responsive to spatial compilation in this evaluation.

  2. Acquiring evidence and using it effectively are distinct capabilities. Unlock rates count whether any media item was obtained, treating a decisive finding and a redundant observation equally. They therefore measure access rather than diagnostic value. Diagnostic review addresses how acquired findings support or contradict competing diagnoses. Adding review increases accuracy from 72.65% to 75.64%, while image and audio unlock rates change by only 0.27% and 0.08%. This pattern suggests that diagnostic progress can involve better use of evidence, beyond more frequent media access.


  • Q3: Appendix B.2 on page 18 states that the released audio contains only cardiac sounds, whereas Figure 5 and the accompanying paragraph on page 9 describe obtaining audio through abdominal auscultation. Please briefly reconcile these descriptions and clarify the actual scope of the audio data.

Reply to Q3: Thank you for identifying this inconsistency. Our audio collection includes cardiac sounds and rhythms, respiratory sounds, and bowel sounds. The previous cardiac-only description was incomplete. We have revised the description in Appendix B.2 to clarify this scope, including bowel sounds obtained through abdominal auscultation.


  • Q4: Lines 247–249 on page 5 refer to “126 cases,” whereas Appendix E.1, lines 1315–1317 on page 25, specifies 42 cases rated by 3 physicians, yielding 126 evaluations. Please use consistent wording to distinguish the number of cases from the number of evaluations. This is a wording correction only and does not affect the rating.

Reply to Q4: Thank you for pointing out this wording error. We have revised Section 3.2 and Appendix E.1 to consistently state that 42 distinct cases were each assessed by three physicians, yielding 126 case–physician evaluations. This wording correction does not change the evaluation sample, scores, or agreement statistics.

  • Q5: Figure 4 compares the complete method, while Table 10 examines the available modalities. Please add ablations of spatial compilation and independent diagnostic review, keeping the other components and inference budgets comparable. For spatial compilation, report diagnostic accuracy and evidence-unlocking rates. For diagnostic review, compare diagnostic accuracy on the same acquired evidence. Appendix C.3 specifies a 45-call allowance for OmniDoctorX. Please clarify the baseline's allowance and report actual model-call counts and token consumption for both systems.

Reply to Q5: More details refer to [Reply to Q2] and Table A for the component-ablation results and analysis. OmniDoctorX operates within the same model-call allowance as the baseline. Both systems share the same 45-call allowance in the evaluation reported below. This allowance includes format repairs. OmniDoctorX allocates it across acquisition, diagnostic review, and repairs, so review receives no additional call allowance. These measurements show actual resource consumption under the shared call ceiling. They apply to the evaluation in Table C below.

Table C: Actual inference cost for Qwen3-Omni-Instruct in a separate completed evaluation.

SettingDiagnostic accuracyMean model callsMean input tokensMean output tokens
Baseline72.65%7.1733,255892
Full OmniDoctorX82.25%8.7747,3724,569

  • Q6: Please specify the number of candidate diagnoses per case and how distractors are obtained. If one diagnostic option is a broader category that includes another, such as a disease category and its specific subtype, or if options use synonymous labels, how is a unique correct answer determined? A brief description of the procedure used in the current experiments would suffice.

Reply to Q6: We clarify the option construction and scoring procedure below. We have revised Appendix D.1 to clarify the option counts, distractor construction, and scoring rules for determining the unique reference answer.

  1. Cases in the same department share the same options. Each department’s option set contains its distinct primary-diagnosis labels plus “None of the above”. The number of options varies across departments from 6 to 30, while all cases within the same department share the same option set. For each case, the recorded primary diagnosis is the reference answer, and the remaining options serve as distractors. The option set stays unchanged across experimental conditions. We use three option-order seeds (0, 1, and 2). For each case and seed, all compared methods receive the same options in the same order.

  2. Strict accuracy requires matching the recorded reference label. We normalize capitalization, punctuation, and whitespace before comparison. Strict accuracy requires selecting the case’s recorded diagnosis label. Related categories or alternative names are not accepted as substitutes. This ensures a unique answer under the scoring rule.


  • Q7: Page 8 reports 10,054 eligible episodes, which is 5 fewer than 479×3×7. Were all five exclusions due to the infrastructure failures described in Appendix D.1, and were the same exclusion rules applied to all models? Please also provide case-paired confidence intervals for the main performance differences in Figure 4 and specify the resampling or pairing unit.

Reply to Q7: Yes. Of the 10,059 planned episodes (479 cases × 3 seeds × 7 models), five Gemini episodes were excluded because of timeout or connection failures, leaving 10,054 eligible episodes. The same exclusion rule applies to all models. We have revised Section 5.2 and Appendix D.1 to update the following results. For the paired analysis, outcomes are matched by case and option-order seed. The bootstrap procedure resamples cases 10,000 times, retaining both methods and all available paired seeds within each sampled case. Each resample recomputes the equally weighted mean difference across seeds. The 2.5th and 97.5th percentiles define the 95% confidence interval as shown in Table C below.

Table C: Diagnostic accuracy and media unlock rates, with paired differences and 95% confidence intervals.

ModelMetricBaselineOmniDoctorXDifference [95% CI]
Gemini-3.7-FlashDiagnostic accuracy82.74%86.36%+3.62 [+1.47, +4.42]
Gemini-3.7-FlashImage unlock rate25.07%69.18%+44.11 [+40.63, +49.89]
Gemini-3.7-FlashAudio unlock rate3.31%51.90%+48.59 [+43.71, +53.44]
Qwen3-Omni-InstructDiagnostic accuracy72.65%85.25%+12.60 [+7.71, +17.22]
Qwen3-Omni-InstructImage unlock rate8.91%30.91%+22.00 [+14.57, +24.39]
Qwen3-Omni-InstructAudio unlock rate28.38%32.68%+4.30 [+0.78, +7.96]

Comments on the paper

1 comment

We sincerely thank all reviewers for their thorough evaluation, constructive feedback, and encouraging remarks. We greatly appreciate the recognition of our work's contribution and the insightful questions raised, which have led to several key improvements to the paper. We provide this consolidated summary to highlight how we have effectively resolved all major concerns across the review period. We appreciate that multiple reviewers (S9NW and xnGD) have explicitly confirmed their decision to raise their scores following our detailed responses and new experiments. All revisions have been incorporated into the updated manuscript, with key changes highlighted in blue text for clarity.

We are pleased to know that:

  • Reviewers S9NW, xnGD, and MA3g recognized the value of our interactive multimodal clinical benchmark (OmniClinic) and the effectiveness of our agentic framework (OmniDoctorX) in bridging active evidence acquisition with diagnostic reasoning.
  • Reviewer MA3g acknowledged the validity of our experimental comparisons across conditions and noted that our core environment design provides a solid foundation for studying multimodal diagnostic behavior.

Below, we summarize the major improvements and how they resolved the shared concerns of the reviewers.

  • Component-Wise Ablations and Inference Efficiency (Reviewers S9NW and xnGD):
    • The reviewers inquired about isolating the individual contributions of OmniDoctorX’s modules and whether gains stem from coordination rather than extra compute.
    • We conducted new single-component ablations and resource accounting (Section 5.3.2 and Appendix D.4): 1) Module Contributions: Under the same 45-call allowance, every module positively contributes to raising Qwen3-Omni-Instruct's diagnostic accuracy from 72.65% to 85.25%, with Spatial Compilation breaking the execution bottleneck (raising image-unlock rates from 8.91% to 19.45%) and Independent Diagnostic Review improving accuracy (+2.99%) on identical acquired evidence. 2) Compute Efficiency: Candidate Planning reduces actual mean model calls by 15.8% and token consumption by 21.7% (93,842 →\rightarrow 73,517) while maintaining accuracy, proving that improvements stem from structured coordination rather than extra computation.
  • Multimodal Diagnostic Value and Expert Validation Scope (Reviewers S9NW and xnGD):
    • The reviewers asked about the diagnostic impact of minor media defects, low Krippendorff’s α\alpha alongside high mean scores, the 42-case expert audit scale, and whether acquiring more evidence improves reasoning.
    • We updated Section 3.2, Section 5.3.3, and Appendix E.3: 1) Multimodal Impact & Media Quality: Only 2.13% of reviewed media were flagged, and case-level audits confirm that patient history and lab reports preserve diagnostic validity; moreover, Table 4 (+11.60% accuracy for Gemini-3.7-Flash and +6.82% for Qwen3-Omni-Instruct over text-only baselines) and matched case trajectories confirm that relevant multimodal evidence directly improves diagnosis. 2) Inter-Rater Reliability: Gwet’s AC2 shows strong agreement on media integrity (0.867 for images, 0.937 for audio), while lower α\alpha reflects differing scoring strictness among senior physicians (mean image sign-fidelity scores of 3.79, 3.55, and 4.62/5 across 126 case–physician evaluations and 1,008 criterion ratings).
  • Reproducibility, Model Selection, and Evaluation Transparency (Reviewers MA3g and S9NW):
    • The reviewers requested a resource release commitment, rationale for the evaluated models, details on option construction and statistical rigor, and minor text/table corrections.
    • We revised Reproducibility Statement, Section 3.3, Section 5.1, Section 5.2, Section 6, and Appendix D.1: 1) Open Release & Model Scope: We committed to publicly releasing the environment, case data, redistributable media, evaluation code, and clarified that our model suite targets unified tri-modal (text, image, audio) models including top-ranked open-source and frontier systems (Qwen3-Omni and Gemini-3.7-Flash). 2) Protocol & Corrections: We detailed department-level option count, 10,000-resample case-paired 95% confidence intervals, and the infrastructure-excluded episodes.

In summary, the extensive additional experiments and clarifications have not only resolved the reviewers' concerns but have fundamentally strengthened the paper's rigor and transparency. By isolating component-level coordination gains, validating clinical media quality, and committing to full open-source release, we believe OmniClinic and OmniDoctorX establish a reproducible and effective paradigm for active multimodal clinical diagnosis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.