acceptodds
Under review as a conference paper at ICLR 2027

MedOmni: Benchmarking Omni Modality Integration in Medical Agentic Systems

17 views

Abstract

Clinical decisions require reasoning over heterogeneous medical evidence, yet existing benchmarks typically restrict each task to a small set of predefined modalities. This leaves medical agents' ability to select, access, and integrate broad evidence insufficiently evaluated. We introduce MedOmni, a clinically grounded benchmark developed with a multidisciplinary clinical consortium. To our knowledge, under our counting criterion, MedOmni provides the broadest modality coverage per task and is the first benchmark designed specifically for omni-modal medical agent evaluation. Its specialised infrastructure decouples tasks, modalities, and encoders through an extensible modality perception service, allowing new modalities to be incorporated without redesigning existing tasks. A structured assurance mechanism connects benchmark requirements to auditable checks across data, perception, tasks, and evaluation runs. Experiments reveal substantial differences in how agents benefit from and reason over heterogeneous evidence, identifying evidence selection and integration as central challenges for omni-modal medical intelligence.

Primary area:
datasets and benchmarks
Keywords:
Medicine, Multi-modal, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u9bvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedOmni evaluates medical agents that can access several types of patient data and call specialist encoders.

The benchmark combines tasks from three clinical datasets. It measures prediction performance, evidence access, task completion, and cost. The authors also introduce checks for leakage and task validity.

Critical strengths

S1. Useful research infrastructure

The platform combines heterogeneous patient data and specialist encoders in a reproducible, extensible environment. It enables research on agent-directed evidence selection

S2. Explicit attention to leakage

Prediction boundaries, target-specific exclusions, and documented rejected proposals make task construction more transparent. These are valuable safeguards

Critical weaknesses

M1. Some tasks lack a clear clinical purpose

Table 7 mixes diagnosis, historical conditions, physiological measurements, cognitive scores, and demographic prediction. These are not interchangeable clinical decisions.

For example, predicting cancer history from metabolic and lifestyle data may measure associations within the cohort. It does not necessarily test cancer diagnosis or a useful screening decision. Predicting demographic attributes from sleep physiology also needs a clearer rationale within a benchmark of clinical reasoning.

The intended use should be explicit. Is the agent screening, estimating a missing measurement, identifying an existing diagnosis, or informing a treatment decision? Who would use the answer, and what would they do differently?

Clinician involvement in task construction is a strength. It does not, by itself, resolve these questions.

M2. The scores do not establish clinical usefulness

The overall score averages different prediction tasks and metrics. This is useful for a leaderboard, but its clinical interpretation is limited.

A model's ability to rank patients does not show whether its predicted risks are reliable or whether its decisions are acceptable at a chosen threshold. Average performance can also hide clinically important failures on individual tasks.

The paper reports task-level results, which helps. However, describing the aggregate as decision correctness goes beyond what these measures establish. The evaluation does not distinguish the clinical consequences of different errors.

M3. The behavioural measures may reward unnecessary evidence access

The modality-choice score rewards agreement with the task proposer's list of relevant modalities. It does not establish whether those modalities were necessary for the individual patient.

Clinical records and structured measurements are excluded from this measure. An agent could make an appropriate decision from these inputs and still receive a low modality-choice score. Conversely, consulting every available modality already receives a relatively high score. The first-answer metric is also difficult to interpret because Figure 9 explicitly instructs agents to save predictions early and refine them

From a medical perspective, more evidence access is not automatically better reasoning. The relevant question is whether the agent uses enough evidence to make an appropriate decision and recognises when it needs more.

The benchmark measures inspection of existing data. It should not be interpreted as evaluating whether an agent orders appropriate investigations.

M4. The benefit of broader evidence and specialist tools is not established clearly enough

The evidence ladder adds modalities in a fixed order. Some of the largest gains occur when records and laboratory results become available. This leaves open whether the agent combines modalities or mainly relies on the most informative input.

A matched records-and-measurements baseline would help answer this. Without it, the value of the broader evidence collection remains difficult to judge.

The perception-service comparison also needs more cautious interpretation. The confidence intervals in Table 18 include no improvement. These results are suggestive, but do not establish a consistent benefit from offering the service.

Questions for authors
  • M1: State the intended clinical use of each task. Distinguish diagnostic or screening tasks from historical-label prediction and exploratory phenotype estimation. Explain the role of tasks without a clear clinical action.
  • M2: Keep the aggregate score as a benchmark measure. Narrow claims about clinical decision correctness. For a small set of clinically actionable tasks, report clinically interpretable errors and, where the sample allows, calibration or threshold-based performance.
  • M3: Describe modality choice as agreement with proposer-selected inputs. Explain how appropriate decisions based on records and measurements are treated. A small clinician review of selected traces would help assess whether additional evidence access was useful.
  • M4: Report a matched comparison with records and structured measurements alone. Use existing ladder results where possible. Revise the perception-service claims to reflect their uncertainty.

As a positioning suggestion, the authors could distinguish agent-directed evidence selection from prior modular prediction approaches, such as HAIM (DOI: 10.1038/s41746-022-00689-4), MoDN (DOI: 10.1371/journal.pdig.0000108), and MultiModN (arXiv: 2309.14118). HEALNet ( https://doi.org/10.48550/arXiv.2311.09115).

4 comments

Response [1/5]

M1: State the intended clinical use of each task. Distinguish diagnostic or screening tasks from historical-label prediction and exploratory phenotype estimation. Explain the role of tasks without a clear clinical action. (Question M1. Weakness M1 raises the same point, with cancer history and demographic prediction as examples.)

We sincerely thank the reviewer for this kind, careful, and constructive point. We agree that the tasks are not interchangeable clinical decisions. In the revision, we state the intended use of every task and who would act on it, label the two tasks the reviewer names as exploratory and show that the headline scores barely change without them, and clarify the role of tasks without a direct clinical action.

  1. Intended use of every task (Table 10). We now state the intended use of each task, derived from its target and prediction time, and who would act on the answer: 14 tasks detect a current condition (diagnosis or screening), 8 estimate a missing measurement, 4 identify a recorded diagnosis, 2 predict an outcome, and 2 are exploratory. We agree with the reviewer that cancer history from metabolic and lifestyle data, and demographic attributes from sleep physiology, test associations rather than a clinical decision, and we label both exploratory. Excluding them changes every agent's headline score by at most 1.8 points (Table 11).

  2. Role of tasks without a clear clinical action. MedOmni is designed as a platform for evaluating agent behaviour in general medical settings, where each patient offers a different combination of modalities. Our task selection therefore favoured tasks that can draw on as many modalities as possible over clinically important tasks that rest on a single modality, which is why some tasks lack a direct clinical action. We keep them as capability probes, report them as a separate group, and make no clinical claim for them. To support clinically targeted use, we provide a repeatable task construction pipeline (Appendix B.4). Any practitioner with credentialed access to MIMIC-IV, BDSP or AI-READI can define a task configuration for their own clinical question and build it with the same pipeline, without changing the agent or the scoring.

Response [2/5]

M2: Keep the aggregate score as a benchmark measure. Narrow claims about clinical decision correctness. For a small set of clinically actionable tasks, report clinically interpretable errors and, where the sample allows, calibration or threshold-based performance. (Question M2. Weakness M2 raises the same point, and adds that averages can hide failures on individual tasks and that different errors have different clinical consequences.)

We sincerely thank the reviewer for this thoughtful point. Following the suggestion, we have added an analysis with clinically interpretable metrics (calibration error and specificity at a fixed sensitivity) on a small set of clinically actionable tasks, together with a list of every below-chance result on individual tasks. We have revised how the paper states its metrics accordingly: the aggregate is presented as a benchmark score, not as clinical decision correctness.

  1. Clinically interpretable results (Table R12, Tables 25 and 26). On the 11 tasks that detect a current condition with a binary target (Table 10), the best agent, Astra, reaches a median specificity of 0.56 at 90% sensitivity, against 0.10 for a random ranking. Astra leads on specificity, and nano is worst on both metrics among the six agents. On individual tasks, apart from nano (14 of 29 tasks), each agent falls below chance on at most 2 tasks.

Table R12: Calibration and fixed-sensitivity performance (11 detection tasks, 21 targets, 20 patients per task)

AgentCalibration error (ECE) ↓Specificity at 90% sensitivity ↑
Astra0.1550.56
MiMo0.0770.39
Sol0.1800.31
DeepSeek0.1610.25
Luna0.1780.10
nano0.1950.00
Random ranking0.10

Thresholds are chosen on the same patients, so specificity is optimistic.

  1. Reframed clinical claims and added limitations.
    • Clinical claims, reframed (Section 1, Section 3.1, Appendix B.5). "Decision correctness" is now "task performance", and the aggregate is a benchmark score, not a measure of clinical decision quality.
    • Limitations on clinical readiness and error consequences, added (new Appendix F.3). Even Astra remains below the 82.5% specificity required of the first FDA-cleared autonomous screening AI (Abràmoff et al., npj Digital Medicine, 2018). The benchmark also does not weigh the clinical consequences of different errors, which needs clinician-set thresholds for each task.

Response [3/5]

M3: Describe modality choice as agreement with proposer-selected inputs. Explain how appropriate decisions based on records and measurements are treated. A small clinician review of selected traces would help assess whether additional evidence access was useful. (Question M3. Weakness M3 also notes that consulting every modality scores relatively high, that the first-answer metric is hard to interpret given the Figure 9 instruction, and that the benchmark measures inspection of existing data, not ordering of investigations.)

We sincerely thank the reviewer for this insightful point. Following the suggestion, we ran a blinded clinician review of agent traces, and revised the descriptions of the behavioural measures.

  1. Blinded clinician review (Table R13, new Appendix F.4). Five licensed clinicians rated 25 blinded traces from 9 tasks, 24 of them by two clinicians. Each trace pairs the agent that opened the most data with the one that opened the least.
    • Unneeded access was rare. Where agents opened data, raters judged it needed in 36 of 41 ratings.
    • Clinicians clearly favoured Astra over nano. None of Astra's 12 traces was judged inappropriate, against 10 of nano's 12 (p < 0.0001). Extra access helped by making the evidence sufficient.

Table R13: Clinician ratings by agent (24 ratings each)

QuestionAstranano
Opened data was needed1916
Evidence was enough224
Reasoning used it2312
Decision inappropriate014
  1. Revised measures.
    • Modality choice is renamed "agreement with proposer-selected inputs", a descriptive measure. Decisions based only on records and measurements score 0 on it but are still scored in full by task performance (see our response to Reviewer hexs, Q2).
    • First answer is renamed "provisional answer", since the prompt asks agents to save one early.
    • Scope. We state that the benchmark evaluates inspection of existing data, not the ordering of investigations.

Response [4/5]

M4: Report a matched comparison with records and structured measurements alone. Use existing ladder results where possible. Revise the perception-service claims to reflect their uncertainty. (Question M4. Weakness M4 also asks whether the agent combines modalities or mainly relies on the most informative input.)

We sincerely thank the reviewer for this constructive point. Following the suggestion, we report a matched comparison with records and structured measurements alone, from the existing ladder and from a new evidence-ablation experiment, and we have revised the perception-service claims to reflect their uncertainty.

  1. Matched comparison (Table R14, new Appendix G.2). We agree that records and measurements carry most of the gain on MIMIC-IV and AI-READI. On MIMIC-IV, the effect of adding chest radiographs and ECGs is moderate, and on AI-READI the gain mostly comes from records and labs. BDSP is different. There, agents reach most of their score from physiological signals before any record is added. A new evidence ablation on the nine clinician-reviewed tasks agrees (Appendix G.2, Table 34). On BDSP, Sol scores significantly lower with records only than with only the primary file type removed (by 19.6 points [7.1, 37.6]), while removing any single file type leaves it close to the full surface, so agents combine signals rather than rely on a single input.

This does not make the study of other modalities less meaningful. In real clinical settings, records and structured measurements are not always available. Even without them, the other modalities carry real information about the target (AI-READI: Astra 29.5, Sol 30.8, Table R14), which is why MedOmni studies how each potentially relevant modality contributes to the task.

Table R14: Matched comparison from the existing ladder (MedOmni score, 20 patients per task)

SourceEvidence addedAstraSolLuna
MIMIC-IVRecords and measurements → + chest radiograph and ECG82.0 → 81.579.0 → 74.068.5 → 70.5
AI-READIOther modalities → + record and labs29.5 → 65.130.8 → 61.916.3 → 29.6
BDSPDemographics → + sleep signals, no record26.8 → 63.627.9 → 63.418.7 → 44.5
  1. Perception-service claims, revised. The MP service improves Astra (+1.8) and Sol (+3.3) overall, with its largest gains on raw physiological signals in BDSP (Astra +6.2, Sol +7.6, Luna +2.1). We now report each gain with its effect size and confidence interval, which at 20 patients per task still includes zero (Introduction, Section 4.3, Table 37). The main role of the MP service is to make non-native modalities accessible to agents, which holds regardless of the size of the gain.

Response [5/5]

Positioning: "the authors could distinguish agent-directed evidence selection from prior modular prediction approaches, such as HAIM, MoDN, and MultiModN. HEALNet."

We sincerely thank the reviewer for this helpful suggestion, and we have added a positioning section (Appendix A.2). HAIM, MoDN, MultiModN and HEALNet are trained models that fuse a fixed, pre-selected set of modalities for a specific setting or condition. MedOmni follows an agentic paradigm: a zero-shot agent chooses for each patient what to open from a much broader spectrum of modalities, many of them irrelevant to the target. It is designed to be generic across settings and conditions and extensible, so models like these can be added as tools in the MedOmni perception service.

We sincerely thank the reviewer again for the constructive and thoughtful comments, which have helped us improve the paper. We would be very happy to address any further questions or concerns the reviewer may have.

AuthorsOpenReview ↗

An Invitation for Any Further Questions or Clarifications

Thank you for your time and thoughtful engagement with the review process. As the discussion period draws to a close, please feel free to let me know if there are any questions about my review or points I could clarify. I would be happy to help and will do my best to respond before the deadline. Thank you again for your attention and efforts.

Reviewer_hexsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MedOmni evaluates medical agents on patient-specific combinations of clinical evidence. It combines standardized tasks, an optional modality perception service, and an assurance framework. Experiments cover 30 task types and include a modality ladder and matched perception-service ablations to examine how agents benefit from additional evidence.

Critical strengths
  • The benchmark evaluates evidence selection and tool use over patient-specific modality combinations. Shared modality contracts and an optional perception service make the infrastructure potentially useful beyond the reported tasks.

  • The modality ladder and matched service ablations go beyond leaderboard comparisons and provide useful evidence about when additional inputs or specialist encoders help.

Critical weaknesses
  • The reliability of the benchmark comparisons needs further support. The 600 instances represent 314 unique patients, with only 20 patients per task type. The bootstrap in Appendix E.4 resamples patients within tasks but does not describe preserving dependence when a patient appears in several tasks. Clipping below-chance skill scores to zero also gives random predictions an expected score of about 10. Although acknowledged, the effect on model comparisons in these small cohorts remains unclear.

  • The behavioral measures do not directly establish evidence integration. Modality-choice F1 compares accessed families with a task proposer's relevance set, potentially penalizing alternative sufficient evidence routes. File access does not establish that the evidence affected the answer. The modality ladder provides complementary outcome evidence, but covers five task types in a fixed cumulative order. These limits should be reflected in the interpretation of omni-modal reasoning.

Questions for authors
  • How many patients are reused across task types, and do the leading paired comparisons remain significant when resampling preserves all task instances belonging to the same patient?

  • How were the proposer-defined relevance sets checked across clinicians? Can an agent following a different but sufficient evidence route receive a low modality-choice F1?

  • Using the existing traces or ablations, can the authors distinguish access to a modality from use of its information in the final decision? Which integration claims are supported beyond the five ladder tasks?

  • Are the main rankings and paired differences stable when below-chance scores are retained rather than clipped to zero? Please also give a matched random-prediction reference for interpreting the aggregate score.

  • The current repository releases five encoders and a masked task example, while the task emitter, MP gateway, scoring harness, and assurance implementation are deferred until acceptance. Could the authors provide the data-independent code and configurations needed to reproduce the reported evaluation, together with instructions for credentialed users to regenerate the task instances?

5 comments

Response [1/5]

Q1: How many patients are reused across task types, and do the leading paired comparisons remain significant when resampling preserves all task instances belonging to the same patient?

W1 (part): The 600 instances represent 314 unique patients, with only 20 patients per task type. The bootstrap in Appendix E.4 resamples patients within tasks but does not describe preserving dependence when a patient appears in several tasks.

We sincerely thank the reviewer for this careful and constructive question. We have expanded the cohort to 568 distinct patients, patient reuse is moderate, and under the suggested patient-clustered resampling most leading comparisons remain significant, including Astra's lead over Sol and MiMo and every agent's lead over nano. Three close pairs become borderline, and we now describe them as consistent rather than significant.

  1. A larger cohort. To address the limited cohort, we added a new 40-patient cohort on all 30 tasks for Sol, Luna and nano (see Table R1 in our response to Reviewer Vr7Q). Combined with the submission, these agents are now evaluated on 568 distinct patients (up from 314) and 1,628 task–patient pairs (up from 600), with up to 60 patients per task, and the ranking of the three agents is unchanged.

  2. Patient reuse (Table R7). Over the 29 headline tasks, 300 patients fill 580 instances: 58% appear in one task type and 42% in two or more.

Table R7: Number of task types per patient (submission cohort, 29 headline tasks)

Task types per patient1234 or more
Patients175622538
  1. Patient-clustered resampling (Table R8). Following the suggestion, we now resample each patient once across all tasks, so all instances of a patient move together. Intervals widen as expected. Astra's lead over Sol and MiMo, and every agent's lead over nano (e.g., Luna − nano +30.6 [+13.7, +42.9]), remain significant. Three close pairs (Astra − DeepSeek, Sol − Luna, DeepSeek − Luna) now include zero but stay positive in 94–97% of draws. The same holds in our new 40-patient cohort (Sol − Luna positive in 97% of draws, and Sol − nano and Luna − nano significant).

Table R8: Leading paired differences, published vs patient-clustered 95% CIs (20 patients per task)

PairDifferencePublished CIClustered CIP(difference > 0)
Astra − Sol+8.8[+2.5, +15.5][+1.1, +17.2]0.99
Astra − MiMo+10.6[+3.0, +17.4][+0.9, +19.0]0.99
Astra − DeepSeek+8.6[+2.3, +15.2][−0.4, +17.4]0.97
Sol − Luna+10.5[+0.2, +18.9][−1.6, +20.6]0.96
DeepSeek − Luna+10.7[+0.3, +20.5][−2.8, +23.0]0.94
Luna − nano+30.6[+17.4, +39.3][+13.7, +42.9]1.00

We have added the patient-clustered bootstrap to Appendix E.4 as the reference for paired comparisons and revised the wording of these comparisons accordingly, highlighted in blue.

Response [2/5]

Q2: How were the proposer-defined relevance sets checked across clinicians? Can an agent following a different but sufficient evidence route receive a low modality-choice F1?

W2 (part): Modality-choice F1 compares accessed families with a task proposer's relevance set, potentially penalizing alternative sufficient evidence routes.

We sincerely thank the reviewer for this insightful question. Each relevance set was proposed by one clinician and then cross-checked by an LLM review and two further clinicians. We also agree that an agent following a different sufficient route can receive a low score: our traces show that agents often decide correctly through routes clinicians did not select, and we now present the measure as agreement with proposer-selected inputs.

  1. How relevance sets were checked. For each task, the proposing clinician marked the modality families relevant to the decision. Each set was then cross-checked by an LLM review and by two further clinicians before the task was accepted. We now describe this procedure in the appendix of the revised paper.

  2. Yes, a sufficient alternative route can score low (Table R9). Using the saved traces, we computed the measure for every trial and compared it with whether the answer was correct (better than the no-information prediction for that patient). For Astra, Sol and Luna, 62–75% of correct answers score below 0.5, and for five of six agents wrong answers score slightly higher than correct ones. Two reference policies make the same point: opening every modality scores 0.61, above all agents but one, while relying only on records and measurements scores 0, because these are excluded from the measure.

Table R9: Modality-choice score against answer correctness (submission runs)

AgentMean scoreCorrect answers scoring < 0.5Mean score, correct / wrong
MiMo0.6239%0.60 / 0.65
DeepSeek0.5947%0.56 / 0.66
Astra0.3862%0.35 / 0.44
Sol0.3171%0.27 / 0.39
Luna0.3075%0.24 / 0.39
nano0.1787%0.17 / 0.13
Open every modality0.61
Records and measurements only0.00
  1. A promising finding. We are truly grateful for this question. It shows that agents can reach good decisions through evidence routes that differ from clinicians', a promising direction for future study. We now describe B2 as agreement with proposer-selected inputs rather than a measure of reasoning quality, and discuss this finding in the revised appendix (highlighted in blue).

Response [3/5]

Q3: Using the existing traces or ablations, can the authors distinguish access to a modality from use of its information in the final decision? Which integration claims are supported beyond the five ladder tasks?

W2 (part): File access does not establish that the evidence affected the answer. The modality ladder provides complementary outcome evidence, but covers five task types in a fixed cumulative order. These limits should be reflected in the interpretation of omni-modal reasoning.

We sincerely thank the reviewer for this precise question. Yes, the traces allow us to separate access from use, and they confirm that access alone does not establish use. We have revised our interpretation of omni-modal reasoning accordingly.

  1. From access to use (Table R10). The saved traces record every encoder call, the agent's terminal output and its messages, so we can follow each encoder answer into the decision at three levels: called, seen (its output values appear on the agent's screen), and cited (the agent's reasoning quotes the output before it writes the final answer). Most encoder outputs are seen, but only 12–47% are cited in the decision. Similarly, in 3–10% of trials agents open a binary image or signal file (e.g., DICOM) without calling any encoder, so that content could not have informed the answer.

Table R10: How far encoder answers reach the decision (trials that called an encoder)

AgentTrials calling an encoderOutput seenOutput cited in the decision
Astra16267%12%
DeepSeek36389%47%
MiMo46168%42%
Sol13387%17%
Luna11680%39%
  1. Evidence of use beyond the ladder. Use is tested by withholding evidence. Beyond the five ladder tasks (Tables 31–32), a new evidence ablation withholds files on the nine clinician-reviewed tasks (Appendix G.2, Table 34).

  2. Where integration is supported. Whether agents combine several modalities, rather than relying on the most informative one, is supported on the ladder tasks and, in a new evidence ablation, on BDSP (Appendix G.2). We now state the scope of these results explicitly, describe B2 as evidence access rather than use, and have softened our claims about omni-modal integration (highlighted in blue).

Response [4/5]

Q4: Are the main rankings and paired differences stable when below-chance scores are retained rather than clipped to zero? Please also give a matched random-prediction reference for interpreting the aggregate score.

W1 (part): Clipping below-chance skill scores to zero also gives random predictions an expected score of about 10. Although acknowledged, the effect on model comparisons in these small cohorts remains unclear.

We sincerely thank the reviewer for this helpful suggestion. The rankings and paired differences are stable without clipping, and a matched random predictor scores about 8 with clipping and about −3 without, close to the reviewer's estimate. We now report both alongside the leaderboard.

  1. Rankings without clipping (Table R11). Retaining below-chance scores leaves the ranking unchanged, except that DeepSeek and Sol swap places. The two were already statistically tied (0.2 points apart), and DeepSeek has slightly more below-chance targets.

Table R11: Suite score with and without clipping, and a matched random predictor (20 patients per task, 29 headline tasks)

AgentClipped (published)Unclipped
Astra62.762.7
DeepSeek54.151.9
Sol53.952.7
MiMo52.149.9
Luna43.440.9
nano12.7−0.2
Random predictor7.9 [4.2, 12.0]−3.2 [−10.0, +4.0]
  1. Paired differences without clipping. Every leading paired difference keeps its sign and becomes slightly larger, e.g., Astra − Sol +8.8 → +9.9, Astra − MiMo +10.6 → +12.7 and Sol − Luna +10.5 → +11.8. In our three repeated runs, Sol also leads Luna in every run without clipping (see our response to Reviewer Vr7Q, W2).

  2. Matched random reference. A random predictor (gold labels permuted across patients, same pipeline) scores 7.9 with clipping and −3.2 without (Table R11), consistent with the reviewer's estimate. We have added it to the leaderboard in the revised paper, highlighted in blue.

Response [5/5]

Q5: The current repository releases five encoders and a masked task example, while the task emitter, MP gateway, scoring harness, and assurance implementation are deferred until acceptance. Could the authors provide the data-independent code and configurations needed to reproduce the reported evaluation, together with instructions for credentialed users to regenerate the task instances?

We sincerely thank the reviewer for this request, and we fully agree that reproducibility is essential. We have reorganized and expanded our anonymized code release, with the data-independent code and configurations, together with instructions for credentialed users to regenerate the task instances: https://anonymous.4open.science/r/MedOmni-858C.

The snapshot contains the task construction pipeline with every task configuration (studio/), the MP service with its encoder cards (perception/), the run and scoring harness (bench/), the shared contracts (core/), and the documentation, including a step-by-step guide to rebuilding a task set (docs/guides/). It contains no patient data. Credentialed users can regenerate every task instance from the source databases by following the included guide.

We sincerely thank the reviewer again for the constructive and thoughtful comments, which have helped us improve the paper. We would be very happy to address any further questions or concerns the reviewer may have.

Reviewer_Vr7QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedOmni Bench, a clinically grounded evaluation platform designed to assess whether medical agents can dynamically select, access, and integrate heterogeneous patient evidence across 34 modality families and 30 task types. To support diverse non-native inputs without redesigning tasks, the platform implements an extensible Modality Perception service alongside a structured assurance mechanism that rigorously audits clinical validity, container isolation, and data leakage. Extensive empirical evaluations across six frontier agent systems demonstrate that while specialized perception tools significantly boost reasoning over complex physiological and imaging signals, broader modality exposure alone does not ensure superior clinical decisions due to persistent bottlenecks in evidence selection and integration.

Critical strengths
  1. MedOmni Bench provides unprecedented clinical evidence diversity by incorporating thirty-four modality families across thirty patient-specific diagnostic tasks designed in collaboration with licensed medical practitioners.
  2. The platform introduces an extensible Modality Perception service that standardizes heterogeneous input contracts and equips agents to dynamically leverage specialized external tools for non-native medical modalities.
Critical weaknesses
  1. Evaluating only twenty patient instances per task type limits the statistical power of the benchmark and leaves performance estimates susceptible to patient-level variance.
  2. Executing each evaluation trial configuration only once fails to account for the substantial run-to-run stochasticity inherent in long-horizon agent trajectories.
  3. Confining experiments to a single agent scaffold obscures whether observed performance patterns stem from base model intelligence or scaffold-specific design choices.
  4. Uneven accuracy and potential failure modes across the specialized perception encoders confound the evaluation by conflating tool errors with flawed agent reasoning.
  5. Sourcing clinical data exclusively from three public databases restricts the geographic, institutional, and demographic generalizability of the benchmark.
Questions for authors

Please refer to the weaknesses.

4 comments

Response [1/5]

W1: Evaluating only twenty patient instances per task type limits the statistical power of the benchmark and leaves performance estimates susceptible to patient-level variance.

We sincerely thank the reviewer for this valuable comment, and we fully agree that a larger cohort gives a more robust evaluation. While 20 patients per task is in line with the scale of recent agentic medical benchmarks, we have followed the reviewer's suggestion and added a new 40-patient cohort on all 30 tasks for three representative agents, nearly tripling the cohort per task (up to 60 patients). We are pleased to report that the ranking remains the same as with 20 patients.

  1. A trade-off between cost and robustness. Each instance is a full agent episode on one patient (up to 20 minutes, many tool calls), not a single model query. Agentic medical benchmarks are therefore built at this scale: MedAgentBench (Jiang et al., NEJM AI 2025) evaluates 300 tasks over 100 patients (about 30 per category), AgentClinic (Schmidgall et al., npj Digital Medicine 2026) about one case per disease, and HealthAgentBench (Liu et al., 2026) 5–10 tasks per category. We hope this context is helpful: our 20 patients per task, now extended with a further 40, is consistent with this practice.

  2. A larger, mostly new cohort (Table R1). The new 40-patient draw shares only 154 of 1,182 task–patient pairs with the submission, mostly on AI-READI, whose eligible pool is about 100 participants. Sol, Luna and nano are now evaluated on 568 distinct patients (up from 314) and 1,628 task–patient pairs (up from 600).

Table R1: Patients evaluated before and after the rebuttal for Sol, Luna and nano: patients (task–patient pairs)

SourceOriginal cohortExpanded cohortCombinedPer task, combined
MIMIC-IV130 (180)182 (360)310 (539)59–60
BDSP98 (120)62 (240)160 (360)60
AI-READI86 (300)81 (582)98 (729)41–56
Total314 (600)325 (1,182)568 (1,628)
  1. The ranking holds (Table R2). Sol > Luna > nano at both cohort sizes. At 40 patients, Sol and Luna lead nano by 46.1 and 38.2 points. Sol leads Luna in 96–97% of patient-clustered bootstrap draws, though the 95% interval touches zero, so we report this lead as consistent but modest. nano completes most trials (80.5% at 40 patients, 78.7% at 20) but scores low.

Table R2: Suite score at 20 and 40 patients per task (29 headline tasks, 95% CIs resample whole patients across tasks)

Agent20 patients (submission)40 patients (new)
Sol53.9 [42.2, 66.6]54.4 [43.4, 65.3]
Luna43.4 [34.2, 56.6]46.4 [35.0, 58.2]
nano12.7 [8.7, 26.4]8.2 [5.0, 18.5]
Sol − Luna+10.5 [−1.6, +20.6]+8.0 [−0.4, +15.9]

We have added these results to the appendix of the revised paper, highlighted in blue. We thank the reviewer again for prompting this analysis.

Response [2/5]

W2: Executing each evaluation trial configuration only once fails to account for the substantial run-to-run stochasticity inherent in long-horizon agent trajectories.

We sincerely thank the reviewer for raising this important point. Following the suggestion, we repeated the evaluation three times, and although the scores vary slightly from run to run, the relative performance of the agents holds.

  1. Three independent runs (Table R3). We ran the nine clinician-reviewed tasks (20 patients each) three more times with fresh, independent executions for Sol and Luna. All 1,080 trials completed and answered every case.

Table R3: Suite score over three independent runs (9 tasks, 20 patients per task)

AgentRun 1Run 2Run 3Mean
Sol49.046.552.649.4
Luna45.041.645.143.9
Sol − Luna+4.0+4.9+7.5+5.5
  1. Repeated runs in medical benchmarks. We would also like to share some context on why single runs have been common in this area. Each agent run is a long, tool-using episode, so repetition multiplies an already high evaluation cost. Representative benchmarks therefore spend their budget on breadth of tasks and patients rather than repeats. MedAgentBench (Jiang et al., NEJM AI 2025) reports a single attempt per task with deterministic decoding and leaves repeated runs to future work, and HealthAgentBench (Liu et al., 2026) scores one attempt per task. Recent work has argued that this practice should change (Beyond the Leaderboard, ACL 2026), and PhysicianBench (Liu et al., 2026) is a notable example that reports three runs. We fully agree with this direction and now report repeated runs as well.

We have added these results to the appendix of the revised paper, highlighted in blue.

Response [3/5]

W3: Confining experiments to a single agent scaffold obscures whether observed performance patterns stem from base model intelligence or scaffold-specific design choices.

We sincerely thank the reviewer for this insightful comment. Following the suggestion, we evaluated the same models under two further widely used agent scaffolds, codex and openhands-sdk. Each model's score changes only slightly across the three scaffolds, while the differences between models remain large, so the ranking reflects the model rather than the scaffold (Table R4).

Table R4: Suite score of each model under three scaffolds (7 clinician-reviewed tasks, 20 patients per task)

ScaffoldSolLunanano
terminus-2 (submission)48.339.74.5
codex51.633.916.9
openhands-sdk48.838.319.4
Spread across scaffolds3.35.714.9

We have added this comparison to the appendix of the revised paper, highlighted in blue.

Response [4/5]

W4: Uneven accuracy and potential failure modes across the specialized perception encoders confound the evaluation by conflating tool errors with flawed agent reasoning.

We sincerely thank the reviewer for this thoughtful comment, and we agree that encoder accuracy is uneven across modalities. We would like to clarify that choosing which encoders to trust is part of the capability MedOmni evaluates, and our new analyses show that encoder errors account for very few of the agents' mistakes and do not drive the ranking.

  1. Choosing reliable encoders is part of the task, by design. The MP service is an open, extensible registry, so encoders will always differ in reliability. Agents are therefore neither required nor forbidden to call an encoder, and each encoder card declares its intended use, limitations and reliability evidence (Section 3.2). A capable agent should be selective: trust an encoder where it is reliable and fall back on other evidence where it is not. Agents indeed differ widely in how often they call encoders, from 20% to 78% of trials (Table R5), and calling more often does not lead to higher scores (Figure 5).

  2. Encoder errors explain very few wrong answers (Table R5). Using the saved traces, we judged each encoder output whose label is the target or an established clinical sign of it (e.g., chest X-ray cardiomegaly for heart failure) against the patient's label, and attributed every wrong answer to its cause. Following an incorrect encoder accounts for at most 2% of all wrong answers for every agent (10 of 870, Table R5), and agents overrode a correct encoder more often (32) than they followed an incorrect one (10). The full procedure is in the revised appendix.

Table R5: Encoder use and wrong answers on encoder-calling trials, by cause (submission runs, 29 headline tasks, 20 patients per task)

AgentTrials calling an encoderWrong answersNo encoder output bears on the targetFollowed an incorrect encoderOverrode a correct encoderRecovered from an incorrect encoder
Astra28%136132133
DeepSeek61%243230589
MiMo78%20919331320
Sol23%140138020
Luna20%142135160
  1. The ranking does not appear to depend on encoder outputs. The submission includes a matched comparison with the MP service switched off (Table 37 and Figure 6, bottom), in which offering the service does not significantly change any agent's score. In addition, on the patient–task pairs where neither agent called an encoder, the Sol − Luna difference remains similar (+10.1 vs +8.0 on all pairs at 40 patients, details in the revised appendix).

We have added these analyses to the appendix of the revised paper, highlighted in blue.

Response [5/5]

W5: Sourcing clinical data exclusively from three public databases restricts the geographic, institutional, and demographic generalizability of the benchmark.

We sincerely thank the reviewer for raising this point, and we fully agree that data diversity is very important. Our three sources are large and demographically diverse, and to broaden geographic coverage we have added a fourth, fully open data source from Korea.

  1. Diversity of the existing sources. The three sources are large clinical archives rather than single cohorts (Table 2). MIMIC-IV covers more than 360,000 patients of diverse ethnic backgrounds. AI-READI was designed to balance race and ethnicity, sex and diabetes severity across three US sites. BDSP is a large multi-site sleep archive, from which the benchmark currently draws one site. Together they span three care settings (hospital, outpatient research study and sleep laboratory).

  2. A fourth source from another country (Table R6). We agree that geographic coverage is limited, as all three sources are from the United States. We have therefore added VitalDB (surgical patients at Seoul National University Hospital, Korea), which brings a second country and an intraoperative care setting. VitalDB is also fully open access, so users can obtain it without a credentialed data-use agreement. Both agents score far above a random predictor on these tasks.

Table R6: MedOmni score on each VitalDB task (60 patients per task)

TaskLunaSol
In-hospital death61.656.8
Postoperative anaemia87.091.4
Postoperative creatinine56.353.3
ICU stay (exploratory)82.668.3

A random predictor (gold labels permuted across patients) averages 4.4 [0.0, 13.0] over the four tasks.

  1. Coverage compared with other agentic medical benchmarks. With VitalDB, MedOmni offers the broadest modality and care-setting coverage on real patient data among the agentic medical benchmarks we compared. For example, MedAgentBench (Jiang et al., NEJM AI 2025) draws on electronic health records from one hospital, and AgentClinic (Schmidgall et al., npj Digital Medicine 2026) uses simulated cases.

We have added VitalDB and this comparison to the appendix of the revised paper, highlighted in blue.

We sincerely thank the reviewer again for the constructive and thoughtful comments, which have helped us improve the paper. We would be very happy to address any further questions or concerns the reviewer may have.

AuthorsOpenReview ↗

An Invitation for Any Further Questions or Clarifications

Thank you for your time and thoughtful engagement with the review process. As the discussion period draws to a close, please feel free to let me know if there are any questions about my review or points I could clarify. I would be happy to help and will do my best to respond before the deadline. Thank you again for your attention and efforts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition that MedOmni provides "unprecedented clinical evidence diversity" across 34 modality families with an "extensible Modality Perception service" (Vr7Q), that it "evaluates evidence selection and tool use over patient-specific modality combinations", with infrastructure "potentially useful beyond the reported tasks" and analyses that "go beyond leaderboard comparisons" (hexs), and that it is "useful research infrastructure" in "a reproducible, extensible environment" that "enables research on agent-directed evidence selection" (u9bv).

We have addressed each reviewer's concerns individually below and incorporated all new experiments and clarifications into our revised manuscript (highlighted in blue). Below is a summary of the major revisions:

  1. A More Robust Leaderboard: A new 40-patient cohort on all 30 tasks extends the evaluation to up to 60 patients per task and 568 distinct patients. The ranking holds at both cohort sizes, across three independent repeated runs, and under two further agent scaffolds, codex and openhands-sdk (Tables R1–R4).
  2. Blinded Review by Licensed Clinicians: Five licensed clinicians rated blinded agent traces for evidence need, sufficiency, use and decision appropriateness, and strongly favoured the leading agent over the weakest (p < 0.0001, Table R13).
  3. A New Fully Open Dataset: We added VitalDB, a fully open intraoperative dataset from Korea, which brings a second country and care setting. Both agents score far above a random predictor (Table R6).
  4. Encoder Errors and Evidence Integration: We added an attribution of every wrong answer to its cause and several new evidence-ablation experiments. Encoder errors account for very few wrong answers, and each modality's contribution is heterogeneous and depends on the task, with agents combining inputs where the task calls for it (Tables R5, R14, new Appendix G.2).
  5. Clinical Grounding and Statistical Robustness: We state the intended use of every task, report clinically interpretable metrics on actionable tasks, and confirm the rankings under patient-clustered resampling and without clipping (Tables R8, R11, R12).

We have also reorganized and expanded our anonymized code release, so that the full evaluation is now straightforward to reproduce (https://anonymous.4open.science/r/MedOmni-858C).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.