- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MedOmni evaluates medical agents that can access several types of patient data and call specialist encoders.
The benchmark combines tasks from three clinical datasets. It measures prediction performance, evidence access, task completion, and cost. The authors also introduce checks for leakage and task validity.
Critical strengths
S1. Useful research infrastructure
The platform combines heterogeneous patient data and specialist encoders in a reproducible, extensible environment. It enables research on agent-directed evidence selection
S2. Explicit attention to leakage
Prediction boundaries, target-specific exclusions, and documented rejected proposals make task construction more transparent. These are valuable safeguards
Critical weaknesses
M1. Some tasks lack a clear clinical purpose
Table 7 mixes diagnosis, historical conditions, physiological measurements, cognitive scores, and demographic prediction. These are not interchangeable clinical decisions.
For example, predicting cancer history from metabolic and lifestyle data may measure associations within the cohort. It does not necessarily test cancer diagnosis or a useful screening decision. Predicting demographic attributes from sleep physiology also needs a clearer rationale within a benchmark of clinical reasoning.
The intended use should be explicit. Is the agent screening, estimating a missing measurement, identifying an existing diagnosis, or informing a treatment decision? Who would use the answer, and what would they do differently?
Clinician involvement in task construction is a strength. It does not, by itself, resolve these questions.
M2. The scores do not establish clinical usefulness
The overall score averages different prediction tasks and metrics. This is useful for a leaderboard, but its clinical interpretation is limited.
A model's ability to rank patients does not show whether its predicted risks are reliable or whether its decisions are acceptable at a chosen threshold. Average performance can also hide clinically important failures on individual tasks.
The paper reports task-level results, which helps. However, describing the aggregate as decision correctness goes beyond what these measures establish. The evaluation does not distinguish the clinical consequences of different errors.
M3. The behavioural measures may reward unnecessary evidence access
The modality-choice score rewards agreement with the task proposer's list of relevant modalities. It does not establish whether those modalities were necessary for the individual patient.
Clinical records and structured measurements are excluded from this measure. An agent could make an appropriate decision from these inputs and still receive a low modality-choice score. Conversely, consulting every available modality already receives a relatively high score. The first-answer metric is also difficult to interpret because Figure 9 explicitly instructs agents to save predictions early and refine them
From a medical perspective, more evidence access is not automatically better reasoning. The relevant question is whether the agent uses enough evidence to make an appropriate decision and recognises when it needs more.
The benchmark measures inspection of existing data. It should not be interpreted as evaluating whether an agent orders appropriate investigations.
M4. The benefit of broader evidence and specialist tools is not established clearly enough
The evidence ladder adds modalities in a fixed order. Some of the largest gains occur when records and laboratory results become available. This leaves open whether the agent combines modalities or mainly relies on the most informative input.
A matched records-and-measurements baseline would help answer this. Without it, the value of the broader evidence collection remains difficult to judge.
The perception-service comparison also needs more cautious interpretation. The confidence intervals in Table 18 include no improvement. These results are suggestive, but do not establish a consistent benefit from offering the service.
Questions for authors
- M1: State the intended clinical use of each task. Distinguish diagnostic or screening tasks from historical-label prediction and exploratory phenotype estimation. Explain the role of tasks without a clear clinical action.
- M2: Keep the aggregate score as a benchmark measure. Narrow claims about clinical decision correctness. For a small set of clinically actionable tasks, report clinically interpretable errors and, where the sample allows, calibration or threshold-based performance.
- M3: Describe modality choice as agreement with proposer-selected inputs. Explain how appropriate decisions based on records and measurements are treated. A small clinician review of selected traces would help assess whether additional evidence access was useful.
- M4: Report a matched comparison with records and structured measurements alone. Use existing ladder results where possible. Revise the perception-service claims to reflect their uncertainty.
As a positioning suggestion, the authors could distinguish agent-directed evidence selection from prior modular prediction approaches, such as HAIM (DOI: 10.1038/s41746-022-00689-4), MoDN (DOI: 10.1371/journal.pdig.0000108), and MultiModN (arXiv: 2309.14118). HEALNet ( https://doi.org/10.48550/arXiv.2311.09115).
4 comments
Response [3/5]
M3: Describe modality choice as agreement with proposer-selected inputs. Explain how appropriate decisions based on records and measurements are treated. A small clinician review of selected traces would help assess whether additional evidence access was useful. (Question M3. Weakness M3 also notes that consulting every modality scores relatively high, that the first-answer metric is hard to interpret given the Figure 9 instruction, and that the benchmark measures inspection of existing data, not ordering of investigations.)
We sincerely thank the reviewer for this insightful point. Following the suggestion, we ran a blinded clinician review of agent traces, and revised the descriptions of the behavioural measures.
- Blinded clinician review (Table R13, new Appendix F.4). Five licensed clinicians rated 25 blinded traces from 9 tasks, 24 of them by two clinicians. Each trace pairs the agent that opened the most data with the one that opened the least.
- Unneeded access was rare. Where agents opened data, raters judged it needed in 36 of 41 ratings.
- Clinicians clearly favoured Astra over nano. None of Astra's 12 traces was judged inappropriate, against 10 of nano's 12 (p < 0.0001). Extra access helped by making the evidence sufficient.
Table R13: Clinician ratings by agent (24 ratings each)
| Question | Astra | nano |
|---|---|---|
| Opened data was needed | 19 | 16 |
| Evidence was enough | 22 | 4 |
| Reasoning used it | 23 | 12 |
| Decision inappropriate | 0 | 14 |
- Revised measures.
- Modality choice is renamed "agreement with proposer-selected inputs", a descriptive measure. Decisions based only on records and measurements score 0 on it but are still scored in full by task performance (see our response to Reviewer hexs, Q2).
- First answer is renamed "provisional answer", since the prompt asks agents to save one early.
- Scope. We state that the benchmark evaluates inspection of existing data, not the ordering of investigations.
Response [4/5]
M4: Report a matched comparison with records and structured measurements alone. Use existing ladder results where possible. Revise the perception-service claims to reflect their uncertainty. (Question M4. Weakness M4 also asks whether the agent combines modalities or mainly relies on the most informative input.)
We sincerely thank the reviewer for this constructive point. Following the suggestion, we report a matched comparison with records and structured measurements alone, from the existing ladder and from a new evidence-ablation experiment, and we have revised the perception-service claims to reflect their uncertainty.
- Matched comparison (Table R14, new Appendix G.2). We agree that records and measurements carry most of the gain on MIMIC-IV and AI-READI. On MIMIC-IV, the effect of adding chest radiographs and ECGs is moderate, and on AI-READI the gain mostly comes from records and labs. BDSP is different. There, agents reach most of their score from physiological signals before any record is added. A new evidence ablation on the nine clinician-reviewed tasks agrees (Appendix G.2, Table 34). On BDSP, Sol scores significantly lower with records only than with only the primary file type removed (by 19.6 points [7.1, 37.6]), while removing any single file type leaves it close to the full surface, so agents combine signals rather than rely on a single input.
This does not make the study of other modalities less meaningful. In real clinical settings, records and structured measurements are not always available. Even without them, the other modalities carry real information about the target (AI-READI: Astra 29.5, Sol 30.8, Table R14), which is why MedOmni studies how each potentially relevant modality contributes to the task.
Table R14: Matched comparison from the existing ladder (MedOmni score, 20 patients per task)
| Source | Evidence added | Astra | Sol | Luna |
|---|---|---|---|---|
| MIMIC-IV | Records and measurements → + chest radiograph and ECG | 82.0 → 81.5 | 79.0 → 74.0 | 68.5 → 70.5 |
| AI-READI | Other modalities → + record and labs | 29.5 → 65.1 | 30.8 → 61.9 | 16.3 → 29.6 |
| BDSP | Demographics → + sleep signals, no record | 26.8 → 63.6 | 27.9 → 63.4 | 18.7 → 44.5 |
- Perception-service claims, revised. The MP service improves Astra (+1.8) and Sol (+3.3) overall, with its largest gains on raw physiological signals in BDSP (Astra +6.2, Sol +7.6, Luna +2.1). We now report each gain with its effect size and confidence interval, which at 20 patients per task still includes zero (Introduction, Section 4.3, Table 37). The main role of the MP service is to make non-native modalities accessible to agents, which holds regardless of the size of the gain.
Response [5/5]
Positioning: "the authors could distinguish agent-directed evidence selection from prior modular prediction approaches, such as HAIM, MoDN, and MultiModN. HEALNet."
We sincerely thank the reviewer for this helpful suggestion, and we have added a positioning section (Appendix A.2). HAIM, MoDN, MultiModN and HEALNet are trained models that fuse a fixed, pre-selected set of modalities for a specific setting or condition. MedOmni follows an agentic paradigm: a zero-shot agent chooses for each patient what to open from a much broader spectrum of modalities, many of them irrelevant to the target. It is designed to be generic across settings and conditions and extensible, so models like these can be added as tools in the MedOmni perception service.
We sincerely thank the reviewer again for the constructive and thoughtful comments, which have helped us improve the paper. We would be very happy to address any further questions or concerns the reviewer may have.
An Invitation for Any Further Questions or Clarifications
Thank you for your time and thoughtful engagement with the review process. As the discussion period draws to a close, please feel free to let me know if there are any questions about my review or points I could clarify. I would be happy to help and will do my best to respond before the deadline. Thank you again for your attention and efforts.
Response [1/5]
We sincerely thank the reviewer for this kind, careful, and constructive point. We agree that the tasks are not interchangeable clinical decisions. In the revision, we state the intended use of every task and who would act on it, label the two tasks the reviewer names as exploratory and show that the headline scores barely change without them, and clarify the role of tasks without a direct clinical action.
Intended use of every task (Table 10). We now state the intended use of each task, derived from its target and prediction time, and who would act on the answer: 14 tasks detect a current condition (diagnosis or screening), 8 estimate a missing measurement, 4 identify a recorded diagnosis, 2 predict an outcome, and 2 are exploratory. We agree with the reviewer that cancer history from metabolic and lifestyle data, and demographic attributes from sleep physiology, test associations rather than a clinical decision, and we label both exploratory. Excluding them changes every agent's headline score by at most 1.8 points (Table 11).
Role of tasks without a clear clinical action. MedOmni is designed as a platform for evaluating agent behaviour in general medical settings, where each patient offers a different combination of modalities. Our task selection therefore favoured tasks that can draw on as many modalities as possible over clinically important tasks that rest on a single modality, which is why some tasks lack a direct clinical action. We keep them as capability probes, report them as a separate group, and make no clinical claim for them. To support clinically targeted use, we provide a repeatable task construction pipeline (Appendix B.4). Any practitioner with credentialed access to MIMIC-IV, BDSP or AI-READI can define a task configuration for their own clinical question and build it with the same pipeline, without changing the agent or the scoring.
Response [2/5]
We sincerely thank the reviewer for this thoughtful point. Following the suggestion, we have added an analysis with clinically interpretable metrics (calibration error and specificity at a fixed sensitivity) on a small set of clinically actionable tasks, together with a list of every below-chance result on individual tasks. We have revised how the paper states its metrics accordingly: the aggregate is presented as a benchmark score, not as clinical decision correctness.
Table R12: Calibration and fixed-sensitivity performance (11 detection tasks, 21 targets, 20 patients per task)
Thresholds are chosen on the same patients, so specificity is optimistic.