acceptodds
Withdrawn from ICLR 2027

Toward Virtual Patients: A Benchmark for Forecasting Future Radiology Reports

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrospective electronic health records (EHRs) contain outcomes only for the tests clinicians historically ordered, so interactive clinical agents cannot be trained or evaluated on queries that depart from recorded trajectories. We study an evaluable surrogate—query-conditioned clinical world modeling: given a patient's leakage-controlled EHR context and a requested imaging study, generate the study's report, scored by its structured findings under a frozen extractor. Building on MIMIC-IV and MIMIC-IV-Note, we align longitudinal clinical events with 117K radiology reports across four modalities and pose 1,500 sealed masked-observed queries (every target study was actually performed, so every query is gradable). Evaluating frontier LLMs, structured supervised pipelines, and a budgeted LLM-policy search over model design, we find a robust dissociation: end-to-end LLMs reach micro-F1 while structured predictors over a prior-imaging finding-state representation plateau at , invariant to search budget, policy strength, and rendering strategy. We trace the plateau to the supervision itself: report labels measure radiologists' mention behavior, not patient state—a finding documented as present goes unmentioned in the next report of the time even within seven days, nearly independent of elapsed time. Under an omission-tolerant reading of the same predictions—forgiving a predicted finding that prior reports document and the target report merely leaves unmentioned—all systems gain 0.10–0.12 and the verdict changes: a structured pipeline that significantly trails the best end-to-end LLM under the mention convention ties it under that reading—the evaluation convention, not only the model, determines the conclusion. We formalize the task as state estimation composed with mention selection and intervene on both factors, obtaining a double dissociation: mention-trained models win under the mention convention while densely state-supervised models win under the omission-tolerant reading—where a dense-supervised world model nearly matches the best end-to-end LLM—and, at matched context budget, structured state supplements lift an end-to-end report generator by . Our model-design search runs under an integrity-by-construction protocol: the LLM policy acts in a typed, budgeted action space with the sealed test structurally unreachable; one committed configuration is scored once by an external evaluator. Validation-to-sealed gaps grow with search intensity while sealed scores stay flat, and policy behavior (early saturation, cost-insensitivity, avoidance of semantic actions) is reported as part of the evaluation. We release the benchmark, findings schema, extraction pipeline, environment, and evaluation harness.

Primary area:
datasets and benchmarks
Keywords:
Virtual Patients, Clinical World Models, Electronic Health Records, Radiology Report Generation, Automated Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Dw37OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a benchmark that evaluates whether a model can generate a radiology report based on the patient's prior EHR history. The authors compare two report-generation approaches, an end-to-end model and a structured-state pipeline. To account for omissions in the reference radiology reports, which can unfairly penalize otherwise correct outputs, they report omission-tolerant performance alongside standard evaluation metrics. Under standard metrics, the structured pipelines plateau at a micro-F1 of about 0.50, slightly below end-to-end LLMs (about 0.55). Under the omission-tolerant metric, however, all systems improve by 0.10–0.12 and perform comparably. The authors argue that the evaluation convention, not only the model, determines the conclusion.

Critical strengths

None

Critical weaknesses

Weakness 1: Clarity and organization

The paper is hard to follow, and in my view, it is difficult to see how its components connect because the paper switches between topics too frequently. (1) Beyond the evaluation, the contributions extend to loosely related components such as auto-research, which are mixed throughout the abstract, introduction, and related work. (2) Many components lack detail. For example, although the paper mentions some of the inputs in passing, it does not provide a complete specification of the benchmark, such as the full list of modalities and EHR history components used (e.g., whether prior images are included). It is also unclear what the 73 union concepts in the structured pipeline refer to. (3) Em dashes are overused (about 100 times). Overall, the paper reads as a mix of loosely connected parts, which makes its potentially valuable core finding difficult to evaluate.

Weakness 2: Limited contribution of the evaluation itself

In my view, the main contribution of this paper is the finding that evaluation should account for errors inherent in the reference radiology reports. However, this finding is relatively straightforward, and there are already prior attempts to fill in missing information by referencing other reports for evaluation. As a result, the novelty of the proposed evaluation is limited. In addition, the task itself requires generating a radiology report without seeing the actual image (Lines 198-201), which raises concerns about its validity. Since the reference reports are written by radiologists based on findings observed in the images, many findings cannot be inferred from the EHR history alone. For example, if the reference report describes lung cancer, a history of coughing alone is not enough for a model to infer that finding. Therefore, errors caused by the absence of images may have a larger effect on the results than the report omissions the paper focuses on.

Questions for authors
  1. What components make up the EHR history, and from which MIMIC source (e.g., module or table) was each component extracted?
  2. What is the specific structure of the structured pipeline? What do the 73 union concepts refer to?
  3. What prompts are used for each model, including the system prompt and the input prompt? How is the EHR history formatted and presented to the model within the prompt?
Reviewer_4oQaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for forecasting radiology reports. Given a patient's leakage-controlled EHR context from MIMIC-IV and MIMIC-IV-Note before time t and a requested imaging study, a system writes the report for that study. A frozen LLM extractor (gpt-5.4-mini) maps generated and true reports to a four-state findings schema with 73 concepts, and the score is micro-F1 between the two sets of positive findings. The sealed test set has 1,500 queries (1,419 scorable), all for studies that were actually performed. End-to-end LLMs reach micro-F1 of 0.54 to 0.56, while structured pipelines that predict a prior-imaging finding state and render it into a report plateau near 0.50; this includes configurations found by an LLM policy searching a typed action space without access to the sealed set. The authors attribute the plateau to the labels, since a finding documented as present goes unmentioned in the next report about half the time. Under an omission-tolerant reading that forgives such carry-forwards, all systems gain about 0.11 and the best structured pipeline ties the best LLM. Validation consists of paired bootstrap tests, an extractor sensitivity check, human audits that include two radiologists, dense-supervision and rendering experiments, and an analysis of the search policy's behavior.

Critical strengths

The evaluation is designed with more care than most benchmark papers. Contexts are leakage-controlled and audited for same-study leaks, the splits are patient-disjoint, every committed configuration is scored once on the sealed set, and the comparisons the text interprets come with paired bootstrap tests. Negative results are reported plainly, including that the agent search found little beyond well-tuned ensembles. This weighs in the paper's favor because it makes the reported numbers trustworthy as measurements, whatever one concludes about their interpretation.

A second strength is the observation that report-derived labels record what radiologists choose to mention as well as what is present. A finding documented as present often goes unmentioned in the next report, even within a week, and changing how such silences are scored changes which systems look best. People who train or evaluate models on report-derived labels need to know this, and the paper measures it on a public dataset, although the exact rate is less certain than presented (see weaknesses).

Critical weaknesses

My main concern is that the central diagnosis rests on labels and a secondary metric whose validity is not established. The flicker rate is computed from extracted labels, and the paper's own audits suggest that a sizable share of that silence may be extraction error. The author audit found full coverage of key findings in only 60% of reports. In the radiologists' adjudication, 13% of forgiven false positives were in fact mentioned in the target report, and a further 8% rested on unsupported prior labels. The omission-tolerant reading, on which the claims that the verdict changes and that the best structured pipeline ties the best LLM depend, was not supported by that adjudication either. Radiologists judged 6% of forgiven findings likely persistent and 71% likely resolved. The authors argue that these judgments reflect the reporting convention, which may be true, but it leaves the reading untested. The double dissociation in Section 5.4 does not settle the question, because models trained on dense labels that remove silence from the loss will predict more carry-forward positives, and the omission-tolerant reading forgives exactly those by definition. An independent reference for state is needed. For chest radiographs, roughly 60% of the benchmark, MIMIC-CXR images are linked to MIMIC-IV and could be used to adjudicate a sample of forgiven findings.

Several claims also go further than the evidence. The abstract traces the structured plateau to the supervision itself, yet the paper's own controls point at the input representation. Copying the most recent prior report scores 0.495, an end-to-end LLM restricted to a 4k context scores 0.503, and the ablation in Section 5.5 attributes the end-to-end advantage almost entirely to verbatim prior reports, which the 73-concept state discards. Label noise affects both paths and the end-to-end models still reach 0.56, so noise alone does not explain a ceiling at 0.50. The paper says repeatedly that the masked-observed design lets it measure clinician-driven selection, citing Section 5.3, but that section analyzes radiologists' reporting behavior, not which studies clinicians order. Because only performed studies are queried, the benchmark cannot test the counterfactual queries that motivate the virtual-patient framing, and the query itself reveals that a clinician chose to order the study. Finally, differences of 0.015 to 0.02 carry several headline claims, including the +0.02 gain from structured supplements in the abstract, while swapping the extractor alone moves between-arm gaps by up to 0.009 (Appendix C). The same gpt-5.4-mini serves as gold-label extractor, Path B state extractor and renderer, and Path A generator, and the bootstrap intervals cover case sampling but not generation variance.

Questions for authors

What fraction of the present-to-not-mentioned transitions turn out to be extraction misses when a human checks them? A flicker rate measured on a verified sample would show how much of the mention-versus-state effect is real. Could the forgiven false positives on chest radiographs be adjudicated against the linked MIMIC-CXR images, so that the omission-tolerant reading is tested against an independent reference? How do the between-arm differences hold up with an extractor from a different model family (or CheXbert labels on chest radiographs), with repeated generations, and with a bootstrap clustered by patient, given that the sealed set may contain several queries per patient? Where in the paper is clinician-driven ordering selection measured? Why is the target study's indication excluded from the query, when an agent ordering a test would know its reason and 65% of false negatives are described as inferable only from the indication? Which deployment was used for each model that received MIMIC text (see the ethics note)? Finally, the AI-use statement says that claims about related work are being verified against the original papers. Has that been completed, in particular for the statements about EHRWorld and MIRA?

The auto-research thread is only loosely connected to the benchmark's main question, and its evidence comes from few runs (two policies, thirteen trajectories, three random seeds). Its sealed protocol is sound holdout practice, but a validation-to-sealed gap that grows with search intensity shows adaptive overfitting to the validation split, a well-documented effect; it is not evidence that leakage was prevented, as Section 5.2 states. A tighter paper on the benchmark and the mention-versus-state analysis, with the search study reported separately, would be easier to follow. The prose is also very dense, packing many coined terms and numbers into each sentence, and some figures are hard to read at print size; the axis labels in the left panel of Figure 5 overlap. Reporting macro-F1 next to micro-F1, since chest radiographs dominate the score, and giving dated model snapshots for every LLM would also help.

Reviewer_n1d6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a MIMIC-IV benchmark for predicting a future radiology report from information available before an examination. It compares direct LLM generation with structured predictors followed by an LLM renderer. The main empirical observation is that report labels reflect what radiologists mention, not necessarily the underlying patient state, and changing the scoring convention consequently changes the comparison between the two approaches. A separate experiment lets an LLM policy search over a fixed menu of structured model configurations.

Critical strengths
  1. The benchmark construction is careful. The temporal cutoff, patient-disjoint split, sealed evaluation, fixed extractor, and leakage audit make the reported comparisons more credible than a simple retrospective EHR experiment.

  2. The paper does substantial diagnostic work. In particular, the error breakdown and the supervision and rendering interventions provide useful evidence that predicting report mentions is different from estimating a latent clinical state.

Critical weaknesses
  1. The omission-tolerant score does not validate patient-state prediction. The authors appropriately call it a heuristic sensitivity analysis, but they also describe it as an upper bound on state accuracy and use it to attribute much of the performance gap to mention selection. Appendix H cannot support that interpretation: the radiologists state that reliable persistence judgments require the missing target images, and their report-based judgments remain entangled with the same restatement convention under study. The 0.667 versus 0.668 result therefore shows that the ranking changes under a different scoring rule, not that the two systems recover patient state equally well.

  2. The contribution to virtual-patient modeling remains unclear. The authors explicitly position the benchmark as a surrogate, and a benchmark paper need not deliver a complete patient simulator. However, the experiments concern single-step report prediction for examinations that actually occurred; they do not establish whether performance on this task carries over to unobserved examination requests or patient trajectories. The report-mention analysis is careful, but the paper needs to explain what it establishes about clinical world modeling beyond the limitations of report-based supervision.

Questions for authors
  1. Under what assumptions does the omission-tolerant score upper-bound true state accuracy? Can this interpretation be tested with image-grounded labels or a more conservative analysis stratified by finding type, time interval, and prior-evidence reliability?

  2. What evidence connects performance on this report-forecasting surrogate to the ability to simulate a patient's state or answer queries beyond the recorded trajectory?

  3. How many unique patients contribute the 1,419 queries? If patients contribute multiple queries, do the key differences remain significant under patient-clustered bootstrap? Given the 900 chest-radiography queries and the differing rankings in Table 2, do the conclusions also hold under namespace-macro averaging or after excluding chest radiography?

  4. The Auto-Research policy beats random search, but does it outperform standard AutoML or Bayesian optimization under the same action space and budget? Would such a comparison support the claim that the roughly 0.50 plateau reflects supervision and representation rather than insufficient search?

Comments on the paper

1 comment

We sincerely thank all reviewers and the AC for their careful and constructive feedback. The reviews identified important issues—including the validation of the omission-tolerant reading, patient-clustered statistics and the clarity of the benchmark specification—that we believe require substantial revision and additional experiments beyond the scope of this discussion period. We have therefore decided to withdraw the submission and will incorporate these suggestions into a substantially revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.