- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents CLINENV, a benchmark constructed automatically from MIMIC-IV with an LLM agent extracting clinical decisions from discharge summaries, anchoring them to the event timeline, and segmenting each admission into ordered decision stages. Seven models are evaluated on 899 stages and the best reaches 0.306 decision F1, matches diagnosis targets better than medication targets, and shows little stage-level association between coverage and decision F1.
Critical strengths
-
The idea of building an automated EHR to case data construction pipeline with multiple decision stages is interesting.
-
The paper is well formulated and easy to follow.
Critical weaknesses
-
The motivation of the paper is unclear. It is unclear how an agent’s performance on CLINENV can be interpreted and transferred to real clinical practice to guide future agent development. The benchmark is curated from public datasets, and patient data are processed as static table dumps that are designed for retrospective analysis and are far from real clinical practice. In real practice, clinicians review the chart directly, examine the patient, and place orders via the standard EMR systems. However, in CLINENV the model can only ask LLM-simulated agents about a fixed record, cannot obtain any test the original team did not order, and is told the type and exact number of decisions to submit.
-
It is unclear how the quality of the automated EHR conversion pipeline is ensured, as most procedures are purely driven by LLMs, and the paper itself states that no clinician audit, inter-annotator agreement, or extraction error rate is available (Appendix D.2). In evaluation, only a small model (GPT-5.4-mini) is used to simulate all four information agents, and the LLM judges used for coverage and for ICD/ATC mapping are not specified.
-
It is unclear how well the developed data construction pipeline generalizes to other EHR data (e.g., private hospital or medical institute data) beyond the well-structured MIMIC datasets.
-
The evaluated models are outdated (Llama-3.1, Gemma-3) and do not cover a broad range of frontier models; only one proprietary family (GPT-5.4) is included. Each model is also run only once, on a subset of all the stages, without illustration of the sampling process.
-
It would be useful to include a few representative failure cases, manually analyze the failure reasons (e.g., genuine error vs. ground-truth error), and provide insights for future agent development.
Questions for authors
besides comments above,
-
How were the 899 evaluated stages selected? How many runs per model? Please provide the prompts for the model under test and the agents.
-
What fraction of diagnosis targets already appear in the past medical history or prior discharge summaries available through the patient/history agents? Is there any potential label leakage issue?
2 comments
We invited an ICU clinician to assess ClinEnv's reference decisions and agent responses. We report the audit and case analyses below in response to W2 and W5.
W2. Clinical audit of the construction pipeline.
We invited one ICU clinician to review 20 stages from 20 patients and 20 admissions, covering 59 reference decisions. We sampled five stages for each combination of task type (diagnosis or management) and admission length (short or long), drawing from 847 eligible stages. We built a local web interface that displayed stage-visible records, role-specific views, reference decisions, discharge notes, and archived interactions, with fields for the clinician's judgments.
| Dimension | Supported / correct | Identified errors | Additional evidence required | Not applicable |
|---|---|---|---|---|
| Source support | 55 | 2 | 2 | 0 |
| Temporal anchoring | 29 | 1 | 4 | 25 |
| Decision type | 59 | 0 | 0 | 0 |
| Code/field consistency | 58 | 1 | 0 | 0 |
| Clinical appropriateness | 56 | 2 | 1 | 0 |
The clinician judged 56/59 reference decisions (94.9%) clinically appropriate, including all 11 medication and procedure targets in the sample. At stage level, three were rated ready for evaluation, eight had minor issues, and nine required substantive revision.
The model sees observations preceding the target decision. In A06, it receives events 0–113 and predicts the catheterization recorded at events 114–115. These target records are withheld so they do not reveal the answer. We located all 11 sampled management anchors and the separate EGD case's iron-treatment anchor in the full source timelines. This source check locates the targets; the clinical audit evaluates their timing.
The same case illustrates a record-supported interaction. The following excerpts show the patient agent answering questions about symptoms at presentation and after treatment:
Model: "What symptoms are you experiencing?"
Patient agent: "I came in because I had chest pain. It started as a sharp pain on the left side of my chest and went up to my jaw. It felt a lot like the pain I had with my heart attack before. I also had some shortness of breath with earlier episodes, and I felt very cold with this one. I took nitroglycerin at home, but it didn’t help this time."
Model: "How are you feeling right now?"
Patient agent: "Right now I’m chest pain free. Earlier I did have some chest pressure, but it got better after nitro. The right-sided chest pressure I had on the floor also went away with morphine."
The clinician rated both responses as supported by the history of present illness, including the distinction between symptoms at presentation and subsequent treatment response.
W5. Representative failures and development implications.
| Case | Audit finding | Development implication |
|---|---|---|
| A06: model decision | The clinician rejected the proposed EGD because the stage-visible record did not support gastrointestinal bleeding. | Require evidence for each proposed procedure. |
| A12: reference diagnosis | The clinician rejected the primary CNS lymphoma label and accepted secondary CNS involvement described in the history. | Validate diagnostic qualifiers against the clinical narrative. |
| A17: reference fields | The pantoprazole description specifies IV administration, but the structured route is PO. | Check consistency across descriptions and enriched fields. |
| A19: laboratory response | A blood-culture query returns urine and wound-test entries. | Match specimen and test type together. |
The interaction examples come from archived GPT-5.4 runs and are reported separately from the seven-model benchmark.
Summary.
The audit supports the clinical appropriateness of most sampled targets and distinguishes errors in references, information-agent responses, and model decisions. We will include the clinician evaluation, its protocol, and the case analyses in the camera-ready version.
Thank you for your feedback on the multi-stage construction pipeline and the paper’s clarity. We respond point by point below.
W1. Motivation and practical use.
ClinEnv targets clinical decision-support agents that gather information and make decisions as care progresses. Clinicians build a patient’s clinical picture step by step. ClinEnv evaluates this process through evidence acquisition and successive decisions along a recorded clinical course. Type/count hints define the decision scope, while replay keeps later patient states comparable across models. This helps identify weaknesses in information gathering and compare performance on diagnosis and management tasks.
W2. Construction quality, information agents, and judges.
We have engaged an ICU clinician to audit the constructed cases. The prepared sample contains 20 stages and 59 reference actions. The audit will examine source support, temporal anchoring, diagnosis grounding, medication-action labels, and clinical appropriateness. We will report the findings and uncertain cases in the follow-up. With one clinician, inter-rater agreement cannot be estimated.
History tools retrieve stored records directly, without an LLM. In the current release,
gpt-5.4-minigenerates patient/nurse responses, matches lab labels, and performs coverage judging and ICD reranking. The lab interface then returns the corresponding stored results. ICD retrieval usestext-embedding-3-small; drug mapping uses RxNorm/ATC lookup with an LLM normalization fallback. Prompts and configurations are provided separately. Repeated coverage judging will test consistency; clinical fidelity requires clinician review.W3. Generalization beyond MIMIC.
The pipeline has not yet been validated on another hospital’s EHR. Transfer requires adapting local tables, terminology, timestamps, and note formats to the case schema and validating the resulting stages. We will state these requirements explicitly. We are also exploring RL training, but training gains and transfer to other datasets remain unverified. Agent transfer and pipeline portability require separate validation.
W4. Model coverage and single runs.
We agree that the model set is limited, particularly its coverage of proprietary model families. We will limit the conclusions to the seven evaluated systems. The planned coverage-judge repeats will not measure agent stability across runs.
W5. Representative failure cases.
The clinician will assess whether mismatches reflect unsupported reference actions, inconsistent information-agent responses, model errors, or clinically acceptable alternatives. In the EGD example, the displayed agent answers do not reflect a procedure present in the record. The follow-up will present the supporting records and clinical judgments for each case, with implications for agent development.
Q1. Sampling, runs, and prompts.
We stratified complete admissions by diagnosis categories and task labels, retaining all stages of each selected admission. The evaluation contains 229 patients, 393 admissions, and 899 stages from the 9,297-stage benchmark. All seven models used the same stages, with one run per model.
The full model-under-test, patient, nurse, lab, and judge prompts are provided separately. The supplement also includes tool schemas and explains stage hints and replay context. These are current release texts and configurations; historical per-run prompt snapshots have not been recovered.
Q2. Diagnosis exposure and leakage.
The pipeline excludes the current admission’s discharge-diagnosis table from agent views and limits history retrieval to earlier admissions. It retains HPI and PMH, where previously established diagnoses can be valid context.
We screened complete diagnosis-label strings in the current case export. We excluded 18 admissions whose stage or target counts differ from the evaluation summary. The remaining 375 admissions contain 1,983 diagnosis targets across 847 stages:
All 1,194 linked prior stays in this subset ended at or before the current admission started. The screen misses abbreviations and synonyms, and the matches still require clinical and temporal review. These results describe the current export; they do not establish exposure or leakage rates for the historical runs. A shared cohort alone cannot establish that possible leakage leaves model rankings unchanged. We will report clinician-reviewed examples in the follow-up.
Summary.
We have clarified ClinEnv’s intended use, construction assumptions, sampling, and prompts, and provided an initial diagnosis-exposure screen. We will follow up with the clinician audit, representative cases, and coverage-judge repeatability results.