acceptodds
Under review as a conference paper at ICLR 2027

ClinEnv: An Interactive Multi-Stage EHR Environment for Agents

Abstract

Clinical practice is not the selection of an answer from enumerated options: a physician gathers heterogeneous information incrementally and commits to sequential, irreversible decisions under uncertainty. Static benchmarks cannot probe and existing interactive medical benchmarks each compromise on at least one of them. We present ClinEnv, an interactive benchmark that evaluates LLMs as attending physicians over real inpatient admissions under a paradigm we term Longitudinal Inpatient Simulation. Each case is automatically constructed into an ordered sequence of decision stages; at every stage the model must actively query four specialized agents before committing to medications, procedures, and diagnoses. ClinEnv scores both what the model decides, through deterministic ontology-grounded matching, and how it gathers information. Across seven models, the strongest reaches only 0.31 decision F1, and outcome quality is sharply decoupled from process quality. Difficulty concentrates in management decisions and later stages, where models recover discharge diagnoses far more reliably than management actions (0.51 vs. 0.17 F1) and continue to issue redundant queries as cases progress. ClinEnv makes this information-acquisition gap, invisible to outcome-only evaluation, directly measurable.

Primary area:
datasets and benchmarks
Keywords:
Clinical NLP, Large Language Model Agents, Interactive Benchmark, Electronic Health Records, Medical Decision Making

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CcLFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CLINENV, a benchmark constructed automatically from MIMIC-IV with an LLM agent extracting clinical decisions from discharge summaries, anchoring them to the event timeline, and segmenting each admission into ordered decision stages. Seven models are evaluated on 899 stages and the best reaches 0.306 decision F1, matches diagnosis targets better than medication targets, and shows little stage-level association between coverage and decision F1.

Critical strengths
  • The idea of building an automated EHR to case data construction pipeline with multiple decision stages is interesting.

  • The paper is well formulated and easy to follow.

Critical weaknesses
  • The motivation of the paper is unclear. It is unclear how an agent’s performance on CLINENV can be interpreted and transferred to real clinical practice to guide future agent development. The benchmark is curated from public datasets, and patient data are processed as static table dumps that are designed for retrospective analysis and are far from real clinical practice. In real practice, clinicians review the chart directly, examine the patient, and place orders via the standard EMR systems. However, in CLINENV the model can only ask LLM-simulated agents about a fixed record, cannot obtain any test the original team did not order, and is told the type and exact number of decisions to submit.

  • It is unclear how the quality of the automated EHR conversion pipeline is ensured, as most procedures are purely driven by LLMs, and the paper itself states that no clinician audit, inter-annotator agreement, or extraction error rate is available (Appendix D.2). In evaluation, only a small model (GPT-5.4-mini) is used to simulate all four information agents, and the LLM judges used for coverage and for ICD/ATC mapping are not specified.

  • It is unclear how well the developed data construction pipeline generalizes to other EHR data (e.g., private hospital or medical institute data) beyond the well-structured MIMIC datasets.

  • The evaluated models are outdated (Llama-3.1, Gemma-3) and do not cover a broad range of frontier models; only one proprietary family (GPT-5.4) is included. Each model is also run only once, on a subset of all the stages, without illustration of the sampling process.

  • It would be useful to include a few representative failure cases, manually analyze the failure reasons (e.g., genuine error vs. ground-truth error), and provide insights for future agent development.

Questions for authors

besides comments above,

  • How were the 899 evaluated stages selected? How many runs per model? Please provide the prompts for the model under test and the agents.

  • What fraction of diagnosis targets already appear in the past medical history or prior discharge summaries available through the patient/history agents? Is there any potential label leakage issue?

2 comments

Thank you for your feedback on the multi-stage construction pipeline and the paper’s clarity. We respond point by point below.

W1. Motivation and practical use.

ClinEnv targets clinical decision-support agents that gather information and make decisions as care progresses. Clinicians build a patient’s clinical picture step by step. ClinEnv evaluates this process through evidence acquisition and successive decisions along a recorded clinical course. Type/count hints define the decision scope, while replay keeps later patient states comparable across models. This helps identify weaknesses in information gathering and compare performance on diagnosis and management tasks.

W2. Construction quality, information agents, and judges.

We have engaged an ICU clinician to audit the constructed cases. The prepared sample contains 20 stages and 59 reference actions. The audit will examine source support, temporal anchoring, diagnosis grounding, medication-action labels, and clinical appropriateness. We will report the findings and uncertain cases in the follow-up. With one clinician, inter-rater agreement cannot be estimated.

History tools retrieve stored records directly, without an LLM. In the current release, gpt-5.4-mini generates patient/nurse responses, matches lab labels, and performs coverage judging and ICD reranking. The lab interface then returns the corresponding stored results. ICD retrieval uses text-embedding-3-small; drug mapping uses RxNorm/ATC lookup with an LLM normalization fallback. Prompts and configurations are provided separately. Repeated coverage judging will test consistency; clinical fidelity requires clinician review.

W3. Generalization beyond MIMIC.

The pipeline has not yet been validated on another hospital’s EHR. Transfer requires adapting local tables, terminology, timestamps, and note formats to the case schema and validating the resulting stages. We will state these requirements explicitly. We are also exploring RL training, but training gains and transfer to other datasets remain unverified. Agent transfer and pipeline portability require separate validation.

W4. Model coverage and single runs.

We agree that the model set is limited, particularly its coverage of proprietary model families. We will limit the conclusions to the seven evaluated systems. The planned coverage-judge repeats will not measure agent stability across runs.

W5. Representative failure cases.

The clinician will assess whether mismatches reflect unsupported reference actions, inconsistent information-agent responses, model errors, or clinically acceptable alternatives. In the EGD example, the displayed agent answers do not reflect a procedure present in the record. The follow-up will present the supporting records and clinical judgments for each case, with implications for agent development.

Q1. Sampling, runs, and prompts.

We stratified complete admissions by diagnosis categories and task labels, retaining all stages of each selected admission. The evaluation contains 229 patients, 393 admissions, and 899 stages from the 9,297-stage benchmark. All seven models used the same stages, with one run per model.

The full model-under-test, patient, nurse, lab, and judge prompts are provided separately. The supplement also includes tool schemas and explains stage hints and replay context. These are current release texts and configurations; historical per-run prompt snapshots have not been recovered.

Q2. Diagnosis exposure and leakage.

The pipeline excludes the current admission’s discharge-diagnosis table from agent views and limits history retrieval to earlier admissions. It retains HPI and PMH, where previously established diagnoses can be valid context.

We screened complete diagnosis-label strings in the current case export. We excluded 18 admissions whose stage or target counts differ from the evaluation summary. The remaining 375 admissions contain 1,983 diagnosis targets across 847 stages:

Available contextLiteral matchesFraction
Past medical history34/1,9831.71%
Prior admission summaries/full notes83/1,9834.19%
Either source90/1,9834.54%

All 1,194 linked prior stays in this subset ended at or before the current admission started. The screen misses abbreviations and synonyms, and the matches still require clinical and temporal review. These results describe the current export; they do not establish exposure or leakage rates for the historical runs. A shared cohort alone cannot establish that possible leakage leaves model rankings unchanged. We will report clinician-reviewed examples in the follow-up.


Summary.

We have clarified ClinEnv’s intended use, construction assumptions, sampling, and prompts, and provided an initial diagnosis-exposure screen. We will follow up with the clinician audit, representative cases, and coverage-judge repeatability results.

We invited an ICU clinician to assess ClinEnv's reference decisions and agent responses. We report the audit and case analyses below in response to W2 and W5.


W2. Clinical audit of the construction pipeline.

We invited one ICU clinician to review 20 stages from 20 patients and 20 admissions, covering 59 reference decisions. We sampled five stages for each combination of task type (diagnosis or management) and admission length (short or long), drawing from 847 eligible stages. We built a local web interface that displayed stage-visible records, role-specific views, reference decisions, discharge notes, and archived interactions, with fields for the clinician's judgments.

DimensionSupported / correctIdentified errorsAdditional evidence requiredNot applicable
Source support55220
Temporal anchoring291425
Decision type59000
Code/field consistency58100
Clinical appropriateness56210

The clinician judged 56/59 reference decisions (94.9%) clinically appropriate, including all 11 medication and procedure targets in the sample. At stage level, three were rated ready for evaluation, eight had minor issues, and nine required substantive revision.

The model sees observations preceding the target decision. In A06, it receives events 0–113 and predicts the catheterization recorded at events 114–115. These target records are withheld so they do not reveal the answer. We located all 11 sampled management anchors and the separate EGD case's iron-treatment anchor in the full source timelines. This source check locates the targets; the clinical audit evaluates their timing.

The same case illustrates a record-supported interaction. The following excerpts show the patient agent answering questions about symptoms at presentation and after treatment:

Model: "What symptoms are you experiencing?"

Patient agent: "I came in because I had chest pain. It started as a sharp pain on the left side of my chest and went up to my jaw. It felt a lot like the pain I had with my heart attack before. I also had some shortness of breath with earlier episodes, and I felt very cold with this one. I took nitroglycerin at home, but it didn’t help this time."

Model: "How are you feeling right now?"

Patient agent: "Right now I’m chest pain free. Earlier I did have some chest pressure, but it got better after nitro. The right-sided chest pressure I had on the floor also went away with morphine."

The clinician rated both responses as supported by the history of present illness, including the distinction between symptoms at presentation and subsequent treatment response.

W5. Representative failures and development implications.

CaseAudit findingDevelopment implication
A06: model decisionThe clinician rejected the proposed EGD because the stage-visible record did not support gastrointestinal bleeding.Require evidence for each proposed procedure.
A12: reference diagnosisThe clinician rejected the primary CNS lymphoma label and accepted secondary CNS involvement described in the history.Validate diagnostic qualifiers against the clinical narrative.
A17: reference fieldsThe pantoprazole description specifies IV administration, but the structured route is PO.Check consistency across descriptions and enriched fields.
A19: laboratory responseA blood-culture query returns urine and wound-test entries.Match specimen and test type together.

The interaction examples come from archived GPT-5.4 runs and are reported separately from the seven-model benchmark.


Summary.

The audit supports the clinical appropriateness of most sampled targets and distinguishes errors in references, information-agent responses, and model decisions. We will include the clinician evaluation, its protocol, and the case analyses in the camera-ready version.

Reviewer_TB7fOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the paper proposes a clinical environment that is interactive and allows llm agents to actively select actions over time, and indicates the lacks of current models and ways of evaluation.

Critical strengths

impactful topic, the whole clinical llm agent field needs work like this to go beyond static snapshot benchmarks.

Critical weaknesses
  1. whether the benchmark is clinically realistic really depends on how the patient, nurse, laboratory, and history agents are formed, which could inevitably inherit the bias from the agent's prior.

  2. it is unclear whether the low f1 reflect legitimate practice variation or indeed agent error.

Questions for authors
  1. how can we ensure the patient, nurse, laboratory, and history agents' clinical realism?

  2. how do you distinguish wrong actions from acceptable alternatives that were simply not the recorded choice? especially for management actions.

  3. can u discuss your work versus the recent clinical environment works that comes within method papers like this year's nips MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments, and this year's colm MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs.

  4. i noticed that all llms here are general domain, do we have comparison of medgemma, that could give us a very good way to isolate lack of domain prior.

  5. how do we isolate the confounding factor of long context forgetting when analyzing the failures? i didnt play with your actual dataset but expect these to be very long tasks.

some models like qwen also have well-known issues of overreasoning, although its not in the main table. I personally never use llama but suspect similar issues.

2 comments

Thank you for recognizing the need for clinical-agent evaluation beyond static snapshots. We appreciate your support and your questions about clinical realism and the meaning of our scores. We address each point below.

Q1. Clinical realism of the information agents.

Patient and nurse agents receive role-specific records. Their prompts require them to use only the supplied information and acknowledge what is missing. Nurse and lab records are limited by stage boundaries. The lab interface uses an LLM to match test labels before returning stored results. History tools retrieve prior-admission records directly, without an LLM.

The EGD discrepancy shows why these safeguards need validation. We have engaged an ICU clinician to review reference quality and selected agent responses. We have prepared 20 stages containing 59 reference actions for review. The follow-up will report the findings, including unsupported or contradictory responses.

Q2. Wrong actions versus acceptable alternatives.

The current score cannot fully separate errors from acceptable alternatives. ICD/ATC hierarchy matching awards partial credit for related diagnoses, procedures, and drugs; medication-action matching also checks the action type. LLMs assist terminology normalization, but do not judge whether an unrecorded treatment is clinically appropriate.

Partial credit reduces dependence on exact matches, but related ontology codes do not necessarily imply interchangeable clinical decisions. F1 therefore measures concordance with recorded care, not a clinical-error rate. The clinician will assess reference appropriateness and selected alternative actions separately. We will provide case-level explanations in the follow-up.

Q3. MedExAgent and MedAction.

MedExAgent and MedAction also study interactive clinical diagnosis. The table compares their environments with ClinEnv.

WorkCase sourceTask spanInformation accessDecision targetsProcess measures
MedExAgentDDxPlus/PMC/MedQADiagnostic episodePatient/examsDiagnosisTool quality/cost
MedActionPMC reportsDiagnostic episodeTest oracleDiagnosis/testsDTC/RAC
ClinEnv (ours)MIMIC-IV EHRAdmission stagesRole toolsDiagnosis/managementCoverage/resources

DTC and RAC measure disease-trajectory and reasoning–action consistency, respectively. MedAction's RL extension was added in arXiv v2 and is outside its peer-reviewed COLM version.

Both works train diagnostic agents to gather information actively. ClinEnv evaluates successive diagnosis, medication, and procedure targets within recorded inpatient care. Type/count hints and recorded-care replay keep the evaluation scope consistent across models. We will add this comparison to Related Work. We have not reported training gains in ClinEnv.

Q4. MedGemma and domain knowledge.

MedGemma would be a useful comparison. We are considering this experiment and will share results if it is completed. No results are available yet. Isolating the contribution of medical pretraining would require matched architecture, scale, and inference settings. The current results do not identify this contribution.

Q5. Long-context forgetting.

Figure 2(a) shows generally lower decision F1 at later management stages. Its horizontal axis is stage index, not context length. Cases, target actions, and sample sizes differ across indices, so this trend does not isolate forgetting. Section 6.2 therefore treats these curves as descriptive.

The runtime standardizes prior decisions using recorded care while retaining each model's information-seeking dialogue. A controlled comparison would keep the case, stage, available facts, model, and inference budget fixed. It would compare full dialogue history with a shorter history containing the same facts. Without this control, we cannot separate memory effects from clinical complexity or information access.

Q6. Overreasoning.

Overreasoning could also contribute to poor performance. Qwen was not evaluated, and we have no controlled evidence that overreasoning explains Llama's results. Testing this explanation would require trace analysis under matched inference budgets.


Summary.

Thank you again for supporting this direction and for helping us clarify how ClinEnv's results should be interpreted. We have addressed the environment design, scoring scope, related work, and long-horizon analysis. Our follow-up will provide clinician-reviewed cases and coverage-judge consistency results.

Thank you again for supporting the move beyond static clinical benchmarks. We invited an ICU clinician to assess ClinEnv's reference decisions and agent responses, and report the findings for Q1 and Q2 below.


Q1. Information-agent fidelity and clinical realism.

We invited one ICU clinician to review 20 stages from 20 patients and 20 admissions. We sampled five stages for each combination of task type (diagnosis or management) and admission length (short or long), drawing from 847 eligible stages. We built a local web interface that displayed stage-visible records, role-specific views, reference decisions, discharge notes, and archived interactions, with fields for the clinician's judgments. The interaction audit covered 260 responses from archived GPT-5.4 runs:

Clinician assessmentResponses
Supported by the records182
Incomplete45
Contradicted by the records17
No definitive judgment16

The clinician identified errors such as denying documented procedures, reporting expired orders as active, and mixing specimen types in laboratory retrieval. In the separate EGD case, the nurse denied procedures already present in its view.

ClinEnv withholds the current target action from pre-decision observations. In A06, the model sees events 0–113 and predicts the catheterization recorded at 114–115. We located all 11 sampled management anchors and the separate EGD case's iron-treatment anchor in the full timelines. This separation applies to the current target; it does not justify denying procedures already documented in earlier stages.

Q2. Clinical appropriateness and acceptable alternatives.

Decisions assessedClinically appropriateClinically inappropriateAdditional evidence required
Reference decisions5621
Archived model decisions4938

Of 20 model decisions classified as alternatives, 17 were judged clinically appropriate and three required additional evidence. In A04, the clinician accepted the model's "Anastomotic stricture of esophagus," documented in the radiology record, as a more specific diagnosis than the reference "Other digestive system complications." Both were judged clinically appropriate. A12 illustrates a separate issue: the clinician accepted the model's secondary CNS involvement diagnosis and rejected the reference's primary CNS lymphoma label. These judgments explain why concordance alone cannot determine clinical correctness.

Stage-level ratings were three ready for evaluation, eight with minor issues, and nine requiring substantive revision. These results cover the audited sample; the archived model decisions are reported separately from the seven-model benchmark scores.


Summary.

The audit directly addresses the distinction you raised: disagreement with recorded care can reflect an acceptable alternative or a reference error. It also assesses information-agent responses against their source records. We will include the clinician evaluation, its protocol, and the case analyses in the camera-ready version. Thank you again for your support and for helping us make the evaluation more clinically informative.

Reviewer_7wQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ClinEnv, an interactive EHR benchmark that reconstructs clinical decisions across stages of real hospital admissions. It evaluates seven models using decision concordance, information acquisition, and estimated resource-use metrics, finding stronger performance on diagnosis than on management decisions.

Critical strengths

The paper provides an automated environment for evaluating information acquisition and clinical decision reconstruction, with evidence across seven models.

Critical weaknesses
  • The novel contributions require further clarification. Existing benchmarks—LongMedBench, EHR-Complex, and PhysAssistBench—have already explored long-horizon decision-making, interactive clinical reasoning, and physician-patient-EHR collaborative tasks using real-world EHR data. The paper fails to compare its approach with these works, leaving the added value of ClinEnv—in terms of task design and evaluation capabilities—insufficiently articulated. The authors should provide such comparisons and explicitly state how their contributions extend beyond existing frameworks.

  • The target application scenarios and their relationship to the evaluation tasks remain unclear. While the model under evaluation acts as an agent that queries information and proposes clinical decisions, the tasks require it to reconstruct historical medical decisions—constrained by specified decision types and counts—within a progression dictated by actual patient records. The authors should clarify which types of clinical agents this setup targets, what practical needs it addresses, and how the stage-level F1 score serves as a valid performance metric in this context.

Questions for authors
  • The paper focuses primarily on using ClinEnv for evaluation, without demonstrating results from training models within this environment. Do the authors intend for it to serve as a training environment as well? It would be beneficial to assess whether performance gains from training in ClinEnv generalize to external benchmarks with similar tasks. This would help distinguish between genuine improvements in general clinical reasoning and mere adaptation to the specific tasks and scoring rules of ClinEnv.
  • Figure 1 requires improvements in readability and clarity. The current layout is cluttered with icons that fail to clearly illustrate key mechanisms, such as the provision of decision type and count hints or the injection of ground-truth clinical decisions. Furthermore, there is an inconsistency between the figure, which labels Phase E as “Classification & Deduplication,” and the main text, which refers to it as “Classification and Enrichment.” These terms should be aligned.
  • Processes such as decision extraction, temporal anchoring, diagnosis grounding, and the classification of medication actions rely on an LLM. The appendix notes that no clinician audit was conducted and that neither extraction error rates nor inter-annotator agreement estimates are available. An audit of a sample from the automatically constructed dataset is necessary.
  • There are discrepancies between the agents’ responses and the recorded clinical course. For instance, Appendix E indicates that the patient had already undergone an esophagogastroduodenoscopy (EGD) by Stage 1. Yet in the Stage 1 dialogue in Appendix F, the patient agent says they do not think they have had an endoscopy during the admission, and the nurse agent states that no such procedure is documented in the available bedside data. The authors should explain how these responses arise and whether the agents can accurately reflect events that have already occurred. Such conflicts could potentially skew the subsequent reasoning and judgments of the model being evaluated.
  • The scoring method for medication adjustments may inadvertently reward incomplete responses. According to Equation (8), the score for a medication adjustment is halved when the submission and ground truth explicitly specify opposing adjustment directions, such as increase versus decrease. However, no direction penalty applies if the direction is omitted. Consequently, omitting the adjustment direction could yield a higher score than providing an explicit but incorrect direction. The authors should justify this rule and explain how the omission of the adjustment direction affects model comparisons.
  • ICD mapping relies on embedding-based retrieval followed by LLM-based reranking, and information coverage is also assessed by an LLM. While the subsequent matching algorithm is deterministic, the overall score remains influenced by these model components. The authors should report the accuracy of the ICD mappings, the consistency of the coverage assessments, and whether potential scoring errors might alter the relative ranking of the models.
  • The selection process for the evaluation subset and the handling of failure cases remain unclear. Although the dataset contains 9,297 stages, the experiments use only 899 of them. The paper does not sufficiently explain how this subset was selected, the number of patients and admissions included, or whether complete clinical trajectories were preserved. Furthermore, the paper should clarify how cases are scored when the model fails to finalize its decisions within 60 turns, makes unsuccessful tool calls, or produces invalid submission formats.
  • The efficiency of information acquisition may be sensitive to how queries are formulated. Since the efficiency metric penalizes the number of tool calls, and a single call can encompass multiple queries, is it permissible to request all information visible to a specific agent in a single query?
  • Neither the main text nor the appendix provides sufficient detail regarding the sampling parameters and inference budgets used for the models under evaluation. They also do not specify the configurations of the embedding models, rerankers, and LLM-based coverage evaluator.
3 comments

Thank you for recognizing ClinEnv’s value for evaluating information acquisition and clinical decision reconstruction. We respond point by point below.

W1. Novelty and comparison with existing benchmarks.

We agree that these comparisons are needed. We group related task families and compare LongMedBench, EHR-Complex, PhysAssistBench, and ClinEnv separately below.

BenchmarkTask focusDataTask spanAccessPatient dialogueOutputOutcome metricsProcess metrics
MedQA, MedMCQA, PubMedQA, MultiMedQAMedical QAExams/abstracts/QAOne questionProvided✖️Choice/textAccuracy/ratings—
EHRSQL, MIMIC-SQLText-to-SQLEHROne querySQL generation✖️SQLSQL/execution—
FHIR-AgentBench, MedAgentBenchEHR tasksEHROne taskFHIR✖️Answers/ordersLLM/rulesRetrieval (some)
AgentClinic, MedDialBench, MAQuEDiagnostic dialogueCases/EHROne encounterPatient/agents☑️DiagnosisAccuracy/successCoverage/experience
HELPMed, AMIEHuman consultationClinical scenariosOne consultationHuman dialogue☑️Clinical assessmentOutcomes/ratingsDialogue analysis
MedCalc-Env, MedAgentGymMedical computationBiomedical tasksOne taskInputs/tools✖️Numbers/codeRules/executionEfficiency (some)
LongMedBenchLongitudinal reasoningEHRMulti-visitContext/RAG/memory✖️QA/ordering/decisionsTask scores—
EHR-ComplexDatabase reasoningEHRPatient/cohort querySQL/Python✖️AnswersExact matchTurns/failures
PhysAssistBenchPhysician assistanceEHR4 requestsFHIR/patient☑️AssistanceRubric/pass ratesImplicit rubric
ClinEnv (ours)Decision reconstructionEHRWithin-admission stagesRole agents☑️Diagnosis/managementConcordanceCoverage/resources

Grouped rows summarize related tasks; slash-separated entries cover variants across their members. Patient dialogue: ☑️ included; ✖️ absent. Process metrics cover information acquisition, interaction, or resource use; “—” indicates no separate measure listed.

LongMedBench emphasizes multi-visit memory and decisions, EHR-Complex database reasoning, and PhysAssistBench physician-request fulfillment. ClinEnv evaluates successive diagnosis and management decisions within an admission, with role-specific evidence gathering, recorded-care replay, and type/count hints. Its contribution is this controlled protocol and the joint measurement of decision concordance and information acquisition.

W2. Intended application and interpretation of F1.

Our target is clinical decision-support agents that gather information as care progresses, through questioning, record review, and test results. ClinEnv evaluates this stepwise process and successive decisions using real inpatient records. Type/count hints define the decision scope, while replay keeps subsequent patient states comparable across models. Stage-level F1 measures concordance with recorded actions; coverage measures information acquired. Clinically reasonable alternatives can receive low concordance, so F1 should not be interpreted as a clinical-error rate.

Q1. Training and external transfer.

Yes. We are exploring RL training in ClinEnv using its structured actions and stage-level feedback. Training gains and external transfer have not yet been established. External benchmark evaluation is needed to distinguish general clinical reasoning improvements from adaptation to ClinEnv’s tasks and scoring rules.

Q2. Figure 1.

We have revised Figure 1 to simplify the layout, show type/count hints and recorded-decision injection, and align Phase E with “Classification and Enrichment.” These changes will appear in the revised manuscript.

Q3-Q4. Clinician audit and EGD discrepancy.

We have engaged an ICU clinician to audit the constructed cases. The prepared sample includes 20 stages and 59 reference actions, covering source support, temporal anchoring, diagnosis grounding, medication-action labels, and clinical appropriateness. We will report the completed results, including actual denominators and uncertain judgments, over the next few days.

EGD appears in the recorded prior-stage context, but the displayed agent answers conflict with that record. The nurse view also contains endoscopy/colonoscopy records, so role-specific visibility alone does not explain the discrepancy. The clinician is reviewing the reference events, role views, and answers. We will report the findings and any correction separately.

Q5. Omitted medication-adjustment direction.

We agree that omitting direction can avoid the current penalty. There are 41 adjustment targets across 39/899 stages. To obtain a conservative bound, we set the entire stage-level F1 to zero for those 39 stages and leave all other stages unchanged:

ModelOriginal mean F1Updated mean F1
GPT-5.40.3063570.305801
GPT-5.4-mini0.2582290.256394
GPT-5.4-nano0.2541740.250058
Gemma-3-27B0.2284670.228245
Llama-3.1-70B0.2262690.226158
Llama-3.1-8B0.2016920.200913
Gemma-3-12B0.1972930.197070

For every adjacent pair, the higher-ranked model’s lower bound exceeds the lower-ranked model’s original score. Thus, penalty-only changes to adjustment-direction scoring cannot reverse the overall ranking when outputs, mappings, and other scoring components are fixed. These bounds do not give exact scores under a revised rule or establish the stability of medication-specific rankings.

Q6. ICD mapping and coverage-judge reliability.

ICD normalization uses embedding retrieval of 15 candidates and LLM reranking, followed by rule-based hierarchical matching. We will clarify that only the final matching step is deterministic. Mapping accuracy remains unquantified, and auditing reference items does not validate all mappings of model outputs. We will report repeated coverage assessments on fixed inputs in the follow-up. The ranking bound in Q5 applies only to adjustment-direction scoring, not to arbitrary scoring errors.

Q7. Evaluation subset and failure handling.

We stratified complete admissions by diagnosis categories and task labels, retaining all stages of each selected admission. The subset contains 229 patients, 393 admissions, and 899 stages from the 9,297-stage constructed benchmark. All seven models used the same stages and were run once.

At 60 turns, the released runtime returns the submissions collected so far for scoring. Missing required fields produce tool errors; malformed payloads can cause runtime failures. We will document these paths separately, since failures do not uniformly produce zero-scored stages.

Q8. Query bundling and efficiency.

Our protocol prohibits bundled questions. The prompt requires one tool call at a time and states: “Do not attempt to ask multiple questions or call multiple tools simultaneously.” The free-text interface does not enforce one fact per call, however, so broad requests could exploit the call-count metric. We will clarify this limitation and test such requests in future sandbox evaluations.

Q9. Model configurations and inference budgets.

The current released settings are:

ComponentModelTemperatureCompletion-token cap per call
Patient agentgpt-5.4-mini0300
Nurse agentgpt-5.4-mini0400
Lab agentgpt-5.4-mini0200
Coverage judgegpt-5.4-mini04096
ICD rerankergpt-5.4-mini0200
ICD embeddingtext-embedding-3-smallN/AN/A
History toolsDirect record retrievalN/AN/A

ICD retrieval returns 15 candidates; the stage limit is 60 turns. top_p and seed are not explicitly set in these generation calls. The model-under-test path uses the backend’s default token limit. These are release defaults; historical per-run overrides have not been recovered. The coverage repeat experiment will record the actual model identifier and settings.


Summary.

We have clarified ClinEnv’s evaluation target and bounded the adjustment-direction issue’s effect on overall ranking. We will follow up over the next few days with the clinician-audit findings and coverage-repeatability results.

We invited an ICU clinician to assess ClinEnv's reference decisions and agent responses. We report the findings below in response to Q3 and Q4.


Q3. Clinician audit and temporal anchoring.

We invited one ICU clinician to review 20 stages from 20 patients and 20 admissions, covering 59 reference decisions. We sampled five stages for each combination of task type (diagnosis or management) and admission length (short or long), drawing from 847 eligible stages. We built a local web interface for the review. It displayed stage-visible records, role-specific views, reference decisions, discharge notes, and archived interactions, with fields for the clinician's judgments.

DimensionSupported / correctIdentified errorsAdditional evidence requiredNot applicable
Source support55220
Temporal anchoring291425
Decision type59000
Code/field consistency58100
Clinical appropriateness56210

The clinician judged 56/59 reference decisions (94.9%) clinically appropriate, including all 11 medication and procedure targets in the sample. The two rejected targets concerned primary versus secondary CNS lymphoma and the remission status of multiple myeloma. At stage level, three were rated ready for evaluation, eight had minor issues, and nine required substantive revision.

Each target action follows the observation window provided to the model. In A06, the model receives events 0–113 and predicts catheterization and coronary angiography, recorded at events 114–115. These procedure records are withheld so they do not reveal the answer. We located all 11 sampled management anchors and the iron-treatment anchor from the separate EGD case in the full source timelines. The source check locates each target; the clinical audit evaluates its timing.

Q4. EGD discrepancy.

The targeted review confirmed that EGD and colonoscopy appear at events 59 and 60 in the nurse view. The clinician judged the nurse's denial to contradict these records. Both procedures had already occurred in the preceding stage, so withholding the current target does not explain their denial.

The clinician made no definitive judgment on the two patient denials, noting dementia in the record. We report this case separately from the 20-stage sample.


Summary.

The audit adds the clinician assessment missing from the original submission: most sampled targets were judged clinically appropriate, and the remaining label and timing issues are documented. It also identifies the EGD denial as an agent-response error, separate from the deliberate withholding of current targets. We will include the clinician evaluation, its protocol, and the case analyses in the camera-ready version.

Comments on the paper

1 comment

We thank the Area Chair and all reviewers for their careful feedback. We summarize the main clarifications and new clinical evidence from our responses below.

1. ClinEnv's contribution and intended use.

ClinEnv evaluates how agents gather evidence and reconstruct successive diagnosis, medication, and procedure decisions within real inpatient admissions. Role-specific queries, stage-level targets, and recorded-course replay allow information acquisition and decision concordance to be examined together. Our expanded benchmark comparisons clarify this task's relationship to existing longitudinal and interactive clinical evaluations. Type/count hints and replay define a controlled evaluation of recorded care; clinical correctness requires separate assessment.

2. New clinician audit.

We invited one ICU clinician to review 20 stages from 20 admissions, sampled across task types and admission lengths. A local review interface provided records, role-specific views, reference decisions, and archived interactions. The clinician judged 56/59 reference decisions (94.9%) clinically appropriate, including all 11 medication and procedure targets; two were inappropriate and one required additional evidence.

Among 260 information-agent responses from archived GPT-5.4 runs, 182 were supported by the records, 45 were incomplete, 17 contradicted the records, and 16 received no definitive judgment. At stage level, three were ready for evaluation, eight had minor issues, and nine required substantive revision. The audit therefore provides both clinical support for most sampled targets and specific evidence of construction and interface problems.

3. Clinical alternatives and the meaning of F1.

Of 20 archived model decisions classified as alternatives, the clinician judged 17 clinically appropriate. For example, “Anastomotic stricture of esophagus” was accepted alongside the broader reference “Other digestive system complications.” This directly addresses the concern that disagreement with recorded care can reflect a reasonable clinical alternative. We interpret F1 as record concordance and report clinical appropriateness separately. These archived interactions are assessed separately from the seven-model benchmark results.

4. Sampling and scoring clarifications.

We clarified that the evaluation covers 899 stages from 393 complete admissions and 229 patients, with one run per model, and provided the released prompts and configuration defaults. For the medication-adjustment direction issue, our conservative bound shows that increasing only this penalty cannot reverse the overall model ordering when outputs, mappings, and other scoring rules are fixed. This bound addresses that specific rule; Coverage-judge repeatability remains outstanding.


We will incorporate the clinician evaluation, audit protocol, case analyses, benchmark comparisons, and clarified evaluation setup into the camera-ready version. We respectfully ask the committee to consider these clarifications and new clinical evidence in its final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.