acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Not Acting: WARD Diagnoses the Evidence-to-State-to-Action Gap in Longitudinal Clinical Agents

Abstract

Longitudinal clinical decision-making requires agents to translate accumulating evidence into updated patient-state judgments and appropriate actions over time. However, benchmarks centered on isolated responses or final task outcomes provide limited insight into intermediate failures in evidence use, state updating, and action selection. We introduce WARD, a benchmark for evaluating how clinical agents translate accumulating evidence into patient-state judgments and timely, safe actions. WARD comprises 4,560 clinical tasks across seven clinical output domains, organized into 240 checkpoints from 40 longitudinal clinical trajectories spanning 95 hospitalizations. Its Longitudinal Track evaluates these tasks under controlled cumulative evidence. Its Interactive Track evaluates end-to-end, tool-mediated clinical management across 271 verifiable evidence items and 141 reference transitions. Across 12 models with scorable checkpoint outputs, evidence recognition consistently outpaces downstream state and action decisions, with linked-evidence nonresponse reaching 74.47–90.90%. Models also struggle to retain and update information over time, while in the Interactive Track only 11.46% of executed actions advance a reference transition. These results reveal a persistent evidence-to-state-to-action gap and highlight the need for process-level evaluation of longitudinal clinical agents. Code and data are available at https://anonymous.4open.science/r/WARD-7C3A/.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Clinical Agents, Longitudinal Clinical Reasoning, Electronic Health Records, Process-Level Evaluation, Interactive Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZbsyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is a benchmark that tests how longitudinal medical agents translate accumulating evidence into actions, i have no question about the soundness and comprehensiveness of this work but the its current form is preventing me from fully getting what the authors want to present.

Critical strengths
  1. Trendy topic as only agents can really help transform healthcare.

  2. the scale and dimension of the dataset is comprehensive enough for an iclr benchmark.

Critical weaknesses

===presentation issues===

  1. before evaling "longitudinal clinical agents", i think its better we clarify what these "agents" are, more concretely their action. Are they "agents" because they are expected to give task instructions, treatment plans? or are they agents because they should interact with EHR, python and other tools? Or a combination of both?

You can back it up with real clinical use case examples.

  1. Severe presentation issue. Whoever write the results section, obviously just copied what claude/codex gave them over. This leads to several issues: a) overload of numbers, invented phrases. If you don't buy my comments, read last year's best paper. https://arxiv.org/pdf/2505.06120 see how many numbers they repeat from the figs/tables v.s. how many your current paper does. It's both a torture to audience and a waste of space.

b) Because you wasted so much space with meaningless repetition of numbers that already in table/fig , the sec 6.3 explanation for "Pending-prerequisite recovery" or " prerequisite Pending" is non-existent. Therefore I have no idea what they mean in your context.

c) similar issues for sec 7.

in general i like the topic very much but " Generally understandable, but requires substantial effort in places." is the highest score I can give for presentation, and this poor presentation limits me from "I gained a substantially deeper understanding of the problem or question."

===content issues==== i will specify in the question for authors section.

Questions for authors

===presentation issues===

  1. Can you please remake fig 1 and fig 2 when you arxiv/ after acceptance, or if unlucky in ICLR, moving to other venues? These are examples of poor presentation. Literally 2 most important figures in paper and you are literally storming it with full text. Current form is no better way than just writing them down in bullet points in text. "Let the figure do the talk" not let the audience read the text.

  2. can we please have a human researcher write the results section? the purpose is to help me understand your findings and only then is possible for me to give fair scores on presentation and informativeness.

===content issues====

  1. this is a medical related topic, and i see both medgemma and other general domain llms are being compared. however, i (may have missed due to appendix being too long) dont see a head to head same family medical v.s. general models? like medical gemma has its general domain sister right?

  2. in results u mentioned about repeated or prolonged stagnation, are these like overthinking loops (this can be especially an issue for qwen) or just the llm not making any substantial moves despite being aware of the "prereq not completed"? if so does it mean its a long context issue or forgetting issue, or model capability issue? i want to clearly understand why they fail to continue, like are the models aware of these pereq and still refuse to resolve?

same questions confound both 6.2 and 6.3.

5 comments

W1: Define what the clinical agents do in each track, using a concrete clinical example.

We thank the reviewer for this question. In WARD, the term refers to a model that uses the progressively available records of the same patient to assess the current clinical state, determine the next action, and identify unresolved clinical and safety prerequisites. “Longitudinal” refers to how the available evidence and appropriate decisions change over time.

In the Longitudinal Track, each checkpoint provides the cumulative clinical record available up to that time. The model generates each response independently, without tool access or access to its earlier responses. We describe this setup in Section 4. and Appendix C.1. For example, the input at Checkpoint 8 includes a negative axillary cytology result, while the pathology report for the breast lesion itself is not yet available. The model must submit a structured response that identifies the key evidence, states that the diagnosis remains unresolved, recommends awaiting the pathology result, and explains why the evaluation should continue.

In the Interactive Track, the model decides what to query or do next and makes one tool call per turn. The interface supports patient communication, EHR search and retrieval, order placement, procedures, treatment, and care transitions. We describe the interaction protocol in Section 4.2 and list the tools in Appendix C.2, Table B2. In the same scenario, the model could first query the EHR for the breast lesion pathology report. If the tool indicates that the report is still unavailable, the model uses the current record to decide what else it can check. If the tool returns the report, the model uses the result to choose its next action.

W2 & Q2: Revise the Results and §7 to reduce repeated numbers, clarify Pending and recovery, and focus on what the findings mean.

We thank the reviewer for the interest in this research question and for specific suggestions on how to analyze and present the results. We carefully read the paper you recommended. Its Results section first uses key aggregate results to establish the overall trend, then uses comparisons and follow-up analyses to show which aspects of performance decline and whether the decline depends on the experimental setup. We also want to clarify how we originally structured our analysis.. We aimed to state the main finding, support it with data, and then discuss possible explanations. The main text therefore includes cross-model summaries and correlation analyses, but it also repeats some values already shown in figures and tables, making the key comparisons and their implications less prominent.

The original manuscript explains the relevant metrics and environment feedback, but the definitions and calculation details are spread across the Methods, figure captions, and appendices. This makes them harder to look up while reading the results. For example, Table B3 and Appendix C.3 explain when the environment returns Prerequisite pending. Appendix D.4 and Table C3 explain how Pending-prerequisite recovery is calculated. We will explain these terms where we first discuss the corresponding results and keep only the numbers needed to support the key comparisons in the text.

Following the approach in the paper you suggested, we will organize our results around a central question: to what extent can models use the evidence they recognize to make appropriate clinical judgments and take appropriate follow-up actions? §6.1 first examines whether models make the corresponding judgments or recommend the appropriate actions after recognizing relevant evidence. §6.2 then asks whether models revise their judgments when needed and retain those that remain appropriate as the clinical record grows. §6.3 compares action recommendations at individual checkpoints with actual progress over successive interactions. It also uses a trajectory example to show how a task can stall again after an earlier blockage has been resolved.

In §7, we will follow the approach to analyzing model behavior in Appendix F of the recommended paper. We will spend less space recounting the tool calls and responses shown in Figure 5 and use the audit of behavior after Pending feedback, which you suggested in Q4, to interpret the cases. In Figure 5(a), for example, the environment blocks a premature surgery request, after which the model checks the consent record and proceeds. We will use this case to illustrate how a model adjusts its actions in response to feedback and put it in the context of the audit. Models often attempt to check prerequisites after being blocked, but those attempts do not always lead to further progress.

We provide the revised text of §6.3 below:

6.3 Interactive Execution and Sustained Progress

Among interactive trajectories that did not reach a terminal state, 76.92% exhibited repeated or prolonged stagnation. To examine how stagnation arose, we analyzed 1,204 executed tool calls (Figure 4a), of which 11.46% advanced the reference path, approximately half were valid explorations that returned no recorded outcome, and 27.66% were blocked by unmet prerequisites. The environment returns Pending when it identifies a candidate transition for a request but the prerequisites for executing that transition remain unmet. We tracked 78 Pending points and found that 45 were followed by at least one completed reference transition. At 19 of those 45 points, later actions advanced the workflow without completing any of the original request’s candidate transitions.

To determine which step a subsequent action advanced and whether that step resolved the original blockage, we executed a candidate action proposed by an independent auditing model from the environment state at which the request received Pending. When the action produced progress, we retried the original request. Among the 26 blocking points where a single action produced progress, 16 completed a preparatory step required for a direct prerequisite, 4 satisfied one direct prerequisite while others remained unmet, and 5 allowed the original request to succeed on retry. In the original trajectories, at 12 of the 19 points where only other transitions were completed, the model made progress and then issued another request that matched one of the original candidate transitions, again receiving Pending. These results show that a successful action often resolves only part of the dependency chain. After making progress, the model must determine which prerequisites remain unmet and when to retry the original request.

However, even after resolving a prerequisite blockage, a model can enter a prolonged period of stagnation when a tool returns no result. In WARD-001, GPT-5.6 SOL satisfied the imaging prerequisite, completed the blocked biopsy, and obtained the pathology report. It then received no result for an immunohistochemistry request and immediately repeated it. The 31 calls after pathology retrieval comprised 20 explorations with no recorded outcome and 11 calls targeting completed steps. None advanced the reference path or returned Pending.

Reviewer_pKGkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WARD, a benchmark for testing how clinical agents turn growing evidence into patient state assessments and suitable actions. WARD is built from real clinical records. It has 40 care trajectories covering 95 hospitalizations and 240 decision checkpoints. Physicians reviewed all reference annotations.WARD offers two evaluation tracks. The first assesses each checkpoint on its own using cumulative evidence. The second requires interactive tool use. The agent must acquire evidence, resolve prerequisites and complete tasks in order.The evaluation covers evidence recognition and use, patient state assessment, action execution, safety, stopping decisions and progress along the care trajectory.For reliability, the authors compare results against clinician baselines, calibrate the semantic judge, run sensitivity analyses and validate the environment.

Critical strengths

1.WARD checks whether patient assessments and recommended actions match the evidence. This reveals clinical contradictions that a correct final diagnosis can hide. 2.Its two tracks test clinical judgment when evidence is given and step by step execution when agents must use tools. They cover both deciding what to do and carrying out the steps.

Critical weaknesses

1.Evidence Utilization requires clinical predictions to match the reference before their evidence links count. This limits its ability to separate clinical prediction errors from evidence linking errors. 2.The reference annotations are not directly validated. Physician agreement before adjudication is not reported and the agreement statistics for semantic judging and trajectory quality ratings measure different evaluation stages.

Questions for authors
  1. When physicians independently reviewed at least 40% of the checkpoints Please report their agreement before adjudication on patient states, critical actions and dependencies and list the omissions found in the preliminary annotations. 2.Does the reported human adjudication cover the final unmatched patient states and actions? Among these unmatched items, what share are real clinical errors or critical omissions, what share are incomplete answers with safety risks and what share are acceptable answers that get no credit because of reference coverage or wording detail? 3.How much of the score loss in Evidence Utilization comes from unmatched clinical targets and how much comes from wrong evidence selection or links? To separate these sources, please report evidence link precision and recall for the cases where the clinical targets are correctly matched.
  2. For targets that need several pieces of evidence or prerequisites, does the nonresponse rate stay high even when physicians confirm that enough supporting evidence was found? 5.The reference policy uses the hidden dependency graph to finish the interactive tasks. How can the key prerequisites be inferred from the records available at each decision point? Could you provide physician execution results under the same observations, tools and feedback, to show that these workflows can be found and completed using only permitted information?
6 comments

W1 & Q3: Separate Evidence Utilization losses from unmatched clinical targets and incorrect evidence links; report link precision and recall on matched targets.

We thank the reviewer for pointing out that Evidence Utilization (EU) is affected by both target matching and evidence linking. In the camera-ready version, we will jointly report target coverage, conditional link quality, and error decomposition in the appendix.

To distinguish the effects of unmatched clinical targets from evidence-linking errors on EU scores, we conducted an additional decomposition analysis across all 12 models and 240 checkpoints. First, to assess evidence-linking quality after target matching, we computed link Precision, Recall, and F1 only for clinical targets matched by the original scorer, alongside reference-target matching coverage. All metrics are macro-averaged across cases.

ModelReference-target matching coverageOriginal EU F1 (%)Conditional link P (%)Conditional link R (%)Conditional link F1 (%)
GPT-5.6 SOL25.6418.8788.8458.3769.40
GPT-5.6 Terra21.1814.4288.3244.7858.38
Kimi K324.2617.3683.7967.1373.99
GLM-5.219.4615.0786.1776.1580.43

To further quantify where errors occur, we decomposed the original EU false positives (FPs) and false negatives (FNs) into five categories, distinguishing invalid outputs, unmatched targets, and missing or incorrect links on matched targets:

Error sourceGPT-5.6 SOLGPT-5.6 TerraKimi K3GLM-5.2
Reference links missed due to invalid checkpoint outputs (FN)6451,349321688
Reference links missed due to unmatched reference targets in valid outputs (FN)8,2157,9878,6148,893
Links attached to unmatched predicted targets (FP)4,2772,1007,8408,489
Missing links on matched targets (FN)1,3021,4211,019607
Incorrect links on matched targets (FP)209148383263

W2 & Q1: Clarify whether physicians reviewed annotations without seeing GPT drafts, and report pre-adjudication agreement and draft omissions.

We thank the reviewer for asking us to clarify the physician review process and report pre-adjudication agreement and potential omissions in the draft annotations.

Twenty physicians participated in the annotation process. Ten primary reviewers covered all 240 checkpoints, and six secondary reviewers reviewed 145 of them (60.42%). Secondary reviewers had access to both the GPT-4o draft and the primary reviewer’s annotations and reviewed them against the clinical records available at the checkpoint. Disagreements were resolved by one of four adjudicating physicians who had participated in neither review stage. We will clarify which materials secondary reviewers could access and report the coverage of each review stage in Section 3.1 and the appendix of the camera-ready version.

To quantify pre-adjudication agreement under this review protocol, we matched primary and secondary annotations one-to-one across the 145 checkpoints and report both case-macro F1 and pooled item-level F1.

DimensionPrimary review itemsSecondary review itemsMatched itemsCase-macro F1 (%)Pooled item-level F1 (%)
Patient State2,3271,4111,31273.2370.20
Critical Actions87781361171.9672.31
Dependencies48953228356.8755.44

To investigate potential omissions, we compared both sets of annotations with the archived candidate annotations for the same checkpoints. The initial comparison flagged 62 potentially important items. A second item-by-item review found that 25 were already covered by equivalent or broader wording, leaving 37 candidates for clinical confirmation.

DimensionAlready covered in the archivePotential omissions awaiting clinical confirmation
Patient State05
Critical Actions1214
Dependencies1318
Total2537

Q2: Audit final unmatched outputs and quantify clinical errors, unsafe incomplete answers, and clinically valid responses missed by scoring.

We thank the reviewer for highlighting the need for human adjudication of final unmatched outputs. We sampled from GPT-5.6 SOL, GPT-5.6 Terra, and Kimi K3: for each model and each dimension (Patient State and Next Actions), we selected 10 unmatched predictions and 10 unmatched reference targets, yielding 120 items. We reviewed the complete responses alongside the reference annotations and the clinical records visible at each checkpoint.

The unmatched predictions were classified as follows:

ClassificationPatient State, n=30Next Actions, n=30
Substantive clinical error13
Incomplete clinical content, missing important elements or conditions2117
Fully expressed an existing reference target, but not matched by the scorer54
Clinically reasonable additional content, further assessed for acceptability as an alternative answer36
Total3030

The unmatched reference targets were classified separately:

ClassificationPatient State, n=30Next Actions, n=30
Fully covered in the complete response, but not matched by the scorer42
Partially covered, with important content or conditions missing2223
Completely omitted45
Total3030

To address the reviewer’s safety concern, we further assessed whether incomplete content posed a safety risk:

Audit groupSafety classificationPatient StateNext Actions
Incomplete unmatched predictionsSafety risk identified83
No clear safety risk identified1314
Subtotal2117
Partially covered unmatched reference targetsSafety risk identified43
No clear safety risk identified1820
Subtotal2223

We also distinguished critical from noncritical omissions:

Omission classificationPatient StateNext Actions
Critical omission23
Noncritical omission22
Total complete omissions45

Finally, we distinguished acceptable alternative answers from reasonable additional content that did not substitute for the current target:

ClassificationPatient StateNext Actions
Acceptable alternative answer for the current task13
Clinically reasonable additional content, but not an alternative to the current target23
Total reasonable additional items36
Reviewer_pLuUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WARD, a benchmark evaluating how clinical agents translate accumulating evidence into patient-state judgments and appropriate actions. It contains 40 longitudinal trajectories spanning 95 hospitalizations, with 240 checkpoints and 4,560 field-level evaluation tasks. GPT-4o-generated reference annotations are reviewed by a panel of 20 physicians.

WARD provides two evaluation tracks: a Longitudinal Track assessing clinical judgments at independently prompted checkpoints, and an Interactive Track evaluating tool-mediated execution across ten trajectories.

Experiments reveal substantial differences between evidence recognition and downstream state/action performance, frequent inconsistencies across checkpoints, and limited progress toward reference transitions during interactive execution.

Critical strengths

WARD presents a valuable benchmark with substantial clinical annotation and thoughtful process-level evaluation. I particularly appreciate its focus on temporal evidence availability and the distinction between clinical judgments and executed actions. Key strenghts:

  • Valuable dataset and clinical annotation. Real longitudinal inpatient records, clinically meaningful checkpoints, and extensive physician involvement provide a useful foundation for evaluating clinical agents.

  • Careful temporal and process-level design. Distinguishing when evidence occurred from when it became available is particularly important. Evaluating evidence use, state judgments, action decisions, and stopping behavior provides more diagnostic information than outcome-only scoring.

  • Substantial evaluation effort. The authors provide detailed metric definitions, judge calibration, clinician baselines, sensitivity analyses, and interactive failure audits. The case studies illustrate clinically meaningful errors that terminal-outcome evaluation may overlook.

Critical weaknesses
  1. The evidence-to-state-to-action gap is not sufficiently separated from differences in metric difficulty (Sections 5–6).

Evidence Recognition relies primarily on structured record matching, whereas Patient State and Next Actions require matching more open-ended clinical predictions. Consequently, their F1 scores are not directly comparable, and the observed decline does not necessarily isolate a reasoning failure.

The linked-evidence nonresponse metric is more targeted, but lacks a comparison with the unconditional downstream miss rate. Moreover, the clinician baseline omits Evidence Recognition and reports relatively low absolute State and Action F1 scores (28.15% and 35.01%), raising questions about reference coverage and task difficulty.

Please report clinician Evidence Recognition on the matched subset and compare downstream response rates conditional on matched versus unmatched evidence. A clinician audit of clinically valid predictions not captured by the reference would further strengthen the interpretation.

  1. The Longitudinal Track measures consistency rather than persistent memory or autonomous updating (Section 4.1).

Each checkpoint is evaluated independently, with no access to the model's previous responses. Therefore, subsequent recovery or regression reflects changes in predictions under evolving input contexts, rather than an agent maintaining or revising its own memory.

The authors acknowledge this distinction in the appendix, but the abstract and conclusions make broader claims about retaining and updating information over time.

I suggest framing these results as longitudinal judgment consistency. A repeated-evaluation control on identical checkpoint inputs would also help distinguish evidence-driven changes from ordinary output variability.

  1. Reference annotation reliability and acceptable alternatives require further validation (Section 3.1).

The physician-reviewed annotation pipeline is a strength, but pre-adjudication agreement and aggregate annotation revision statistics are not reported. Since GPT-4o generated the initial reference drafts, it would be useful to establish how extensively clinicians modified them and whether independently constructed references produce comparable evaluations.

Please report inter-physician agreement on double-reviewed checkpoints and summarize annotation acceptance, revision, and rejection rates. A targeted audit of clinically reasonable predictions rejected by the matching pipeline would help assess reference completeness.

  1. Interactive results conflate clinical planning, reference-graph coverage, and tool-protocol failures (Sections 4.2 and 6.3).

The Interactive Track evaluates only ten trajectories with four models and one run each. Nine Gemini trajectories terminate due to invalid tool outputs, while four Kimi trajectories are transport-censored. These failures complicate the interpretation of the reported stagnation and progress metrics.

Furthermore, the authors identify clinically reasonable actions that do not advance the reference graph. Figure 5(c) also illustrates surgery advancing the environment state before the agent retrieves the required pathology, suggesting that reference progression and safe, evidence-grounded action sequencing are not equivalent.

Please clarify the resolver's matching and prerequisite rules, distinguish clinical failures from execution-protocol failures in the main results, and explain how clinically appropriate alternative actions and unsafe-but-advancing actions affect scoring.

  1. Action granularity and budget design (Section 4.2).

The Interactive Track restricts agents to one tool call per turn, with a finite action budget shared by evidence retrieval and clinical interventions. While this simplifies sequential execution and prerequisite checking, the design may affect measured trajectory progress and stagnation. The 60-action budget extension does not isolate this effect. Please justify the single-call design and clarify whether results are sensitive to action granularity or to separately budgeting read-only and state-changing operations.

Minor Comments

  • Clarify the handling of schema-invalid outputs, particularly for models with substantial invalidity rates, so clinical performance can be distinguished from format compliance.

  • Correct reporting inconsistencies, including the Interactive Track's 20- versus 40-action budgets and the checkpoint numbering in Figure 5(b).

  • Clarify the data-release terms and access safeguards for the de-identified institutional EHR trajectories.

Questions for authors

see weakness section

7 comments

W1: Report clinicians’ evidence recognition performance, compare downstream responses with and without evidence recognition, and audit missed matches of reasonable answers.

Thank you for the suggestion! We initially presented F1 scores across the three dimensions side by side to show model performance on different tasks. In the revision, we will use evidence recognition and target matching for the same clinical target as the primary basis for analysis.

To directly examine the relationship between evidence recognition and downstream responses, we inspected each reference state and critical action with evidence links. We first determined whether the model recognized at least one linked piece of evidence, and then whether it matched the target.

The results show that even when linked evidence was recognized, models often failed to match the corresponding clinical target. For Patient State, the hit rate was 4.36 percentage points higher when evidence was recognized, but reached only 15.77%. For Critical Actions, at least one linked piece of evidence was recognized for approximately 92% of targets, yet the hit rate was only 15.26%, with no improvement associated with evidence recognition. We will present the conditional results for states and actions separately in the camera-ready version.

Target TypeEvidence Recognition StatusNumber of Model–Target PairsDownstream Hit Rate (%)
Patient StateAll42,34515.11
Patient StateAt least one linked piece of evidence recognized35,07115.77
Patient StateNo linked evidence recognized7,27411.41
Critical ActionsAll7,17615.10
Critical ActionsAt least one linked piece of evidence recognized6,59915.26
Critical ActionsNo linked evidence recognized57718.66

To complete the clinician baseline on the same subset, we added an Evidence Recognition evaluation on the 50 checkpoints from 27 cases used in the original baseline. For each dimension, F1 was first calculated for each case based on item matching and then averaged with equal weight across cases.

ClinicianEvidence Recognition (%)Patient State (%)Next Actions (%)
Doctor 156.5330.0434.43
Doctor 252.1228.2943.21
Doctor 371.4326.1127.40
Average60.0328.1535.01

To examine whether clinically reasonable answers were missed by the scorer, we sampled 30 unmatched predictions and 30 unmatched reference targets for each of Patient State and Next Actions, and reviewed the complete answers alongside the corresponding reference content. Answers often mentioned the relevant clinical condition or action direction, but did not fully express the necessary state elements, execution conditions, or timing required by the target.

Classification of Unmatched PredictionsPatient StateNext Actions
Substantive clinical error13
Incomplete clinical content, missing important elements or conditions2117
Fully expressed an existing reference target, but not matched by the scorer54
Focused on reasonable additional content not confirmed to be a required target36
Classification of Unmatched Reference TargetsPatient StateNext Actions
The complete answer covered the target, but the scorer did not match it42
Partially covered, but still missing important content or conditions2223
Completely omitted45

W2: Clarify the rationale for independent checkpoint evaluation and assess the contribution of output variability to longitudinal changes.

We thank the reviewer for highlighting the important distinction between independent checkpoint evaluation and memory capabilities. We agree that the scope of capabilities assessed by this design should be defined more clearly in the main text.

Regarding the evaluation design and its scope, we use independent checkpoint evaluations to provide different models with consistent, controlled patient information at each time point and prevent earlier erroneous responses from entering subsequent contexts and affecting comparisons of current judgment. Our goal is to assess whether models make appropriate judgments at each checkpoint and exhibit changes consistent with the evolving clinical evidence for the same patient.

We acknowledge that some wording in the abstract and conclusion may suggest memory capabilities beyond what this design assesses. We will revise these descriptions consistently. For example, replacing “updating its own memory” with “forming a current patient-state assessment from updated visible evidence.”

Following your suggestion, we added a fixed-input repeated-evaluation control. We selected one pair of adjacent checkpoints from each of 40 cases and independently generated three responses per checkpoint input, yielding 720 responses across three models. We compared hit-status flip rates between repeated responses to fixed inputs and responses to adjacent checkpoint inputs, using evidence targets shared across both checkpoints with unchanged reference labels, and macro-averaged the rates across cases. We will extend this experiment to all models in the camera-ready version.

For Evidence, we measured changes in whether each “evidence ID + role category” target was matched, restricting the analysis to shared targets with unchanged reference annotations:

ModelIncluded casesFixed-input flip rate (%)Adjacent-input flip rate (%)
Qwen3.7-Max262.5119.22
Kimi K3216.2130.00
GLM-5.2175.0711.00

For Stop, we measured changes in the model’s predicted stopping-decision category among cases where the reference stopping decision changed:

ModelIncluded casesFixed-input category flip rate (%)Adjacent-input category flip rate (%)
Qwen3.7-Max207.5060.00
Kimi K3169.3846.88
GLM-5.2147.1446.43

W3: Report physician agreement and annotation revision rates, and audit clinically valid unmatched predictions.

We thank the reviewer for this suggestion. We compiled review coverage and item-level revision statistics and will report them in Section 3.1 and the appendix of the camera-ready version.

Twenty physicians participated in the annotation process. Ten physicians served as primary reviewers, collectively reviewing all 240 checkpoints. Six physicians served as secondary reviewers, reviewing 145 of the 240 checkpoints (60.42%). Disagreements between primary and secondary reviewers were resolved by one of four adjudicating physicians who had participated in neither review stage.

The table shows how reviewers handled the GPT-4o draft annotations by type.

Annotation typeDraft itemsAccepted without modification, n (%)Accepted after revision, n (%)Rejected or removed, n (%)
Clinical Claims1,019713 (69.97%)255 (25.02%)51 (5.00%)
Patient State4,8803,318 (67.99%)1,269 (26.00%)293 (6.00%)
Actions823469 (56.99%)305 (37.06%)49 (5.95%)
Total6,7224,500 (66.94%)1,829 (27.21%)393 (5.85%)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary of Contributions and Rebuttal Responses

Dear Area Chair and All Reviewers,

Greetings! We sincerely appreciate the reviewers' time and thoughtful comments.

We would like to take this opportunity to clarify and highlight our contributions. In real-world clinical practice, patient evidence becomes available over time, requiring models to update their assessments of patient state and take timely, safe actions. Static benchmarks typically evaluate a single response, while existing interactive benchmarks focus largely on final outcomes. They can therefore miss errors in evidence use, state updates, and actions, while scoring clinically sound and risky trajectories equally. To address these gaps, we propose WARD:

  • Clinical tasks with temporal information boundaries. WARD reconstructs 40 longitudinal trajectories from real inpatient records, spanning 95 hospitalizations, 240 clinical decision checkpoints, and 4,560 tasks. Each checkpoint provides only records available at that time; physicians review reference judgments and actions and link them to evidence and prerequisites.
  • Decision evaluation at each checkpoint. In the Longitudinal Track, models independently read the cumulative record available at each checkpoint. We evaluate evidence recognition and utilization, clinical judgments, patient state, next actions, safety constraints, and stopping decisions, then examine whether these judgments change appropriately as new evidence appears.
  • Process evaluation of sequential actions. In the Interactive Track, models use tools to obtain information and execute actions. We record whether prerequisites are resolved, actions advance the case, and progress is sustained. Together, the two tracks help distinguish failures to recognize evidence, form the corresponding judgment, or translate that judgment into effective action.

During the rebuttal discussion, Reviewers pLuU and pKGk expressed appreciation for our responses, indicated that their concerns had been addressed, and maintained their positive scores. We thank them for their continued engagement. We summarize our responses and additional evidence below:

  1. Reliability of clinical annotations and scoring. (Reviewer pLuU, pKGk) Physicians reviewed all 240 checkpoints, with secondary review for 145 and independent adjudication of disagreements. We report pre-adjudication agreement, draft-annotation revision statistics, and a human audit of unmatched answers. The audit found that many unmatched answers omitted necessary clinical elements, conditions, or timing.
  2. Evidence recognition and clinical decisions. (Reviewer pLuU, pKGk, Zbsy) We separated unmatched clinical targets from incorrect evidence links and added a physician Evidence Recognition baseline. A stricter analysis required models to actually recognize sufficient evidence to support a target; even then, they often missed the corresponding state or action. Gemma–MedGemma comparisons likewise showed that upstream evidence gains did not extend to Next Actions.
  3. Longitudinal Track and output variability. (Reviewer pLuU, Zbsy) Independent checkpoints test judgments as new patient evidence appears; this does not directly test memory of models’ earlier responses. For three tested models, adjacent-checkpoint changes exceeded repeated-response changes on fixed inputs. We also used one clinical case to explain both tracks and redesigned Figures 1 and 2.
  4. Interactive stagnation and protocol. (Reviewer pLuU, pKGk, Zbsy) We clarified tool matching, Pending, invalid outputs, and the separate scoring of clinical quality and termination. At 78 Pending blocking points, we conducted a behavior audit, a blinded audit using only public information, and action replay. Models usually investigated prerequisites, but public information did not always reveal the required step, and advancing a node did not always clear the blockage. We also compared single and batched information queries and shared and separate budgets to examine protocol effects. We will define Pending and recovery metrics at first use.

We believe WARD aligns well with the ICLR community’s interest in long-horizon agent evaluation, process-level diagnosis, and reliable decision-making. It provides researchers with an evaluation framework that distinguishes failures in evidence recognition, state assessment, and sequential action, and identifies more specific targets for improving longitudinal clinical agents.

We sincerely thank the reviewers for their valuable feedback, which helped us clarify our contributions and strengthen the supporting analyses.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.