- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This is a benchmark that tests how longitudinal medical agents translate accumulating evidence into actions, i have no question about the soundness and comprehensiveness of this work but the its current form is preventing me from fully getting what the authors want to present.
Critical strengths
-
Trendy topic as only agents can really help transform healthcare.
-
the scale and dimension of the dataset is comprehensive enough for an iclr benchmark.
Critical weaknesses
===presentation issues===
- before evaling "longitudinal clinical agents", i think its better we clarify what these "agents" are, more concretely their action. Are they "agents" because they are expected to give task instructions, treatment plans? or are they agents because they should interact with EHR, python and other tools? Or a combination of both?
You can back it up with real clinical use case examples.
- Severe presentation issue. Whoever write the results section, obviously just copied what claude/codex gave them over. This leads to several issues: a) overload of numbers, invented phrases. If you don't buy my comments, read last year's best paper. https://arxiv.org/pdf/2505.06120 see how many numbers they repeat from the figs/tables v.s. how many your current paper does. It's both a torture to audience and a waste of space.
b) Because you wasted so much space with meaningless repetition of numbers that already in table/fig , the sec 6.3 explanation for "Pending-prerequisite recovery" or " prerequisite Pending" is non-existent. Therefore I have no idea what they mean in your context.
c) similar issues for sec 7.
in general i like the topic very much but " Generally understandable, but requires substantial effort in places." is the highest score I can give for presentation, and this poor presentation limits me from "I gained a substantially deeper understanding of the problem or question."
===content issues==== i will specify in the question for authors section.
Questions for authors
===presentation issues===
-
Can you please remake fig 1 and fig 2 when you arxiv/ after acceptance, or if unlucky in ICLR, moving to other venues? These are examples of poor presentation. Literally 2 most important figures in paper and you are literally storming it with full text. Current form is no better way than just writing them down in bullet points in text. "Let the figure do the talk" not let the audience read the text.
-
can we please have a human researcher write the results section? the purpose is to help me understand your findings and only then is possible for me to give fair scores on presentation and informativeness.
===content issues====
-
this is a medical related topic, and i see both medgemma and other general domain llms are being compared. however, i (may have missed due to appendix being too long) dont see a head to head same family medical v.s. general models? like medical gemma has its general domain sister right?
-
in results u mentioned about repeated or prolonged stagnation, are these like overthinking loops (this can be especially an issue for qwen) or just the llm not making any substantial moves despite being aware of the "prereq not completed"? if so does it mean its a long context issue or forgetting issue, or model capability issue? i want to clearly understand why they fail to continue, like are the models aware of these pereq and still refuse to resolve?
same questions confound both 6.2 and 6.3.
5 comments
W2 & Q2: Revise the Results and §7 to reduce repeated numbers, clarify Pending and recovery, and focus on what the findings mean.
We thank the reviewer for the interest in this research question and for specific suggestions on how to analyze and present the results. We carefully read the paper you recommended. Its Results section first uses key aggregate results to establish the overall trend, then uses comparisons and follow-up analyses to show which aspects of performance decline and whether the decline depends on the experimental setup. We also want to clarify how we originally structured our analysis.. We aimed to state the main finding, support it with data, and then discuss possible explanations. The main text therefore includes cross-model summaries and correlation analyses, but it also repeats some values already shown in figures and tables, making the key comparisons and their implications less prominent.
The original manuscript explains the relevant metrics and environment feedback, but the definitions and calculation details are spread across the Methods, figure captions, and appendices. This makes them harder to look up while reading the results. For example, Table B3 and Appendix C.3 explain when the environment returns Prerequisite pending. Appendix D.4 and Table C3 explain how Pending-prerequisite recovery is calculated. We will explain these terms where we first discuss the corresponding results and keep only the numbers needed to support the key comparisons in the text.
Following the approach in the paper you suggested, we will organize our results around a central question: to what extent can models use the evidence they recognize to make appropriate clinical judgments and take appropriate follow-up actions? §6.1 first examines whether models make the corresponding judgments or recommend the appropriate actions after recognizing relevant evidence. §6.2 then asks whether models revise their judgments when needed and retain those that remain appropriate as the clinical record grows. §6.3 compares action recommendations at individual checkpoints with actual progress over successive interactions. It also uses a trajectory example to show how a task can stall again after an earlier blockage has been resolved.
In §7, we will follow the approach to analyzing model behavior in Appendix F of the recommended paper. We will spend less space recounting the tool calls and responses shown in Figure 5 and use the audit of behavior after Pending feedback, which you suggested in Q4, to interpret the cases. In Figure 5(a), for example, the environment blocks a premature surgery request, after which the model checks the consent record and proceeds. We will use this case to illustrate how a model adjusts its actions in response to feedback and put it in the context of the audit. Models often attempt to check prerequisites after being blocked, but those attempts do not always lead to further progress.
We provide the revised text of §6.3 below:
6.3 Interactive Execution and Sustained Progress
Among interactive trajectories that did not reach a terminal state, 76.92% exhibited repeated or prolonged stagnation. To examine how stagnation arose, we analyzed 1,204 executed tool calls (Figure 4a), of which 11.46% advanced the reference path, approximately half were valid explorations that returned no recorded outcome, and 27.66% were blocked by unmet prerequisites. The environment returns Pending when it identifies a candidate transition for a request but the prerequisites for executing that transition remain unmet. We tracked 78 Pending points and found that 45 were followed by at least one completed reference transition. At 19 of those 45 points, later actions advanced the workflow without completing any of the original request’s candidate transitions.
To determine which step a subsequent action advanced and whether that step resolved the original blockage, we executed a candidate action proposed by an independent auditing model from the environment state at which the request received Pending. When the action produced progress, we retried the original request. Among the 26 blocking points where a single action produced progress, 16 completed a preparatory step required for a direct prerequisite, 4 satisfied one direct prerequisite while others remained unmet, and 5 allowed the original request to succeed on retry. In the original trajectories, at 12 of the 19 points where only other transitions were completed, the model made progress and then issued another request that matched one of the original candidate transitions, again receiving Pending. These results show that a successful action often resolves only part of the dependency chain. After making progress, the model must determine which prerequisites remain unmet and when to retry the original request.
However, even after resolving a prerequisite blockage, a model can enter a prolonged period of stagnation when a tool returns no result. In WARD-001, GPT-5.6 SOL satisfied the imaging prerequisite, completed the blocked biopsy, and obtained the pathology report. It then received no result for an immunohistochemistry request and immediately repeated it. The 31 calls after pathology retrieval comprised 20 explorations with no recorded outcome and 11 calls targeting completed steps. None advanced the reference path or returned Pending.
We thank the reviewer for this question. In WARD, the term refers to a model that uses the progressively available records of the same patient to assess the current clinical state, determine the next action, and identify unresolved clinical and safety prerequisites. “Longitudinal” refers to how the available evidence and appropriate decisions change over time.
In the Longitudinal Track, each checkpoint provides the cumulative clinical record available up to that time. The model generates each response independently, without tool access or access to its earlier responses. We describe this setup in Section 4. and Appendix C.1. For example, the input at Checkpoint 8 includes a negative axillary cytology result, while the pathology report for the breast lesion itself is not yet available. The model must submit a structured response that identifies the key evidence, states that the diagnosis remains unresolved, recommends awaiting the pathology result, and explains why the evaluation should continue.
In the Interactive Track, the model decides what to query or do next and makes one tool call per turn. The interface supports patient communication, EHR search and retrieval, order placement, procedures, treatment, and care transitions. We describe the interaction protocol in Section 4.2 and list the tools in Appendix C.2, Table B2. In the same scenario, the model could first query the EHR for the breast lesion pathology report. If the tool indicates that the report is still unavailable, the model uses the current record to decide what else it can check. If the tool returns the report, the model uses the result to choose its next action.