- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CodeClinic, a benchmark designed to evaluate the ability of large language models (LLMs) to automatically generate and reuse programming skills for clinical reasoning tasks. Built on the MIMIC-IV dataset, the benchmark consists of two complementary tasks: Longitudinal ICU Surveillance and Compositional Information Seeking.
The authors also propose a Clinical Autoformalization approach that transforms natural-language clinical guidelines into a reusable library of Python functions through iterative code generation, execution-based validation, and error feedback. Experimental results demonstrate that this approach outperforms both Zero-shot Code Generation and a manually designed EHR toolbox on compositional medical question-answering tasks, while significantly reducing token consumption during inference.
One particularly noteworthy finding is that, although the generated function library improves prediction accuracy at individual time points, these gains do not translate into substantial improvements in accuracy over complete patient trajectories. Based on this observation, the authors argue that maintaining longitudinal patient state remains a major challenge for current clinical agents.
Critical strengths
- The benchmark design is practically meaningful. This paper considers two important dimensions of clinical reasoning: compositional clinical reasoning and longitudinal patient state tracking, addressing a limitation of existing EHR Agent benchmarks that primarily focus on static question-answering tasks. In particular, the Longitudinal ICU Surveillance task incorporates the temporal semantics of different clinical states, including persistent states, cumulative states, validity intervals, and expired measurements. These design choices make the benchmark more representative of real-world clinical monitoring scenarios.
- The Autoformalization approach has practical value. By transforming medical knowledge into executable and reusable Python functions, the authors provide a feasible approach to reducing the manual effort required for tool development. Experimental results demonstrate that this approach can improve accuracy on complex medical question-answering tasks while reducing token consumption during inference.
- The paper highlights a substantial gap between accuracy at individual time points and accuracy over complete patient trajectories. It also analyzes the failure modes of agents in maintaining longitudinal patient states. These findings offer valuable insights into the limitations of current LLM-based agents when performing long-horizon tasks.
Critical weaknesses
My primary concern is the mismatch between the data used for Skill development and the Longitudinal Evaluation task. The authors use static Compositional QA examples to generate and validate the Skill Library. Therefore, the relatively poor Trajectory Accuracy observed in the Longitudinal task may not be solely attributable to the Agent's limited ability to track patient states over time. It could also result from the lack of Long-horizon Supervision during Skill development. The current experiments do not sufficiently disentangle these two factors.
I suggest that the authors conduct a 2×2 controlled experiment, in which Skill Libraries are developed using either Static QA Examples or Longitudinal Trajectory Examples, and then evaluated on both Static Compositional QA and Longitudinal Surveillance tasks. By controlling for the underlying model and Skill development budget, this experiment could help determine whether the bottleneck in Long-horizon Performance stems from a mismatch between Skill development and the target task, or from the Agent's inherent limitations in State Management. Such an experiment would make the paper's analysis of Longitudinal Failure and its conclusions more convincing.
Questions for authors
See the weaknesses.
Author Response
Thank you for your time and constructive feedback! We address the raised concerns below:
1. Skill-data mismatch and the suggested 2x2 experiment Thank you for this suggestion. We ran a targeted controlled comparison to test whether building skills from longitudinal rather than static examples improves longitudinal performance. Experimental design. We selected suspicion_of_infection, which targets the infection family where the longitudinal gap is largest. Using the same builder (Qwen3.5-9B), 100 labeled examples and identical development budgets, we constructed two versions of this function. Static (S): built from compositional QA examples, as in the paper. Longitudinal (L): built from time-indexed checkpoint questions sampled from 380 development trajectories, with no patient overlap with evaluation. The libraries differ only in this function, and we also kept the original function (R) as a reference. We evaluated the same Qwen3.5-9B agent with each library on 100 held-out static questions and on the longitudinal test set.
Longitudinal supervision produces no substantial improvement in infection detection or escalation. Both constructed functions remain imperfect, so this does not show that skill construction is solved. Rather, changing the supervision source alone does not resolve the observed failure in this controlled setting.
Where does the failure occur? We inspected the function outputs across all test trajectories. All three versions expose signals associated with infection.
has_suspicion_of_infectionreceived_antibioticsantibiotics_givenThe agent calls the function at about 90% of checkpoints with every library. However, even at infected checkpoints where these signals are present in the function output, the agent reports suspected infection in only 0%, 0% and 1.5% of cases, respectively. The functions expose relevant evidence, but the agent rarely converts it into an explicitly reported clinical state. This points to a downstream failure in identifying and updating states, rather than a failure attributable solely to skill construction.
The longitudinal gap also occurs without any skill library. Zero-shot Claude-Sonnet-4.6 drops from 76.2% checkpoint accuracy to 34.0% trajectory accuracy, and Gemini-3.1-Pro drops from 72.5% to 34.0%. Skill-data mismatch cannot explain this zero-shot gap. Our new finding-level analysis further distinguishes under-detection from forgetting.
Carry recall measures detection while a finding remains active. Conditional retention measures continued detection given that the finding was reported at the preceding checkpoint. Once findings are detected, they are usually retained. The larger failure is that relevant states are not consistently identified in the first place.
Scope and revision. The controlled experiment covers one concept and a 9B agent, and we are extending it to the remaining functions. We added the experiment to the appendix and revised Limitations to state that the current results point to a downstream bottleneck but do not isolate every source of longitudinal error.