acceptodds
Desk rejected at ICLR 2027

CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Clinical reasoning agents based on large language models (LLMs) aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tracking from electronic health records (EHRs). Existing systems typically rely on manually curated clinical tools or skills for concepts such as sepsis detection and organ failure assessment. However, maintaining these tool libraries requires substantial expert effort, while zero-shot querying or code generation often produces inefficient and unreliable reasoning chains, especially under institution-specific clinical policies. We introduce CodeClinic, a benchmark built on MIMIC-IV for evaluating whether LLM agents can synthesize and compose reusable clinical skills instead of relying on fixed toolboxes. The benchmark contains two complementary tasks: longitudinal ICU surveillance and compositional information seeking. The longitudinal setting simulates monitoring patient trajectories with structured decisions every four hours across 25 findings and eight clinical families, while the compositional setting spans 63k instances across 259 tasks in nine domains and is stratified by compositional dependency depth to evaluate increasingly complex multi-step reasoning. We further propose an offline autoformalization pipeline that converts natural-language clinical guidelines into reusable and verified Python skill libraries through iterative LLM refinement. With the same backbone, the resulting library outperforms both zero-shot code generation and a published expert-engineered EHR toolbox while using up to 47% fewer inference tokens per query, yet leaves substantial headroom on both tasks.

Primary area:
datasets and benchmarks
Keywords:
LLM, Agent, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zprWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CodeClinic, a benchmark designed to evaluate the ability of large language models (LLMs) to automatically generate and reuse programming skills for clinical reasoning tasks. Built on the MIMIC-IV dataset, the benchmark consists of two complementary tasks: Longitudinal ICU Surveillance and Compositional Information Seeking.

The authors also propose a Clinical Autoformalization approach that transforms natural-language clinical guidelines into a reusable library of Python functions through iterative code generation, execution-based validation, and error feedback. Experimental results demonstrate that this approach outperforms both Zero-shot Code Generation and a manually designed EHR toolbox on compositional medical question-answering tasks, while significantly reducing token consumption during inference.

One particularly noteworthy finding is that, although the generated function library improves prediction accuracy at individual time points, these gains do not translate into substantial improvements in accuracy over complete patient trajectories. Based on this observation, the authors argue that maintaining longitudinal patient state remains a major challenge for current clinical agents.

Critical strengths
  1. The benchmark design is practically meaningful. This paper considers two important dimensions of clinical reasoning: compositional clinical reasoning and longitudinal patient state tracking, addressing a limitation of existing EHR Agent benchmarks that primarily focus on static question-answering tasks. In particular, the Longitudinal ICU Surveillance task incorporates the temporal semantics of different clinical states, including persistent states, cumulative states, validity intervals, and expired measurements. These design choices make the benchmark more representative of real-world clinical monitoring scenarios.
  2. The Autoformalization approach has practical value. By transforming medical knowledge into executable and reusable Python functions, the authors provide a feasible approach to reducing the manual effort required for tool development. Experimental results demonstrate that this approach can improve accuracy on complex medical question-answering tasks while reducing token consumption during inference.
  3. The paper highlights a substantial gap between accuracy at individual time points and accuracy over complete patient trajectories. It also analyzes the failure modes of agents in maintaining longitudinal patient states. These findings offer valuable insights into the limitations of current LLM-based agents when performing long-horizon tasks.
Critical weaknesses

My primary concern is the mismatch between the data used for Skill development and the Longitudinal Evaluation task. The authors use static Compositional QA examples to generate and validate the Skill Library. Therefore, the relatively poor Trajectory Accuracy observed in the Longitudinal task may not be solely attributable to the Agent's limited ability to track patient states over time. It could also result from the lack of Long-horizon Supervision during Skill development. The current experiments do not sufficiently disentangle these two factors.

I suggest that the authors conduct a 2×2 controlled experiment, in which Skill Libraries are developed using either Static QA Examples or Longitudinal Trajectory Examples, and then evaluated on both Static Compositional QA and Longitudinal Surveillance tasks. By controlling for the underlying model and Skill development budget, this experiment could help determine whether the bottleneck in Long-horizon Performance stems from a mismatch between Skill development and the target task, or from the Agent's inherent limitations in State Management. Such an experiment would make the paper's analysis of Longitudinal Failure and its conclusions more convincing.

Questions for authors

See the weaknesses.

1 comment
AuthorsOpenReview ↗

Author Response

Thank you for your time and constructive feedback! We address the raised concerns below:

1. Skill-data mismatch and the suggested 2x2 experiment Thank you for this suggestion. We ran a targeted controlled comparison to test whether building skills from longitudinal rather than static examples improves longitudinal performance. Experimental design. We selected suspicion_of_infection, which targets the infection family where the longitudinal gap is largest. Using the same builder (Qwen3.5-9B), 100 labeled examples and identical development budgets, we constructed two versions of this function. Static (S): built from compositional QA examples, as in the paper. Longitudinal (L): built from time-indexed checkpoint questions sampled from 380 development trajectories, with no patient overlap with evaluation. The libraries differ only in this function, and we also kept the original function (R) as a reference. We evaluated the same Qwen3.5-9B agent with each library on 100 held-out static questions and on the longitudinal test set.

LibraryStatic QA ↑Suspected infection recall ↑Confirmed infection recall ↑Escalation BA ↑
Original (R)35%0.0%0.0%54.3%
Static (S)37%0.0%0.0%54.5%
Longitudinal (L)33%0.9%0.0%56.3%

Longitudinal supervision produces no substantial improvement in infection detection or escalation. Both constructed functions remain imperfect, so this does not show that skill construction is solved. Rather, changing the supervision source alone does not resolve the observed failure in this controlled setting.

Where does the failure occur? We inspected the function outputs across all test trajectories. All three versions expose signals associated with infection.

LibraryOutput fieldInfected checkpointsNon-infected checkpoints
Rhas_suspicion_of_infection69%18%
Sreceived_antibiotics66%12%
Lantibiotics_given66%12%

The agent calls the function at about 90% of checkpoints with every library. However, even at infected checkpoints where these signals are present in the function output, the agent reports suspected infection in only 0%, 0% and 1.5% of cases, respectively. The functions expose relevant evidence, but the agent rarely converts it into an explicitly reported clinical state. This points to a downstream failure in identifying and updating states, rather than a failure attributable solely to skill construction.

The longitudinal gap also occurs without any skill library. Zero-shot Claude-Sonnet-4.6 drops from 76.2% checkpoint accuracy to 34.0% trajectory accuracy, and Gemini-3.1-Pro drops from 72.5% to 34.0%. Skill-data mismatch cannot explain this zero-shot gap. Our new finding-level analysis further distinguishes under-detection from forgetting.

Temporal ClassModelCarry Recall ↑Conditional Retention ↑
PersistentClaude42%90%
PersistentGemini26%85%
PersistentGPT-5.410%98%
CumulativeClaude14%86%
CumulativeGemini22%98%
CumulativeGPT-5.46%100%

Carry recall measures detection while a finding remains active. Conditional retention measures continued detection given that the finding was reported at the preceding checkpoint. Once findings are detected, they are usually retained. The larger failure is that relevant states are not consistently identified in the first place.

Scope and revision. The controlled experiment covers one concept and a 9B agent, and we are extending it to the remaining functions. We added the experiment to the appendix and revised Limitations to state that the current results point to a downstream bottleneck but do not isolate every source of longitudinal error.

Reviewer_LuqMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CodeClinic, a MIMIC-IV benchmark for longitudinal ICU surveillance and compositional EHR queries. It also proposes an offline pipeline that generates, tests, and reuses Python functions for clinical concepts. Results across eight LLMs show gains in compositional accuracy and inference efficiency, but little improvement in full-trajectory monitoring.

Critical strengths
  • The two tasks expose useful limitations of EHR agents: temporal state tracking and multi-step concept computation.

  • Evaluation spans eight LLM backbones and reports performance by dependency depth and across ICU checkpoints.

  • The reusable function library improves compositional accuracy and reduces inference tokens; offline construction costs are reported.

Critical weaknesses
  1. The contribution of the proposed pipeline is unclear. Code generation, execution feedback, repair, and function reuse are established techniques. The main comparison uses different levels of tool support: AgentEHR supplies retrieval primitives, while CodeClinic supplies prebuilt concept-specific functions. Comparisons with single-pass offline function generation and refinement without verification feedback, under matched supervision and offline budgets, are needed to show what iterative verification adds.

  2. Function reliability is not directly established. Downstream QA accuracy conflates function correctness with tool selection and interpretation. Functions can enter the library even below the 90% verification threshold. Report held-out accuracy per function, threshold pass rates, and errors from composing functions. Tests on unseen concept combinations would better support the composability claim.

  3. The longitudinal results do not isolate temporal state-tracking failures. Trajectory accuracy requires the global action to be correct at all 13 checkpoints; it can fall with horizon length even without a memory problem. The existing temporal error analysis should be extended with finding-level state-transition metrics, stratified by model and temporal-semantic class. Global-action accuracy needs a majority-action baseline, balanced accuracy, and class-wise results on identical test subsets.

  4. Some evaluation comparisons are hard to interpret. Qwen3.5-4B zero-shot accuracy differs sharply between Tables 2 and 4 (21.0% versus 38.1%); the paper should explain the sampling, inference, and aggregation protocols. GRPO’s overall accuracy in Table 4 also appears inconsistent with its level-wise results under the stated concept-level aggregation, potentially affecting its ranking against Autoformalization.

Questions for authors
  • How many generated functions pass verification, and how accurate are they on held-out cases? Can you separate function errors from agent tool-use errors?

  • How does the full pipeline compare with single-pass offline function generation and refinement without verification feedback under comparable supervision and computational budgets?

  • Could the authors extend the existing temporal error analysis with finding-level state-transition metrics stratified by model and temporal-semantic class? Please also clarify the granularity of suspected-condition and alert F1, and report a majority-action baseline and class-balanced metrics on the same evaluation subsets.

  • What explains the zero-shot discrepancy between Tables 2 and 4, and how is GRPO’s overall accuracy in Table 4 obtained from its level-wise results? Does the claimed advantage over GRPO remain when all methods use consistent evaluation and aggregation procedures?

2 comments
AuthorsOpenReview ↗

Author Response (Part 1)

We appreciate the detailed feedback and questions! Below we address the key concerns:

1. Contribution of the pipeline: single-pass and no-feedback baselines We ran these ablations under the same supervision (~100 examples per concept), builder model and iteration budget, on a fixed subset of 12 concepts with 100 held-out questions each.

VariantAccuracy (%)Build tokens per concept
Full pipeline52.41.24M to 1.71M
Refinement without verification feedback50.41.33M
Single-pass generation46.20.50M

Each component contributes at a comparable or lower budget. Iterative refinement adds 6.2 points over single-pass generation (52.4% vs. 46.2%). Label-based verification feedback adds a further 2.0 points over refinement without it (52.4% vs. 50.4%) at a similar offline cost, since refinement without feedback spends the full 100-iteration budget on every concept. Guideline access contributes 4.3 points (48.1% without it).

Regarding tool support, AgentEHR supplies retrieval primitives rather than concept implementations. We now present it as a reference point for expert-built tools, and these ablations as the controlled comparison.

2. Function reliability Thank you for raising this. We now report it directly. Only 4 of 63 functions reach the 90% verification threshold. The other 59 enter the library as the best candidate found, after a mean of 96.3 of 100 repair iterations. Section 5 now states this in the main text, and the appendix reports these statistics.

The lowest end-to-end accuracies are on composite or rare concepts (sepsis3 6%, suspicion_of_infection 8%, sapsii 12%, apsiii 18%). Errors in upstream concepts propagate into these composites without being flagged, so verification succeeds on common cases and fails on the rest. The ablation runs show the same pattern. On the 12-concept subset, 0 to 3 concepts per configuration reach the threshold, with mean verification accuracy between 47.7% and 65.0%. Of 72 concept builds across six configurations, 7 produced no usable function, and 6 of these are Level 3+ composite scores (SOFA, SAPS II, LODS, Sepsis-3).

We agree that end-to-end accuracy mixes function errors with tool-use errors, and we have not run an oracle-call analysis. Two existing results bound the question. First, on the longitudinal task, tool calls succeed on 98.7% to 100% of checkpoints. Second, despite the low pass rate, the library outperforms zero-shot code generation by 18.9 points and the expert toolbox by 15.5 points with the same backbone (Table 3), so best-effort functions still provide most of the gain. The paper now states explicitly that the functions are imperfect, and lists oracle-call analysis and abstention from poorly verified functions as future work.

We have not tested unseen concept combinations. We narrowed the composability claim to the dependency structure covered by the benchmark and added this point to Limitations.

AuthorsOpenReview ↗

Author Response (Part 2)

3. Longitudinal metrics We recomputed all available longitudinal runs on identical subsets of 100 ICU stays, with 13 checkpoints per stay (1,300 checkpoints in total) and identical reference labels.

Majority baseline, balanced accuracy and class-wise results. Escalation is the majority action and accounts for 75.8% of checkpoints.

ModelAction Acc.Balanced Acc.Escalate RecallContinue RecallTraj. Acc.
Always escalate75.850.0100.00.010.0
Claude-Sonnet-4.676.275.277.273.234.0
Gemini-3.1-Pro72.574.071.276.834.0
GPT-5.466.570.263.177.425.0
Qwen3.5-27B31.849.814.984.73.0
+ AF39.059.020.397.86.0
Qwen3.5-9B24.950.31.299.46.0
+ AF34.655.615.096.26.0
Qwen3.5-4B27.051.44.298.75.0
+ AF34.855.016.093.97.0

The majority-action baseline shows that raw action accuracy can be misleading under class imbalance. Balanced accuracy is more informative. Frontier models reach 70% to 75%, whereas zero-shot Qwen models perform near chance. Autoformalization improves the balanced accuracy of Qwen models by 3.6 to 9.2 points, although escalation recall remains low.

Finding-level state transitions. We added three metrics for each temporal-semantic class. Onset recall measures detection when a finding first becomes active. Carry recall measures detection while the finding remains active. Conditional retention measures continued detection given that the finding was reported at the preceding checkpoint. Each cell reports onset / carry / retention (%).

Temporal ClassClaudeGeminiGPT-5.4Qwen-27B + AF
Persistent episode39/42/9016/26/853/10/980/0/–
Cumulative max stage20/14/8614/22/983/6/1000/1/–
Active interval69/91/9876/90/9817/44/986/6/41
Time-to-live65/89/9769/90/958/26/945/6/59
Composite current state53/39/7960/43/8713/21/870/0/–

A dash indicates that the finding was never reported, so retention is undefined. The results distinguish two failure modes. Claude detects only 42% of active persistent findings, but retains 90% of those it has already reported. Low carry recall is therefore driven mainly by under-detection rather than by systematic forgetting. We also observe the opposite failure, in which models keep reporting findings after their reference states have ended. For active-interval and time-to-live findings, Claude and Gemini still report 24% to 57% of expired findings at the next checkpoint.

F1 granularity. The reported suspected-condition and alert F1 scores are checkpoint-averaged set F1, not class-macro F1. Set F1 is computed independently at each checkpoint and averaged across checkpoints, with a score of 1 when both the predicted and reference sets are empty. As a result, an empty output already achieves 18.3% suspected-condition F1 and 24.2% alert F1. We corrected the metric description in the paper and added the empty-output baseline to the results table.

Finally, we agree that trajectory accuracy alone cannot isolate temporal state-tracking failures, since it compounds errors across 13 checkpoints. The transition-level analysis characterizes detection, retention and resolution directly, without attributing all failures to memory.

4. Table 2 vs. Table 4, and GRPO aggregation

  • Zero-shot gap: Table 2 is evaluated on the full held-out test set, while Table 4 is evaluated on a subset of 100 questions per concept. We now state this in both captions.
  • GRPO aggregation: Thank you, you are correct, and we apologize for the error. Overall accuracy is the macro average over concepts, so levels are weighted by their concept counts (34, 17 and 12). We miscalculated the GRPO overall, and the correct value is 46.0%, not 40.5%. The zero-shot (38.1%) and autoformalization (45.3%) overalls were computed correctly.

We revised the claim accordingly. GRPO and autoformalization reach similar overall accuracy (46.0% vs. 45.3%) but differ by depth. GRPO leads at Level 1 (+5.9 points), while autoformalization leads at Level 2 (+5.0) and Level 3+ (+6.2), where composition matters most. Autoformalization achieves this without parameter updates, its gains over zero-shot hold across model families, and the same pattern holds when the library is rebuilt from the same 50 examples per concept used for GRPO (45.9% overall and 36.8% vs. 27.9% at Level 3+).

Reviewer_JSzcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CodeClinic, a benchmark for evaluating the ability of LLM agents to generate and utilize reusable coding skills for clinical reasoning over electronic health records (EHRs). Built on MIMIC-IV, the benchmark consists of two tasks: (1) longitudinal ICU surveillance, which requires agents to track evolving patient conditions across multiple checkpoints, and (2) compositional information seeking, comprising 63,000 queries across 63 clinical concepts with varying dependency depths. The authors further propose a clinical autoformalization pipeline that converts natural-language clinical guidelines into reusable Python function libraries through iterative LLM-based code generation, execution, verification, and refinement using limited labeled examples. The resulting libraries are constructed offline and subsequently provided to inference agents without updating model parameters. Experiments across multiple LLM backbones compare this approach against zero-shot code generation, an expert-engineered toolbox, and GRPO fine-tuning. Results show that autoformalization improves accuracy and reduces inference token consumption on compositional reasoning tasks, while improving checkpoint-level accuracy without substantially improving full-trajectory accuracy in longitudinal ICU surveillance.

Critical strengths
  1. The analysis about the benchmark is solid.
  2. The study problem may be beneficial for human use.
Critical weaknesses

Weaknesses

1. Presentation

  • Figure quality. Figure 3 is presented as a raster image rather than a vector graphic. The concept labels on the right are heavily crowded, while the decorative icons on the left introduce unnecessary visual clutter. The figure should be redesigned to improve readability and clarity.

  • Writing clarity. Several statements are unnecessarily verbose or insufficiently motivated. For example, Line 58 explicitly clarifies that ICU surveillance is "not a forecasting task," although this distinction has not been established as a source of ambiguity. Furthermore, the authors acknowledge that temporal retrieval is a "specific, reproducible weakness" of their method (Line 452), but this limitation is not sufficiently discussed in the Limitations section.

  • Unclear motivation for the benchmark design. The Introduction does not sufficiently explain how the compositional information-seeking task directly supports the central motivation of synthesizing reusable coding skills. The relationship between the two benchmark tasks and the proposed autoformalization framework should be better established.

2. Soundness

  • Insufficient comparisons with existing methods. The primary comparisons focus on zero-shot code generation, the AgentEHR toolbox, and GRPO fine-tuning. However, the paper lacks comparisons with established approaches to automatic tool creation, skill learning, and iterative refinement. Consequently, it is difficult to determine whether the improvements arise from the specific autoformalization framework or from already established techniques.

  • Weak and potentially unfair GRPO baseline. The GRPO baseline is insufficiently investigated. In Figure 14(a), the training reward increases only marginally after approximately 300–400 steps, suggesting that optimization may not have been fully effective. One possible explanation is that an increasing fraction of sampled prompt groups produces uniform rewards and therefore contributes little or no policy-gradient signal. The authors should report the fraction of effective training groups and consider stronger RL training recipes, such as DAPO with dynamic sampling, under a controlled training budget.

    More importantly, the comparison does not control the training resources: GRPO uses only 50 training examples per concept, whereas autoformalization uses approximately 100 verification examples per concept and relies on a stronger Qwen3.5-27B model for library construction. Therefore, the reported superiority of library learning over weight adaptation cannot be attributed solely to the learning paradigm.

  • Missing ablation studies. The authors acknowledge that the verification set used during autoformalization is relatively small. An ablation study varying the number of verification examples would help determine how the approach scales with additional supervision. Moreover, the individual contributions of guideline retrieval, iterative refinement, verification feedback, and function reuse are not sufficiently isolated.

  • Limited generalization and static skill libraries. The authors explicitly motivate their work by noting that clinical definitions vary across institutions and evolve over time. However, once the generated library is complete, it is deployed as a static resource without an explicit mechanism for adapting to updated guidelines or institutional policies. This leaves an important part of the original motivation unresolved.

    Furthermore, most experiments evaluate reuse across different patient records for the same clinical concepts, rather than generalization to unseen concepts or updated clinical definitions. The preliminary eICU experiments cover only two relatively simple concepts and do not establish generalization to more complex, institution-dependent clinical criteria.

  • Limited longitudinal improvement. Although the autoformalized functions improve checkpoint-level accuracy and reduce token consumption, they do not substantially improve full-trajectory accuracy. This raises questions about whether the proposed method meaningfully addresses the longitudinal state-tracking problem, which is one of the two central evaluation settings.

3. Novelty

  • Limited novelty of reusable code-based skill learning. Converting natural-language instructions or task descriptions into reusable executable code has been extensively investigated in prior work, including Voyager [1], LATM [2], CRAFT [3], CREATOR [4], and SkillWeaver [5]. In particular, LATM already separates offline tool construction from subsequent tool usage, while CRAFT constructs and validates reusable code tools from task examples. The authors should more clearly articulate what methodological contributions distinguish clinical autoformalization from these established approaches, beyond applying them to the EHR domain.

  • Limited novelty of iterative refinement. Iterative improvement through feedback and execution-based verification is also well established, as demonstrated by Self-Refine [6], Reflexion [7], and the iterative code-repair mechanism in Voyager [1]. The proposed autoformalization loop appears to follow a similar generate–verify–refine paradigm. The authors should clarify whether there is a substantive algorithmic contribution beyond adapting this paradigm to clinical data.

  • Missing comparison with RetroSum. AgentEHR introduces RetroSum, which uses retrospective summarization and evolving experience memory to improve long-horizon EHR interactions. However, the present paper compares against the AgentEHR toolbox without evaluating RetroSum as a reasoning baseline. Given the significant degradation in longitudinal trajectory accuracy, a comparison with RetroSum would be relevant for evaluating the benchmark's long-horizon reasoning capabilities.

References

[1] Voyager

[2] Large Language Models as Tool Makers (LATM)

[3] CRAFT

[4] CREATOR

[5] SkillWeaver

[6] Self-Refine

[7] Reflexion

[8] AgentEHR / RetroSum

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the detailed feedback! Below, we acknowledge and address the raised concerns:

1. Figure 3 Presentation: We redrew Figure 3 as a vector graphic and removed the decorative icons. The left panel shows the example question's dependency graph, colored by difficulty level. The crowded circular taxonomy is replaced by a bar chart of concept counts per domain, split by difficulty level. The full list of 63 concepts now appears in an appendix table.

2. Writing: L58 "not a forecasting task" and temporal retrieval at L452 We removed the "not a forecasting task" sentence. The temporal-retrieval weakness is now discussed in Limitations. AgentEHR outperforms our method on temporal questions (41.7% vs. 37.8%) and on 11 of 63 concepts dominated by irregular time-series retrieval. Fixed-signature functions handle common time windows well but not unusual ones, and adding parameterized temporal primitives is a natural remedy.

3. Motivation: link between compositional QA and skill synthesis We added an Introduction paragraph to address this concern. Compositional QA is where skills are built and verified. Each concept has many labeled instances across variants and dependency depths, so a function can be checked against ground truth and reused by every concept that depends on it. Longitudinal surveillance is where the same skills are deployed over a whole ICU stay. Building skills directly from trajectories is harder, because each trajectory provides only one coarse label per checkpoint. This split lets the benchmark separate synthesizing correct skills from using them over a long horizon.

4. No comparison with tool-making and refinement methods We agree that generate-verify-refine is established, and the revised Related Work cites and discusses all seven works. However, a direct empirical comparison is outside the scope of our main contribution, which is a new agentic EHR benchmark. These systems mainly target Minecraft, web, or math and puzzle environments, and would need substantial re-engineering for EHR schemas and clinical labels. In particular, they assume task demonstrations rather than guideline specifications, and they do not verify against labeled patient cohorts. Instead, our component ablations (point 6) remove the parts these methods share or lack. Single-pass generation approximates LATM- and CREATOR-style one-shot tool making and reaches 46.2%, compared with 52.4% for the full pipeline. Removing guideline access lowers accuracy to 48.1%. Both guideline grounding, which these methods do not use, and iterative refinement contribute to the final accuracy.

5. GRPO baseline

  • Matched supervision: GRPO was trained on 50 examples per concept, so we rebuilt the function library from the same 50 examples per concept. With the same Qwen3.5-4B inference agent, the rebuilt library reaches 45.9% overall, comparable to GRPO's 46.0% (corrected from 40.5%, see our response to Reviewer LuqM). The two methods differ by depth. GRPO leads at Level 1 (57.1% vs. 52.7%), while the library leads at Level 3+ (36.8% vs. 27.9%), consistent with our finding that the library helps most where concepts must be composed. The library achieves this without parameter updates.
  • Training dynamics: We agree that the reward plateau after 300 to 400 steps in Figure 14(a) deserves explanation. Unfortunately, our training logs did not record per-rollout rewards, so we cannot report the fraction of prompt groups with non-uniform rewards for this run. With 4 rollouts per prompt, groups in which all rollouts receive the same reward contribute no gradient, and dynamic sampling as in DAPO targets exactly this problem. The revised paper notes this and lists DAPO-style training as future work.
  • Builder model: Building the library with Qwen3.5-27B while GRPO trains Qwen3.5-4B is a genuine confound. We now state this explicitly and scope the claim to equal supervision and an identical inference model, rather than claiming that library learning outperforms weight adaptation in general. Building the library is a one-time cost of about 88 GPU-hours, compared with 144 GPU-hours for GRPO training.

6. Missing ablations We selected a fixed subset of 12 concepts (4 at Level 1, 3 at Level 2 and 5 at Level 3+), six of which have an associated guideline. Each variant rebuilds only these 12 functions with the Qwen3.5-27B builder and keeps all other library functions fixed. The full pipeline is the library reported in the paper. All variants use the same 100-iteration budget and are evaluated with the Qwen3.5-27B inference agent on the same 100 held-out questions per concept.

VariantAccuracy (%)
Full pipeline52.4
No guideline access48.1
No verification feedback50.4
Single-pass generation46.2
Verification examples per concept102550100
Accuracy (%)48.551.252.252.4
AuthorsOpenReview ↗

Author Response (Part 2)

Every component contributes: Iterative refinement adds 6.2 points over single-pass generation, verification feedback adds 2.0 points and guideline access adds 4.3 points. Accuracy rises quickly up to 25 verification examples and saturates near 50, which supports our low-supervision setting.

Guideline access matters where a clinical definition exists. On the six subset concepts with an associated guideline, removing guideline access lowers verification accuracy from 54.2% to 28.7%, and two of these concepts (SAPS II and LODS) produce no function at all. On the six concepts without a guideline, accuracy is essentially unchanged (63.3% vs. 63.8%).

We did not isolate function reuse, but existing results provide indirect evidence. The margin over AgentEHR is largest at Level 3+ (42.8% vs. 23.8%, a gain of 19.0 points compared with 15.5 overall), and the library outperforms GRPO only at Levels 2 and 3+, where concepts must be composed.

7. Static libraries and generalization A guideline or policy update is handled by re-running the pipeline offline on the changed concept and its dependents in the concept graph, without retraining any model. This costs about 6.5M tokens per concept, including verification. Rebuilding could also be triggered automatically when monitored performance drops after deployment. We added this procedure to Section 4. We agree that generalization to unseen concepts and to complex eICU criteria is not established, and Limitations now states this explicitly.

8. Limited longitudinal improvement We distinguish two complementary components of CodeClinic. Autoformalization provides reusable functions for computing clinical concepts, while longitudinal surveillance evaluates an agent's ability to reason over evolving patient states across an ICU stay. We do not claim that autoformalization alone solves long-horizon reasoning. The results illustrate this distinction. Adding the function library consistently improves checkpoint-level accuracy, but the gains do not translate into reliable full-trajectory performance.

ModelCheckpoint Acc. ZS → AFBalanced Acc. ZS → AFTrajectory Acc. ZS → AF
Qwen3.5-27B31.8 → 39.049.8 → 59.03 → 6
Qwen3.5-9B24.9 → 34.650.3 → 55.66 → 6
Qwen3.5-4B27.0 → 34.851.4 → 55.05 → 7

All values are percentages. ZS: zero-shot. AF: autoformalization. To better understand the remaining failures, we added finding-level state-transition metrics. Persistent and cumulative findings have low detection recall while they remain active, but retention is high once a finding has been reported. Under-detection, rather than forgetting, is therefore a major source of error. A controlled experiment (see our response to Reviewer zprW) supports this. Rebuilding the infection function from longitudinal checkpoints does not close the gap, because the agent rarely converts the infection evidence returned by the function into a reported state.

9. RetroSum RetroSum improves long-horizon interaction through retrospective summarization and memory. This is complementary to verified concept functions, and the two could be combined. We could not reproduce RetroSum within the discussion period. It is now discussed in Related Work, and combining the two approaches is listed as future work.

10. Ethics We agree. To quantify the clinical risk, we added a false-negative analysis of 100 ICU stays, which contain 986 escalation-required checkpoints across 94 stays. Even frontier agents miss 22.8% to 36.9% of required escalations.

ModelEscalation FNR ↓Alert-finding FNR ↓Stays never escalated ↓
Claude-Sonnet-4.622.8%51.1%8.5%
Gemini-3.1-Pro28.8%56.5%8.5%
GPT-5.436.9%83.5%27.7%
Qwen3.5-27B + AF79.7%98.1%58.5%
Qwen3.5-9B + AF85.0%98.4%59.6%
Qwen3.5-4B + AF84.0%98.4%58.5%

Escalation FNR is the fraction of required escalations that were missed. Alert-finding FNR is the fraction of true alert-level findings omitted from both output lists.

We revised the Ethics Statement to distinguish retrospective benchmarking from clinically validated decision support, to report these false-negative risks, and to require clinician review, prospective validation and subgroup audits before any clinical use. We also expanded Limitations to discuss demographic bias and limited generalizability arising from MIMIC-IV's single-center population and coding practices. Finally, we removed the Conclusion's suggestion of deployment in real ICU workflows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.