acceptodds
Under review as a conference paper at ICLR 2027

EHR-Evo: Evolving Harnesses for EHR Agents through Execution Experience

Abstract

Large language model agents provide a promising interface for accessing electronic health records (EHRs), but reliable execution remains difficult because database queries require complex schema navigation, tool use, and multi-step program construction. Existing systems can recover from individual failures through retries and execution feedback, yet typically rely on fixed execution scaffolds and make limited use of accumulated experience. We introduce , an experience-driven execution harness around a frozen language-model agent. An outer-loop proposer uses sanitized execution traces to evolve model-external planning, retrieval, memory, and recovery mechanisms, after which the selected harness is frozen for evaluation. During execution, the harness retrieves reusable operational knowledge and solution traces, constructs question-specific guidance, and uses execution feedback for targeted recovery and cross-question experience reuse. We evaluate on MIMIC-III, eICU, and TREQS, comparing the complete evolved harness with fixed-retry, persistent-memory, and existing agent/harness baselines. The results show consistent improvements in answer success while revealing distinct effects of recovery and persistent experience on execution completion. Our study demonstrates that EHR-agent capability can be improved by evolving and adapting the model-external harness while keeping the underlying solver frozen.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
EHR agent, harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CgKPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an outer inner loop harness tuning procedure for EHR tasks. The base model of the ehr agent is gpt4 and frozen. The inner agent (model and harness) is depending on the setting allowed to retry (with seeing outcome traces of earlier events) or previous trajectories from the batch. The outer loop is driven by gpt 5.6 sol and proposed harnesses within the restricted design space of planning, memory formation and retrieval and recovery mechanisms based on failures.

Critical strengths
  • Ehr is a relevant problem
  • Large empirical over EHR agent and baselines
  • Code provided (no data) but 3 relevant EHR datasets
Critical weaknesses
  • Missing Baselines, related work and novelty. Meta harness is a good baseline. Lack of related work and baselines (gepa for static prompt tying, ace style for memory formation (and retrieve, form, management). This is a form of learning from experience in text space. Related work should mention EHR/FHIR finetuning methods as alternative approach to learn from experience. Just Claude Code or Codex in outer loop to improve the harness of inner model as baseline . It is not shown that the restricted evo-ehr search space is not just an artefact of the gpt 5.6 sol abilities.
  • Leakage. Retry is triggered on benchmark eval as wrong. This leaks the outcome, instead of only the execution trace. If there is a perfect “filter” giving correct answers, the agent could generate parallel candidates and just analyse. Compute match a parallel + filter baseline against the sequential version.
  • Experimental gains unclear attributed to components ( like outer loop optimisation, ground truth feedback and memory )
Questions for authors
  • points in weaknesses
  • Why did you choose gpt4 as basemodel in the inner agent. Health is sensitive, so a model that can be self hosted seems an obvious choice.
  • Did you experiment with dynamic eval budgets inside the outer loop?
3 comments

We thank the reviewer for the thoughtful questions about methodological novelty, correctness feedback, and the evidence supporting harness evolution. The comments also highlight the importance of making model choices and evaluation budgets explicit. We clarify our contribution relative to existing methods and explain the scope of the experimental evidence below.

W1: Novelty and comparison with existing methods. We agree that reflective prompt optimisation, experience memory, and harness adaptation are established research directions. Our contribution is their execution-specific integration for EHR database agents, with two distinct adaptation stages and two separately maintained forms of experience.

First, the outer loop uses execution traces to select model-external execution instructions covering EHR schema navigation, planning, and error recovery. Second, after harness selection, the execution system maintains both reference-filtered solution trajectories and operational knowledge. The latter can incorporate lessons from failed executions—such as invalid schema references, patient-versus-admission scope errors, and tool-return-type mismatches—without retaining those executions as successful solution examples.

This differs in optimisation target and implementation from general prompt optimisation and evolving context management, although the underlying ideas are related. We do not claim that individual components are unprecedented or that our experiments establish superiority over GEPA or ACE.

Our EHR-specific Meta-Harness adaptation uses the same named proposer and solver models and matched candidate-search schedules. EHR-Evo achieves higher benchmark success rates across all three datasets (+22.07, +28.28, and +17.57 pp on MIMIC-III, eICU, and TREQS). Our eICU confidence-prompt reconstruction of TrustEHRAgent covers all 580 eICU questions under the identical evaluator and achieves 41.38% SR / 93.79% CR. We interpret these comparisons within their stated search spaces, implementation choices, and evaluation protocols.

W2: Correctness feedback and the parallel-generation baseline. We acknowledge that the main evaluation uses reference judgments for retry allocation and solution-memory admission. Consequently, these results establish performance under a benchmark-conditioned protocol, not fully reference-free deployment performance.

We have not performed the requested compute-matched parallel generation and oracle-filtering comparison (nor do we treat Full − w/o retry as a substitute for independent parallel sampling). Therefore, we do not claim that sequential repair is superior to independent candidate generation under equal computation and access to correctness feedback.

W3: Contribution of harness evolution and component attribution. We provide an additional controlled analysis of harness instruction content. On the same 580 eICU questions, the initial and selected configurations use the same frozen solver, initial examples, read-only memory, planning settings, and runtime-error-based retry policy. The selected instructions improve answer success from 223/580 (38.45%) to 236/580 (40.69%), with 17 previously unsuccessful questions corrected and 4 previously successful questions regressed (exact McNemar p=0.0072p = 0.0072).

This comparison supports the contribution of the selected execution instructions under the shared frozen-memory setting. Because the memory originated from an earlier benchmark-conditioned run, it does not establish reference-free learning or the superiority of automated search over handcrafted instructions.

Table 1 additionally shows configuration-level performance differences under the full evaluation protocol:

SR difference, ppMIMIC-III (580)eICU (580)TREQS (996)
Full − EHRAgent+18.28+17.07+13.46
Full − w/o memory+1.55+5.52+1.31
Full − w/o retry+1.03+6.90+3.11

Because w/o memory retains initial seed examples and w/o retry retains internal debugging, these full-system comparisons demonstrate performance differences associated with enabling recovery and memory updates, but do not independently identify their causal contributions.

Q2–Q3: Model selection and evaluation budgets.

  • Q2 (Model selection): GPT-4-0613 was selected to maintain compatibility with the EHRAgent experimental setting on de-identified benchmarks, rather than as an optimal model for clinical deployment. Validation with self-hosted models remains future work.
  • Q3 (Evaluation budgets): Harness search uses fixed candidate-screening and development-evaluation schedules, rather than dynamically varying evaluation budgets. The selected configuration is frozen before final evaluation.

Dear Reviewer CgKP,

Thank you again for your detailed review. We would appreciate your reconsideration of our submission in light of the additional evidence and clarifications provided during the rebuttal.

1. Controlled evidence for harness evolution. Our paired eICU experiment holds the frozen solver, examples, read-only memory, and retry policy constant across 580 questions. The selected harness improves success from 223/580 (38.45%) to 236/580 (40.69%), with 17 corrected versus four regressed outcomes (exact McNemar p = 0.0072). This isolates the effect of harness instruction selection under the shared-memory setting, without benchmark-triggered additional retries or online memory updates during the comparison.

2. Contribution beyond existing approaches. EHR-Evo combines execution-driven harness selection with two distinct experience stores: successful solution traces and reusable operational knowledge. Crucially, failed executions can inform schema-navigation and recovery rules without being admitted as successful examples. Our EHR-specific Meta-Harness adaptation uses the same proposer and solver models and matched candidate-search schedules; EHR-Evo achieves +22.07, +28.28, and +17.57 percentage points higher benchmark success across MIMIC-III, eICU, and TREQS.

3. Clarification of oracle feedback. We acknowledge that the main evaluation uses benchmark-conditioned feedback and does not establish fully reference-free operation. However, EHR-Evo without the additional top-level retry still exceeds our EHRAgent rerun by 17.25, 10.17, and 10.35 percentage points, respectively. Both configurations retain reference-filtered solution-memory admission, so this comparison does not isolate every mechanism, but shows that the additional retry alone cannot explain the observed differences.

We recognise that a compute-matched independent-generation baseline remains outstanding. Nevertheless, we believe the controlled experiment, methodological distinctions, and consistent empirical evidence substantively address the concerns about novelty and attribution.

We respectfully ask whether these additional results and clarifications warrant reconsideration of your overall score. We would appreciate knowing if a particular remaining concern is decisive.

Thank you for your consideration.

The Authors

I thank the authors for their comments and additonal runs and increase the score.

Reviewer_cfMfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EHR-Evo, an execution harness for EHR database agent. Before evaluation, an outer-loop proposer reads sanitized execution traces and proposes harness configurations, which are scored on development data and the best is frozen. During evaluation, the harness retrieves persistent memory split into operational knowledge and solution traces, builds a question-specific plan, allocates one top-level retry with a failure-classified repair context, and writes selected experience back for later questions. Experiments are conducted on three EHR datasets and compare the proposed harness against several existing work.

Critical strengths
  • The studied problem of improving the reliability of EHR agents in executing against EHR databases is important and practical.
  • Experiments are conducted on three datasets and different model variants are compared.
Critical weaknesses
  • The results depend on benchmark feedback that the baselines do not receive and that would not be available in practice. The system gets a second attempt only when the benchmark judge says its first answer is wrong, and solution traces are saved to memory only when the judge confirms they are correct. This effectively gives the method access to test-set correctness labels: it knows exactly when to retry and can reuse verified solutions from earlier test questions on later ones. It is unclear how much of the improvement comes from the proposed harness rather than this extra feedback.

  • The overall technical contribution and novelty of the framework are limited. Most components are directly adopted from existing work (past-solution retrieval and self-debugging with execution feedback are already in EHRAgent; experience memory and reflection are standard). The paper needs to clearly state its contribution beyond integrating existing methods.

  • Implementation details of the baselines are lacking. The reproduced Meta-Harness baseline underperforms plain EHRAgent on all three datasets. Details such as the proposer, search budget, and development data used for it are not described.

Questions for authors

see comments above

1 comment

We thank the reviewer for the detailed comments on correctness feedback, the distinction between our memory components, and the implementation of the Meta-Harness baseline. This feedback helped us make the evaluation protocols more explicit and sharpen the description of our methodological contribution. We respond to these points in turn below.

For W1: Access to correctness feedback and baseline parity. Publicly released EHRAgent (ehragent/main.py / standard memory-update loop) also uses reference judgments to filter which solution trajectories enter memory. However, we do not claim that shared benchmark conditioning eliminates deployment concerns. To clarify protocol differences across baselines, we summarize the feedback and top-level pass settings below:

SystemMemory Admission FeedbackTop-Level Retry TriggerTop-Level Pass Policy
EHRAgentReference-filtered trajectoriesNone (internal debug only)1 top-level pass
Meta-HarnessFixed read-only initial examplesExecution error signals only1 top-level pass + error retries
EHR-Evo (w/o retry)Reference-filtered trajectories + operational rulesNone (internal debug only)1 top-level pass
EHR-Evo (Full)Reference-filtered trajectories + operational rulesReference judgment on Attempt 1Up to 2 top-level passes

The w/o retry setting and EHRAgent each permit one top-level solving attempt, retain the base agent's internal debugging, and use benchmark correctness for solution-trace admission under our evaluated protocol. Their internal model turns, prompt length, and tool-call costs are not established as identical. Consequently, the observed eICU difference (63.79% versus 53.62% over 580 questions) is evidence of a configuration-level advantage under shared benchmark-conditioned feedback access, not a fully compute-matched, reference-free comparison.

For W2: Novelty beyond combining existing components. We acknowledge conceptual overlap with reflective prompt optimization and memory-curation frameworks. Concretely, our implemented system separates two distinct stores with different update logic (Section 4.4, Appendix A): (1) a solution-trace store that admits only reference-verified trajectories, and (2) an operational-knowledge store updated via structured rules tracking schema relations, tool-return types, and supporting/contradictory execution evidence—allowing failed executions to update operational constraints without polluting worked examples. While Appendix D illustrates rules extracted from writable-memory runs and Table 1 shows that Full exceeds w/o memory by up to 5.52 pp (eICU), isolating the independent causal contribution of each store remains future work.

For W3: Meta-Harness implementation and search-budget parity. Our EHR-specific Meta-Harness adaptation used the reported proposer/solver identities (GPT-5.6-sol / GPT-4-0613) and a fixed search schedule matching EHR-Evo's candidate/split budget (3 rounds × 3 candidates over an 11-setting search space, with disjoint search, development, and test splits, promoting the top 3 candidates by SR then CR). It achieved 23/36 correct development answers across the three development subsets (with 34/36 completed executions and 19 runtime-triggered retries, 9 producing accepted answers), followed by test SRs of 57.93%, 42.41%, and 62.05%. Because its completion rates are relatively high, low SR cannot be explained solely by unfinished executions. However, the observed difference may reflect search-space design, adaptation choices, benchmark scoring, or statistical variability, and we do not ascribe it uniquely to weak generalisation.

Reviewer_Rf9XOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces EHR-Evo, an experience-driven execution harness for electronic health record (EHR) question answering that aims to improve the performance of a frozen language model without updating its parameters. The method operates at two timescales: an outer loop uses execution traces to propose and select harness configurations, while an inner loop uses retrieved operational knowledge, previous solution traces, and feedback-guided retries to improve subsequent executions. The approach maintains separate stores for reusable operational knowledge and concrete solution traces. The central contribution is a practical framework for improving EHR-agent execution through model-external adaptation and accumulated experience. However, the current evaluation does not adequately isolate the contribution of outer-loop harness evolution, and aspects of the benchmark-conditioned feedback and answer-matching procedure weaken the evidence for the claimed improvements.

Critical strengths
  1. The distinction between per-question recovery and cross-question operational memory is useful and well motivated for this domain. This is the most transferable idea in the paper and the main reason I see potential here.

2.Consistent empirical gains across three EHR benchmarks, with complementary execution metrics.

Critical weaknesses
  1. The experimental protocol does not cleanly establish the contribution of harness evolution, and benchmark-conditioned feedback complicates the interpretation of the results. The paper's primary claim concerns evolving the execution harness using experience. However, the reported component variants primarily compare retry behavior and persistent-memory updates. The evidence does not clearly isolate the benefit of the outer-loop evolution procedure itself from the benefits of additional execution attempts, planning guidance, retrieval, and memory.

  2. The paper reports limited repeated-run evaluation and explicitly acknowledges that memory performance may depend on question ordering, initialization, and write policies. The model-routing records also do not establish model snapshot across all conditions. Without stronger answer validation, variance estimates, and controlled comparisons, it is difficult to determine how much of the reported improvement reflects a robust methodological advantage rather than benchmark-specific evaluation behavior or execution variability. These limitations are material because the reported numerical gains are the main evidence supporting acceptance.

Questions for authors

Can the authors isolate the contribution of outer-loop harness evolution?

How much do benchmark-conditioned decisions affect the reported gains?

How robust are the reported success rates under stricter answer evaluation and repeated runs?

10 comments

We thank the reviewer for the careful assessment of our experimental design and the questions regarding component attribution, robustness, and evaluation criteria. These comments helped us clarify what the controlled comparisons establish and distinguish the contributions of harness instructions, memory, and retries. We address each concern below.

For W1 & Q1: Separating harness evolution, retries, and memory. We agree that Table 1 reflects the combined system rather than outer-loop search alone. Appendix C, Table 2 provides a paired harness-content comparison on the exact same 580 eICU questions, holding the frozen solver (GPT-4-0613), four initial examples, planner/compression settings, read-only memory state, and runtime-error-based retry policy constant while varying only the selected harness instructions:

Configuration (eICU, N=580N=580)Overall SROverall CRRuntime-Error RetriesPaired Discordant Outcomes (Init → Sel)
Initial harness38.45% (223/580)88.79% (515/580)67—
Selected harness (after evolution)40.69% (236/580)91.03% (528/580)6917 fixed / 4 regressed (exact McNemar p=0.0072p = 0.0072)
  • Frozen-Memory Provenance & Scope: On the same 580 eICU questions, the selected harness increased success from 223 to 236 answers, with 17 questions changing from incorrect to correct and four changing in the opposite direction (two-sided exact McNemar p=0.0072p = 0.0072). Both conditions shared the frozen read-only memory (produced by an earlier benchmark-conditioned run under a different random seed), the solver, four initial examples, and the runtime-error retry policy. As the memory was obtained from an earlier benchmark-conditioned run, this comparison estimates the effect of harness-instruction content conditional on that memory and is not a sealed, test-disjoint estimate of autonomous adaptation. Nor does it establish superiority to manually written prompts or cross-seed robustness.
  • Protocol Bridge (40.69% in Table 2 vs. 70.69% in Table 1): Table 2 evaluates a restricted ablation (read-only memory, runtime-error retries only, no benchmark-conditioned top-level retry or cross-question memory growth) solely to isolate harness instruction content. Table 1 (70.69% SR) reflects the complete online protocol with dynamic memory admission and benchmark-conditioned top-level retry enabled.

For W2: Robustness and TREQS complexity-stratum alignment.

  • TREQS Complexity-Stratum Denominators: Both methods were evaluated on the same 996 TREQS question identifiers. While we re-ran EHRAgent within our evaluation framework, Table 1 reports the original published per-level breakdown for EHRAgent (527 / 453 / 16, yielding 71.54%, 61.59%, and 18.75%) alongside our EHR-Evo complexity-group mapping (525 / 449 / 22, yielding 80.95%, 79.06%, and 59.09%). Because the per-level assignments differ between these two mappings, we rely on the directly comparable overall success rates over the identical 996 questions (66.16% versus 79.62%) and do not treat the published per-level differences as matched-stratum estimates. We will align the complexity-level mapping across both methods in the revised manuscript.
  • Ablations & Multi-Seed Caveat: Disabling cross-question memory growth (w/o memory) still yields 16.73, 11.55, and 12.15 pp gains over EHRAgent on MIMIC-III (580), eICU (580), and TREQS (996). Because our runs use a single random seed (42) and fixed question order, cross-seed and question-order variance remain open limitations.

For Q2: Contribution of correctness feedback. We agree that our main protocol is benchmark-conditioned: reference judgments govern allocation of the extra top-level attempt and admission of solution traces to memory (though final scoring applies the benchmark matcher strictly to the final-attempt transcript rather than selecting the best attempt). Removing the extra top-level attempt (w/o retry) still outperforms EHRAgent by 17.25, 10.17, and 10.35 pp, showing that the additional top-level attempt is not required for all observed gains under the reported protocol. However, because w/o retry still retains reference-conditioned memory admission and base-agent internal debugging, it does not establish reference-free performance; a reference-free evaluation remains necessary to quantify deployment-level performance.

For Q3: Stricter scoring and CR−SR diagnostic. Success is computed using the original benchmark matcher (Appendix B.5) over the shared 996-question TREQS set and 580-question MIMIC-III/eICU sets. While CR−SR decreases across all three datasets (MIMIC-III: 27.74% → 17.41%; eICU: 30.00% → 22.99%; TREQS: 23.30% → 10.46%), this operational decomposition only tracks completed executions rejected by the benchmark matcher and cannot demonstrate strict semantic correctness. No independent strict-denotation audit or multi-seed variance estimate is available in the current results; we restrict our claims to improvement under the stated benchmark protocol.

Thanks for the additional experiments . I am not convinced how " additional top-level attempt is not required for all observed gains under the reported protocol." And after reading the comments from other reviewers I feel there is some issue in the current version and for now I will maintain my overall score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.