- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes an outer inner loop harness tuning procedure for EHR tasks. The base model of the ehr agent is gpt4 and frozen. The inner agent (model and harness) is depending on the setting allowed to retry (with seeing outcome traces of earlier events) or previous trajectories from the batch. The outer loop is driven by gpt 5.6 sol and proposed harnesses within the restricted design space of planning, memory formation and retrieval and recovery mechanisms based on failures.
Critical strengths
- Ehr is a relevant problem
- Large empirical over EHR agent and baselines
- Code provided (no data) but 3 relevant EHR datasets
Critical weaknesses
- Missing Baselines, related work and novelty. Meta harness is a good baseline. Lack of related work and baselines (gepa for static prompt tying, ace style for memory formation (and retrieve, form, management). This is a form of learning from experience in text space. Related work should mention EHR/FHIR finetuning methods as alternative approach to learn from experience. Just Claude Code or Codex in outer loop to improve the harness of inner model as baseline . It is not shown that the restricted evo-ehr search space is not just an artefact of the gpt 5.6 sol abilities.
- Leakage. Retry is triggered on benchmark eval as wrong. This leaks the outcome, instead of only the execution trace. If there is a perfect “filter” giving correct answers, the agent could generate parallel candidates and just analyse. Compute match a parallel + filter baseline against the sequential version.
- Experimental gains unclear attributed to components ( like outer loop optimisation, ground truth feedback and memory )
Questions for authors
- points in weaknesses
- Why did you choose gpt4 as basemodel in the inner agent. Health is sensitive, so a model that can be self hosted seems an obvious choice.
- Did you experiment with dynamic eval budgets inside the outer loop?
3 comments
Dear Reviewer CgKP,
Thank you again for your detailed review. We would appreciate your reconsideration of our submission in light of the additional evidence and clarifications provided during the rebuttal.
1. Controlled evidence for harness evolution. Our paired eICU experiment holds the frozen solver, examples, read-only memory, and retry policy constant across 580 questions. The selected harness improves success from 223/580 (38.45%) to 236/580 (40.69%), with 17 corrected versus four regressed outcomes (exact McNemar p = 0.0072). This isolates the effect of harness instruction selection under the shared-memory setting, without benchmark-triggered additional retries or online memory updates during the comparison.
2. Contribution beyond existing approaches. EHR-Evo combines execution-driven harness selection with two distinct experience stores: successful solution traces and reusable operational knowledge. Crucially, failed executions can inform schema-navigation and recovery rules without being admitted as successful examples. Our EHR-specific Meta-Harness adaptation uses the same proposer and solver models and matched candidate-search schedules; EHR-Evo achieves +22.07, +28.28, and +17.57 percentage points higher benchmark success across MIMIC-III, eICU, and TREQS.
3. Clarification of oracle feedback. We acknowledge that the main evaluation uses benchmark-conditioned feedback and does not establish fully reference-free operation. However, EHR-Evo without the additional top-level retry still exceeds our EHRAgent rerun by 17.25, 10.17, and 10.35 percentage points, respectively. Both configurations retain reference-filtered solution-memory admission, so this comparison does not isolate every mechanism, but shows that the additional retry alone cannot explain the observed differences.
We recognise that a compute-matched independent-generation baseline remains outstanding. Nevertheless, we believe the controlled experiment, methodological distinctions, and consistent empirical evidence substantively address the concerns about novelty and attribution.
We respectfully ask whether these additional results and clarifications warrant reconsideration of your overall score. We would appreciate knowing if a particular remaining concern is decisive.
Thank you for your consideration.
The Authors
I thank the authors for their comments and additonal runs and increase the score.
We thank the reviewer for the thoughtful questions about methodological novelty, correctness feedback, and the evidence supporting harness evolution. The comments also highlight the importance of making model choices and evaluation budgets explicit. We clarify our contribution relative to existing methods and explain the scope of the experimental evidence below.
W1: Novelty and comparison with existing methods. We agree that reflective prompt optimisation, experience memory, and harness adaptation are established research directions. Our contribution is their execution-specific integration for EHR database agents, with two distinct adaptation stages and two separately maintained forms of experience.
First, the outer loop uses execution traces to select model-external execution instructions covering EHR schema navigation, planning, and error recovery. Second, after harness selection, the execution system maintains both reference-filtered solution trajectories and operational knowledge. The latter can incorporate lessons from failed executions—such as invalid schema references, patient-versus-admission scope errors, and tool-return-type mismatches—without retaining those executions as successful solution examples.
This differs in optimisation target and implementation from general prompt optimisation and evolving context management, although the underlying ideas are related. We do not claim that individual components are unprecedented or that our experiments establish superiority over GEPA or ACE.
Our EHR-specific Meta-Harness adaptation uses the same named proposer and solver models and matched candidate-search schedules. EHR-Evo achieves higher benchmark success rates across all three datasets (+22.07, +28.28, and +17.57 pp on MIMIC-III, eICU, and TREQS). Our eICU confidence-prompt reconstruction of TrustEHRAgent covers all 580 eICU questions under the identical evaluator and achieves 41.38% SR / 93.79% CR. We interpret these comparisons within their stated search spaces, implementation choices, and evaluation protocols.
W2: Correctness feedback and the parallel-generation baseline. We acknowledge that the main evaluation uses reference judgments for retry allocation and solution-memory admission. Consequently, these results establish performance under a benchmark-conditioned protocol, not fully reference-free deployment performance.
We have not performed the requested compute-matched parallel generation and oracle-filtering comparison (nor do we treat
Full − w/o retryas a substitute for independent parallel sampling). Therefore, we do not claim that sequential repair is superior to independent candidate generation under equal computation and access to correctness feedback.W3: Contribution of harness evolution and component attribution. We provide an additional controlled analysis of harness instruction content. On the same 580 eICU questions, the initial and selected configurations use the same frozen solver, initial examples, read-only memory, planning settings, and runtime-error-based retry policy. The selected instructions improve answer success from 223/580 (38.45%) to 236/580 (40.69%), with 17 previously unsuccessful questions corrected and 4 previously successful questions regressed (exact McNemar p=0.0072).
This comparison supports the contribution of the selected execution instructions under the shared frozen-memory setting. Because the memory originated from an earlier benchmark-conditioned run, it does not establish reference-free learning or the superiority of automated search over handcrafted instructions.
Table 1 additionally shows configuration-level performance differences under the full evaluation protocol:
Because
w/o memoryretains initial seed examples andw/o retryretains internal debugging, these full-system comparisons demonstrate performance differences associated with enabling recovery and memory updates, but do not independently identify their causal contributions.Q2–Q3: Model selection and evaluation budgets.
GPT-4-0613was selected to maintain compatibility with the EHRAgent experimental setting on de-identified benchmarks, rather than as an optimal model for clinical deployment. Validation with self-hosted models remains future work.