acceptodds
Under review as a conference paper at ICLR 2027

EvoMedAgent: Auditable Reasoning and Failure-Driven Memory Evolution for Medication Recommendation

Abstract

Medication decisions often involve multiple conditions, ongoing therapies, and patient-specific safety constraints. Effective medication recommendations therefore must go beyond generating clinically plausible outputs to support auditable and safe real-world use. This calls for two complementary capabilities that, however, are often overlooked in existing methods: (1) auditability, enabling clinicians to trace how patient evidence and competing treatment needs lead to individual decisions; and (2) correctability, enabling the system to identify recurrent recommendation errors in comparable clinical contexts, derive targeted corrections, and validate them before updating for safe decisions.We introduce EvoMedAgent, an agentic medication recommender that couples an auditable reasoning pipeline with failure-driven memory evolution. The pipeline makes medication selection auditable through evidence-grounded unit decisions, patient-level reconciliation, and independent review, linking these judgments to each final medication in a reasoning trace. Using this trace and recorded medication sets, the evolution process attributes observed errors, proposes clinically scoped corrections, and validates them before incorporating them into memory. Extensive experiments show that EvoMedAgent achieves substantial improvements over state-of-the-art baselines on real-world datasets, with ablations supporting both mechanisms, and human and LLM evaluations supporting improved auditability.Our code is available at https://anonymous.4open.science/r/EvoMedAgent/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medication Recommendation, LLM Agents, Auditable Reasoning, Failure-Driven Memory Evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_25ToOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoMedAgent, a multi-agent medication recommendation framework that combines auditable reasoning with failure-driven memory evolution. Experiments on MIMIC-III and MIMIC-IV demonstrate relative Jaccard improvements of 36.20% and 33.24% over other baselines. Ablation studies validate the contributions of both components, while evaluations by two physicians and an LLM judge on 50 cases demonstrate improved reasoning auditability. However, the contributions of failure attribution and memory evolution are not sufficiently isolated from existing agentic methods.

Critical strengths
  1. Identify two important challenges in medication recommendation: the lack of decision auditability and the limited ability to learn from observed failures, both of which are particularly relevant in clinical settings.
  2. Propose EvoMedAgent framework to employ a structured multi-agent architecture with specialized stages and clearly defined responsibilities, enabling systematic medication reasoning and decision review.
  3. Extensive evaluation results demonstrate strong empirical performance against other baselines on MIMIC-III and MIMIC-IV, supported by ablation studies that highlight the contributions of key architectural components.
Critical weaknesses
  1. Learning from failures and evolving reusable memory have been explored in existing agent frameworks, such as ReasoningBank. The novelty of typed failure attribution and memory evolution should be better articulated.
  2. Medication decisions are often patient-specific and subject to clinical judgment. Accumulating failure experiences may introduce clinical biases, conflicting knowledge, or irrelevant information, potentially affecting future recommendations.
  3. Figure 1(a) and Line 088 emphasize clinician auditability, but Section 3.1 mainly describes automated interactions among specialized LLM agents. The authors should clarify the distinction between automated reasoning trace generation and actual clinician auditability, and carefully revise this contribution.
  4. Figure 2 shows unusually close agreement between physician and LLM evaluations, raising concerns about the evaluation protocol. In particular, the review and revision accountability criterion in Table 21 involves reconstructing complex decision processes, where large differences between physicians and LLMs might reasonably be expected. Such close alignment raises questions about whether the evaluation criteria are sufficiently rigorous and discriminative to meaningfully assess reasoning quality.
  5. MIMIC-III and MIMIC-IV originate from the same institution, this makes assessment of the generalizability of the proposed framework difficult.
Questions for authors
  1. The main novelty appears to lie in integrating existing agentic mechanisms with typed failure attribution and scoped memory updates. However, the distinction from methods such as SafeRx- Agent and SkeMex remains unclear. Table 19 provides only one relatively simple case, where the improvements might also be achievable through prompt refinement. Could the authors provide more detailed comparative case studies?
  2. The framework identifies recommendation failures by comparing predicted medications with historical prescriptions and validates memory updates using the same training cases that motivated these corrections. However, historical prescriptions may not always represent optimal clinical decisions. Could the authors clarify how they distinguish recommendation errors from differences in prescribing practices.
12 comments
AuthorsOpenReview ↗

Response to Reviewer#25To PART-1

We thank the reviewer for recognizing the research problem, structured multi-Agent reasoning, and component ablations, and for the constructive suggestions regarding methodological novelty, the reliability of supervision from historical prescriptions, clinical auditability, and cross-institutional generalization. We address Q1, Q2, and the remaining Weaknesses in turn, distinguishing conclusions supported by the current evidence from clinical claims that remain unverified.

Q1. Methodological Novelty and Distinctions from Existing Agent Frameworks (W1)

Q1.1 What is the principal methodological novelty?

EvoMedAgent's novelty lies in attributing fine-grained medication recommendation failures to specific decision stages and correcting them through clinically scoped memory updates—not in being the first to introduce multiple Agents or long-term memory.

Fine-grained medication recommendation must accommodate multiple disease-specific treatment goals, candidate-medication constraints, inter-condition treatment conflicts, and patient-specific safety requirements. The same final medication-set discrepancy may arise at different stages and require different corrective actions. For example, an omitted drug may never enter the entity-owned candidate set (Candidate Failure), or it may be present as a candidate but excluded during Unit Reasoning or patient-level reconciliation (Selection Failure). Although the output discrepancy is identical, the appropriate points of correction differ. EvoMedAgent therefore forms the closed loop Typed Failure Attribution → Layer-Specific/Scoped Update → Cross-Case Validation → Subsequent Reuse, rather than reducing all failures to clinically unscoped reflections or general summaries of past trajectories.

In Table 2, removing Typed Failure Attribution, Scoped Routing (Unified Memory), or Corpus-Level Validation reduces Jaccard from 48.61% to 45.78%, 45.16%, and 44.49%, respectively. To distinguish the respective contributions of the architecture, auxiliary tools, candidate priors, and validation strategy, we conducted three controlled experiments and a rule ablation on patients excluded from Memory Evolution.

(a) Test-Time Auxiliary Tool Isolation. With the final evolved Memory, Candidate Boundary, and full Agent reasoning pipeline fixed, we disable the ARMR Retriever, Calibration Tool, and Residual Tool individually or all three together:

ConfigurationJaccard ↑ (%)F1 ↑ (%)DDI ↓ (%)
Full EvoMedAgent48.6164.187.18
w/o ARMR Retriever48.4063.987.27
w/o Calibration45.4961.177.26
w/o Residual48.4764.057.38
w/o All Three44.2959.978.11

With all three auxiliary tools disabled, the system still achieves 44.29% Jaccard / 59.97% F1. Thus, the tools contribute to performance but do not fully explain the system's predictive ability.

(b) Shared Candidate Boundary Comparison. We preserve the native reasoning pipelines and tools of SkeMex, RareAgents, and PRTA, while imposing EvoMedAgent's training-derived case-level Candidate Boundary on each baseline. The following results show changes relative to each native baseline configuration:

BaselineΔJaccard\Delta\mathrm{Jaccard}ΔF1\Delta F_1ΔDDI\Delta\mathrm{DDI}ΔAvgMed\Delta\mathrm{AvgMed}
SkeMex−0.3245−0.3983+0.3865−0.613
RareAgents−1.2916−1.5467−0.5350−6.786
PRTA−0.6859−0.7235−0.1227−0.516

None of the three baselines improves Jaccard or F1, indicating that the effectiveness of candidate constraints depends on their compatibility with each method's native reasoning pipeline.

[Q1.1 continues in Part 2.]

AuthorsOpenReview ↗

Response to Reviewer#25To PART-2

[Q1.1 continued from Part 1.]

(c) Out-of-Sample Rule Validation Strategy. Holding the rule-proposal mechanism and screening criteria fixed, we replace full-training-corpus screening with a validation set disjoint from the proposal-source patients, and compare it with a configuration without corpus-level validation on the same MIMIC-III test set:

Validation StrategyJaccard ↑ (%)F1 ↑ (%)DDI ↓ (%)
No-validation44.4960.147.88
Out-of-sample46.1961.937.10
In-sample (Original)48.6164.187.18

Out-of-sample screening improves Jaccard/F1 over No-validation by 1.70/1.79 pp, but remains below In-sample screening, which has broader eligible-case coverage.

Additionally, on validation patients excluded from Memory Evolution, we fix the full test-time configuration and disable only the evolved Experiential Rules: Jaccard is 48.9948% vs. 46.9492%, and F1 is 64.4757% vs. 62.7182%, with and without those rules, respectively.

Together, these experiments clarify the limits of attributing performance gains to tools, candidate priors, and memory updates.

Q1.2 How does EvoMedAgent differ from ReasoningBank, SafeRx-Agent, and SkeMex, and what ablation evidence supports the distinction?

ReasoningBank already extracts reusable experience from reasoning trajectories; SafeRx-Agent employs specialty collaboration and safety verification; and SkeMex includes a skill repository and utility-based skill governance. We do not characterize these established capabilities as absent from the prior methods. EvoMedAgent's key distinction is to localize medication recommendation discrepancies to specific decision stages and combine Typed Failure Attribution, Layer-Specific/Scoped Memory Update, and Cross-Case Validation into a clinically scoped failure-driven memory evolution mechanism.

The Table 2 ablations directly support the performance contributions of these core design choices. The Jaccard scores for Full, Static Memory, w/o Typed Failure Attribution, Unified Memory, and w/o Corpus-Level Validation are 48.61%, 43.73%, 45.78%, 45.16%, and 44.49%, respectively. These results indicate that the gains are associated not only with experiential memory itself but also with typed failure attribution, targeted scope management, and cross-case rule validation, supporting the importance of these mechanisms within the full system.

In addition, Test-Time Auxiliary Tool Isolation in Q1.1(a) shows that, with evolved memory and the complete reasoning pipeline fixed, the system still achieves 44.29% Jaccard / 59.97% F1 when all three auxiliary tools are disabled. This further shows that the full system's performance advantage cannot be explained solely by learned auxiliary tools. Together with the mechanism ablations, these findings provide complementary experimental evidence for EvoMedAgent's structured, failure-driven memory evolution design.

Q1.3 Additional detailed cross-patient case study

The original Table 19 primarily illustrates reasoning coordination for a complex patient and does not sufficiently demonstrate Memory Evolution. We therefore provide actual cross-patient execution traces linking Failure Attribution → Scoped Update → Cross-Case Validation → Memory Reuse, and compare the reasoning behavior of three Agents on the same test case.

(a) Patient A: From a specific medication omission to a targeted memory update.

Training case 3362 underwent single-vessel coronary artery bypass grafting (CABG). Metoclopramide was already present in the procedure's Candidate Memory but was not selected during Unit Reasoning and did not appear in the final recommendation set. The system therefore attributed this omission to Selection Failure rather than Candidate Failure.

Together with a similar omission in training case 3207, the system proposed a prefer Metoclopramide rule scoped to single-vessel CABG units with the role Primary or Comorbidity-coupled. The rule acts only on an existing candidate; it is not an unconditional medication recommendation.

Memory Evolution stageObserved result
Failure AttributionExisting candidate omitted during Unit Reasoning → Selection Failure
Scoped UpdateSingle-vessel CABG + specified unit role → prefer Metoclopramide
Cross-Case Validation43 eligible / 43 supporting, confidence = 1.0
Counterfactual UtilityΔJ\Delta J = +0.003751, ΔF1\Delta F_1 = +0.003472
Memory UpdatePassed validation; written to Experiential Memory and kept Active

[Q1.3 continues in Part 3.]

AuthorsOpenReview ↗

Response to Reviewer#25To PART-3

[Q1.3 continued from Part 2.]

After excluding the two proposal-source patients, all 41 remaining eligible cases still support the rule. Thus, the observed experience pattern is not supported solely by its source patients. The reported ΔJ\Delta J / ΔF1\Delta F_1 represent cached set-level counterfactual estimates on training cases, not performance improvements measured on the test set.

(b) Patient B: Matching and reusing the validated rule in a different clinical context.

Independent test case 5404 primarily involved mitral and aortic valve surgery, together with single-vessel CABG and pulmonary comorbidities. Unlike Patient A, the CABG unit in this case had the Comorbidity-coupled rather than Primary role.

The case satisfied the rule's scoped applicability conditions. The coronary unit retrieved the rule; Metoclopramide appeared in the matched experiential rules, was selected during Unit Reasoning, and remained in the final medication set. This provides traceable evidence of training-case failure → scoped correction → conditional matching in a new patient → actual reuse.

(c) Comparison of actual reasoning workflows for the same patient.

To further clarify the design differences, we examined the recorded execution traces of all three methods on Patient B:

MethodActual module sequence and key observationsFinal result
SafeRx-AgentRoute activates 5 experts → Summarize / Generate proposes 20 drugs, excluding Metoclopramide → Critique retains 19 drugs → FindFlags / Verify checks interactions and removes some label-positive medications13 drugs; 8 TP / 5 FP / 14 FN; Jaccard 29.63%
SkeMexRead retrieves no applicable skill (retrieved_skill_ids=[]) → drug_info / ddi_check queries candidates and safety evidence without involving Metoclopramide → final outputs 14 drugs14 drugs; 12 TP / 2 FP / 10 FN; Jaccard 50.00%
EvoMedAgentClinical Decomposition identifies valve, CABG, and pulmonary clinical units → Unit Reasoning retrieves and invokes the validated CABG experiential rule, selecting Metoclopramide → Evidence Reconciliation integrates cross-unit evidence, removes duplicates, and addresses local omissions → Trace Review / Faithful Summarization records unresolved risks and preserves consistency of the final set26 drugs; 22 TP / 4 FP / 0 FN; Jaccard 84.62%

In this case, neither SafeRx-Agent's expert proposals nor SkeMex's tool interactions involved Metoclopramide, whereas EvoMedAgent retrieved and used the validated rule in the CABG unit. EvoMedAgent matched all 22 historically recorded medication labels, including propofol, glycopyrrolate, and neostigmine—anesthetic/reversal agents omitted by both baselines—as well as other perioperative supportive medications. This illustrates how Clinical Decomposition, Unit Reasoning, and Patient-Level Reconciliation jointly organize medication decisions for a complex, multimorbid patient.

Importantly, neither the final selection of Metoclopramide nor the case-level Jaccard of 84.62% can be attributed solely to this rule: memory from other clinical units and auxiliary tools also contributed. The case demonstrates the traceable process of rule generation, validation, retrieval, and cross-patient reuse, alongside differences in the actual decision behavior of the frameworks.

In summary, EvoMedAgent differs from existing Agent frameworks not merely by storing past failures but by identifying the decision stage responsible, turning that failure into a clinically scoped correction, validating it across cases, and reusing it in subsequent reasoning. Together with clinical decomposition and patient-level evidence reconciliation, this mechanism constitutes the central design distinction of our work.

Reviewer_iXW5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoMedAgent, a multi-agent framework for fine-grained electronic health record medication recommendation that couples an auditable reasoning pipeline with failure-driven memory evolution. At inference time, specialized LLM agents and deterministic operators generate drug recommendations through structured reasoning and safety review. With training feedback, the system identifies recurring errors and updates hierarchical memory through layer-specific validation and counterfactual screening. Experiments on public medical datasets show substantial performance improvements over existing baselines, supported by component ablations and a human-in-the-loop auditability evaluation.

Critical strengths
  • Decomposing multimorbid visits into clinical problem units followed by patient-level reconciliation is a well-motivated approach to competing treatment objectives, and the deterministic stage contracts keep the reasoning trace structurally consistent across handoffs.
  • The failure-driven memory design is intuitive and effective: typed failure attribution, scoped routing across four memory layers, and a validation gate each contribute roughly 3 to 4 Jaccard points in the Table 2 ablations.
  • The empirical study is broad, covering MIMIC-III and MIMIC-IV, diverse neural, LLM, and agent baselines, long-tail cohorts.
  • Clinical auditability is evaluated explicitly using blinded trace dossiers, two physician raters with strong inter-rater agreement, an LLM judge, and an objective defect-localization task.
Critical weaknesses
  • While the paper addresses an important problem, its technical novelty remains limited, as the proposed approach does not sufficiently demonstrate fundamental methodological advances beyond existing agentic reasoning and memory evolution techniques.
  • The source of the large performance margin is hard to isolate. Beyond the proposed reasoning and evolution mechanisms, EvoMedAgent restricts decisions to a training-label-derived candidate boundary (covering 96.3% of observed test labels on MIMIC-III, App. E.1) and uses a training-only ARMR-based similar-case retriever and two learned auxiliary tools, a calibration tool and a residual tool (Table 10). App. C.3 states that baselines keep their original tools and that the comparison is system-level rather than tool-normalized, but it does not say whether any baseline receives a comparable candidate boundary or learned components. On MIMIC-III, even the weakest variants in Table 2 (Pipeline only and Holistic reasoning) exceed the strongest baselines in Jaccard and F1, and Sec. 4.3 notes that stage ablations also remove stage-specific tools, which apparently confounds the effect of the reasoning stages with that of the estimators. It is also not stated which tools and candidate boundary each variant retains.
  • The memory evolution and validation pipeline may suffer from in-sample optimism and a train-test mismatch. Per App. E.4, training traces used for failure attribution and rule validation are generated with memory bootstrapped from those same cases’ labels (unlike the similar-case retriever, which excludes the query patient; no analogous exclusion is described), and counterfactual gains are computed over all eligible training cases, including those that motivated the rule (App. E.3). Moreover, because the learned estimators are applied only at test time (App. H.1), rules are validated on a pipeline configuration different from the deployed one, and the validation-split labels, which Table 5 excludes from evolution, are never used as an out-of-sample check for rule retention. The aggregate benefit of the validation gate in Table 2 does not address whether individual rules’ ΔJ estimates are biased.
  • Some ablation and auditability claims are not fully established. In Table 2, the DDI gain that Sec. 4.3 credits to Trace Review–Correction (7.42 vs. 7.18) is hard to distinguish from run-to-run noise. Table 2 appears to be a single run (its full-model row matches Table 1, not the five-run mean in Table 6), and a gap of 0.24 is comparable to the noise expected between two independent single runs given the full model’s test-time std of 0.17 (Table 6). The paper does not state whether this variant reuses the full run’s upstream outputs (which would make the comparison paired), so this improvement is not established without repeated runs or a paired test. In addition, the auditability rubric appears closely aligned with EvoMedAgent’s own stage contracts (e.g., invariants I1/I2 and the summary contract in Table 11 map onto the rubric’s traceability, grounding, and faithfulness dimensions). Since the physicians and the LLM judge score against the same rubric, their agreement (Table 22) does not rule out rubric bias, and the LLM judge shares EvoMedAgent’s backbone.
Questions for authors
  • Which tools and candidate boundary does each Table 2 variant retain, and how does EvoMedAgent perform when the ARMR retriever and the calibration and residual tools are disabled at test time (each separately and jointly)?
  • Do all LLM and agent baselines share the same backbone, and how do the strongest agent baselines (e.g., SkeMex, PRTA) perform if given the same training-label-derived candidate boundary?
  • Was any leave-one-out or exclusion applied when bootstrapping candidate and experiential memory for the training predictions used in failure attribution? Do the retained experiential rules still improve agreement with the observed medications on held-out validation patients under the full test-time configuration?
  • Is the DDI improvement from Trace Review–Correction robust across repeated runs or a paired patient-level test, and does the auditability ranking hold under an LLM judge from a different model family?
11 comments
AuthorsOpenReview ↗

Response to Reviewer#iXW5 PART-1

We thank the reviewer for recognizing the clinical problem decomposition, patient-level reconciliation, failure-driven memory evolution, and auditability design. To address concerns about performance attribution, baseline fairness, in-sample effects of memory updates, and the reliability of auditability evaluation, we provide controlled experiments for the four main questions and clarify the methodological novelty.

Q1. Contributions of Auxiliary Tools and Ablation Configurations

Q1.1 Which tools and Candidate Boundaries are retained in each ablation? (W2.2, W2.3)

We agree that the original paper did not sufficiently distinguish stage ablations from tool ablations. The original experiments in Table 2 are System-Level Ablations: removing a reasoning stage also removes its stage-specific tool interfaces. Memory-Level Ablations retain the normal reasoning stages and tool permissions. The configurations are as follows:

VariantARMR RetrieverCalibrationResidual
Full EvoMedAgent✓✓✓
Holistic Reasoning———
w/o Patient-Level Reconciliation✓—✓
w/o Trace Review–Correction✓✓—
Pipeline Only✓✓✓
Static Memory✓✓✓
w/o Typed Failure Attribution✓✓✓
Unified Memory✓✓✓
w/o Corpus-Level Validation✓✓✓

✓ indicates that the tool retains access at its original permitted stage, not that it is invoked in every inference run; — indicates that the corresponding interface is removed.

The Candidate Boundary is primarily provided by Candidate Memory. Static Memory and Pipeline Only both use Bootstrap Candidate Memory; the latter disables long-term guidance from Experiential, Strategic, and General-Heuristic Memory. Other variants use the evolved Candidate Memory associated with their respective configurations.

In Q1.2, we additionally hold the full reasoning pipeline and Candidate Boundary fixed to isolate the contributions of the three tools.

Q1.2 What are the separate and combined contributions of the three auxiliary tools? (W2.2)

We conducted Test-Time Auxiliary Tool Isolation: with evolved Memory, Candidate Boundary, and the complete Agent reasoning pipeline fixed, we disable the ARMR Retriever, Calibration Tool, and Residual Tool individually or all three together.

ConfigurationJaccard ↑ (%)F1 ↑ (%)DDI ↓ (%)
Full EvoMedAgent48.6164.187.18
w/o ARMR Retriever48.4063.987.27
w/o Calibration45.4961.177.26
w/o Residual48.4764.057.38
w/o All Three44.2959.978.11

Calibration has the largest individual contribution: disabling it reduces Jaccard/F1 by 3.12/3.01 pp, whereas disabling ARMR or Residual reduces Jaccard by approximately 0.21/0.14 pp, respectively. Disabling all three tools still yields 44.29% Jaccard / 59.97% F1, approximately 4.32/4.21 pp below Full. Thus, auxiliary tools account for some of the gains, but the core reasoning pipeline with evolved memory retains substantial predictive ability. We will clarify the distinction between Stage-Level System Ablations and Test-Time Tool Isolation in the revision.

AuthorsOpenReview ↗

Response to Reviewer#iXW5 PART-2

Q2. Fairness of Baseline Backbones and Candidate Boundaries

Q2.1 Do baselines use the same Backbone, and which native tools are retained? (W2.1, W2.2)

As described in Section 4.1 and Appendix C.3, all methods use the same gpt-5.6-terra Backbone (medium reasoning effort), patient information boundaries, output medication vocabulary, and evaluation protocol, while retaining their native tools and reasoning mechanisms. Thus, matching the Backbone does not imply identical candidate generation or non-LLM auxiliary tools. We summarize the main tools, memory mechanisms, and candidate-generation strategies below:

MethodNative Tools / MemoryCandidate Generation / Boundary
Vanilla LLM / CoTDirect prompting / structured reasoningShared medication vocabulary; no entity-scoped candidate memory
MARC v1Evidence Extractor, Medication Set Reasoner, optional DrugBank retrievalReasoner-generated medication set
SafeRx-AgentSpecialty experts, indication evidence, DDI and contraindication checks, Critique/VerifyExpert-generated proposal union
RareAgentsMultidisciplinary specialists, treatment memory, DrugBank and DDI evidenceSpecialist proposals
D2MDTAdaCare, MCGRU, RETAIN predictors; explanation and consultation modulesMedication-wise predictions over the full vocabulary
ClinicalAgentsWorking/experience memory, biomedical retrieval, DDI checking, iterative verificationIteratively generated medication hypotheses
SkeMexSkill repository, skill retrieval, DrugBank and DDI tools, ReAct reasoningSkill-guided selection; no entity-scoped medication boundary
PRTAGAMENet, ARMR, SafeDrug; LLM tool routing, weighting and reflection memoryTool-generated Top-20 candidates for LLM reasoning; final fusion over the full vocabulary by default
MACRecManager, Analyst, Searcher and Reflector agentsCollaborative medication recommendation
EvoMedAgentFour scoped memory layers; ARMR retrieval, biomedical evidence, DDI checking, Calibration and Residual ToolsDiagnosis/procedure-owned Candidate Memory with explicit unit-level constraints

All methods use the same standardized medication output vocabulary, but their internal candidate mechanisms differ. SkeMex retrieves reasoning Skills rather than entity-scoped medication sets; PRTA provides the LLM with Top-20 candidates from GAMENet, ARMR, and SafeDrug. EvoMedAgent instead enforces a diagnosis/procedure-scoped Candidate Boundary before Unit Reasoning, with subsequent selection and correction constrained by that boundary. The original experiments therefore compare native end-to-end systems under a shared Backbone, rather than fully matching their candidate-generation mechanisms.

AuthorsOpenReview ↗

Response to Reviewer#iXW5 PART-3

Q2.2 What happens when strong baselines receive the same training-label-derived Candidate Boundary? (W2.1)

To test whether the Candidate Boundary constructed from training labels independently explains the performance advantage, we conducted Shared Candidate Boundary Comparison. We retain the native reasoning pipelines and tools of SkeMex, RareAgents, and PRTA while imposing the same case-level candidate boundary on all three. For SkeMex and RareAgents, it replaces the medication-choice vocabulary in the prompt; PRTA ranks candidates within the shared boundary.

BaselineΔJaccard\Delta\mathrm{Jaccard}ΔF1\Delta F_1ΔDDI\Delta\mathrm{DDI}ΔAvgMed\Delta\mathrm{AvgMed}
SkeMex−0.3245−0.3983+0.3865−0.613
RareAgents−1.2916−1.5467−0.5350−6.786
PRTA−0.6859−0.7235−0.1227−0.516

Δ\Delta = Shared Candidate Boundary − Native Baseline. Negative Jaccard/F1 values indicate reduced accuracy; negative DDI values indicate a lower interaction-rate proxy.

None of the three baselines improves Jaccard or F1 with the shared Candidate Boundary (see the Δ\Delta values); RareAgents also reduces AvgMed by approximately 6.79. Candidate constraints can exclude some recorded medications and alter each Agent's native proposal generation, tool fusion, and final selection behavior. Their usefulness therefore depends on how well they integrate with the underlying reasoning pipeline. These findings do not negate the role of Candidate Memory within EvoMedAgent; together with Q1.2, they indicate that EvoMedAgent's performance cannot be explained simply by providing the same candidate set or auxiliary tools in isolation.

Q3. Bootstrap Self-Influence, Rule Validation, and Training–Test Configuration Differences

Q3.1 Does Bootstrap perform Leave-One-Out construction, and how might a training patient's own labels affect it? (W3.1)

The original Bootstrap does not perform patient-wise Leave-One-Out construction. It builds shared entity–medication Candidate Memory and initial Positive Experiential Rules from the training corpus (Appendix E.1), using smoothed statistics, multi-case support, and per-entity rule-count limits to provide population-level priors, rather than patient-specific label inputs.

To assess potential self-influence from a training patient's labels, we conducted a statistical sensitivity analysis excluding that patient's own contribution. None of the 404 initial Experiential Rules was supported by only one patient; the mean number of supporting patients was 53.42. For Candidate Memory, the mean number of supporting patients was 4.59. The mean candidate count for training cases changed from 244.73 to 244.10, while the coverage of recorded medications remained 96.40%, close to the Bootstrap coverage of 96.32% on the independent test set.

These statistics suggest that Bootstrap primarily provides shared cross-patient priors, and its high coverage does not depend entirely on the current training patient's own labels. Nevertheless, the original training predictions cannot be characterized as strictly Leave-One-Out or as providing unbiased Failure Attribution. Q3.2 further evaluates the incremental contribution of retained Experiential Rules on patients excluded from Memory Evolution.

Q3.2 Are retained Experiential Rules useful for independent patients under the full test-time configuration? (W3.2, W3.3)

We conducted Held-out Full-Configuration Experiential Rule Ablation on validation patients excluded from Memory Evolution. We held the other memory layers, Calibration, Residual, and full reasoning pipeline fixed, disabling only the evolved Experiential Rules.

ConfigurationJaccard ↑ (%)F1 ↑ (%)DDI ↓ (%)AvgMed
w/o Evolved Experiential Rules46.949262.71826.932217.8770
Full Evolved Memory48.994864.47577.034418.0916
Δ (Full − w/o Rules)+2.0456+1.7575+0.1022+0.2146

ΔJaccard\Delta\mathrm{Jaccard}, F1F_1, and DDI are measured in percentage points.

Retaining the evolved rules improves Jaccard/F1 by 2.05/1.76 pp, demonstrating incremental predictive value for patients not used in Memory Evolution under the full test-time configuration.

Reviewer_3y2iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoMedAgent, an intelligent agent for medication recommendation that integrates an auditable inference framework with failure-driven memory evolution. The inference framework breaks down the patient's clinical state, generates evidence-based medication recommendations, and performs role-based review. The memory module categorizes errors and iteratively updates memories based on the discrepancy between predictions and actual medication use. Performance on MIMIC-III/IV is superior to baseline, and its effectiveness and auditability are validated by ablation experiments and a combined evaluation of 50 cases. While the research topic is cutting-edge and the results are promising, further improvements are needed in verifying the independence of memory updates, as well as the sufficiency of clinical interpretation and audit evidence.

Critical strengths
  1. The system integrates clinical decomposition, unit reasoning, patient-level reconciliation, and verification into a structured process, preserving the complete reasoning trajectory and overcoming the limitations of relying solely on textual interpretation to achieve verifiable decision-making.

  2. Significantly outperforms baseline on MIMIC-III; ablation experiments confirm the effectiveness of evolutionary memory and verification mechanisms; cost, uncertainty analysis, and physician evaluation further support the conclusions.

Critical weaknesses
  1. Updates are based on the training set and validation data from the same source, which easily introduces selection bias; utility calculations do not run the entire pathway and only a single evolutionary trajectory, raising questions about memory stability.

  2. The usability of features in medication decisions cannot be confirmed, and task attributes are ambiguous; auditability only checks trajectory completeness; greater computational overhead interferes with the effectiveness of mechanism attribution.

Questions for authors

Q1: How are information boundaries defined during prediction? Please explain the temporal sequence of various MIMIC features, how post-encoding is handled, and clarify whether the task is retrospective reconstruction or look-ahead support.

Q2: Why use the entire training set to validate memory updates? Is the cached counterfactual estimation consistent with the end-to-end results? Please provide multi-round evolutionary variance and compare different validation settings.

Q3: Does the gain come from the architecture itself, rather than computational power or trajectory format? Please provide evidence that doctors use trajectories to identify clinical errors, and the results under the same token budget.

8 comments
AuthorsOpenReview ↗

Response to Reviewer#3y2i PART-1

We thank the reviewer for the specific concerns regarding information boundaries, the reliability of memory validation, and performance attribution. We clarify the experimental setting, present additional controlled experiments, and delineate the scope of the conclusions below.

Q1. Prediction-Time Information Boundaries and Retrospective Task Definition

Q1.1 Task definition (W2.1)

Our evaluation concerns retrospective visit-level medication-set reconstruction, not prospective prescription prediction. Following the MIMIC-based medication recommendation paradigm (Section 4.1; Appendices C.1–C.2), EvoMedAgent uses the visit-level context of the target hospitalization and the patient's prior history to reconstruct the medication set recorded for that admission. It does not simulate real-time decisions made at a specific clinical time point using only information available at that time. Accordingly, these results do not establish deployment effectiveness for autonomous prescribing. We will clarify this task definition in the revision.

Q1.2 Temporal boundaries of structured fields (W2.1)

Structured inputs include demographics, diagnoses and procedures for the target visit, restricted laboratory results, and prior visit history. Some diagnosis and procedure codes and Lab Events may become available during the hospitalization or only be finalized at discharge. They are therefore compatible with retrospective reconstruction but cannot be assumed to have been available before every prescribing decision. We will explicitly state this temporal boundary in the revision.

For Lab Events, we do not use all tests from the entire hospitalization. Instead, we apply the following filters:

  • Temporal constraint: Retain only abnormal laboratory results within the 48-hour window preceding the start of any prescription during the admission.
  • Abnormality filter: Retain only laboratory results flagged as abnormal.
  • Item deduplication: For the same ITEMID within one admission, retain only the earliest qualifying abnormal result.

This filtering preserves medication-relevant abnormal findings while limiting the influence of subsequent repeated tests. However, because the window is constructed using retrospectively recorded prescription times, it does not guarantee information availability at an arbitrary prospective decision time point. Prior studies on fine-grained medication recommendation [1, 2] provide empirical motivation for the 48-hour window; specifically, [1] reports that performance is no longer substantially sensitive to window length beyond one day. We will provide the filtering details and clarify this limitation in the revision.

Q1.3 Clinical text filtering and label isolation (W2.1)

The model does not directly read complete discharge summaries. Appendix C.2 specifies two filtering steps: we extract only five sections—Chief Complaint, History of Present Illness, Past Medical History, Family History, and Allergies—and then use constrained LLM summarization to exclude information about treatment, medications administered, and subsequent care during the target admission. Although the source documents are generated at discharge, the inference input is a clinical summary restricted by both section selection and content filtering.

Table 5 further separates the uses of medication labels: training labels are used for memory initialization, evolution, and auxiliary-tool fitting; validation labels for configuration selection; and test labels only for final evaluation. The target visit's medication labels are not provided as input when generating its recommendations. These safeguards define the retrospective reconstruction setting; they do not establish a strictly prospective medication-decision setting.

[1] Li Y, Zhang Q, Wang S, et al. Medication mapping and diagnosis enhancement for fine-grained medication recommendation[J]. Information Sciences, 2026: 123930.

[2] Bhoi S, Lee M L, Hsu W, et al. REFINE: A fine-grained medication recommendation system using deep learning and personalized drug interaction modeling[J]. Advances in Neural Information Processing Systems, 2023, 36: 24013-24024.

AuthorsOpenReview ↗

Response to Reviewer#3y2i PART-2

Q2. Independent Generalization of Memory Updates, Counterfactual Estimation, and Evolution Stability

Q2.1 Why validate rules on the full training corpus rather than a disjoint validation set? (W1.1)

The full training corpus provides more case-level support for population-level rule-utility estimation, including eligible cases for long-tail rules; the independent validation experiment directly tests whether the benefits of screening depend on in-sample data.

(a) Validation covers the rule-eligible population, not only the proposal-source cases

FailureCards trigger rule proposals, but validation in Appendix E.3 retrieves all eligible cases in the training corpus that satisfy the rule's clinical entity, unit role, and applicability conditions; it does not merely revisit the cases that generated the proposal. Standard rules must meet joint thresholds for support, confidence, and cached set-level utility:

\Delta J(q) > 0,\quad \Delta F_1(q) \ge 0.$$ Provenance and scope consistency are also checked. Restricted single-case tentative `prefer` rules must satisfy the additional conditions in Appendix E.3. These constraints reduce the risk that a local correction harms other eligible cases, although **validation on the same training corpus may still be optimistically biased**. **(b) The full training corpus provides greater sample support for population-level rule-utility estimation** The MIMIC-III training and validation splits contain 4,197 and 1,081 patients, respectively. Holding fixed the first experiential rule proposals generated during training, we obtain the following eligibility and support coverage: | **Rule eligibility / support** | **Full train** | **Validation** | **Size-matched train** | | :-------------------------------------------- | -------------: | -------------: | ---------------------: | | $\ge 1$ eligible case | 86.45% | 42.74% | 45.92 ± 0.40% | | $\ge 2$ eligible cases | 38.73% | 27.28% | 27.79 ± 0.18% | | $n_s \ge 2$ and $\operatorname{conf} \ge 0.5$ | 15.53% | 10.74% | 10.81 ± 0.11% | The full training corpus achieves greater coverage of eligible cases. To distinguish sample size from data composition, we randomly sampled 1,081 patients from the training set, repeating this procedure 100 times. For the criteria of at least two eligible cases and simultaneous satisfaction of the support/confidence thresholds, the size-matched training subsets (27.79% / 10.81%) closely match the independent validation set (27.28% / 10.74%). **The coverage advantage is primarily associated with the larger training sample size**: this not only provides sufficient eligible cases for more long-tail rules but also supplies a larger population-level basis for estimating rule support, confidence, and average cached utility. **(c) Out-of-Sample Rule Validation Strategy** On the same MIMIC-III held-out test set, we compare No-validation (Table 2), the original In-sample validation, and Out-of-sample validation, which screens rules on a validation set disjoint from their source patients. The rule-proposal mechanism and screening criteria remain unchanged. | **Validation strategy** | **Jaccard ↑** | **F1 ↑** | **DDI ↓** | | :---------------------- | ------------: | ---------: | --------: | | No-validation | 44.49% | 60.14% | 7.88% | | Out-of-sample | 46.19% | 61.93% | **7.10%** | | In-sample (Original) | **48.61%** | **64.18%** | 7.18% | **Out-of-sample validation improves Jaccard/F1 over No-validation by 1.70/1.79 percentage points (pp)**, showing that the system-level benefit of rule screening does not depend entirely on in-sample validation. It remains 2.42/2.25 pp below In-sample validation, potentially in part because the smaller validation set offers less coverage for long-tail rules. As complementary evidence, on validation patients excluded from Memory Evolution, we retain the full test-time configuration and disable only the evolved Experiential Rules: Jaccard is 48.99% vs. 46.95%, and F1 is 64.48% vs. 62.72%, with and without these rules, respectively. This provides additional evidence of the incremental value of retained rules.
AuthorsOpenReview ↗

Response to Reviewer#3y2i PART-3

Q2.2 How closely does the Cached Counterfactual agree with full Agent replay? (W1.2)

The Cached Counterfactual is a low-cost, population-level proxy for rule screening, not an accurate predictor of the gain from a full Agent replay for an individual patient. In Eq. (5) and Appendix E, we apply cached set-level transformations to all eligible training cases, calculate mean ΔJaccard\Delta\mathrm{Jaccard} / ΔF1\Delta F_1, and combine these estimates with support, confidence, and provenance to screen reusable correction rules. It is not an exact prediction of the counterfactual gain from the full Agent pipeline for an individual test patient. The computation reuses existing predictions without additional LLM calls, whereas replaying the entire pipeline for every rule and eligible case would be computationally expensive.

To quantify the limits of this approximation, we conducted the Cached-vs-Replayed Counterfactual Study:

  • We sampled 100 distinct MIMIC-III test patients, covering accepted rules/rejected proposals and single-rule/two-rule interventions, yielding 200 patient–rule interventions. Test labels were used only for offline scoring.
  • We calculated cached set-level transformation gains from the same baseline predictions. After applying the rules, we reran downstream Unit Reasoning, Calibration, Reconciliation, Audit, and Correction. Clinical Decomposition was held fixed because the tested rules do not enter that stage.
  • We executed 310 downstream reasoning chains and additionally repeated the unchanged baselines for 10 cases to assess generation variability.
MetricΔJaccard\Delta\mathrm{Jaccard}ΔF1\Delta F_1
Direction agreement48.0%48.0%
MAE0.019650.01801
Spearman ρ\rho0.1220.119

Case-level fidelity of the cached estimate is limited. Of the 200 interventions, 164 have zero cached gain; direction agreement is only 48.0%, and rank correlations are weak. Only 3 of the 10 repeated unchanged baselines produced exactly the same medication set, indicating generation variability.

We do not interpret these results as confirmation that full replay validates the cached approximation. Fidelity to per-patient replay and system-level utility of population-level rule screening are distinct questions: this study quantifies the limits of the local approximation, whereas the validation-strategy ablation in Q2.1 provides system-level evidence for the memory-screening mechanism. The Cached Counterfactual should therefore be regarded as a computationally efficient population-level screening proxy, not a precise substitute for the full multistage reasoning gain.

Q2.3 Is memory evolution stable across independent runs? (W1.3)

The five repeated tests in Appendix C.5 hold the final Memory State fixed and therefore measure only test-time generation variability. To assess the reproducibility of evolution itself, we added five independent Memory Evolution Runs: each evolves memory independently under the same data split and configuration and is evaluated on the same MIMIC-III test patients.

MetricOriginalFive independent evolutions (mean ± std)
Jaccard ↑48.61%48.77 ± 0.51%
F1 ↑64.18%64.35 ± 0.52%
DDI ↓7.18%7.10 ± 0.17%

Across the five independent evolutions, the standard deviations of Jaccard, F1, and DDI are 0.5124, 0.5269, and 0.1768 pp, respectively. Overall performance varies only modestly when memory is independently re-evolved under the current experimental configuration. Together with fixed-memory repeated inference, these results provide evidence of stability at two distinct levels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.