- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoMedAgent, a multi-agent medication recommendation framework that combines auditable reasoning with failure-driven memory evolution. Experiments on MIMIC-III and MIMIC-IV demonstrate relative Jaccard improvements of 36.20% and 33.24% over other baselines. Ablation studies validate the contributions of both components, while evaluations by two physicians and an LLM judge on 50 cases demonstrate improved reasoning auditability. However, the contributions of failure attribution and memory evolution are not sufficiently isolated from existing agentic methods.
Critical strengths
- Identify two important challenges in medication recommendation: the lack of decision auditability and the limited ability to learn from observed failures, both of which are particularly relevant in clinical settings.
- Propose EvoMedAgent framework to employ a structured multi-agent architecture with specialized stages and clearly defined responsibilities, enabling systematic medication reasoning and decision review.
- Extensive evaluation results demonstrate strong empirical performance against other baselines on MIMIC-III and MIMIC-IV, supported by ablation studies that highlight the contributions of key architectural components.
Critical weaknesses
- Learning from failures and evolving reusable memory have been explored in existing agent frameworks, such as ReasoningBank. The novelty of typed failure attribution and memory evolution should be better articulated.
- Medication decisions are often patient-specific and subject to clinical judgment. Accumulating failure experiences may introduce clinical biases, conflicting knowledge, or irrelevant information, potentially affecting future recommendations.
- Figure 1(a) and Line 088 emphasize clinician auditability, but Section 3.1 mainly describes automated interactions among specialized LLM agents. The authors should clarify the distinction between automated reasoning trace generation and actual clinician auditability, and carefully revise this contribution.
- Figure 2 shows unusually close agreement between physician and LLM evaluations, raising concerns about the evaluation protocol. In particular, the review and revision accountability criterion in Table 21 involves reconstructing complex decision processes, where large differences between physicians and LLMs might reasonably be expected. Such close alignment raises questions about whether the evaluation criteria are sufficiently rigorous and discriminative to meaningfully assess reasoning quality.
- MIMIC-III and MIMIC-IV originate from the same institution, this makes assessment of the generalizability of the proposed framework difficult.
Questions for authors
- The main novelty appears to lie in integrating existing agentic mechanisms with typed failure attribution and scoped memory updates. However, the distinction from methods such as SafeRx- Agent and SkeMex remains unclear. Table 19 provides only one relatively simple case, where the improvements might also be achievable through prompt refinement. Could the authors provide more detailed comparative case studies?
- The framework identifies recommendation failures by comparing predicted medications with historical prescriptions and validates memory updates using the same training cases that motivated these corrections. However, historical prescriptions may not always represent optimal clinical decisions. Could the authors clarify how they distinguish recommendation errors from differences in prescribing practices.
12 comments
Response to Reviewer#25To PART-2
[Q1.1 continued from Part 1.]
(c) Out-of-Sample Rule Validation Strategy. Holding the rule-proposal mechanism and screening criteria fixed, we replace full-training-corpus screening with a validation set disjoint from the proposal-source patients, and compare it with a configuration without corpus-level validation on the same MIMIC-III test set:
| Validation Strategy | Jaccard ↑ (%) | F1 ↑ (%) | DDI ↓ (%) |
|---|---|---|---|
| No-validation | 44.49 | 60.14 | 7.88 |
| Out-of-sample | 46.19 | 61.93 | 7.10 |
| In-sample (Original) | 48.61 | 64.18 | 7.18 |
Out-of-sample screening improves Jaccard/F1 over No-validation by 1.70/1.79 pp, but remains below In-sample screening, which has broader eligible-case coverage.
Additionally, on validation patients excluded from Memory Evolution, we fix the full test-time configuration and disable only the evolved Experiential Rules: Jaccard is 48.9948% vs. 46.9492%, and F1 is 64.4757% vs. 62.7182%, with and without those rules, respectively.
Together, these experiments clarify the limits of attributing performance gains to tools, candidate priors, and memory updates.
Q1.2 How does EvoMedAgent differ from ReasoningBank, SafeRx-Agent, and SkeMex, and what ablation evidence supports the distinction?
ReasoningBank already extracts reusable experience from reasoning trajectories; SafeRx-Agent employs specialty collaboration and safety verification; and SkeMex includes a skill repository and utility-based skill governance. We do not characterize these established capabilities as absent from the prior methods. EvoMedAgent's key distinction is to localize medication recommendation discrepancies to specific decision stages and combine Typed Failure Attribution, Layer-Specific/Scoped Memory Update, and Cross-Case Validation into a clinically scoped failure-driven memory evolution mechanism.
The Table 2 ablations directly support the performance contributions of these core design choices. The Jaccard scores for Full, Static Memory, w/o Typed Failure Attribution, Unified Memory, and w/o Corpus-Level Validation are 48.61%, 43.73%, 45.78%, 45.16%, and 44.49%, respectively. These results indicate that the gains are associated not only with experiential memory itself but also with typed failure attribution, targeted scope management, and cross-case rule validation, supporting the importance of these mechanisms within the full system.
In addition, Test-Time Auxiliary Tool Isolation in Q1.1(a) shows that, with evolved memory and the complete reasoning pipeline fixed, the system still achieves 44.29% Jaccard / 59.97% F1 when all three auxiliary tools are disabled. This further shows that the full system's performance advantage cannot be explained solely by learned auxiliary tools. Together with the mechanism ablations, these findings provide complementary experimental evidence for EvoMedAgent's structured, failure-driven memory evolution design.
Q1.3 Additional detailed cross-patient case study
The original Table 19 primarily illustrates reasoning coordination for a complex patient and does not sufficiently demonstrate Memory Evolution. We therefore provide actual cross-patient execution traces linking Failure Attribution → Scoped Update → Cross-Case Validation → Memory Reuse, and compare the reasoning behavior of three Agents on the same test case.
(a) Patient A: From a specific medication omission to a targeted memory update.
Training case 3362 underwent single-vessel coronary artery bypass grafting (CABG). Metoclopramide was already present in the procedure's Candidate Memory but was not selected during Unit Reasoning and did not appear in the final recommendation set. The system therefore attributed this omission to Selection Failure rather than Candidate Failure.
Together with a similar omission in training case 3207, the system proposed a prefer Metoclopramide rule scoped to single-vessel CABG units with the role Primary or Comorbidity-coupled. The rule acts only on an existing candidate; it is not an unconditional medication recommendation.
| Memory Evolution stage | Observed result |
|---|---|
| Failure Attribution | Existing candidate omitted during Unit Reasoning → Selection Failure |
| Scoped Update | Single-vessel CABG + specified unit role → prefer Metoclopramide |
| Cross-Case Validation | 43 eligible / 43 supporting, confidence = 1.0 |
| Counterfactual Utility | = +0.003751, = +0.003472 |
| Memory Update | Passed validation; written to Experiential Memory and kept Active |
[Q1.3 continues in Part 3.]
Response to Reviewer#25To PART-3
[Q1.3 continued from Part 2.]
After excluding the two proposal-source patients, all 41 remaining eligible cases still support the rule. Thus, the observed experience pattern is not supported solely by its source patients. The reported / represent cached set-level counterfactual estimates on training cases, not performance improvements measured on the test set.
(b) Patient B: Matching and reusing the validated rule in a different clinical context.
Independent test case 5404 primarily involved mitral and aortic valve surgery, together with single-vessel CABG and pulmonary comorbidities. Unlike Patient A, the CABG unit in this case had the Comorbidity-coupled rather than Primary role.
The case satisfied the rule's scoped applicability conditions. The coronary unit retrieved the rule; Metoclopramide appeared in the matched experiential rules, was selected during Unit Reasoning, and remained in the final medication set. This provides traceable evidence of training-case failure → scoped correction → conditional matching in a new patient → actual reuse.
(c) Comparison of actual reasoning workflows for the same patient.
To further clarify the design differences, we examined the recorded execution traces of all three methods on Patient B:
| Method | Actual module sequence and key observations | Final result |
|---|---|---|
| SafeRx-Agent | Route activates 5 experts → Summarize / Generate proposes 20 drugs, excluding Metoclopramide → Critique retains 19 drugs → FindFlags / Verify checks interactions and removes some label-positive medications | 13 drugs; 8 TP / 5 FP / 14 FN; Jaccard 29.63% |
| SkeMex | Read retrieves no applicable skill (retrieved_skill_ids=[]) → drug_info / ddi_check queries candidates and safety evidence without involving Metoclopramide → final outputs 14 drugs | 14 drugs; 12 TP / 2 FP / 10 FN; Jaccard 50.00% |
| EvoMedAgent | Clinical Decomposition identifies valve, CABG, and pulmonary clinical units → Unit Reasoning retrieves and invokes the validated CABG experiential rule, selecting Metoclopramide → Evidence Reconciliation integrates cross-unit evidence, removes duplicates, and addresses local omissions → Trace Review / Faithful Summarization records unresolved risks and preserves consistency of the final set | 26 drugs; 22 TP / 4 FP / 0 FN; Jaccard 84.62% |
In this case, neither SafeRx-Agent's expert proposals nor SkeMex's tool interactions involved Metoclopramide, whereas EvoMedAgent retrieved and used the validated rule in the CABG unit. EvoMedAgent matched all 22 historically recorded medication labels, including propofol, glycopyrrolate, and neostigmine—anesthetic/reversal agents omitted by both baselines—as well as other perioperative supportive medications. This illustrates how Clinical Decomposition, Unit Reasoning, and Patient-Level Reconciliation jointly organize medication decisions for a complex, multimorbid patient.
Importantly, neither the final selection of Metoclopramide nor the case-level Jaccard of 84.62% can be attributed solely to this rule: memory from other clinical units and auxiliary tools also contributed. The case demonstrates the traceable process of rule generation, validation, retrieval, and cross-patient reuse, alongside differences in the actual decision behavior of the frameworks.
In summary, EvoMedAgent differs from existing Agent frameworks not merely by storing past failures but by identifying the decision stage responsible, turning that failure into a clinically scoped correction, validating it across cases, and reusing it in subsequent reasoning. Together with clinical decomposition and patient-level evidence reconciliation, this mechanism constitutes the central design distinction of our work.
Response to Reviewer#25To PART-1
We thank the reviewer for recognizing the research problem, structured multi-Agent reasoning, and component ablations, and for the constructive suggestions regarding methodological novelty, the reliability of supervision from historical prescriptions, clinical auditability, and cross-institutional generalization. We address Q1, Q2, and the remaining Weaknesses in turn, distinguishing conclusions supported by the current evidence from clinical claims that remain unverified.
Q1. Methodological Novelty and Distinctions from Existing Agent Frameworks (W1)
EvoMedAgent's novelty lies in attributing fine-grained medication recommendation failures to specific decision stages and correcting them through clinically scoped memory updates—not in being the first to introduce multiple Agents or long-term memory.
Fine-grained medication recommendation must accommodate multiple disease-specific treatment goals, candidate-medication constraints, inter-condition treatment conflicts, and patient-specific safety requirements. The same final medication-set discrepancy may arise at different stages and require different corrective actions. For example, an omitted drug may never enter the entity-owned candidate set (Candidate Failure), or it may be present as a candidate but excluded during Unit Reasoning or patient-level reconciliation (Selection Failure). Although the output discrepancy is identical, the appropriate points of correction differ. EvoMedAgent therefore forms the closed loop Typed Failure Attribution → Layer-Specific/Scoped Update → Cross-Case Validation → Subsequent Reuse, rather than reducing all failures to clinically unscoped reflections or general summaries of past trajectories.
In Table 2, removing Typed Failure Attribution, Scoped Routing (Unified Memory), or Corpus-Level Validation reduces Jaccard from 48.61% to 45.78%, 45.16%, and 44.49%, respectively. To distinguish the respective contributions of the architecture, auxiliary tools, candidate priors, and validation strategy, we conducted three controlled experiments and a rule ablation on patients excluded from Memory Evolution.
(a) Test-Time Auxiliary Tool Isolation. With the final evolved Memory, Candidate Boundary, and full Agent reasoning pipeline fixed, we disable the ARMR Retriever, Calibration Tool, and Residual Tool individually or all three together:
With all three auxiliary tools disabled, the system still achieves 44.29% Jaccard / 59.97% F1. Thus, the tools contribute to performance but do not fully explain the system's predictive ability.
(b) Shared Candidate Boundary Comparison. We preserve the native reasoning pipelines and tools of SkeMex, RareAgents, and PRTA, while imposing EvoMedAgent's training-derived case-level Candidate Boundary on each baseline. The following results show changes relative to each native baseline configuration:
None of the three baselines improves Jaccard or F1, indicating that the effectiveness of candidate constraints depends on their compatibility with each method's native reasoning pipeline.
[Q1.1 continues in Part 2.]