- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EvoMedAgent supports continual medical QA with a frozen generator through selective memory admission and conditional reuse. It combines clinical scope checks, source-calibrated support, and conflict isolation. Experiments across six benchmarks and patient-disjoint transfer show improved accuracy and AULC over strong baselines, with ablations supporting the design.
Critical strengths
-
The paper addresses the important problem of reusing supported medical claims that may be inappropriate for another patient. The method preserves patient identity and clinical conditions during memory writing and checks them before retrieval.
-
The paper provides comprehensive experimental validation of the proposed method.
Critical weaknesses
-
The methodological novelty appears incremental. The approach mainly combines established memory-writing and graph-retrieval mechanisms with clinical scope checks, source calibration, and conflict isolation. The paper could better establish what distinguishes this combination from safeguards that existing memory systems could adopt.
-
Comparisons with recent memory methods are incomplete. ACE and AMG-RAG receive only qualitative comparisons, while A-Mem and ReasoningBank are not evaluated experimentally. The paper therefore provides limited evidence that its assertion-based memory offers advantages over methods that accumulate reusable reasoning strategies.
-
The gains over closely matched controls are modest. Compared with generic rechecking under matched budgets, the full method improves AULC by approximately 1.06 percentage points. The final-accuracy gain is 0.62 points, with a confidence interval that includes zero.
-
Source-support calibration uses another LLM’s judgments, which do not establish clinical correctness. The memory audit also relies partly on model judgments and limited human correction. Stronger independent clinical validation would help establish the reliability of accepted assertions.
-
The evidence for generalization remains limited. The six-benchmark comparison uses frozen memory, while online adaptation is evaluated on two datasets with one generator backbone. Patient-disjoint transfer remains within EHRNoteQA, leaving performance across institutions and clinical settings unclear.
Questions for authors
-
Could you compare EvoMedAgent with ACE and ReasoningBank under matched feedback and computation budgets, and add the same clinical checks to a baseline to isolate their contribution?
-
Could independent clinicians assess accepted assertions for factual correctness, preservation of clinical conditions, and inappropriate reuse across patients?
-
Do the online gains persist with additional generator backbones and under shifts in clinical tasks or institutions?
3 comments
Remaining Clarifications
Thank you for the detailed responses and additional experiments. The independent clinical audit and expanded baseline comparisons substantially address several of my concerns. However, I retain some reservations about the incremental contribution and the robustness of the modest gains over generic rechecking. Could you report confidence intervals for the new matched comparisons and clarify whether the EHRNoteQA transfer results use a common evaluation set across methods? A generic-recheck control on CliCR would also help isolate the contribution of clinical conditioning.
Thank you for the follow-up.
Matched comparisons. In Table 3B, we held the verification budget fixed on MedQA. Full ADMIT gained 1.0637 AULC points over Generic (95% CI [0.6060, 1.5480]) and 0.62 Final points (95% CI [−0.02, 1.26]). In the Table 12 diagnostic, the clinical check accepted 4/100 assertions with altered conditions, compared with 20/100 for Generic. For intact assertions, acceptance was 98/100 with the clinical check and 95/100 with Generic.
The paired differences and their 95% CIs are in percentage points.
| Comparison | ΔAULC [95% CI] | ΔFinal [95% CI] |
|---|---|---|
| Full ADMIT − ACE-adapted Clinical | +0.77 [+0.14, +1.43] | +1.12 [+0.12, +2.12] |
| Full ADMIT − ReasoningBank-supervised-adapted Clinical | +0.85 [+0.31, +1.42] | +1.36 [+0.20, +2.52] |
| ACE-adapted (Clinical − Generic) | +0.48 [+0.12, +0.82] | +0.30 [−0.14, +0.76] |
| ReasoningBank-supervised-adapted (Clinical − Generic) | +0.16 [−0.10, +0.45] | −0.02 [−0.72, +0.68] |
| MedQA with Ministral (Full ADMIT − Generic) | +0.36 [+0.01, +0.72] | +0.50 [+0.06, +0.94] |
| CliCR (Full ADMIT − Generic) | +0.57 [+0.14, +1.00] | +0.44 [+0.04, +0.84] |
EHRNoteQA evaluation set. The main transfer results use each method's own evaluable set (Section 4.4). All methods started with the same 500 held-out patients. Full ADMIT has 492 evaluable patients and No-update has 491, with 489 in common. Three patients are evaluable only under Full ADMIT, two only under No-update, and six under neither method.
| Scoring set | No-update (%) | Full ADMIT (%) | ΔAccuracy [95% CI] |
|---|---|---|---|
| Method-specific evaluable | 71.00 | 74.50 | +3.50 [2.8, 4.2] |
| Common evaluable | 70.96 | 74.43 | +3.47 [2.7, 4.2] |
| All 500, exclusions scored incorrect | 69.72 | 73.31 | +3.59 [2.8, 4.2] |
CliCR. For Full ADMIT and Generic, we kept the candidate assertions, feedback, READ settings, and verification budget fixed. The only change was to replace the clinical scope check with a generic recheck. Across 25 paired trajectories, Full ADMIT gained 0.57 EM-AULC points over Generic. In the final windows, it answered 11 more of the 2,500 requests correctly, a gain of 0.44 Final EM points.
| Method | EM-AULC | Final EM (%) | Final correct / total |
|---|---|---|---|
| No-update | 62.48 | 62.84 | 1,571/2,500 |
| Write-all | 64.73 | 65.28 | 1,632/2,500 |
| Generic recheck | 67.28 | 67.92 | 1,698/2,500 |
| Full ADMIT | 67.85 | 68.36 | 1,709/2,500 |
Thank you for taking the time to read this follow-up. Your questions helped us explain the matched comparisons and the transfer evaluation more clearly. We hope the additional results address your remaining concerns, and we would appreciate your considering them in your final assessment.
Beyond auditing accepted assertions and their reuse, we tested whether the online gains extend across answer generators and clinical task types.
Online gains persist with a second answer generator and on clinical reading comprehension.
The original online results cover MedQA and MedMCQA (Table 2B, p. 6); the nine-backbone comparison is frozen evaluation (Section 4.6/Figure 6, p. 9). To test online generator dependence, we replace only the answer generator with Ministral-3-8B-Instruct-2512-BF16, retaining Qwen extraction, MedGemma verification, and calibration. All arms use Always-Deep, 25 paired 1,000-question trajectories, and Final over the last 200:
Table R5. Online MedQA results with Ministral.
For CliCR, development/calibration use 2,000 official training cases with source-family separation, frozen before testing. Evaluation uses 500 distinct official test cases, one question each, in 25 paired orders, with memory resets and near-duplicate leakage excluded. Always-Deep commits/scores before feedback. Official normalized EM uses released references; missing/invalid outputs score zero. EM-AULC averages trailing windows of up to 100 items; Final uses the last 100.
Table R6. Online CliCR results.
ADMIT leads Write-all by 3.12 EM-AULC/3.08 Final points under the independently calibrated task protocol.
The original patient-disjoint EHRNoteQA transfer, with downstream writing disabled, reports +3.5 points [2.8, 4.2] over No-update on method-specific evaluable sets (Section 4.4/Figure 4, p. 8; Appendix N, pp. 43–44).
We are grateful for your careful engagement with our work and for the questions that have helped us present its contribution more clearly. We hope the responses above address your concerns and clarify the evidence supporting our conclusions. If so, we respectfully invite you to reconsider your assessment. We would be glad to discuss any remaining questions, and we sincerely appreciate your time and consideration.