acceptodds
Under review as a conference paper at ICLR 2027

WHEN SHOULD RAG WRITE? SELECTIVE MEMORY FOR CONTINUAL MEDICAL QA

Abstract

Medical question answering systems can learn from answer feedback, but a source-supported claim may still be inapplicable to a later patient. We introduce EvoMedAgent, a selective admission policy for graph memory with a frozen generator. The policy couples three mechanisms: executable clinical scope that must pass before retrieval ranking, source-family calibration of assertion support, and joint admission checks for source conditions and unresolved conflicts. The policy requires patient-bound observations to remain patient-specific and general assertions to retain reuse conditions. Within a test-then-learn protocol, accepted assertions become visible to later questions. Evaluation spans six frozen medical QA benchmarks, paired online streams, and transfer to disjoint EHRNoteQA patients after memory is frozen. Relative to no update, the full policy improves MedQA AULC by 5.3 points and EHRNoteQA transfer accuracy by 3.5 points. Matched-editor comparisons and component ablations establish the contribution of admission checks, while feedback-schedule controls examine evidence availability. Together, these results show that selective admission improves later predictions with a frozen generator and explicit checks on the reuse conditions of accepted evidence.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Retrieval-augmented generation, Non-parametric memory, Selective retrieval, Post-answer feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_veEAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoMedAgent supports continual medical QA with a frozen generator through selective memory admission and conditional reuse. It combines clinical scope checks, source-calibrated support, and conflict isolation. Experiments across six benchmarks and patient-disjoint transfer show improved accuracy and AULC over strong baselines, with ablations supporting the design.

Critical strengths
  1. The paper addresses the important problem of reusing supported medical claims that may be inappropriate for another patient. The method preserves patient identity and clinical conditions during memory writing and checks them before retrieval.

  2. The paper provides comprehensive experimental validation of the proposed method.

Critical weaknesses
  1. The methodological novelty appears incremental. The approach mainly combines established memory-writing and graph-retrieval mechanisms with clinical scope checks, source calibration, and conflict isolation. The paper could better establish what distinguishes this combination from safeguards that existing memory systems could adopt.

  2. Comparisons with recent memory methods are incomplete. ACE and AMG-RAG receive only qualitative comparisons, while A-Mem and ReasoningBank are not evaluated experimentally. The paper therefore provides limited evidence that its assertion-based memory offers advantages over methods that accumulate reusable reasoning strategies.

  3. The gains over closely matched controls are modest. Compared with generic rechecking under matched budgets, the full method improves AULC by approximately 1.06 percentage points. The final-accuracy gain is 0.62 points, with a confidence interval that includes zero.

  4. Source-support calibration uses another LLM’s judgments, which do not establish clinical correctness. The memory audit also relies partly on model judgments and limited human correction. Stronger independent clinical validation would help establish the reliability of accepted assertions.

  5. The evidence for generalization remains limited. The six-benchmark comparison uses frozen memory, while online adaptation is evaluated on two datasets with one generator backbone. Patient-disjoint transfer remains within EHRNoteQA, leaving performance across institutions and clinical settings unclear.

Questions for authors
  1. Could you compare EvoMedAgent with ACE and ReasoningBank under matched feedback and computation budgets, and add the same clinical checks to a baseline to isolate their contribution?

  2. Could independent clinicians assess accepted assertions for factual correctness, preservation of clinical conditions, and inappropriate reuse across patients?

  3. Do the online gains persist with additional generator backbones and under shifts in clinical tasks or institutions?

3 comments

Beyond auditing accepted assertions and their reuse, we tested whether the online gains extend across answer generators and clinical task types.

W5 & Q3. Additional generators, clinical tasks, and transfer.

Online gains persist with a second answer generator and on clinical reading comprehension.

The original online results cover MedQA and MedMCQA (Table 2B, p. 6); the nine-backbone comparison is frozen evaluation (Section 4.6/Figure 6, p. 9). To test online generator dependence, we replace only the answer generator with Ministral-3-8B-Instruct-2512-BF16, retaining Qwen extraction, MedGemma verification, and calibration. All arms use Always-Deep, 25 paired 1,000-question trajectories, and Final over the last 200:

Table R5. Online MedQA results with Ministral.

PolicyAULCFinal (%)
No-update70.4870.90
Write-all71.3771.74
Generic recheck73.8674.36
ADMIT74.2274.86

For CliCR, development/calibration use 2,000 official training cases with source-family separation, frozen before testing. Evaluation uses 500 distinct official test cases, one question each, in 25 paired orders, with memory resets and near-duplicate leakage excluded. Always-Deep commits/scores before feedback. Official normalized EM uses released references; missing/invalid outputs score zero. EM-AULC averages trailing windows of up to 100 items; Final uses the last 100.

Table R6. Online CliCR results.

PolicyEM-AULCFinal EM (%)
No-update62.4862.84
Write-all64.7365.28
ADMIT67.8568.36

ADMIT leads Write-all by 3.12 EM-AULC/3.08 Final points under the independently calibrated task protocol.

The original patient-disjoint EHRNoteQA transfer, with downstream writing disabled, reports +3.5 points [2.8, 4.2] over No-update on method-specific evaluable sets (Section 4.4/Figure 4, p. 8; Appendix N, pp. 43–44).

We are grateful for your careful engagement with our work and for the questions that have helped us present its contribution more clearly. We hope the responses above address your concerns and clarify the evidence supporting our conclusions. If so, we respectfully invite you to reconsider your assessment. We would be glad to discuss any remaining questions, and we sincerely appreciate your time and consideration.

Reviewer_veEAOpenReview ↗

Remaining Clarifications

Thank you for the detailed responses and additional experiments. The independent clinical audit and expanded baseline comparisons substantially address several of my concerns. However, I retain some reservations about the incremental contribution and the robustness of the modest gains over generic rechecking. Could you report confidence intervals for the new matched comparisons and clarify whether the EHRNoteQA transfer results use a common evaluation set across methods? A generic-recheck control on CliCR would also help isolate the contribution of clinical conditioning.

Thank you for the follow-up.

Matched comparisons. In Table 3B, we held the verification budget fixed on MedQA. Full ADMIT gained 1.0637 AULC points over Generic (95% CI [0.6060, 1.5480]) and 0.62 Final points (95% CI [−0.02, 1.26]). In the Table 12 diagnostic, the clinical check accepted 4/100 assertions with altered conditions, compared with 20/100 for Generic. For intact assertions, acceptance was 98/100 with the clinical check and 95/100 with Generic.

The paired differences and their 95% CIs are in percentage points.

ComparisonΔAULC [95% CI]ΔFinal [95% CI]
Full ADMIT − ACE-adapted Clinical+0.77 [+0.14, +1.43]+1.12 [+0.12, +2.12]
Full ADMIT − ReasoningBank-supervised-adapted Clinical+0.85 [+0.31, +1.42]+1.36 [+0.20, +2.52]
ACE-adapted (Clinical − Generic)+0.48 [+0.12, +0.82]+0.30 [−0.14, +0.76]
ReasoningBank-supervised-adapted (Clinical − Generic)+0.16 [−0.10, +0.45]−0.02 [−0.72, +0.68]
MedQA with Ministral (Full ADMIT − Generic)+0.36 [+0.01, +0.72]+0.50 [+0.06, +0.94]
CliCR (Full ADMIT − Generic)+0.57 [+0.14, +1.00]+0.44 [+0.04, +0.84]

EHRNoteQA evaluation set. The main transfer results use each method's own evaluable set (Section 4.4). All methods started with the same 500 held-out patients. Full ADMIT has 492 evaluable patients and No-update has 491, with 489 in common. Three patients are evaluable only under Full ADMIT, two only under No-update, and six under neither method.

Scoring setNo-update (%)Full ADMIT (%)ΔAccuracy [95% CI]
Method-specific evaluable71.0074.50+3.50 [2.8, 4.2]
Common evaluable70.9674.43+3.47 [2.7, 4.2]
All 500, exclusions scored incorrect69.7273.31+3.59 [2.8, 4.2]

CliCR. For Full ADMIT and Generic, we kept the candidate assertions, feedback, READ settings, and verification budget fixed. The only change was to replace the clinical scope check with a generic recheck. Across 25 paired trajectories, Full ADMIT gained 0.57 EM-AULC points over Generic. In the final windows, it answered 11 more of the 2,500 requests correctly, a gain of 0.44 Final EM points.

MethodEM-AULCFinal EM (%)Final correct / total
No-update62.4862.841,571/2,500
Write-all64.7365.281,632/2,500
Generic recheck67.2867.921,698/2,500
Full ADMIT67.8568.361,709/2,500

Thank you for taking the time to read this follow-up. Your questions helped us explain the matched comparisons and the transfer evaluation more clearly. We hope the additional results address your remaining concerns, and we would appreciate your considering them in your final assessment.

Reviewer_Jar8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoMedAgent, a framework for continual medical question answering that improves subsequent predictions by updating an external knowledge graph while keeping the language model frozen. The framework consists of three components: ACCESS determines whether retrieval is necessary based on prediction uncertainty and medical concept density, READ retrieves relevant evidence under clinical applicability constraints, and ADMIT selectively incorporates medical assertions extracted from question–answer feedback through source-support calibration, clinical scope verification, and conflict checks. Experiments on six medical QA benchmarks demonstrate improvements over existing RAG and memory-update baselines, including gains in online adaptation and cross-patient transfer.

Critical strengths
  1. The paper addresses an important issue in continual medical QA: knowledge supported by a particular source or patient context may not be applicable to other patients. Preserving clinical applicability conditions during memory updates is a meaningful research direction.
  2. The experiments cover multiple medical QA benchmarks, online adaptation, component ablations, feedback availability and noise, and cross-patient transfer. The authors also provide comparisons with existing dynamic memory methods and budget-matched verification controls.
  3. The distinction between source-supported assertions and clinically applicable assertions is useful. The ablation results indicate that explicitly checking clinical applicability conditions can improve memory admission decisions and downstream QA performance.
Critical weaknesses
  1. The memory update process primarily relies on historical questions and their ground-truth answers, without directly incorporating the model's own predictions or retrieved evidence. Consequently, the approach closely resembles supervised incremental knowledge base construction, and its advantages over simpler offline or incremental knowledge construction methods remain unclear.
  2. Prediction uncertainty does not necessarily reflect the need for external knowledge, as models may confidently produce incorrect answers, causing questions that would benefit from retrieval to be routed to the retrieval-free path. Although the paper evaluates routing efficiency and latency, it provides limited analysis of such high-confidence routing errors and their impact on downstream performance.
  3. Medical assertions are extracted from historical QA pairs, which may lack complete clinical evidence or necessary applicability conditions. Since the subsequent verification process largely relies on the same QA feedback, it may establish consistency with the source without adequately ensuring clinical correctness or reliable reuse across patients.
Questions for authors
  1. What is the key methodological contribution of EvoMedAgent beyond existing dynamic memory and knowledge graph updating methods such as KEDKG, AMG-RAG, Dynamic Cheatsheet, and ACE?
  2. Since memory updates rely only on historical QA feedback, how does the proposed online formulation differ from incremental knowledge base construction? Could simpler approaches using the same feedback achieve comparable results?
  3. How can ADMIT ensure the clinical correctness and completeness of extracted assertions when the original QA feedback lacks necessary medical conditions?
  4. How does the framework handle clinical constraints beyond the predefined schema, and what is the total computational overhead of the memory-update process?
4 comments

To Reviewer Jar8:

Thank you for your careful review and constructive feedback. We appreciate your recognition of the importance of preserving clinical conditions, the distinction between source support and clinical applicability, and the breadth of our evaluation. We are grateful for the time you have devoted to examining both the method and its implications for medical QA.

W1 & Q1 & Q2. Methodological contribution and incremental knowledge construction.

EvoMedAgent couples feedback-driven memory construction with the clinical conditions governing reuse, and ADMIT outperforms simpler admission policies under the same feedback.

The motivation is that a claim can remain source-supported yet become unsuitable for another patient, clinical state, or time. KEDKG [1] emphasizes revised facts and graph retrieval; AMG-RAG [2] updates medical graphs; Dynamic Cheatsheet [3] and ACE [4] accumulate reusable strategies or playbooks. EvoMedAgent makes clinical applicability explicit in the memory unit and in both decisions governing its use: assertions store source-linked conditions, ADMIT checks condition preservation at writing, and READ checks patient and condition compatibility before semantic ranking. Source-support calibration and conflict isolation further govern which assertions become visible.

Memory construction is incremental; the online formulation evaluates its effect on subsequent predictions. Each answer is committed and scored before feedback is released, and an update becomes usable only afterward. Extraction uses the completed question and released reference—not the model's answer as truth.

The matched MedQA comparison uses 25 paired trajectories of 1,000 questions. All policies share Always-Deep, initial memory, generator, candidates, READ, and verification-budget ceilings, then evolve independent memories. AULC averages trailing-window accuracy (up to 100 questions); Final uses the last 200.

Admission policyAULCFinal (%)
No-update80.9281.46
Write-all82.883.3
Raw-confidence gate85.085.6
ConsistencyGate-adapted85.1185.76
A-MAC-adapted85.2485.90
ADMIT85.6886.38

ADMIT leads Write-all by 2.9 AULC/3.1 Final points and the tested A-MAC adaptation by 0.44 AULC/0.48 Final points. ConsistencyGate averages five MedGemma support scores (temperature 0.7, top-p 0.95; threshold 0.70); A-MAC uses one deterministic utility judgment and a weighted five-factor score (threshold 0.60). These adaptations retain common structural, authorization, patient, time, conflict, and READ checks, replacing ADMIT's support threshold and separate scope veto. Raw-confidence retains the scope check. The ceiling is one common score plus five extra calls/candidate, eight candidates/feedback.

The condition-check comparison under selective ACCESS is in Table 3B (p. 7), with diagnostics in Table 12 (p. 27). Within adapted ACE and ReasoningBank, Clinical replaces Generic's additional check with an independent scope check for necessary conditions, negation, uncertainty, temporal qualifiers, and improper patient generalization; it adds neither ADMIT source calibration nor conflict isolation. Base/Generic/Clinical share scope, patient-ID, and source-version metadata and the same pre-prompt patient/condition matching. Clinical–Generic differences are +0.48/+0.16 AULC, testing this write-side check inside other writers.

These matched comparisons address the admission policy. Part 2 turns to ACCESS and high-confidence routing errors.

W2. High-confidence routing errors and retrieval allocation.

Entropy measures answer uncertainty, not the need for external medical knowledge. ACCESS therefore combines two complementary signals: model uncertainty and medical concept density.

Prediction uncertainty alone is not a sufficient proxy for retrieval need. Entropy captures how dispersed the model’s answer distribution is, but it does not indicate whether the most likely answer is correct or whether the question requires external medical knowledge. A model can be highly confident in an incorrect answer, producing low entropy and taking the retrieval-free path.

ACCESS incorporates linked-concept density as an independent input-side signal. D(q) measures the density of medical concept mentions in the question. Entropy and density are normalized using within-length-bucket percentile ranks before combination (Section 3.2; Appendix G, pp. 32–33).

The two signals capture different aspects of retrieval need: H(q) reflects uncertainty about the answer, whereas D(q) indicates how strongly the question is grounded in identifiable medical concepts that can benefit from external knowledge retrieval. A question with high medical concept density can therefore receive a sufficiently high routing score despite relatively low entropy. Density is an input-side signal and is not used as a correctness predictor.

The paired diagnostic covers 1,273 MedQA questions and five generation seeds (6,365 requests). Fast and Deep share memory snapshots and paired seeds; both routers reuse these paired potential answers.

Combining the signals improves retrieval allocation. Among 1,152 Fast-wrong/Deep-correct requests, entropy-only misses 529 (45.92%), whereas Full H+D misses 194 (16.84%). This reduction occurs while Deep usage decreases from 51.0% to 46.7%. Across the full evaluation set, H+D achieves 82.1% accuracy, compared with 74.7% for entropy-only, 75.0% for density-only, and 80.5% for Always-Deep (Table 4, p. 7). Neither signal alone provides the same accuracy–computation trade-off as their combination.

The high-confidence subgroup shows the limitation of entropy-only routing. Among 1,880 requests with probe confidence ≥0.90, 72 errors can be corrected by Deep retrieval. Entropy-only misses all 72; H+D retrieves and corrects 25. H+D also introduces 31 errors in this subgroup, giving 91.38% accuracy versus 91.70% for Always-Fast. H+D’s overall advantage comes from more effective retrieval allocation across the overall request distribution.

Answer-side tokens include the uncertainty probe: H+D uses 2,119.7 tokens/request versus Always-Deep’s 832.5; extra diagnostic executions are counted separately.

The next part examines the evidence available to ADMIT and the clinical quality of the assertions it accepts.

W3 & Q3. Source completeness, condition preservation, and clinical reliability.

Conditions omitted from a short answer can still be checked against the complete source when retained in the question stem; conditions absent from all available sources require additional evidence for assessment.

Verification reads the full completed question stem plus released answer, so a condition omitted from a short answer remains checkable when present in the stem (Appendix D, pp. 19–20). ADMIT assesses whether the proposed assertion retains source-required restrictions. READ then checks the retained restrictions against the current request, rejecting unresolved constrained attributes before ranking (Appendix H.1, pp. 34–35). This separates condition preservation during writing from condition matching during reuse; the following experiments measure how accurately those decisions are made.

The original 100-pair diagnostic reduces erroneous acceptance of condition-modified assertions from 20/100 to 4/100—80%—while retaining 98/100 intact assertions versus Generic's 95/100 (Table 12B–D, p. 27). These are isolated check-call decisions. The complete-source experiment below uses 200 sources, shared candidates, and different additional checks:

PolicyAdmittedOvergeneralizedValid referencesInapplicable exposure (%)
No-scope-veto176151615.50
Generic recheck16251572.00
ADMIT16111600.50

Exposure uses 200 inapplicable queries and actual final prompts. ADMIT preserves more valid references than Generic while reducing overgeneralization and inappropriate exposure.

The separate deletion test removes a necessary condition from every runtime input and cached reference, making it non-inferable. All 179 admitted assertions then omit it, with exposure in 148/200 inapplicable queries (74%); reconstructing the deleted information requires independent evidence. The complete-source controls isolate condition preservation.

We take your request for independent clinical validation seriously. Recruiting clinicians for independent review and allowing sufficient time for two assessments per item and adjudication requires careful coordination; completing a large-scale evaluation within the limited rebuttal period is therefore difficult in practice. We prioritized the focused blinded assessment described below to provide concrete clinical evidence addressing your concern, and we commit to expanding the human audit sample for the camera-ready version.

The original EHRNoteQA audit reports 92/100 source-supported entries with intact/applicable conditions, five scope errors, and three unresolved; eight entries received human review with model labels visible (Appendix N.1, Table N.4, pp. 44–45).

Independent blinded clinicians additionally assessed the MedQA ADMIT run with 30% incorrect feedback labels and Always-Deep: separate uniform samples without replacement of 200 unique assertion–scope–source-version units and 200 actual exposure pairs. Two clinicians independently assigned Yes/No/Unresolved, blinded to method, score, and corruption flags; a third adjudicated disagreements.

Reviewers received each assertion, its complete scope, the unaltered original case, and clinical reference materials with recorded versions. Factual correctness was judged against medical evidence, with answer labels withheld during initial review. Condition completeness was checked against the full original source rather than the corrupted feedback; reuse applicability was judged using information available at the target request. Insufficient evidence received an Unresolved label.

Factual correctness was 188/200 (94.00%), condition completeness 179/200 (89.50%), and appropriate reuse 184/200 (92.00%). Unresolved counts were 3/200, 2/200, and 8/200, respectively; eight exposure pairs were inappropriate. All remain in their respective denominators, and the criteria are reported separately. Pre-adjudication agreement was 95.50%/93.50%/93.50%, with unweighted three-category Cohen's κ of 0.682/0.682/0.657.

Part 4 addresses constraints outside the schema and reports the complete online computation costs.

Q4. Out-of-schema constraints and full online cost.

Out-of-schema conditions are addressed at two stages: extraction is instructed to abstain, and ADMIT checks any emitted candidates for omitted or broadened conditions.

The extraction prompt requires an empty list when necessary conditions cannot be retained in the closed ten-field schema. For emitted candidates, structure checks and a separate scope judgment examine the representation; an unresolved or failed scope call blocks publication. Schema extensions require explicit revision and renewed quality checks. READ matches retained constraints before semantic ranking (Table 9, p. 17; Appendix D, pp. 19–20; Appendix H.1, pp. 34–35).

In the 200-source out-of-schema test, extraction abstains for 141 sources. Of 59 shared candidates, ADMIT rejects 56 (94.92%) and admits three, versus Generic's 16 and No-scope-veto's 47. All admissions in this test omit the unrepresentable condition. Extraction plus admission withhold updates for 197/200 sources (98.50%), with their contributions counted separately.

Source reacquisition/span-parsing failures are a different case: preserving cached quotations and other inputs allows 161 of 180 candidates to pass in each test. This follows the manuscript's continuation rule; scope-model call failures instead block publication.

The following token means cover all 25,000 requests with immediate correct feedback and shared selective ACCESS/READ:

MethodAnswer tokensWrite tokensTotal tokens
No-update2,106.802,106.8
Write-all2,123.33,729.75,852.9
Raw-confidence gate2,116.04,344.66,460.6
ADMIT2,113.14,344.66,457.7

Writing includes extraction, common scoring, and admission checks; repeated prefixes count repeatedly. Totals use unrounded counts, explaining Write-all's 0.1 rounding difference.

ADMIT's measured mean durations per request and total times over 25,000 requests are:

ConfigurationAnswer (s)Update (s)Complete step (s)Total online time (min)
Always-Deep1.845.927.763,233.3
Selective ACCESS1.335.927.253,020.8

Measurements use one A800 80GB, with Qwen3.5-9B and MedGemma-27B-Text-IT preloaded in BF16. Request concurrency is one: each answer is committed and scored, feedback is processed, and accepted updates become readable before the next request starts. Complete-step timing runs from before prompt construction until updates are visible to subsequent READ. All 25,000 requests and the final update are included; model loading, warm-up, and trajectory resets are excluded. The mean update duration is 5.92 seconds: 3.30 for extraction-output generation, 1.82 for scope-check output generation, and 0.80 for the remaining processing. Complete-step means and total times are calculated from request-level elapsed durations.


Thank you again for the thoughtful questions and the opportunity to explain our approach in greater detail. We hope the responses above clarify the methodological contribution, the role of clinical conditions, and the interpretation of the results. Should these clarifications address your concerns, we would appreciate your reconsideration of the rating. We welcome further discussion and are grateful for your time and careful consideration.

References

[1] Lu Y, et al. Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering. arXiv preprint arXiv:2412.13782, 2024.

[2] Rezaei M R, et al. Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge. Findings of EMNLP, 2025: 12682–12701.

[3] Suzgun M, et al. Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory. arXiv preprint arXiv:2504.07952, 2025.

[4] Zhang Q, et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. arXiv preprint arXiv:2510.04618, 2025.

Reviewer_csHjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoMedAgent, a selective memory-writing framework for medical QA with a frozen generator. Under a test-then-learn protocol, it extracts assertions from answer feedback and selectively admits them based on calibrated support, clinical scope, source validity, and conflict checks. An uncertainty-based router and scope-aware retriever govern memory access. Experiments cover six medical QA benchmarks, online adaptation, ablations, and patient-disjoint transfer.

Critical strengths
  1. Rigorous evaluation. The paper provides carefully controlled comparisons, paired statistical tests, extensive ablations, and robustness analyses. The experimental methodology is generally convincing.
Critical weaknesses
  1. Limited novelty and missing relevant baselines. The main contribution is essentially a memory admission policy rather than a RAG method. Similar ideas have been explored in recent works. These works are not adequately discussed or compared. Given the relatively small improvement over supervised Dynamic Cheatsheet (+1.24 AULC), stronger memory-writing baselines are necessary to establish novelty. The clinical scope mechanism is potentially interesting but is mainly evaluated in the EHR transfer setting.

[1] ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control

[2] MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance

[3] Adaptive Memory Admission Control for LLM Agents (A-MAC)

[4] Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory

  1. Insufficient evidence supporting the claimed mechanism. First, frozen QA improvements appear largely attributable to ACCESS routing rather than memory writing (Table 1). Second, online admission experiments use only one backbone, with limited hyperparameter sensitivity analysis. More importantly, just 10% feedback recovers most of the improvement, while incorrect or cross-domain feedback causes surprisingly little degradation (Figure 5, Appendix M.1). These observations cast doubt on whether selective admission of correct, relevant assertions is truly responsible for the reported gains.
Questions for authors
  1. How do the authors explain the strong performance with sparse, noisy, or cross-domain feedback? Can improved predictions be linked to specific admitted assertions?

  2. How does ADMIT compare with existing memory admission methods under matched conditions? Do the retrieval baselines in Table 4B use the same ACCESS router?

3 comments

W2 & Q1. Attribution to writing, sparse/noisy feedback, and generalization.

Sparse feedback limits opportunities to add knowledge on top of a shared initial memory; it does not reduce the system's existing knowledge to 10%. Matched admission controls and fixed-context interventions test the contribution of these incremental memories.

Freezing memory during testing preserves the effects of the policy that constructed it. In the original shared-historical-proposal controls, each write variant rebuilds memory before frozen QA: Full reaches 82.1%, versus 80.1% for Raw-confidence, 79.2% without source authorization, and 79.6% without conflict isolation (Table 5, p. 8; Appendix E.1, p. 28). These controls directly test how historical admission affects the memory later used for prediction.

All conditions use 25 paired trajectories of the same 1,000 questions and reuse frozen per-item routes: 471 Deep and 529 Fast questions, or 11,775 Deep requests out of 25,000 (47.10%). Answers are committed and scored before feedback; AULC averages trailing windows of up to 100 items and Final uses the last 200.

Table R2A. Feedback conditions under selective ACCESS.

FeedbackNew exposure (%)Corrupt-origin exposure (%)AULCFinal (%)
None / No-update0.000.0081.582.0
100% correct29.780.0086.887.4
10% sparse correct12.140.0085.586.1
10% incorrect labels28.972.9186.086.2
30% incorrect labels27.1611.5285.386.0
Domain-order shift24.980.0085.786.4

AULC/Final reproduce Figure 5 and Appendix M.1; exposure counts actual prompts once per request over all 25,000 requests, including Fast and incorrect answers. Corrupt-origin exposure means a new assertion has any corrupted supporting source at request time, not that it is necessarily factually wrong.

Sparse feedback. The experiment measures incremental learning on top of existing knowledge. All conditions share PrimeKG's static background and memory constructed before evaluation (Section 3.1, p. 3; Table 14, p. 29), with the generator frozen. The 10% mask selects which completed questions supply feedback for external-memory updates; an unselected question produces no new proposals, rather than removing existing knowledge (Appendix M.1, p. 43). Feedback availability therefore controls opportunities to acquire additional evidence, not the fraction of initial knowledge available to the system.

No-update already reaches 81.5 AULC/82.0 Final using the shared initial knowledge. The 10% feedback condition adds 4.0/4.1 points, compared with 5.3/5.4 points from complete feedback, retaining 75.5%/75.9% of the gains. Feedback rate counts questions that release labels, whereas new-assertion exposure counts later prompts using acquired knowledge. An admitted assertion can serve multiple subsequent, clinically compatible questions. Approximately 83 admission events per trajectory (Table 34, p. 43) accompany 12.14% new-assertion exposure, compared with 29.78% under complete feedback: 10% feedback therefore retains 40.8% of full-feedback exposure.

The benefit of an exposure depends on whether the evidence changes an otherwise incorrect answer. Additional exposures may supply overlapping information or leave an already correct answer unchanged. Prediction gains therefore need not scale linearly with either feedback volume or exposure frequency. Reusable knowledge can make sparse updates effective while additional feedback yields smaller marginal gains.

The matched masking experiment tests whether these new assertions actually affect predictions. On the 1,983 sparse-feedback requests in R4, masking relevant new assertions lowers accuracy by 2.15 points relative to matched random masking, with routing and the rest of the context fixed. Together, the exposure and masking results support continued acquisition and use of helpful evidence under sparse feedback, on top of the shared initial knowledge.

The sparse-feedback results connect incremental memory to later predictions. Part 3 examines noisy feedback and the effects of masking admitted assertions.

W2 & Q1 (continued). Noisy feedback and assertion-level attribution.

Noisy feedback. With 30% label corruption, approximately 70% of feedback remains correct; the generator is frozen and original answers score predictions. Table R2B compares exposure under R2A's shared selective routes, feedback and READ. Corrupt-origin exposure means any corrupted supporting source, including mixed-support assertions.

Table R2B. Exposure with 30% incorrect labels; selective ACCESS. Counts use all 25,000 requests.

PolicyNew-exposure requestsCorrupt-origin requestsCorrupt/new (%)
Write-all10,0375,30652.86
ADMIT6,7912,87942.39

All-request new/corrupt-origin exposure rates are 40.15%/21.22% for Write-all and 27.16%/11.52% for ADMIT. ADMIT reduces corrupt-origin exposure requests by 45.74%; their share among new-exposure requests falls by 10.47 percentage points (19.81% relatively). Thus the reduction includes both fewer new-exposure requests and a lower corrupt-origin share within them. A prompt can contain multiple assertions, so the corrupt-origin share among new-exposure requests can exceed the feedback corruption rate.

The separate Always-Deep control shares candidates, corruption masks, replacement answers, and READ:

Table R3A. Admission events with 30% incorrect labels; Always-Deep. Counts are means per trajectory; source shares use pooled event counts.

MetricWrite-allADMIT
All admission events/trajectory1,821.68771.28
Corrupt-origin admission events/trajectory541.44196.72
Uncorrupted-origin admission events/trajectory1,280.24574.56
Corrupt-origin share of admissions (%)29.7225.51

Table R3B. QA performance with 30% incorrect labels; Always-Deep.

PolicyAULCFinal (%)
Write-all80.9680.74
ADMIT83.9584.58

Relative to Write-all, ADMIT reduces total admission events by 57.66% and corrupt-origin events by 63.67%; their share falls from 29.72% to 25.51%. Source flags describe feedback provenance. AULC/Final improve by 2.99/3.84 points under Always-Deep, complementing the selective-ACCESS exposure comparison.

Domain changes. This condition orders the same questions into four 250-item clinical-domain blocks, with identically ordered controls; feedback still answers its originating question. It tests changing question domains, not unrelated-domain answer substitution. READ selects evidence for the current query and retained clinical conditions. Adding 50% feedback availability and a 240-position delay reduces AULC/Final from 85.7/86.4 to 84.5/85.2 (Figure 5), consistent with the importance of timely evidence.

Assertion-level attribution. In R2A's trajectories, frozen rules select relevant, clinically compatible new assertions already in the prompt without examining labels or outcomes. Random controls match non-target evidence on count, length, position, new-assertion status, source, and relation.

Table R4. MedQA matched masking; accuracy differences are percentage points.

FeedbackRequestsTarget masked (%)Random masked (%)Random − target
100% correct4,93783.5186.71+3.20
10% sparse correct1,98383.8085.95+2.15
30% incorrect labels3,96183.4585.28+1.83
Domain-order shift3,47883.2485.86+2.62

Coverage is 19.75%, 7.93%, 15.84%, and 13.91% of the 25,000 original requests. Branches share requests and five generation seeds; random masking uses ten mask seeds. Repetitions average within requests, then requests receive equal weight. No rerouting, retrieval, refill, or learning occurs. The sparse-feedback contrast directly links predictions to admitted content.

Separately, masking exclusively corrupted-support assertions in R3B's runs uses the common 1,973-request intersection (7.89%), selected by source lineage without observing outcomes. Each request has one generation per branch, using paired seeds and matched count/length/position controls. Write-all's full-context/target-masked/random-masked accuracies are 78.16%/81.15%/77.70%; ADMIT's are 83.68%/84.19%/83.43%. Target masking therefore improves on full context by 2.99/0.51 points, respectively, and on random masking by 3.45/0.76 points. On this matched subset, target removal improves prediction; random masking tests the specificity of the effect.

The admission and masking results address the mechanism behind the online gains. Part 4 turns to a second generator, a new clinical task, and threshold sensitivity.

W2 & Q1 (continued). Additional generator, clinical task, and threshold sensitivity.

Additional generator and task. Replacing only the answer generator with Ministral-3-8B-Instruct-2512-BF16 [5], while retaining Qwen extraction, MedGemma verification, and calibration, gives:

Table R5. Ministral; Always-Deep, 25 × 1,000 MedQA questions.

PolicyAULCFinal (%)
No-update70.4870.90
Write-all71.3771.74
Generic recheck73.8674.36
ADMIT74.2274.86

For CliCR [6], 2,000 official training cases support source-family-separated development/calibration, frozen before testing. Evaluation uses 500 distinct official test cases, one question each, and 25 paired orders, with memory reset and near-duplicate leakage excluded. Always-Deep uses commit–score–feedback ordering. Official normalized EM [7] uses released references; invalid outputs score zero. EM-AULC uses trailing windows of up to 100; Final uses the last 100.

Table R6. CliCR online results.

PolicyEM-AULCFinal EM (%)
No-update62.4862.84
Write-all64.7365.28
ADMIT67.8568.36

Threshold sensitivity. With calibration, candidates, other gates, and Always-Deep fixed, the 0.60/0.65/0.70/0.75/0.80 sweep yields AULC 85.04–85.72 and Final 85.86–86.38. The main-run threshold remains the pre-locked 0.70 (85.68/86.38, as reported in R1); each sensitivity arm evolves its own memory. This complements the ACCESS sensitivity analysis in Appendix G, Figure 9 (p. 32).

We sincerely appreciate the time and care you have devoted to reviewing our work. We hope these responses clarify the contribution and address your questions about the admission mechanism and its supporting evidence. If they resolve your concerns, we would be grateful if you would reconsider your assessment. We would be pleased to discuss any remaining questions. Thank you again for your thoughtful feedback.

References

[1] Zhang Y, Li S. ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control. arXiv preprint arXiv:2607.22962, 2026.

[2] GuilinDev. Adaptive Memory Admission Control for LLM Agents (A-MAC). Software repository.

[3] Wang H, et al. MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance. arXiv preprint arXiv:2608.21867, 2026.

[4] Sun H, et al. Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory. arXiv preprint arXiv:2606.09365, 2026.

[5] Mistral AI. Ministral-3-8B-Instruct-2512-BF16. Model checkpoint.

[6] Šuster S, Daelemans W. CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension. NAACL-HLT, 2018: 1551–1563.

[7] Šuster S, Daelemans W. CliCR official evaluation script. Software, dataset-code/evaluate.py, 2018.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.