acceptodds
Under review as a conference paper at ICLR 2027

EvoTD: Decoupled Memory Evolution for Specialty Triage and Diagnostic Inference

Abstract

Clinical practice requires clinicians to synthesize heterogeneous specialty-specific knowledge before making diagnostic decisions. Medical question answering has therefore become a widely employed task for such knowledge synthesis, owing to its controllable evaluation. However, most prior approaches either treat specialty triage as a one-time preprocessing step without leveraging it during diagnostic inference or update notes without enforcing consistency with the selected specialty-specific context. To this end, we propose EvoTD, a decoupled memory evolution framework for specialty triage and diagnostic inference. Specifically, DualEvo is first designed to maintain separate specialty-boundary patches and notes, preventing diagnostic-experience updates from altering specialty triage. Then, SkillRoute adaptively selects deterministic boundary rules, graph-based scoring, and LLM reasoning to construct specialty-specific context via case evidence. To improve the reliability of memory evolution, we develop ReliHarness to synthesize entity-indexed notes from predictions and harness their usage through warmup, single-pass pre-injection, and harmful-memory quarantine. Comprehensive experiments show that EvoTD improves over direct prompting by 7.1 and 6.1 points (3-seed means) on NEJMQA and MMLU-Pro Health, respectively.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Specialty triage, Diagnostic inference, Memory evolution, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iqjuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EVOTD is a train-free memory-evolution framework for medical QA that treats specialty triage and diagnostic inference as inference-coupled but evolution-decoupled. DUALEVO maintains separate specialty-boundary patches (TriageEvo) and entity-indexed diagnostic notes (DiagnosisEvo) under a one-way constraint (Eq. 4) that bars notes from the router; SKILLROUTE builds specialty-conditioned context at evidence-adaptive depth (deterministic rules → graph scoring → LLM reasoning); RELIHARNESS governs updates via warmup, single-pass pre-injection, and harmful-memory quarantine. With DeepSeek-v4-Flash, EVOTD improves over direct prompting by +7.1/+6.1 points (3-seed means) on NEJMQA/MMLU-Pro Health at ~1.6 serving-time LLM calls per item (vs ~13 for GSEM).

Critical strengths

1,The asymmetric blast radius of routing vs. note errors motivates channel-specific acceptance rules (zero-regression replay for patches vs. harm-based quarantine for notes). The argument is coherent and Eqs. 4/5/7 are precise. The residual coupling through the answer-correctness filter on BR_t is honestly acknowledged (§3.3). 2,Item-level bootstrap CIs and paired sign-flip/exact McNemar tests, explicitly avoiding the false precision of concatenating repeated runs; W2R/R2W flip counts (87/35, 58/13) directly address whether evolution overturns already-correct answers. 3,Removing warmup costs 2 points; replacing single-pass pre-injection with predict-then-correct collapses accuracy to the Pure LLM level (68.44 vs 68.03) and inverts flips to 11/18 — clean evidence that a stochastic corrective pass rewrites more right answers than it repairs, a useful cautionary result for Reflexion-style methods generally.

Critical weaknesses

1,The authors report answer-entity recurrence of 37.7% (dev) vs 100% (test). Notes keyed by (dept, gold entity) fire when the entity appears among options — under full test-side recurrence, a "memory hit" may effectively telegraph the answer. No entity-disjoint partition, shuffled-key control, or plain entity-cache baseline is provided (acknowledged). This is the sharpest threat to validity of the diagnostic channel; the non-significant +3.0 open-ended transfer does not dispel it. 2,NEJMQA test is only 244 items; all Table 4 ablations are single runs; the one-way vs. bidirectional contrast includes zero in both CIs (p=0.172/0.343); warmup p=0.096; standalone DiagnosisEvo p=0.078; all three MedQA-Open McNemar tests non-significant. In short: the headline gains are significant, but almost every mechanism-attribution test is not — the one-way constraint's benefit is currently an architectural argument, not empirical evidence. 3,BR_t holds only 10 traces — the patch gate may be noisy on such a small replay set (sensitivity analysis needed); proprietary endpoints drift, weakening reproducibility; heavy naming overhead (four coined components) and an overloaded Figure 2;

Questions for authors

see Weaknesses

1 comment
AuthorsOpenReview ↗

Response to iqju

W1. Same controls as 4zEn-W4: entity-cache 34.5%, shuffled-key 1.0%, note-pool entity-disjoint 72/46 items, same-entity distractor 52 cases with 4/52 distractor predictions. No full train/test entity-disjoint partition; limitation stated.

W2. We agree the mechanism tests are exploratory; the headline is significant. The one-way constraint is structural reliability, not an accuracy claim: toggling the diagnosis channel moves routing by only -0.61, as Eq. 4 predicts. We will add item-level statistics and per-seed flips; a 3-seed warmup/DiagnosisEvo-only test can be added if the AC requests.

W3. For the 8 accepted TriageEvo patches, the 10-case window is always full and has zero overlap with the patch source cases; source cases are routing errors used as positive evidence, not replay. We will provide the 3/5/10/20 composition if requested. Hosted-API drift is why we report 3 seeds and flip ratios; we will simplify naming/Figure 2.

Reviewer_EJ8yOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes EvoTD, a self-evolving memory framework for medical question answering that separates specialty triage from diagnostic inference. The framework has three components: DualEvo, which maintains diagnostic memories; SkillRoute, which selects routing strategies based on the case complexity; and ReliHarness, which controls memory updates. EvoTD gives small accuracy gains over existing methods while using fewer LLM calls at inference time.

Critical strengths
  1. The single-pass memory injection strategy in ReliHarness effectively reduces inference overhead while maintaining diagnostic performance.
Critical weaknesses
  1. The paper is hard to follow, and needs substantial improvement in clarity and organisation. Many terms, notations, and components are introduced without definitions or implementation details, and key steps such as graph-based routing, KG construction, and memory evolution are not clearly described.
  2. Basic dataset statistics, data splits, and the evaluation protocol are missing. Also, the gain over the strongest baselines are marginal. On NEJMQA in thinking mode, the gain is +0.52. That is about 1 extra correct answer on the 244 test questions mentioned in the L451 or about 3 if all 655 questions were used. Same on the MMLU-Pro health dataset, the gain is only +0.19, which is roughly 4 questions.
Questions for authors

See the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to EJ8y

We thank the reviewer; we agree clarity is the main issue. Below we define the terms, describe KG/graph routing and memory evolution, and give the dataset/protocol table. We will add a notation table, a compact algorithm, a KG paragraph, a dataset/protocol table, and this glossary to the revision.

1. Terms and components. EvoTD = the overall two-stage framework (specialty triage + diagnostic inference). DualEvo = the two evolution channels. TriageEvo = routing channel: it proposes specialty-boundary patches from recurrent confusion pairs; a patch is accepted only after zero-regression replay. DiagnosisEvo = diagnosis channel: it distills entity-indexed notes from correctly answered traces. ReliHarness = the screening layer: warmup, top-1 entity retrieval, single-pass pre-injection, harmful-memory quarantine, and replay verification. SkillRoute = the routing context builder: deterministic boundary rules -> graph scoring -> LLM decision cards, at an evidence-adaptive depth. TriageCard = the per-case trace (evidence frame, route, specialty context, injected notes, answer, rationale, execution spans), shared read-only by both channels. Patch = a routing boundary rule; note = an entity-indexed diagnostic rule; memory = the umbrella term for both. KG = department knowledge graph. Boundary rule = a rule with if_all/unless_any conditions that boosts or suppresses a specialty. Cue = a clinical keyword/phrase; option-only cues are weak. Boundary fast path = deterministic route when exactly one safe boundary rule fires. Warmup = first 30% no note injection. Top-1 retrieval = keep only the highest-scoring entity-matched note. Single-pass pre-injection = inject the note once before one answer prediction; no second corrective pass. Quarantine = retire a note when observed harm exceeds help; retire a patch when R2W exceeds W2R. Replay set B^R_t = the most recent 10 LLM-path traces that were both routing- and answer-correct. R-Acc/A-Acc = routing/answer accuracy; W2R/R2W = wrong-to-right/right-to-wrong flips. Offline = frozen train-derived memory; online = memory evolves from the stream.

2. KG and graph-based routing. The KG has 47 domains, 8 boundary rules, 149 cue terms, plus age/sex/life-stage. Evidence extraction pulls age/sex/life-stage and stem cues. Domain scoring uses exact and partial keyword matches; boundary rules add strong positive weights and can add contraindication penalties. SkillRoute has three depths: (i) boundary fast path when exactly one safe boundary rule fires (confidence about 0.95); risky boundary rules are excluded; (ii) deterministic graph path when the KG/evidence score is strong enough (confidence >=0.85 and a score margin of 2.0, or 4.0 when multiple specialties have cues); (iii) LLM slow path for ambiguous cases, using top-2 department decision cards. A verifier flags low-confidence or low-margin cases for review. Progressive disclosure loads controller-only, graph-only, full, or top-k skills; k=2 for NEJMQA, k=3 for the second corpus.

3. Memory evolution. TriageEvo triggers on recurrent (gold,pred) confusion pairs; a teacher writes a short boundary rule; the patch is injected only into the LLM routing prompt. Acceptance requires zero regression on the most recent 10 LLM-path traces that were both routing- and answer-correct; the patch's own source cases are used as positive evidence, not as the replay set. DiagnosisEvo distills positive notes from correctly answered train traces; each note stores (department, gold_entity, applicability cues, reasoning rule, source traces, harm flag). Retrieval scores question and option overlap; top-1 is injected before a single prediction. Warmup suppresses notes for the first 30% of the stream; a note is quarantined when observed harm exceeds help. Offline freezes train-derived patches and notes; online evolves both from the stream.

4. Data and protocol. NEJMQA: 411 train / 244 test, 5 departments, 4 options, MCQ. MMLU-Pro Health: 572 / 246, 8 subjects, 10 options, MCQ. MedQA-Open: 230 evaluated per seed, option-free. Seeds: NEJMQA 42/43/44; MMLU 42/45/46; open-ended 44/45/46. We report structured and CoT prompts, no-think and think cells, 3-seed means, item-level bootstrap (10k), paired sign-flip/exact McNemar, W2R/R2W, R-Acc/A-Acc, F1/MCC. 5. What the gain means. +0.52 = NEJMQA think CoT 81.40 - GSEM 80.88; +0.19 = MMLU think struct 81.90 - SkillOpt 81.71. These are single-cell differences vs the strongest baseline, not the headline. The headline is +7.10/+6.10 vs direct prompting (87/35, 58/13; CI [2.87,11.48]/[2.71,9.76]; p=.0016/.0007). We do not claim SOTA; we claim comparable accuracy at ~1.6 vs ~13 calls/item plus reliability.

6. Revision. We will add the notation table, compact algorithm, KG paragraph, dataset/protocol table, and glossary appendix, and simplify Figure 2 and the component names.

Reviewer_4zEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoTD addresses the inconsistency between specialty triage and downstream diagnostic inference in medical question answering. It introduces DualEvo to separately evolve specialty-boundary patches and diagnostic notes, preventing diagnostic experience from interfering with specialty routing. SkillRoute adaptively combines deterministic rules, graph-based scoring, and LLM reasoning, while ReliHarness improves memory reliability through entity-indexed note retrieval, warmup, single-pass pre-injection, and harmful-memory quarantine. Experiments on NEJMQA and MMLU-Pro Health show that EvoTD outperforms direct prompting by 7.1 and 6.1 percentage points, respectively.

Critical strengths
  1. EvoTD maintains separate specialty-boundary patches and diagnostic notes, preventing diagnostic experience from improperly influencing specialty triage. Its warmup, single-pass pre-injection, and harmful-memory quarantine mechanisms further mitigate the risk of performance degradation caused by unreliable memories.

  2. SkillRoute adaptively selects among deterministic rules, graph-based scoring, and LLM reasoning according to case complexity, reducing unnecessary LLM calls while consistently outperforming direct prompting on two medical question-answering benchmarks.

Critical weaknesses
  1. Although EvoTD decomposes medical QA into specialty routing and diagnostic inference, the paper does not establish its incremental benefit over a strong LLM that may implicitly identify and apply the relevant specialty perspective. It is therefore unclear whether the framework addresses a broadly meaningful problem or mainly a dependency introduced by its own workflow design.

  2. EvoTD primarily separates and regulates triage and diagnostic memories through stage-specific access control, warmup, retrieval, replay screening, and memory quarantine. While this is a potentially useful systems contribution, it introduces neither new medical knowledge nor a fundamentally new diagnostic reasoning mechanism. The paper should therefore be positioned more clearly as a memory-governance framework rather than an advance in clinical reasoning.

  3. Most of the reported gains appear to come from DiagnosisEvo, while TriageEvo’s incremental contribution is not established through controlled factorial ablations or paired significance tests.

  4. Because diagnostic notes are indexed by gold answer entities that are often visible in the multiple-choice options, the gains may reflect entity lookup or answer caching rather than generalizable reasoning transfer. Entity-disjoint evaluation, entity-cache and shuffled-key baselines, and counterfactual cases requiring different decisions for the same entity are needed to disentangle these effects.

  5. EvoTD assumes immediate and reliable gold-label feedback, whereas real clinical feedback is often delayed, incomplete, noisy, or disputed. Its quarantine mechanism is correlational rather than counterfactual, and its small replay set provides only limited in-distribution protection. Thus, the results demonstrate benchmark effectiveness but not robustness to distribution shift, clinical reliability, or deployment readiness.

Questions for authors

Please refer to the Weaknesses section for details.

1 comment
AuthorsOpenReview ↗

Response to 4zEn

TITLE: Response to 4zEn

W1. Pure LLM is a strong implicit-routing baseline (same backbone/prompt). Headline +7.10/+6.10 (87/35, 58/13 flips; CI [2.87,11.48]/[2.71,9.76]; p=.0016/.0007). SkillRoute is the cold-start substrate, not the contribution.

W2. We agree; we will reposition as a memory-governance/reliability framework; novelty is stage ownership + asymmetric gating, not distillation/retrieval.

W3. The offline full cell preloads frozen train-derived TriageEvo patches + DiagnosisEvo notes; neither updates on test, by design. The Table 4 2x2 gives 71.31 / 71.72 / 74.09 / 76.64. TriageEvo's answer effect is +0.41 (within 2.05pp routing noise); its routing effect is +1.23 R-Acc. DiagnosisEvo carries accuracy; TriageEvo is reliability/auditability. Artifacts/config available on request.

W4. 84 injected cases: entity-cache 34.5%, random 25.1%, shuffled-key 1.0% (95% CI 0-3.6%), paired SkillRoute 76.2%, EvoTD 84.5%. Note-pool entity-disjoint (no train-note entity in any option): 72 NEJMQA items, EvoTD 75.0% vs 69.4% SR vs 63.9% Pure; 46 MMLU items, 76.1% vs 65.2% Pure. Same-entity counterfactual: 52 distractor-entity cases, EvoTD 84.6% vs 80.8% SR; only 4/52 predict the distractor. We do not provide a full train/test entity-disjoint partition; the note-pool subset is the closest audit.

W5. Gold is appended only after prediction; online is sequential supervised adaptation, not deployment. Quarantine is outcome-level because a counterfactual second pass is stochastic and harmful (two-pass 68.44 vs 68.03; flips 11/18). B^R_t=10 is a cost-controlled zero-regression replay.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview + Terminology Table — detailed responses under each review

We thank the reviewers. EvoTD's contribution is an ownership boundary: triage and diagnosis are inference-coupled but evolution-decoupled. TriageEvo updates boundary patches under a strict zero-regression replay gate; DiagnosisEvo updates entity-indexed notes under warmup/top-1/harm quarantine; the one-way constraint keeps notes out of the router. This asymmetry is deliberate: strict triage buys reliability/auditability, loose diagnosis carries accuracy, so DiagnosisEvo dominates by design. All EvoTD rows use the same engine; offline preloads train-derived patches+notes (no test-stream update), online evolves both channels; ablations disable components by design.

Detailed point-by-point responses are posted under each review (4zEn, EJ8y, iqju).

Revisions: reframe around the ownership boundary/asymmetric gating; add notation/algorithm/KG and a dataset/protocol table; add entity-cache/shuffled-key/note-pool-disjoint/same-entity controls; state the entity-disjoint and TriageEvo factorial limitations explicitly; add BR_t 3/5/10/20 if requested.

Terminology Table

TermMeaningWhere it appears
EvoTDOverall two-stage framework: specialty triage + diagnostic inference, with decoupled memory evolution.Title/Abstract
DualEvoThe two evolution channels: TriageEvo (routing patches) and DiagnosisEvo (diagnostic notes).§3.3, Fig. 2
TriageEvoRouting channel. Proposes specialty-boundary patches from recurrent confusion pairs; accepted only after zero-regression replay.§3.3, Table 4
DiagnosisEvoDiagnosis channel. Distills entity-indexed notes from correctly answered traces.§3.3, Table 4
ReliHarnessScreening layer: warmup, top-1 entity retrieval, single-pass pre-injection, harmful-memory quarantine, replay verification.§3.5
SkillRouteRouting context builder: deterministic boundary rules -> graph scoring -> LLM decision cards, at evidence-adaptive depth.§3.4, Fig. 2
TriageCardPer-case trace: evidence frame, route, specialty context, injected notes, answer, rationale, execution spans. Shared read-only by both channels.Eq. 3, Fig. 2
PatchA routing boundary rule, injected into the LLM routing prompt.§3.3
NoteAn entity-indexed diagnostic rule: source specialty, answer entity, applicability cues, reasoning rule, source traces, harm flag.§3.3, §3.5
MemoryUmbrella term for patches and notes.§3.3
KGDepartment knowledge graph: positive domains, boundary rules, cue aliases, life-stage signals.§3.4
Boundary ruleA KG rule with if_all/unless_any conditions that boosts or suppresses a specialty.§3.4
CueA clinical keyword or phrase extracted from the case stem; option-only cues are weak.§3.4
Boundary fast pathDeterministic route when exactly one safe boundary rule fires; confidence about 0.95.§3.4
Deterministic graph pathGraph-scored route accepted only at confidence >=0.85 and a score margin of 2.0 (4.0 with multiple specialty cues).§3.4
LLM slow pathSkill-injected LLM routing for ambiguous cases, using top-2 department decision cards.§3.4
Progressive disclosureFour skill-loading modes: controller-only, graph-only, full, top-k. k=2 for NEJMQA, k=3 for the second corpus.§3.4
WarmupSuppresses note injection over the first 30% of the stream.§3.5
Top-1 retrievalKeeps only the highest-scoring entity-matched note.§3.5
Single-pass pre-injectionInjects the selected note once before a single answer prediction; no second corrective pass.§3.5
QuarantineRetires a note when observed harm exceeds help; retires a patch when observed R2W exceeds W2R.Eq. 7
Replay set B^R_tThe most recent 10 LLM-path traces that were both routing- and answer-correct; used for zero-regression patch acceptance.Eq. 5
Entity keyNote identity (department, gold_entity); the target-side gold label is never read before prediction.§3.3
R-AccRouting accuracy: specialty assignment matches the corpus label.Experiments
A-AccAnswer accuracy: exact-option or exact-set match; malformed outputs count as errors.Experiments
W2R / R2WWrong-to-right / right-to-wrong flips, computed on paired per-case outcomes.Table 3, Table 4
OfflineMemory is frozen from the development/train pool: preloaded patches + notes; no test-stream update.Table 1, Table 4
OnlineMemory evolves from the evaluation stream: both channels accumulate updates.Table 2
Think / no-thinkWhether the backbone is allowed to emit reasoning tokens before the answer.Table 1
Area_Chair_m66VOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.