- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EVOTD is a train-free memory-evolution framework for medical QA that treats specialty triage and diagnostic inference as inference-coupled but evolution-decoupled. DUALEVO maintains separate specialty-boundary patches (TriageEvo) and entity-indexed diagnostic notes (DiagnosisEvo) under a one-way constraint (Eq. 4) that bars notes from the router; SKILLROUTE builds specialty-conditioned context at evidence-adaptive depth (deterministic rules → graph scoring → LLM reasoning); RELIHARNESS governs updates via warmup, single-pass pre-injection, and harmful-memory quarantine. With DeepSeek-v4-Flash, EVOTD improves over direct prompting by +7.1/+6.1 points (3-seed means) on NEJMQA/MMLU-Pro Health at ~1.6 serving-time LLM calls per item (vs ~13 for GSEM).
Critical strengths
1,The asymmetric blast radius of routing vs. note errors motivates channel-specific acceptance rules (zero-regression replay for patches vs. harm-based quarantine for notes). The argument is coherent and Eqs. 4/5/7 are precise. The residual coupling through the answer-correctness filter on BR_t is honestly acknowledged (§3.3). 2,Item-level bootstrap CIs and paired sign-flip/exact McNemar tests, explicitly avoiding the false precision of concatenating repeated runs; W2R/R2W flip counts (87/35, 58/13) directly address whether evolution overturns already-correct answers. 3,Removing warmup costs 2 points; replacing single-pass pre-injection with predict-then-correct collapses accuracy to the Pure LLM level (68.44 vs 68.03) and inverts flips to 11/18 — clean evidence that a stochastic corrective pass rewrites more right answers than it repairs, a useful cautionary result for Reflexion-style methods generally.
Critical weaknesses
1,The authors report answer-entity recurrence of 37.7% (dev) vs 100% (test). Notes keyed by (dept, gold entity) fire when the entity appears among options — under full test-side recurrence, a "memory hit" may effectively telegraph the answer. No entity-disjoint partition, shuffled-key control, or plain entity-cache baseline is provided (acknowledged). This is the sharpest threat to validity of the diagnostic channel; the non-significant +3.0 open-ended transfer does not dispel it. 2,NEJMQA test is only 244 items; all Table 4 ablations are single runs; the one-way vs. bidirectional contrast includes zero in both CIs (p=0.172/0.343); warmup p=0.096; standalone DiagnosisEvo p=0.078; all three MedQA-Open McNemar tests non-significant. In short: the headline gains are significant, but almost every mechanism-attribution test is not — the one-way constraint's benefit is currently an architectural argument, not empirical evidence. 3,BR_t holds only 10 traces — the patch gate may be noisy on such a small replay set (sensitivity analysis needed); proprietary endpoints drift, weakening reproducibility; heavy naming overhead (four coined components) and an overloaded Figure 2;
Questions for authors
see Weaknesses
Response to iqju
W1. Same controls as 4zEn-W4: entity-cache 34.5%, shuffled-key 1.0%, note-pool entity-disjoint 72/46 items, same-entity distractor 52 cases with 4/52 distractor predictions. No full train/test entity-disjoint partition; limitation stated.
W2. We agree the mechanism tests are exploratory; the headline is significant. The one-way constraint is structural reliability, not an accuracy claim: toggling the diagnosis channel moves routing by only -0.61, as Eq. 4 predicts. We will add item-level statistics and per-seed flips; a 3-seed warmup/DiagnosisEvo-only test can be added if the AC requests.
W3. For the 8 accepted TriageEvo patches, the 10-case window is always full and has zero overlap with the patch source cases; source cases are routing errors used as positive evidence, not replay. We will provide the 3/5/10/20 composition if requested. Hosted-API drift is why we report 3 seeds and flip ratios; we will simplify naming/Figure 2.