acceptodds
Under review as a conference paper at ICLR 2027

From Case Feedback to Auditable Memory: Context Evolution for Complete Blood Count-Based Risk-Oriented Reasoning

Abstract

Large language models (LLMs) show promise in medical reasoning, yet accumulating reusable and auditable experience from structured laboratory findings across case streams remains challenging. To address these issues, we propose a feedback-driven context evolution framework that adapts frozen LLMs through structured memory and inference time context updates. Specifically, we first construct a hierarchical memory architecture that separates stable knowledge from dynamic experience and organizes audit events, memory cards, and retrieval indices to preserve clinical guidance and traceable case-derived experience. We then design a deterministic write gate that checks consistency, merges compatible rules, isolates conflicts, and routes ambiguous candidates for human review, thereby controlling how new experience enters memory. At inference time, section-first retrieval dynamically adjusts the number and composition of retrieved cards to the configured context capacity, integrating relevant stable knowledge with case-derived experience. We evaluate the framework on complete blood count-based case streams for disease detection, tumor risk assessment, and candidate tumor group ranking. Compared with retrieval-augmented generation and other context optimization baselines, the framework improves disease detection and candidate tumor group ranking, while exhibiting sustained performance across a long-horizon case stream.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agent memory, context evolution, retrieval-augmented generation, evidence-grounded reasoning, medical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bMkmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a memory system that lets a frozen LLM learn from a stream of blood test (CBC) cases, using a rule-based Gate to decide which rules learned from feedback are added to memory. On hospital and MIMIC-IV data with Qwen3.8-27B, it reports better disease detection and tumor-group ranking than RAG, DC, GEPA and ACE.

Critical strengths
  1. The problem matters. In medicine, one bad rule in memory can affect many later cases, so studying how to control what gets written into an LLM's memory is worthwhile.

  2. The system is carefully built and well documented. Background knowledge is read-only, each learned rule can be traced back to the cases that produced it, test cases never change the memory, and the appendix gives the full prompts and settings.

Critical weaknesses
  1. For MSA S3 (510 test cases), the reported 22.63% means 115.4 correct predictions, but I think it should be an integer number of cases. Should the accuracy be either 22.55% (115/510) or 22.75% (116/510)? Similar problems appear in HMT S2 (50.06% of 800 = 400.48 cases) and HMT S3 (21.20% of 400 = 84.8 cases). In contrast, RAG's 22.55% on MSA S3 corresponds to exactly 115 correct cases. The claimed 0.08-point gain over RAG is therefore smaller than one test case. Appendix D.3 does not clearly explain these discrepancies. Why?

  2. Most of the improvement over the Frozen LLM already comes from static knowledge (RAG). The only Gate ablation is on HMT (Table 3). The paper also gives no counts of accepted, merged, or rejected rules. "Quarantine" and "human review" are only decision labels, not actual operations. Please report the Gate's decisions and test whether it can reject bad rules produced by deliberately incorrect update labels.

  3. On HMT, S2 accuracy is 50.06%, barely above always predicting tumor. The S3 gains over GEPA and ACE mainly come from higher coverage. Among cases that reach S3, both baselines actually rank better. There is also a data shortcut: all normal cases are from the hospital, while MIMIC cases all have disease and lack four platelet fields. Please report S2/S3 accuracy with correct upstream routing, and give separate results for hospital and MIMIC cases.

  4. Memory is built only with Qwen3.8-27B. Table 2 tests two other models using this memory, but gives no results without memory. Both also perform worse on S3. It is therefore unclear whether the method helps other LLMs. Important baselines are missing: memory without the Gate on both datasets, retrieval of similar labeled cases, and logistic regression trained on the same cases. The comparison is also unfair: DC receives no true labels, and DC, GEPA, and ACE have a 2,400-token memory limit, while the proposed method uses more memory.

  5. Section 3 shows no real learned rule or Gate decision, and key implementation details are left to Appendix B. Please add an example showing how one labeled case produces a rule, how the Gate handles it, and how it is later retrieved.

  6. The arrows in Figure 1 are weird. Some of them are not placed properly. The elements and layout in this image look very much like they were automatically generated.

  7. The Supplementary Material is just the same appendix contained in the current submitted PDF, not any other evidence like code or some additional experiments to strengthen the claim of this paper. This is not appropriate.

Questions for authors

Please address the concerns in the Weaknesses.

5 comments

(1/4): Integer counts and the revised RAG/Memory comparison

W1. Why do the reported percentages imply fractional numbers of correct cases, and how should the 0.08-point difference from RAG be interpreted?

Response. The earlier summary combined inconsistent evaluation denominators. We have recompiled the results with explicit valid-output counts and run-wise aggregation; the old entries and the 0.08-point comparison are superseded. We do not round the old percentages into an inferred number of correct patients.

For each stage and run, N is the fixed reference-eligible population, A the number of valid stage outputs, C the number correct, and H the S3 Top-3 hits. Incorrect but valid answers remain in A. RAG and Memory are reported as C/A and H/A, accompanied by coverage A/N. Stage non-arrivals, explicit non-answers, and invalid outputs remain in N. F1 and balanced accuracy use the stated scoring population.

RAG is a single evaluation. Memory reports the equal-weight mean of five recorded learning-order scores, with sample SD: mean accuracy = (1/5) sum_r(C_r/A_r). Each run has integer counts, but the mean percentage need not correspond to an integer obtained by multiplying it by the full reference N. The repeated orders use the same test patients.

The revised comparison uses RAG that retrieves knowledge together with case examples, and Memory, our evolving-memory method after 700 updates. Both are scored on valid stage outputs. Values below follow the revised result summaries (%; +/- denotes between-order sample SD). Memory uses the confirmed 16K context, anchors ON, and Gate. The methods retain their recorded resource settings.

Protocol/methodS1 AccS2 AccS2 macro-F1S3 Top-1S3 Top-3
MSA RAG (knowledge + cases)88.6189.5947.2626.2050.57
MSA Memory89.85 +/- 2.6283.71 +/- 0.7657.2028.42 +/- 0.4853.73 +/- 0.55
HMT RAG (knowledge + cases)81.4162.4539.7922.1950.30
HMT Memory82.91 +/- 3.6554.27 +/- 2.5253.1440.88 +/- 9.1059.97 +/- 4.55

The integer bindings are:

Protocol/stageN per runRAG C/A, one evaluationMemory sum(C)/sum(A), five orders
MSA S1780669/7553358/3740
MSA S2600439/4902065/2467
MSA S3510115/439569/2003
HMT S11200946/11624825/5830
HMT S2800341/5461536/2831
HMT S340075/338302/781

The last column provides aggregate integer traceability, not the formula used for the displayed means. For example, HMT S3 has pooled accuracy 302/781 = 38.67%, while the equal-weight run mean is 40.88%. S3 Top-3 hit totals are 222/170 for RAG and 1076/460 across our five orders on MSA/HMT.

Our RAG calculation, 115/510 = 22.55%, is the full-population result. Its revised answered-case score is 115/439 = 26.20%. This change in denominator is explicit; it is not a new prediction gain. Likewise, the earlier values for our method, 22.63%, 50.06%, and 21.20%, are replaced, not reinterpreted as single-run counts.

S1/S2/S3 coverage is 96.79/81.67/86.08% for MSA RAG and 95.90/82.23/78.55% for MSA Memory; corresponding HMT values are 96.83/68.25/84.50% and 97.17/70.78/39.05%. We interpret the conditional rates with these answered subsets and do not infer statistical significance from the old sub-case margin.

Manuscript changes. The revised PDF defines these quantities in Section 4.1 (p. 6, lines 308-314), reports the revised scores in Table 1, and provides aggregation and integer summaries in Appendix D.3 (p. 31, Table 13), with endpoint SD in Appendix E.1 (p. 32, Table 14).

(2/4): The additional role of memory and the Gate's actual operations

W2. What is added beyond knowledge-plus-case RAG, what does the Gate actually do, and what are its admission decisions?

Response. Our retrieval reference already includes both knowledge and case examples. The added mechanism is a controlled transition from case feedback to reusable rules: a Reflector proposes an edit, an independently executed Gate decides its admission, and the committed card retains its provenance and version. Part 1 compares the implemented RAG and Memory settings on valid stage outputs; the action records below document the memory-update mechanism.

1. Concrete write-path operations. HUMAN_REVIEW withholds a proposed candidate from automatic admission and records its reason. QUARANTINE_CONFLICT blocks a conflicting candidate write and preserves the audit record. These decisions affect what enters active memory. Completed clinician adjudication and re-entry are separate operations, and capacity deactivation is recorded separately from admission. Section 3.3 (p. 5, lines 230-249) now specifies these semantics.

2. Recorded action counts. The following rows concern the Protected architecture in the completed two-protocol supplement. They include initialization plus 700 updates, totaling 1050 events on MSA and 1400 on HMT. New, Update, Merge, Review, and Invalid denote WRITE_NEW, UPDATE_EXISTING, MERGE_EXISTING, HUMAN_REVIEW, and REJECT_INVALID.

ProtocolNewUpdateMergeNo changeReviewInvalidTotal
MSA Protected42626455325161050
HMT Protected72325762014111400

These are observed operations; a no-change is not counted as detection of a medically incorrect rule. Appendix E.6, Table 19 (p. 35), retains the complete action counts for all architecture arms.

3. Controls on both protocols. The added architecture comparison evaluates Protected, E-only, and Unified using the same current decoder, learning cases, 700 subsequent clean updates, test patients, and retrieval grouping in one learning order. All three retain Gate. Protected keeps K read-only; E-only removes K and its anchors and retains the E allocation; Unified permits K edits while preserving K/E retrieval groups.

Protocol/armNatural S3 Top-3 (%)Coverage (%)
MSA Protected50.3584.90
MSA E-only39.6342.55
MSA Unified43.3068.82
HMT Protected48.3776.50
HMT E-only27.272.75
HMT Unified40.8669.75

Protected has higher natural-routing S3 Top-3 and coverage than these two alternatives in both protocols. This is a comparison of the implemented configurations, including their allocation and anchor conditions. It is reported separately from a pure Gate switch; stage-wise results and action counts are reported in Appendix E.6, Tables 17-19 (p. 35).

4. Decision-level verification. A fixed-candidate replay reproduces 400 recorded decisions and their pre-update states. Enabling the admission checks changes four WRITE_NEW decisions to HUMAN_REVIEW through internal-consistency checks. This demonstrates that the checks affect admission actions. We interpret the replay as operational evidence for the implemented checks, rather than certification of medical label correctness.

Manuscript changes. Section 3.3 now describes the executable write semantics. Section 4.3 and Appendix E.6 distinguish the historical configuration comparison from the two-protocol architecture controls, with their observed action counts and interpretation.

(3/4): HMT S2, S3 coverage, and Hospital/MIMIC results

W3. Is HMT S2 essentially always-tumor prediction, are S3 scores driven by coverage, and what happens under correct upstream routing and within each data source?

Response. We now report class-balanced behavior and coverage alongside the revised stage scores, and provide the requested source breakdown and correct-upstream analysis. These quantities address different parts of the concern and are kept distinct.

1. S2 class behavior. The previous HMT S2 value 50.06% is superseded by the valid-output mean 54.27 +/- 2.52%, conditional BA 56.42%, and coverage 70.78%. An always-tumor rule has BA 50% when both classes are represented. Figure 3 and the class-wise analysis report HMT non-tumor/tumor recall of 66.97/45.86% among valid outputs, with class coverage 55.90/85.65%. This describes the selected configuration more precisely than accuracy alone. The definitions and class-wise results appear in Section 4.4 and Appendix E.4, Table 16 (p. 34).

2. S3 scoring and correct upstream conditions. We pair Top-1/Top-3 with coverage and distinguish valid-output quality from success on the complete reference cohort. The earlier aggregate ranking is not treated as a same-patient stage comparison.

The correct-upstream memory diagnostic is reported separately in Appendix E.8, Table 22 (p. 37). It supplies disease eligibility for S2 and tumor eligibility for S3, without revealing the target tumor group. The reference S2/S3 populations are 600/510 on MSA and 800/400 on HMT. This diagnostic uses the current decoder and a fixed historical memory snapshot. The RAG/Memory comparison in Part 1 uses the revised valid-output result summaries; each analysis retains its stated protocol and denominator.

3. Memory results by source. Following the request, the source breakdown below contains the memory method alone. Scores are percentages among valid stage outputs from the fixed source-stratified memory snapshot. They are not a decomposition of the main-table five-order means.

ProtocolSourceS1 AccS2 AccS3 Top-1S3 Top-3
MSAHospital86.9291.1924.8342.07
MSAMIMIC81.5485.2221.5650.19
HMTHospital85.0072.7335.0046.25
HMTMIMIC71.5058.9721.8355.95

S1 coverage is 100%. S2/S3 coverage is 75.71/81.01% for MSA Hospital, 81.54/81.27% for MSA MIMIC, 55.00/80.00% for HMT Hospital, and 71.50/84.00% for HMT MIMIC. The Hospital source has higher conditional S1/S2 accuracy, while MIMIC has higher conditional S3 Top-3 in both protocols. The source gap is therefore task-dependent.

All normal controls are Hospital-derived, and field availability differs between sources. We state these cohort properties explicitly and interpret the table as source-specific performance, not evidence that the source/class association has been removed. Diagnosis labels and explicit source identifiers are excluded from the retrieval query; missingness structure is treated as a separate input-property issue. The source analysis is documented in Appendix E.8.

Manuscript changes. The revised source analysis includes the natural-routing breakdown and coverage in Appendix E.8, Table 21 (p. 36), and correct-upstream results in Table 22 (p. 37). The class-balance and coverage discussion is separated from the method's memory-control contribution.

Reviewer_4UZnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a context evolution framework for sequential complete blood count (CBC) reasoning using frozen LLM. To resolve the stability–plasticity dilemma during inference time, the work establishes a hierarchical memory that decouples read-only "stable knowledge" from updatable "dynamic experience cards". Case-derived feedback generates candidate rules that must pass through a deterministic Write Gate enforcing structural checks, Jaccard similarity constraints, and conflict isolation before entering active memory. At inference time, section-first retrieval constructs bounded prompts for a three-stage diagnostic pipeline (S1: disease detection, S2: tumor risk assessment, S3: 8-class tumor group ranking).

Critical strengths
  1. The work separates read-only clinical knowledge from revisable dynamic rules, while maintaining immutable audit logs that link every memory edit directly to its originating case feedback. This guarantees complete inspectability and traceability without updating underlying LLM weights.
  2. By routing candidate experience through a deterministic Write Gate that enforces structural validation, keyword similarity thresholds, and conflict isolation, the system effectively shields active memory from single-case noise and knowledge corruption.
  3. The work design realistic, multi-stage case streams across hospital and MIMIC-IV cohorts. The evaluation incorporates longitudinal checkpoint tracking, controlled ablation studies, and cross-backbone transfer experiments.
Critical weaknesses
  1. The core Write Gate relies on surface-level Python heuristic: such as Jaccard string similarity thresholds, character caps, and exact keyword filtering, rather than learned semantic verification. It functions as a rule-based engineering wrapper over existing Reflector and RAG paradigms rather than a novel learning algorithm.
  2. Under the MSA protocol, Stage 3 Top-1 Accuracy and Acc@3 show negligible gains over static RAG. Furthermore, in Stage 2, static RAG achieves higher accuracy on MSA, while ACE and GEPA achieve higher F1 scores on HMT.
  3. Confusion matrices reveal an extremely low recall for non-tumor disease cases, with many non-tumor cases misclassified under tumor risk or missing outputs. The feedback loop fails to resolve this non-tumor confounder bias.
  4. Although the Write Gate routes ambiguous rules to a "Human Review" status, no clinical experts participated during automated evaluation. Consequently, this routing acts as a rejection slot rather than a functional human-in-the-loop review mechanism.
  5. Transferring memory cards evolved on Qwen3.8-27B to recipient backbones (Gemma 4 26B and MedGemma-27B) leads to noticeable performance drops in HMT Stage 2 and 3, exposing a strong dependency on the originating model.
Questions for authors
  1. The innovation claimed in this work revolves around the "Deterministic Write Gate" and "Context Evolution Framework". However, the Write Gate appears to rely fundamentally on deterministic Python heuristics: such as Jaccard string similarity thresholds, word-count caps, and exact keyword filtering, to gate and merge prompt cards. What is the fundamental algorithmic or theoretical novelty here beyond wrapping standard Reflector/RAG paradigms in a deterministic rule engine? How does this design differ from conventional rule-based prompt filtering?
  2. The Write Gate screens rules primarily based on surface-level textual overlap and formatting constraints. How can the authors ensure that the rules approved by the Write Gate are medically sound? Since a rule with high structural compliance and low lexical overlap could still encode false clinical heuristics, how does the system prevent non-lexical medical fallacies from corrupting the active memory pool without human expert intervention?
  3. In Table 2 under the MSA protocol, the proposed framework achieves a Stage 3 Top-1 Accuracy of 22.63% and Acc@3 of 43.55%, which is virtually identical to the baseline static RAG. Moreover, in Stage 2, the method's accuracy under MSA (70.53%) falls below static RAG. Given the substantial overhead of memory reflection and write-gate management, how do the authors justify the practical necessity and benefit of dynamic context evolution?
  4. According to Figure 3, the system exhibits a severe false-positive bias toward tumor risk, resulting in an extraordinarily low recall for non-tumor diseases. Why does the feedback-driven reflection mechanism consistently fail to correct this bias against non-tumor confounders, and what causes the evolved rules to over-index on tumor features at the expense of non-tumor precision?
4 comments

Response 1/4: Q1 - Innovation beyond prompt filtering

Q1. What is the algorithmic contribution beyond deterministic heuristics, and how does it differ from rule-based prompt filtering?

Response. Our contribution is a feedback-driven algorithm for controlled evolution of persistent reasoning memory. It turns case-level feedback into reusable rules while separately specifying their write permissions, admission, revision and subsequent retrieval. This directly addresses the stability-plasticity problem formulated in the Introduction: incorporating experience without overwriting the protected knowledge substrate.

The paper makes three connected contributions:

1. Adaptation as a constrained memory transition. The backbone stays frozen. Stable knowledge K is read-only, while experience E is editable. Consequently, K_t = K_0 throughout adaptation by construction; case feedback can change E without directly rewriting K. This places an explicit constraint on the reachable memory states while permitting case-derived learning.

2. A separate proposal-to-admission algorithm. For each learning case, the Reflector uses the prediction, applicable reference feedback and retrieved cards to propose an edit. A separate Gate applies operation-specific structural, permission, duplicate and implemented consistency checks before the edit can change reusable rule content. Accepted edits carry provenance and version links. The cycle is retrieval -> prediction -> feedback -> candidate -> admission/update -> later retrieval (Section 3.3-3.4).

3. Evaluation of evolving memory across CBC case streams. Section-first retrieval combines protected knowledge and active experience within bounded context. Saved checkpoints, read-only testing and cross-backbone reuse make the evolving state inspectable at both update and prediction time. The evaluation therefore measures task outcomes together with how experience is admitted and reused.

Conventional prompt filtering selects or removes content for the current input. Here the controlled object is a persistent state that can influence later cases: a rejected candidate does not become an active rule, an admitted update has a recorded predecessor and source, and later predictions can be linked to the retrieved version. Jaccard similarity implements one admission check within this state-transition procedure.

Relation to the closest methods. Static RAG retrieves existing knowledge. Our method additionally abstracts stage-specific feedback into new experience and controls its persistent reuse. ACE (ICLR 2026) evolves playbooks through generation, reflection and curation with structured incremental updates. Dynamic Cheatsheet (EACL 2026) maintains self-curated memory without requiring reference labels. Our evaluated design couples stage-specific reference feedback with protected K/editable E permissions and a version-linked admission-to-retrieval trace. Reflection and incremental memory have prior precedents; the specific contribution is this controlled write-and-reuse formulation and its evaluated operation.

Behavioral evidence. The completed architecture comparison tests Protected, E-only and writable-K Unified configurations after 700 updates. Natural-routing S3 Top-3 / coverage (%) is:

ProtocolProtectedE-onlyUnified
MSA50.35 / 84.9039.63 / 42.5543.30 / 68.82
HMT48.37 / 76.5027.27 / 2.7540.86 / 69.75

Protected has higher Top-3 and coverage than these two alternatives in both protocols under natural routing. These are complete configuration comparisons: the supplement allocates 1,600/800 tokens to K/E, and E-only leaves K's allocation unused. The appendix retains the allocation details and oracle-stage results. Fixed-candidate replay provides a complementary mechanism test: four admission decisions change from WRITE_NEW to HUMAN_REVIEW with the Gate, holding the recorded candidate and pre-state fixed.

We will foreground these three contributions, the transition procedure and the closest-method distinctions in the Introduction, Section 3 and related work.

Response 2/4: Q2 - Medical content and incorrect feedback

Q2. How are medically incorrect rules addressed when surface compliance alone cannot establish clinical soundness?

Response. Medical content generation and permission to modify memory are separate steps. The Reflector uses the case, applicable clinical reference feedback, prediction error and retrieved knowledge to formulate guidance with applicability and limitations. The Gate determines whether that candidate may alter reusable memory. Lexical compliance is not treated as clinical adjudication.

What the controls do. Learning writes cannot overwrite read-only K. Candidate edits to E undergo the checks defined for their operation; candidates assigned HUMAN_REVIEW are withheld from active reuse and their reasons are logged. Versions and source links make admitted edits traceable to the feedback that produced them. HUMAN_REVIEW is an automated candidate-withholding status in these experiments; the automated evaluation did not execute clinician adjudication or re-entry.

A separately documented clean HMT trace at update position 37 illustrates the content-to-reuse path: incorrect tumor-risk reasoning led to guidance about anemia and thrombocytopenia as non-tumor confounders. The candidate received WRITE_NEW, remained active at version 2 after 100 updates, and was retrieved in S2/S3 requests for one held-out case.

Controlled incorrect-feedback test. The experiment uses the same initialized M0 and first 100 HMT update positions, with S3 feedback labels changed for 10 of the 48 tumor cases (20.83%). S2 labels and held-out references remain unchanged. Controlled NoGate retains shared schema, target/permission and capacity constraints. The complete updated results are below (%, 800 oracle-S2 and 400 oracle-S3 cases; coverage 100%):

StateS2 AccS3 Top-1S3 Top-3
M052.0019.2545.75
Gate Clean50.62520.0045.75
NoGate Clean52.5019.0045.25
Gate Noisy50.2519.5044.75
NoGate Noisy51.7515.2538.25

S3 performance under incorrect feedback. With noisy feedback, Gate yields 19.50% Top-1 and 44.75% Top-3, compared with 15.25% and 38.25% for NoGate: differences of 4.25 and 6.50 percentage points. The Top-3 counts are 179/400 versus 153/400, and Top-1 counts are 78/400 versus 61/400.

From clean to noisy feedback, Gate Top-3 changes from 45.75% to 44.75% (-1.00 point), while NoGate changes from 45.25% to 38.25% (-7.00 points). Top-1 changes by -0.50 and -3.75 points, respectively. S2 accuracy changes from 50.625% to 50.25% with Gate and from 52.50% to 51.75% without it.

The perturbation targets the feedback used to construct experience, while held-out references stay fixed. Admission governs which proposed edits may alter persistent memory and thus the context available to subsequent reasoning. The observed comparison shows better preservation of S3 ranking performance with Gate in this controlled test. The result concerns this single-order, 100-update feedback perturbation; rule-level clinical validation remains a separate evaluation target.

We will retain all five states in the revised presentation, report the S3 differences and clean-to-noisy changes explicitly, and distinguish downstream feedback robustness from independently verified medical correctness of individual rules.

Response 3/4: Q3 - Why dynamic context evolution is useful

Q3. How do the updated RAG comparison and the added memory operations justify the practical value of dynamic context evolution?

Response. Dynamic evolution provides a way to construct, revise and reuse experience from feedback, alongside improvements in the reported S3 valid-answer scores. The updated comparison below uses the same CBC evaluation cases and clinical knowledge resource, with a 16K context window for both RAG and Ours. The main Ours configurations use anchors ON, Gate and 700 updates.

Updated RAG/Ours results. Acc and Top-3 are evaluated among valid stage outputs; Cov is valid outputs divided by reference-eligible cases. Valid but incorrect answers remain in the evaluated subset. All entries are percentages; F1 is macro-F1.

Protocol / methodS2 Acc / macro-F1 / CovS3 Top-1 / Top-3 / Cov
MSA RAG89.59 / 47.26 / 81.6726.20 / 50.57 / 86.08
MSA Ours83.71 / 57.20 / 82.2328.42 / 53.73 / 78.55
HMT RAG62.45 / 39.79 / 68.2522.19 / 50.30 / 84.50
HMT Ours54.27 / 53.14 / 70.7840.88 / 59.97 / 39.05

Relative to static RAG, the reported MSA S3 Top-1/Top-3 values increase from 26.20/50.57% to 28.42/53.73% (+2.22/+3.16 percentage points). HMT increases from 22.19/50.30% to 40.88/59.97% (+18.69/+9.67 points). MSA S2 macro-F1 increases from 47.26% to 57.20%; HMT increases from 39.79% to 53.14%. The table also retains S2 accuracy and each stage's coverage, including the lower S3 coverage of Ours.

Ours is summarized across five saved case orders; static RAG is a single evaluation. The selected S3 Top-3 means ± sample SD are 53.7258 ± 0.5481% for MSA and 59.9662 ± 4.5505% for HMT. These scores describe valid-answer quality together with the displayed service coverage.

Why comparable retrieval accuracy does not make the update mechanism redundant. With access to the same evaluation data and knowledge resource, static RAG can retrieve relevant existing guidance. The additional capability we study is what happens after feedback reveals a new error or exception:

  1. The Reflector abstracts the case-specific feedback into reusable experience with scope and limitations.
  2. The write controller decides whether that experience may be added or used to revise E while leaving K protected.
  3. Versions and later retrieval identify how the accepted guidance is reused across subsequent cases.

The tested static RAG baseline keeps its retrieval resource fixed; it does not synthesize or revise experience through this feedback-driven write path. It also has no candidate-admission stage for checking newly proposed feedback-derived edits before persistent reuse. Those operations would require an additional update policy. Here, they are explicit parts of the proposed framework, supported by the admission and reuse records in Responses 1-2.

Overhead and reuse. Reflection and admission operate on the designated learning stream. Held-out evaluation loads a fixed snapshot with reflection and memory writing disabled. The adaptation cost therefore constructs a reusable resource for subsequent read-only reasoning. We will report initialization, updating and read-only inference separately, so the added construction work is visible alongside the task scores, coverage and reusable-memory functionality.

We will replace the earlier result presentation with this updated RAG/Ours comparison and explain the distinction between retrieving existing guidance and learning new reusable experience.

Response 4/4: Q4 - Non-tumor confounders; transfer weakness

Q4. What do the updated Figure 3 and the feedback mechanism show about non-tumor confounders and the apparent tumor-risk bias?

Response. The updated observed-count Figure 3 shows different S2 class-wise profiles in MSA and HMT. Among valid S2 outputs, non-tumor recall is 23.37% in MSA and 66.97% in HMT, while tumor recall is 91.00% and 45.86%, respectively.

For each saved order, the figure normalizes each reference-class row within its valid outputs and then averages equally across five orders. Mean S2 coverage is 82.23% for MSA and 70.78% for HMT. The class-wise values and coverage are:

ProtocolNon-tumor recallTumor recallNon-tumor CovTumor Cov
MSA23.3791.0059.1186.31
HMT66.9745.8655.9085.65

In MSA, 76.63% of answered non-tumor cases are assigned tumor risk; the corresponding HMT value is 33.03%. HMT also has a 54.14% non-tumor prediction rate among answered tumor cases. The class-coverage columns account for eligible cases that did not produce a valid S2 output and are excluded from the plotted rows.

Possible mechanisms. First, the reference disease population differs: MSA contains 510 tumor and 90 non-tumor cases, whereas HMT contains 400 of each. The valid-output subsets can further differ through upstream routing. These differences can change which patterns are reinforced and which cases reach S2.

Second, CBC abnormalities such as anemia or thrombocytopenia may occur in both tumor and non-tumor conditions. Reference feedback identifies a stage-level error, but converting one correction into a reusable rule still requires choosing its applicability. A rule that generalizes a shared abnormality too broadly is one possible path to false-positive predictions.

Third, the Reflector and reasoner operate on the available CBC evidence and model knowledge. Revising guidance can change the context used later, while heterogeneous disease composition, missing fields and source-related patterns may affect how that guidance transfers. These are possible explanations to examine against the observed counts; the matrices alone do not identify their causal contributions.

We will discuss these possibilities alongside the updated class-wise results and preserve both missing-output counts and coverage in the supporting tables. The revised figure is used to characterize the current error profiles, rather than treating a change of denominator as clinical correction.

Weakness: HMT performance drops after transfer to Gemma4/MedGemma.

Response. The recipient runs demonstrate that the frozen external memory can be consumed by both backbones, with stage- and model-dependent performance. The same saved recipient runs give the following valid-output results (%, Acc=C/A, Top-3=H/A, Cov=A/N):

Protocol / recipientS2 Acc / CovS3 Top-1 / Top-3 / Cov
MSA Gemma4-26B88.42 / 90.6720.79 / 46.36 / 94.31
MSA MedGemma-27B87.68 / 92.0015.50 / 41.12 / 94.90
HMT Gemma4-26B59.90 / 78.8819.46 / 44.32 / 92.50
HMT MedGemma-27B53.79 / 92.2513.10 / 38.29 / 99.25

All MSA and HMT recipients import source memory evolved under 16K/ON/Gate700, matching the stated source-memory configuration of the main experiments. The reported results describe cross-backbone performance under this common source-memory configuration.

Natural-language cards provide a common representation that the recipient can read, but the interpretation of a card and the resulting routing decision still depend on the backbone. This explains why executable memory reuse and equal predictive performance are separate properties. The current results address cross-backbone compatibility and observed performance differences; estimating the incremental effect of the transferred memory requires a matched within-recipient no-memory comparison.

We will state the source snapshots, recipient-specific S2/S3 results and coverage together in the transfer analysis, maintaining the distinction between reusable representation and model-independent performance.

Reviewer_S3VCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a feedback-driven context evolution framework that improves CBC-based medical reasoning using frozen large language models without updating model parameters. Its key contributions include a hierarchical memory architecture separating stable knowledge from evolving experience, a deterministic write gate for auditable and conflict-aware memory updates, and section-first retrieval for adaptive context construction. The framework is validated on hospital and MIMIC-IV datasets under mixed-source adaptation and cross-domain transfer settings, covering disease detection, tumor risk assessment, and tumor-group ranking. Ablation studies, long-horizon evaluations, and cross-backbone experiments further demonstrate the benefits of controlled memory evolution, although tumor-risk classification improvements remain inconsistent across metrics.

Critical strengths
  1. The proposed framework provides good traceability of memory updates. Structured memory cards, provenance tracking, and versioned snapshots make it easier to inspect how experience is incorporated and reused.

  2. The experimental evaluation covers multiple tasks and adaptation settings.** The paper includes comparisons with several baselines, long-horizon evaluation, ablation studies, and cross-backbone memory reuse.

Critical weaknesses
  1. The novelty is somewhat limited. While the overall framework is carefully designed, its core components including hierarchical memory organization, feedback-driven reflection, controlled memory updates, and section-first retrieval, are closely related to techniques explored in prior memory-augmented and self-evolving LLMs and agents. The contribution appears to lie more in the systematic integration of these components.
  2. The main experiments lacks an important Case-based RAG baseline. The current RAG baseline retrieves from a fixed knowledge-card collection. However, a key motivation of the proposed method is to utilize case-level experience. Therefore, it is important to comparing the method with a case-based RAG approach that retrieves similar historical cases.
  3. The ablation study is insufficient. Although the paper investigates the effects of the write gate, statistical anchors, and context capacity, two important ablations are missing. First, a w/o Stable Knowledge setting would help determine how much the preserved knowledge contributes to the overall performance, beyond the dynamic experience. Second, a Unified Mutable Memory setting, in which all the memory can be updated or modified through feedback, would directly evaluate the necessity of maintaining a hierarchical memory architecture. Without these ablations, the effectiveness of the architectural design remains insufficiently validated.
Questions for authors
  1. Could the author clarify why do MSA and HMT use different test set sizes and substantially different class distributions? In particular, tumor cases account for 85% (510/600) of disease cases in MSA, compared to 50% (400/800) in HMT.
  2. Could the author clarify why are the results not reported separately for the Hospital and MIMIC test subsets? Such an analysis would help assess whether the proposed method successfully adapts to the new data sourcewhile preserving its performance on the original source.
4 comments

Part 1

(1/4): Why MSA and HMT use different test cohorts

Thank you for the constructive questions and for recognizing the traceability of the framework. We first clarify the two evaluation questions, then address novelty, case-based retrieval, and the architectural controls.

Q1. Why do MSA and HMT have different test sizes and class distributions?

The two protocols were designed to probe complementary adaptation and case-mix regimes. MSA studies adaptation over a mixed-source stream with a tumor-enriched disease population. HMT initializes memory on Hospital cases and updates it on MIMIC cases, then evaluates both sources under a balanced class design. Their test cohorts were constructed using protocol-specific quotas rather than a common prevalence-matched sampling rule.

Specifically, MSA contains 180 normal, 90 non-tumor-disease, and 510 tumor cases, giving 780 test cases and 510/600=85% tumors among disease cases. HMT contains 400 cases in each category, giving 1,200 test cases and 400/800=50% tumors among disease cases. The different totals follow these different quota designs. The intent was to examine a tumor-enriched regime alongside a balanced regime, not to estimate clinical prevalence or make the two protocols interchangeable replications.

This distinction matters for interpretation. An always-tumor predictor attains 85% versus 50% accuracy on the full disease cohorts. We therefore interpret accuracy together with class-balanced and class-specific metrics. For valid-output scoring, the relevant class mixture is that among valid stage answers, which can differ from the full cohort. Within each protocol, methods use the same reference test cohort and eligibility definitions. Cross-protocol differences also involve learning-source composition and case difficulty, so they cannot be attributed to prevalence alone.

We will make the quota rationale and the distinction between full-cohort and valid-output metrics explicit in the experimental setup and appendix.

Part 2

(2/4): Hospital and MIMIC results for memory

Q2. Why were Hospital and MIMIC results not reported separately?

The original presentation emphasized pooled protocol results and did not make the source-specific behavior sufficiently visible. We agree that the breakdown is important for understanding performance on the original and update sources. Below we report the memory results alone, using the same stage definitions: S1 disease detection, S2 tumor risk among disease cases, and S3 tumor-group ranking.

Memory performance by source (%). Accuracy and Top-3 are calculated among valid stage outputs. These figures use the fixed memory snapshot from the source-stratified evaluation; they are reported separately from the main-table five-order means.

ProtocolSourceS1 AccS2 AccS3 Top-1S3 Top-3
MSAHospital86.9291.1924.8342.07
MSAMIMIC81.5485.2221.5650.19
HMTHospital85.0072.7335.0046.25
HMTMIMIC71.5058.9721.8355.95

S1 coverage is 100%. S2/S3 coverage is 75.71/81.01% for MSA Hospital, 81.54/81.27% for MSA MIMIC, 55.00/80.00% for HMT Hospital, and 71.50/84.00% for HMT MIMIC. Coverage is measured against the reference-eligible population at each stage.

The source gap depends on the task: Hospital has higher conditional S1/S2 accuracy, whereas MIMIC has higher conditional S3 Top-3 in both protocols. This breakdown makes the behavior concealed by pooled averages explicit. It describes final-state source performance; changes during adaptation require a comparison with an earlier snapshot. Also, all normal controls are Hospital-derived, so the S1 source gap reflects different class composition as well as source differences, and MIMIC normal specificity is not estimable.

We will include this memory-only source breakdown and its coverage in the revised source analysis and distinguish final-state performance from longitudinal adaptation and retention.

Part 3

(3/4): What is new beyond existing memory and reflection methods?

The contribution is a controlled, inspectable transition from case feedback to reusable reasoning memory, with explicit protection and admission rules. Section 3 specifies the memory state, the permitted updates, and how admitted experience is linked to later use. This provides a concrete basis for comparison with earlier methods.

1. A different object of adaptation. Knowledge-and-case RAG retrieves existing material and labeled examples. Our framework additionally converts feedback on learning cases into reusable, section-specific rules with applicability conditions and limitations. The model remains frozen, while this external state changes across the case stream. The added capability is persistent rule evolution with an inspectable write history, beyond retrieving a different set of examples.

2. Explicit separation of proposal, permission, and admission. GEPA uses reflection to propose and evaluate prompt changes; ACE already uses generation, reflection, curation, and structured incremental playbook updates. We build on these established ideas. Our specific design separates read-only K from revisable E and places an independently executed deterministic Gate between the Reflector's proposal and a committed memory change. A proposal can be admitted, consolidated, or withheld with a recorded reason, while K remains protected throughout the learning stream. This makes the update boundary explicit and reproducible.

3. Inspectability extends through subsequent retrieval. A-MEM dynamically links structured notes and evolves related memories. Our design emphasizes the case-feedback-to-write-to-reuse chain: source feedback, admission decision, card versions and status, snapshot, and the cards actually selected for a later prediction. Section-first retrieval selects active stable and dynamic cards within the configured guidance budget and records their inclusion. This enables inspection of both how experience entered memory and where it was subsequently used.

For example, a recorded HMT S2 error produced a non-tumor-confounder rule about anemia and thrombocytopenia. The Gate admitted the rule; a later snapshot retained it with version and provenance information; its identifier and text then appeared in held-out request messages. This recorded chain demonstrates that the claimed inspectability is exercised during learning and retrieval. Prompt inclusion documents reuse; it does not by itself establish a causal prediction gain.

These design differences provide practical advantages in controlling what may change, withholding a proposed update before it affects active memory, and tracing the experience used in later reasoning. Their empirical role is evaluated through the knowledge-plus-case retrieval comparison and the requested E-only/Unified controls in Part 4. The comparison reports where the operational design is accompanied by measurable benefits and where the alternatives remain stronger.

We will sharpen the contribution and related-work discussion around these specific differences, explicitly credit the shared mechanisms, and connect the design claims to the added controls and recorded execution evidence.

Part 4

(4/4): Knowledge-plus-case RAG and architectural ablations

We address the missing baseline and the two requested architectural controls below.

1. Knowledge-plus-case retrieval versus evolving memory.

The revised comparison uses RAG that retrieves knowledge together with case examples, and Memory after 700 updates. Both are scored on valid stage outputs. Memory uses the confirmed 16K/anchors-ON/Gate configuration. The two methods retain their recorded method-specific resource settings. Memory entries are means +/- sample SD across five saved learning orders of the same test cohort; RAG is a single evaluation. All values are percentages.

Protocol/methodS1 AccS2 AccS2 macro-F1S3 Top-1S3 Top-3
MSA RAG (knowledge + cases)88.6189.5947.2626.2050.57
MSA Memory89.85 +/- 2.6283.71 +/- 0.7657.2028.42 +/- 0.4853.73 +/- 0.55
HMT RAG (knowledge + cases)81.4162.4539.7922.1950.30
HMT Memory82.91 +/- 3.6554.27 +/- 2.5253.1440.88 +/- 9.1059.97 +/- 4.55

Memory has higher conditional S1 accuracy and S3 Top-1/Top-3 in both protocols. Its S3 Top-3 differences are +3.16 points in MSA and +9.67 in HMT. S2 accuracy favors RAG, while S2 macro-F1 favors Memory. This supports stage-specific benefits rather than uniform improvement.

S1/S2/S3 coverage is 96.79/81.67/86.08% for MSA RAG and 95.90/82.23/78.55% for MSA Memory; it is 96.83/68.25/84.50% for HMT RAG and 97.17/70.78/39.05% for HMT Memory. Conditional scores describe different answered subsets and do not imply higher full-cohort success. The appendix will retain the full-population results and distinguish context, guidance, storage, and construction costs; this is not an equal-total-compute claim.

2. Removing stable knowledge and allowing all initial knowledge to change.

We evaluated Protected, E-only, and Unified with the same current decoder, protocol-specific initialization cases, 700 subsequent clean-feedback cases, and test patients in one learning order. These supplementary controls are separate from the main-table five-order comparison above.

Protected keeps K read-only and updates E. E-only removes K and its anchors from initialization onward, retains the 800-token E quota, and does not transfer the unused 1,600-token K quota to E. Unified allows initial K cards to be edited under the same admission procedure while preserving K/E retrieval groups. Thus the controls test the implemented configurations with their stated capacity and anchor conditions.

The compact table reports both natural routing and correct-upstream evaluation. Oracle supplies disease eligibility at S2 or tumor eligibility at S3 without revealing the target tumor group. BA denotes balanced accuracy.

Protocol/armNatural S3 Top-3 / CovOracle S2 BAOracle S3 Top-3 / Cov
MSA Protected50.35 / 84.9051.5747.25 / 100.00
MSA E-only39.63 / 42.5551.1854.71 / 100.00
MSA Unified43.30 / 68.8254.7439.22 / 100.00
HMT Protected48.37 / 76.5061.0046.00 / 100.00
HMT E-only27.27 / 2.7550.5048.17 / 95.50
HMT Unified40.86 / 69.7550.3839.50 / 100.00

Protected has higher natural S3 conditional Top-3 and coverage than both controls in both protocols. Oracle results locate the limits: E-only ranks higher in MSA oracle Top-3; in HMT, E-only and Protected each hit 184/400 cases, with conditional scores 184/382=48.17% and 184/400=46.00%. HMT E-only natural S3 has only 11 valid answers. Unified also has higher natural S1 in both protocols and slightly higher natural HMT S2.

Unified edited three/four K cards in MSA/HMT, but none appeared in final test prompts. We therefore interpret the comparison at the configuration and memory-trajectory level, without attributing the score gap to direct exposure to edited K. We will add these controls and retain the full stage-wise results and edit-exposure details in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.