- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a memory system that lets a frozen LLM learn from a stream of blood test (CBC) cases, using a rule-based Gate to decide which rules learned from feedback are added to memory. On hospital and MIMIC-IV data with Qwen3.8-27B, it reports better disease detection and tumor-group ranking than RAG, DC, GEPA and ACE.
Critical strengths
-
The problem matters. In medicine, one bad rule in memory can affect many later cases, so studying how to control what gets written into an LLM's memory is worthwhile.
-
The system is carefully built and well documented. Background knowledge is read-only, each learned rule can be traced back to the cases that produced it, test cases never change the memory, and the appendix gives the full prompts and settings.
Critical weaknesses
-
For MSA S3 (510 test cases), the reported 22.63% means 115.4 correct predictions, but I think it should be an integer number of cases. Should the accuracy be either 22.55% (115/510) or 22.75% (116/510)? Similar problems appear in HMT S2 (50.06% of 800 = 400.48 cases) and HMT S3 (21.20% of 400 = 84.8 cases). In contrast, RAG's 22.55% on MSA S3 corresponds to exactly 115 correct cases. The claimed 0.08-point gain over RAG is therefore smaller than one test case. Appendix D.3 does not clearly explain these discrepancies. Why?
-
Most of the improvement over the Frozen LLM already comes from static knowledge (RAG). The only Gate ablation is on HMT (Table 3). The paper also gives no counts of accepted, merged, or rejected rules. "Quarantine" and "human review" are only decision labels, not actual operations. Please report the Gate's decisions and test whether it can reject bad rules produced by deliberately incorrect update labels.
-
On HMT, S2 accuracy is 50.06%, barely above always predicting tumor. The S3 gains over GEPA and ACE mainly come from higher coverage. Among cases that reach S3, both baselines actually rank better. There is also a data shortcut: all normal cases are from the hospital, while MIMIC cases all have disease and lack four platelet fields. Please report S2/S3 accuracy with correct upstream routing, and give separate results for hospital and MIMIC cases.
-
Memory is built only with Qwen3.8-27B. Table 2 tests two other models using this memory, but gives no results without memory. Both also perform worse on S3. It is therefore unclear whether the method helps other LLMs. Important baselines are missing: memory without the Gate on both datasets, retrieval of similar labeled cases, and logistic regression trained on the same cases. The comparison is also unfair: DC receives no true labels, and DC, GEPA, and ACE have a 2,400-token memory limit, while the proposed method uses more memory.
-
Section 3 shows no real learned rule or Gate decision, and key implementation details are left to Appendix B. Please add an example showing how one labeled case produces a rule, how the Gate handles it, and how it is later retrieved.
-
The arrows in Figure 1 are weird. Some of them are not placed properly. The elements and layout in this image look very much like they were automatically generated.
-
The Supplementary Material is just the same appendix contained in the current submitted PDF, not any other evidence like code or some additional experiments to strengthen the claim of this paper. This is not appropriate.
Questions for authors
Please address the concerns in the Weaknesses.
5 comments
(2/4): The additional role of memory and the Gate's actual operations
W2. What is added beyond knowledge-plus-case RAG, what does the Gate actually do, and what are its admission decisions?
Response. Our retrieval reference already includes both knowledge and case examples. The added mechanism is a controlled transition from case feedback to reusable rules: a Reflector proposes an edit, an independently executed Gate decides its admission, and the committed card retains its provenance and version. Part 1 compares the implemented RAG and Memory settings on valid stage outputs; the action records below document the memory-update mechanism.
1. Concrete write-path operations. HUMAN_REVIEW withholds a proposed candidate from automatic admission and records its reason. QUARANTINE_CONFLICT blocks a conflicting candidate write and preserves the audit record. These decisions affect what enters active memory. Completed clinician adjudication and re-entry are separate operations, and capacity deactivation is recorded separately from admission. Section 3.3 (p. 5, lines 230-249) now specifies these semantics.
2. Recorded action counts. The following rows concern the Protected architecture in the completed two-protocol supplement. They include initialization plus 700 updates, totaling 1050 events on MSA and 1400 on HMT. New, Update, Merge, Review, and Invalid denote WRITE_NEW, UPDATE_EXISTING, MERGE_EXISTING, HUMAN_REVIEW, and REJECT_INVALID.
| Protocol | New | Update | Merge | No change | Review | Invalid | Total |
|---|---|---|---|---|---|---|---|
| MSA Protected | 426 | 26 | 4 | 553 | 25 | 16 | 1050 |
| HMT Protected | 723 | 25 | 7 | 620 | 14 | 11 | 1400 |
These are observed operations; a no-change is not counted as detection of a medically incorrect rule. Appendix E.6, Table 19 (p. 35), retains the complete action counts for all architecture arms.
3. Controls on both protocols. The added architecture comparison evaluates Protected, E-only, and Unified using the same current decoder, learning cases, 700 subsequent clean updates, test patients, and retrieval grouping in one learning order. All three retain Gate. Protected keeps K read-only; E-only removes K and its anchors and retains the E allocation; Unified permits K edits while preserving K/E retrieval groups.
| Protocol/arm | Natural S3 Top-3 (%) | Coverage (%) |
|---|---|---|
| MSA Protected | 50.35 | 84.90 |
| MSA E-only | 39.63 | 42.55 |
| MSA Unified | 43.30 | 68.82 |
| HMT Protected | 48.37 | 76.50 |
| HMT E-only | 27.27 | 2.75 |
| HMT Unified | 40.86 | 69.75 |
Protected has higher natural-routing S3 Top-3 and coverage than these two alternatives in both protocols. This is a comparison of the implemented configurations, including their allocation and anchor conditions. It is reported separately from a pure Gate switch; stage-wise results and action counts are reported in Appendix E.6, Tables 17-19 (p. 35).
4. Decision-level verification. A fixed-candidate replay reproduces 400 recorded decisions and their pre-update states. Enabling the admission checks changes four WRITE_NEW decisions to HUMAN_REVIEW through internal-consistency checks. This demonstrates that the checks affect admission actions. We interpret the replay as operational evidence for the implemented checks, rather than certification of medical label correctness.
Manuscript changes. Section 3.3 now describes the executable write semantics. Section 4.3 and Appendix E.6 distinguish the historical configuration comparison from the two-protocol architecture controls, with their observed action counts and interpretation.
(3/4): HMT S2, S3 coverage, and Hospital/MIMIC results
W3. Is HMT S2 essentially always-tumor prediction, are S3 scores driven by coverage, and what happens under correct upstream routing and within each data source?
Response. We now report class-balanced behavior and coverage alongside the revised stage scores, and provide the requested source breakdown and correct-upstream analysis. These quantities address different parts of the concern and are kept distinct.
1. S2 class behavior. The previous HMT S2 value 50.06% is superseded by the valid-output mean 54.27 +/- 2.52%, conditional BA 56.42%, and coverage 70.78%. An always-tumor rule has BA 50% when both classes are represented. Figure 3 and the class-wise analysis report HMT non-tumor/tumor recall of 66.97/45.86% among valid outputs, with class coverage 55.90/85.65%. This describes the selected configuration more precisely than accuracy alone. The definitions and class-wise results appear in Section 4.4 and Appendix E.4, Table 16 (p. 34).
2. S3 scoring and correct upstream conditions. We pair Top-1/Top-3 with coverage and distinguish valid-output quality from success on the complete reference cohort. The earlier aggregate ranking is not treated as a same-patient stage comparison.
The correct-upstream memory diagnostic is reported separately in Appendix E.8, Table 22 (p. 37). It supplies disease eligibility for S2 and tumor eligibility for S3, without revealing the target tumor group. The reference S2/S3 populations are 600/510 on MSA and 800/400 on HMT. This diagnostic uses the current decoder and a fixed historical memory snapshot. The RAG/Memory comparison in Part 1 uses the revised valid-output result summaries; each analysis retains its stated protocol and denominator.
3. Memory results by source. Following the request, the source breakdown below contains the memory method alone. Scores are percentages among valid stage outputs from the fixed source-stratified memory snapshot. They are not a decomposition of the main-table five-order means.
| Protocol | Source | S1 Acc | S2 Acc | S3 Top-1 | S3 Top-3 |
|---|---|---|---|---|---|
| MSA | Hospital | 86.92 | 91.19 | 24.83 | 42.07 |
| MSA | MIMIC | 81.54 | 85.22 | 21.56 | 50.19 |
| HMT | Hospital | 85.00 | 72.73 | 35.00 | 46.25 |
| HMT | MIMIC | 71.50 | 58.97 | 21.83 | 55.95 |
S1 coverage is 100%. S2/S3 coverage is 75.71/81.01% for MSA Hospital, 81.54/81.27% for MSA MIMIC, 55.00/80.00% for HMT Hospital, and 71.50/84.00% for HMT MIMIC. The Hospital source has higher conditional S1/S2 accuracy, while MIMIC has higher conditional S3 Top-3 in both protocols. The source gap is therefore task-dependent.
All normal controls are Hospital-derived, and field availability differs between sources. We state these cohort properties explicitly and interpret the table as source-specific performance, not evidence that the source/class association has been removed. Diagnosis labels and explicit source identifiers are excluded from the retrieval query; missingness structure is treated as a separate input-property issue. The source analysis is documented in Appendix E.8.
Manuscript changes. The revised source analysis includes the natural-routing breakdown and coverage in Appendix E.8, Table 21 (p. 36), and correct-upstream results in Table 22 (p. 37). The class-balance and coverage discussion is separated from the method's memory-control contribution.
(1/4): Integer counts and the revised RAG/Memory comparison
W1. Why do the reported percentages imply fractional numbers of correct cases, and how should the 0.08-point difference from RAG be interpreted?
Response. The earlier summary combined inconsistent evaluation denominators. We have recompiled the results with explicit valid-output counts and run-wise aggregation; the old entries and the 0.08-point comparison are superseded. We do not round the old percentages into an inferred number of correct patients.
For each stage and run, N is the fixed reference-eligible population, A the number of valid stage outputs, C the number correct, and H the S3 Top-3 hits. Incorrect but valid answers remain in A. RAG and Memory are reported as C/A and H/A, accompanied by coverage A/N. Stage non-arrivals, explicit non-answers, and invalid outputs remain in N. F1 and balanced accuracy use the stated scoring population.
RAG is a single evaluation. Memory reports the equal-weight mean of five recorded learning-order scores, with sample SD: mean accuracy = (1/5) sum_r(C_r/A_r). Each run has integer counts, but the mean percentage need not correspond to an integer obtained by multiplying it by the full reference N. The repeated orders use the same test patients.
The revised comparison uses RAG that retrieves knowledge together with case examples, and Memory, our evolving-memory method after 700 updates. Both are scored on valid stage outputs. Values below follow the revised result summaries (%; +/- denotes between-order sample SD). Memory uses the confirmed 16K context, anchors ON, and Gate. The methods retain their recorded resource settings.
The integer bindings are:
The last column provides aggregate integer traceability, not the formula used for the displayed means. For example, HMT S3 has pooled accuracy 302/781 = 38.67%, while the equal-weight run mean is 40.88%. S3 Top-3 hit totals are 222/170 for RAG and 1076/460 across our five orders on MSA/HMT.
Our RAG calculation, 115/510 = 22.55%, is the full-population result. Its revised answered-case score is 115/439 = 26.20%. This change in denominator is explicit; it is not a new prediction gain. Likewise, the earlier values for our method, 22.63%, 50.06%, and 21.20%, are replaced, not reinterpreted as single-run counts.
S1/S2/S3 coverage is 96.79/81.67/86.08% for MSA RAG and 95.90/82.23/78.55% for MSA Memory; corresponding HMT values are 96.83/68.25/84.50% and 97.17/70.78/39.05%. We interpret the conditional rates with these answered subsets and do not infer statistical significance from the old sub-case margin.
Manuscript changes. The revised PDF defines these quantities in Section 4.1 (p. 6, lines 308-314), reports the revised scores in Table 1, and provides aggregation and integer summaries in Appendix D.3 (p. 31, Table 13), with endpoint SD in Appendix E.1 (p. 32, Table 14).