- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a learner-relative memory framework that selects reasoning examples based on a frozen LLM’s estimated capability and then retrieves memories through candidate recall, structured matching, and an LLM-based applicability gate. The goal is to use a smaller, more targeted memory bank while avoiding harmful or unusable retrieved information.
Critical strengths
-
The problem is interesting and practically relevant because useful memory is not necessarily equivalent to semantically relevant memory, and indiscriminate context injection can degrade LLM reasoning.
-
The separation between memory-bank construction and inference-time selection is sensible, and the counterfactual experiment showing that rejected memories can actually hurt performance provides useful evidence for selective memory injection.
Critical weaknesses
-
The overall performance improvement on the main reasoning benchmark is modest. On BBEH, PureMem improves accuracy from 72.75% without memory to only 74.78%, around 2% of improvement. It needs to quantify whether this improvement justifies the additional computational and system complexity.
-
The role of the “Full Memory” baseline needs clearer interpretation. The paper reports Full Memory as the highest-performing configuration and describes this as evidence that the complete bank is useful when capacity is unconstrained. However, this should not be interpreted as a general upper bound on memory-augmented reasoning. Whether retaining a full memory bank is beneficial depends strongly on the model’s context capability, retrieval strategy, memory-bank size and quality, task characteristics, and the amount of irrelevant or conflicting information. Indeed, the paper’s own motivation is that more context can sometimes hurt performance. The authors need to clarify that Full Memory is an empirical reference under this particular experimental protocol, not a generally optimal or upper-bound strategy.
Questions for authors
-
Given that PureMem improves BBEH accuracy by only 2% over the no-memory baseline, how do the authors justify the additional cost and complexity of capability probing, teacher-based memory generation, multi-stage retrieval, and LLM verification?
-
How robust is the claimed compactness–performance advantage across different context-window sizes, memory-bank scales, retrieval budgets, and models?