acceptodds
Withdrawn from ICLR 2027

PureMem: Learner-Relative Memory Curation for Reliable Retrieval-Augmented Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-augmented language models commonly assume that relevant and correct context is useful. This assumption can fail when retrieved memory is redundant, difficult for the target model to exploit, or induces an incompatible reasoning strategy. We introduce PureMem, a learner-relative memory framework that asks not only what is relevant, but what a frozen language model can reliably use. PureMem constructs a compact reasoning memory bank from examples in the model's empirically estimated zone of proximal development (ZPD), and retrieves memory through three stages: candidate recall, structured procedure matching, and LLM-based applicability verification. A conservative injection policy falls back to the base model when no candidate is judged applicable. On BBEH, PureMem improves accuracy from 72.75% to 74.78% while retaining only 26.4% of the full memory bank. Counterfactual placebo interventions show that the verifier separates relatively beneficial memory from harmful memory: injected memory improves accepted cases but significantly degrades rejected cases. On RealMemBench, the three-stage retrieval harness improves NDCG@10 by 0.097 over the strongest executed embedding baseline and also improves downstream answer quality. These results distinguish a model-specific memory-curation procedure from a reusable retrieval architecture, while showing that injection policies must be adapted to the target memory domain. Code and data are provided as anonymised supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, External Memory, Language-Model Reasoning, Learner-Relative Curation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cA7TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a learner-relative memory framework that selects reasoning examples based on a frozen LLM’s estimated capability and then retrieves memories through candidate recall, structured matching, and an LLM-based applicability gate. The goal is to use a smaller, more targeted memory bank while avoiding harmful or unusable retrieved information.

Critical strengths
  • The problem is interesting and practically relevant because useful memory is not necessarily equivalent to semantically relevant memory, and indiscriminate context injection can degrade LLM reasoning.

  • The separation between memory-bank construction and inference-time selection is sensible, and the counterfactual experiment showing that rejected memories can actually hurt performance provides useful evidence for selective memory injection.

Critical weaknesses
  • The overall performance improvement on the main reasoning benchmark is modest. On BBEH, PureMem improves accuracy from 72.75% without memory to only 74.78%, around 2% of improvement. It needs to quantify whether this improvement justifies the additional computational and system complexity.

  • The role of the “Full Memory” baseline needs clearer interpretation. The paper reports Full Memory as the highest-performing configuration and describes this as evidence that the complete bank is useful when capacity is unconstrained. However, this should not be interpreted as a general upper bound on memory-augmented reasoning. Whether retaining a full memory bank is beneficial depends strongly on the model’s context capability, retrieval strategy, memory-bank size and quality, task characteristics, and the amount of irrelevant or conflicting information. Indeed, the paper’s own motivation is that more context can sometimes hurt performance. The authors need to clarify that Full Memory is an empirical reference under this particular experimental protocol, not a generally optimal or upper-bound strategy.

Questions for authors
  • Given that PureMem improves BBEH accuracy by only 2% over the no-memory baseline, how do the authors justify the additional cost and complexity of capability probing, teacher-based memory generation, multi-stage retrieval, and LLM verification?

  • How robust is the claimed compactness–performance advantage across different context-window sizes, memory-bank scales, retrieval budgets, and models?

Reviewer_a23gOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript proposes PureMem, a learner-relative memory framework for retrieval-augmented reasoning. It selects source examples with intermediate success rates under a frozen language model, converts teacher-generated solution traces into concrete precedents and procedural schemas, and uses a three-stage retrieval pipeline with an applicability verifier to control memory injection. The manuscript is clearly written and addresses a relevant problem. However, the central source-curation hypothesis is not sufficiently validated, and the experiments do not convincingly establish the incremental value or cost-effectiveness of the proposed pipeline.

Critical strengths

The prompt-matched controls and counterfactual placebo analysis are useful. In particular, the results provide evidence that the applicability gate can reduce harmful memory injection.

Critical weaknesses

(1) The central hypothesis connecting source difficulty to memory utility is not sufficiently established. A model’s ability to solve a source problem without assistance does not necessarily imply that the procedural steps extracted from that problem offer little benefit for another query. Similarly, failing to solve a source problem independently does not establish that its solution cannot be exploited with appropriate support. The intermediate no-memory success rate is therefore an insufficiently validated proxy for useful memory sources. Importantly, the conclusion acknowledges that mastered-only and out-of-reach-only controls have not been conducted. Comparisons among easy-, intermediate-, and hard-source memory banks under matched task composition and memory budgets are needed to directly test the proposed hypothesis.

(2) The comparisons with relevant existing methods are insufficient. The BBEH experiments mainly compare memory-bank variants, while the RealMemBench experiments primarily compare embedding baselines and variants of the proposed pipeline. These controls do not establish an advantage over existing demonstration-selection or utility-aware retrieval approaches. The manuscript also cites Cui and Sachan (2025), which already investigates ZPD for in-context learning. Although source-level memory construction is a different setting, its incremental contribution over related learner-aware selection ideas requires clearer justification. A standard LLM reranker or applicability filter using the same candidate pool and computational budget would also be an important baseline.

(3) The computational cost of Stage III is not adequately evaluated in the main paper. The verifier performs repeated LLM evaluations for each shortlisted candidate, and this overhead remains even when all candidates are rejected. Offline student probing and teacher-based memory construction incur additional costs. Reducing the number of stored chunks therefore does not by itself establish end-to-end efficiency. This is particularly important because, under dense retrieval, enabling the gate improves accuracy only from 74.35% to 74.78% in Table 1. Latency, token usage, and comparisons against alternatives with matched inference budgets are needed to assess whether this gain justifies the additional computation.

(4) The experimental results leave the necessity and robustness of the proposed components unclear. In Table 1, random retrieval with the gate achieves 75.65%, exceeding dense retrieval with the gate at 74.78%. This raises questions about the contribution of the carefully designed retrieval pipeline beyond the verifier. Moreover, ZPD curation improves over the task-stratified, size-matched bank by only 0.87 percentage points, without reported paired significance or variability across memory-sampling seeds in the main text. Further component ablations and stability analyses are needed. RealMemBench does not apply ZPD curation, so its results cannot resolve the limited validation of the central source-selection principle.

Questions for authors

same as weaknesses

Reviewer_TyQcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PureMem asks which external reasoning memories a frozen language model can use. It selects source problems with intermediate empirical success rates, distils answer-consistent teacher traces into linked concrete precedents and procedural schemas, and combines dual-view recall, structured matching and applicability verification. Rejected candidates trigger a prompt-matched no-memory fallback. BBEH evaluates curation and injection control: accuracy increases from 72.75% to 74.78% while retaining 955 of 3,616 chunks. RealMem evaluates the retrieval harness over historical conversations, without ZPD construction; its conservative reasoning gate does not transfer. The paper positions curation as a compactness-performance trade-off, not superiority over full memory.

Critical strengths
  1. Informative negative-transfer test. Freezing the retrieved Top-1 and gate decision, the real-versus-placebo intervention separates accepted (+2.81 percentage points, n=178) from rejected (-3.23 points, n=433) cases, with a +6.04-point interaction (95% CI [1.82,10.47], reported permutation p=.0065; section 5.2, p. 7; Figure 3, p. 8). The authors appropriately disclose that accepted benefit alone is nonsignificant (p=.227), supporting a negative-transfer filter rather than guaranteed positive utility. Prompt-matched fallback also addresses a meaningful comparison confound (section 3.5, p. 6).
  2. Useful evidence with explicit boundaries. The compact bank outperforms no memory and both size-matched controls (Figure 2, p. 7). The independent RealMem embedding comparison improves NDCG@10 from .586 to .683, with higher held-out QA marginals (section 5.3, pp. 8-9; Tables 2-3). The authors acknowledge that full memory is more accurate, random retrieval with gating exceeds dense retrieval with gating, and conversational injection needs a different policy. These qualifications make the findings more informative.
Critical weaknesses
  1. Missing essential experimental specifications. The Reproducibility Statement (p. 10, lines 505–517) assigns key details—including the model checkpoint, data splits, ZPD probe settings, retrieval and gating parameters, metric definitions, and placebo and statistical-testing protocols—to Appendix A.1–A.9. However, these sections are absent from the available submission materials. Consequently, the method cannot be fully instantiated, and consequential experimental choices underlying the reported comparisons and significance claims cannot be adequately assessed. These omissions substantially limit confidence in the central empirical conclusions.
  2. Key scientific specification is unavailable. The complete supplied PDF ends at references on p. 13, while the Reproducibility Statement (p. 10, lines 505-517) assigns the model checkpoint, source split, probe count/band, retrieval/gate settings, metric definitions and paired-test details to absent Appendix A.1-A.9. Thus even the learner-relative band cannot be instantiated from the provided material. This may be an assembly omission and does not imply false experiments. It nevertheless prevents checking consequential design choices. Related unresolved reporting includes stage2_only/rerank Recall@20 of .954/.956 despite the stated depth-20 invariance (Table 2; p. 9, lines 451-455), and a reported paired QA gain of .117 versus marginal subtraction .120 (p. 8; Table 3). Different subsets or weighting could explain the latter.
  3. The distinctive curation benefit is weakly identified. PureMem exceeds the task-stratified matched bank by .87 points, equivalent to six additional correct answers among 690, but the supplied PDF gives no paired uncertainty or bank-sampling variation for this comparison (sections 5.1-5.2; Figure 2, p. 7). Size and task-stratified controls are valuable, yet do not distinguish a robust intermediate-band effect from a favorable bank realization. The conclusion itself says mastered-only/out-of-reach-only controls remain unrun (p. 9, lines 478-484). I therefore credit a promising compact bank, while finding the additional-signal claim stronger than its current support; I am not requiring universal band optimality.
Questions for authors
  1. Could you provide access to Appendix A.1–A.9 cited in the Reproducibility Statement? These sections are not included in the supplied PDF and contain essential experimental specifications needed for assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.