acceptodds
Under review as a conference paper at ICLR 2027

Learn to Remember: Geometric Memory for Inference-Time Self-Improvement in Language Models

Abstract

Large language models (LLMs) are used in a stateless way, discarding what they learn from one query to the next. Memory-augmented LLMs adapt at inference time, but existing systems either inject the entire memory into every prompt, so cost grows with it, or retrieve by similarity under a fixed encoder that never learns whether an entry helped. We propose Learn to Remember (LeRe), in which a frozen LLM cycles through plan, solve, and curate roles and learns which memory to retrieve rather than how much to store. A Planner rewrites each query into a structured retrieval key; the Contrastive Contextual Memory Encoder (CCME), lightweight heads on a frozen encoder, is trained online from the Curator's attribution of which entries helped, so retrieval reflects measured utility; and a Guard–Consolidate–Maintain (GCM) gate rejects answer leakage, merges near-duplicates, and quarantines unreliable entries. LeRe learns without ground-truth labels or weight updates. Across twelve mathematical, scientific, and multimodal streams and three backbones, LeRe improves average accuracy over a no-memory baseline by 6.9%, 9.0%, and 4.6%, exceeding the strongest memory-based baseline on every backbone and gaining up to 24.7% on competition mathematics, while its per-query token cost stays flat as memory grows, at near-lowest cost. These consistent gains suggest that selecting proven memory, rather than accumulating more, offers a scalable, efficient, and effective way to support inference-time adaptation in LLMs. Our code is available in an anonymous repository at https://anonymous.4open.science/r/learn-to-remember-AD37.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, inference-time adaptation, contrastive representation learning, utility-aware retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_LkDFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learn to Remember (LeRe), a memory-augmented framework in which a frozen LLM cycles through Planner, Solver, and Curator roles and learns which memory entries to retrieve instead of accumulating more memory. Its core component, the Contrastive Contextual Memory Encoder (CCME), trains lightweight heads on a frozen encoder online, using the Curator's attribution of which entries actually helped as the training signal. A Guard–Consolidate–Maintain (GCM) gate filters answer leakage, merges near-duplicates, and quarantines unreliable entries.

Critical strengths
  1. The question is clear, the limitations indeed exist in related work.
  2. The methods have great transferabilities, can be used with different LLMs
  3. Comprehensive experiments on 12 streams and 3 backbones
Critical weaknesses
  1. Can the author add one ablation variant that uses the solver (without memories)? I'd like to see how much of the performance will drop under such circumstances.
  2. Many results drop on MATH and MMLU Eng. The gain is not stable across tasks. Even compared with other baselines, LeRe is weaker on these two datasets.
  3. In the related work section, the authors talk about many related methods, but very few of them are compared to LeRe in the experimental section.
  4. More hyperparameter analysis needs to be added, as there are so many hyperparameters in Table 4.

minor:

  1. Abstract can be revised, some words, for example, entry, do not have a clear meaning under such a semantic background.
Questions for authors

Please refer to the weakness section

3 comments
AuthorsOpenReview ↗

Response to Reviewer LkDF

Dear Reviewer LkDF,

Thank you for your careful evaluation of our work and your constructive suggestions for improving the manuscript's clarity. We have addressed your comments in the revised manuscript, with all corresponding changes highlighted in red. Below, we respond to each of your comments and questions point by point:

Performance of the ablation variant without memories

The ablation results, including the baseline that isolates the contribution of memory, are presented below:

Table B: Ablation accuracy comparison with newly added memory isolating baseline

MethodGemini AIMEGemini Std-10Gemini HLEGPT-4.1-mini AIMEGPT-4.1-mini VisionGPT-4.1-mini MMLU Eng.GPT-4o-mini AIMEGPT-4o-mini Std-4GPT-4o-mini GPQA
w/o Planner65.4 (−4.4)70.0 (−6.8)10.8 (−1.6)38.9 (−6.2)58.0 (−1.6)59.6 (−4.4)19.1 (−2.5)52.8 (−2.4)36.4 (−4.5)
w/o CCME66.7 (−3.1)74.0 (−2.8)10.4 (−2.0)38.3 (−6.8)56.4 (−3.2)61.6 (−2.4)20.8 (−0.8)53.6 (−1.6)34.8 (−6.1)
w/o Execution43.8 (−26.0)72.4 (−4.4)9.2 (−3.2)31.5 (−13.6)52.4 (−7.2)58.8 (−5.2)4.9 (−16.7)49.2 (−6.0)37.9 (−3.0)
w/o Memories62.4 (−7.4)75.6 (−1.2)8.4 (−4.0)33.3 (−11.8)57.2 (−2.4)58.0 (−6.0)19.1 (−2.5)52.0 (−3.2)36.4 (−4.5)
LeRe69.876.812.445.159.664.021.655.240.9

As shown in Table B, disabling cross-query memory consistently reduces accuracy across all backbones and benchmarks, with an average drop of 4.8 percentage points and a maximum drop of 11.8 points on AIME. These results show that cross-query memory provides substantial additional gains beyond those achieved by the Solver and code execution alone.

LeRe’s performance on specific datasets

Thank you for this observation. We agree that LeRe does not improve performance on every stream, and we recognize this as a limitation of the current framework. Our results suggest two possible explanations:

  • Verification bottleneck: On GPT-4o-mini, the Curator accepts only 29% of correct answers (Table 7, ζ1=0.29\zeta_1 = 0.29), limiting the number of useful experiences retained in memory. In contrast, on GPT-4.1-mini, where verification is more reliable, LeRe outperforms all baselines on both streams (MATH +4.4 and Engineering +6.4).
  • Limited room for improvement: On Gemini-3.1-flash-lite, the baseline accuracy on MATH is already 94.0%, leaving little room for further gains from cross-query memory.

Despite these limitations, LeRe achieves the highest average improvement across all 12 streams on each backbone (+6.9, +9.0, and +4.6 percentage points, compared with +3.7, +2.4, and +3.2 for the strongest competing method, DC RS). The improvements are particularly clear on challenging reasoning tasks, where LeRe outperforms the baselines in all 9 AIME settings and every HLE-Exact evaluation.

These results suggest that LeRe is effective for complex reasoning, although its performance depends in part on the backbone's verification capability and the difficulty of the task. Improving self-verification to make memory updates more reliable, particularly on weaker backbones, is an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer LkDF

Baseline selection criteria clarification

Thank you for raising this question. We appreciate the opportunity to clarify our baseline selection. Our evaluation focuses on frozen, API-served models that adapt online to a single-pass stream of reasoning problems, without access to ground-truth labels or token log-probabilities. Dynamic Cheatsheet (both variants) and ACE are directly applicable to this setting, while RAG-ICL provides a standard retrieval-based baseline.

Several other methods discussed in Section 2 are closely related but operate under different assumptions:

  1. Weight-based test-time adaptation (Wang et al., 2021; Niu et al., 2022; Zuo et al., 2025) requires gradient access to update model parameters, which is unavailable for the API-served models considered in our experiments.
  2. Learned retrievers (EPR, LLM-R, REPLUG) typically rely on offline training using gold outputs or model likelihoods. These requirements are not compatible with our setting, where ground-truth labels and token likelihoods are unavailable.
  3. Interactive agent memory methods (e.g., Reflexion and workflow memory) generally involve repeated attempts at the same task or interactions within multi-turn environments. Our evaluation instead considers independent reasoning problems processed in a single pass (Pass@1).
  4. Planning and query-reformulation methods (ReAct, Plan-and-Solve, HyDE) focus on improving reasoning or retrieval rather than maintaining and updating cross-query memory. LeRe incorporates planning through its Planner, whose contribution is separately evaluated in the w/o Planner ablation (Table 2).

We agree that a clearer comparison with these methods would help readers understand our baseline selection. In the revised manuscript, we will add an appendix table summarizing the related methods, their key requirements, and how they differ from our experimental setting.

Hyperparameter analysis

Thank you for this helpful suggestion. Our current evaluation focuses on LeRe's overall effectiveness, including accuracy comparisons across backbones, cost-efficiency analysis, component ablations, and analyses of the memory mechanism, with additional experiments provided in the Appendix.

For hyperparameter sensitivity, we evaluated the number of retrieved entries K∈{1,3,5,10}K \in \{1, 3, 5, 10\} (Appendix C.3.1, Table 8). The results show that accuracy varies by at most 3.2 percentage points across these settings. All other hyperparameters listed in Table 4, including retrieval weights, consolidation thresholds, quarantine rules, and the CCME training schedule, were fixed in advance and kept consistent across all backbones and task streams, without task-specific tuning.

We agree that a more comprehensive sensitivity analysis would provide further insight into LeRe's behavior. We have acknowledged this limitation in the revised manuscript and identified a systematic study of these hyperparameters as a direction for future work.

Minor Comment: Abstract Clarity

Thank you for pointing out this wording issue. We have revised the Abstract (with changes highlighted in red) to improve clarity, replacing ambiguous terms such as "entry" with more specific descriptions, including "retrieved reasoning strategies" and "memory items," where appropriate.

Thank you for your response. After reading the full rebuttal, I find that some of my concerns have been addressed. I am inclined to keep my current score.

Reviewer_W7BuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LeRe to learn how to retrieve external memories online. A Planner turns each question into a retrieval key, a Solver answers using the top-3 memories plus up to three code runs, and a Curator checks the answer, labels which memories helped, and writes new ones. Those labels train two small projection heads (CCME) online, so retrieval shifts toward entries that helped before. Across 12 streams and 3 backbones, it beats DC RS on average while per-query tokens stay flat.

Critical strengths
  1. The method leaves the LLM untouched and only trains two small projection heads for retrieval. Since it pulls in just a few memories per query, the prompt doesn't keep growing as memory accumulates, which seems practical for real use.

  2. The comparison with DC RS is fair on tools, since both get the same code-execution budget, and LeRe still comes out ahead. Removing CCME also lowers accuracy, so the learned retriever seems to be helping, not just the overall pipeline.

Critical weaknesses
  1. I'm not convinced the method is really label-free. When filtering new memories, the guard checks whether a proposal contains the reference answer, so gold answers still affect what gets stored. Also, the Curator's verdicts drive both memory updates and CCME training, and they are fairly noisy, with the Curator accepting more than half of the wrong answers on two backbones.

  2. It remains unclear how much of the gain actually comes from memory. All results are single runs, and the lead over DC RS on some backbones is small enough to be noise. The baseline also can't run code, and dropping execution hurts far more than dropping CCME. A run with the same tools but no memory would help.

Questions for authors
  1. If the guard never sees the gold answer, keeping only the n-gram and answer-phrasing checks, do the results change?

  2. Could you run the same pipeline with the same tools but memory turned off or kept empty? That would show how much of the gain comes from memory rather than from the Planner and code execution.

  3. Section 4.3 denotes CCME wasn't updated on GPT-4o-mini/AIME 2020–2025, but Table 6 lists that run as updated. Which is right?

2 comments
AuthorsOpenReview ↗

Response to Reviewer W7Bu

Dear Reviewer W7Bu,

we thank you for your thoughtful evaluation and constructive comment of LeRe. Below, we provide point-by-point responses to each concern (C) and question (Q). All updates in the revised text are highlighted in red font.

Label-free claim clarification & curator precision justification

Answer to Q1: Results do not meaningfully change when reference access is removed.

As noted in Remark A.15 (Appendix A.8), the Guard is the only component in LeRe that accesses reference answers. We have revised the manuscript to clarify the scope of our "label-free" claim. To assess the impact of this access, we examined all 4,115 proposals screened by the Guard across the 36 runs reported in Table 1:

  • 0.53% intervention rate: The gold check affected the screening outcome for only 22 of the 4,115 proposals (0.53%). All 22 cases involved spurious token matches (e.g., (F) referring to physical force rather than an answer label), and none revealed the actual target answer.
  • Ablation with reference-answer access removed: These 22 cases occurred in only 6 runs: MATH, MMLU-Pro (Engineering and Physics), and MMMU-Pro Std-10 on Gemini-3.1-flash-lite; and AIME 2020–2025 and MMLU-Pro Engineering on GPT-4o-mini. We reran these 6 experiments with reference-answer access completely removed, retaining only the n-gram and phrasing checks:

Table A: Re-evaluation on streams where gold check intervention triggered

SettingGemini MATHGemini MMLU Eng.Gemini MMLU Phys.Gemini Std-10GPT-4o-mini AIMEGPT-4o-mini MMLU Eng.Avg.
Access removed93.680.090.076.022.833.265.9
Table 194.078.090.076.821.631.665.3
Difference−0.4+2.00.0−0.8+1.2+1.6+0.6

Removing reference-answer access yields results statistically indistinguishable from those in Table A, confirming that LeRe's performance gains do not depend on this access.

Answer to C1: We agree that the Curator's signal is noisy. As shown in Table 7, it accepts more than half of the incorrect proposals on Gemini-3.1-flash-lite and GPT-4.1-mini (ζ0=0.53\zeta_0 = 0.53 and 0.590.59, respectively). We recognize that reliable automated verification remains an open challenge, particularly for self-improvement frameworks. Nevertheless, our experimental results suggest that the Curator still provides a useful learning signal:

  • Higher acceptance of correct proposals: Across all backbones, the Curator consistently accepts correct proposals more often than incorrect ones (ζ1>ζ0\zeta_1 > \zeta_0), satisfying the condition in Proposition A.9 for preserving the ordering of memory reliability by true helpfulness.
  • Significant correlation with correctness: Curator acceptance is significantly correlated with actual correctness in 26 of the 36 evaluated configurations.

Despite this noise, LeRe achieves consistent performance improvements across diverse benchmarks (Table 1), suggesting that the framework can benefit from imperfect verification signals without requiring a fully reliable verifier.

AuthorsOpenReview ↗

Response to Reviewer W7Bu

Additional baseline with memory turned off

Answer to Q2 & C2: Thank you for this constructive suggestion. To address this point, we evaluated the recommended w/o Memories baseline across all 9 benchmark settings. This baseline uses the same Solver, prompts, tools, and code-execution budget as full LeRe, but disables all cross-query memory reads and writes. The accuracy results are shown below:

Table B: Ablation accuracy comparison with newly added memory isolating baseline

MethodGemini AIMEGemini Std-10Gemini HLEGPT-4.1-mini AIMEGPT-4.1-mini VisionGPT-4.1-mini MMLU Eng.GPT-4o-mini AIMEGPT-4o-mini Std-4GPT-4o-mini GPQA
w/o Planner65.4 (−4.4)70.0 (−6.8)10.8 (−1.6)38.9 (−6.2)58.0 (−1.6)59.6 (−4.4)19.1 (−2.5)52.8 (−2.4)36.4 (−4.5)
w/o CCME66.7 (−3.1)74.0 (−2.8)10.4 (−2.0)38.3 (−6.8)56.4 (−3.2)61.6 (−2.4)20.8 (−0.8)53.6 (−1.6)34.8 (−6.1)
w/o Execution43.8 (−26.0)72.4 (−4.4)9.2 (−3.2)31.5 (−13.6)52.4 (−7.2)58.8 (−5.2)4.9 (−16.7)49.2 (−6.0)37.9 (−3.0)
w/o Memories62.4 (−7.4)75.6 (−1.2)8.4 (−4.0)33.3 (−11.8)57.2 (−2.4)58.0 (−6.0)19.1 (−2.5)52.0 (−3.2)36.4 (−4.5)
LeRe69.876.812.445.159.664.021.655.240.9

As shown in Table B, removing cross-query memory consistently reduces performance across all models and benchmarks, demonstrating that LeRe's improvements are not solely due to planning or code execution, but also benefit substantially from the memory mechanism.

Clarification of CCME Update Status on GPT-4o-mini / AIME 2020–2025

Answer to Q3: We thank the reviewer for pointing out this inconsistency. The statement in Section 4.3 is correct: the projection heads were not updated in the standard GPT-4o-mini AIME run (K=3K=3), as only 2 CCME contrastive pairs were collected, fewer than the 4 required to trigger an update (Table 4). The discrepancy arose because Table 6 also included results from the K=10K=10 sensitivity run (Table 8), in which 37 updates occurred.

We have corrected Table 6 to include only the standard Table 1 runs and recomputed the pooled metrics accordingly. This correction has little effect on GPT-4o-mini's pooled geometric results: the number of anchors changes from 199 to 189, Δmargin\Delta\text{margin} from +0.39+0.39 to +0.38+0.38, and the helpful/harmful shifts from +0.07/−0.32+0.07/-0.32 to +0.07/−0.30+0.07/-0.30. The geometric separation pattern and our main conclusions remain unchanged.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer nZqZ

Dear Reviewer nZqZ,

We thank you for your constructive feedback and for recognizing the strengths of our work. All corresponding changes in the revised manuscript are highlighted in red. Below, we address each of your questions and concerns.

Clarification on "label-free" claim and Guard access

We thank you for carefully examining our framework. As noted in Remark A.15 (Appendix A.8), the Guard is the only component in LeRe that uses reference answers. We have revised the main text to make this distinction clearer and specify the scope of our label-free claim.

Answer to Q1-1: Yes, the Guard had access to reference answers during the Table 1 experiments.

Answer to Q1-2: This access has a negligible impact on the results, and all main conclusions remain unchanged without it. To verify this, we examined all 4,115 proposals screened by the Guard across the 36 runs reported in Table 1:

  • 0.53% intervention rate: The gold check affected the screening outcome for only 22 of the 4,115 proposals (0.53%). All 22 cases involved spurious token matches (e.g., (F) referring to force rather than an answer label), and none revealed the actual target answer.

  • Invariant runs: In 30 of the 36 runs, the gold check intervened zero times, meaning memory writes in those runs were completely untouched. The 22 false triggers were confined to 6 runs: MATH, MMLU-Pro (Engineering & Physics), and MMMU-Pro Std-10 on Gemini-3.1-flash-lite; and AIME 2020–2025 and MMLU-Pro Engineering on GPT-4o-mini. We reran these 6 experiments with reference-answer access completely removed:

Table A: Re-evaluation on streams where gold check intervention triggered

SettingGemini MATHGemini MMLU Eng.Gemini MMLU Phys.Gemini Std-10GPT-4o-mini AIMEGPT-4o-mini MMLU Eng.Avg.
Access removed93.680.090.076.022.833.265.9
Table 194.078.090.076.821.631.665.3
Difference−0.4+2.00.0−0.8+1.2+1.6+0.6

As shown in the Table A, removing ground-truth access causes no statistically meaningful shift in performance, confirming that LeRe’s gains do not rely on such access.

AuthorsOpenReview ↗

Response to Reviewer nZqZ

Additional baseline without memories

We agree that this is an important control. Per your suggestion, we added the w/o Memories ablation baseline across all 9 benchmark settings. This baseline uses the identical Solver, prompts, and code-execution budget as full LeRe, but disables cross-query memory read and write entirely.

Table B: Ablation accuracy comparison with newly added memory isolating baseline

MethodGemini AIMEGemini Std-10Gemini HLEGPT-4.1-mini AIMEGPT-4.1-mini VisionGPT-4.1-mini MMLU Eng.GPT-4o-mini AIMEGPT-4o-mini Std-4GPT-4o-mini GPQA
w/o Planner65.4 (−4.4)70.0 (−6.8)10.8 (−1.6)38.9 (−6.2)58.0 (−1.6)59.6 (−4.4)19.1 (−2.5)52.8 (−2.4)36.4 (−4.5)
w/o CCME66.7 (−3.1)74.0 (−2.8)10.4 (−2.0)38.3 (−6.8)56.4 (−3.2)61.6 (−2.4)20.8 (−0.8)53.6 (−1.6)34.8 (−6.1)
w/o Execution43.8 (−26.0)72.4 (−4.4)9.2 (−3.2)31.5 (−13.6)52.4 (−7.2)58.8 (−5.2)4.9 (−16.7)49.2 (−6.0)37.9 (−3.0)
w/o Memories62.4 (−7.4)75.6 (−1.2)8.4 (−4.0)33.3 (−11.8)57.2 (−2.4)58.0 (−6.0)19.1 (−2.5)52.0 (−3.2)36.4 (−4.5)
LeRe69.876.812.445.159.664.021.655.240.9

Table C: Cost comparison between newly added memory isolating baseline and full LeRe

Gemini AIMEGemini Std-10Gemini HLEGPT-4.1-mini AIMEGPT-4.1-mini VisionGPT-4.1-mini MMLU Eng.GPT-4o-mini AIMEGPT-4o-mini Std-4GPT-4o-mini GPQATotal
w/o Memory$0.75$0.74$0.70$0.89$1.40$0.90$0.32$1.99$0.27$7.96
LeRe (Table 1 run)$1.18$1.33$1.62$1.50$2.14$1.84$0.45$2.77$0.43$13.26
w/o Memory as % of LeRe64%56%43%59%65%49%71%72%63%60%

Answer to Q2: Table B shows that removing cross-query memory consistently reduces performance across all models and benchmarks, confirming that LeRe's improvements cannot be attributed to code execution alone. Table C shows that disabling memory also reduces token usage by eliminating memory retrieval and update overhead. We have revised the ablation in Section 4.3 analysis to distinguish the contributions of code execution and cross-query memory.

AuthorsOpenReview ↗

Response to Reviewer nZqZ

Results of GPT-4o-mini on AIME 2020–2025 clarification

We thank the reviewer for pointing out this inconsistency. The two statements refer to different runs, and we have revised the text to clarify this distinction.

Answer to Q3:
In the standard Table 1 and 2 runs (K=3K=3), GPT-4o-mini collected only 2 CCME pairs on AIME, fewer than the 4 required to trigger an update (Table 4). Thus, the statement in the main text that these heads were not updated is correct. The discrepancy arose because the K=10K=10 run from our sensitivity analysis (Table 8), which recorded 37 updates, was also included in Table 6. We have corrected Table 6 by removing this run and recomputing all pooled metrics using only the standard Table 1 runs. This correction has little effect on GPT-4o-mini's pooled results: the number of anchors changes from 199 to 189, Δmargin\Delta\text{margin} from +0.39+0.39 to +0.38+0.38, and the helpful/harmful shifts from +0.07/−0.32+0.07/-0.32 to +0.07/−0.30+0.07/-0.30. The geometric separation pattern and our main conclusions remain unchanged.

AuthorsOpenReview ↗

Summary of revisions for the reviewers and AC

We thank all reviewers for their valuable feedback and constructive suggestions. We have revised our manuscript accordingly and uploaded the latest version, with all modifications highlighted in red font. Below is a summary of our revision details:

Label use & CCME update consistency (Reviewers nZqZ, Reviewer W7Bu)

  1. Remark A.15 (L986–991): states that reference answers never enter the Curator's verdict or the credit, reliability, or CCME updates, that they are read only by the Guard's leakage filter, which can remove but never add content, and defines this regime as learning without label supervision.
  2. "Without ground-truth labels" replaced by "without label supervision" in the Abstract (L024), Introduction (L077), Figure 2 caption (L123), Conclusion (L480–481), and Limitations (L2009).
  3. Baseline descriptions clarified so that no method, including LeRe, receives correctness feedback from reference answers: §4.1 (L320–322) and App. B.2 (L1056, L1128–1129).
  4. App. C.2.2 (L1347–1349): corrected the statement on ground truth; it is used for the verdict-agreement analysis and, within the loop, only by the leakage filter (Remark A.15).
  5. Table 6: removed the GPT-4o-mini AIME 2020–2025 entry, and recomputed the pooled GPT-4o-mini row (189 anchors, margin +0.38).
  6. App. C.2.1 (L1313–1318): updated the text accordingly (29 model–stream pairs; significant on 26; pooled shifts and p-value bound).

Abstract Revision: Replacing Ambiguous Terms (Reviewer LkDF)

Ambiguous terms such as "entry" is replaced with more specific descriptions in Abstract (L016-L023).

Ablation (all reviewers)

Table 2 and §4.3 (L449–472): Added the w/o Memories ablation, which disables cross-query memory while retaining the Planner, Solver, and code execution, along with the corresponding accuracy results and analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.