acceptodds
Under review as a conference paper at ICLR 2027

Continual Self-Improvement with Lightweight Experiential Latent Memories

Abstract

Large language models achieve strong reasoning performance by scaling inference-time compute, yet remain fundamentally stateless, discarding the rich, self-produced reasoning traces generated during this process. We investigate whether models can instead learn online from this experience, converting transient computation (reasoning traces) into persistent reusable knowledge, and without external supervision or access to future data. We show that In-Context Learning (ICL) over raw reasoning traces fails to generalize, reflecting a fundamental limitation of token-level reuse: individual traces lack the abstraction needed for transfer, even after refinement (e.g. self-reflection). In contrast, drawing inspiration from recent works on unsupervised reinforcement learning, we find that lightweight per-instance training with self-generated test-time signals as rewards could yield substantial gains, often surpassing full-dataset offline training, motivating a shift from raw traces to learned latent representations. Building on this insight, we propose an online method that distills inference-time compute spent on encountered problems into compact modular latent memories capturing the underlying reasoning structure. These memories are stored and retrieved for future inputs, enabling continual improvement while avoiding catastrophic forgetting through modular design. Importantly, our method is lightweight, parametrized as extremely small and lightly trained soft prompt memories (0.001% of model parameters), yet achieving performance competitive with full parametric updates and offline training. Across challenging reasoning benchmarks, our approach significantly outperforms textual ICL methods, enabling reliable self improvement from experience.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
self-improvement, memory, continual learning, learning from experience, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UidHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ELM, which keeps a language model frozen and trains a separate soft prompt for each encountered problem. After answering a problem, it trains the prompt with GRPO using rewards based on agreement with the majority answer among sampled solutions, then stores it for later use. For subsequent problems, the system retrieves a stored prompt and generates answers both with and without it. If the answers differ, a process reward model selects the final response.

Critical strengths
  1. Practical research question. The paper studies how to reuse experience from past problems to help a model solve future ones, which is relevant to models used over time.

  2. Evidence of transfer across problems. The experiments show that some prompts trained on a single problem can help solve other problems, with further evidence from transfer between datasets.

  3. Useful analysis of the results. The paper examines how memories help or hurt performance and how different answer selection methods affect the results. These analyses help explain both the gains and the failure cases.

Critical weaknesses
  1. Unclear value of the extra training. ELM uses majority-vote answers to train its memories, but its main results on Qwen2.5-Math-7B are worse than simply using majority voting on all four benchmarks. This raises doubts about whether the extra memory training and retrieval are worth the effort.

  2. Missing basic comparisons. The paper does not compare its method with randomly choosing a memory, always using the same memory, or using an untrained prompt. Without these comparisons, it is unclear whether creating a separate memory for each question and retrieving a relevant one are actually necessary.

  3. Unclear advantage over textual memories. In Table 3, ELM uses a verifier to choose between two answers, but the textual memory baselines do not. Appendix E.4 reports that adding the same selection step allows textual memories to outperform ELM on Llama-3.1-8B-Instruct on MATH500 (52.9 vs. 50.9) and GPQA (27.4 vs. 19.7). The claimed advantage of latent memories is therefore less convincing when the methods use the same answer selection process.

  4. Dependence on a separate verifier. For Qwen2.5-Math-1.5B, the input-retrieval version performs worse than the original model on both AMC23 and AIME24 without the verifier (Table 12). Asking the model to judge its own answers also fails to reproduce the gains obtained with the separate 7B PRM, leaving limited evidence that the model can improve reliably on its own.

  5. Inconsistent numbers in Table 4. The zero-shot averages in Table 4 are 12.9 for Llama and 31.7 for Qwen, but averaging the AMC23 and AIME24 results in Tables 3 and 12 gives 11.65 and 30.0. These numbers do not match, which also affects the reported 3.8-point improvement without routing.

Questions for authors

See the weaknesses above.

Reviewer_osUiOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Experiential Latent Memory (ELM), an online method that converts experience from previously encountered problems into lightweight soft-prompt memories. These memories are trained using self-generated rewards, retrieved for future inputs, and combined with response routing. Experiments on three models and four reasoning benchmarks show that ELM generally outperforms textual memory methods and can approach or exceed offline training while keeping the base model frozen.

Critical strengths
  1. ELM provides a novel way to convert test-time reasoning experience into reusable latent memories without updating the base model.

  2. The method improves performance in most model–benchmark settings, with particularly clear gains over textual-memory baselines on the two base Qwen models.

Critical weaknesses
  1. The scope of the claim of “self-improvement without external supervision” requires clarification. Although ELM trains its memories using majority voting over the model’s own outputs, its strongest online results rely on an externally trained Qwen2.5-Math-PRM-7B router. Table 12 shows that replacing this router with Self-Judge substantially reduces performance and can produce results below the zero-shot baseline in some settings. Thus, memory construction is internally supervised, but the complete inference system is not fully self-contained.

  2. Appendix A reports wall-clock costs for the evaluated streams, including roughly one day of memory extraction for MATH500 on a single H200. However, the experiments contain at most 500 stream elements and do not evaluate how total storage, retrieval latency, or serving cost scale with a substantially larger memory pool. Since every sample adds a separate memory and the paper does not implement merging or eviction, long-term scalability remains unclear.

  3. The paper acknowledges that incorrect self-supervision can create harmful memories, but it does not directly quantify pseudo-label accuracy or examine whether such memories accumulate over longer streams. Freezing the base model and using modular memories limit interference, while the external PRM can reject some harmful outputs; however, these mechanisms neither detect nor remove incorrect memories from the memory pool. This unresolved issue limits confidence in the long-term reliability of continual self-improvement.

Questions for authors
  1. How should the claim of self-improvement without external supervision be interpreted given that the final response selection relies on an externally trained PRM? Can ELM maintain its gains without this verifier?

  2. How does ELM scale to much longer data streams, considering that every sample requires GRPO training and adds a permanent memory? Could the authors report the expected training, storage, and retrieval costs as the memory pool grows?

  3. In the main comparison, ELM uses a response router while the training-free memory baselines do not. Could the authors clarify why the routed versions of these baselines are not included in the main table, particularly since some routed baselines outperform ELM in the appendix?

  4. How does ELM prevent incorrect pseudo-labels from creating and accumulating harmful memories over time?

1 comment

I would appreciate the authors’ response to the main concerns raised in my review. These points are important to my assessment, and without further clarification, I may need to reconsider my current rating.

Reviewer_A3S4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Experiential Latent Memory for online continual self-improvement of LLMs. For each encountered example, ELM trains a small soft-prompt memory using GRPO with self-generated majority-vote rewards while keeping the base model frozen. For future inputs, the method retrieves a relevant memory. The results show that ELM generally improves over zero-shot and textual memory baselines, using only a very small number of trainable parameters per memory.

Critical strengths
  1. The experiments provide useful evidence that soft prompts generalize better than LoRA or full-model per-instance updates while requiring substantially fewer trainable parameters. Representing each experience as an independently trained soft prompt is an good way to accumulate knowledge without modifying the backbone.

2.Strong empirical results in the online setting. ELM consistently improves over zero-shot on the Qwen models and substantially outperforms several textual-memory baselines, particularly on base models where conventional ICL-style memories often degrade performance.

Critical weaknesses

1.The paper motivates ELM as self-improvement using internally generated supervision, but the final online system uses Qwen2.5-Math-PRM-7B to choose between zero-shot and memory-conditioned responses. The routing ablation shows that the PRM contributes a non-trivial fraction of the improvement, while the Self-Judge alternative is substantially weaker and even hurts performance in several settings. This makes it less clear how much of the claimed reliable self-improvement comes purely from the learned memories.

2.The evidence for scalable continual learning is still limited. largest stream contains only hundreds of examples, and most experiments operate within a single eval benchmark distribution. Since a new independently trained memory is created for every sample using multiple rollouts and GRPO steps, both cumulative storage and training cost grow with the stream length. The paper does not yet characterize retrieval quality, memory cost, or performance as the memory pool becomes much larger or more heterogeneous. This matters because scalability is central to the paper's continual-learning claim. Memory creation itself requires 10 gradient steps with 8 rollouts per example, and the reported extraction cost reaches roughly one day for MATH500.

  1. The evaluation is limited, Llama-3.1-8B-Instr. and Qwen2.5-Math-7B are a little bit out of date. Considering more recent models will make results more convincing.
Questions for authors

1.Can you report the main online results using Self-Judge or another fully self-contained routing mechanism? It would also be useful to clarify the supervision and training data used by the PRM.

  1. Can you report cumulative memory/storage cost, retrieval cost and accuracy as a function of stream length, ideally on a longer or heterogeneous stream? This would be interesting to see what will ELM behave as the memory pool becomes substantially larger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.