acceptodds
Under review as a conference paper at ICLR 2027

Memory-Driven Self-Improving LLM Agents for Sequential Decision-Making

Abstract

Large language models (LLMs) have emerged as effective action policies for sequential decision-making (SDM) tasks due to their extensive prior knowledge. However, this broad yet general knowledge is often insufficient for long-term decision-making tasks with limited task-related experience, making it challenging to efficiently adapt LLMs to specific SDM tasks, especially for generalizing to unseen tasks. To address this challenge, we propose a memory-driven self-improvement framework for LLM agents that combines the general prior knowledge of LLMs with a compact memory of domain-specific experiences. The framework consists of two mutually reinforcing components: memory-driven value estimation and LLM prior refinement. Specifically, the memory stores state-action experiences with Q-values grounded in environmental feedback, enabling retrieval-based, non-parametric value estimation while providing valuable experiences for refining the LLM prior. The refined LLM prior, in turn, generates higher-quality trajectories that further enrich the memory, forming a self-improvement loop. Experiments on multi-step agentic and mathematical reasoning tasks demonstrate that our approach improves sample efficiency and generalization to unseen tasks, substantially outperforming traditional RL and LLM-based baselines.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Sequential Decision-making, Probabilistic Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AnRaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a memory-driven self-improvement framework for LLM agents in sequential decision-making tasks. The framework consists of two components: (1) memory-based non-parametric Q-value estimation, which retrieves similar state-action experiences to guide action selection, and (2) LLM policy refinement, which uses Q-weighted supervised fine-tuning to improve action generation. Experiments on agentic tasks demonstrate improvements in sample efficiency and generalization.

Critical strengths
  1. The paper addressed an interesting topic about utilizing memory to improve LLM agent.

  2. The framework combines experience based value estimation with LLM policy optimization, allowing historical interactions to improve both action selection and future action generation.

Critical weaknesses
  1. The proposed Mem-Q method is closely related to classical episodic control and nearest-neighbor value estimation. Its main difference appears to be the use of pretrained language model embeddings for state-action representations. Compared with a parameterized Q-network, the generalization of the proposed estimator depends heavily on whether semantic similarity in the embedding space reflects similarity in long-term action values.

  2. The contribution of Q estimation and policy optimization is insufficiently isolated. The strongest results seem to come from Q-weighted policy fine-tuning rather than memory-based action selection alone. For example, on Game of 24, Mem-EM without fine-tuning achieves 11.35% test success, compared with 10.67% for the zero-shot LLM. In contrast, the fine-tuned policy achieves 18.78% even without test-time memory access, while adding memory-based action selection further increases performance to 19.65%. These results suggest that the main generalization benefit comes from distilling high-value experiences into the LLM policy.

  3. The paper evaluates methods that access a memory table during inference, including on unseen tasks. However, it is not sufficiently clear whether the memory is frozen after training, whether test-time interactions are added to it, and whether memory is reset between evaluation episodes. This distinction matters because performance improvements from retrieving training experiences are conceptually different from improvements obtained through test-time online adaptation.

Questions for authors

see weaknesses

Reviewer_fgPVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a self-evolving framework for sequential decision-making tasks. The basic idea is to have two mutual components including memory-driven value estimation and memory-driven LLM prior refinement. It introduces a data structure named memory table, which contains the historical interactions and their Q values. The values of new action candidates are estimated by retrieving similar interactions from the memory table. The memory table is also used to optimize the LLM prior by training the model with state-action pairs weighted according to their values. Experiments on various benchmarks have shown performance improvements over zero-shot LLM and traditional RL baselines.

Critical strengths
  • This paper addresses a timely and important problem on combining task-specific memory and LLM prior for sequential decision making.
Critical weaknesses
  • Estimating the values of action candidates based on similarity with existing state-action pairs may be unreliable.
  • Most of the environments used are simple and limited, it cannot yet support outperforming existing baselines in more complex scenarios.
Questions for authors
  • Could you add comparisons with more baselines, such as iteratively fine-tuning the model with value-weighted trajectories, without memory-based Q-value estimation and action selection? It would also be helpful to include more recent papers on LLM-assisted sequential decision making.
  • Most of the environments used in this paper are relatively simple. In contrast, MemRL (https://arxiv.org/pdf/2601.03192), which stands out among the baselines in the paper, was evaluated on tasks involving code generation, operating systems and databases within BigCodeBench and Lifelong Agent Bench, demonstrating the clear value of this approach for real-world applications. Is Mem-EM also applicable to other boarder and real-world scenarios?
Reviewer_ni8hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mem-EM for text-based sequential decision-making. Specifically, it stores state-action pairs with return-based values. Retrieved experiences help select LLM-generated actions and refine the LLM through Q-weighted fine-tuning or prompting. To verify the effectiveness, the authors conduct experiments on three agent environments and two math tasks. The results show improved task performance. Moreover, the experience can be transferred between models.

Critical strengths
  1. The proposed experience reuse is practical, as one memory table supports action scoring, policy learning, and prompting.
  2. The MiniF2F results in the paper show useful transfer from DeepSeek experience to Qwen3-8B, even without test-time memory access.
Critical weaknesses
  1. In Table 1, the full Mem-EM achieves 19.65%, while fine-tuned greedy decoding scores 18.78%. The improved gain is small on this task. Therefore, the authors may conduct SFT experiments under same-data and same-initialization, which would help demonstrate the benefit of Q-weighted training.

  2. In Table 2, Mem-EM uses five tactic proposals, while the DeepSeek baselines only use one. Therefore, it is unclear how much of the gain comes from memory scoring rather than extra sampling.

Questions for authors
  1. Does Mem-EM still outperform the Table 2 baselines when the number of proposals and inference budget are matched?
  2. After dropping the prior correction in lines 266–269, what objective does the update optimize, and which EM guarantees still apply?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.