- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a memory-driven self-improvement framework for LLM agents in sequential decision-making tasks. The framework consists of two components: (1) memory-based non-parametric Q-value estimation, which retrieves similar state-action experiences to guide action selection, and (2) LLM policy refinement, which uses Q-weighted supervised fine-tuning to improve action generation. Experiments on agentic tasks demonstrate improvements in sample efficiency and generalization.
Critical strengths
-
The paper addressed an interesting topic about utilizing memory to improve LLM agent.
-
The framework combines experience based value estimation with LLM policy optimization, allowing historical interactions to improve both action selection and future action generation.
Critical weaknesses
-
The proposed Mem-Q method is closely related to classical episodic control and nearest-neighbor value estimation. Its main difference appears to be the use of pretrained language model embeddings for state-action representations. Compared with a parameterized Q-network, the generalization of the proposed estimator depends heavily on whether semantic similarity in the embedding space reflects similarity in long-term action values.
-
The contribution of Q estimation and policy optimization is insufficiently isolated. The strongest results seem to come from Q-weighted policy fine-tuning rather than memory-based action selection alone. For example, on Game of 24, Mem-EM without fine-tuning achieves 11.35% test success, compared with 10.67% for the zero-shot LLM. In contrast, the fine-tuned policy achieves 18.78% even without test-time memory access, while adding memory-based action selection further increases performance to 19.65%. These results suggest that the main generalization benefit comes from distilling high-value experiences into the LLM policy.
-
The paper evaluates methods that access a memory table during inference, including on unseen tasks. However, it is not sufficiently clear whether the memory is frozen after training, whether test-time interactions are added to it, and whether memory is reset between evaluation episodes. This distinction matters because performance improvements from retrieving training experiences are conceptually different from improvements obtained through test-time online adaptation.
Questions for authors
see weaknesses