- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies what forms of experience memory help an LLM agent to make better decisions in a card game. It compares no memory, weak agent's own summary of a batch of old trajectories, externally computed statistics over many games, LLM written suggestions without trajectories (called Expert) and LLM written suggestions based on past trajectories (called Teacher). Experiments on 50 games per setup show that the methods using a stronger LLM outperform the methods using the weak LLM only. Other high level differences were generally not statistically significant.
Critical strengths
- Memory systems of LLM agents are an important research direction.
- The results are analyzed in unusual depth.
- The paper clearly presents important statistics on the results and the details of the experiment setup.
Critical weaknesses
- The main conclusions of the paper are trivial. As I understand them, they say that a weak model works much better with detailed instructions on a task from a strong model. Results for no memory vs. batch and Teacher vs. Expert are not statistically significant when using 50 games.
- The methods used are not described sufficiently clearly. The authors should provide the exact prompts they used for aggregating the traces in self-batch, Teacher, etc.
- The main results in Figure 2 show that a vast majority of the differences between the variants of the memory lay within the 95% confidence interval and hence cannot be confidently differentiated from random noise. Only reading appendix B establishes actual significance between more pairs. It should be presented differently.
- The paper is about a single card game without any attempt to generalize the results. In particular, the Expert and Teacher method rely on the knowledge of the strong LLM about the particular game which is very hard to generalize.
- As the game is opensource and has a public wiki available, to try to isolate the LLM player from accessing prior knowledge of the wiki, the authors rename the card and relic names with pseudonyms. However, they preserve the exact descriptions of the cards with no transformation. I have observed the models recognizing the transformed names and using them in further reasoning in my own experiments unrelated to this paper. Therefore I think this would deserve some deeper insight into whether the pseudonyms work.
- Directly comparing Self-batch and Teacher, which uses a very strong model on trajectories and 6 guidelines imputed by a human does not make much sense. An intuitive explanation is that the human guidelines provide very powerful signal to learn from and it is not very surprising that it performs better than the Self-batch case where the agent has to learn the memory by itself.
- The phrasing is hard to read and unclear. For example “A writer with its own prior knowledge and six human guidelines read the same games and wrote corrections of the player’s errors beside rules the games do not contain. Statistics reaches the clear rate of Expert for DeepSeek Flash, but it exists only because the simulator scores every game for free, a signal most settings with a memory lack.“
Questions for authors
- What is the novel reusable knowledge, unrelated to the particular game that you study, that the reader should take from reading the paper?
- Why not to compare to other experience memory systems surveyed in your related work?