acceptodds
Under review as a conference paper at ICLR 2027

Experience Is Not Knowledge: Where the Decision Knowledge in an Agent's Memory Comes From

Abstract

LLM agents are commonly improved by writing a memory from their own trajectories and reading it back before the next episode. The gain is credited to experience, but the memory mixes what the writer already knew with what it induced from the trajectories. We ask where the useful decision knowledge in an agent’s memory comes from. The testbed is Slay the Spire, a roguelike deckbuilder: every run is generated fresh and the same decisions recur, so a rule written from a few runs is tested hundreds of times in fresh ones. We hold the player model, the combat executor, the prompt and the form of the memory fixed, and vary only the source of the memory’s content. The self memory is the player’s own summary of its games. The others hold outcome statistics from randomized simulator games, a stronger model’s prior knowledge alone, or, in the teacher memory, the player’s trajectories read by a stronger model under a few human-written guidelines. The player’s own memory does not raise its clear rate, and neither fewer games, a stronger writer nor writing one game at a time changes that. The memories written from statistics, from prior knowledge and by the teacher raise it, and the teacher memory raises it from 14 clears of 50 to 30. Replay at shared decision states shows that the player acts on its self-written memory, and that the memory poisons it. From the play counts in its own logs the writer concluded that the basic attack, which every deck starts with and so plays most, deserves an early upgrade, and the player then made in the exam an error it had almost never made in the games the memory was written from. The statistics and teacher memories reach their gains by different routes: one keeps adding cards to a large deck and the other stops. The teacher memory, written from one player’s trajectories, is the best memory for the other two players, whose own memories again do not help them. The three players converge on what it states simply and stay apart on the rules they must compose. Experience is not knowledge. The player improved when its memory held correct, usable rules, whether a stronger reader extracted them from the player’s own games or brought them from outside, and a wrong extraction from those same games became a new error.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, experience replay and reflection, self-improvement, memory poisoning, roguelike games, Slay the Spire, decision making, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Za8nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies what forms of experience memory help an LLM agent to make better decisions in a card game. It compares no memory, weak agent's own summary of a batch of old trajectories, externally computed statistics over many games, LLM written suggestions without trajectories (called Expert) and LLM written suggestions based on past trajectories (called Teacher). Experiments on 50 games per setup show that the methods using a stronger LLM outperform the methods using the weak LLM only. Other high level differences were generally not statistically significant.

Critical strengths
  • Memory systems of LLM agents are an important research direction.
  • The results are analyzed in unusual depth.
  • The paper clearly presents important statistics on the results and the details of the experiment setup.
Critical weaknesses
  1. The main conclusions of the paper are trivial. As I understand them, they say that a weak model works much better with detailed instructions on a task from a strong model. Results for no memory vs. batch and Teacher vs. Expert are not statistically significant when using 50 games.
  2. The methods used are not described sufficiently clearly. The authors should provide the exact prompts they used for aggregating the traces in self-batch, Teacher, etc.
  3. The main results in Figure 2 show that a vast majority of the differences between the variants of the memory lay within the 95% confidence interval and hence cannot be confidently differentiated from random noise. Only reading appendix B establishes actual significance between more pairs. It should be presented differently.
  4. The paper is about a single card game without any attempt to generalize the results. In particular, the Expert and Teacher method rely on the knowledge of the strong LLM about the particular game which is very hard to generalize.
  5. As the game is opensource and has a public wiki available, to try to isolate the LLM player from accessing prior knowledge of the wiki, the authors rename the card and relic names with pseudonyms. However, they preserve the exact descriptions of the cards with no transformation. I have observed the models recognizing the transformed names and using them in further reasoning in my own experiments unrelated to this paper. Therefore I think this would deserve some deeper insight into whether the pseudonyms work.
  6. Directly comparing Self-batch and Teacher, which uses a very strong model on trajectories and 6 guidelines imputed by a human does not make much sense. An intuitive explanation is that the human guidelines provide very powerful signal to learn from and it is not very surprising that it performs better than the Self-batch case where the agent has to learn the memory by itself.
  7. The phrasing is hard to read and unclear. For example “A writer with its own prior knowledge and six human guidelines read the same games and wrote corrections of the player’s errors beside rules the games do not contain. Statistics reaches the clear rate of Expert for DeepSeek Flash, but it exists only because the simulator scores every game for free, a signal most settings with a memory lack.“
Questions for authors
  1. What is the novel reusable knowledge, unrelated to the particular game that you study, that the reader should take from reading the paper?
  2. Why not to compare to other experience memory systems surveyed in your related work?
Reviewer_EuyqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LLMs improve by looking back at their trajectories and use this when it encounters a similar scenario again by writing that part of memory down. The paper aims to consider whether the written down memory episode was from the trajectory or from the base knowledge itself and ask “where the useful decision knowledge comes from”. The authors use Slay the Spire as the testbed. By keeping the player, prompt and memory format fixed and changing only where the memory comes from, they find out that the player's own memory does not improve its win rate, but the distilled teacher’s memory helps raise the score from raising clears from 14 to 30 out of 50. The teacher also received six human-written guidelines, and its gain does not separate a memory written by a stronger model from prior knowledge alone.

Critical strengths
  1. The harness in the paper provides a very detailed control setting to isolate the contribution of the trajectory memory from the model’s inbuilt state. As the player, combat executor, prompt and memory format are held fixed, the source of the memory’s content is the only distinction.
  2. The finding that self-written memory doesn’t help holds across four DeepSeek variants (fewer games, a stronger writer, sequential writing) and for two other player models. The teacher's memory, written from one player’s games, is the best arm for all three players, which suggests it carries knowledge about the game rather than about one player. They do a thorough analysis of the results using pairwise tests, Holm correction, Wilson intervals, and disclosed failures.
Critical weaknesses
  1. Only one rule is traced end to end. The full chain from the writer input to the behaviour is shown only for “Strike” rule, although there are rules which link the memory to the behaviours but are not traced end-to-end. The self-batch scores 15/50 when compared to 14/50 of no-memory. The claim that memory “poisons” the player rests on 1 example’s changed behavior.
  2. The memory ablation is done thrice for the self-memory, but the teacher’s knowledge is distilled on a single run. Besides only a single teacher is used in the entire paper (Claude Fable 5.1 pipeline with 7 Claude Opus 5 trajectory readers, a Claude Sonnet statistics agent and an Opus card-table agent). This becomes unclear to what contributes to the gain.
  3. Low statistical significance. With 50 games per arm, it becomes very difficult to distinguish the contribution of each of the components to the gain.
  4. Given that the paper claim that the teachers memory combines to locate the player errors and prior knowledge to fix this, the fact that the teacher clearly does not outperform the expert model (30 vs 26 out of 50) means the trajectories’ contribution is not demonstrated. The gain could come entirely from the writer’s prior knowledge the human guidelines.
Questions for authors
  1. Can you trace other self-memory rules end to end, and compare exam games with and without a Strike upgrade ?
  2. Can you show the teacher pipeline run with zero trajectories (or without the six guidelines), to show the trajectories add value beyond the zero-game Expert memory (30 vs 26 of 50) ?
  3. Can you try with a different teacher agent and try to use the memory from its trajectories ?
Reviewer_S4fXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

AI Statement: I don't use AI for generating reviews - that is a abdication of duty, and I do not think LLM generation has a place in academic discourse - LLM generation in papers is plagiarism and poor scholarly practice.

Critical strengths

The paper examines agent processing in a Slay the Spire game.

Critical weaknesses

The paper runs a few experiments in a single agentic setting and tries to draw conclusions from those experiments. The experiments are insufficiently controlled to draw meaningful generalisable scientific conclusions beyond "we ran this and A is better than B". The stated conclusions go way beyond what can be ascertained from the experiments done. I gained no scholarly insight from this, and I do not feel the paper provides actual research value. It analysis a single existing code base rather than a methodological concept, and I have no way of knowing whether any of the observations have any generalisation value.

Questions for authors

I have no questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.