- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how the interaction trajectories of an LLM agent can be turned into memories that improve its later decisions. The authors introduce EvoExp, a framework for this. Memory entries are paired with executable applicability conditions, and only entries whose condition holds are added to the prompt of the frozen actor LLM during inference. They also train a “memory inducer” that builds these memories, first with SFT and then with RL, using a variant of GRPO they call EU-GRPO; RL uses only Dou Dizhu. The trained memory inducer can then construct memories in other environments (Tide and ALFWorld), showing some level of generalization.
Critical strengths
-
I think evaluating a memory update by its effect on the same actor at the same decision states is a useful approach (Section 3.4, Eq. 12). It shows the updater how its advice affects the actor's decisions, rather than whether the next game was won (or, in a more general setting, whether the task was completed). The paper also states that an update gets “zero credit at that state” if it does not change the prompt (Page 6), so updates that never reach the actor are not rewarded.
-
The transfer results on Tide and ALFWorld (Table 3, Page 9) are useful, because these environments were not included in inducer training, and they show some generalization of the technique. This has to be taken with some care, though: Table 3 compares against only one other memory method (ExpeL with DeepSeek-V4), and on Tide the difference is small (0.22 vs 0.20).
Critical weaknesses
-
It is hard to tell how reliable the gains from RL really are, especially in Dou Dizhu, the only game used for RL training. Compared with SFT, RL improves the Dou Dizhu win rate by 0.06 and Hold'em by 2.25 chips (Table 1, Page 8). Both are within the average confidence-interval half-widths the paper reports (±0.07 for win rates, ±3.72 chips; Page 7). The gains are larger in Liar's Dice (+0.20) and Love Letter (+0.17), and Big Two goes down (−0.08). The average intervals describe the reported entries, so they do not directly show how reliable the differences between RL and SFT are. The authors also state that differences on the scale of the interval, including the Dou Dizhu SFT–RL gap, are “numerical trends rather than conclusive pairwise improvements” (Page 7). Since the RL stage (EU-GRPO) is a central part of the method, this makes its benefit over SFT hard to judge.
-
Table 1 (Page 8) does not clearly support the paper's claim that EvoExp is more consistent than the other memory methods when its inducer is not trained. For example, with Qwen3.8-Flash as the experience inducer, EvoExp seems competitive, but I don't see whether it is actually better. It has the highest value in Texas Hold'em (+6.36) and Big Two (+0.12), but it is below ReasoningBank in Dou Dizhu (+0.02 vs +0.17), below Reflexion and ReasoningBank in Liar's Dice (+0.63 vs +0.72 and +0.70), and slightly below three methods in Love Letter (+0.25 vs +0.26 to +0.31); with DeepSeek-V4-Flash it is highest in three of the five games. The paper argues that EvoExp is more consistent because its gains are positive in all five games, and concludes that its interface is useful “even before local training” (Page 7). However, the Dou Dizhu gain of +0.02 is well within ±0.07, and with DeepSeek-V4-Flash, ReasoningBank is also positive in all five games (+0.15, +0.38, +4.12, +0.54, +0.26). Since the table reports only point estimates, it is hard to know how large these differences really are. If EvoExp were clearly better everywhere this might matter less, but as it wins in some games and loses in others, per-entry confidence intervals would help.
Questions for authors
-
(Clarification) All three rewards in Table 2 start from the same SFT initialization (Page 7), whose results are in Table 1: +0.14 in Dou Dizhu, +0.56 in Liar's Dice and +8.05 chips in Hold'em (Page 8). Why do both State Rollout and Next-Game Payoff perform worse than this SFT starting point? Apart from State Rollout in Dou Dizhu (+0.12 vs +0.14), the drops are larger than the average interval half-widths (Page 7).
-
(Clarification) Could you report confidence intervals for the paired differences RL − SFT and Decision Utility − State Rollout? For ALFWorld, how many of the 100 episodes are solved only by SFT and how many only by RL?
-
(Future work) My understanding is that the value evaluator is only used during RL training, and that the trained inducer can be used “without further parameter updates or a deployment-time value evaluator” (Page 9), and ALFWorld already shows it working outside games. For a more complex environment, like a codebase, could the game-trained inducer be used directly after defining the observation fields and memory interface, or would it need more training? If further training is needed, how would the decision-utility signal be obtained? Section 3.1 suggests that action values for more complex environments “may be provided by a pretrained value function” (Page 5), but Appendix A.4 notes that a suitable evaluator may not always be available (Page 16). It would be good to clarify which parts of the method transfer to new environments and which need to be developed separately.
-
(Clarification) For SFT, the paper states, “We retain 5,000 final candidate records” generated using Qwen3.8-Flash (Page 5), and Appendix A.2 describes how invalid updates are rejected (Page 13). How were these 5,000 records selected? Were they tested to see whether they improved the actor's decisions, or only checked for being valid updates?