acceptodds
Under review as a conference paper at ICLR 2027

Learning to Distill Experience: Utility-Driven Experience Induction for LLM Agents

Abstract

Large language model (LLM) agents increasingly distill interaction trajectories into external memory to reuse past experience, yet determining what should actually be learned remains challenging. A single trajectory can support multiple plausible lessons, while its outcome provides limited evidence about which lessons will help in subsequent decision contexts. We formulate this challenge as utility-driven memory induction: learning to transform interaction trajectories into reusable knowledge by optimizing its downstream decision utility, rather than trajectory-level outcomes alone.. We propose EvoExp, a trainable experience inducer that generates actionable guidance paired with explicit applicability conditions. Following supervised initialization, Experience-Utility GRPO (EU-GRPO) optimizes the inducer through paired evaluations of a frozen actor under current and updated memory stores, using estimated action-value improvements across decision contexts as feedback. Once trained, EvoExp constructs new memories from fresh trajectories without further parameter updates or a deployment-time value evaluator. Across five strategic games, the RL-trained inducer outperforms the no-memory control in every game and achieves the best reported performance in three, including a +0.20 paired win-rate gain on Dou Dizhu and +10.30 chips on Texas Hold'em; despite RL training only on Dou Dizhu, EvoExp further improves on ALFWorld and GPT-authored Tide Ledger game.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, External Memory, Experience Induction, Reinforcement Learning, Test-Time Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_As74OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the interaction trajectories of an LLM agent can be turned into memories that improve its later decisions. The authors introduce EvoExp, a framework for this. Memory entries are paired with executable applicability conditions, and only entries whose condition holds are added to the prompt of the frozen actor LLM during inference. They also train a “memory inducer” that builds these memories, first with SFT and then with RL, using a variant of GRPO they call EU-GRPO; RL uses only Dou Dizhu. The trained memory inducer can then construct memories in other environments (Tide and ALFWorld), showing some level of generalization.

Critical strengths
  1. I think evaluating a memory update by its effect on the same actor at the same decision states is a useful approach (Section 3.4, Eq. 12). It shows the updater how its advice affects the actor's decisions, rather than whether the next game was won (or, in a more general setting, whether the task was completed). The paper also states that an update gets “zero credit at that state” if it does not change the prompt (Page 6), so updates that never reach the actor are not rewarded.

  2. The transfer results on Tide and ALFWorld (Table 3, Page 9) are useful, because these environments were not included in inducer training, and they show some generalization of the technique. This has to be taken with some care, though: Table 3 compares against only one other memory method (ExpeL with DeepSeek-V4), and on Tide the difference is small (0.22 vs 0.20).

Critical weaknesses
  1. It is hard to tell how reliable the gains from RL really are, especially in Dou Dizhu, the only game used for RL training. Compared with SFT, RL improves the Dou Dizhu win rate by 0.06 and Hold'em by 2.25 chips (Table 1, Page 8). Both are within the average confidence-interval half-widths the paper reports (±0.07 for win rates, ±3.72 chips; Page 7). The gains are larger in Liar's Dice (+0.20) and Love Letter (+0.17), and Big Two goes down (−0.08). The average intervals describe the reported entries, so they do not directly show how reliable the differences between RL and SFT are. The authors also state that differences on the scale of the interval, including the Dou Dizhu SFT–RL gap, are “numerical trends rather than conclusive pairwise improvements” (Page 7). Since the RL stage (EU-GRPO) is a central part of the method, this makes its benefit over SFT hard to judge.

  2. Table 1 (Page 8) does not clearly support the paper's claim that EvoExp is more consistent than the other memory methods when its inducer is not trained. For example, with Qwen3.8-Flash as the experience inducer, EvoExp seems competitive, but I don't see whether it is actually better. It has the highest value in Texas Hold'em (+6.36) and Big Two (+0.12), but it is below ReasoningBank in Dou Dizhu (+0.02 vs +0.17), below Reflexion and ReasoningBank in Liar's Dice (+0.63 vs +0.72 and +0.70), and slightly below three methods in Love Letter (+0.25 vs +0.26 to +0.31); with DeepSeek-V4-Flash it is highest in three of the five games. The paper argues that EvoExp is more consistent because its gains are positive in all five games, and concludes that its interface is useful “even before local training” (Page 7). However, the Dou Dizhu gain of +0.02 is well within ±0.07, and with DeepSeek-V4-Flash, ReasoningBank is also positive in all five games (+0.15, +0.38, +4.12, +0.54, +0.26). Since the table reports only point estimates, it is hard to know how large these differences really are. If EvoExp were clearly better everywhere this might matter less, but as it wins in some games and loses in others, per-entry confidence intervals would help.

Questions for authors
  1. (Clarification) All three rewards in Table 2 start from the same SFT initialization (Page 7), whose results are in Table 1: +0.14 in Dou Dizhu, +0.56 in Liar's Dice and +8.05 chips in Hold'em (Page 8). Why do both State Rollout and Next-Game Payoff perform worse than this SFT starting point? Apart from State Rollout in Dou Dizhu (+0.12 vs +0.14), the drops are larger than the average interval half-widths (Page 7).

  2. (Clarification) Could you report confidence intervals for the paired differences RL − SFT and Decision Utility − State Rollout? For ALFWorld, how many of the 100 episodes are solved only by SFT and how many only by RL?

  3. (Future work) My understanding is that the value evaluator is only used during RL training, and that the trained inducer can be used “without further parameter updates or a deployment-time value evaluator” (Page 9), and ALFWorld already shows it working outside games. For a more complex environment, like a codebase, could the game-trained inducer be used directly after defining the observation fields and memory interface, or would it need more training? If further training is needed, how would the decision-utility signal be obtained? Section 3.1 suggests that action values for more complex environments “may be provided by a pretrained value function” (Page 5), but Appendix A.4 notes that a suitable evaluator may not always be available (Page 16). It would be good to clarify which parts of the method transfer to new environments and which need to be developed separately.

  4. (Clarification) For SFT, the paper states, “We retain 5,000 final candidate records” generated using Qwen3.8-Flash (Page 5), and Appendix A.2 describes how invalid updates are rejected (Page 13). How were these 5,000 records selected? Were they tested to see whether they improved the actor's decisions, or only checked for being valid updates?

Reviewer_h5aiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EvoExp, a trainable memory inducer that converts interaction trajectories into guidance paired with executable applicability conditions. Following supervised initialization, EU-GRPO trains the inducer by comparing a frozen actor’s actions under current and updated memory stores at matched decision states, using a fixed action-value evaluator. RL training occurs only in Dou Dizhu; the learned inducer subsequently constructs memories in other environments without further parameter updates or a deployment-time evaluator.

Critical strengths
  1. The problem is well motivated. Successful trajectories do not uniquely identify useful lessons. Evaluating a memory through its effect on subsequent decisions is a sensible response to this ambiguity.
  2. A useful representation and several good experimental controls. Pairing guidance with executable, AST-validated applicability conditions (A.2) makes memory exposure explicit and inspectable. The paired reward (Eq. 12) carries this into training by holding the state, actor, evaluator and decoding fixed, so it measures the marginal effect of an update where the memory is reused. The evaluation is also well controlled: the actor is frozen, ten stores are built from disjoint source groups, source episodes and test scenarios are shared across methods, and the prompt-based baselines use the same actor and decoding with two constructor LLMs each. A newly authored game reduces the risk of contamination.
Critical weaknesses
  1. (Main concern) Dependence on an external action-value evaluator limits the method’s demonstrated applicability. EU-GRPO requires Q^env(hs,a)\hat Q_{\mathrm{env}}(h_s,a) to evaluate actor-selected actions at sampled intermediate states (Eqs. 11–12). Section 3.4 assumes access to a tractable transition model or a pretrained value function; all RL training uses DouZero in Dou Dizhu. Reliable action-value estimates are not readily available in many practical agent settings, and obtaining them through branching rollouts may require costly state restoration and repeated interactions. The paper therefore needs to establish how much evaluator quality and infrastructure its approach requires. Table 2 does not resolve this concern. Next-Game Payoff is negative in two of three environments, while State Rollout substantially underperforms Decision Utility—for example, +1.23 versus +10.30 on Texas Hold’em. These results leave unclear how much of the improvement comes from the proposed credit-assignment design versus the external evaluator’s knowledge, continuation-policy differences, or estimator variance. They do not establish that an expert-quality evaluator is necessary, but neither do they demonstrate robustness to weaker, more accessible supervision. The fact that no evaluator is needed at deployment is a meaningful strength. However, the evidence that training with DouZero yields broadly transferable induction remains limited. Big Two, the only main-table game excluded from both SFT and RL, declines from +0.38 to +0.30 after RL; the other non-RL games appear in SFT. On ALFWorld, RL improves over SFT from 61% to 65% on 100 episodes without a reported confidence interval. Tide improves over SFT, but its advantage over ExpeL is only 0.22 versus 0.20, again without uncertainty estimates. No training experiment tests another evaluator or a non-game domain. Although Appendix A.4 acknowledges this limitation, its implications for the method’s scope need stronger empirical treatment.

  2. The most relevant trained baseline is missing. MemoPilot (Cai et al., ICML 2026) trains a memory updater with GRPO for a frozen LLM player in games, including Limit Texas Hold'em, and its code is public. The paper discusses it as related work but only adopts its “Next-Game Payoff” reward unit in Table 2. Other learned-memory methods (Memory-R1, MemSkill, MemRL) are also absent; MemSkill reports ALFWorld results. Separately, the claim that all methods use the “same retrieval procedure” needs clarification: how are baseline memories without executable applicability conditions handled, and are retrieval and context budgets matched across methods?

Questions for authors
  1. Can the dependence on an expert evaluator be overcome in practice?
  2. Does Table 2 isolate the benefit of decision-level credit assignment?
Reviewer_QfpVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EvoExp, a trained memory inducer that converts trajectories into memory items. Each memory entry includes a lesson and applicability condition. EvoExp is trained with a variation of GRPO which rewards a candidate memory update by the change in a fixed value model’s action-value scores, measured with and without the update. Experiments cover five strategic games, including some transfer experiments.

Critical strengths

Credit assignment for trained memory modules is an important problem. The paper is easy to follow and the experiments include multiple games and include ablations on the credit signal and applicability conditions that support the design choices.

Critical weaknesses

1- The related work covers RL-trained memory methods, but none of them appear as baselines in the experiments. The comparisons are only with prompt-based, untrained methods. MemoPilot is discussed only through its reward unit in Table 2 which is not a fully fair comparison as (a) That tests one component, not the full method (b) EvoExp requires a learned action-value model to estimate rewards. Moreover, other memory-augmented works are missing from related work including Mem-pi (Wang et al.) and JEF-Hinter (Nekoei et al.).

2- Many of the claimed improvements are within confidence intervals (only the averaged CI is reported). With matched inducers, EvoExp often does not improve over the baselines. On Dou Dizhu for example, it scores +0.02 against ReasoningBank’s +0.15 to +0.17, and it is behind or within noise on Liar’s Dice and Love Letter. Evo-Exp RL’s gain over Evo-Exp SFT is mostly within noise too. Most of the gain over the base inducer comes from SFT, whose data was generated by Qwen3.8-Flash. Finally, the transfer experiments compare only against ExpeL which is among the weakest methods in Table 1.

Questions for authors

1- How much of the base-to-SFT improvement is distillation versus filtering? Have you tried SFT on Qwen3-8B’s own outputs?

2- EU-GRPO is trained on DouZero’s action-value differences, yet the SFT-to-RL gain on Dou Dizhu itself is small compared to other unseen games. Does a higher delta Q during training actually trasnlates to higher win rate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.