acceptodds
Withdrawn from ICLR 2027

Training Language Agents to Learn from Experience

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language agents can adapt from experience in interactive environments, but current reflection-based methods primarily self-correct by retrying the same task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes actor trajectories and generates system prompts intended to improve the actor's performance on future unseen tasks. We also propose an RL-based pipeline for training reflectors directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform matched untrained baselines on all six held-out task types, showing that the ability to learn from experience can itself be learned. These gains extend across two model families and to a different actor, with initial evidence of cross-benchmark transfer. Finally, we introduce MetaGym, a Python library for constructing meta-environments for self-improving language agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Language Agents, Meta-learning, Self-improvement, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7gdvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a meaningful question: whether a language model can be trained to distill prompt-update rules from interaction experience in order to improve performance on future tasks. Freezing the actor, using verifiable environment rewards, and evaluating across different reflectors and actors provide encouraging empirical signals.

However, the current evaluation selects prompts on a fixed validation set and reports post-selection scores on that same set, without an independent final test. In addition, checkpoint validation is exposed to task types from the meta-test split, which weakens the interpretation of the results as generalization to strictly unseen task types.

Together with the other issues raised in the weaknesses below, in summary, these concerns lead me to conclude that the paper is not yet ready for publication in its current version.

Critical strengths

S1: The paper distinguishes solving the current instance from distilling experience for future instances. This is a more meaningful setting than retrying the same task, since many real-world interactions cannot be rolled back.

S2: Keeping the actor frozen in the main experiments isolates the effect of prompt updates from changes in actor capability.

S3: Task success in practical environments, (i.e., ALFWorld and MiniHack) provides a reliable training signal and reduces evaluation bias.

Critical weaknesses

W1: Training a reflector to improve an actor has precedents in Retroformer [1] and SAMULE [4]; cross-task experience consolidation is explored in ExpeL [2] and MetaReflection [3]; and RL-based prompt generation is established in StablePrompt [5]. The potentially distinctive contribution is the no-retry, cross-instance adaptation setting, rather than these components themselves.

However, comparisons primarily against untrained reflectors do not sufficiently establish the adadamic value over these related approaches. Clearer differentiation and targeted comparisons are needed.

  • [1] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR, 2024.

  • [2] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19632–19642, 2024.

  • [3] Priyanshu Gupta, Shashank Kirtania, Ananya Singha, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, and Sherry Shi. MetaReflection: Learning Instructions for Language Agents using Past Reflections. EMNLP, 2024.

  • [4] Yubin Ge, Salvatore Romeo, Jason Cai, Monica Sunkara, and Yi Zhang. SAMULE: Self-Learning Agents Enhanced by Multi-level Reflection. EMNLP,

  • [5] Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, and Junmo Kim. StablePrompt: Automatic Prompt Tuning using Reinforcement Learning for Large Language Model. EMNLP, 2024.

  • [6] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. ICLR, 2026.

  • [7] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR, 2026, Oral.

  • [8] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. EMNLP, pp. 3369–3391, 2022.

  • [9] Yumin Choi, Jinheon Baek, and Sung Ju Hwang. System Prompt Optimization with Meta-Learning. NeurIPS, 2025.

  • [10] Yulun Jiang, Liangze Jiang, Damien Teney, Michael Moor, and Maria Brbić. Meta-RL Induces Exploration in Language Agents. ICLR, 2026.

W2: The training reward optimizes same-instance retries, not generalization to new instances (Eq. 4, p. 5).

  • The reward Rtrain(sp′)=1k∑e∈BR(actor,e,sp′)R_{\text{train}}(sp') = \frac{1}{k}\sum_{e \in B} R(\text{actor}, e, sp') is computed on the same batch BB whose trajectories the reflector has just seen,
  • The actual goal is Ee′∼F[R(actor,e′,sp′)]\mathbb{E}_{e' \sim F}[R(\text{actor}, e', sp')] on new instances from the same task family.

Because sp′sp' is conditioned on BB, a higher reward on BB does not imply a higher reward on future instances. The model may learn instance-specific fixes rather than transferable rules. Training with a replayable simulator is reasonable, but asking for "future-oriented" prompts in natural language does not constrain generalization.

Please compare observed-batch reward with fresh-batch reward (observe BB, evaluate on an independent B′B') and a mixed variant, and report how retry gains correlate with gains on new instances.

W3: Training in this paper is a one-step GRPO update on the prompt. the four collection rounds provide states from different stages but do not amount to multi-round credit assignment over a long-horizon meta-return.

W4: Best-so-far curves do not show sustained improvement (Fig. 4, p. 8; p. 9). Figure 4 plots Mt=max⁡0<i≤tR^(spi)M_t = \max_{0<i\le t} \widehat{R}(sp_i), which is non-decreasing by construction. It can keep rising even if the current prompt stops improving or degrades. Likewise, the fraction of best prompts found after round 4 does not show that a long-horizon adaptation strategy has been learned, since sampling more candidates of similar quality would also tend to produce a later maximum.

Please report the held-out performance of the current prompt at each round, the final prompt's performance without validation-based selection, a random-candidate/restart baseline with the same budget, and the average round-to-round gain and degradation rate.

W5: Baselines are too weak to establish the need for a trained reflector. The main table compares the initial prompt, an untrained reflector, and the trained reflector. This shows that fine-tuning helps, but not whether the training pipeline is worth adopting over existing experience-summarization or context-update methods.

Please add

  • ExpeL (insights-only)
  • ACE-style online context updates
  • Well-prompted but untrained reflector
  • Multi-candidate prompt search under the same budget.
  • Methods that allow repeated evaluation, such as GEPA

could be reported separately as a more permissive setting, with their extra budget stated explicitly.

W6: The batch-size ablation is confounded. Moving between settings changes several factors at once: the experience batch size at training and at test time, the MiniHack training episode length (reduced to 18 steps for k=5k=5), some context/response lengths, and the checkpoint used. As a result, the failure of k=3k=3 on MiniHack cannot be cleanly attributed to too few successful samples and a weak GRPO signal.

Please vary one factor at a time, e.g., change only the training batch size while holding episode length, context limits, test-time batch size, and checkpoint selection fixed.

W7: The initial prompt is excluded from best-so-far tracking (Alg. 1). Algorithm 1 does not initialize the tracked score with the actual performance of sp0sp_0, and MtM_t takes the maximum over 0<i≤t0 < i \le t only. If every update performs worse than the initial prompt, the curve would still look like steady progress.

Please explain why sp0sp_0 is excluded, and report how often updated prompts underperform sp0sp_0, as well as results with sp0sp_0 included in the maximum.

W8: It is also unclear whether Tables 1–5 report scores on the same validation set used for prompt selection.

If so, results on a fully independent test set are needed.

Questions for authors

See Weakness

Reviewer_F3yEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors formulate the ICT task, an online adaptation framework where an agent must distill reusable lessons across non-repeatable episodes to improve its performance on unseen future tasks. Specifically, they employ a dual-LLM setup with a frozen actor and a trainable reflector. Using GRPO, the reflector is optimized by dynamically replaying actor trajectories on past training batches without human demonstrations. Evaluated on held-out task types across ALFWorld and MiniHack, the trained reflectors show performance improvements over untrained counterparts, demonstrating transferability across different actor models and across benchmarks.

Critical strengths

The paper provides a neat reinforcement learning pipeline that drives prompt optimization directly through downstream execution replay, eliminating the need for human-annotated reflection data.

Critical weaknesses
  1. The core experiments rely exclusively on older, small models like Qwen2.5-7B and Llama-3.1-8B, which naturally suffer from limited baseline reasoning and trajectory synthesis. Since modern base models possess substantially stronger native reasoning and deduction capabilities, it is highly likely they can perform this reflection zero-shot without needing any dedicated, compute-heavy RL post-training.

  2. The proposed approach offers limited practical significance given current frontier architectures (e.g., DeepSeek-V4.1-Flash, GPT-6 Astra) that already excel at in-context failure diagnosis and high-level strategy extraction. Furthermore, the authors do not evaluate larger, modern open-source foundations (e.g., 70B+ variants), leaving it entirely unproven whether this training pipeline yields real gains on strong models or merely compensates for the reasoning deficits of small 7B/8B checkpoints.

  3. The evaluation split between meta-train and meta-test is still within the same narrow domains (ALFWorld and MiniHack), where the operational syntax, discrete actions, and failure modes are highly repetitive and transparent (e.g., finding a fridge or reading a scroll). The underlying distribution gap is relatively small, raising concerns that the reflector simply overfits to domain-specific contrastive patterns rather than learning genuine cross-domain meta-reflection; testing on genuinely complex, open-ended environments is needed to validate real-world generalization.

Questions for authors

Given that your experiments rely solely on older, small-scale models (Qwen2.5-7B and Llama-3.1-8B) tested on homogeneous toy benchmarks: (1) how can you justify the necessity of RL post-training rather than leveraging the zero-shot reasoning priors already embedded in modern or frontier models (e.g., DeepSeek-V4.1-Flash); (2) does this optimization pipeline provide any non-trivial performance gain when applied to larger, modern open-weights foundations (such as 70B+ models); and (3) given the narrow operational gap between task types in ALFWorld and MiniHack, how do you rule out in-domain overfitting to demonstrate that the learned reflector truly generalizes to complex, open-ended environments?

Reviewer_pmm3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether an LLM can learn to turn an agent’s past experience into better instructions for new tasks. In its In-context Training (ICT) loop, a frozen actor attempts a batch of tasks, and a reflector reads the trajectories and rewrites the actor’s system prompt. The loop continues on fresh task instances, while a fixed validation set is used to select the best prompt. The reflector is trained with GRPO. It generates candidate prompts, the actor retries the previous task batch under each candidate, and the average success rate provides the reward. The actor stays frozen, and no human-written reflection examples are needed. On held-out task types in ALFWorld and MiniHack, trained reflectors outperform untrained ones. For Qwen, the average gains are 9.9 and 3.9 percentage points, respectively. The paper also reports gains with another reflector family and actor, plus modest cross-benchmark transfer. The MiniHack gain disappears with smaller episode batches. The authors release MetaGym to support this workflow.

Critical strengths
  1. The question is useful, and the pipeline is straightforward. Turning past experience into instructions that help on new tasks is a meaningful goal for agents. The method needs no human-written reflection examples, and keeping the actor frozen makes it easier to assess what training the reflector adds.

  2. The transfer results are the strongest evidence. The trained reflector also helps a different actor, and a MiniHack-trained reflector improves four of six ALFWorld task types (+3.7 points on average). These results suggest benefits beyond one model–environment pairing. Testing two reflector families and reporting 30 ICT runs per setting also gives the comparisons more support.

Critical weaknesses
  1. The main evaluation does not separate prompt selection from testing. Tables 1, 2, 4 and 5 report the best prompt found over 10 meta-turns, using the same fixed validation set for selection and scoring. Without an independent test set, the scores may be optimistic, and this bias need not cancel between trained and untrained reflectors. Figure 4 also plots best-so-far performance, so its upward trend does not establish continued improvement. Independent test scores and raw per-turn performance are needed.

  2. The “unseen task types” claim is weakened by checkpoint selection. Appendix B states that checkpoints are evaluated using additional instances from the meta-test task types. Although these instances are separate from the final evaluation instances, the evaluated task types still influence model selection. This supports instance-level generalisation more clearly than generalisation to genuinely unseen task types. Checkpoints should instead be selected using separate meta-validation types.

  3. The experiments do not isolate learning from experience. There are no controls that remove the trajectories, replace them with mismatched experiences, or provide only task descriptions and outcome summaries. Figure 3’s lessons, such as reading a scroll or using a fridge, could also come from prior knowledge. Reporting formatting failures, truncation rates and output lengths would help rule out simpler effects of fine-tuning. Without these controls, it remains unclear how much of the gain requires analysing the actual experience.

  4. The method’s added value over close prior work and simpler alternatives is unclear. Retroformer already combines a frozen actor with an RL-trained reflector; the narrower contribution here is cross-instance prompt adaptation. Yet the experiments mainly compare against an untrained reflector, without stronger prompt-search baselines or SFT on high-reward sampled outputs. The paper reports roughly 20 hours of training per reflector on a four-GH200 node, but provides no cost–performance comparison showing what this investment buys over those alternatives.

Questions for authors

Q1. Please evaluate both the selected and final-turn prompts on a disjoint test split, and report non-cumulative success on fresh adaptation batches across turns. Include (sp_0) as a scored selection candidate. Do the main gains and the claimed improvement after turn 4 still hold?

Q2. Please re-select checkpoints without using meta-test task types, using held-out meta-train instances or separate meta-validation types instead. How much do the reported results change?

Q3. Please run controls throughout the ICT loop using task descriptions only, descriptions plus outcomes, and trajectories from unrelated tasks. Apply these to both trained and untrained reflectors. How much of the training gain depends on access to task-relevant execution traces?

Q4. How does GRPO compare with SFT on high-reward reflector completions sampled and scored during training? Please also report format violations, truncation rates and output lengths for trained and untrained reflectors, to help assess how much these simpler changes explain the gains.

Q5. Please report actor calls, environment episodes and GPU time for training and adaptation, including data collection, candidate replays and validation-based selection. Under matched interaction budgets and the same non-repeatable-task constraint, how does the method compare with additional prompt search using an untrained reflector?

Q6. Have you compared replay-based rewards with rewards from a fresh batch of same-type tasks, using the same number of episodes per candidate? Does replay encourage episode-specific instructions, such as fixed object locations, and do these hurt performance on fresh tasks?

Q7. Please specify the validation-set size and seen/unseen split for each ALFWorld task type, along with the actor’s sampling temperature. For MiniHack, does the (k=3) versus (k=5) conclusion hold when training episode horizons and response-length limits are matched?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.