Training Language Agents to Learn from Experience

Language agents can adapt from experience in interactive environments, but current reflection-based methods primarily self-correct by retrying the same task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes actor trajectories and generates system prompts intended to improve the actor's performance on future unseen tasks. We also propose an RL-based pipeline for training reflectors directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform matched untrained baselines on all six held-out task types, showing that the ability to learn from experience can itself be learned. These gains extend across two model families and to a different actor, with initial evidence of cross-benchmark transfer. Finally, we introduce MetaGym, a Python library for constructing meta-environments for self-improving language agents.