Learning Stateful Predictive Knowledge From Experience
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
As large language model (LLM) agents increasingly learn from experience, they primarily rely on trajectory-level reflection to extract insights. Viewed through the lens of predictive knowledge, we argue that this approach operates on episodic hindsight rather than predictive foresight, yielding brittle, path-dependent heuristics. To address this, we propose Stateful Knowledge Learning (SKL). SKL shifts the agent's focus from trajectory-level summarization to maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to state. We first demonstrate a motivating example showing how stateful knowledge brings in granularity, generalization, and enables knowledge bootstrapping. To further scale up the idea, we introduce two algorithms via self-distillation (SKL-SD) and reinforcement learning (SKL-RL), training agents to autonomously extract state-grounded predictive knowledge from experience and learn to leverage it for policy making. Experiments on WebShop, ScienceWorld, and more complex task ChessPuzzles show that SKL provides a scalable way to organize and internalize reflective computation, yielding performance gains under the stated experience and simulation budgets.
- Primary area:
- reinforcement learning
- Keywords:
- Reinforcement Learning, LLM Agents