RICE: Response-Informed Context Evolution for Agent Learning

Interactive agents increasingly summarize completed trajectories as natural-language skills and reuse them across policy updates. This persistence allows past experience to support later decisions, but it also creates a mismatch: stored skill content can remain available while the policy that interprets it continues to change. Task outcomes evaluate the quality of sampled behavior, yet they do not record how retrieved context changes the current policy's predictions. On ALFWorld/Qwen3-1.7B, $52.1\%$ of the recorded response histories associated with stored skills have tied outcome differences; within this subset, the 95th percentile of the response magnitude is $3.22$ times the 5th percentile. This observation shows that outcomes and policy responses describe different properties of the same retrieval event. We define the local policy response as the difference between next-token distributions with and without retrieved context at the same decision prefix. We then introduce Response-Informed Context Evolution (), which accumulates these responses across retrieval events and associates the resulting history with the participating skills. Returns select which observed behavior provides supervision, response history determines how strongly accepted examples contribute, and response history together with past outcome evidence controls which skills remain available for future retrieval. Paused skills remain stored and can return to the available set when their outcome utility recovers. Across three model sizes on ALFWorld, WebShop, and Search-QA, achieves the highest listed overall success on ALFWorld and exact success on WebShop, while the Search-QA comparisons are mixed. In a controlled ALFWorld/Qwen3-1.7B comparison, validation success increases from $92.4\%$ for the Base configuration to $94.5\%$ with response weighting and $96.9\%$ with response-informed retrieval control. These results support treating retrieval as both context provision and feedback about how stored skills affect an evolving policy.