acceptodds
Under review as a conference paper at ICLR 2027

RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

Abstract

Standard reinforcement learning (RL) for large language model (LLM) agents primarily optimizes extrinsic task rewards, often favoring one-off task completion over continual adaptation. This paradigm can cause premature convergence to suboptimal policies and leaves useful experience only implicitly encoded in model parameters, limiting its retrieval and reuse for future decisions. We introduce RetroAgent, an online RL framework that trains agents to master interactive environments not merely by solving tasks, but by evolving across episodes. Inspired by human retrospective self-improvement, RetroAgent augments extrinsic rewards with hindsight-generated dual intrinsic feedback: (1) Intrinsic Numerical Feedback, which rewards beneficial exploration by measuring incremental subtask progress relative to prior attempts; and (2) Intrinsic Language Feedback, which distills successes and failures into reusable textual lessons for explicit experience reuse. To leverage these lessons effectively, we propose Similarity & Utility-Aware Upper Confidence Bound (SimUtil-UCB), a retrieval strategy that balances semantic relevance, historical utility, and exploration. Across four challenging agentic benchmarks, RetroAgent achieves new state-of-the-art (SOTA) performance, outperforming GRPO by +18.3% on ALFWorld, +15.4% on WebShop, +27.1% on Sokoban, and +8.9% on MineSweeper, while demonstrating strong test-time adaptation and out-of-distribution (OOD) generalization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
online reinforcement learning, dual intrinsic feedback, reflective agent reasoning, continual adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9qD6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RetroAgent for sparse-reward agent RL, where terminal success does not distinguish useful partial exploration from unproductive failure and successful experience is difficult to reuse explicitly. After each episode, a reflection module produces two intrinsic signals: numerical feedback rewards progress above a historical task-success threshold, while language feedback distills successes and informative failures into lessons stored in memory. SimUtil-UCB retrieves lessons using semantic similarity, exponential-moving-average utility, and an exploration bonus. Across ALFWorld, WebShop, Sokoban, and Minesweeper, the proposed framework achieves strong and generally consistent improvements over the reported RL, memory-augmented, and meta-RL baselines, with particularly notable gains on the more planning-intensive environments.

Critical strengths
  1. The paper presents a coherent framework that addresses two complementary limitations of sparse-reward agent learning. Its numerical feedback is intended to distinguish promising partial progress from uninformative failures, while its language feedback converts experience into explicit lessons that can be reused across episodes. Integrating directed exploration and persistent experience reuse within one online RL pipeline is practically meaningful and constitutes the paper's main conceptual strength.
  2. The empirical evaluation is broad for an agent-RL study. It covers four environments with different interaction and reasoning demands, compares against prompting, RL, memory-augmented, exploration-oriented, and meta-RL methods, and reports consistently strong improvements. The component ablations, retrieval comparisons, OOD evaluation, and test-time adaptation analysis provide useful evidence that the observed gains are not limited to a single benchmark or evaluation setting.
Critical weaknesses

The intrinsic numerical reward relies on the self-generated potential score ϕ(x,τ)\phi(x,\tau), but the paper does not provide a reliable training-time mechanism for supervising or calibrating this quantity. In the RL-trained reflection variant, the reflection reward directly supervises only the binary success prediction cc, even though ϕ\phi is subsequently used as a reward signal for policy optimization. Moreover, ϕ(x,τ)\phi(x,\tau) represents a trajectory-level estimate of subtask completion, whereas Φx(k)\Phi_x^{(k)} is a task-level empirical success rate over a group of rollouts. Although both are normalized to [0,1][0,1], they do not have the same semantics or necessarily share a calibrated scale. Consequently, the difference [ϕ(x,τ)−Φx(k)]+[\phi(x,\tau)-\Phi_x^{(k)}]_+ lacks a sufficiently justified interpretation as capability improvement, and errors in ϕ\phi may directly induce misleading policy updates. The post-hoc comparison against GPT-4o scores does not resolve this issue during training.

Questions for authors

How does RetroAgent behave when trained from substantially smaller or weaker base models whose self-reflection and progress-estimation abilities are less reliable? Please evaluate the complete training pipeline with weaker backbones under matched interaction and optimization budgets, and report both downstream task performance and the calibration or accuracy of ϕ\phi throughout training. The comparison should include extrinsic-only GRPO and, if feasible, a version using externally estimated or oracle progress scores. In particular, does error in ϕ\phi accumulate or become self-reinforcing during optimization? This experiment would help determine whether RetroAgent genuinely improves weak agents or presupposes a sufficiently capable model that can already generate reliable intrinsic rewards.

Reviewer_DdpKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RetroAgent, an online RL framework for training LLM agents to learn from both task outcomes and retrospective feedback across episodes. Specifically, RetroAgent augments outcome-based rewards with two forms of intrinsic feedback: numerical feedback and language feedback. Intrinsic numerical feedback rewards incremental subtask progress relative to prior attempts, while intrinsic language feedback distills past experiences into reusable lessons through self-reflection. The authors further propose SimUtil-UCB, a memory retrieval strategy that balances semantic relevance, historical utility, and exploration. Experiments on four benchmarks demonstrate the effectiveness of RetroAgent compared to other RL baselines.

Critical strengths

This paper presents an interesting approach to leveraging experiences collected during RL training. In particular, each sampled episode not only contributes to the scalar reward but also serves as a source of retrospective feedback that can guide future training. Moreover, RetroAgent also explicitly trains the self-reflection capability, leading to stronger generalization and test-time scaling of the trained agent.

Besides, the paper provides a comprehensive empirical evaluation across four different environments, comparing RetroAgent against a broad range of RL, memory-augmented, and meta-RL baselines. The consistent improvement indicates good empirical results of the proposed method.

Critical weaknesses

First, The presentation of the method could be substantially improved, particularly in Sections 3.2–3.4. The relationships between intrinsic numerical feedback, memory retrieval, and reflection optimization are not clearly explained. The overall training process is therefore less clear and hard to follow.

Moreover, the design choice of SimUtil-UCB is not well motivated and justified. It remains unclear why SimUtil-UCB should be preferred over other possible memory retrieval strategies. Although the authors provide comparisons with similarity-based and similarity–utility retrieval, comparisons with simpler exploration strategies (e.g., random or frequency-balanced retrieval) are missing.

Finally, I think there exist a mismatch between the refleciton objective and the proposed intrinsic feedback. In RetroAgent, the reflector generates a potential score, a success prediction, and a lesson, i.e., (ϕ\phi,cc,ll). However, the reflection reward only evaluates the correctness of cc. But potential score ϕ\phi and lessons ll are the actual outputs used to guide the future training and sampling.

Questions for authors

What is the connection between RetroAgent and existing RL approaches based on intrinsic curiosity [1, 2] or experience replay [3]? It would be interesting to discuss how RetroAgent relates to these methods and what distinguishes it from them.

How reliable and generalizable is the LLM-estimated potential score ϕ(x,τ)\phi(x,\tau) across different environments? Have the authors considered using a learned value function to estimate progress at each state as an alternative?

[1] Curiosity-driven Exploration by Self-supervised Prediction. ICML, 2017

[2] Never Give up: Learning Directed Exploration Strategies. ICLR, 2020

[3] Efficient RL Training for LLMs with Experience Replay. ICML, 2026

Reviewer_z4SjOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RetroAgent, an online reinforcement learning framework that uses retrospective reflection to generate two complementary forms of feedback: an intrinsic numerical reward based on estimated task progress, and textual lessons/experience stored for subsequent retrieval. The numerical reward compares a trajectory’s completion score against a task-specific historical success-rate threshold. The proposed SimUtil-UCB retrieval mechanism combines semantic relevance and estimated utility. The framework trains the policy with GRPO and optionally trains reflection through a REINFORCE objective rewarding correct outcome predictions. Across four challenging agentic benchmarks, RetroAgent achieves new state-of-the-art (SOTA) performance.

Critical strengths
  1. A coherent integration of learning and memory. The framework connects retrospective evaluation, policy optimization, and explicit experience retrieval. 
  2. Evaluation is broad. The paper evaluates four environments and two model families. It includes analyses of reward construction, retrieval, and reflection mechanisms. It also reports cases where combining feedback or training reflection degrades performance.
Critical weaknesses
  1. The framework appears primarily compositional, showing limited evidence of synergy between the proposed components. First, Table 2 shows that introducing discounted returns alone increases WebShop success from 66.9% to 74.7%, which means that the headline improvement over the original GRPO baseline includes a substantial change to return estimation. Second, the in-context dual-feedback variant achieves 78.9%, compared with 79.7% for numerical feedback alone in Table 3.

  2. The paper’s contribution positioning is incomplete. Previous study Critique-GRPO also introduces the language feedback that overlaps significantly with the contributions of this paper, and Complementary RL also seperates rollout into two sub-groups with and without textual experience.

  3. The models are too old (about 2 years ago). Both Qwen2.5 and the Llama 3 series were released more than 2 years ago. Providing results on new models would be better.

Questions for authors
  1. In Table 9, training reflection reduces Llama’s Sokoban success from 39.1% to 24.5%. What's the reason of substantial degradation?

  2. How is the generated feedback used? Will it be used in the remaining half of the group?

  3. Why do you compute the advantage over all N samples? The prompt differs depending on whether one trajectory is experienced or inexperienced.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.