- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RetroAgent for sparse-reward agent RL, where terminal success does not distinguish useful partial exploration from unproductive failure and successful experience is difficult to reuse explicitly. After each episode, a reflection module produces two intrinsic signals: numerical feedback rewards progress above a historical task-success threshold, while language feedback distills successes and informative failures into lessons stored in memory. SimUtil-UCB retrieves lessons using semantic similarity, exponential-moving-average utility, and an exploration bonus. Across ALFWorld, WebShop, Sokoban, and Minesweeper, the proposed framework achieves strong and generally consistent improvements over the reported RL, memory-augmented, and meta-RL baselines, with particularly notable gains on the more planning-intensive environments.
Critical strengths
- The paper presents a coherent framework that addresses two complementary limitations of sparse-reward agent learning. Its numerical feedback is intended to distinguish promising partial progress from uninformative failures, while its language feedback converts experience into explicit lessons that can be reused across episodes. Integrating directed exploration and persistent experience reuse within one online RL pipeline is practically meaningful and constitutes the paper's main conceptual strength.
- The empirical evaluation is broad for an agent-RL study. It covers four environments with different interaction and reasoning demands, compares against prompting, RL, memory-augmented, exploration-oriented, and meta-RL methods, and reports consistently strong improvements. The component ablations, retrieval comparisons, OOD evaluation, and test-time adaptation analysis provide useful evidence that the observed gains are not limited to a single benchmark or evaluation setting.
Critical weaknesses
The intrinsic numerical reward relies on the self-generated potential score , but the paper does not provide a reliable training-time mechanism for supervising or calibrating this quantity. In the RL-trained reflection variant, the reflection reward directly supervises only the binary success prediction , even though is subsequently used as a reward signal for policy optimization. Moreover, represents a trajectory-level estimate of subtask completion, whereas is a task-level empirical success rate over a group of rollouts. Although both are normalized to , they do not have the same semantics or necessarily share a calibrated scale. Consequently, the difference lacks a sufficiently justified interpretation as capability improvement, and errors in may directly induce misleading policy updates. The post-hoc comparison against GPT-4o scores does not resolve this issue during training.
Questions for authors
How does RetroAgent behave when trained from substantially smaller or weaker base models whose self-reflection and progress-estimation abilities are less reliable? Please evaluate the complete training pipeline with weaker backbones under matched interaction and optimization budgets, and report both downstream task performance and the calibration or accuracy of throughout training. The comparison should include extrinsic-only GRPO and, if feasible, a version using externally estimated or oracle progress scores. In particular, does error in accumulate or become self-reinforcing during optimization? This experiment would help determine whether RetroAgent genuinely improves weak agents or presupposes a sufficiently capable model that can already generate reliable intrinsic rewards.