- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles delayed feedback on experience utility and insufficient adaptation of retrieved experience to the current task. To this end, they introduce Experience-in-the-Loop Learning, which jointly optimizes experience generation and utilization within the rollout, and propose CompeteRL, a framework built on this formulation that improves foresight and hindsight experience through task-matched competitive feedback. The framework first adapts retrieved experience into task-specific foresight guidance, then distills hindsight experience from the resulting execution and evaluates it by rerunning the same task from its initial state. The paired execution returns provide competitive rewards that encourage the two forms of experience to co-improve, while the actor learns from its own execution returns. While the idea to use foresight, hindsight, and their combination is not new, their proposed co-improvement through competitive feedback from paired execution is interesting; however, with some missing ablations that support the effectiveness of each component, it remains unclear whether each of them contribute to the final gain.
Critical strengths
- The proposed idea is well-motivated and clean. While existing works often separate experience generation from its subsequent evaluation, they make foresight and hindsight to co-improve through competition, using direct execution feedback to optimize both experience generation and utilization. The ablations partially support the benefit of each mechanism, with the main results outperforming all the baselines.
Critical weaknesses
-
The existing analyses do not sufficiently validate the effectiveness of the proposed foresight-hindsight co-improvement mechanism. The “w/o hindsight” ablation in Table 3 removes both experience distillation and its guided re-execution, while “2x actor GRPO (w/o skill)” uses no skills, making it unclear whether the gains arise from directly optimizing hindsight generation through re-execution feedback or simply from generating and using the hindsight guidance. As co-improvement is one of the central contributions of this work, the ablation beyond ablating each branch (which supports the need for each branch, not their co-improvement) is needed.
-
The main results are compared with baseline results obtained using different evaluation protocols, which makes it unfair to claim superiority over these baselines without evaluating them under the same conditions. The authors explicitly acknowledge that some results taken from prior work cannot be interpreted as comparisons under identical settings, yet provide no identifiers or explanations. In agentic tasks, differences in protocols and harnesses can severely affect performance even with the same model, and ALFWorld, with only 134 evaluation tasks, is particularly sensitive to changes. Moreover, Skill1’s original evaluation reports better performance than CompeteRL on several tasks (while training config is slightly different), making this difficult to dismiss.
Questions for authors
- CompeteRL seems to require more computation than other baselines due to its additional hindsight-guided pass. How does its training cost compare in terms of wall clock time (or number of tokens) for competitive baselines (like Skill1)? Do the performance gains persist under matched compute budgets?