acceptodds
Under review as a conference paper at ICLR 2027

CompeteRL: Foresight–Hindsight Competitive Reinforcement Learning

Abstract

Experiential learning is becoming tightly integrated with agentic reinforcement learning (RL), which uses task outcomes to directly optimize experience utilization and extraction. However, existing methods separate experience extraction from utilization across rollouts, delaying feedback of newly extracted experience and leaving retrieved historical experience poorly adapted to the current task. To tackle the challenge, we introduce **Experience-in-the-Loop Learning**, a formulation that turns experience extraction and utilization into trainable decisions within the policy rollout. Building on this formulation, we propose CompeteRL, an online RL framework that integrates experience generation, utilization, and validation within a unified rollout, while improving experience quality through foresight–hindsight competition. Specifically, \ours adapts retrieved knowledge into foresight experience before acting and distills trajectory-grounded hindsight experience afterward. The two experiences compete through their guided execution returns, forming a closed loop of task-specific adaptation and immediate feedback. Extensive experiments across four agent benchmarks using Qwen2.5-7B and Qwen3-4B show that CompeteRL **(I) delivers strong task performance**, exceeding Skill1 by 6.1% on average with Qwen3-4B, and **(II) lies on the Pareto-optimal frontier** of cost and performance across four datasets, outperforming SkillRL by 7.4% on ALFWorld at comparable token cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, skill

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9PFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles delayed feedback on experience utility and insufficient adaptation of retrieved experience to the current task. To this end, they introduce Experience-in-the-Loop Learning, which jointly optimizes experience generation and utilization within the rollout, and propose CompeteRL, a framework built on this formulation that improves foresight and hindsight experience through task-matched competitive feedback. The framework first adapts retrieved experience into task-specific foresight guidance, then distills hindsight experience from the resulting execution and evaluates it by rerunning the same task from its initial state. The paired execution returns provide competitive rewards that encourage the two forms of experience to co-improve, while the actor learns from its own execution returns. While the idea to use foresight, hindsight, and their combination is not new, their proposed co-improvement through competitive feedback from paired execution is interesting; however, with some missing ablations that support the effectiveness of each component, it remains unclear whether each of them contribute to the final gain.

Critical strengths
  1. The proposed idea is well-motivated and clean. While existing works often separate experience generation from its subsequent evaluation, they make foresight and hindsight to co-improve through competition, using direct execution feedback to optimize both experience generation and utilization. The ablations partially support the benefit of each mechanism, with the main results outperforming all the baselines.
Critical weaknesses
  1. The existing analyses do not sufficiently validate the effectiveness of the proposed foresight-hindsight co-improvement mechanism. The “w/o hindsight” ablation in Table 3 removes both experience distillation and its guided re-execution, while “2x actor GRPO (w/o skill)” uses no skills, making it unclear whether the gains arise from directly optimizing hindsight generation through re-execution feedback or simply from generating and using the hindsight guidance. As co-improvement is one of the central contributions of this work, the ablation beyond ablating each branch (which supports the need for each branch, not their co-improvement) is needed.

  2. The main results are compared with baseline results obtained using different evaluation protocols, which makes it unfair to claim superiority over these baselines without evaluating them under the same conditions. The authors explicitly acknowledge that some results taken from prior work cannot be interpreted as comparisons under identical settings, yet provide no identifiers or explanations. In agentic tasks, differences in protocols and harnesses can severely affect performance even with the same model, and ALFWorld, with only 134 evaluation tasks, is particularly sensitive to changes. Moreover, Skill1’s original evaluation reports better performance than CompeteRL on several tasks (while training config is slightly different), making this difficult to dismiss.

Questions for authors
  1. CompeteRL seems to require more computation than other baselines due to its additional hindsight-guided pass. How does its training cost compare in terms of wall clock time (or number of tokens) for competitive baselines (like Skill1)? Do the performance gains persist under matched compute budgets?
Reviewer_1VMPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CompeteRL adapts retrieved skills into a foresight skill, acts, distills a hindsight skill from that trajectory, and re-executes the same reset task under the hindsight skill. The two returns compete inside one GRPO update. On four agent benchmarks and two Qwen backbones, it reports the best averages in its tables, including +6.1 over its Skill1 entry with Qwen3-4B.

Critical strengths

The same-task reset seems to give the hindsight skill an immediate return. The ablation looks like more than simply spending another actor pass.

Critical weaknesses

The comparison tables seem to mix this paper’s split and decoding setup with numbers taken from other protocols. The competition argument also looks like it describes a different estimator from the update that is trained.

Questions for authors
  1. Why score the hindsight skill by resetting the same task, rather than testing it on an instance the foresight pass never saw?
  2. Why analyze the competition term as a REINFORCE baseline, rather than under the GRPO update the algorithm actually trains?
  3. Why report the cited skill agents from their published protocols, rather than re-running them under this paper’s task split and decoding setup?
Reviewer_6XSCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They propose a new RL method, CompeteRL for learning skills. In the approach, there are multiple steps: the model first adapts retrieved skills to the current task (foresight) and attempts it. It then writes a new lesson from that attempt (hindsight) and retries the same task. They evaluate two models, Qwen2.5-7B and Qwen3-4B, on ALFWorld, WebShop, MiniHack and ScienceWorld. And show that CompeteRL outperforms existing methods including skill-based RL methods like Skill1 and SkillRise.

Critical strengths
  • Interesting RL method, having foresight and hindsight
  • Comprehensive comparison and evaluation across ALFWorld, WebShop, MiniHack and ScienceWorld
Critical weaknesses

My main concern is that the improvement over the strongest skill-based RL baselines looks quite small (about 1.5-4.6 points), and there are no repeated runs or variance reported including statistical significance. Since the method adds multiple extra steps per rollout (foresight writing, hindsight writing, and two executions), I think it’s important to compare the cost against Skill1 and SkillRise, not only just GRPO and GiGPO (for example, training cost and total tokens including inputs, etc.) It’d also be helpful to discuss the method differences with Skill1 and SkillRise in more detail, and to check whether those baselines would close the gap with the same compute. Also, some baseline numbers seem to be taken from prior work with different settings, so it’s hard to tell how fair the comparison is.

Questions for authors

Please see the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.