acceptodds
Under review as a conference paper at ICLR 2027

SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning

Abstract

Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce State-Rooted Executable Rollout Optimization (SERO), which augments on-policy Agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies policy-reachability weighting, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Long-Horizon Decision Making, Off-Policy Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TMduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SERO augments agentic RL with executable oracle continuations starting from states visited by the current policy. It learns oracle actions through an auxiliary supervised loss, weighting each action by the policy’s compatibility with preceding oracle actions. Results on ALFWorld and WebShop are promising.

Critical strengths

The motivation is clear: better credit assignment cannot supply useful behaviors absent from sampled trajectories. Executing oracle continuations provides additional training targets under the observations those actions actually produce.

Critical weaknesses
  1. The novelty relative to expert-continuation training is insufficiently established. On-policy Expert Corrections (OEC, Lauffer et al., 2025) already lets a student begin a trajectory, switches to an expert, and trains on the expert continuation. SERO adds joint RL optimization and prefix weighting, but does not directly compare against this closely related approach. A matched baseline combining the same RL backbone with OEC-style supervision on successful expert continuations, without compatibility weighting and with a tuned auxiliary coefficient, is needed.

  2. It is unclear why hard-to-reach states should receive less supervision. These states may contain useful behaviors that the policy has not yet learned. Giving them more supervision could therefore also help. The paper compares SERO with uniform, depth-decay, and shuffled weights, but does not test the opposite strategy: assigning more weight to states with low prefix compatibility. Such a comparison would better justify the proposed weighting rule.

Questions for authors

Could the authors provide the additional experiments suggested in the Critical Weaknesses section to address these concerns?

Reviewer_5BZoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the exploration limitations of long-horizon agentic reinforcement learning, where conventional credit assignment methods cannot provide supervision for successful behaviors that the current policy fails to discover. The authors propose SERO, which restores states visited by the student policy and executes multi-step oracle continuations to generate additional supervision. The method introduces a branch-improvement gate to filter non-improving continuations and policy-reachability weighting to adjust supervision based on the student's compatibility with oracle action prefixes. The resulting auxiliary loss is combined with existing RL objectives. Experiments on ALFWorld and WebShop using Qwen2.5-1.5B/7B models demonstrate improvements over several RL baselines, supported by component ablations and additional control experiments.

Critical strengths
  1. The method is clearly formulated and well presented. The paper provides a clear mathematical formulation of executable oracle continuations, branch-improvement gating, and policy-reachability weighting. The overall workflow is illustrated in Figure 2, and the training objective is explicitly defined in Equations (3)–(10), making the overall training procedure easy to follow.

  2. The method integrates naturally with existing RL algorithms. SERO introduces an auxiliary supervision loss without modifying the original reward computation, advantage estimation, or policy optimization objective. This allows it to be incorporated into different RL backbones, as demonstrated with GraphGPO and HGPO. Moreover, oracle access is required only during training, introducing no additional inference-time overhead.

  3. The paper provides targeted ablations to support its design choices. The authors separately evaluate root-only supervision, full oracle continuations, and policy-reachability weighting. The fixed-oracle-bank experiments further compare different weighting strategies under identical collected trajectories, while additional controls examine the effect of extra environment interactions. These experiments help isolate the contributions of the main components.

Critical weaknesses
  1. The evaluation relies on relatively dated models. The main training experiments use only Qwen2.5-1.5B/7B-Instruct, while the closed-source prompting baselines, GPT-4o and Gemini-2.5-Pro, are also not representative of the latest frontier models. Given recent advances in reasoning and agentic capabilities, stronger models may have different exploration limitations and require less oracle supervision. It remains unclear whether SERO would provide comparable benefits on newer or larger-scale models.

  2. The distinction from existing on-policy distillation methods is insufficiently established. SERO combines oracle-generated continuations with a weighted supervision objective, closely resembling existing multi-turn OPD and expert correction methods. Although it introduces branch-improvement gating and policy-reachability weighting, direct comparisons with representative methods are missing. This weakens the evidence for the method's novelty and advantages.

  3. Random root selection raises concerns about recoverability and sampling efficiency. SERO uniformly samples a state from a failed trajectory, but the selected state may no longer be recoverable within the remaining interaction budget. The branch-improvement gate filters non-improving continuations only after oracle execution, potentially incurring unnecessary computation. Appendix A.4 reports 78 positive-gate branches among 271 roots in a fixed-bank experiment, although this does not directly measure online sampling efficiency. The paper provides limited evidence that uniform root sampling is a cost-effective strategy, leaving the trade-off between useful supervision and oracle execution cost insufficiently explored.

  4. The effect of continuation length on policy efficiency is insufficiently investigated. Equation (9) sums supervision losses without normalizing by continuation length, allowing longer valid continuations to contribute more aggregate supervision. Moreover, the branch-improvement gate considers outcome improvement rather than path efficiency. Although policy-reachability weighting attenuates the contribution of later steps, the paper does not experimentally investigate whether these design choices affect the generated trajectory lengths or execution efficiency of the learned policy.

Questions for authors

Please see the weaknesses.

Reviewer_jx3rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SERO augments group-based agentic RL with an auxiliary loss that supplies behavior the policy never samples. It works in three steps:

From a failed on-policy rollout, it restores the simulator at a visited state and lets a training-only oracle execute a full continuation, admitted only if the oracle beats the policy’s own suffix return. Each oracle action is supervised with a likelihood loss, weighted by the frozen policy’s compatibility with the preceding oracle prefix. The base RL estimator is left unchanged.

On ALFWorld and WebShop with Qwen2.5-1.5B/7B and GraphGPO/HGPO backbones, SERO improves success, most notably on ALFWorld unseen tasks.

Critical strengths
  • Clear and well-motivated gap. The distinction between improving credit over sampled behavior Well-motivated gap. Group RL can re-rank sampled behavior but cannot supply missing behavior, and single-action teacher correction leaves the downstream consequences of an intervention unsupervised.
  • Clean, modular design. The auxiliary objective needs no importance correction and plugs into existing credit estimators.
  • Strong controls. The fixed-bank replay with shuffled weights shows that weight-prefix alignment, not just total mass or depth decay, drives the weighting gain (85.94 vs 80.73 unseen). The matched-interaction and short-bridge controls address obvious confounds.
Critical weaknesses
  • Missing key comparisons. The paper does not compare against the methods closest to it: on-policy distillation or DAgger using the same oracle. It also skips the simplest alternative, which is training on full oracle demonstrations from the start of each task. Without this, we cannot tell whether starting the oracle from states the policy actually visited is what helps, or whether the gain just comes from adding more expert data.

  • A surprising result is not explained. Training on only the first oracle action makes performance worse than not using the oracle at all, by 3 to 4 points. A correct expert action should not hurt, so this needs an explanation.

  • Little evidence that long continuations are actually used. The paper’s main idea is learning from many oracle steps. However, 85% of the oracle steps get a weight below 0.05, so in practice the model mostly learns from the first few steps. The extra gain from the weighting scheme over simple uniform weighting is also within noise.

  • Strong requirements, no limitations section. The method needs a simulator that can be reset to any past state and an oracle that is nearly perfect. Many real settings, such as real websites, tool use, or coding, do not have these. The paper does not test what happens with an imperfect oracle, does not report the full compute cost, and does not discuss these limits.

Questions for authors
  • Are the seeds independent training runs? Which baselines were rerun under your configurations?
  • How does SERO compare to OPD/DAgger and to BC on full oracle trajectories from the initial state, at a matched oracle budget?
  • Why does Root-only underperform the base? Is its gate computed from the full continuation’s return?
  • What is the mass-weighted effective supervision depth over training, and how does the fraction of positive-gate branches evolve as the policy improves?
  • How does performance degrade with a suboptimal or noisy oracle?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.