- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SERO augments agentic RL with executable oracle continuations starting from states visited by the current policy. It learns oracle actions through an auxiliary supervised loss, weighting each action by the policy’s compatibility with preceding oracle actions. Results on ALFWorld and WebShop are promising.
Critical strengths
The motivation is clear: better credit assignment cannot supply useful behaviors absent from sampled trajectories. Executing oracle continuations provides additional training targets under the observations those actions actually produce.
Critical weaknesses
-
The novelty relative to expert-continuation training is insufficiently established. On-policy Expert Corrections (OEC, Lauffer et al., 2025) already lets a student begin a trajectory, switches to an expert, and trains on the expert continuation. SERO adds joint RL optimization and prefix weighting, but does not directly compare against this closely related approach. A matched baseline combining the same RL backbone with OEC-style supervision on successful expert continuations, without compatibility weighting and with a tuned auxiliary coefficient, is needed.
-
It is unclear why hard-to-reach states should receive less supervision. These states may contain useful behaviors that the policy has not yet learned. Giving them more supervision could therefore also help. The paper compares SERO with uniform, depth-decay, and shuffled weights, but does not test the opposite strategy: assigning more weight to states with low prefix compatibility. Such a comparison would better justify the proposed weighting rule.
Questions for authors
Could the authors provide the additional experiments suggested in the Critical Weaknesses section to address these concerns?