acceptodds
Under review as a conference paper at ICLR 2027

When Every Rollout Fails: Splicing Learner-Conditioned Options into Long-Horizon Agent RL

Abstract

Group-based reinforcement learning with verifiable rewards often yields all-fail groups on long-horizon agent tasks: every sampled trajectory receives zero reward, leaving group-relative methods without a reward-based policy gradient. A common remedy supplies complete solutions from a stronger model, but full trajectories are expensive and expose the learner to long off-policy histories. We introduce SPLICE, which constructs local training targets from failed current-policy trajectories online. For each all-fail group, a proposal model selects a splice point in a current-policy trajectory and generates several bounded options from that point. We estimate each option's value using repeated current-policy continuations, convert positive estimates into a return-weighted local target, and fit the current policy on the options. We evaluate SPLICE across two policy families on two long-horizon agent tasks, Software Engineering with SWE-bench Verified and Deep Research with BrowseComp-Plus. SPLICE outperforms GRPO and full-trajectory guidance baselines in all four settings, improving over GRPO by 2.2–10.8 percentage points. On initially failed SWE tasks, it raises GRPO's graduation rate by 10.7 percentage points. Further experiments show that our method improves performance while reducing costly proposal-model computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reinforcement Learning, Long-Horizon Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HNiDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SPLICE to address the all-fail group problem in GRPO. A teacher model selects splice points in failed trajectories and generates bounded options, which are evaluated through student continuations to construct return-weighted local supervision. SPLICE is evaluated on two long-horizon agent tasks and outperforms the evaluated baselines. Ablation studies further examine the effectiveness of its key components.

Critical strengths
  1. The paper addresses the all-fail group problem in GRPO and proposes a method to recover learning signals.
  2. The method combines bounded teacher options with student continuations to provide local supervision.
  3. The proposed method outperforms the evaluated baselines on two agentic tasks.
Critical weaknesses
  1. SPLICE may be less effective on trajectories with multiple bottlenecks, since a useful local intervention can still be followed by failure at a later bottleneck.
  2. Under a limited continuation budget, options with low but nonzero success probabilities may be incorrectly discarded.
  3. The ablations do not separate direct teacher completions from learner-assisted recoveries, leaving the contribution of learner-conditioned evaluation unclear. When the teacher completes the task within an option, the benefit may come from supervised learning on that successful option.
Questions for authors
  1. The criteria and mechanism used by the teacher to select splice points need further clarification.
  2. A comparison using student-generated options from the same splice states, under a matched rollout budget, would help establish the benefit of teacher guidance.
Reviewer_eQgfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SPLICE, a method to improve GRPO training for hard prompts that result in all-failure groups. The idea is to use a strong proposal model to determine the splice point and then create 3 different options. The trained model will then sample 3 continuations from each option. If any of the continuations succeed, the option is scored with the mean reward, and the option will be used as an SFT objective for the trained model. Evaluated on SWE-bench Verified (OpenHands) and BrowseComp-Plus, with Qwen3-8B (after a cold-start SFT) and Qwen3.5-9B. SPLICE beats GRPO by 2.2–10.8 points and SFT+RL by 0.6–2.5 points.

Critical strengths
  • All failure groups contributing nothing to the gradient is one of GRPO's core issues. The paper is well motivated.
  • The idea is sensible as in allowing a stronger teacher model to provide a hint that guides the student model towards better completions
  • The effectiveness of the method is tested on two realistic benchmarks in different domains, showing the generalizability of the method.
Critical weaknesses
  • The method involves many hyperparameters that seem rather arbitrarily chosen. Why 3 options and 3 continuations? K and M are never ablated. Only the option horizon α (Table 4) and the splice-point rule (Table 6) are varied, and the snapshot interval C is fixed.
  • Several baselines are mentioned but never tested against, including POPE, LUFFY and DAPO-style oversampling.
  • Every experiment is a single run and given that the rollout to splice is selected randomly, variance could be large.
Questions for authors
  • How does the choice of K and M affect model performance?
Reviewer_KctEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a problem in long-horizon agent RL with GRPO: when every trajectory in a group fails, the rewards have no within-group variation, all advantages are zero, and no effective update can be produced. SPLICE retains the GRPO update for mixed-reward groups, skips all-success groups, and introduces local guidance only for all-fail groups. The proposal model does not generate a complete solution; instead, it produces K bounded options partway through a failed trajectory. Each option is followed by M continuations from the current policy, filtered and weighted according to observed task returns, and only the proposal tokens within the options are supervised. Across the four settings with Qwen3-8B-CS and Qwen3.5-9B on SWE-bench Verified and BrowseComp-Plus, SPLICE achieves 25.8, 29.8, 60.6, and 51.3, respectively, all exceeding GRPO and ReLIFT. On initially failed SWE tasks, its graduation rate is 18.4%, compared with 7.7% for RL and 15.4% for ReLIFT.

Critical strengths
  • Using the current policy rather than the proposal model to evaluate continuations directly tests whether the learner can build on the guidance, and this design choice is clear. In Table 4, switching to proposal continuations increases Resolve only from 25.8 to 26.2, while increasing proposal-model turns from 95.3k to 374.2k, suggesting that the proposed design offers a better trade-off between performance and compute.
  • The ablations of local supervision are relatively comprehensive. Single option, hard top-Q, uniform positive, and the default method achieve 22.0, 24.4, 23.6, and 25.8, respectively; extending supervision to the full suffix reduces performance to 22.8. These results support continuation evaluation, return weighting, and supervising only bounded options, respectively, rather than merely showing that adding teacher data helps.
  • The analysis of failed-task recovery and task difficulty is informative. SPLICE achieves an 18.4% graduation rate on initially failed tasks, compared with 7.7% for RL. On Medium tasks, performance rises from 0.5 for the initial policy to 23.5, whereas RL reaches 9.8. This suggests that the method mainly helps tasks that direct RL struggles to solve but that can still be completed after a local correction.
Critical weaknesses
  • The requirement for exact environment rollback limits generality. Sec 4.1 requires saving the environment state and interaction history every C=4 turns and restoring them from a selected snapshot. This is relatively easy in SWE and controlled retrieval environments, but exact rollback is often impossible for live web pages, external services, or environments with irreversible actions. Although Sec 6 acknowledges this limitation, no experiments examine whether the method remains effective when only partial or approximate restoration is possible. The current conclusions therefore mainly apply to training environments with snapshotting and reproducible states and cannot be directly generalized to broader online agent settings.
  • The method relies on a strong proposal model, and direct comparisons with distillation are missing. Both policy families use Qwen3.5-122B-A10B, with no comparison to a smaller proposal model and no test of having the current policy generate its own options. Sec 2.2 mentions RSTG, which combines GRPO with on-policy distillation for all-fail groups, but the experiments include neither OPD nor RSTG. SFT, SFT+RL, and ReLIFT mainly provide full-trajectory supervision and cannot substitute for these comparisons. It therefore remains unclear how much of the gain depends on the strong teacher and whether local option supervision outperforms on-policy distillation.
  • The key improvements lack evidence of stability. In Table 1, SPLICE outperforms the strongest baseline, SFT+RL, by only 0.6 to 2.5 points; in Table 6, proposal-model splice-point selection outperforms random selection by only 1.2 points. These gaps may be comparable to the natural variance of agent rollouts, making it difficult to determine whether the improvements are stable and whether they are due to direct distillation from the teacher model.
Questions for authors
  • SPLICE requires intermediate states to be saved and restored during training. If an environment can preserve only the interaction history but cannot exactly restore the external state, does the method remain applicable? What modifications are needed for tasks that do not support snapshots or involve irreversible actions?
  • All current experiments use Qwen3.5-122B-A10B to generate options. How much of SPLICE's gain is retained, and how much does performance degrade, when using a weaker proposal model or a teacher model from another family? Compared with other teacher-guided methods, how do SPLICE's performance and cost compare, and what is the source of its advantage?

If the authors can address these questions, I would consider raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.