acceptodds
Under review as a conference paper at ICLR 2027

Reach or Solve? Deep Diving into Agentic RL Gains with Checkpoint Handoffs

Abstract

Reinforcement learning (RL) is widely used to improve language-model agents, and its gains are usually measured by final task success. However, an agent's earlier actions shape the states in which its later decisions are made, so final task success conflates the ability to reach useful states with the ability to complete the task once there. Comparing agents only on the states each one reaches does not separate the two, since each agent is then scored on states selected by its own actions. To address this conflation, we introduce checkpoint handoff, an evaluation protocol that decouples reaching from completing without retraining. One checkpoint acts as a *reacher* up to a handoff point, and another continues as the *solver* from the same replayed history. In detail, (1) **Reach** measures how often a reacher arrives at states that a replayable environment verifies to be a fixed number of actions from success, and (2) **Solve** measures how often a solver completes the task from identical cloned copies of those states. Crossing supervised fine-tuning (SFT) and RL checkpoints in both roles across two benchmarks and two independently released training pipelines, we find that the gain from switching the solver from SFT to RL is consistently larger when RL is the reacher, at all three model scales on TravelPlanner and on both ALFWorld splits. Further analyses on ALFWorld show that RL improves both **Reach** and **Solve**. The solver gain is larger under an RL reacher because RL reaches solvable states more often, and separately measured **Reach** and **Solve** gaps recover most of this difference. Because handoff only requires replaying one checkpoint's history under another, agentic RL evaluations can report arrival and completion alongside final success.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, LLM Agents, Agent Evaluation, Causal Attribution, Environment State

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xzCFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the endogenous state problem in multi-turn agentic reinforcement learning (RL), where final task success conflates an agent's ability to reach useful/solvable states with its ability to solve the task once there. To disentangle these two factors without retraining, the authors introduce checkpoint handoff, an evaluation protocol where one checkpoint (the reacher) acts up to a verified frontier (or recorded point), and another checkpoint (the solver) continues from an exact, environment-verified clone of that state.

Critical strengths

Conceptual Innovation & Clarity: The formalization of the "endogenous state problem" neatly isolates why traditional endpoint metrics fail to attribute RL gains. The proposed checkpoint handoff protocol provides a rigorous, interventionist framework (leveraging potential outcomes and factorial interactions) to separate state visitation (REACH) from state-conditional value (SOLVE) without requiring model retraining.

Thorough Empirical Validation: The claims are tested across multiple scales (1.5B, 3B, 7B models on TravelPlanner) and independent training pipelines (Agent-STAR and SkillRL on ALFWorld), supported by robust audit controls, rigorous replay verification checks, and out-of-sample component predictions.

Critical weaknesses

Scope of Architecture Base: As noted in the reproducibility details, all evaluated checkpoints belong to variations of the broad Qwen family, leaving open whether these specific reach/solve interaction dynamics generalize to completely unrelated base-model architectures.

Late vs. Early Handoff Divergence: The TravelPlanner protocol uses a late handoff rule immediately before the reacher's final model call, whereas ALFWorld uses an early-action verified frontier (F2\mathcal{F}_{2}). While both show positive interactions, the mechanics of state representation and token histories differ significantly between tool-calling text planning and symbolic grid/household environments.

Questions for authors

Clarification on Budget Scaling: Given that the ALFWorld evaluation focuses heavily on a 2-step frontier (F2\mathcal{F}_{2}) under a strict 2-action continuation budget, how sensitive are the relative proportions of REACH versus SOLVE if the handoff frontier distance DD is extended to larger horizons (e.g., D=4D=4 or D=8D=8)?

Future Work Direction: Since the 7B scale on TravelPlanner showed a slight shrinkage in the interaction term due to the RL history carrying more intrinsic weight even for SFT solvers, do you anticipate that scaling beyond 7B will shift the dominant performance bottleneck entirely back to the solver or maintain reacher dominance?

Reviewer_s1kKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates where the gains from RL post-training in LLM agents are because the policy is better at reaching good states that setup the agent to solve the problem, or whether the agent is better at the solve part. The author suggest that the RL-trained agent is better at reaching, while on ALFworld improvements of RL are on both.

Critical strengths

The idea of the paper to try to better understand where the gains from RL come from is good, however the paper fails to account for a vast amount of relevant literature and known results on this.

Critical weaknesses

The result is not novel, and well known from classic online vs offline RL literature (Ross et al. 2011; Levine et al. 2020). The reason for this is that in the online RL (here akin to RL) the agents learn from their own experience, and learn to recover from bad states, while in offline RL (in its simplest form behavioral cloning, akin to SFT here), the agents are trained to follow expert data, or in any case data that comes from another policy, any drift from that policy or data makes the learning agent fall into states that it didn't learn to recover from through its own experience, which explains why the RL policy is better at reaching and recovering (the authors mention that SFT policy gets stuck in loops, which is expected). The solve state here is also somewhat arbitrarily chosen, and seemingly very easy to win from, and most of the difficulty is placed on reach.

Another problem here is that the SFT policy is overall just very weak (5-6% progression on ALFworld).

References:

Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.

Levine, Sergey, et al. "Offline reinforcement learning: Tutorial, review, and perspectives on open problems." arXiv preprint arXiv:2005.01643 (2020).

Questions for authors

For future work, it would be useful for the authors to engage and compare more algorithms from well known RL and imitation learning literature to actually disentangle where the gains are, beyond what's already known. It would be interesting to test a compute matched DAgger, as well as STaR (Zelikman et al. 2022) to RL, and see how they compare. DAgger uses an expert policy to label states that the learning agent visits, which should help it recover from bad positions it reached (without RL reward). STaR basically trains by bootrapping the policy on its own best trajectories, discarding failures, so it does not explicitly teach to to recover from bad states.

References: Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.

Reviewer_3mgeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that final task success for LLM agents mixes two abilities: reaching useful states and completing the task from them. It proposes "checkpoint handoff," where one model checkpoint (the reacher) acts until it is a fixed number of steps from success, and another (the solver) continues from an exact copy of that state. Crossing SFT and RL checkpoints on TravelPlanner and ALFWorld, the authors find that RL's solver advantage is larger when RL is also the reacher, that RL reaches near-goal states far more often than SFT, and that this explains most of the observed difference.

Critical strengths

Careful experimental control. Solvers continue from exact clones of the same environment state, history, and budget with paired seeds, so the solver comparison is clean and well controlled.

Reproducibility. Released checkpoints from two independent pipelines are used without retraining, and code, configurations, and continuation records are provided.

Critical weaknesses

1. The reach/solve distinction is arbitrary. In an MDP, the state is a sufficient statistic of the past: if earlier actions lead to bad states, the agent must deal with them, and the value of a state already captures how well a policy completes the task from it. Splitting a trajectory into "reach" and "solve" is simply a choice of cut point, not a property of the task. The cut is fixed at two steps from success (D = 2) and never varied, so the reacher plays nearly the entire episode; attributing RL's gain to "reaching" is therefore largely built into the design. On TravelPlanner, the cut is instead placed before the final model call, so "solve" means something different across the two benchmarks.

2. Limited contribution. The main takeaway is that RL agents outperform SFT agents, which is already well established. RQ1 essentially asks whether the state distribution matters when following the RL versus the SFT policy; the answer is obviously yes, since a policy's state-visitation distribution depends on the policy (cf. covariate shift in imitation learning and the performance difference lemma). The paper does not show that the reach/solve profile reveals anything actionable beyond the endpoint success rate.

3. Poor clarity.

  • "Checkpoint" is not defined in the introduction. The abstract suggests a model checkpoint, but at L120 ("the checkpoint sees only hth_t") it is unclear whether it refers to the model or something else.
  • The reacher/solver terminology is confusing: in an MDP, what distinguishes a "reach" state from a "solve" state?
  • L166: F2F_2 and the "solvable frontier" are used before being defined (§3.1). Why is being two steps from success special, and why is completing those two steps called "solving"?
Questions for authors

How do you define a reacher and solver for a given domain? What is special about reacher part as opposed to solver part?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.