- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the endogenous state problem in multi-turn agentic reinforcement learning (RL), where final task success conflates an agent's ability to reach useful/solvable states with its ability to solve the task once there. To disentangle these two factors without retraining, the authors introduce checkpoint handoff, an evaluation protocol where one checkpoint (the reacher) acts up to a verified frontier (or recorded point), and another checkpoint (the solver) continues from an exact, environment-verified clone of that state.
Critical strengths
Conceptual Innovation & Clarity: The formalization of the "endogenous state problem" neatly isolates why traditional endpoint metrics fail to attribute RL gains. The proposed checkpoint handoff protocol provides a rigorous, interventionist framework (leveraging potential outcomes and factorial interactions) to separate state visitation (REACH) from state-conditional value (SOLVE) without requiring model retraining.
Thorough Empirical Validation: The claims are tested across multiple scales (1.5B, 3B, 7B models on TravelPlanner) and independent training pipelines (Agent-STAR and SkillRL on ALFWorld), supported by robust audit controls, rigorous replay verification checks, and out-of-sample component predictions.
Critical weaknesses
Scope of Architecture Base: As noted in the reproducibility details, all evaluated checkpoints belong to variations of the broad Qwen family, leaving open whether these specific reach/solve interaction dynamics generalize to completely unrelated base-model architectures.
Late vs. Early Handoff Divergence: The TravelPlanner protocol uses a late handoff rule immediately before the reacher's final model call, whereas ALFWorld uses an early-action verified frontier (). While both show positive interactions, the mechanics of state representation and token histories differ significantly between tool-calling text planning and symbolic grid/household environments.
Questions for authors
Clarification on Budget Scaling: Given that the ALFWorld evaluation focuses heavily on a 2-step frontier () under a strict 2-action continuation budget, how sensitive are the relative proportions of REACH versus SOLVE if the handoff frontier distance is extended to larger horizons (e.g., or )?
Future Work Direction: Since the 7B scale on TravelPlanner showed a slight shrinkage in the interaction term due to the RL history carrying more intrinsic weight even for SFT solvers, do you anticipate that scaling beyond 7B will shift the dominant performance bottleneck entirely back to the solver or maintain reacher dominance?