acceptodds
Under review as a conference paper at ICLR 2027

Depth Before Breadth: Reverse-to-Forward Curriculum with Advantage Decomposition for LLM Agent

Abstract

Reinforcement learning (RL) has become a common approach for training large language model (LLM) agents on long-horizon tasks. However, learning from sparse terminal rewards makes credit assignment and exploration difficult. We approach these challenges through a policy-induced capability region, the set of states from which the current policy can reliably complete a task. Near its boundary, an agent may need only a few actions to reach a more solvable state, even when the goal remains distant. This reduces long-horizon learning to two coupled subproblems of reaching that state and completing the task from there. We address them with paired advantages and a progressive reverse-to-forward curriculum. The advantages provide separate signals for reaching more solvable states and completing the task from them, while the curriculum expands the capability region backward along successful reference trajectories and then into off-reference states through on-policy exploration. On ALFWorld and WebShop, our method outperforms competitive agentic RL methods at both model scales, achieving success rates of 90.1%/96.1% and 80.2%/82.6% with Qwen2.5-1.5B/7B-Instruct, respectively.

Primary area:
reinforcement learning
Keywords:
Long-horizon LLM agents, agentic reinforcement learning, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9gWbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a training method for long-horizon LLM agents that addresses sparse terminal rewards and the resulting credit assignment problem. It splits each rollout into a prefix and a suffix and gives them separate learning signals. Prefixes are trained based on how solvable the reached state is, while suffixes are trained based on task completion from that state. Training first moves backward along successful reference trajectories and then switches to states reached by the policy itself. The method is evaluated on ALFWorld and WebShop with Qwen2.5 1.5B and 7B models.

Critical strengths
  • The credit assignment idea is simple and well-motivated. Separating the signal for reaching a useful state from the signal for completing the task gives more informative supervision than assigning the same terminal reward to the entire trajectory.

  • The empirical results are strong and consistent across both benchmarks and model sizes. The ablation also shows that both the forward stage and paired advantages contribute to the gains. The paper is also clear and well-written.

Critical weaknesses
  • The method requires successful reference trajectories and the ability to reset the environment to intermediate states. The authors acknowledge this limitation, but these requirements may restrict where the method can be applied. It is also unclear how costly the reference trajectories are to obtain.

  • The final results show that the policy improves, but the evidence for the paper’s specific explanation is less direct. The endpoint analysis suggests that the policy learns to handle more off-reference states, but it does not directly measure success from those states. A more direct evaluation would make the capability-region claim more convincing.

Questions for authors
  1. The main contribution is the separation of prefix and suffix credit, but the ablation does not test these two signals separately. How does performance change when using only the prefix advantage or only the suffix advantage, while keeping the curriculum and interaction budget fixed?

  2. Table 1 states that the baseline results are taken from prior work, while the paper also says that all RL methods use the same training settings. Could the authors clarify which baselines were rerun and how closely the training tasks, interaction budgets, and model-selection procedures were matched?

Reviewer_wgxJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Depth Before Breadth, a reinforcement learning framework designed to train large language model agents on long-horizon tasks with sparse rewards by decomposing credit assignment into paired advantages for entering and completing tasks within a policy-induced capability region. The method employs a reverse-to-forward curriculum that first expands the solvable state space backward along successful reference trajectories and then broadens it through on-policy exploration, thereby addressing the challenges of credit assignment and exploration more effectively than existing group-based policy optimization methods. Experimental results on ALFWorld and WebShop benchmarks demonstrate that this approach achieves superior success rates and training efficiency compared to strong baselines such as GRPO and GiGPO across different model scales.

Critical strengths
  1. The selection of related work is relatively comprehensive.
  2. The experiments are thorough.
Critical weaknesses
  1. The motivation presented in the Introduction (lines 59–73) is somewhat obscure, and it is recommended that the authors reorganize this section. In particular, the concept of “policy-induced capability region” introduced in Figure 1 is difficult to understand without an intuitive explanation.

  2. There is a disconnect between the theoretical definition of the “capability region” and its concrete implementation in Algorithm 1. During actual training, how is the condition p^π(c)≥η\hat{p}_\pi(c) \ge \eta estimated?

  3. Appendix B.1 shows that different reliability thresholds η\eta were used for different environments (0.6 for ALFWorld and 0.78 for WebShop). What is the rationale behind selecting these specific values? How should we set this threshold when deploying in real-world scenarios?

  4. Given that the Right Advantage (ARA_R) relies on comparisons among these suffixes, would increasing NRN_R (e.g., to 4 or 8) provide more stable advantage estimates?

  5. It is suggested that the authors report wall-clock time comparison data in the appendix.

Questions for authors

Please refer to "Critical Weaknesses".

Reviewer_bWXAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies credit assignment for long-horizon LLM agents under sparse terminal rewards. The method samples starting points near the capability boundary, divides trajectories into prefixes and suffixes, and proposes Paired Advantage: comparing the completion rate of prefixes reaching contexts across endpoints, and comparing suffix outcomes within the same endpoint; it also employs a backward-to-forward curriculum by first moving the starting point along successful reference trajectories and then expanding prefixes from the original starting points. The contributions are the paired advantage decomposition and the two-stage curriculum. In the ALFWorld and WebShop environments, the method outperforms GRPO, GiGPO, and CRPS in model scale, but it relies on a successful reference trajectory for each task.

Critical strengths
  1. Paired Advantage provides fine-grained credit assignment without requiring process rewards. It decomposes long-term credit into prefix selection across endpoints and suffix comparison within the same endpoint, reusing the suffix for the same endpoint, while singleton endpoints can generate an intermediate signal of 0.5. Under sparse terminal rewards, credit assignment is the core challenge, and this design offers greater discrimination than trajectory-level advantages.
  2. The Forward phase allows the method to operate beyond the reference trajectory. The starting point is fixed at the original initial state, prefixes are sampled by the policy itself, and endpoints can fall outside the reference trajectory. This avoids the limitations of a purely reverse curriculum constrained by a support set of successful references, allowing the capability region to expand toward the policy's own distribution.
Critical weaknesses
  1. The Reverse stage highly overlaps with Backplay and reverse curriculum, and omits Cago, a directly related work; the main text does not sufficiently explain the mechanistic differences, making the "capability region" appear more like a conceptual reconstruction.
  2. There is a lack of Forward-only / reference-free PA ablation and distillation baselines, making it impossible to demonstrate that the gains come from Paired Advantage rather than "one successful reference trajectory per task." This assumption is costly and impractical in production; without isolating the contribution, the claims of method effectiveness and generalizability cannot be supported.
Questions for authors

1.The Reverse stage mechanically overlaps heavily with Backplay and reverse curriculum, and directly related work is omitted from Cago; The main text does not sufficiently explain the essential differences between the "Competency Area" framework and these tasks. 2.The lack of forward-only/reference trajectory PA ablation and a distillation baseline for the same data volume; Therefore, it is impossible to determine whether performance gain comes from a paired advantage or from the strong assumption of "one successful reference trajectory per task." 3.The paper requires each training task to have a successful reference trajectory, but does not discuss performance when references are sparse, quality is degraded, or missing, nor does it provide alternatives. This assumption is expensive in production environments and often does not hold true, directly affecting the generalizability of the method and the assessment of practical value. 4.(N_R), prefix length, threshold (eta), etc., are only fixed values, without reporting the impact of reasonable range variations on performance. For technical contributions centered on advantage computation, lack of sensitivity analysis weakens reliability and reproducibility conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.