- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a training method for long-horizon LLM agents that addresses sparse terminal rewards and the resulting credit assignment problem. It splits each rollout into a prefix and a suffix and gives them separate learning signals. Prefixes are trained based on how solvable the reached state is, while suffixes are trained based on task completion from that state. Training first moves backward along successful reference trajectories and then switches to states reached by the policy itself. The method is evaluated on ALFWorld and WebShop with Qwen2.5 1.5B and 7B models.
Critical strengths
-
The credit assignment idea is simple and well-motivated. Separating the signal for reaching a useful state from the signal for completing the task gives more informative supervision than assigning the same terminal reward to the entire trajectory.
-
The empirical results are strong and consistent across both benchmarks and model sizes. The ablation also shows that both the forward stage and paired advantages contribute to the gains. The paper is also clear and well-written.
Critical weaknesses
-
The method requires successful reference trajectories and the ability to reset the environment to intermediate states. The authors acknowledge this limitation, but these requirements may restrict where the method can be applied. It is also unclear how costly the reference trajectories are to obtain.
-
The final results show that the policy improves, but the evidence for the paper’s specific explanation is less direct. The endpoint analysis suggests that the policy learns to handle more off-reference states, but it does not directly measure success from those states. A more direct evaluation would make the capability-region claim more convincing.
Questions for authors
-
The main contribution is the separation of prefix and suffix credit, but the ablation does not test these two signals separately. How does performance change when using only the prefix advantage or only the suffix advantage, while keeping the curriculum and interaction budget fixed?
-
Table 1 states that the baseline results are taken from prior work, while the paper also says that all RL methods use the same training settings. Could the authors clarify which baselines were rerun and how closely the training tasks, interaction budgets, and model-selection procedures were matched?