- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Gated Successor Calibration (GSC) to improve solver-guided reinforcement learning for LLM agents. Building on CAST, the method introduces two components: Successor Centering, which adjusts solver-based progress rewards by comparing them with the average progress of alternative successor states, and All-Success Gating, which removes process-level rewards when all sampled trajectories in a group succeed. The authors evaluate GSC on three puzzle environments and further test its zero-shot generalization on ALFWorld and WebShop. Compared to CAST, GSC improves average success rates both on the training domains and unseen difficulty levels.
Critical strengths
- The paper addresses an important problem in credit assignment for LLM agents. Comparing solver feedback against alternative actions is a reasonable way to distinguish meaningful progress from plain actions.
- The experimental results are promising and consistent across multiple environments. GSC outperforms the strong CAST baseline across all three games, with improvements on both in-domain and unseen-difficulty tasks.
Critical weaknesses
-
The idea of this paper is not novel, and the analysis of the proposed components is insufficient. Progress-based rewards for credit assignment have already been explored in prior work, and GSC mainly introduces two relatively simple modifications to existing solver-guided RL. More importantly, the ablation results show that neither component alone improves average in-domain performance over CAST, while combining them leads to a substantial 6.8 percentage-point gain. This interaction is interesting but not sufficiently explained. The paper would benefit from a deeper analysis of why the individual components fail to improve performance but work so well together.
-
The generalization improvement is limited. The zero-shot transfer gains on ALFWorld and WebShop are marginal compared to CAST, with an average improvement of only 1.1 percentage points. Moreover, the paper does not report error bars or variance across runs, making it difficult to determine whether these improvements are statistically significant.
Questions for authors
- Could the authors provide more insight into why the two proposed components work well together but not individually?
- Could the author provide some discussion about why their methods novel compared with prior methods using progressive reward? and why it's worth to work on games using this method.
- Could the authors provide results across multiple random seeds, particularly for the zero-shot transfer experiments? The purpose is to make sure the improvement is clean and significant.