- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FORGE (Feedback-Optimized Runtime Guidance Engine), a framework for improving RL of language agents when GRPO encounters all-failure rollout groups. FORGE trains a separate guidance agent that provides adaptive assistance during student execution, supported by an episode workspace and retrievable experience library. Its key contribution is a post-update learning-value metric, which estimates whether guided trajectories improve the student's subsequent unassisted behavior by comparing temporary student updates with and without the candidate trajectories. Experiments on ALFWorld and WebShop show improvements over GRPO and several other baselines, with ablations supporting the benefits of adaptive guidance, retrieval, and learning-value-based feedback.
Critical strengths
-
The paper addresses the sparse-reward limitation of GRPO in difficult interactive tasks, where all-failure rollout groups provide no task-directed learning signal. Adaptive guidance offers a mechanism for recovering useful training experience.
-
Rather than rewarding guidance solely for producing successful trajectories, FORGE evaluates its contribution to the student's subsequent update. This provides a learning-oriented objective for training the guidance agent.
-
FORGE achieves better overall performance on ALFWorld and WebShop. Component ablations and candidate-selection comparisons provide evidence that dynamic guidance, experience retrieval, and learning-value feedback each contribute to performance.
Critical weaknesses
-
FORGE requires multiple guided rollout groups and temporary student updates for candidate evaluation, introducing substantial additional training cost. Since comparisons use equal training iterations rather than matched compute or interaction budgets, it is unclear whether the improvements justify the additional resources.
-
The proposed metric measures local probe-loss improvements on successful trajectories, which may not reliably predict downstream task success. The paper does not directly establish how well this metric correlates with actual unassisted performance improvements across candidates or training stages.
-
Though FORGE achieves the best overall results, its gains over HGPO are relatively small (0.48 and 0.75 percentage points on ALFWorld seen/unseen, respectively). Evaluation is also limited to two benchmarks and one student model size, leaving scalability and robustness uncertain.
Questions for authors
Q1: What is the total training cost of FORGE compared with GRPO, GiGPO, and HGPO, including guided rollouts, temporary student updates, and probe evaluations? How does performance compare under matched compute and environment-interaction budgets?
Q2: How strongly does the proposed post-update learning-value score correlate with actual improvements in unassisted task success? Does this relationship remain consistent across training stages and task difficulties?
Q3: How are the independent successful probe trajectories collected, and how many are used per event? Could their distribution introduce selection bias or favor particular guidance strategies?
Q4: The paper separately evaluates guidance-policy feedback and candidate selection, but what happens when both are replaced with simpler alternatives? How much of the overall improvement comes from learning a better guide versus selecting better training data?
Q5: Are the relatively small improvements over HGPO statistically significant across independent training runs? How does the method scale with larger student models, longer trajectories, and increasing numbers of guidance candidates?