acceptodds
Withdrawn from ICLR 2027

FORGE: Feedback-Optimized Runtime Guidance Engine for Agentic RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) improves language agents through interaction, but sparse terminal rewards can leave difficult tasks without informative training feed-back. In Group Relative Policy Optimization (GRPO), an all-failure rollout group yields zero task-reward advantages, removing its contribution to the task-directed policy gradient. Guidance can open new paths to success, yet its value for train- ing depends on whether the resulting experience improves the students subsequent unassisted behavior. We introduce FORGE( Feedback-Optimized Runtime Guidance Engine) that trains a harnessed guidance agent alongside the student using post-update learning value. When a clean rollout group entirely fails, FORGE launches candidate tutoring interactions, adapting its guidance to the students evolving execution history. An episode workspace and a retrievable experience library support decisions about what guidance to provide, when to intervene, and when to end assistance. To evaluate each interaction, FORGE compares paired temporary student updates with and without its trajectory group, measuring the resulting loss difference on independent, unassisted probe trajectories. This esti- mates the candidates contribution to student learning beyond the common training data. The resulting feedback trains the guidance policy over complete interactions, including retrieval and follow-up decisions. All eligible candidates contribute to guidance-policy training, while at most the highest-value positive group enters the students task-reward update. Experiments on multiple interactive agent benchmarks show that FORGE outperforms GRPO and prior guidance-based baselinesin unassisted task performance, demonstrating the effectiveness of training dy-namic guidance with post-update learning value.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Language Agents, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q3GDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FORGE (Feedback-Optimized Runtime Guidance Engine), a framework for improving RL of language agents when GRPO encounters all-failure rollout groups. FORGE trains a separate guidance agent that provides adaptive assistance during student execution, supported by an episode workspace and retrievable experience library. Its key contribution is a post-update learning-value metric, which estimates whether guided trajectories improve the student's subsequent unassisted behavior by comparing temporary student updates with and without the candidate trajectories. Experiments on ALFWorld and WebShop show improvements over GRPO and several other baselines, with ablations supporting the benefits of adaptive guidance, retrieval, and learning-value-based feedback.

Critical strengths
  1. The paper addresses the sparse-reward limitation of GRPO in difficult interactive tasks, where all-failure rollout groups provide no task-directed learning signal. Adaptive guidance offers a mechanism for recovering useful training experience.

  2. Rather than rewarding guidance solely for producing successful trajectories, FORGE evaluates its contribution to the student's subsequent update. This provides a learning-oriented objective for training the guidance agent.

  3. FORGE achieves better overall performance on ALFWorld and WebShop. Component ablations and candidate-selection comparisons provide evidence that dynamic guidance, experience retrieval, and learning-value feedback each contribute to performance.

Critical weaknesses
  1. FORGE requires multiple guided rollout groups and temporary student updates for candidate evaluation, introducing substantial additional training cost. Since comparisons use equal training iterations rather than matched compute or interaction budgets, it is unclear whether the improvements justify the additional resources.

  2. The proposed metric measures local probe-loss improvements on successful trajectories, which may not reliably predict downstream task success. The paper does not directly establish how well this metric correlates with actual unassisted performance improvements across candidates or training stages.

  3. Though FORGE achieves the best overall results, its gains over HGPO are relatively small (0.48 and 0.75 percentage points on ALFWorld seen/unseen, respectively). Evaluation is also limited to two benchmarks and one student model size, leaving scalability and robustness uncertain.

Questions for authors

Q1: What is the total training cost of FORGE compared with GRPO, GiGPO, and HGPO, including guided rollouts, temporary student updates, and probe evaluations? How does performance compare under matched compute and environment-interaction budgets?

Q2: How strongly does the proposed post-update learning-value score correlate with actual improvements in unassisted task success? Does this relationship remain consistent across training stages and task difficulties?

Q3: How are the independent successful probe trajectories collected, and how many are used per event? Could their distribution introduce selection bias or favor particular guidance strategies?

Q4: The paper separately evaluates guidance-policy feedback and candidate selection, but what happens when both are replaced with simpler alternatives? How much of the overall improvement comes from learning a better guide versus selecting better training data?

Q5: Are the relatively small improvements over HGPO statistically significant across independent training runs? How does the method scale with larger student models, longer trajectories, and increasing numbers of guidance candidates?

Reviewer_3FfkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FORGE addresses all-failure GRPO groups by training a separate guide to provide adaptive help during student rollouts. The guide can retrieve prior experience, intervene as execution unfolds, and stop assistance; its feedback measures how a candidate trajectory group changes the student's loss on independent successful, unassisted probes after a temporary update. The method uses this learning-value signal both to train the guide and to select at most one positive-value group per failure event for the student's update. On ALFWorld and WebShop, the final student is evaluated without guidance or retrieval and outperforms the reported baselines, reaching 78.11% unseen ALFWorld success and 81.0% WebShop success. Ablations examine guide learning, runtime interventions, retrieval, and candidate selection.

Critical strengths
  1. The guide's objective directly addresses a meaningful failure mode of assisted training. Section 3.4 compares candidate and control updates from the same parameters and optimizer state, using a common unassisted probe. This is a sensible way to distinguish guidance that merely rescues an episode from experience that helps the student learn. Keeping assistance out of final evaluation also makes the intended benefit concrete.
  2. The component experiments go beyond comparing the full system with plain GRPO. Table 3 keeps positive-value student selection while changing the guide's reward, which helps separate guidance-policy learning from data selection. Removing follow-up interventions substantially reduces unseen ALFWorld performance, from 78.1% to 61.8%, and Figure 3 separately tests selection rules. These are useful checks on the proposed runtime-guidance design.
Critical weaknesses
  1. The successful probes are not clearly connected to the failed tasks they are meant to evaluate. Section 3.4, lines 270–285, requires successful unassisted probe trajectories but does not establish whether they come from the current failed task. If they do, it is unclear how they are obtained when the student repeatedly fails. If they come from other tasks, a reduction in probe loss may not reflect useful progress on the task that triggered guidance. This distinction is central to interpreting the learning-value signal.
  2. The comparisons leave both the resource trade-off and backbone generality unresolved. Tables 1–2 improve over HGPO by only 0.48/0.75 points on seen/unseen ALFWorld and 2.5 points on WebShop success. FORGE can generate four groups of eight student rollouts per triggered event, plus guide calls, probe collection, and temporary updates, yet comparisons match iterations rather than total interactions or compute (Section 4.1). All results also use a Qwen2.5-7B student with a Qwen2.5-3B guide. The frozen-guide ablation tests guide training, but not dependence on its starting model. Matched-budget comparisons and additional student/guide backbones are needed to establish how broadly the gains hold.
Questions for authors
  1. Do the successful probes come from the same task that triggered guidance? If so, how are they obtained when the current student consistently fails? If they come from other tasks, what evidence shows that their loss changes measure learning value for the current failed task?
  2. How do FORGE and GRPO/HGPO compare at matched total environment interactions or training compute, accounting for discarded candidates, guide inference, probe collection, and temporary student updates?
  3. Please evaluate Qwen3-4B or Qwen3-8B students against the strongest baselines under comparable budgets. With the student fixed, how sensitive are the gains to the guide's base model? These tests would clarify whether the benefit extends beyond the single student–guide pairing in Section 4.1.
Reviewer_kZHfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims for GRPO improvement and specifically, when all rollouts in a group fail, and consequently, no effective advantage is generated. This paper trades training cost for a retry stage when all failure happens in a group. Specifically, they propose to jointly train a student and an auxiliary guidance agent, where the student is the target model, and the guidance agent decides to stop the student, wait, or provide hints, when all fail group appears. Once hints are provided, a group of new rollouts are collected and look-ahead probe is used to determine which rollout is the best. Valid new rollouts are then added to the group and subsequent GRPO training is continued. They evaluated on WebShop and ALFworld benchmark.

Critical strengths
  1. This paper targets a meaningful case, where the reward is sparse and subsequently, GRPO wastes computation resources on groups providing no distinguishing signals.

  2. This paper introduces a novel jointly-trained hint provider to revisit the all-failure group and re-collect valid training data, and then filter the data via look-ahead probe.

Critical weaknesses
  1. The comparison is not fair: FORGE uses extra rollouts and an extra hint model, and it is unclear whether its advantage holds under the same FLOPS, time, or rollout budget. ALFWorld results are also reported at 50 steps, while GiGPO and HGPO train for 150 and 160 and report better results.

  2. HiLL and HINT, the closest predecessors, are discussed but not compared experimentally.

Minor:

  • The two curves in Figure 2 start at different values, and there are no repeated runs or uncertainty estimates.
  • The ablation does not show that analyzing the original failed rollouts is necessary.
  • The related work does not discuss GiGPO and HGPO, which are used as baselines.
  • Figure 1 says "HINT agent/policy" while the text says "guide agent/policy", which is confusing since HINT is also a baseline.
Questions for authors
  1. Does FORGE outperform GiGPO, HGPO, HiLL, and HINT under matched compute or interaction budgets?
  2. How does FORGE perform when the guide cannot see the original failed trajectories?
  3. Why is ALFWorld evaluated at 50 iterations, and does the advantage hold with longer training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.