- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VH-GPO to address the lack of effective policy-learning signals when all rollouts in a group fail and their group-relative advantages are zero. The method introduces a teacher–student handoff mechanism: a teacher recovers from a student-visited state, after which student continuation rollouts and independent confirmation identify an early feasible handoff point within a bounded search budget. Action preferences, teacher-bridge supervision, and successful student-continuation supervision then supplement GRPO training. Only the student is used at inference time. The paper evaluates VH-GPO on ALFWorld and WebShop at two model scales, reporting means and standard deviations across three random seeds. Improvements vary across settings, but the method improves overall performance. Ablations further examine the training objectives, teacher intervention locations, and handoff selection.
Critical strengths
- The method makes a clear and meaningful contribution. It turns the question of when to end teacher assistance into an empirically testable decision based on student continuation outcomes. Teacher recovery, handoff confirmation, and student-continuation learning form a coherent approach that adapts supervision to the student’s current capabilities and enables learning from otherwise uninformative failed rollouts.
- The experimental design is extensive and supports the central idea. The evaluation covers two environments, two model scales, and three random seeds, with improvements in aggregate metric means across settings. Ablations provide further support: on ALFWorld with the 1.5B student, random verified handoff reduces success from 94.79% to 90.10%, while removing the student-resume loss reduces it to 90.62%. These results support the roles of both handoff selection and learning from successful student continuations.
Critical weaknesses
- The mechanism analysis has limited coverage. The relevant ablations are conducted only on ALFWorld with the 1.5B student. Moreover, Terminal Teacher changes the scope of teacher supervision while also removing student-resume supervision, so its performance gap cannot be attributed solely to handoff selection. The results support the overall design, but the individual contributions and their applicability across environments remain less clear.
- The full training cost is not sufficiently characterized. Table 5 reports additional output tokens, which do not capture the complete overhead of environment restoration, replay, and model computation. The extra-rollout GRPO budget is based on the configured upper bound for four confirmation continuations, rather than a compute-matched comparison with the full procedure. This limits the assessment of practical efficiency.
Questions for authors
- Do you have ablation results for handoff selection or the student-resume loss on WebShop? These would help establish whether the mechanisms identified on ALFWorld play similar roles in a different interactive environment.
- Under the same hardware configuration, how much additional training time or GPU-hours does VH-GPO require relative to GRPO? Please distinguish online recovery, search, and replay overhead from the one-time cost of preparing the teacher.
4 comments
Response part 2
W2: Incomplete characterization of training costs
Thank you for pointing out that additional output-token counts and sampling controls based on the maximum number of confirmations do not establish actual computational efficiency. In addition to reporting complete online costs and a breakdown by phase, we provide results for GRPO trained for 200 updates on ALFWorld to examine whether longer training enables standard GRPO to reach VH-GPO's performance. GRPO's cumulative training time is 20 h 31 min 47 s, corresponding to 82.119 A100 GPU-hours on 4 A100 GPUs. The table compares this result with the existing VH-GPO result.
| Method | Updates | Training time (hours) | A100 GPU-hours | Overall ALFWorld success rate (%) |
|---|---|---|---|---|
| GRPO | 200 | 20.530 | 82.119 | 80.47 |
| VH-GPO | 150 | 16.242 | 64.967 | 94.79 |
GRPO reaches 80.47% success after 200 updates. Its recorded cumulative training time is 4.288 hours (approximately 26.40%) longer, corresponding to 17.152 additional A100 GPU-hours, while its success rate remains below the reported VH-GPO mean. Under these records, longer GRPO training does not reach VH-GPO's average success rate.
Response part 3
Q1: Mechanism ablations on WebShop
Thank you for this suggestion. We conducted additional WebShop ablations with a Qwen2.5-1.5B-Instruct student, covering the three auxiliary objectives, the teacher recovery anchor, handoff selection, and full teacher-suffix supervision. The table reports all available WebShop ablation results as means ± standard deviations over three random seeds. Success-rate changes are relative to full VH-GPO, in percentage points, and are computed using unrounded means.
| Method | Success rate (%) | Score | Change in success rate (pp) |
|---|---|---|---|
| Full VH-GPO | 77.34 ± 3.37 | 88.71 ± 1.45 | — |
| w/o action preference loss | 73.70 ± 0.37 | 85.69 ± 1.46 | −3.64 |
| w/o student resume loss | 75.78 ± 3.37 | 87.31 ± 0.34 | −1.56 |
| w/o teacher bridge loss | 70.31 ± 1.69 | 81.75 ± 1.64 | −7.03 |
| Random Failure Anchor | 76.56 ± 3.37 | 88.62 ± 0.86 | −0.78 |
| Random Verified Handoff | 76.30 ± 1.95 | 87.23 ± 1.32 | −1.04 |
| Terminal Teacher | 73.18 ± 1.95 | 85.64 ± 1.09 | −4.17 |
Auxiliary training objectives. Removing the action preference, student resume, and teacher bridge losses reduces mean success by 3.64, 1.56, and 7.03 percentage points, respectively, and also lowers Score. Removing the teacher bridge loss produces the largest decrease, consistent with the ALFWorld observations and supporting the utility of explicitly learning verified teacher recovery behavior.
Handoff selection. Replacing earliest verified handoff with random verified handoff reduces mean success by 1.04 percentage points and also lowers Score. This supports prioritizing earlier handoff points once the student's ability to finish independently has been verified.
Student resume loss. Removing this loss reduces mean success by 1.56 percentage points and Score from 88.71 to 87.31. Explicitly reinforcing verified successful student continuations therefore also helps on WebShop, consistent with the ALFWorld observations.
Recovery anchors and full teacher-suffix supervision. Random Failure Anchor reduces mean success by 0.78 percentage points, suggesting a smaller effect of anchor-selection order in this environment. Terminal Teacher reduces mean success by 4.17 percentage points.
Response part 4
Thank you for requesting measurements of training time and GPU-hours. We compiled timing records for standard GRPO and VH-GPO on ALFWorld and WebShop. Both use Qwen2.5-1.5B-Instruct students and are allocated 4 NVIDIA A100-SXM4-80GB GPUs. VH-GPO's teacher and student share these 4 GPUs.
The complete VH-GPO job takes 16.242 hours and 64.967 A100 GPU-hours on ALFWorld, and 7.041 hours and 28.163 A100 GPU-hours on WebShop.
The recorded differences are 2.187 hours and 8.749 GPU-hours (15.56%) on ALFWorld, and 0.968 hours and 3.874 GPU-hours (15.95%) on WebShop.
| Environment | GRPO time (hours) | VH-GPO time (hours) | GRPO GPU-hours | VH-GPO GPU-hours | Difference between records (hours / GPU-hours) | Relative difference |
|---|---|---|---|---|---|---|
| ALFWorld | 14.055 | 16.242 | 56.219 | 64.967 | +2.187 / +8.749 | +15.56% |
| WebShop | 6.072 | 7.041 | 24.289 | 28.163 | +0.968 / +3.874 | +15.95% |
Online recovery, handoff search, replay, and auxiliary training
The table below reports cumulative phase costs for the same two VH-GPO runs.
| Phase | ALFWorld time (minutes) | ALFWorld GPU-hours | WebShop time (minutes) | WebShop GPU-hours |
|---|---|---|---|---|
| Online teacher recovery | 98.144 | 6.543 | 18.071 | 1.205 |
| Total handoff search | 129.988 | 8.666 | 8.395 | 0.560 |
| Of which: candidate-state testing | 99.197 | 6.613 | 6.631 | 0.442 |
| Of which: independent student confirmation | 30.792 | 2.053 | 1.765 | 0.118 |
| Environment action replay | 5.878 | 0.392 | 0.101 | 0.007 |
| Total recovery + search + replay | 234.010 | 15.601 | 26.568 | 1.771 |
| Auxiliary-training forward and backward passes | 30.318 | 2.021 | 1.874 | 0.125 |
Recovery, search, and replay account for 24.01% of the complete ALFWorld job time and 6.29% of the complete WebShop job time. Cumulative phase costs do not directly equal the net increase relative to GRPO because the time spent in other training components also changes. On ALFWorld, VH-GPO spends 38.83 fewer minutes on standard student trajectory generation, 17.99 fewer minutes on old-policy probability computation, and 12.83 fewer minutes on overall policy updates; validation also takes 35.22 fewer minutes.
One-time teacher-preparation costs
The ALFWorld and WebShop teachers were each trained with GRPO for 250 updates using 4 NVIDIA A100 GPUs. Their one-time training costs are shown below. GPU-hours are calculated as cumulative training time multiplied by 4.
| Teacher environment | Training algorithm | Updates | One-time training duration | Training time (hours) | A100 GPU-hours |
|---|---|---|---|---|---|
| ALFWorld | GRPO | 250 | 25 h 42 min 23 s | 25.706 | 102.826 |
| WebShop | GRPO | 250 | 9 h 25 min 27 s | 9.424 | 37.697 |
These teacher-training costs arise from our specific experimental configuration and are not mandatory for every use of VH-GPO. The method can directly use an existing effective frozen teacher. If a teacher is trained specifically for the method, its one-time preparation cost should be included. Regardless of the teacher's source, its online inference and recovery-search costs are included in the corresponding training-cost measurements.
Response part 1
W1: Limited Scope of Mechanism Analysis
Thank you for the suggestion. We agree that the original Terminal Teacher ablation changed the scope of teacher supervision while removing student-resume supervision, making it difficult to isolate the contribution of early handoff.
To address this concern, we added a Full Teacher Suffix + Student Resume control. This variant retains the earliest-handoff search, independent confirmation, student-resume supervision, and action-preference supervision, while extending teacher imitation to the entire successful teacher suffix. Thus, it allows us to examine the effect of extending teacher imitation without removing student-resume supervision.
On ALFWorld, this variant achieves 91.41% success, compared with 93.75% for Earliest Handoff (VH-GPO) under the same single-seed setting, a difference of 2.34 percentage points. This result suggests that limiting teacher imitation to the verified bridge can be beneficial even when student-resume supervision is retained.
Furthermore, we conducted additional ablations on WebShop with the 1.5B student. Random Verified Handoff and removing the student-resume loss reduce success rates by 1.04 and 1.56 percentage points, respectively, providing further evidence for the roles of handoff selection and student-resume supervision in another environment. Full results are provided in our response to Q1 below.