acceptodds
Under review as a conference paper at ICLR 2027

VH-GPO: VERIFIED HANDOFF FOR LLM AGENT REINFORCEMENT LEARNING

Abstract

Reinforcement learning (RL) plays a key role in training large language models (LLMs) to solve complex tasks through outcome-based feedback. Group-relative RL methods such as GRPO train language-model agents by comparing the outcomes of multiple rollouts for the same task. This mechanism is effective when outcomes vary, but if every trajectory in a group fails with the same reward, relative advantages collapse toward zero and the group supplies no useful learning signal. When the trainable agent's rollouts fail to produce any successful trajectory, one possible solution is to introduce a frozen teacher policy. We refer to the trainable agent as the student. The teacher attempts recovery from a state reached by the student, and its continuation is retained only if the environment verifies success. Yet allowing the teacher to complete the remaining task makes the entire recovered suffix teacher-generated, turning supervision into off-policy imitation and weakening a key benefit of RL: learning on the state distribution induced by the student. We therefore ask: after a teacher has repaired a student failure, when should control return to the student? Our core insight is that the teacher should act only long enough to cross what the student cannot yet handle, then hand back control so that successful experience is again student-generated. We formalize this idea as outcome-verified teacher–student handoff and instantiate it in Verified Handoff for Group Policy Optimization (VH-GPO). Given a verified teacher recovery, VH-GPO returns control to the student at the earliest independently confirmed handoff point found within a bounded search budget. With Qwen2.5-1.5B, VH-GPO increases success rates over GRPO from 59.38% to 94.79% on ALFWorld and from 71.62% to 77.34% on WebShop. Improvements also hold with the 7B student, for which additional teacher and student output tokens amount to 1.77% and 2.61% of regular student rollout output tokens on ALFWorld and WebShop, respectively. Ablation studies validate the design of VH-GPO and demonstrate the effectiveness of its key components. The teacher need not solve every problem for the student; instead, it should provide just enough support when the student struggles to help it find a way forward and gradually learn to solve problems independently. Code is available at https://anonymous.4open.science/r/VO-86EE.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Reinforcement Learning, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HfSsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VH-GPO to address the lack of effective policy-learning signals when all rollouts in a group fail and their group-relative advantages are zero. The method introduces a teacher–student handoff mechanism: a teacher recovers from a student-visited state, after which student continuation rollouts and independent confirmation identify an early feasible handoff point within a bounded search budget. Action preferences, teacher-bridge supervision, and successful student-continuation supervision then supplement GRPO training. Only the student is used at inference time. The paper evaluates VH-GPO on ALFWorld and WebShop at two model scales, reporting means and standard deviations across three random seeds. Improvements vary across settings, but the method improves overall performance. Ablations further examine the training objectives, teacher intervention locations, and handoff selection.

Critical strengths
  1. The method makes a clear and meaningful contribution. It turns the question of when to end teacher assistance into an empirically testable decision based on student continuation outcomes. Teacher recovery, handoff confirmation, and student-continuation learning form a coherent approach that adapts supervision to the student’s current capabilities and enables learning from otherwise uninformative failed rollouts.
  2. The experimental design is extensive and supports the central idea. The evaluation covers two environments, two model scales, and three random seeds, with improvements in aggregate metric means across settings. Ablations provide further support: on ALFWorld with the 1.5B student, random verified handoff reduces success from 94.79% to 90.10%, while removing the student-resume loss reduces it to 90.62%. These results support the roles of both handoff selection and learning from successful student continuations.
Critical weaknesses
  1. The mechanism analysis has limited coverage. The relevant ablations are conducted only on ALFWorld with the 1.5B student. Moreover, Terminal Teacher changes the scope of teacher supervision while also removing student-resume supervision, so its performance gap cannot be attributed solely to handoff selection. The results support the overall design, but the individual contributions and their applicability across environments remain less clear.
  2. The full training cost is not sufficiently characterized. Table 5 reports additional output tokens, which do not capture the complete overhead of environment restoration, replay, and model computation. The extra-rollout GRPO budget is based on the configured upper bound for four confirmation continuations, rather than a compute-matched comparison with the full procedure. This limits the assessment of practical efficiency.
Questions for authors
  1. Do you have ablation results for handoff selection or the student-resume loss on WebShop? These would help establish whether the mechanisms identified on ALFWorld play similar roles in a different interactive environment.
  2. Under the same hardware configuration, how much additional training time or GPU-hours does VH-GPO require relative to GRPO? Please distinguish online recovery, search, and replay overhead from the one-time cost of preparing the teacher.
4 comments
AuthorsOpenReview ↗

Response part 1

W1: Limited Scope of Mechanism Analysis

Thank you for the suggestion. We agree that the original Terminal Teacher ablation changed the scope of teacher supervision while removing student-resume supervision, making it difficult to isolate the contribution of early handoff.

To address this concern, we added a Full Teacher Suffix + Student Resume control. This variant retains the earliest-handoff search, independent confirmation, student-resume supervision, and action-preference supervision, while extending teacher imitation to the entire successful teacher suffix. Thus, it allows us to examine the effect of extending teacher imitation without removing student-resume supervision.

On ALFWorld, this variant achieves 91.41% success, compared with 93.75% for Earliest Handoff (VH-GPO) under the same single-seed setting, a difference of 2.34 percentage points. This result suggests that limiting teacher imitation to the verified bridge can be beneficial even when student-resume supervision is retained.

Furthermore, we conducted additional ablations on WebShop with the 1.5B student. Random Verified Handoff and removing the student-resume loss reduce success rates by 1.04 and 1.56 percentage points, respectively, providing further evidence for the roles of handoff selection and student-resume supervision in another environment. Full results are provided in our response to Q1 below.

AuthorsOpenReview ↗

Response part 2

W2: Incomplete characterization of training costs

Thank you for pointing out that additional output-token counts and sampling controls based on the maximum number of confirmations do not establish actual computational efficiency. In addition to reporting complete online costs and a breakdown by phase, we provide results for GRPO trained for 200 updates on ALFWorld to examine whether longer training enables standard GRPO to reach VH-GPO's performance. GRPO's cumulative training time is 20 h 31 min 47 s, corresponding to 82.119 A100 GPU-hours on 4 A100 GPUs. The table compares this result with the existing VH-GPO result.

MethodUpdatesTraining time (hours)A100 GPU-hoursOverall ALFWorld success rate (%)
GRPO20020.53082.11980.47
VH-GPO15016.24264.96794.79

GRPO reaches 80.47% success after 200 updates. Its recorded cumulative training time is 4.288 hours (approximately 26.40%) longer, corresponding to 17.152 additional A100 GPU-hours, while its success rate remains below the reported VH-GPO mean. Under these records, longer GRPO training does not reach VH-GPO's average success rate.

AuthorsOpenReview ↗

Response part 3

Q1: Mechanism ablations on WebShop

Thank you for this suggestion. We conducted additional WebShop ablations with a Qwen2.5-1.5B-Instruct student, covering the three auxiliary objectives, the teacher recovery anchor, handoff selection, and full teacher-suffix supervision. The table reports all available WebShop ablation results as means ± standard deviations over three random seeds. Success-rate changes are relative to full VH-GPO, in percentage points, and are computed using unrounded means.

MethodSuccess rate (%)ScoreChange in success rate (pp)
Full VH-GPO77.34 ± 3.3788.71 ± 1.45—
w/o action preference loss73.70 ± 0.3785.69 ± 1.46−3.64
w/o student resume loss75.78 ± 3.3787.31 ± 0.34−1.56
w/o teacher bridge loss70.31 ± 1.6981.75 ± 1.64−7.03
Random Failure Anchor76.56 ± 3.3788.62 ± 0.86−0.78
Random Verified Handoff76.30 ± 1.9587.23 ± 1.32−1.04
Terminal Teacher73.18 ± 1.9585.64 ± 1.09−4.17

Auxiliary training objectives. Removing the action preference, student resume, and teacher bridge losses reduces mean success by 3.64, 1.56, and 7.03 percentage points, respectively, and also lowers Score. Removing the teacher bridge loss produces the largest decrease, consistent with the ALFWorld observations and supporting the utility of explicitly learning verified teacher recovery behavior.

Handoff selection. Replacing earliest verified handoff with random verified handoff reduces mean success by 1.04 percentage points and also lowers Score. This supports prioritizing earlier handoff points once the student's ability to finish independently has been verified.

Student resume loss. Removing this loss reduces mean success by 1.56 percentage points and Score from 88.71 to 87.31. Explicitly reinforcing verified successful student continuations therefore also helps on WebShop, consistent with the ALFWorld observations.

Recovery anchors and full teacher-suffix supervision. Random Failure Anchor reduces mean success by 0.78 percentage points, suggesting a smaller effect of anchor-selection order in this environment. Terminal Teacher reduces mean success by 4.17 percentage points.

AuthorsOpenReview ↗

Response part 4

Thank you for requesting measurements of training time and GPU-hours. We compiled timing records for standard GRPO and VH-GPO on ALFWorld and WebShop. Both use Qwen2.5-1.5B-Instruct students and are allocated 4 NVIDIA A100-SXM4-80GB GPUs. VH-GPO's teacher and student share these 4 GPUs.

The complete VH-GPO job takes 16.242 hours and 64.967 A100 GPU-hours on ALFWorld, and 7.041 hours and 28.163 A100 GPU-hours on WebShop.

The recorded differences are 2.187 hours and 8.749 GPU-hours (15.56%) on ALFWorld, and 0.968 hours and 3.874 GPU-hours (15.95%) on WebShop.

EnvironmentGRPO time (hours)VH-GPO time (hours)GRPO GPU-hoursVH-GPO GPU-hoursDifference between records (hours / GPU-hours)Relative difference
ALFWorld14.05516.24256.21964.967+2.187 / +8.749+15.56%
WebShop6.0727.04124.28928.163+0.968 / +3.874+15.95%
Online recovery, handoff search, replay, and auxiliary training

The table below reports cumulative phase costs for the same two VH-GPO runs.

PhaseALFWorld time (minutes)ALFWorld GPU-hoursWebShop time (minutes)WebShop GPU-hours
Online teacher recovery98.1446.54318.0711.205
Total handoff search129.9888.6668.3950.560
Of which: candidate-state testing99.1976.6136.6310.442
Of which: independent student confirmation30.7922.0531.7650.118
Environment action replay5.8780.3920.1010.007
Total recovery + search + replay234.01015.60126.5681.771
Auxiliary-training forward and backward passes30.3182.0211.8740.125

Recovery, search, and replay account for 24.01% of the complete ALFWorld job time and 6.29% of the complete WebShop job time. Cumulative phase costs do not directly equal the net increase relative to GRPO because the time spent in other training components also changes. On ALFWorld, VH-GPO spends 38.83 fewer minutes on standard student trajectory generation, 17.99 fewer minutes on old-policy probability computation, and 12.83 fewer minutes on overall policy updates; validation also takes 35.22 fewer minutes.

One-time teacher-preparation costs

The ALFWorld and WebShop teachers were each trained with GRPO for 250 updates using 4 NVIDIA A100 GPUs. Their one-time training costs are shown below. GPU-hours are calculated as cumulative training time multiplied by 4.

Teacher environmentTraining algorithmUpdatesOne-time training durationTraining time (hours)A100 GPU-hours
ALFWorldGRPO25025 h 42 min 23 s25.706102.826
WebShopGRPO2509 h 25 min 27 s9.42437.697

These teacher-training costs arise from our specific experimental configuration and are not mandatory for every use of VH-GPO. The method can directly use an existing effective frozen teacher. If a teacher is trained specifically for the method, its one-time preparation cost should be included. Regardless of the teacher's source, its online inference and recovery-search costs are included in the corresponding training-cost measurements.

Reviewer_3dpAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces training-time teacher recovery to address groups in relative RL where all trajectories fail, rewards have no within-group variation, and a useful learning signal is absent. VH-GPO lets a frozen teacher attempt to complete the task from restorable states visited by the student and retains recoveries verified as successful by the environment. It then searches, within a limited budget, for states from which the student can complete the task, confirms success through independent sampling, and returns control at the earliest tested state that passes confirmation. The resulting trajectory comprises a student prefix, a teacher bridge, and a student resume, and must pass full replay verification.

The method retains the original GRPO update and constructs three auxiliary losses from these trajectories: preferences between actions at the same state, imitation of teacher bridges, and imitation of successful student resumes. Experiments use Qwen2.5-1.5B/7B students on ALFWorld/WebShop, with a frozen, GRPO-trained 1.5B teacher. Table 1 reports improvements over GRPO and the selected group-relative RL baselines in all four settings. The most pronounced gain is the 1.5B ALFWorld success rate increasing from 59.38% to 94.79%. Only the student is used at inference.

Critical strengths
  1. The intervention has explicit verification criteria and keeps deployment simple. Only groups meeting the all-failure/zero-signal criteria are processed. Teacher recovery, independent student confirmation, and full replay filter different failure modes. Successful examples do not simply treat an entire teacher trajectory as a student RL trajectory; auxiliary supervision is constructed separately for bridges, resumes, and action comparisons. In particular, the 7B student receives local support from a task-trained 1.5B teacher, showing that the design does not necessarily require a larger teacher. All recovery and teacher mechanisms are removed after training, leaving student-only execution at deployment.
  2. The ablations are substantial and support the full procedure's utility. Table 2 removes each of the three auxiliary losses. Table 3 changes recovery anchors and handoff selection while retaining verification, and also compares full teacher suffixes. Table 4 examines additional student sampling. Random Verified Handoff uses the same search budget and performs 4.69 percentage points below the full method, providing direct evidence for handoff selection. Appendix C also includes an OVCSD reimplementation. The large gains over ordinary GRPO and these ablations together support a contribution beyond simply adding a few unfiltered teacher demonstrations.
Critical weaknesses
  1. The independent contribution of early handoff and the advantage over strong baselines need stronger evidence. Terminal Teacher in Table 3 switches to full teacher suffixes while also removing handoff search and student-resume supervision. Its 7.81-percentage-point gap therefore mixes supervision scope, successful student-resume data, and changes in data selection. Random handoff is a more direct control, but Appendix D states that it explicitly excludes the earliest candidate. This may change confirmation rates and the distribution of retained trajectories, neither of which is reported. The OVCSD reimplementation in Appendix C is useful, but lacks sufficient detail about teacher-suffix sampling, loss budgets, and key settings. Moreover, the margins over the strongest RL baseline on ALFWorld are 94.79% versus 92.71% at 1.5B and 95.57% versus 94.53% at 7B. The WebShop margins are also smaller than some reported standard deviations. The evidence clearly supports improvement over ordinary GRPO, but stronger paired statistics and controls with comparable auxiliary supervision are needed to establish gains over strong baselines and the specific contribution of early handoff.
  2. Training resources and the scope of the triggering conditions are insufficiently documented. Section 4.1 uses a GRPO-trained 1.5B teacher, but does not report its training data, cost, checkpoint selection, or standalone performance. Matching 150 student updates therefore does not establish equal total resources. Table 5's 1.77%/2.61% figures count only additional output tokens relative to student root rollouts in the 7B runs. They exclude input processing, auxiliary-loss updates, state restoration/replay, and teacher preparation, so they do not establish equally small overall training overhead. Appendix B also replaces WebShop's original partial-credit score with a binary reward for strict success, which increases the prevalence of all-failure groups with equal rewards. The paper does not test whether intervention is similarly valuable when training with the original partial-credit score. These issues affect the practical relevance of the method under realistic training budgets and different reward settings.
Questions for authors
  1. Handoff mechanism: Can you vary only where teacher imitation ends, while keeping teacher recovery, student-resume collection, the preference loss, and auxiliary-loss normalization unchanged? Please report confirmation rates, retained trajectory counts, supervised token counts, and paired gains for earliest/random/full-suffix comparisons. This would affect my assessment of early handoff's independent contribution.
  2. Statistical reliability and baselines: Please provide paired results by training seed and confidence intervals using per-task outcomes for VH-GPO versus GraphGPO/HGPO, and specify the training/evaluation splits and checkpoint-selection rule. What recovery settings, objective weights, and budgets were used in the OVCSD reimplementation? This information would help establish whether the margins over strong baselines are stable.
  3. Resources and rewards: How was the teacher trained, did it use the same training split, and what are its standalone performance and preparation cost? What are the total time/token costs after including teacher preparation, environment restoration/replay, and auxiliary updates? Under WebShop's original partial-credit reward, are zero-signal groups still frequent enough, and the gains large enough, to support the current conclusions?
7 comments
AuthorsOpenReview ↗

Response part 1

W1: The independent contribution of early handoff and comparisons with strong baselines

Thank you for these suggestions. We added three controls: random handoff that allows selection of the earliest candidate, full teacher-suffix supervision that retains student resumes, and a variant that randomizes only the endpoint of teacher imitation. The overall success rates for earliest handoff, random handoff, full teacher suffix with student resume, and random teacher-imitation endpoint with earliest student resume are 93.75%, 92.97%, 91.41%, and 79.69%, respectively. The paired gains of earliest handoff over the three controls are 0.78, 2.34, and 14.06 percentage points. In our response to Q2, we provide the OVCSD configuration, results paired by training seed against GraphGPO/HGPO, and confidence intervals based on per-task outcomes.

AuthorsOpenReview ↗

Response part 2

W2: Total resource costs and reward settings

Thank you for raising these issues. We separately report the one-time cost of teacher training and the online student-training job cost, including teacher recovery, handoff search, independent confirmation, environment replay, and auxiliary updates. We also distinguish generation input/output tokens from tokens processed by auxiliary updates. The GRPO teachers in the paper were each trained for 250 updates on 4 A100 GPUs, with one-time costs of 102.826 and 37.697 A100 GPU-hours for ALFWorld and WebShop, respectively. The complete online jobs used 64.967 and 28.163 A100 GPU-hours. The recorded time differences relative to GRPO were 15.56% and 15.95%, respectively.

VH-GPO does not require a separately trained teacher or GRPO as the teacher-training algorithm: an existing frozen model capable of providing valid recoveries can be reused. Dedicated teacher preparation is therefore not an intrinsic step of the method, although its cost must be counted when incurred. Q3 provides further cost details and results under WebShop's original partial-credit reward.

AuthorsOpenReview ↗

Response part 3

Q1: Handoff mechanism ablations

We agree that the original Terminal Teacher variant changes teacher-supervision scope, handoff search, and student-resume supervision simultaneously, and therefore does not isolate the contribution of early handoff. We added the following four conditions to distinguish the actual handoff location from the endpoint of teacher imitation.

ConditionEndpoint of teacher imitationLocation used for student resume, independent confirmation, and preference supervision
Earliest HandoffEarliest successful nonterminal handoff candidate tested within the search budgetEarliest handoff point
Random Handoff Including the Earliest CandidateUniformly sampled from successful nonterminal candidates tested, including the earliest candidateChanges with the selected random handoff point
Full Teacher Suffix + Student ResumeEnd of the complete teacher recovery trajectoryEarliest handoff point retained; student resume is preserved
Random Teacher-Imitation Endpoint + Earliest Student ResumeUniformly sampled from successful nonterminal candidates tested by the earliest-handoff procedure, including the earliest candidateRemains at the earliest handoff point, independently of the teacher-supervision endpoint

All conditions use the same auxiliary-loss settings and normalization. The last variant randomizes only the endpoint of teacher imitation. Teacher recovery, earliest-handoff search, independent student confirmation, replay, retention rules, successful student-resume collection, and preference-pair construction remain unchanged.

Recovery and retention statistics during training

The table below reports the numbers of groups for which teacher recovery was attempted, teacher recovery succeeded, and trajectories were ultimately retained for auxiliary training. The confirmation rate is the number of retained groups divided by the number of groups with successful teacher recovery.

ConditionGroups with recovery attemptedGroups with successful teacher recoveryRetained groupsRetained / attemptedConfirmation rate
Earliest Handoff106685551.89%80.88%
Random Handoff Including the Earliest Candidate116665446.55%81.82%
Full Teacher Suffix + Student Resume107594945.79%83.05%
Random Teacher-Imitation Endpoint118695748.31%82.61%
Table A1: Auxiliary supervision tokens over 150 training updates
ConditionTeacher supervision tokensStudent-resume tokensPositive and Negative Preference TokensTotal supervision tokens
Earliest Handoff20,17711,390465 + 46032,492
Random Handoff Including the Earliest Candidate25,6447,161437 + 42833,670
Full Teacher Suffix + Student Resume34,4749,652473 + 47345,072
Random Teacher-Imitation Endpoint + Earliest Student Resume31,0609,945611 + 61942,235
Reviewer_BamwOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

When every rollout in a GRPO group fails, the group-relative advantages are all zero and the group gives no learning signal. This paper proposes VH-GPO to address this problem for LLM agents. For such a group, a frozen teacher policy starts from a state the student reached in one of its failed trajectories (the “anchor”). The teacher’s continuation is kept only if the environment confirms the task succeeds. Within a bounded search budget, the method then finds the earliest state along that teacher continuation from which the frozen student can independently finish the task, and hands control back to the student there. Experiments use Qwen2.5-1.5B and 7B students with a frozen, GRPO-trained Qwen2.5-1.5B teacher, on ALFWorld and WebShop.

Critical strengths

S1. A clean, simple fix for a known GRPO problem. The all-failure case in batched GRPO rollouts is a known problem: advantages collapse and the group is wasted. The paper proposes a clean, simple solution. The teacher acts only on these groups, and the method returns control to the student at the earliest point where independent success is confirmed. This keeps most of the learning on the student’s own state distribution instead of turning it into off-policy imitation.

S2. Clear motivation and easy-to-follow writing. The motivation is well presented, and the paper is easy to follow overall. The guiding question of when control should return to the student is stated early, and Fig. 1 lays out each step of the pipeline.

Critical weaknesses

W1. An unstated assumption that the teacher can already solve the task, with no guidance on how to vet a teacher beforehand. My biggest concern is the unstated assumption that the teacher is good enough on these tasks to solve the bridge and help the student. It also raises the question of what to evaluate about a teacher, and how, to be reasonably confident beforehand that it will help the student during RL. The paper current doesn’t say:

  • if and what data or environment the teacher was trained on;

  • how good the teacher is on its own.

W2. It’s unclear whether the gains carry over beyond ALFWorld and WebShop. It is not clear whether the method would give similar gains on tasks other than ALFWorld and WebShop. Both are well-studied benchmarks with fixed templates and limited action spaces. On ALFWorld, the RL baselines in Table 1 already reach 92–94% success, so much of the headline gain is measured against plain GRPO in a nearly saturated setting.

Questions for authors
  1. (W1) How was the teacher trained: on which environments, for how many updates, and what is its own success rate on ALFWorld and WebShop?

  2. (W1) How does VH-GPO perform with weaker teachers (e.g., the untrained Qwen2.5-1.5B-Instruct, an earlier GRPO checkpoint, or a teacher trained on a different environment)? Is there a teacher-quality threshold below which the method stops helping?

  3. (W2) Can you evaluate on a less templated or harder benchmark (e.g., ScienceWorld, WebArena, search-based QA), or explain how the method works when states during the RL training can’t be cheaply restored?

7 comments
AuthorsOpenReview ↗

Response part 1

Q1: Teacher training and standalone performance (W1)

Thank you for this question. We trained Qwen2.5-1.5B-Instruct with GRPO for 250 updates separately on ALFWorld and WebShop to obtain a teacher for each environment. The teachers used the same training splits as the students: the official train split for ALFWorld, and goals with indices of 500 or above for WebShop.

We evaluated each teacher's standalone task performance in its corresponding environment:

Teacher training environmentTraining algorithmUpdatessuccess rate
ALFWorldGRPO25089.06%
WebShopGRPO25076.56%

The WebShop teacher achieved an average task score of 87.32.

AuthorsOpenReview ↗

Response Part 2

Q2: Weaker teachers and the teacher-quality threshold (W1)

Thank you for this important question. We agree that understanding VH-GPO's dependence on teacher capability, including any minimum quality requirement, is essential to characterizing its applicability.

Additional experiments. We conducted a teacher-quality ablation on ALFWorld using the original Qwen2.5-1.5B-Instruct model, without additional GRPO training, and checkpoints after 50, 100, 150, 200, and 250 GRPO updates as frozen teachers. The following table reports each teacher's standalone success rate, the final success rate of the corresponding VH-GPO student, and teacher-intervention statistics during training. For the GRPO-250 teacher, we report the student result for the seed used in this experiment, 93.75%, rather than the original paper's three-seed average of 94.79%.

Frozen teacherTeacher success rate (%)VH-GPO student success rate (%)Groups requiring recoveryGroups ultimately used for auxiliary training
Original model (0 updates)7.0376.5620817
GRPO-5014.0678.1219338
GRPO-10041.4180.4715661
GRPO-15059.3885.1615762
GRPO-20080.4789.0611657
GRPO-25089.0693.7510655

(1) VH-GPO can provide substantial gains even with a weak teacher.

With the original Qwen2.5-1.5B-Instruct teacher, whose standalone success rate is only 7.03%, the VH-GPO student achieves a success rate of 76.56%. This is 17.18 percentage points higher than the GRPO baseline of 59.38% in Table 1 of the paper. Using the teacher trained for only 50 GRPO updates further increases student success to 78.12%.

These results suggest that VH-GPO does not require a teacher with a high standalone success rate on complete tasks. The teacher attempts local recovery from states already visited in failed student trajectories. A trajectory is used for auxiliary training only when the environment verifies successful teacher recovery and the student can independently complete the remaining task from a candidate handoff state. Thus, even a teacher with limited overall capability can occasionally produce valid recoveries that provide useful learning signals.

(2) Teacher capability affects the final gains, but the amount of auxiliary supervision alone does not determine performance.

As standalone teacher success increases from 7.03% to 89.06%, the corresponding VH-GPO student's success rate increases from 76.56% to 93.75%. Stronger teachers therefore yield better student performance in the tested settings.

However, the amount of auxiliary supervision does not have a simple linear relationship with final performance. For example, 17 groups of auxiliary supervision yield a student success rate of 76.56%. With GRPO-50 and GRPO-100 teachers, the number of groups providing auxiliary supervision increases to 38 and 61, while student success reaches 78.12% and 80.47%, respectively.

The number of groups contributing valid auxiliary supervision does not fully capture their learning value. Inspection of teacher recovery trajectories also revealed that environment verification guarantees task success, but does not directly guarantee that every reasoning statement in the teacher's response is consistent with the environment state. Consequently, successful recovery trajectories can still differ in supervision quality. This may help explain why more recovered trajectories do not necessarily translate into proportionally larger gains, and why noisy supervision could potentially impair learning.

We also observe that stronger teachers are generally associated with fewer groups requiring recovery. From GRPO-150 to GRPO-250, for example, the number of groups selected for recovery decreases from 157 to 106, and the number contributing auxiliary supervision decreases from 62 to 55, while final student success increases from 85.16% to 93.75%. One possible explanation is that, as the student improves, fewer all-failure groups require intervention. Valid recoveries produced by different teachers may also have different learning value. Thus, the quantity and quality of auxiliary supervision, together with the opportunities for teacher intervention during training, may all affect the final outcome.

AuthorsOpenReview ↗

Response part 3

Q2: Weaker teachers and the teacher-quality threshold (W1)

(3) Teacher-quality thresholds and evaluation before training.

Within the tested range of standalone teacher success rates, 7.03%–89.06%, we did not observe a fixed success-rate threshold below which VH-GPO ceased to help. This does not establish that no such threshold exists more generally.

In practice, teacher selection should also consider recovery capability on student failure states, the ability to produce valid handoffs to the student, and the reliability of generated reasoning.

First, a teacher should have some local recovery capability. VH-GPO asks the teacher to recover from student-visited failure states, rather than solve every task from the initial state. Before training, one can use the initial student to collect a representative set of all-failure, zero-signal groups and obtain candidate recovery states using the VH-GPO procedure. Under the same recovery budget, the proportion of successful recoveries, or Verified Recovery Rate, can be measured to assess local error-correction capability.

Second, the consistency of teacher reasoning also matters. The teacher bridge loss supervises the teacher's full responses, including the Think reasoning content as well as its actions. Incorrect reasoning can introduce noise and potentially reduce performance.

Overall, teacher selection for VH-GPO should consider both the ability to produce environment-verified recoveries from failed student trajectories and the ability to provide reliable reasoning consistent with the environment state. A teacher need not have a high standalone success rate, but should provide a sufficient number of valid examples with useful supervision. We will clarify these teacher-evaluation principles and their limitations in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have revised the manuscript to address the main concerns raised during the discussion.

The revised manuscript includes:

  1. Teacher capability and preparation (Appendix A): Teacher training details, standalone performance, and an ablation across six teacher checkpoints.
  2. Mechanism analysis (Section 4.4 and Appendix H): Additional WebShop ablations and controlled comparisons of teacher-imitation endpoints and student-resume supervision.
  3. Statistical reliability (Appendix F): Per-seed results and paired bootstrap confidence intervals against GraphGPO and HGPO.
  4. Generalization (Appendix G): Additional evaluation on ScienceWorld.
  5. Computational costs and reproducibility (Section 4.5 and Appendices C–E): Online training time, GPU-hours, additional generation costs, OVCSD implementation details, and handoff diagnostics.

Further experimental results and detailed explanations are provided in our point-by-point responses to the reviewers.

We appreciate the reviewers' time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.