acceptodds
Withdrawn from ICLR 2027

Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning for long-horizon agents typically relies on sparse outcome-based rewards. This leads to a severe cold-start problem, as early-stage policies often fail to solve sampled tasks, leaving little useful reward signal for learning. To mitigate this problem, we use on-policy distillation (OPD) to provide token-level guidance on the student’s own rollouts. We find that the benefit of this guidance depends on the performance gap between the teacher and the stu- dent. When the teacher substantially outperforms the student, distillation helps guide the student through the early training stage where outcome rewards provide little learning signal. As the gap narrows and eventually reverses, however, con- tinued distillation becomes less beneficial and may hinder further improvement. Motivated by this observation, we propose Gap-Adaptive Teacher Scheduling (GATS), which augments the student’s RL objective with an OPD term whose weight adapts to the teacher–student performance gap. Specifically, GATS grad- ually reduces teacher guidance as the student approaches the teacher’s reference performance and withdraws it once that reference is reached. This enables GATS to leverage task-trained teachers smaller than the student, since teacher guidance is primarily needed during early training. Across ALFWorld, WebShop, and Sci- enceWorld with three Qwen2.5 teacher–student configurations, GATS achieves the highest average success rate among the compared methods in all three config- urations, improving over reward-only GRPO by 4.37%–11.87% under matched student rollout budgets. Code is available at https://anonymous.4open.science/r/guide-then-let-go-review-CCF4/.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, on-policy distillation, sparse rewards, exploration, knowledge distillation, GRPO, large language model agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tETbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains a student agent with a combination of outcome-reward RL and token-level supervision from a teacher, and makes the weight on the teacher term depend on the measured gap between the teacher's and the student's success rates. The weight shrinks as the gap closes, and the teacher is switched off once the student reaches the teacher's reference score, after which training continues with reward only. The authors argue that guidance helps during the sparse-reward cold start but hurts later, and they support this with a small study of OPD gain against the teacher–student gap, experiments on ALFWorld, WebShop and ScienceWorld with three teacher–student pairs, and comparisons against fixed, annealed and divergence-based supervision schedules.

Critical strengths
  1. The idea is simple and the motivation is easy to follow: guidance while the student is behind, none once it catches up. The rule itself is stated plainly enough.

  2. The schedule comparisons are more thorough than usual. The paper sweeps the horizon of a linear schedule and also calibrates it to match the withdrawal steps.

Critical weaknesses
  1. The two distillation baselines do much worse than plain RL without any teacher, in two of the three settings by a wide margin. The paper explains this as the student being held at the teacher's ceiling, but a drop of that size on a benchmark where the teacher itself does fine does not look like a ceiling effect. If the baseline implementations are undertuned, the gain attributed to the adaptive schedule is inflated, and this is the comparison the whole paper rests on.

  2. Each configuration appears to be trained once, and the reported variation comes from repeated evaluation of one checkpoint rather than from repeated training. Given that the gap-utility study behind the design uses short runs on a few checkpoints, I would want to see the ordering hold across seeds before reading much into the margins between GATS and the tuned linear schedule, which are small.

Questions for authors
  1. Why does fixed OPD fall so far below plain RL in the first two configurations, and were those baselines tuned to the same degree as GATS?

  2. Do the main comparisons come from single training runs? If so, does the ordering survive additional seeds?

  3. A short linear schedule chosen by sweeping comes close to GATS on ALFWorld. What distinguishes the adaptive rule from a well-tuned fixed one, beyond not needing the sweep?

Reviewer_7WqHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Gap-Adaptive Teacher Scheduling (GATS) for sparse-reward agentic RL. It combines GRPO with on-policy distillation from a frozen, task-trained teacher. The distillation weight is determined by the gap between a fixed teacher training-performance reference and a lagged moving average of student success, and teacher supervision is permanently withdrawn once the student reaches that reference. Evaluations on ALFWorld, WebShop, and ScienceWorld use three Qwen2.5 teacher–student configurations. The paper reports improved final success over GRPO, comparisons against distillation and fixed-schedule alternatives, and teacher-inclusive compute analyses.

Critical strengths
  1. A simple, operational response to the cold-start/teacher-ceiling trade-off. GATS uses readily available training feedback and avoids teacher inference after withdrawal. The rule is sufficiently explicit to implement, and the teacher reference is carefully distinguished from held-out teacher performance.

  2. The experiments address several immediate confounds. The paper includes smaller teachers, multiple student configurations, ID/OOD results, decay-shape and horizon comparisons, and GPU-hour accounting that includes teacher costs. In particular, the matched-withdrawal linear schedule provides a useful control beyond merely comparing against permanent distillation.

Critical weaknesses
  • The major concern is that the method is an emperical methods without solid theroy background and generalization. Indetail:

  • The reliability of the withdrawal signal is insufficiently established. The remaining concern is whether aggregate success is a reliable proxy for the marginal value of guidance, not that an acknowledged empirical motivation was presented as a universal theorem.

  • Generalization of the scheduling rule remains limited. The three configurations and ID/OOD evaluations provide meaningful evidence, but all models belong to Qwen2.5-Instruct and teachers are trained on their corresponding environments. Held-out tasks under a fixed training setup do not establish robustness when the training distribution changes, the teacher has different strengths, or model families differ.

Questions for authors
  • How many independent training seeds underlie each result? Can you provide seed-level final held-out success and withdrawal-time variability for GATS, GRPO, and the closest fixed-schedule baseline, rather than only repeated evaluations of a single model?

  • At the GRPO compute budgets in Table 3, what are the held-out success rates of the corresponding checkpoints? The existing teacher-inclusive accounting should be retained; this question concerns generalization at those budgets.

Reviewer_WNNHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the cold-start problem in outcome-based agentic RL, where sparse rewards can prevent an initially weak policy from discovering successful trajectories. Given access to a teacher model, the paper asks how teacher guidance should be scheduled during RL. Existing RL+OPD approaches either retain teacher guidance throughout training or remove it according to a predefined schedule, risking either premature withdrawal or excessive reliance on the teacher. The paper observes that the benefit of OPD correlates with the teacher–student performance gap and proposes GATS, which dynamically weights the OPD objective according to this gap. Guidance is stronger when the student substantially underperforms the teacher and decreases as the gap closes. Experiments on ALFWorld, WebShop, and ScienceWorld with several Qwen2.5 teacher–student configurations show improvements over GRPO and existing GRPO+OPD baselines.

Critical strengths
  • Addresses an important practical problem in agentic RL: learning efficiently from sparse outcome rewards when teacher supervision is available.
  • GATS is simple and intuitive, replacing manually specified OPD schedules with an adaptive criterion based on the teacher–student performance gap.
  • The method achieves strong empirical results relative to GRPO and the considered GRPO+OPD baselines.
  • The paper is clear and well written, and explicitly reports teacher training cost.
Critical weaknesses
  • Limited novelty. Relative to prior RL+OPD methods, the main contribution is an adaptive rule for weighting OPD based on the teacher–student performance gap. This appears relatively incremental without stronger comparisons to other adaptive teacher-guidance approaches.
  • Potential brittleness under heterogeneous task difficulty. GATS relies on aggregate performance to determine the amount of teacher guidance. If the task pool contains substantially different difficulty levels, strong performance on easy tasks could reduce guidance prematurely for harder tasks; conversely, difficult tasks could cause unnecessary teacher usage on already-solved easy tasks. A task- or instance-dependent scheduling mechanism may be more appropriate.
  • Dependence on teacher competence. The claim that a relatively small teacher is sufficient may not generalize to harder task distributions.
Questions for authors
  • Why are the comparisons primarily restricted to GRPO+OPD variants? How does GATS compare with methods that adaptively provide teacher hints or other forms of online teacher guidance?
  • In Table 1, why do the GRPO+OPD methods, including GATS, not consistently improve with stronger students and/or teachers, whereas GRPO and prompting appear to benefit more consistently from increased student capacity?
  • How does GATS behave when task difficulty is heterogeneous?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.