- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper trains a student agent with a combination of outcome-reward RL and token-level supervision from a teacher, and makes the weight on the teacher term depend on the measured gap between the teacher's and the student's success rates. The weight shrinks as the gap closes, and the teacher is switched off once the student reaches the teacher's reference score, after which training continues with reward only. The authors argue that guidance helps during the sparse-reward cold start but hurts later, and they support this with a small study of OPD gain against the teacher–student gap, experiments on ALFWorld, WebShop and ScienceWorld with three teacher–student pairs, and comparisons against fixed, annealed and divergence-based supervision schedules.
Critical strengths
-
The idea is simple and the motivation is easy to follow: guidance while the student is behind, none once it catches up. The rule itself is stated plainly enough.
-
The schedule comparisons are more thorough than usual. The paper sweeps the horizon of a linear schedule and also calibrates it to match the withdrawal steps.
Critical weaknesses
-
The two distillation baselines do much worse than plain RL without any teacher, in two of the three settings by a wide margin. The paper explains this as the student being held at the teacher's ceiling, but a drop of that size on a benchmark where the teacher itself does fine does not look like a ceiling effect. If the baseline implementations are undertuned, the gain attributed to the adaptive schedule is inflated, and this is the comparison the whole paper rests on.
-
Each configuration appears to be trained once, and the reported variation comes from repeated evaluation of one checkpoint rather than from repeated training. Given that the gap-utility study behind the design uses short runs on a few checkpoints, I would want to see the ordering hold across seeds before reading much into the margins between GATS and the tuned linear schedule, which are small.
Questions for authors
-
Why does fixed OPD fall so far below plain RL in the first two configurations, and were those baselines tuned to the same degree as GATS?
-
Do the main comparisons come from single training runs? If so, does the ordering survive additional seeds?
-
A short linear schedule chosen by sweeping comes close to GATS on ALFWorld. What distinguishes the adaptive rule from a well-tuned fixed one, beyond not needing the sweep?