- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how weaker student policies can internalize guidance from stronger LLM teachers. It proposes an adaptive framework for LLM-guided reinforcement learning in which teacher guidance serves two roles: action intervention and direct supervision. A controller adjusts the intervention probability based on policy entropy, teacher-free success, and student–teacher agreement. Experiments on six MiniGrid environments and a custom continuous-control environment show that sparse intervention combined with replay accelerates learning relative to PPO baselines.
Critical strengths
-
The framework explicitly separates the teacher’s role in guiding exploration from its role in providing supervision. Combining these roles is conceptually well motivated, and the empirical results suggest that this combination can improve learning efficiency.
-
The appendices include substantial experiments examining the effects of intervention, supervision, and prompt components. These analyses help clarify which components contribute to performance and provide useful context for interpreting the main results.
Critical weaknesses
-
The novelty relative to prior work is insufficiently established. The related-work discussion and comparisons focus primarily on LLM4Teacher while omitting several relevant lines of research. For example, POPE[1], BREAD[2], and PrefixRL[3] study how partial expert guidance can support productive exploration and improve subsequent unguided performance. These connections raise the question of whether the proposed two-role teacher design provides benefits beyond partial guidance alone. Int also investigates targeted interventions that promote student internalization. In the broader RL literature, HG-DAgger[4], ThriftyDAgger[5], and TGRL[6] examine how to balance intervention, direct teacher supervision, and reward-based learning. Without clearer distinctions and comparisons with relevant baselines, the novelty of combining intervention, learning from intervention data, and adaptive teacher influence remains unclear.
-
The evidence for competence-adaptive control is inconclusive. Competence-adaptive control is presented as a central contribution, but the reported ablations do not clearly establish its advantage over a fixed intervention probability. In Table 8, fixing alpha at 0.089 yields higher median final success. In Table 9, fixed alpha matches the full method’s median of 11k steps to reach 95% success. These results weaken the empirical justification for the adaptive controller.
-
Apparent inconsistencies and evaluation choices limit confidence in the results. Figure 4 reports an 18× speedup at 75% success on DoorKey 5×5, which appears inconsistent with the corresponding values in Table 2. In Tables 8 and 9, removing the supervised loss and removing stored teacher actions produce different results, although the method description appears to imply that stored teacher actions are used only for supervision. The distinction between these ablations therefore needs clarification. Finally, measuring steps to success by the first evaluation with at least 19 successful episodes out of 20 may introduce substantial variability. This is particularly consequential when comparing diagnostic variants with similar performance.
Questions for authors
-
How does the method differ from prior work combining intervention, learning from intervention data, and adaptive teacher influence, including the work listed above? Can the authors provide comparisons with relevant baselines that isolate the benefits of the two-role teacher design and adaptive controller?
-
Can the authors reconcile the DoorKey 5×5 speedup reported in Figure 4 with the corresponding values in Table 2? Can they also explain why removing the supervised loss and removing stored teacher actions yield different results in Tables 8 and 9?
-
What motivates defining steps to success by the first evaluation with at least 19 successful episodes out of 20? How sensitive are the main conclusions to evaluation variability and to requiring sustained success across multiple evaluations?