acceptodds
Under review as a conference paper at ICLR 2027

How Should LLMs Teach RL Agents? Action Assistance and Competence-Adaptive Release

Abstract

Reinforcement learning (RL) in complex decision-making tasks often requires extensive environment interaction. Large language models (LLMs) can provide state-conditioned action advice, but success with teacher assistance does not imply that an RL agent can solve the task independently. We propose a competence-adaptive framework for LLM-guided RL that distinguishes two roles of teacher guidance: action intervention, which changes the trajectories collected from the environment, and demonstration supervision, which supports RL policy updates. During training, the framework executes the LLM teacher’s action with an adaptive probability. State–action pairs from actual interventions are stored and replayed to provide supervision alongside the RL objective. A fuzzy controller adjusts the intervention probability based on policy uncertainty, teacher-free evaluation success, and agreement between the RL agent’s proposed actions and the teacher’s actions, while retaining a positive base weight for demonstration supervision. We evaluate the framework on three MiniGrid task families, DoorKey, UnlockPickup, and MultiRoom, and Env2. Across six principal MiniGrid configurations, median final teacher-free success ranges from 90% to 100%. On DoorKey 5×5, the median number of environment steps to first reach 95% teacher-free success decreases from 64,000 to 6,000. Ablations show that persistent teacher-action takeover can impair independent performance, and that removing demonstration supervision slows learning on some tasks.

Primary area:
reinforcement learning
Keywords:
large language model, deep reinforcement learning, learning efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_epw4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how weaker student policies can internalize guidance from stronger LLM teachers. It proposes an adaptive framework for LLM-guided reinforcement learning in which teacher guidance serves two roles: action intervention and direct supervision. A controller adjusts the intervention probability based on policy entropy, teacher-free success, and student–teacher agreement. Experiments on six MiniGrid environments and a custom continuous-control environment show that sparse intervention combined with replay accelerates learning relative to PPO baselines.

Critical strengths
  1. The framework explicitly separates the teacher’s role in guiding exploration from its role in providing supervision. Combining these roles is conceptually well motivated, and the empirical results suggest that this combination can improve learning efficiency.

  2. The appendices include substantial experiments examining the effects of intervention, supervision, and prompt components. These analyses help clarify which components contribute to performance and provide useful context for interpreting the main results.

Critical weaknesses
  1. The novelty relative to prior work is insufficiently established. The related-work discussion and comparisons focus primarily on LLM4Teacher while omitting several relevant lines of research. For example, POPE[1], BREAD[2], and PrefixRL[3] study how partial expert guidance can support productive exploration and improve subsequent unguided performance. These connections raise the question of whether the proposed two-role teacher design provides benefits beyond partial guidance alone. Int also investigates targeted interventions that promote student internalization. In the broader RL literature, HG-DAgger[4], ThriftyDAgger[5], and TGRL[6] examine how to balance intervention, direct teacher supervision, and reward-based learning. Without clearer distinctions and comparisons with relevant baselines, the novelty of combining intervention, learning from intervention data, and adaptive teacher influence remains unclear.

  2. The evidence for competence-adaptive control is inconclusive. Competence-adaptive control is presented as a central contribution, but the reported ablations do not clearly establish its advantage over a fixed intervention probability. In Table 8, fixing alpha at 0.089 yields higher median final success. In Table 9, fixed alpha matches the full method’s median of 11k steps to reach 95% success. These results weaken the empirical justification for the adaptive controller.

  3. Apparent inconsistencies and evaluation choices limit confidence in the results. Figure 4 reports an 18× speedup at 75% success on DoorKey 5×5, which appears inconsistent with the corresponding values in Table 2. In Tables 8 and 9, removing the supervised loss and removing stored teacher actions produce different results, although the method description appears to imply that stored teacher actions are used only for supervision. The distinction between these ablations therefore needs clarification. Finally, measuring steps to success by the first evaluation with at least 19 successful episodes out of 20 may introduce substantial variability. This is particularly consequential when comparing diagnostic variants with similar performance.

Questions for authors
  1. How does the method differ from prior work combining intervention, learning from intervention data, and adaptive teacher influence, including the work listed above? Can the authors provide comparisons with relevant baselines that isolate the benefits of the two-role teacher design and adaptive controller?

  2. Can the authors reconcile the DoorKey 5×5 speedup reported in Figure 4 with the corresponding values in Table 2? Can they also explain why removing the supervised loss and removing stored teacher actions yield different results in Tables 8 and 9?

  3. What motivates defining steps to success by the first evaluation with at least 19 successful episodes out of 20? How sensitive are the main conclusions to evaluation variability and to requiring sustained success across multiple evaluations?

Reviewer_tarFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how an LLM teacher can improve the sample efficiency of RL while allowing the trained student to act independently. It separates teacher guidance into action intervention during data collection and replay-based supervision and uses a fixed fuzzy controller to regulate both based on student uncertainty, teacher-free success, and student-teacher action agreement. Experiments on six MiniGrid configurations and Env2 report faster learning than teacher-free PPO and an LLM4Teach-style baseline.

Critical strengths
  1. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Limited novelty. Teacher-guided RL [1], online expert/student action mixing and gradual transfer of control [2,3], combining RL with teacher-policy supervision [4], and supervised reuse of collected teacher actions [5] have all been studied previously. Jump-Start RL further uses a guide policy to alter the states encountered during exploration while progressively reducing its influence [6], and TS2C studies intervention-based RL with imperfect online teachers [7]. LLM4Teach has already established the use of an LLM as a policy teacher for an RL student [8]. The proposed method mainly combines these existing components using a fixed fuzzy controller, while comparisons are restricted to PPO and an LLM4Teach-style baseline. In particular, the lack of empirical comparison with intervention-based RL methods such as TS2C makes it difficult to determine whether the contribution extends beyond replacing a conventional teacher with a heavily prompted LLM and introducing a heuristic intervention schedule.

  2. Limited evaluation of teacher quality. Although the method states that teacher actions need not be optimal, all three teacher models used for RL training first pass an open-loop screening test with success rates of 90%–100%, while the three models scoring 0%–5% are excluded. The only direct comparison between a trained student and its teacher acting alone is reported on Env2; no analogous comparison is provided for the six MiniGrid configurations. On Env2, Qwen3-14B acting alone succeeds in only 7/20 episodes, but all failures are timeouts rather than interceptions. This result establishes that the student outperforms this particular teacher interface, but does not fully characterize the underlying LLM's capability: the teacher is restricted to stateless, per-step numerical actions and may perform differently with a richer planning or action interface. Overall, the experiments neither characterize how learning degrades with moderately capable or unreliable teachers nor establish the student-teacher performance relationship beyond Env2.

  3. Incomplete prompt specification and reliance on readily verbalizable environments. Appendix J reports the frozen human instructions but describes parts of the system input only at a high level; the exact state serialization, MiniGrid action mapping, parsing procedure, and complete system prompt are not provided. At the same time, the disclosed prompts inject substantial task-specific information. In Env2, the prompt already specifies the desired direction and qualitative control magnitude for each missile, leaving the LLM largely to translate hand-engineered coaching instructions into numerical actions. In MiniGrid, the teacher receives a symbolic grid, orientation, currently available actions, and phase-specific objectives. When such a description exposes most of the relevant state and solution structure, especially in deterministic grid environments with simple known dynamics, a simulator may be constructed to act directly without interacting with the environment. The reported sample-efficiency gains therefore conflate the proposed learning framework with privileged symbolic state and task knowledge. As currently formulated, the method cannot generalize to environments whose states, dynamics, and useful control decisions cannot be faithfully and compactly expressed in natural language.

References for Weakness 1

[1] Torrey, L. and Taylor, M. Teaching on a Budget: Agents Advising Agents in Reinforcement Learning. AAMAS, 2013. ACM

[2] Ross, S., Gordon, G., and Bagnell, D. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011. PMLR

[3] Menda, K., Driggs-Campbell, K., and Kochenderfer, M. J. EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning. IROS, 2019. arXiv and publication metadata

[4] Schmitt, S. et al. Kickstarting Deep Reinforcement Learning. arXiv:1803.03835, 2018. arXiv

[5] İlhan, E., Gow, J., and Pérez-Liébana, D. Action Advising with Advice Imitation in Deep Reinforcement Learning. AAMAS, 2021. ACM

[6] Uchendu, I. et al. Jump-Start Reinforcement Learning. ICML, 2023. PMLR

[7] Xue, Z., Peng, Z., Li, Q., Liu, Z., and Zhou, B. Guarded Policy Optimization with Imperfect Online Demonstrations. ICLR, 2023. OpenReview

[8] Zhou, Z., Hu, B., Zhao, C., Zhang, P., and Liu, B. Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents. IJCAI, 2024. IJCAI

Questions for authors
  1. Appendix I reports an open-loop success rate of 8/8 for Qwen3-14B, whereas Appendix E reports only 7/20 success when the same model acts at every Env2 step. What differences in evaluation protocol explain this discrepancy?
Reviewer_GmQqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers an RL agent (PPO) with available online expert actions from a teacher (LLM). The paper quesitons how to combine online action selection by the teacher during the rollouts with imitation learning of the teacher actions on rollouts played using the agent policy. A heuristic is proposed, based on fuzzy selection rules and hand-selected features of the succeess and alignment between teacher and agent.

Critical strengths
  1. On the experiments the proposed method outperforms a baseline that uses the teacher diferently, and a naive baseline that ignores the teacher.
Critical weaknesses
  1. The idea is almost trivial, with almost no insight beyond "we tried something and it improved results".
  2. There is no discussion of on-policy/off-policy tradeoffs, and it is strange to use an on-policy algorithm (PPO) without on-policy action selction.
  3. The experiments are on toy domains.
  4. Overall, both the question explored, the investigation approach, the depth of the investigation, and the writing of the paper is more suitable for a class project than for a top-tier ML conference.
Questions for authors

I don't have any questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.