acceptodds
Withdrawn from ICLR 2027

Adaptive Expert Guidance For Efficient On-Policy Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

With massively parallel simulation, on-policy RL methods such as PPO have become standard in many domains. However, learning from scratch is sample-inefficient and fails to exploit the potential existence of a suboptimal expert, such as a heuristic, a model-based controller, or a policy trained on a related task. Such an expert is often available and can guide early training, but its sub-optimality limits final performance. The challenge then becomes balancing expert guidance against learning from rewards. Existing methods set the expert's influence through a blending weight, a schedule, or an evaluation-driven curriculum. Alternatively, they adapt it with additional learned components such as critics over expert actions or auxiliary agents. However, none optimizes it using the same on-policy objective as the policy itself. We propose a method in which the learner and the expert alternate control within each training episode, and the expert's share of control is a single learnable parameter optimized jointly with the policy. The learner benefits from the expert early in training, but its share of control declines as the learner becomes more competent, until eventually vanishing completely. This leaves the learner acting alone and better than the suboptimal expert. We evaluate our method on 34 tasks across two benchmarks, spanning discrete and continuous action spaces, using both learned and model-based experts. Our method improves sample efficiency over guided and unguided baselines while requiring minimal hyperparameter variation. The expert's share decays to zero as the learner improves, vanishing when the expert is no longer useful.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization, On-Policy Learning, Policy Guidance, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FgRhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of efficient on-policy reinforcement learning. To address this issue, the paper presents a new method, in which expert and learner alternate control in fixed-length blocks, with a learnable probability determining expert participation. An expert-use penalty encourages withdrawal, while imitation on expert-controlled blocks trains the learner from unused experience. Experiments across 25 DMControl and nine MiniGrid tasks show improved sample efficiency and aggregated final performance.

Critical strengths
  • The paper provides a simple interpretable mechanism for adapting expert participation.
  • The experimental results are sufficient and evaluate the learner independently of the expert.
Critical weaknesses
  • The paper reports the improved performance in terms of normalized return. However, it seems the improvement is marginal.
  • The suboptimal policy from the expert affects the final policy generated by both the expert and the learner when adopting blocks. It may result in long training time and high computational costs.
  • It could discuss how to balance the block size.
  • Eqs. (4) – (13) use complete block rewards, boundary-state values, and scores accumulated over an entire control block. However, Table 5 specifies a PPO rollout length of 30 steps and a control-block length of 100 steps. Thus, a complete block extends across multiple PPO collection windows. The paper does not explain this and its implications.
Questions for authors
  1. How to define the price of expert use? Please clarify this.
  2. How is the expert share automatically adjusted during the process?
  3. It is not clear how to form a block. How many steps are formed as a block? How the block size affects the performance? It would be better to add an experiment to assess this.
  4. It is unclear whether policy parameters remain fixed across PPO collection windows. Please explain it.
Reviewer_Y5CzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate an RL setting where an expert is available to perform actions during rollouts. The authors employ a switching function that switches from using the trained policy and the expert to carry out rollouts. The training signal consists of a combination of typical RL loss terms (PPO) and imitation learning terms. To incentivize decreasing expert intervention, the authors introduce a further penalty term that increases as the standard deviation of the advantage increases.

Critical strengths

The paper aims to improve RL performance with the help of an expert, which is a relevant topic.

The paper is well-written and easy to read. The main ideas are clearly stated and motivated.

The paper provides a theoretical analysis of the proposed method.

The method is compared against relevant baselines.

Critical weaknesses

The setting assumes the existence of an expert that can be introduced at will during rollouts. This limits the applicability of the proposed method.

The DMC control tasks are fairly saturated at this point. While the authors report an improvement in average performance in DMC, it is mostly marginal, and there is no benchmark where the proposed approach results in a definitive advantage. It would have been interesting to explore more complex benchmarks, such as humanoidbench, which are have been less investigated.

Questions for authors

In section 4, Q2 is unclear to me: in what sense does Eq. 13 predict anything? It is only an update rule, not a theoretical result that follows from the proposed algorithm.

The choice of experts in the benchmarks seems somewhat specific. Is there a particular reason for the chosen experts? Could a richer mix of experts have been used? Why do the authors choose experts that reach 60/70% of performance at convergence?

Why is the standard deviation of the advantage a good scaling term for the expert cost in (11)?

Reviewer_DPDBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for training an RL agent with a fixed, potentially suboptimal expert. Every kk steps, the method randomly selects either the expert or learner to act for the next block. It learns the probability of selecting the expert while simultaneously learning policy. An additional penalty encourages the agent to stop using the expert. An imitation loss is used to teach the learner from the expert's actions. The authors test their approach using PPO on 34 DMControl and MiniGrid tasks, reporting improvements over guided and unguided baselines.

Critical strengths
  • The method combines expert-guided exploration and imitation in a useful/intuitive way. Specifically, the expert can bring the learner to states it would otherwise struggle to reach, while imitation allows the learner to benefit from the expert's actions. The method only requires action queries, so the expert can also be a controller or planner.

  • The experiments provide useful evidence across continuous and discrete action spaces. In particular, varying the MPC expert's model error while keeping the task fixed helps show how the method responds to different expert strengths. The experiment with an expert that never completes the task is also interesting.

Critical weaknesses
  • The benefit of learning how much to use the expert isn't clearly separated from differences in the baselines. The scheduled baseline starts with λ=1\lambda=1, while the proposed method starts at 0.50.5 and limits it to 0.70.7. The schedules also stop using the expert within 5M steps but the proposed method sometimes uses it much longer. Without the price, λ\lambda doesn't reach zero on many of the DMC tasks. Also, many trajectories in Fig. 8 look like near-linear decays to zero within 2–4M steps. So this makes it seem like withdrawal may be driven mostly by the constant price rather than the block-advantage comparison.

  • It's unclear how the algorithm described in Section 3 is implemented with PPO. Equations 4–5 use advantages computed over complete blocks, but Table 5 gives a block length of 100 and a rollout length of 30. The paper doesn't explain how blocks spanning multiple rollouts are handled, including whether the policy changes before a block ends. It also doesn't specify how the block advantages are used in PPO's clipped objective.

Questions for authors
  1. How are the 100-step block advantages computed from 30-step rollouts? Can you clarify whether the policy and λ\lambda change during a block, and how Equations 4–5 are implemented with PPO clipping?

  2. Did you compare against schedules that start at the same λ\lambda and allow the expert to be used for longer than 5M steps (keeping everything else unchanged)?

  3. Did you measure sample efficiency and final return for different values of kk? Figure 6c uses an untrained learner without a critic, so it's hard to tell whether the preferred block length remains useful as the policy learns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.