acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Elite Replay: Learning Beyond Suboptimal Experts in Human-in-the-Loop Reinforcement Learning

Abstract

Human-in-the-loop reinforcement learning (HIL-RL) uses expert intervention and demonstrations to improve training safety and learning efficiency. Many existing methods use expert behavior as an important source of positive supervision, which is effective when reliable expert guidance is available. However, when the expert is suboptimal, continued reliance on such supervision can bias policy learning toward the expert's behavior and make it difficult for the agent to fully exploit better behaviors discovered through its own exploration. To address this problem, we propose Dynamic Elite Replay (DER), an intervention learning framework that integrates elite sample selection and weighted elite learning, enabling the agent to fully exploit high-quality behaviors discovered through its own exploration. DER first identifies high-quality agent-generated experience according to task performance and safety. When complete successful trajectories are scarce, it further extracts safe progress segments that capture useful local improvements. During replay, retained elite samples are re-evaluated using current Q-value advantage, twin-critic agreement, sample freshness, and disagreement with the supervisor, reducing the influence of outdated or unreliable experience. The reweighted elite experience is then used for behavior-cloning and critic updates to reinforce validated high-quality agent behaviors. Experiments in MetaDrive show that DER can achieve performance above a fixed suboptimal supervisor while reducing safety failures. These results suggest that expert intervention can guide training without necessarily defining the final performance ceiling when high-quality behaviors discovered by the agent are effectively identified and reused.

Primary area:
reinforcement learning
Keywords:
human-in-the-loop reinforcement learning, suboptimal expert, elite experience selection, weighted experience replay, self-imitation learning, safe reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jCYpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses HIL-RL where the supervisor is useful but suboptimal, and argues that continuing to imitate such a supervisor caps the policy below behaviors the agent discovers on its own. It proposes Dynamic Elite Replay (DER): maintain an agent-only elite replay path that excludes takeover transitions; select high-quality agent experience at two granularities (complete successful episodes and shorter safe progress segments); and re-evaluate retained elite transitions at replay time using current Q-support/advantage, twin-critic agreement, sample freshness, and supervisor disagreement, driving weighted self-imitation and critic TD updates. On MetaDrive with a fixed suboptimal PPO proxy supervisor, DER exceeds the supervisor and the TD3, HACO, PVP, PVP4Real, and reward-matched PVP+Reward baselines on success, route completion, and reward, with a cumulative ablation and an eight-run stability study.

Critical strengths

(i)One genuinely inverted design: unlike most HIL methods that prioritize takeover data, DER excludes takeover transitions and reuses only validated agent behavior, so the supervisor governs safety without fixing the ceiling; the two-level (episode + segment) selection sensibly handles sparse early successes. (ii)The reward-matched PVP+Reward control rules out reward access alone as the source of the gain, strengthening the causal account.

Critical weaknesses

(i)The contribution is mainly a recombination, and the most direct competitors go uncompared: surpassing a suboptimal expert appears in RLIF/TS2C/T-REX–D-REX, HIL sample selection in E2HiL, and re-evaluating replay against the current policy in Lucid Dreaming/ReF-ER — all cited but none used as baselines, so the central claim is untested where it matters most. (ii)Weak evaluation: a single simulator and single fixed supervisor, endpoints drawn from the validation stream used for model selection rather than a held-out test, with four runs, large variance, and no significance testing.

Questions for authors

(i)Can you add a direct comparison with RLIF, TS2C, and E2HiL under the same MetaDrive protocol? These are the most direct competitors for the paper's central claims, and their absence materially affects the attribution of the gain. (ii)Can you report endpoints on seeds/episodes fully disjoint from the validation stream used for checkpoint selection? How much does the DER-versus-supervisor margin change? (iii) Does the gain hold under a different supervisor quality (stronger or noisier)?

Reviewer_RGz4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Dynamic Elite Replay (DER) for human-in-the-loop RL. DER replays the agent's own best episodes and short progress segments for self-imitation, leaving takeover data out. In MetaDrive with a fixed, fairly weak PPO supervisor, DER ends with much higher success than PVP and the other baselines and with fewer safety failures. The paper credits elite selection for letting the agent learn beyond its expert.

Critical strengths
  • The framing stands out. Instead of treating supervisor actions as the target, DER tries to preserve good behavior the agent found itself. That changes what human-in-the-loop RL is aiming for, rather than tuning how takeovers are used.
  • The improvement over prior methods is reliable. DER leads PVP by a wide margin even with only four runs per method, and all eight reported DER runs clear the supervisor on every metric.
  • Segment-level selection is the one component that clearly pays off. In Table 2, adding progress segments on top of episode-level elite replay is the only step whose effect stands out from seed variance, and that comparison is like-for-like.
Critical weaknesses
  • Table 1 does not look like a like-for-like comparison. DER uses the same validation stream that produces the reported numbers to roll the actor back to its confirmed best checkpoint whenever performance drops, over the last 40% of training. Nothing indicates the baselines get this, and a shared validation schedule isn't the same thing. With no DER run that has rollback turned off, there's no way to split the gap between elite selection and this stabilization. The paper lists the missing no-rollback run as a limitation but does not treat it as a threat to the main result.
  • The mechanism is a stack of heuristics, self-imitation plus several replay weights and gates, not far from SIL, PER and ReF-ER. So the novelty rides on extra knobs, and nothing separates elite selection from the rest of the system. The training script already implements a reward-matched uniform replay control that keeps DER's backbone and rollback and switches off only the elite branches, but no result for it is reported. Past segment selection, the Table 2 steps are within noise, so the ablation doesn't back the full two-stage design.
  • This setup can't separate DER from PVP on the headline claim. With a supervisor at 0.28 success, PVP already exceeds the supervisor on all three main metrics, so learning beyond the expert is not unique to DER. The decisive test, a supervisor strong enough that PVP stays below it, isn't run.
Questions for authors
  1. Could you report DER with rollback turned off, or apply the same checkpoint selection to every baseline, together with the reward-matched uniform replay control? If rollback-free DER stays well above PVP and that control falls clearly below DER, that would address most of my concern about elite selection.
  2. Did any baseline get validation-based checkpoint selection, and were all methods evaluated on the same 50 scenarios?
  3. Does DER still outperform PVP with a supervisor strong enough that PVP doesn't exceed it? If not, would you scope the title and abstract to this weak-supervisor setting?
Reviewer_ttraOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the performance ceiling issue of human-in-the-loop reinforcement learning with suboptimal supervisors. It proposes Dynamic Elite Replay (DER), which performs two-level admission selection over full episodes and local segments, and reweights retained transitions at replay time based on multiple reliability signals. The weighted elite samples are used for weighted behavior cloning and weighted TD learning. Evaluated in MetaDrive, DER outperforms several HIL baselines and surpasses the suboptimal supervisor. Ablation studies validate the contribution of key components.

Critical strengths

The clean separation of admission selection and replay-time reweighting is a principled architectural choice. It addresses the risk that previously good trajectories may become outdated for the current policy. This is the primary reason I consider the work worthy of revision.

Critical weaknesses
  1. The performance gains attributed to DER cannot be reliably isolated from the actor rollback stabilization mechanism. Without ablation results disabling rollback, we cannot confirm whether the reported improvements stem from the proposed elite replay algorithm or this auxiliary stabilization trick. This directly undermines the validity of the core empirical evidence and weakens confidence in the claimed contribution.
  2. Prior HIL baselines erase self-discovered advantageous agent behaviors. This key motivation is only supported indirectly by final performance comparisons. There is no targeted diagnostic experiment to verify this central phenomenon. Since the whole design of DER is built around solving this exact problem, the absence of direct empirical proof creates a fundamental gap between the paper’s motivation and its experimental validation.
Questions for authors
  1. Can you provide results for DER with the actor rollback mechanism disabled?
  2. Do you have diagnostic measurements or visualizations that directly demonstrate baseline HIL methods erase the agent’s self-discovered high-performance behaviors, rather than only showing final reward gaps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.