- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses HIL-RL where the supervisor is useful but suboptimal, and argues that continuing to imitate such a supervisor caps the policy below behaviors the agent discovers on its own. It proposes Dynamic Elite Replay (DER): maintain an agent-only elite replay path that excludes takeover transitions; select high-quality agent experience at two granularities (complete successful episodes and shorter safe progress segments); and re-evaluate retained elite transitions at replay time using current Q-support/advantage, twin-critic agreement, sample freshness, and supervisor disagreement, driving weighted self-imitation and critic TD updates. On MetaDrive with a fixed suboptimal PPO proxy supervisor, DER exceeds the supervisor and the TD3, HACO, PVP, PVP4Real, and reward-matched PVP+Reward baselines on success, route completion, and reward, with a cumulative ablation and an eight-run stability study.
Critical strengths
(i)One genuinely inverted design: unlike most HIL methods that prioritize takeover data, DER excludes takeover transitions and reuses only validated agent behavior, so the supervisor governs safety without fixing the ceiling; the two-level (episode + segment) selection sensibly handles sparse early successes. (ii)The reward-matched PVP+Reward control rules out reward access alone as the source of the gain, strengthening the causal account.
Critical weaknesses
(i)The contribution is mainly a recombination, and the most direct competitors go uncompared: surpassing a suboptimal expert appears in RLIF/TS2C/T-REX–D-REX, HIL sample selection in E2HiL, and re-evaluating replay against the current policy in Lucid Dreaming/ReF-ER — all cited but none used as baselines, so the central claim is untested where it matters most. (ii)Weak evaluation: a single simulator and single fixed supervisor, endpoints drawn from the validation stream used for model selection rather than a held-out test, with four runs, large variance, and no significance testing.
Questions for authors
(i)Can you add a direct comparison with RLIF, TS2C, and E2HiL under the same MetaDrive protocol? These are the most direct competitors for the paper's central claims, and their absence materially affects the attribution of the gain. (ii)Can you report endpoints on seeds/episodes fully disjoint from the validation stream used for checkpoint selection? How much does the DER-versus-supervisor margin change? (iii) Does the gain hold under a different supervisor quality (stronger or noisier)?