acceptodds
Under review as a conference paper at ICLR 2027

HAIL: Heterogeneity-Aware Learning from Human Interventions with Suboptimal Experts

Abstract

Human-in-the-loop reinforcement learning (HITL-RL) has recently attracted increasing attention as a means to improve the learning efficiency and safety of reinforcement learning in complex tasks by incorporating human expert interventions. However, existing methods often implicitly assume that human experts can consistently provide high-quality guidance during training. In practice, human experts are often suboptimal, and the supervision they provide exhibits both intervention-level heterogeneity and temporal-level heterogeneity. To address these issues, we propose HAIL (Heterogeneity-Aware Learning from Human Interventions with Suboptimal Experts), a framework that explicitly models the heterogeneity of human supervision provided by suboptimal experts. Specifically, Quality-Weighted Behavioral Learning (QWBL) estimates the relative quality of different intervention behaviors and dynamically adjusts their optimization weights, allowing intervention samples with different quality levels to contribute differently to policy optimization. Effective-Horizon Retention (EH) selects preference information with higher learning value from different predicted future states while retaining part of the remaining candidates through sparse tail sampling, reducing the interference from preference information with lower learning value while maintaining trajectory-level supervision coverage. Experimental results in autonomous driving and continuous control environments demonstrate that HAIL improves policy performance with suboptimal human experts.

Primary area:
reinforcement learning
Keywords:
Human-in-the-loop Reinforcement Learning, Reinforcement Learning, Preference Learning, Human Intervention, Deep Reinforcement Learning, Autonomous Driving, Suboptimal Expert

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3b8TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the heterogeneity of human interventions in human-in-the-loop reinforcement learning. It proposes HAIL, a framework that incorporates Quality-Weighted Behavioral Learning (QWBL) and Effective-Horizon Retention (EH) to handle heterogeneity at the intervention and temporal levels, respectively. Experiments on MetaDrive and Hopper-v5 show improved performance over representative baselines while reducing intervention costs.

Critical strengths
  1. HAIL significantly outperforms baselines while requiring fewer interventions. Ablation studies also empirically support the proposed QWBL and EH modules.
Critical weaknesses
  1. While expert demonstrations exhibit heterogeneity—meaning they are not necessarily optimal—this does not imply that the interventions are erroneous or unreliable. The paper repeatedly emphasizes the negative impact of low-quality interventions but fails to adequately explain the real-world scenarios in which experts would make such severely flawed interventions. Consequently, the practical significance of the problem being addressed appears overstated.
  2. There are concerns regarding the validity of using an SAC policy to simulate human experts. A pre-trained SAC policy generates human interventions rather than actual human participants. In MetaTrader 5, the simulated expert's success rate is only 36%; would a real human expert have a success rate of only 36%? This raises doubts about whether the simulation accurately represents human expert behavior. Furthermore, when a higher-quality SAC expert is used, the performance gap between HAIL and PPL narrows significantly. This suggests that HAIL's advantages might stem primarily from handling poor decisions made by an unconstrained RL policy, rather than from addressing the specific characteristics of heterogeneity found in real human expert interventions.
  3. Issues regarding the experimental environments. MetaDrive lacks clearly defined environment settings. Although the paper introduces Hopper-v5 as a supplementary continuous control task, it fails to adequately justify its relevance for evaluating the proposed "heterogeneity-aware mechanism." The paper provides limited evidence of generalization across tasks with different dynamics, observation spaces, and intervention characteristics.
Questions for authors
  1. While expert demonstrations exhibit heterogeneity—meaning they are not necessarily optimal—this does not imply that the interventions are erroneous or unreliable. The paper repeatedly emphasizes the negative impact of low-quality interventions but fails to adequately explain the real-world scenarios in which experts would make such severely flawed interventions. Consequently, the practical significance of the problem being addressed appears overstated.
  2. There are concerns regarding the validity of using an SAC policy to simulate human experts. A pre-trained SAC policy generates human interventions rather than actual human participants. In MetaTrader 5, the simulated expert's success rate is only 36%; would a real human expert have a success rate of only 36%? This raises doubts about whether the simulation accurately represents human expert behavior. Furthermore, when a higher-quality SAC expert is used, the performance gap between HAIL and PPL narrows significantly. This suggests that HAIL's advantages might stem primarily from handling poor decisions made by an unconstrained RL policy, rather than from addressing the specific characteristics of heterogeneity found in real human expert interventions.
  3. QWBL assigns quality labels to intervention segments based on execution outcomes (including task completion, collisions, and episode returns). However, unsuccessful segments may contain appropriate individual actions, while successful segments may include suboptimal decisions. Although the evaluator ranks segments with high accuracy, this does not sufficiently demonstrate its ability to assess individual action quality. Additionally, the local consistency loss in Equation (3) tends to homogenize quality scores within a segment, potentially masking meaningful differences between individual interventions.
  4. EH uses the Euclidean distance between the predicted future state and the state resulting from the intervention as a proxy for preference validity. However, geometric similarity in the state space does not necessarily imply similar optimal actions or behavioral preferences, especially in safety-critical scenarios.
  5. QWBL reduces the contribution of low-quality interventions to behavioral learning, whereas EH constructs preference pairs based on the assumption that expert actions are superior to agent actions. Consequently, even if an intervention is rated as low-quality, potentially erroneous expert preferences can still influence policy optimization. This creates a potential inconsistency between the two optimization objectives: behavioral learning accounts for intervention quality, while preference optimization does not directly incorporate it.
  6. Issues regarding the experimental environments. MetaDrive lacks clearly defined environment settings. Although the paper introduces Hopper-v5 as a supplementary continuous control task, it fails to adequately justify its relevance for evaluating the proposed "heterogeneity-aware mechanism." The paper provides limited evidence of generalization across tasks with different dynamics, observation spaces, and intervention characteristics.
  7. HAIL requires an additional offline dataset for quality ranking and a pre-trained quality evaluator, whereas baseline comparisons primarily control for the number of online training steps. The additional costs associated with offline data collection, quality evaluator training, and trajectory processing have not been sufficiently considered.
Reviewer_uiMKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In Human-in-the-Loop Reinforcement Learning, the quality of human expert supervision data is not always consistently high. To address this issue, the authors propose QWBL (Quality-Weighted Behavioral Learning), which dynamically adjusts the learning weights of expert intervention behaviors according to their quality. Additionally, the authors introduce an Effective-Horizon Retention (EH) mechanism to select future states with high learning value for preference learning. Finally, the authors compare their proposed HAIL (Heterogeneity-Aware Learning from Human Interventions with Suboptimal Experts) framework against multiple baselines, including PVP and PPL, in autonomous driving scenarios. Experimental results demonstrate that HAIL achieves a higher Success Rate and a lower Takeover Cost on MetaDrive and Hopper-v5, respectively.

Critical strengths
  1. The paper introduces an innovative approach that maps the quality of expert behaviors to model learning weights, thereby accelerating model convergence.

  2. The paper incorporates future state prediction from PPL and selects expert interventions and future states with high learning value, which can be regarded as a denoising process.

  3. The authors simulate expert behaviors of varying quality using SAC models at different training stages. Under the same training budget, HAIL outperforms PPL.

  4. HAIL outperforms multiple baselines, including PVP and PPL. The authors also conduct extensive ablation studies to demonstrate the effectiveness of each component.

Critical weaknesses
  1. The Quality Evaluator is trained to score segments but is used to score state-action pairs during inference. It remains unclear whether the training and inference procedures are consistent, as the paper does not explain this discrepancy. Additionally, how is the final task outcome propagated back to intermediate segments? Outcome-based labels cannot accurately reflect the quality of individual expert actions.

  2. The experimental results for the Quality Evaluator only demonstrate ranking consistency, without establishing a relationship between the quality of different segments and the final task reward.

  3. The EH module selects high-value learnable states based on the similarity between the original state and future states while retaining one tail state. However, similarity measured by Euclidean distance does not necessarily indicate preference effectiveness. Experimental validation is needed to support this assumption.

  4. The ablation studies on EH only demonstrate that the EH module is effective, but do not establish the effectiveness of selecting Core Future States based on Euclidean distance. Additional ablation experiments should be conducted using only Core, Tail, and Random future states, respectively.

Questions for authors

Ref. Weaknesses 1–4.

Reviewer_jDssOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HAIL, which combines Quality-Weighted Behavioral Learning (QWBL) and Effective-Horizon Retention (EH) to address variation in intervention quality and the applicability of propagated preferences across future states. Experiments on MetaDrive and Hopper report improved policy performance with lower online takeover costs.

Critical strengths

The paper investigates both the intervention quality and how far an intervention’s preference remains applicable. HAIL treats intervention quality and preference propagation as complementary problems and improves the ideas of PPL, which is well motivated. Ablations comparing continuous weighting, uniform down-weighting, binary filtering, and different future-state selection strategies provide useful support for the design choices.

Critical weaknesses
  1. The paper lacks trajectory-level visualizations connecting QWBL’s quality assessment to actual intervention behavior. Figure 4 only shows average quality scores and weights. To better understand QWBL, I'd like to see representative high-quality and low-quality intervention trajectories with per-step quality scores.
  2. The evaluation does not involve real human experts. All interventions come from pretrained SAC policies, and the expert-quality comparison uses different checkpoints from the same training run. This does not establish robustness to fatigue, delayed reactions, or occasional incorrect corrections from human participants.
Questions for authors
  1. In Line 96-98, you stressed that "Future states deviate from the original intervention state to different degrees, resulting in the preference information from the same intervention having different learning values." Can you evaluate whether smaller state deviation corresponds to more reliable propagated preferences?
  2. Since HAIL targets heterogeneous intervention quality, could you compare QWBL-only and EH-only variants under increasing proportions of low-quality interventions to determine which is more robust?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.