- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the heterogeneity of human interventions in human-in-the-loop reinforcement learning. It proposes HAIL, a framework that incorporates Quality-Weighted Behavioral Learning (QWBL) and Effective-Horizon Retention (EH) to handle heterogeneity at the intervention and temporal levels, respectively. Experiments on MetaDrive and Hopper-v5 show improved performance over representative baselines while reducing intervention costs.
Critical strengths
- HAIL significantly outperforms baselines while requiring fewer interventions. Ablation studies also empirically support the proposed QWBL and EH modules.
Critical weaknesses
- While expert demonstrations exhibit heterogeneity—meaning they are not necessarily optimal—this does not imply that the interventions are erroneous or unreliable. The paper repeatedly emphasizes the negative impact of low-quality interventions but fails to adequately explain the real-world scenarios in which experts would make such severely flawed interventions. Consequently, the practical significance of the problem being addressed appears overstated.
- There are concerns regarding the validity of using an SAC policy to simulate human experts. A pre-trained SAC policy generates human interventions rather than actual human participants. In MetaTrader 5, the simulated expert's success rate is only 36%; would a real human expert have a success rate of only 36%? This raises doubts about whether the simulation accurately represents human expert behavior. Furthermore, when a higher-quality SAC expert is used, the performance gap between HAIL and PPL narrows significantly. This suggests that HAIL's advantages might stem primarily from handling poor decisions made by an unconstrained RL policy, rather than from addressing the specific characteristics of heterogeneity found in real human expert interventions.
- Issues regarding the experimental environments. MetaDrive lacks clearly defined environment settings. Although the paper introduces Hopper-v5 as a supplementary continuous control task, it fails to adequately justify its relevance for evaluating the proposed "heterogeneity-aware mechanism." The paper provides limited evidence of generalization across tasks with different dynamics, observation spaces, and intervention characteristics.
Questions for authors
- While expert demonstrations exhibit heterogeneity—meaning they are not necessarily optimal—this does not imply that the interventions are erroneous or unreliable. The paper repeatedly emphasizes the negative impact of low-quality interventions but fails to adequately explain the real-world scenarios in which experts would make such severely flawed interventions. Consequently, the practical significance of the problem being addressed appears overstated.
- There are concerns regarding the validity of using an SAC policy to simulate human experts. A pre-trained SAC policy generates human interventions rather than actual human participants. In MetaTrader 5, the simulated expert's success rate is only 36%; would a real human expert have a success rate of only 36%? This raises doubts about whether the simulation accurately represents human expert behavior. Furthermore, when a higher-quality SAC expert is used, the performance gap between HAIL and PPL narrows significantly. This suggests that HAIL's advantages might stem primarily from handling poor decisions made by an unconstrained RL policy, rather than from addressing the specific characteristics of heterogeneity found in real human expert interventions.
- QWBL assigns quality labels to intervention segments based on execution outcomes (including task completion, collisions, and episode returns). However, unsuccessful segments may contain appropriate individual actions, while successful segments may include suboptimal decisions. Although the evaluator ranks segments with high accuracy, this does not sufficiently demonstrate its ability to assess individual action quality. Additionally, the local consistency loss in Equation (3) tends to homogenize quality scores within a segment, potentially masking meaningful differences between individual interventions.
- EH uses the Euclidean distance between the predicted future state and the state resulting from the intervention as a proxy for preference validity. However, geometric similarity in the state space does not necessarily imply similar optimal actions or behavioral preferences, especially in safety-critical scenarios.
- QWBL reduces the contribution of low-quality interventions to behavioral learning, whereas EH constructs preference pairs based on the assumption that expert actions are superior to agent actions. Consequently, even if an intervention is rated as low-quality, potentially erroneous expert preferences can still influence policy optimization. This creates a potential inconsistency between the two optimization objectives: behavioral learning accounts for intervention quality, while preference optimization does not directly incorporate it.
- Issues regarding the experimental environments. MetaDrive lacks clearly defined environment settings. Although the paper introduces Hopper-v5 as a supplementary continuous control task, it fails to adequately justify its relevance for evaluating the proposed "heterogeneity-aware mechanism." The paper provides limited evidence of generalization across tasks with different dynamics, observation spaces, and intervention characteristics.
- HAIL requires an additional offline dataset for quality ranking and a pre-trained quality evaluator, whereas baseline comparisons primarily control for the number of online training steps. The additional costs associated with offline data collection, quality evaluator training, and trajectory processing have not been sufficiently considered.