- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReGIL, an intervention method for interactive imitation learning that considers whether a novice can recover after leaving expert support. It combines a learned support-persistence value, short dynamics rollouts, and a sequential intervention rule. Experiments on MetaDrive and a MetaWorld drawer-opening task show higher mean autonomous success under expert-data budgets than the compared methods.
Critical strengths
-
The distinction between leaving demonstration support and needing expert assistance is useful. A temporary deviation may resolve under the novice’s own actions. Considering both the predicted return and behavior after that return gives the intervention decision information that current-state novelty alone does not provide.
-
The budgeted results support an expert-data efficiency benefit. The methods share the behavior-cloning architecture and update schedule, and initial demonstrations are counted in the budget. Reporting both budgeted and final performance also makes the performance/data trade-off visible.
Critical weaknesses
-
I think there is a mismatch between Eq. (3) and the takeover rule. The stated value is zero outside support. If takeover does not occur at the first unsupported step, a persistent deviation would not produce the consecutive decreases required for later intervention. It is unclear whether the implemented discounted target resolves this.
-
I am not yet convinced of the HJ formulation’s benefit at a fixed expert-data budget. The no-HJ variant achieves higher success (0.856 vs. 0.830), but uses more expert data. A comparison at the same expert budget would make its contribution clearer.
Questions for authors
-
What is the exact discounted, terminal-aware value target used in the experiments? How does the implemented gate request takeover when the novice remains outside support, no return is predicted, and persistence estimates remain zero or nearly constant?
-
At matched expert-transition budgets, how does the full method compare with the no-HJ and no-sequential-evidence variants? Do the existing learning curves support an advantage for the full gate when expert supervision is controlled?
-
Please specify the RND target and predictor architectures, feature dimension, and input normalization. Are there diagnostics showing how well the resulting errors separate held-out expert states from states encountered during novice deviations? This would help assess the support indicator underlying both recovery prediction and persistence learning.
5 comments
W2 & Q2: Ablation Performance at Matched Expert-Data Budgets
We additionally evaluate the ablation variants under different expert-transition budgets, following the same experimental protocol and using four random seeds.
The results are as below:
| Expert-data Budget | Variant | Human Data | Success ↑ | Return ↑ | Progress ↑ |
|---|---|---|---|---|---|
| 800 | Full ReGIL | 800 ± 0 | 0.564 ± 0.171 | 291.6 ± 40.8 | 0.785 ± 0.105 |
| w/o HJ | 800 ± 0 | 0.544 ± 0.272 | 287.6 ± 66.4 | 0.766 ± 0.168 | |
| w/o Sequential Evidence | 800 ± 0 | 0.570 ± 0.222 | 294.0 ± 56.2 | 0.791 ± 0.139 | |
| 900 | Full ReGIL | 900 ± 0 | 0.770 ± 0.076 | 335.4 ± 16.1 | 0.892 ± 0.039 |
| w/o HJ | 900 ± 0 | 0.550 ± 0.273 | 296.5 ± 50.2 | 0.796 ± 0.129 | |
| w/o Sequential Evidence | 900 ± 0 | 0.595 ± 0.189 | 300.7 ± 39.9 | 0.809 ± 0.099 | |
| 1,000 (cap) | Full ReGIL | 945.5 ± 75.9 | 0.800 ± 0.063 | 335.2 ± 14.8 | 0.898 ± 0.031 |
| w/o HJ | 1,000 ± 0 | 0.805 ± 0.034 | 343.3 ± 8.7 | 0.915 ± 0.018 | |
| w/o Sequential Evidence | 1,000 ± 0 | 0.715 ± 0.209 | 319.8 ± 43.1 | 0.862 ± 0.108 |
All results are reported as mean ± standard deviation over four random seeds. For each budget, we use the corresponding checkpoint when available; otherwise, we report the final checkpoint together with its actual expert-data consumption.
At the lower budget, the three variants exhibit comparable performance, whereas the full method achieves consistently better results across all metrics at the intermediate budget.
Without the HJ formulation, replacing the binary support signal and HJ-style min backup with a continuous normalized RND cost and a standard Bellman backup leads to less informative intervention selection. This is reflected in the substantially lower average performance and larger across-seed variability at the 900 budget, suggesting that the collected corrections vary considerably in their usefulness for policy learning. Although its performance eventually approaches that of the full method with additional expert supervision, it requires more human interventions, indicating that additional and potentially redundant corrective data are needed to compensate for less informative selection.
In contrast, the HJ-based support-persistence formulation, combined with sequential recovery assessment, enables more selective expert interventions and more effective utilization of limited human supervision. These results demonstrate the complementary contributions of the two components to ReGIL's data efficiency. We have included these additional ablation results in the manuscript and appendix.
Q3: RND Implementation Details and Support Detection Performance
We thank the reviewer for requesting these implementation details.
The RND target and predictor use the same MLP architecture: two hidden layers of 512 units with ReLU activations, followed by a 128-dimensional linear output. The target is randomly initialized and kept fixed, while the independently initialized predictor is trained only on expert-controlled observations using feature-averaged mean squared error. Both networks take the environment’s vector observation directly; the MetaDrive input dimension is 259. We retain the environment’s native feature scaling and apply no additional running-mean/variance normalization, whitening, or normalization layers in the RND networks. The appendix provides the hidden widths, output dimension, and optimization settings; We will add the activations and input preprocessing in the revision.
We further evaluate the discrimination capability of the learned RND errors using held-out expert states and novice deviations. We collected 25,803 states from rollouts fully controlled by the expert policy. To construct the deviation dataset, we first let the expert drive and then transferred control to an early-training, low-quality novice policy. We identified deviation events using simulator-reported failures, including collisions and road departures, and retained ten states immediately before each trajectory's first event and ten consecutive states from its onset, yielding 700 deviation states across 35 scenarios. Crucially, deviation states were selected independently of RND scores, avoiding circular evaluation.
We randomly split the expert dataset into training (80%) and held-out (20%) subsets. A fresh RND predictor was trained exclusively on the training subset and frozen for evaluation, with the OOD threshold calibrated using only training expert states. The resulting discrimination performance is summarized below:
| Held-out Expert (Non-OOD) | Novice Deviations (OOD) | ROC-AUC |
|---|---|---|
| 90.31% | 91.29% | 0.963 |
These results confirm that RND provides an effective support indicator for distinguishing expert-supported states from novice deviations. We have updated the main manuscript and appendix to incorporate the above RND diagnostics.
W1 & Q1:Clarification of value target and takeover rule
We thank the reviewer for identifying this important limitation. We agree that the current sequential criterion does not guarantee intervention when the novice remains outside human support with no predicted recovery and persistence estimates have saturated near zero. The exact discounted, terminal-aware target used in our experiments is yexp=mH(s,a) for terminal transitions, and yexp=minmH(s,a),0.98Vψˉ(s′,a′) otherwise, where a′∼πN(⋅∣s′). Since this target remains zero outside human support, discounting does not resolve the saturation issue. Once persistence estimates saturate, the consecutive-decrease condition cannot be satisfied, potentially preventing takeover. A potential remedy is to introduce a persistence-independent fallback that automatically requests takeover after L consecutive out-of-support steps. Such a mechanism would provide a finite-step intervention guarantee under sustained detected OOD while allowing a bounded opportunity for autonomous recovery. We have added this limitation and potential extension to the revised manuscript as future work.
Nevertheless, the current ReGIL gate can trigger takeover before persistence saturation occurs. Specifically, the sequential criterion operates on learned persistence estimates Vψ, using a history window spanning the initial OOD crossing rather than requiring continued decreases deep inside the OOD region. Persistence estimates may begin decreasing before crossing, allowing the gate to detect deteriorating behavior at the support boundary, although this behavior is not guaranteed. The new qualitative analysis in Section 6.4 (Figure 2) illustrates this mechanism: persistent deviations (B1–B3) trigger takeover at the initial OOD crossing, while recoverable deviations (A1–A3) continue autonomously.
To assess the practical significance of this limitation, we evaluated persistent OOD detection across 50 training scenarios and 50 held-out test scenarios. We used a 60-step observation window, defining recovery as returning to human support for at least 10 consecutive steps.
These results show that 103 of 104 persistent deviations triggered takeover requests, indicating that the identified failure mode did not lead to frequent missed interventions in the evaluated scenarios, despite the absence of a theoretical guarantee.