acceptodds
Under review as a conference paper at ICLR 2027

Wait or Take Over: Forecasting OOD Persistence and Recovery for Efficient Human-in-the-Loop Imitation Learning

Abstract

Robot-gated interactive imitation learning allows an agent to request human intervention and learn from the resulting corrective demonstrations. Deciding when to request human takeover is central to using limited human effort effectively. Existing intervention mechanisms commonly rely on pointwise novelty, uncertainty, or disagreement, leaving an ambiguity: a novice outside human support may recover autonomously or continue along a persistent deviation. Intervening in the former case wastes limited human effort. We formulate robot-gated interactive imitation learning as a sequential intervention decision problem and develop Recovery-aware Gating for Interactive Imitation Learning (ReGIL). Drawing on Hamilton–Jacobi reachability analysis, we learn a support-persistence value that estimates whether the novice can remain within human support under its own policy. A short-horizon dynamics model predicts a return to support, while the persistence value assesses behavior after that return. A sequential recovery assessment combines these predictions with observed changes in persistence to determine when human intervention is warranted. This allows the novice to continue when autonomous recovery remains plausible and requests human takeover when predicted recovery is insufficient and observed persistence declines. Experiments in MetaDrive and MetaWorld demonstrate the effectiveness of the framework in learning autonomous policies from limited human intervention. The corresponding source code is available at [https://anonymous.4open.science/r/ReGIL-7864](https://anonymous.4open.science/r/ReGIL-7864).

Primary area:
applications to robotics, autonomy, planning
Keywords:
Interactive Imitation Learning, Human-in-the-Loop Learning, Robot-Gated Intervention, Recovery-Aware Gating, Hamilton– Jacobi Reachability, Human Data Efficiency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3wUGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReGIL, an intervention method for interactive imitation learning that considers whether a novice can recover after leaving expert support. It combines a learned support-persistence value, short dynamics rollouts, and a sequential intervention rule. Experiments on MetaDrive and a MetaWorld drawer-opening task show higher mean autonomous success under expert-data budgets than the compared methods.

Critical strengths
  • The distinction between leaving demonstration support and needing expert assistance is useful. A temporary deviation may resolve under the novice’s own actions. Considering both the predicted return and behavior after that return gives the intervention decision information that current-state novelty alone does not provide.

  • The budgeted results support an expert-data efficiency benefit. The methods share the behavior-cloning architecture and update schedule, and initial demonstrations are counted in the budget. Reporting both budgeted and final performance also makes the performance/data trade-off visible.

Critical weaknesses
  • I think there is a mismatch between Eq. (3) and the takeover rule. The stated value is zero outside support. If takeover does not occur at the first unsupported step, a persistent deviation would not produce the consecutive decreases required for later intervention. It is unclear whether the implemented discounted target resolves this.

  • I am not yet convinced of the HJ formulation’s benefit at a fixed expert-data budget. The no-HJ variant achieves higher success (0.856 vs. 0.830), but uses more expert data. A comparison at the same expert budget would make its contribution clearer.

Questions for authors
  • What is the exact discounted, terminal-aware value target used in the experiments? How does the implemented gate request takeover when the novice remains outside support, no return is predicted, and persistence estimates remain zero or nearly constant?

  • At matched expert-transition budgets, how does the full method compare with the no-HJ and no-sequential-evidence variants? Do the existing learning curves support an advantage for the full gate when expert supervision is controlled?

  • Please specify the RND target and predictor architectures, feature dimension, and input normalization. Are there diagnostics showing how well the resulting errors separate held-out expert states from states encountered during novice deviations? This would help assess the support indicator underlying both recovery prediction and persistence learning.

5 comments

W1 & Q1:Clarification of value target and takeover rule

We thank the reviewer for identifying this important limitation. We agree that the current sequential criterion does not guarantee intervention when the novice remains outside human support with no predicted recovery and persistence estimates have saturated near zero. The exact discounted, terminal-aware target used in our experiments is yexp=mH(s,a)y_{\mathrm{exp}}=m_{\mathcal H}(s,a) for terminal transitions, and yexp=min⁡mH(s,a),0.98Vψˉ(s′,a′)y_{\mathrm{exp}}=\min{m_{\mathcal H}(s,a),0.98V_{\bar\psi}(s',a')} otherwise, where a′∼πN(⋅∣s′)a'\sim\pi_N(\cdot|s'). Since this target remains zero outside human support, discounting does not resolve the saturation issue. Once persistence estimates saturate, the consecutive-decrease condition cannot be satisfied, potentially preventing takeover. A potential remedy is to introduce a persistence-independent fallback that automatically requests takeover after LL consecutive out-of-support steps. Such a mechanism would provide a finite-step intervention guarantee under sustained detected OOD while allowing a bounded opportunity for autonomous recovery. We have added this limitation and potential extension to the revised manuscript as future work.

Nevertheless, the current ReGIL gate can trigger takeover before persistence saturation occurs. Specifically, the sequential criterion operates on learned persistence estimates VψV_\psi, using a history window spanning the initial OOD crossing rather than requiring continued decreases deep inside the OOD region. Persistence estimates may begin decreasing before crossing, allowing the gate to detect deteriorating behavior at the support boundary, although this behavior is not guaranteed. The new qualitative analysis in Section 6.4 (Figure 2) illustrates this mechanism: persistent deviations (B1–B3) trigger takeover at the initial OOD crossing, while recoverable deviations (A1–A3) continue autonomously.

To assess the practical significance of this limitation, we evaluated persistent OOD detection across 50 training scenarios and 50 held-out test scenarios. We used a 60-step observation window, defining recovery as returning to human support for at least 10 consecutive steps.

MetricTraining (50)Test (50)
Persistent OOD events4757
Detection recall100% (47/47)98.25% (56/57)
Missed takeover rate0%1.75%
Median detection delay0 steps1 step

These results show that 103 of 104 persistent deviations triggered takeover requests, indicating that the identified failure mode did not lead to frequent missed interventions in the evaluated scenarios, despite the absence of a theoretical guarantee.

W2 & Q2: Ablation Performance at Matched Expert-Data Budgets

We additionally evaluate the ablation variants under different expert-transition budgets, following the same experimental protocol and using four random seeds.

The results are as below:

Expert-data BudgetVariantHuman DataSuccess ↑Return ↑Progress ↑
800Full ReGIL800 ± 00.564 ± 0.171291.6 ± 40.80.785 ± 0.105
w/o HJ800 ± 00.544 ± 0.272287.6 ± 66.40.766 ± 0.168
w/o Sequential Evidence800 ± 00.570 ± 0.222294.0 ± 56.20.791 ± 0.139
900Full ReGIL900 ± 00.770 ± 0.076335.4 ± 16.10.892 ± 0.039
w/o HJ900 ± 00.550 ± 0.273296.5 ± 50.20.796 ± 0.129
w/o Sequential Evidence900 ± 00.595 ± 0.189300.7 ± 39.90.809 ± 0.099
1,000 (cap)Full ReGIL945.5 ± 75.90.800 ± 0.063335.2 ± 14.80.898 ± 0.031
w/o HJ1,000 ± 00.805 ± 0.034343.3 ± 8.70.915 ± 0.018
w/o Sequential Evidence1,000 ± 00.715 ± 0.209319.8 ± 43.10.862 ± 0.108

All results are reported as mean ± standard deviation over four random seeds. For each budget, we use the corresponding checkpoint when available; otherwise, we report the final checkpoint together with its actual expert-data consumption.

At the lower budget, the three variants exhibit comparable performance, whereas the full method achieves consistently better results across all metrics at the intermediate budget.

Without the HJ formulation, replacing the binary support signal and HJ-style min backup with a continuous normalized RND cost and a standard Bellman backup leads to less informative intervention selection. This is reflected in the substantially lower average performance and larger across-seed variability at the 900 budget, suggesting that the collected corrections vary considerably in their usefulness for policy learning. Although its performance eventually approaches that of the full method with additional expert supervision, it requires more human interventions, indicating that additional and potentially redundant corrective data are needed to compensate for less informative selection.

In contrast, the HJ-based support-persistence formulation, combined with sequential recovery assessment, enables more selective expert interventions and more effective utilization of limited human supervision. These results demonstrate the complementary contributions of the two components to ReGIL's data efficiency. We have included these additional ablation results in the manuscript and appendix.

Q3: RND Implementation Details and Support Detection Performance

We thank the reviewer for requesting these implementation details.

The RND target and predictor use the same MLP architecture: two hidden layers of 512 units with ReLU activations, followed by a 128-dimensional linear output. The target is randomly initialized and kept fixed, while the independently initialized predictor is trained only on expert-controlled observations using feature-averaged mean squared error. Both networks take the environment’s vector observation directly; the MetaDrive input dimension is 259. We retain the environment’s native feature scaling and apply no additional running-mean/variance normalization, whitening, or normalization layers in the RND networks. The appendix provides the hidden widths, output dimension, and optimization settings; We will add the activations and input preprocessing in the revision.

We further evaluate the discrimination capability of the learned RND errors using held-out expert states and novice deviations. We collected 25,803 states from rollouts fully controlled by the expert policy. To construct the deviation dataset, we first let the expert drive and then transferred control to an early-training, low-quality novice policy. We identified deviation events using simulator-reported failures, including collisions and road departures, and retained ten states immediately before each trajectory's first event and ten consecutive states from its onset, yielding 700 deviation states across 35 scenarios. Crucially, deviation states were selected independently of RND scores, avoiding circular evaluation.

We randomly split the expert dataset into training (80%) and held-out (20%) subsets. A fresh RND predictor was trained exclusively on the training subset and frozen for evaluation, with the OOD threshold calibrated using only training expert states. The resulting discrimination performance is summarized below:

Held-out Expert (Non-OOD)Novice Deviations (OOD)ROC-AUC
90.31%91.29%0.963

These results confirm that RND provides an effective support indicator for distinguishing expert-supported states from novice deviations. We have updated the main manuscript and appendix to incorporate the above RND diagnostics.

Reviewer_RhpcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReGIL, which combines expert-support estimation, short-horizon dynamics prediction, and changes in support persistence to decide when to request human takeover. Experiments in MetaDrive and MetaWorld drawer opening show its effectiveness under limited expert-data budgets.

Critical strengths

An interesting innovation is its use of an HJ-style Bellman backup to extend RND pointwise support detection into an estimate of future support persistence. Combined with short-horizon dynamics prediction, this separates whether the novice can return to expert support from whether it can remain supported afterward. This is a well-motivated design for distinguishing transient from persistent deviations and avoiding unnecessary expert interventions.

Critical weaknesses
  1. The paper provides no visualization of the learned expert-support region or trajectory-level plots of RND scores, HJ persistence values, and takeover timings. There is no demonstration of which states the model considers familiar to the expert or how the HJ value distinguishes recoverable deviations from those requiring intervention. This challenges the specific role of the HJ formulation relative to a standard Bellman update.
  2. The paper contains no experiments with human participants. All interventions come from neural or scripted experts, with zero reaction delay. Counting expert-controlled transitions does not adequately capture actual human workload, including monitoring, understanding the current situation, and switching control.
Questions for authors
  1. What is the learned persistence value's empirical prediction horizon? With the policy and networks frozen, how far ahead can it anticipate an actual departure from expert support?
  2. Can you provide a few qualitative examples illustrating when ReGIL allows autonomous recovery and when it requests human takeover?
3 comments

W1 & Q2: Qualitative Analysis of Recovery and HJ-Based Intervention Decisions

We thank the reviewer for highlighting the importance of illustrating recovery-aware intervention decisions.

We provide a trajectory-level qualitative analysis in the revised manuscript (Figure 2) to demonstrate how ReGIL distinguishes autonomously recoverable deviations from persistent deviations requiring human intervention, together with the evolution of the signals underlying these decisions. Since the learned support region lies in a high-dimensional space, we visualize support membership along trajectories, accompanied by normalized RND scores, current persistence values vtv_t, predicted recovery scores ρt\rho_t, and takeover timings.

The figure contrasts two distinct outcomes of out-of-support deviations: autonomous recovery (A1–A3) and persistent deviation (B1–B3), each illustrated by three trajectories aligned at their initial OOD crossing (t=0)(t=0). Both exhibit elevated RND scores upon leaving support, showing that pointwise novelty alone cannot distinguish whether intervention is warranted. However, their recovery-related signals exhibit markedly different behaviors:

  • Autonomous recovery (A1–A3): Despite temporarily leaving expert support, the persistence estimates remain non-negligible, while predicted recovery scores ρt\rho_t stay above the threshold η\eta. ReGIL therefore permits continued autonomous execution, allowing the novice to return to supported behavior without intervention.
  • Persistent deviation (B1–B3): Following OOD crossing, persistence estimates rapidly decline toward zero, with predicted recovery scores falling below η\eta. Together with sustained decreases in observed persistence, these signals trigger takeover requests at the marked steps.

These contrasting behaviors illustrate how HJ-style support persistence, combined with dynamics-based recovery prediction and sequential evidence, enables ReGIL to distinguish transient deviations from persistent ones, rather than triggering interventions solely based on instantaneous signals. We have added a new qualitative analysis subsection (Section 6.4) and Figure 2 to the revised manuscript.

Q1: Empirical Prediction Horizon of the Support-Persistence Value

We thank the reviewer for the question regarding the empirical prediction horizon of the learned persistence value. To quantify its predictive capability, we conduct a frozen-network evaluation across all 50 MetaDrive scenarios, keeping the novice policy, support estimator, and persistence network fixed. We analyze 18,579 currently in-support states. For each state, we record the number of subsequent control steps until its first observed departure from expert support. We separately evaluate states whose first departure occurs 6–10, 11–20, 21–50, or 51–100 steps later, comparing each group against states that remain continuously supported for the next 100 steps. These lead-time groups are non-overlapping. Lower persistence estimates indicate greater departure risk.

Steps before departureValue AUC95% CI (scene bootstrap)
6–100.887[0.846, 0.924]
11–200.811[0.766, 0.859]
21–500.676[0.614, 0.734]
51–1000.591[0.539, 0.642]

We can see that the persistence value retains useful predictive discrimination 11–20 control steps before the first departure, with an AUC of 0.811. Discrimination gradually weakens at longer lead times, reaching 0.676 at 21–50 steps and 0.591 at 51–100 steps. At the configured control interval of 0.1 seconds, 11–20 steps correspond to approximately 1–2 seconds of simulated driving. These results show that the learned value contains predictive information about future support departure while the novice is still within support, with stronger discrimination at shorter lead times.

W2: Real Human-in-the-Loop Evaluation and Practical Intervention Costs

We thank the reviewer for raising this important concern. To examine ReGIL under real human participation, we conducted additional experiments with five human participants, all holding valid driving licenses. Each participant received 30 minutes of familiarization with MetaDrive and completed one run under each of two settings.

  1. Real Human-in-the-Loop Experiments.
  • Human Reaction + Human Demonstration (Table A): Participants respond to takeover requests and provide corrective demonstrations through keyboard control. During reaction delays, the novice remains in control while the environment continues evolving.
  • Human Reaction + Neural Expert Demonstration (Table B): To reduce the effects of individual differences and imprecise keyboard control, participants respond to takeover requests while a neural expert provides the corrective demonstrations. This setting allows us to examine the impact of real human reaction delays with more consistent demonstration quality.

Table A. Human reaction + human demonstrations.

ParticipantHuman Data ↓Success ↑Progress ↑Return ↑SwitchesReaction (s)
P189686%93.23%309.621630.667
P298174%88.86%317.531280.636
P3157942%69.84%194.541880.800
P495366%82.99%236.391570.585
P5140064%79.45%272.662170.848
Mean ± Std1162 ± 30766.4 ± 16.1%82.87 ± 9.01%266.15 ± 51.44170.6 ± 33.60.707 ± 0.112

Table B. Human reaction + neural expert demonstrations.

ParticipantHuman Data ↓Success ↑Progress ↑Return ↑SwitchesReaction (s)
P169392%93.47%322.861630.527
P288870%87.03%314.171690.804
P363282%90.84%327.911810.635
P463194%97.70%347.711620.708
P597684%92.15%291.952050.659
Mean ± Std764 ± 15884.4 ± 9.5%92.24 ± 3.89%320.92 ± 20.34176.0 ± 17.90.667 ± 0.101
  1. Results and Analysis. In the fully human-controlled setting, performance is lower and exhibits substantial variability across participants. Several factors may contribute to this observation: (i) human reaction delays before takeover, (ii) limited precision of keyboard-based control in MetaDrive, and (iii) differences in participants' familiarity with the environment and control proficiency. These factors can affect the quality and consistency of corrective demonstrations, making it difficult to attribute the observed performance differences solely to the intervention-gating mechanism.

In contrast, when human reaction delays are retained but corrective demonstrations are provided by a neural expert, ReGIL achieves 84.4% success and 92.24% route progress, with an average reaction delay of 0.667 seconds. These results suggest that human response delays alone do not necessarily cause substantial performance degradation under the evaluated conditions. The difference between the two settings is consistent with demonstration quality being an important factor, although the pilot study does not establish the individual causal contributions of these factors.

We also additionally report intervention-related measurements: an average takeover duration of 1.108 s, a total human control time of 96.08 s per participant, and six failures occurring during human response delays across five runs. These measurements provide further insights into the practical intervention process under real human participation.

  1. Scope and Future Work. Our primary objective is to improve human demonstration-data efficiency, rather than to minimize overall human workload (Clarified in the revised Introduction). Assuming access to a competent expert capable of providing reliable demonstrations, we study when to request corrective demonstrations, rather than how to improve their quality. We therefore use standardized neural or scripted experts in the main comparisons to isolate the gating mechanism from variations in human control.

Our human study is limited in scale, and variations in operator familiarity and keyboard-control precision may affect demonstration quality and performance. Future work will involve larger-scale human studies to systematically evaluate the impact of demonstration quality and overall human workload, including monitoring, situation awareness, and control-switching costs. We revised the manuscript and appendix accordingly: (i) added real human-in-the-loop experiments, (ii) clarified in the Introduction that our primary objective is to improve human demonstration-data efficiency rather than minimize total human workload, and (iii) expanded the Conclusion to discuss broader human workload and demonstration quality as directions for future work.

Reviewer_fA6FOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A human-in-the-loop imitation learning method of similar style as DAgger, with a different method of deciding when to solicit human input, and when to stop taking it; resulting in good performance, and less human input. In contrast to previous methods, this method estimates the probability of returning to known states, and uses that to only ask for expert input if the probability of recovery is low, and the probability is dropping in recent fixed window.

Critical strengths
  • Very clear writing. Was a joy to read after many papers in different conferences that perhaps utilized AI-slop writing. The story is clearly motivated. Related work is described very well. (Except some small weaknesses - see next section.)
  • Simple but solid solution. Estimating (via a network) the probability, results in (usually) better human-input-sample-complexity, empirically.
  • Ablation studies generally support the use of all the components of the solution proposed in this paper.
Critical weaknesses
  • I could not figure out where exactly is Hamilton-Jacobi estimation being used. There is a section about it, yes. And I could follow the formulation and method sections, but they did not say anywhere that this is now Hamilton-Jacobi or inspired by it.
  • Results
    • Need more clarity about what is really meant by 'under a budget', and 'final'. Because for example, is there a hard limit on budget to be under 1000 for table 1? If so, that means the first ReGIL result must be skewed and there is hard limit on 1000, and since this is reporting the standard deviation the deviation is skewed in one direction. The text also does not tell us if it standard deviation, or something else. (Table 3 does, but these should too)
    • Table 1 does not tell us over how many runs are the standard deviations being reported
    • Table 1 perhaps mentions completion percentage while table 2 has completion times hence smaller is better. This needs clarity.
  • Line 13, in Algorithm 1. The entire policy has to be trained every time a human sample is obtained.
    • (perhaps this could be changed to just a batch / single-sample update if the user needs?)
Questions for authors
  • Need some clarity on results. What is meant by final? Since both columns say 'Completion' while mean different things, maybe clarify this. Why do we not see return and completion for final results?
  • Eq(3). Does it not have a horizon? or is the horizon episode termination?
4 comments

W1: Clarifying the Connection to Hamilton–Jacobi Reachability

We thank the reviewer for the opportunity to clarify the connection to Hamilton–Jacobi (HJ) reachability.

Classical HJ reachability characterizes whether a dynamical system can reach or avoid a prescribed region over time. ReGIL builds on this temporal, set-based perspective, but adopts a complementary formulation: we estimate the probability that it will remain within this region under its own policy.

Specifically, the HJ-inspired component is the support-persistence estimator introduced in Section 5.1. Eq. (3) defines the probability of remaining within expert support over future novice-controlled trajectories; Eq. (4) derives its min-based recursive formulation through a one-step decomposition; and Eq. (5) provides the corresponding neural learning objective using replay transitions.

To make the HJ connection explicit, we treat the complement of human support as the violation set in reachability analysis. The corresponding value ΦHπN=1−VHπN\Phi_{\mathcal H}^{\pi_N}=1-V_{\mathcal H}^{\pi_N} estimates the probability of eventually reaching this set and satisfies a max-based Bellman recursion, consistent with the stochastic HJ reachability formulation in RESPO (Ganai et al., 2023). ReGIL instead learns its complement, the support-persistence value, which yields the min-based recursion in Eq. (4). Thus, our method adopts the HJ-inspired reachability formulation through a data-driven Bellman approximation.

We have also revised the paragraph following Eq. (4) in Section 5.1 to explicitly discuss the Connection to HJ Reachability and clarifying how the stochastic reachability formulation leads to our complementary support-persistence value.

Feference:

Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36:69764–69797, 2023.

Q1 & W2: Clarifying Evaluation Protocols, Budget Constraints, and Reported Metrics

We thank the reviewer for these helpful questions. We clarify the experimental protocol and provide additional results below. Table A. MetaDrive (Human-data budget: 1,000)

SeedUnder BudgetFinal
0839839
11,0001,147
2943943
31,0001,098
Mean ± SD945.5 ± 75.91,006.8 ± 141.7

Table B. MetaWorld (Human-data budget: 400)

SeedUnder BudgetFinal
0400474
1400400
2375375
3400454
Mean ± SD393.8 ± 12.5425.8 ± 46.1

The expert-data budget specifies which checkpoint is evaluated; all training runs continue to the full interaction horizon. For each seed, we evaluate the checkpoint at the specified expert-transition budget if that budget is reached. Otherwise, we evaluate the final checkpoint and report its actual expert-data usage. Expert-data counts include the initialization demonstrations.

Consequently, the expert-data counts at the selected checkpoints are bounded by the budget and may be asymmetrically distributed across seeds. We report mean ± sample standard deviation to summarize the observed variation across seeds, rather than as a confidence interval or a range of possible values. The per-seed counts below make this distribution explicit. “Final” denotes the policy at the end of training: after 8,000 interaction steps including initialization in MetaDrive, and after 1,500 online steps following 50 initialization expert transitions in MetaWorld.

Table C. MetaDrive Final Results

MethodReturn ↑Progress ↑
EnsembleDAgger350.06 ± 8.00.9344 ± 0.0197
ThriftyDAgger346.37 ± 13.10.9215 ± 0.0318
RND-DAgger359.20 ± 2.80.9564 ± 0.0063
AIM350.51 ± 9.00.9341 ± 0.0207
ReGIL342.00 ± 17.30.9143 ± 0.0391

Table D. MetaWorld Final Results

MethodCompletion Time ↓
EnsembleDAgger0.1759 ± 0.0096
ThriftyDAgger0.1751 ± 0.0129
RND-DAgger0.1652 ± 0.0032
AIM0.1801 ± 0.0114
ReGIL0.1749 ± 0.0148

We rename the two metrics to distinguish their meanings and units. Progress in MetaDrive measures the fraction of the route traversed (higher is better). Completion Time in MetaWorld measures the number of environment steps required to satisfy the task’s success criterion, divided by the 500-step episode horizon; unsuccessful episodes receive a value of 1 (lower is better). We also provide the previously omitted final performance metrics in Tables C and D. We have revised the manuscript and appendix accordingly.

W3: Clarifying the Scheduled Mini-Batch Policy Updates

We thank the reviewer for this suggestion. Our framework allows data aggregation and policy optimization to follow different schedules. In our experiments, expert transitions are added to the dataset as they are collected, while policy optimization is triggered after every 25 newly collected expert transitions. Each update consists of 125 mini-batch gradient steps with a batch size of 1,024, continuing from the current policy parameters. All compared methods share this policy-update schedule. We have revised Algorithm 1 to make this flexibility and the scheduled mini-batch updates explicit.

Q2: The Horizon of the Support-Persistence Value

We thank the reviewer for this question. Eq. (3) is defined over an infinite horizon, evaluating whether the novice remains within the human-supported region for all future steps ∀k≥0\forall k \geq 0. In episodic environments, this assessment naturally ends upon episode termination.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.