acceptodds
Under review as a conference paper at ICLR 2027

TollGate: Safe Reset-Free Reinforcement Learning via Feasibility and Budget Gating

Abstract

Reinforcement learning typically relies on resets between attempts, which are readily available in simulation but costly in the real world. Prior work pairs a task-directed forward policy with a reset policy that learns to return the agent to the initial state, using estimated recoverability to determine handoffs. However, both task execution and recovery can incur safety costs, and within each attempt they draw on a single, shared safety budget. Recoverability therefore does not imply affordability: a state can remain recoverable while the remaining budget no longer covers the cost of recovery. We introduce TollGate, a budget-aware handoff framework that combines constrained learning of both policies with a dual-gate switching mechanism. An outcome-supervised feasibility gate predicts whether recovery will succeed, and a budget gate checks whether the predicted recovery cost would exhaust the remaining budget; either gate can trigger a handoff to the reset policy. On a grid world and on continuous-control tasks, TollGate trains both policies without demonstrations or a separately designed reset reward. TollGate learns the tasks, replaces the vast majority of manual resets, and drives safety costs below their thresholds. Together, these capabilities bring reinforcement learning a step closer to the real world, where neither manual resets nor unsafe exploration comes cheap.

Primary area:
reinforcement learning
Keywords:
reset-free reinforcement learning, safe reinforcement learning, constrained MDP, policy switching, safety budget

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dZCEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Reset-free RL removes manual resets by alternating a task policy with a learned reset policy, but existing handoff rules only ask whether recovery is feasible, not whether the agent can afford it: task execution and recovery draw on one shared per-attempt safety budget, so recoverability does not imply affordability. TollGate makes the handoff budget-aware with two gates: an outcome-supervised feasibility monitor, and a budget gate that triggers when spent cost plus a predicted recovery reserve (ensemble cost-critic mean plus an uncertainty margin κσ) crosses a soft gate line. Both policies are trained with SAC-Lagrangian under a shared cost threshold d, and a hard budget C_hard terminates any interval that exceeds it. On a grid world and Safety-Gymnasium Point/Car, TollGate learns the tasks with 99.5–100% autonomous resets in late training and per-phase costs below threshold.

Critical strengths
  • The problem is well-motivated and realistic: recoverability does not imply affordability, and prior reset-free methods track no cumulative cost.
  • The outcome-supervised feasibility monitor is a sound design: it is trained directly on observed recovery outcomes, involves no bootstrapped targets, and its gradient does not flow into either policy's learning objective.
Critical weaknesses
  • The headline comparisons are against an LNT-style (2018) reimplementation plus ablations of TollGate's own gates. Other reset-free methods like VaPRL, MEDAL, RISC, IBC, SRRL, Back to Base are cited but never evaluated, and EARL-scale or manipulation benchmarks are absent.
  • The theory does not cover the implementation. Proposition 3 requires a reserve with simultaneous coverage of undiscounted recovery costs, but the implemented reserve is a discounted expected critic plus ensemble spread, and the authors concede the empirical results establish neither assumption. In practice, the hard budget, not the gates, is what actually enforces safety.
  • The sensitivity analysis is very limited: sweeps cover only the gate line C_soft on the grid, while more important hyper-parameters such as θ_F, κ, and C_hard, as well as additional tasks, are left unexamined.
Questions for authors

Missing references.

  • Reset-Free Reinforcement Learning via Multi-Task Learning (ICRA 2021)
  • Reset-Free Lifelong Learning with Skill-Space Planning (ICLR 2021)
  • Reset-free Reinforcement Learning with World Models (TMLR 2025)
3 comments
AuthorsOpenReview ↗

Response to dZCE(1/2)

Thank you for the detailed review.

W1. Baselines and benchmarks

TollGate addresses when task execution should give way to recovery under a shared safety budget. We have expanded the grid comparisons to cover additional handoff mechanisms and budget-aware learning (App. D.1, D.4, D.5):

  • Learned recovery without predictive handoffs (NoGate). TollGate solves 5/5 vs 4/5 layouts and reduces manual resets (37.4 vs 46.0, lower on 4/5) and hard-cap rates (4.1% vs 8.9%, lower on 5/5), at total cost 950 vs 903. Both arms learn recovery; the difference is predictive handoff timing.
  • RISC-style switching. We use forward-to-reset switching on our learners, with parameters from RISC's gridworld setting. Switching away once confident ends about half of the forward phases before the goal. Task success in the final 50 intervals is 0.45 vs TollGate's 0.99 (lower on 5/5); whole-run hard-cap rates are 6.2% vs 4.1% (higher on 5/5). Manual resets are lower (34.8 vs 37.4, 4/5), as is total cost (726 vs 950, 5/5), alongside lower task completion.
  • Saute-style learning. We give both constrained policies the remaining interval budget and test whether explicit gates still help. Adding the gates lowers the hard-cap rate from 10.1% to 3.5% and manual resets from 49.0 to 34.4 (both on 5/5 layouts), while solved layouts rise from 4/5 to 5/5. App. D.5 gives implementation details.
  • LNT-style thresholds. Of 8 tested values of qmin⁡q_{\min}, none matches both the task success and hard-cap rate of feasibility gating.

Other methods.

  • VaPRL, MEDAL and IBC. VaPRL uses a subgoal curriculum, MEDAL learns recovery through state-distribution matching, and IBC builds an initial-state and goal curriculum. VaPRL and MEDAL use demonstrations, while IBC does not. Comparing these full systems would require aligning their data and curriculum choices with cost constraints on both phases, adding differences beyond the handoff decision studied here.
  • SRRL. SRRL combines Saute-based learning with an LNT-style value-threshold handoff rule and evaluates safety through fatal constraints, while TollGate accounts for accumulated cost across task execution and recovery. A full-system comparison would require aligning the cost and termination protocols. Our Saute-style and LNT-style comparisons test the two relevant design choices under shared learning conditions.
  • Back to Base. Back to Base uses a dynamics model and known unsafe set for reachability-based action filtering, while TollGate learns recovery signals from interaction and uses them to time handoffs. Applying that system to our setting would require supplying an environment dynamics model and incorporating the shared cumulative cost into its reachability formulation.

These differences motivate our comparison of representative handoff mechanisms using the same constrained learners, phase horizons and cost limits. This design isolates the contribution of deciding when to recover under a shared safety budget.

Benchmarks. We chose navigation tasks where both phases encounter hazards and recovery to the initial state is measurable, allowing us to study the shared safety budget. EARL provides reset-free tasks but does not define this safety cost. Extending our evaluation to manipulation tasks is an important direction for future work.

W2. Theory and the role of the hard budget

In our setting, reaching the hard budget terminates the current attempt and requires a manual reset. This stopping mechanism relies on external intervention and does not itself teach the policies low-cost behaviour. Constrained learning trains the task and reset policies under cost constraints. The gates use recovery capability, cost already spent and predicted recovery cost to trigger early handoffs and reduce forced manual resets.

Prop. 3 gives sufficient conditions for avoiding budget exhaustion through a recovery reserve and a handoff margin. The reserve must cover undiscounted recovery costs across possible handoff times, and the gap between the gate line and the hard budget must accommodate growth in spent cost and the reserve between checks. In practice, we construct the reserve from a discounted cost critic and ensemble disagreement. We use error diagnostics to assess the consequences of prediction errors and comparisons under the same budget to evaluate the gates' practical contribution (App. D.2–D.4).

On Car, recomputing the budget-gate decisions at recorded trigger states shows that 37% would not have fired at that step without the κσ\kappa\sigma term. After 400k steps, among budget-gate handoffs where realised recovery cost exceeds the reserve, 98.3–99.8% avoid budget exhaustion. Under the same learners, hard budget and number of environment steps, adding the budget gate to feasibility gating reduces whole-run budget exhaustion during recovery from 263 to 121 and manual resets from 496 to 352 (IQMs, with both reductions in all five seeds).

AuthorsOpenReview ↗

Response to dZCE(2/2)

W3. Sensitivity

We added sweeps of the parameters the reviewer lists (grid, 5 layouts each, App. D.4):

  • θF∈{0.1,0.2,0.3,0.5}\theta_F \in \lbrace 0.1, 0.2, 0.3, 0.5 \rbrace: all solve 5/5 layouts.
  • κ∈{0,1,2,8,32}\kappa \in \lbrace 0, 1, 2, 8, 32 \rbrace: 4–5/5 solved, 36–40 manual resets, hard-cap rate 3.6–4.3%. On Car, 37% of budget-gate triggers depend on the κσ\kappa\sigma term at the trigger step (34.8–39.0% per seed; 23% in the final 200k steps).
  • ChardC_{\rm hard} with Csoft=7C_{\rm soft} = 7: hard budgets of 9, 10, 12 and 15 all solve 5/5 layouts. The full margin sweep is in App. D.4.
  • The gate line Csoft∈{7,8,9}C_{\rm soft} \in \lbrace 7, 8, 9 \rbrace solves 5/5 (Table 6).

Missing references

We thank the reviewer and will add all three to Related Work. MTRF (Gupta et al., ICRA 2021) lets learned tasks serve as resets for one another. LiSP (Lu et al., ICLR 2021) plans in a learned skill space to avoid sink states. MoReFree (Yang et al., TMLR 2025) adapts model-based exploration and imagination training to the reset-free setting. None of the three accounts for a safety cost shared by task execution and recovery, which is the problem TollGate addresses.

Thank you for providing more detailed explanation. As given by my original review, I still think more reset-free baselines and benchmarks are needed, so I would remain my original score.

Reviewer_k92NOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important challenge in real-world reinforcement learning, where resetting the environment between attempts can require manual human intervention. Its central focus is enabling an agent to perform a task and return to its starting region in a hazardous environment while controlling the combined safety cost and reducing human intervention. The proposed TollGate framework uses SAC-Lagrangian (SAC-LR) to train a forward policy to accomplish the task and a recovery policy to return to the starting region. Two gates determine when to switch to recovery policy, with either gate sufficient to trigger a handoff. In continuous tasks, the feasibility monitor is trained using binary cross-entropy on observed recovery outcomes to predict whether recovery reaches a neighborhood around the starting point. In the discrete grid env, it instead uses a per-cell table updated toward binary recovery outcomes through an exponential moving average. The feasibility gate triggers recovery when the monitor’s output falls below a threshold. The budget gate triggers when the safety cost spent so far plus a predicted recovery reserve exceeds another threshold. This reserve combines the cost-critic ensemble’s mean prediction of expected discounted recovery cost with a scaled measure of ensemble disagreement as an uncertainty margin.

Contributions

  • The study considers task execution and recovery under a shared safety budget, recognizing that returning to the start in a hazardous environment may be possible but could exceed the remaining safety budget.
  • Proposes a budget-aware handoff rule that combines outcome-supervised recovery-feasibility estimates with accumulated safety cost and predicted return cost to determine when to initiate recovery.

TollGate is evaluated on a discrete grid env and continuous-control Point and Car env adapted from Safety-Gymnasium. Comparisons include episodic SAC-LR on the grid and an LNT-style switching rule on the grid and Car, alongside feasibility-only and budget-only ablations. Performance is assessed through task success or return, manual resets, and safety costs.

Critical strengths

TollGate’s key strength is accounting for both predicted recovery feasibility and estimated recovery cost relative to the remaining safety budget. This provides a meaningful basis for initiating recovery before further task execution makes returning unaffordable. However, this strength does not outweigh the concerns about prediction reliability and whether the comparisons sufficiently establish the added benefit over learned return-to-start behavior.

Critical weaknesses
  • TollGate’s handoff decisions depend on learned feasibility and recovery-cost estimates, but their reliability is insufficiently evaluated. Underestimated feasibility can trigger premature recovery, while overestimated feasibility or underestimated cost can delay necessary handoffs. Although the paper acknowledges prediction and calibration limitations, it does not quantify their effects or examine sensitivity to the uncertainty coefficient. This leaves the reliability of the switching mechanism under prediction errors unclear.
  • The experiments provide limited evidence of broader effectiveness and practical value. Evaluation covers only goal-reaching navigation tasks with comparisons against episodic SAC-LR, LNT-style switching, and gate ablations. A recovery enabled baseline without predictive handoffs is missing, making it difficult to separate the benefit of learned recovery from the proposed switching rule. Comparisons with established safe-recovery methods under matched interaction budgets and safety constraints are also limited. Substantial manual resets remain necessary during training, so encouraging late-training results alone do not establish a manageable human intervention burden during early learning.
Questions for authors
  • Fig.3 shows a linearly increasing reset count for episodic SAC-LR. Could the authors clarify its termination and reset conditions, particularly whether task completion or timeout triggers an external reset? The supplied code omits the grid implementation needed to verify this.
  • Was the SAC-LR baseline in Fig.3 trained only to reach the task goal and then reset externally, or was it also trained to return to s0s_0? Since TollGate itself uses SAC-LR for policy learning, giving only TollGate learned recovery would prevent this comparison from isolating the benefit of predictive handoffs. A recovery-enabled baseline that returns after task completion or forward timeout, without predictive handoffs, would provide a more informative comparison under matched interaction budgets and safety constraints.
6 comments
AuthorsOpenReview ↗

Response to k92N(1/2)

Thank you for the detailed review.

W1. Reliability of the learned estimates

We measured prediction accuracy and the consequences of prediction errors in the submitted runs (App. D.2–D.4).

Accuracy against ground truth. On the grid, deterministic transitions and a frozen reset policy that depends only on the cell allow exact recovery values to be computed by dynamic programming. At the end of training, across 5 layouts and 180 cells, the cost critic has MAE 0.08 and correlation 0.98 against exact expected discounted recovery cost. The monitor's accept/reject decision at θF\theta_F agrees in 180/180 cells with whether the frozen policy returns within the reset horizon with probability at least θF\theta_F.

On Car, at handoffs triggered only by the budget gate in the final 200k steps, the cost critic predicts an average discounted recovery cost of about 13. Its MAE against realised discounted recovery costs is 0.77–0.94 per seed.

Consequences of underestimated recovery cost. The gap between the gate line and the hard budget (Chard−Csoft=8C_{\rm hard} - C_{\rm soft} = 8 on Car) leaves room for growth between checks and for recovery costs above the reserve. After 400k steps, among budget-gate handoffs whose realised recovery cost exceeded the reserve, 98.3–99.8% did not reach the hard budget (five seeds pooled in each 200k window). Across all 18,039 handoffs after 400k, 13 (0.07%) reached it during recovery.

Premature handoffs and recovery feedback. On Car, the per-seed median recovery cost after feasibility-gate handoffs is 0 in every 200k window. Their failure rate (timeout or hard budget, IQM across seeds) falls from 11.7% in 0–200k to 3.2% in 200–400k and 0 thereafter. On both Car and Point, successful recoveries supply positive monitor labels because Gstop⊆GFG_{\rm stop} \subseteq G_F (Prop. 1). From the first to the last 200k steps:

  • Forward decisions with QF<θFQ_F < \theta_F fall from 3.5% to 0.03% on Car and from 3.1% to 0.02% on Point (lower in 5/5 seeds on both).
  • Interaction spent on handoffs made inside the success region falls from 7.2% to 0.01% on Car and from 2.4% to 0.06% on Point.

Fig. 6 illustrates this feedback on Point: the monitor's accepted region grows along with the distances from which recoveries actually succeed.

Sensitivity to κ\kappa. On Car, 37% of recorded budget-gate triggers (34.8–39.0% per seed) would not have fired at that step without the κσ\kappa\sigma term. This share falls to 23% in the final 200k steps as ensemble disagreement narrows. On the grid, training with κ∈{0,1,2,8,32}\kappa \in \lbrace 0, 1, 2, 8, 32 \rbrace gives 4–5/5 solved layouts, 36–40 manual resets and a hard-cap rate of 3.6–4.3%.

W2. Benefit over learned recovery and training autonomy

To distinguish the contribution of handoff timing from that of learning to recover, our switching comparisons hold the constrained learners, phase horizons and cost limits fixed. We compare representative handoff rules within this common safety setting.

Recovery-enabled baseline without predictive handoffs (also Q2). We added NoGate as suggested. Its reset policy learns to return to s0s_0, starting after task completion or forward timeout. Reaching the hard budget requires a manual reset. On the grid, both arms use the same 5 layouts, with 300 intervals per run:

MetricTollGateNoGate
Layouts solved5/54/5
Manual resets (whole run)37.446.0
Intervals reaching ChardC_{\rm hard}4.1%8.9%
Environment steps11,35814,511

On the grid, predictive handoffs more than halve the fraction of intervals reaching the hard budget and reduce manual resets, even when both methods learn recovery. TollGate also uses fewer environment steps on every layout. Grid comparisons match the number of intervals; continuous handoff-rule comparisons use 1M environment steps per arm.

Why recovery-cost information helps. On Car, at gate handoffs with predicted recovery cost in [12,16)[12, 16), recovery succeeds in 98.8% of cases with TollGate and 22.8% with the feasibility gate alone. TollGate hands off at a median spent-cost-plus-reserve sum of about 17, leaving about 7 below the hard budget of 24. With feasibility gating alone, that sum is already about 22. This comparison shows why recovery capability must be considered together with the remaining budget.

AuthorsOpenReview ↗

Response to k92N(2/2)

Further comparisons under the same cost limits (grid; App. D.5).

Task outcomes use the final 50 intervals; other metrics cover the whole run. Tables report means across five layouts, except for solved-layout counts.

RISC-style switching. We use our learners, with forward-to-reset switching and parameters from RISC's gridworld setting. The rule ends 120 of 250 forward phases before the goal in the final 50 intervals.

MetricTollGateRISC-style
Task success0.990.45
Hard-cap rate4.1%6.2%
Manual resets37.434.8
Total environment cost950726

Saute-style learning. Both constrained policies observe the remaining interval budget. Adding the gates improves task completion and reduces reliance on manual resets:

MetricWithout gatesWith TollGate's gates
Layouts solved4/55/5
Hard-cap rate10.1%3.5%
Manual resets49.034.4

Recovery RL addresses a different recovery objective: reaching safety rather than returning to s0s_0 for another task attempt (Section 2).

Intervention burden during early learning (W2). Whole-run totals are 37 manual resets on the grid vs 300 for episodic training, and 352 on Car vs 8,242 for an episodic reference that resets after every attempt over the same 1M steps. On Car, TollGate's reset count falls from 301 in the first 200k steps to 0 in the final 200k (IQMs across seeds). We will make the distinction between whole-run and late-training results explicit in the main-text reporting.

Q1. The episodic SAC-Lagrangian baseline in Fig. 3 is a single policy trained on the forward task. Each interval ends with an external reset when the agent reaches the goal or after Tf=50T_f = 50 steps. There is therefore exactly one manual reset per interval, and the count grows linearly. It has no per-interval hard budget, so it is never reset because of cost. The grid implementation has been added to the anonymized supplementary material.

Q2. The episodic baseline was trained only to reach the task goal and was then reset externally; it never learns to return to s0s_0. The NoGate comparison in W2 supplies the requested learned-recovery baseline under the same learners, hard budget and number of intervals as TollGate.

I thank the authors for the additional experiments and clarifications. However, my concerns about TollGate’s effectiveness and methodological novelty remain, given its reliance on SAC-LR. The new comparison reports an average of 37.4 manual resets for TollGate versus 46.0 for NoGate. Although this shows a reduction, TollGate does not guarantee reset-free operation and still requires frequent human intervention limiting its practicality for unattended deployment. The baseline comparisons are also limited in number and problematic in their settings as the baselines were reset externally without first being allowed to attempt to return to s0s_0. I therefore maintain my original score.

Reviewer_DmapOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

The paper studies reset-free reinforcement learning under safety constraints. A forward policy attempts the task and a learned reset policy returns the agent to the initial state, so that training can continue without manual resets. Unlike prior reset-free work, both phases incur safety costs that accumulate within an interval (a forward attempt plus the subsequent recovery) and draw on one shared per-interval budget ChardC_{\text{hard}}; exceeding it forces a manual reset. The key observation is that recoverability does not imply affordability: a state can be recoverable with certainty, yet the remaining budget may not cover the cost of the recovery. Existing handoff rules (e.g., Leave no Trace, RISC) threshold a capability estimate and do not account for cost.

The proposed method, TollGate, trains both policies with SAC-Lagrangian under per-phase expected-cost constraints. It hands control from the forward policy to the reset policy when either of two gates fires:

  • Feasibility gate. A monitor QFQ_F predicts whether recovery will succeed. It is trained with binary cross-entropy on labels computed from observed recovery outcomes, with no bootstrapping and no influence on either policy's objective.
  • Budget gate. It fires when the cost spent so far plus a predicted recovery reserve exceeds a gate line Csoft≤ChardC_{\text{soft}} \le C_{\text{hard}}: At+V^c+κσ>Csoft,A_t + \hat V_c + \kappa\sigma > C_{\text{soft}}, where AtA_t is the cost spent so far in the interval, and V^c\hat V_c and σ\sigma are the mean and standard deviation of the reset cost critic's ensemble.

The paper gives three propositions:

  • a condition under which successful recoveries yield positive monitor labels;
  • a geometric bound on how many handoff opportunities are needed to obtain positive feedback;
  • a conditional bound under which the budget gate prevents budget exhaustion, assuming the reserve upper-bounds the true recovery cost with high probability.

Experiments use a 6×6 grid world (5 layouts) and two fixed-layout continuous navigation tasks adapted from Safety-Gymnasium (Point and Car, 5 seeds each). TollGate learns the tasks without demonstrations or a hand-designed reset reward. On the grid it uses 37 manual resets vs. 300 for an episodic learner. In the final 200k steps, autonomous resets account for 99.5–100% (Point) and 99.9–100% (Car) of resets, and the mean per-phase cost of every seed is below its threshold. Ablations compare TollGate with feasibility-only, budget-only, and LNT-style handoff rules.

Critical strengths

Pros

  1. The problem formulation is clear and well motivated, and the method is simple.
    • The observation that recoverability does not imply affordability when task execution and recovery share one per-attempt safety budget has not been addressed by prior reset-free methods.
    • The paper makes it concrete with a two-state counterexample (Section 4.1) and isolates it directly in a constructed hazard-band grid (Fig. 2): every state passes the feasibility gate, yet the budget gate separates states by their measured recovery costs.
    • The two gates only decide when control is handed over and require no demonstrations or separately designed reset reward.
Critical weaknesses

Cons

  1. The experimental scale is very limited relative to the motivation.

    • The method is motivated by the cost of manual resets on physical systems, but is evaluated only on a 6×6 grid (5 layouts, one run each) and two fixed-layout 2D navigation tasks (Point and Car, 5 seeds each).
    • There are no manipulation tasks, no tasks with higher-dimensional state or more complex recovery dynamics, no varied layouts in the continuous tasks, and no real-robot experiments.
    • In these navigation tasks the initial state is reachable by simply driving back, and the label set GF\mathcal{G}_F is easy to define. It is unclear whether the outcome-supervised monitor and the cost-critic reserve would remain reliable where recovery is harder. How would GF\mathcal{G}_F and the monitor be defined when returning to the initial state involves more than position, e.g., manipulation with objects?
  2. The baselines are weak.

    • The only external baseline is an LNT-style reset-value threshold, which almost never lets the forward policy act on Car (118k handoffs, forward return 0.02). This makes it an easy comparison.
    • More recent reset-free methods are discussed in the related work but not compared: RISC (Patil et al., 2024), example-based resets (Kim et al., 2022), MEDAL and IBC (Sharma et al., 2022a; Kim et al., 2023), and SRRL (Gottwald et al., 2025).
    • There is no comparison with budget-aware constrained RL that puts the remaining budget into the state, such as Saute RL (Sootla et al., 2022), which could address the same affordability problem without a separate gate. How does TollGate compare with RISC and a Saute-style approach in the same environments?
    • The Car "budget gate only" arm is a post-hoc reuse of runs with configuration differences, as the paper itself notes.
  3. The safety benefits are mixed, and some claims hold only late in training.

    • On the grid, TollGate accumulates more total environment cost than the episodic learner (950 vs. 556).
    • On Car, adding the budget gate to the feasibility gate increases whole-run total cost (19.9k vs. 15.4k) and reset-phase cost (1.82 vs. 0.59), and lowers forward-phase return (2.50 vs. 2.66), in all five seeds.
    • In the constructed-grid diagnostic, the budget gate lowers task success during training (0.82–0.94 vs. 1.00, Table 5).
    • "Drives safety costs below their thresholds" applies only to final-window means. During training, Point's reset-phase cost peaks at about 13 against a threshold d=2d = 2 (Fig. 4c).
    • "Replaces the vast majority of manual resets" is also a late-training statement. Over the whole run, Point still requires 1430–2085 manual resets (72–78% autonomous).
    • The main text should report whole-run totals alongside final-window statistics (total cost, manual resets, and the fraction of intervals reaching the hard budget) so that the trade-offs over training are visible.
  4. The theoretical results provide limited support for the implemented method.

    • Prop. 3, the main guarantee for the budget gate, assumes the reserve is a simultaneous high-probability upper bound on the undiscounted recovery cost. The implemented reserve, a discounted expected cost plus ensemble standard deviation, does not satisfy this, as the authors acknowledge in App. A.3.
    • The margin condition Csoft+η<ChardC_{\text{soft}} + \eta < C_{\text{hard}} cannot hold on Point, where Csoft=Chard=16C_{\text{soft}} = C_{\text{hard}} = 16 (Table 3). Why was this setting chosen, and how does performance change when a gap is introduced, as the analysis suggests?
    • Props. 1 and 2 are elementary: a set-inclusion argument and a geometric tail bound. As a result, the stated contribution on the conditions under which each gate works does not translate into guarantees for the algorithm that is actually run.
  5. The method has many environment-specific thresholds, and their sensitivity is largely unreported.

    • TollGate introduces several hyperparameters on top of the constrained learners: the feasibility threshold θF\theta_F, the gate line CsoftC_{\text{soft}}, the reserve coefficient κ\kappa, the label set GF\mathcal{G}_F, and the gate warmup. These differ across environments (Table 3).
    • CsoftC_{\text{soft}} for the grid was selected from a sweep (Table 6), but no sensitivity analysis is given for Point and Car.
    • Choosing CsoftC_{\text{soft}} well seems to require knowing typical recovery costs in advance, which is in tension with the goal of autonomous training. How should a practitioner set these thresholds on a new task?
Questions for authors

Already mentioned in the weaknesses section

3 comments
AuthorsOpenReview ↗

Response to Dmap(1/3)

Thank you for the careful summary and concrete suggestions.

1. Experimental scale

  • Choice of tasks. We chose navigation tasks with measurable recovery to the initial state and hazards that incur cost during both task execution and recovery. This lets us study how the two phases share a safety budget. Existing reset-free benchmarks such as EARL provide recovery tasks but do not define this safety cost.
  • GFG_F beyond position. The monitor uses a binary recovery-success criterion GstopG_{\rm stop} and a label set GF⊇GstopG_F \supseteq G_{\rm stop} (Prop. 1). For manipulation, this criterion can include both robot and object configurations, using the return-to-initial-configuration criteria already defined in reset-free manipulation benchmarks such as EARL.
  • Extending our evaluation to manipulation tasks and real robots is an important direction for future work.

2. Baselines

  • TollGate's contribution is deciding when to hand off under a shared safety budget. To isolate this decision from differences in policy learning, our switching comparisons use representative rules from prior methods with the same constrained learners, phase horizons and cost limits. We have expanded these comparisons with RISC-style switching and separately tested whether gates remain useful when policies observe the remaining budget.
  • LNT-style threshold selection. On the grid, we tested 8 thresholds, qmin⁡∈{0,−0.5,−1,−1.5,−2,−2.5,−3,−4}q_{\min} \in \lbrace 0, -0.5, -1, -1.5, -2, -2.5, -3, -4 \rbrace. None matches both the task success and hard-cap rate of feasibility gating. At −3 and −4, the rule never fires and matches NoGate. On Car, matching TollGate's per-step handoff rate on its own states requires thresholds from −4.8 to 0.58 across seeds and checkpoints, showing how the value scale changes during learning. This highlights the extra reset-reward design and calibration burden of value-based handoffs. LNT's original experiments use task-specific reset rewards and reward normalisation. Our grid sweep addresses the concern that the baseline result comes from a single unfavourable threshold.
  • RISC-style switching (grid, App. D.5). We use forward-to-reset switching with parameters from RISC's gridworld setting. RISC switches away once confident to redirect practice. Here it ends 120 of 250 forward phases before the goal in the final 50 intervals. The resulting comparison is:
MetricTollGateRISC-style
Task success, final 50 intervals0.990.45
Hard-cap rate, whole run4.1%6.2%
Manual resets, whole run37.434.8
Total cost, whole run950726

TollGate's higher total cost partly reflects the additional interaction from allowing task exploration to continue: 11,358 vs 10,145 environment steps over training, averaged over five layouts.

  • Saute-style learning (grid, App. D.5). We add the remaining interval budget to both constrained policies' observations and compare learning with and without TollGate's gates. Adding the gates lowers the hard-cap rate from 10.1% to 3.5% and manual resets from 49.0 to 34.4 (both on 5/5 layouts), while solved layouts rise from 4/5 to 5/5. Explicit handoff decisions thus remain useful when the policies themselves observe the budget. Implementation details are in App. D.5.
  • Car budget-gate-only arm. The two batches differ only in the feasibility-label radius and the monitor threshold, and the budget-only rule reads neither. These configuration differences therefore do not affect the comparison (App. C.4).
AuthorsOpenReview ↗

Response to Dmap(2/3)

3. Safety benefits over training

  • Grid total cost (950 vs 556). TollGate's forward-phase cost is 483, lower than the episodic learner's 556 (−13%). The rest is the cost of driving back to s0s_0, which the episodic learner avoids by being reset externally. Fig. 3c shows the price per manual reset at which the two break even.
  • Car, TollGate vs feasibility gate only. Adding the budget gate reduces manual resets from 496 to 352 over training and from 15.3 to 0 in the final 200k steps (IQMs; fewer in all 5 seeds). This comes with higher recovery cost. The final-window reset-phase cost is 1.82, below d=5d = 5, and forward return is 2.50 vs 2.66. The benefit is reduced dependence on manual intervention under the same hard budget.
  • Table 5. The constructed grid isolates gate decisions with zero Lagrange multipliers and hard-budget resets disabled (App. C.1). Earlier handoffs reduce both expenditure and task completion in this diagnostic. In the main constrained-learning comparison (Table 1), TollGate and feasibility gate only reach success 0.99 and 0.98, both solving 5/5 layouts.

Whole-run and late-training results. The cost threshold dd is the policies' learning target. The statements about costs below dd refer to the final window. We will distinguish whole-run totals from late-training statistics in the main text:

  • Manual resets. Over training, TollGate uses 37 on the grid vs 300 for episodic training, and 352 on Car vs 8,242 for an episodic reference that resets after every attempt over the same 1M steps.
  • Autonomous resets. On Car, the autonomous-reset fraction rises from 94–97% over the whole run to 99.9–100% in the final 200k steps; on Point, it rises from 72–78% to 99.5–100% over the same windows.
  • Hard-cap rates. Fractions of intervals reaching ChardC_{\rm hard} are listed below (grid mean; continuous-task IQMs across five seeds).
TollGate environmentWhole runFinal 200k steps
Grid4.1%—
Car1.6%0.0%
Point17.4%0.05%

On Car, whole-run hard-cap rates are 3.7% with feasibility gating only and 7.1% with budget gating only; TollGate's rate is lower in all 5 seeds against each.

4. Theory

  • Prop. 3 identifies sufficient conditions for a recovery reserve to prevent budget exhaustion (Section 4, App. A.3). We also measured the consequences of reserve errors in the implemented method. On Car after 400k steps, 98.3–99.8% of budget-gate handoffs whose realised recovery cost exceeds the reserve avoid reaching the hard budget (five seeds pooled per 200k window). Across all 18,039 handoffs in this period, 13 (0.07%) reach it during recovery.
  • Csoft=ChardC_{\rm soft} = C_{\rm hard} on Point. These values were chosen empirically during development. They are permitted by the switching rule but fall outside Prop. 3's strict-margin condition. The gate checks spent cost plus predicted recovery cost, so it can still trigger before spent cost reaches the hard budget. The gate comparisons in Tables 1–2 use positive margins (grid 7/10, Car 16/24). Section 5 below reports the grid margin sweep.
  • Role of Props. 1 and 2. These propositions explain how handoffs can sustain the feedback needed for an automatic curriculum. Prop. 1 makes accepted recovery success yield positive labels. Without compatible success and label sets, a loop can arise in which recoveries succeed but supply only negative labels and no task progress (App. A.1). Prop. 2 addresses feedback opportunities generated adaptively by the monitor's own handoffs. Its bound conditions on the learning history and requires no independence. The regional extension identifies the feedback needed beyond previously explored states. Fig. 6 illustrates the corresponding expansion of observed recovery distances and the monitor's accepted region.
AuthorsOpenReview ↗

Response to Dmap(3/3)

5. Thresholds and their sensitivity

  • Parameter roles. dd and ChardC_{\rm hard} specify the cost target and per-attempt allowance. TollGate adds the feasibility threshold, reserve coefficient, gate line and warmup. The label set is chosen to contain the recovery-success set (Prop. 1).
  • Feasibility threshold. On the grid, θF∈{0.1,0.2,0.3,0.5}\theta_F \in \lbrace 0.1, 0.2, 0.3, 0.5 \rbrace all solve 5/5 layouts.
  • Reserve coefficient. Grid training with κ∈{0,1,2,8,32}\kappa \in \lbrace 0, 1, 2, 8, 32 \rbrace gives 4–5/5 solved layouts, 36–40 manual resets and a hard-cap rate of 3.6–4.3%. On Car, 37% of recorded budget-gate triggers depend on the κσ\kappa\sigma term at that step (23% in the final 200k).
  • Budget margin. With grid Csoft=7C_{\rm soft} = 7, hard budgets of 9, 10, 12 and 15 all solve 5/5 layouts. The full sweep is below; all settings are reported in App. D.4.
Chard−CsoftC_{\rm hard} - C_{\rm soft}012358
Layouts solved4/54/55/55/55/55/5
Whole-run manual resets63.662.644.437.432.434.0

This supports the rationale in Prop. 3 for leaving room between the gate line and the hard limit.

Setting the gate line. For a new task, start from its cost allowance ChardC_{\rm hard}, then choose Csoft<ChardC_{\rm soft} < C_{\rm hard} to leave a switching margin. The reset cost critic estimates recovery costs online, so no typical recovery cost needs to be supplied in advance. The margin allows for growth of spent cost plus reserve between checks and for prediction error. Prop. 3 formalises the reserve and margin conditions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for recognising the motivation and novelty of our work. To address their concerns about prediction reliability, baseline comparisons and sensitivity, we have added recovery diagnostics, new baseline comparisons and parameter sweeps.

To address the baseline concerns raised by Reviewers Dmap and dZCE, we hold the constrained learners, phase horizons and cost limits fixed and vary the switching rule. This separates the contribution of handoff timing from differences in policy learning, curricula and safety settings. A separate Saute-style comparison tests whether gates still help when policies observe the remaining budget.

Switching or learning principlePrior methodsComparison
Reset-value thresholdLNT; SRRLLNT-style rule; sweep of 8 thresholds (new)
Success-classifier thresholdKim et al. (2022)Feasibility gate only
Recovery after goal or timeoutForward–backward alternation (FBRL)NoGate (new)
Competence-based switchingRISCRISC-style switching (new)
Remaining budget in the policy stateSauteSaute-style learner, with and without gates (new)

Learning and curriculum methods such as VaPRL (Sharma et al., 2021), MEDAL (Sharma et al., 2022), IBC (Kim et al., 2023) and MoReFree (Yang et al., 2025) can complement handoff design. The individual replies discuss their relationship to TollGate.

New results :

  1. Predictive handoffs beyond learned recovery. To address Reviewer k92N's request for a recovery-enabled baseline without predictive handoffs, we compare TollGate with NoGate on the grid. TollGate solves 5/5 vs NoGate's 4/5 layouts, with fewer manual resets (37.4 vs 46.0, lower on 4/5) and lower hard-cap rates (4.1% vs 8.9%, lower on 5/5), at total cost 950 vs 903.
  2. Prediction accuracy. To address k92N's reliability concern, we compared learned predictions with recovery measurements. At the end of grid training, the cost critic has MAE 0.08 and correlation 0.98 against exact expected discounted costs of the frozen policy. Feasibility decisions agree in 180/180 cells. On Car, at handoffs triggered only by the budget gate in the final 200k steps, the cost critic predicts an average discounted recovery cost of about 13. Its MAE against realised discounted recovery costs is 0.77–0.94 per seed.
  3. Consequences of prediction errors. We examined reserve errors in response to k92N's reliability concern and Dmap's and dZCE's questions about the implemented reserve. On Car after 400k steps, 98.3–99.8% of budget-gate handoffs whose realised recovery cost exceeds the reserve avoid reaching the hard budget (five seeds pooled per 200k window). On Car and Point, interaction spent on handoffs inside the recovery-success region decreases over training as recovery feedback updates the monitor.
  4. Gates with budget-aware policies. Dmap asked whether budget-aware learning could address affordability without separate gates. On the grid, adding the gates to the Saute-style learner lowers hard-cap rates from 10.1% to 3.5% and manual resets from 49.0 to 34.4, both on all 5 layouts.
  5. Parameter sensitivity. All three reviewers requested sensitivity analysis. Grid feasibility thresholds of 0.1, 0.2, 0.3 and 0.5 all solve 5/5 layouts. Reserve coefficients of 0, 1, 2, 8 and 32 solve 4–5/5. With Csoft=7C_{\rm soft} = 7, hard budgets of 9, 10, 12 and 15 all solve 5/5.

Manuscript revisions.

  • Following Dmap's request for whole-run reporting and k92N's concern about early intervention, we have distinguished whole-run totals from late-training statistics in the main text. Whole-run hard-cap rates are provided in the reply to Dmap.
  • For the baseline concerns raised by all three reviewers, we have added the grid comparisons to App. D and explained the comparison design there.
  • For the prediction concerns raised by k92N and the reserve questions raised by Dmap and dZCE, we have added diagnostics on the submitted runs to App. D. For k92N's concern about premature handoffs, Section 5.3 now points to Fig. 6 for the monitor's recovery feedback.
  • To address all three reviewers' sensitivity concerns, we have added the full grid parameter sweeps to App. D.
  • We have added the three references requested by dZCE.

The grid implementation requested by k92N has been added to the anonymized supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.