- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Reset-free RL removes manual resets by alternating a task policy with a learned reset policy, but existing handoff rules only ask whether recovery is feasible, not whether the agent can afford it: task execution and recovery draw on one shared per-attempt safety budget, so recoverability does not imply affordability. TollGate makes the handoff budget-aware with two gates: an outcome-supervised feasibility monitor, and a budget gate that triggers when spent cost plus a predicted recovery reserve (ensemble cost-critic mean plus an uncertainty margin κσ) crosses a soft gate line. Both policies are trained with SAC-Lagrangian under a shared cost threshold d, and a hard budget C_hard terminates any interval that exceeds it. On a grid world and Safety-Gymnasium Point/Car, TollGate learns the tasks with 99.5–100% autonomous resets in late training and per-phase costs below threshold.
Critical strengths
- The problem is well-motivated and realistic: recoverability does not imply affordability, and prior reset-free methods track no cumulative cost.
- The outcome-supervised feasibility monitor is a sound design: it is trained directly on observed recovery outcomes, involves no bootstrapped targets, and its gradient does not flow into either policy's learning objective.
Critical weaknesses
- The headline comparisons are against an LNT-style (2018) reimplementation plus ablations of TollGate's own gates. Other reset-free methods like VaPRL, MEDAL, RISC, IBC, SRRL, Back to Base are cited but never evaluated, and EARL-scale or manipulation benchmarks are absent.
- The theory does not cover the implementation. Proposition 3 requires a reserve with simultaneous coverage of undiscounted recovery costs, but the implemented reserve is a discounted expected critic plus ensemble spread, and the authors concede the empirical results establish neither assumption. In practice, the hard budget, not the gates, is what actually enforces safety.
- The sensitivity analysis is very limited: sweeps cover only the gate line C_soft on the grid, while more important hyper-parameters such as θ_F, κ, and C_hard, as well as additional tasks, are left unexamined.
Questions for authors
Missing references.
- Reset-Free Reinforcement Learning via Multi-Task Learning (ICRA 2021)
- Reset-Free Lifelong Learning with Skill-Space Planning (ICLR 2021)
- Reset-free Reinforcement Learning with World Models (TMLR 2025)
3 comments
Response to dZCE(2/2)
W3. Sensitivity
We added sweeps of the parameters the reviewer lists (grid, 5 layouts each, App. D.4):
- : all solve 5/5 layouts.
- : 4–5/5 solved, 36–40 manual resets, hard-cap rate 3.6–4.3%. On Car, 37% of budget-gate triggers depend on the term at the trigger step (34.8–39.0% per seed; 23% in the final 200k steps).
- with : hard budgets of 9, 10, 12 and 15 all solve 5/5 layouts. The full margin sweep is in App. D.4.
- The gate line solves 5/5 (Table 6).
Missing references
We thank the reviewer and will add all three to Related Work. MTRF (Gupta et al., ICRA 2021) lets learned tasks serve as resets for one another. LiSP (Lu et al., ICLR 2021) plans in a learned skill space to avoid sink states. MoReFree (Yang et al., TMLR 2025) adapts model-based exploration and imagination training to the reset-free setting. None of the three accounts for a safety cost shared by task execution and recovery, which is the problem TollGate addresses.
Thank you for providing more detailed explanation. As given by my original review, I still think more reset-free baselines and benchmarks are needed, so I would remain my original score.
Response to dZCE(1/2)
Thank you for the detailed review.
W1. Baselines and benchmarks
TollGate addresses when task execution should give way to recovery under a shared safety budget. We have expanded the grid comparisons to cover additional handoff mechanisms and budget-aware learning (App. D.1, D.4, D.5):
Other methods.
These differences motivate our comparison of representative handoff mechanisms using the same constrained learners, phase horizons and cost limits. This design isolates the contribution of deciding when to recover under a shared safety budget.
Benchmarks. We chose navigation tasks where both phases encounter hazards and recovery to the initial state is measurable, allowing us to study the shared safety budget. EARL provides reset-free tasks but does not define this safety cost. Extending our evaluation to manipulation tasks is an important direction for future work.
W2. Theory and the role of the hard budget
In our setting, reaching the hard budget terminates the current attempt and requires a manual reset. This stopping mechanism relies on external intervention and does not itself teach the policies low-cost behaviour. Constrained learning trains the task and reset policies under cost constraints. The gates use recovery capability, cost already spent and predicted recovery cost to trigger early handoffs and reduce forced manual resets.
Prop. 3 gives sufficient conditions for avoiding budget exhaustion through a recovery reserve and a handoff margin. The reserve must cover undiscounted recovery costs across possible handoff times, and the gap between the gate line and the hard budget must accommodate growth in spent cost and the reserve between checks. In practice, we construct the reserve from a discounted cost critic and ensemble disagreement. We use error diagnostics to assess the consequences of prediction errors and comparisons under the same budget to evaluate the gates' practical contribution (App. D.2–D.4).
On Car, recomputing the budget-gate decisions at recorded trigger states shows that 37% would not have fired at that step without the κσ term. After 400k steps, among budget-gate handoffs where realised recovery cost exceeds the reserve, 98.3–99.8% avoid budget exhaustion. Under the same learners, hard budget and number of environment steps, adding the budget gate to feasibility gating reduces whole-run budget exhaustion during recovery from 263 to 121 and manual resets from 496 to 352 (IQMs, with both reductions in all five seeds).