- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new way of integrating an online RL system with a baseline controller in order to adapt the online controller. The authors identify two challenges when doing that. First, improving the RL and handing over the control needs different value estimates. Second, value estimates of the RL carries extrapolation error.
Critical strengths
- The problem authors trying to solve is well-motivated and has real-world practical use cases.
- The paper reads well and is easy to follow.
- The idea, ie arbitrator gated, is well defined.
Critical weaknesses
- Missing literature discussion and comparison as baseline: [1],[2],[3],[4]
- How can one make sure that the blending action formula (eq 1) leads to a valid action? Each a_base or a_rl might be valid actions individually but when combined, they might either be invalid eg by violating the system's limitations or lead to bad outcomes that are not identifiable by the critic ensemble.
- In line 106, what's pi_beta? It's not defined earlier.
- Tasks in the experiments section are simple and do not show the full potential and limitations of the proposed method. Also, comparing against just one baseline that is specifically designed for this problem (RL-AR) is limited. More baselines that are designed for this problem should be added to the experiment section.
[1] Zhang et al. "Policy expansion for bridging offline-to-online reinforcement learning." ICLR 2023. [2] Laroche et al. "Safe policy improvement with baseline bootstrapping." ICML 2019. [3] Danesh et al. "Safe domain randomization via uncertainty-aware out-of-distribution detection and policy adaptation." arXiv:2507.06111. [4] Hu et al. "Imitation bootstrapped reinforcement learning." arXiv:2311.02198.
Questions for authors
See above.
1. We appreciate these suggestions, which helped us broaden both the literature discussion and the empirical evaluation. We now cite PEX [1] and IBRL [4] in the introduction as approaches that select between a pretrained reference policy and a learner using value estimates. IBRL also incorporates this selection into its bootstrap targets, which connects to our discussion of continuation policies. We additionally cite these approaches in the conclusion when discussing discrete selection as an alternative where action blending may be unsuitable.
SPIBB [2] and UARL [3] share our concern with reliable policy improvement under uncertainty, although their training settings differ. SPIBB learns from a fixed dataset and seeks improvement relative to the data-collecting baseline. UARL refines policies through domain randomization in simulation and uses target-domain data to assess deployment suitability. Neither studies online handover from an existing controller while that controller keeps the system running.
Following your suggestion to broaden the comparisons, we have added CORE-RL, BCF and RPL in Section 6. We prioritized methods designed for online learning alongside an existing baseline controller, using its action proposals without a separate offline pretraining stage. The results are reported in Section 6.1, Table 1 of the revised manuscript.
2. In all our experiments, the action space has independent lower and upper bounds for each input. Equation (1) blends each component with a weight in [0,1], so it remains between the two proposed values and within those bounds. The blended action therefore respects the actuator limits by construction. For action spaces with coupled or nonconvex input constraints, feasibility would need to be enforced separately. Respecting input bounds does not guarantee satisfaction of state constraints.
For outcomes beyond input feasibility, the arbitrator estimates the return of the blended policy that actually controls the system. Training the gate against this estimate can discourage blends predicted to perform poorly. The actor prior encourages learner actions near the baseline proposals in the replay buffer, and LayerNorm stabilizes critic estimates. These mechanisms limit estimation error, but learned values can still miss unsafe outcomes during training. Even accurate return estimates do not certify constraint satisfaction.
We therefore assess training safety empirically through the failure counts in Section 6.1, Table 1 and the extended ablation in Section 6.2, Table 2. We explicitly acknowledge the limitation in the conclusion (Section 7)
Blending action proposals that correspond to different successful strategies can also cause poor outcomes. We have added this limitation to the conclusion (Section 7)
For these settings, the conclusion identifies discrete selection between baseline and learner actions as a possible extension.
3. πβ denotes the blended policy that executes aβ(s)(s) at each state s. It was introduced immediately after Eq. (1) in Section 2. We have reworded the definition to make the notation clearer.
4. The expanded baseline comparison is reported in Section 6.1, Table 1 of the revised manuscript. We have also extended the ablation to all four environments and moved it into the main paper, in Section 6.2, Table 2.
We chose these tasks because they provide existing MPC controllers and interpretable model mismatches, allowing us to study training safety and final performance during handover. They include three benchmarks used in RL-AR, made more challenging here, and a quadrotor task from safe-control-gym. Together, they cover balancing, process control, medical dosing and aerial robotics. Although low-dimensional, they present a challenging online adaptation problem. RL-AR incurs more than one hundred mean training failures on each task.
References
[1] Zhang et al. “Policy expansion for bridging offline-to-online reinforcement learning.” ICLR 2023.
[2] Laroche et al. “Safe policy improvement with baseline bootstrapping.” ICML 2019.
[3] Danesh et al. “Safe domain randomization via uncertainty-aware out-of-distribution detection and policy adaptation.” arXiv:2507.06111.
[4] Hu et al. “Imitation bootstrapped reinforcement learning.” arXiv:2311.02198.