acceptodds
Under review as a conference paper at ICLR 2027

Arbitrated Handover from Baseline Control to Reinforcement Learner

Abstract

A running system cannot pause for a new controller to learn. Online controller adaptation avoids the pause by keeping a fixed baseline in control while a reinforcement learning policy, the learner, trains on the live system. Gating blends the two actions, passing control gradually from baseline to learner over training. The gate must judge when the learner is skilled enough to take over; handing over too early costs performance and can cause failures the baseline would have avoided. We identify two core challenges in this setting. The first is that improving the learner and judging the handover need different value estimates: one of the learner acting alone, one of the blend that actually runs. The second is that, because the learner does not act on its own until the end of training, its value estimate carries extrapolation error. We find that neglecting either results in premature handover, causing performance degradation and task failure during training. We propose arbitrator-gated handover: an arbitrator judges the executed blend with value estimates of its own, and we apply techniques to limit the effects of value estimation error in this setting.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, online control adaptation, hybrid learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_z7p7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new way of integrating an online RL system with a baseline controller in order to adapt the online controller. The authors identify two challenges when doing that. First, improving the RL and handing over the control needs different value estimates. Second, value estimates of the RL carries extrapolation error.

Critical strengths
  • The problem authors trying to solve is well-motivated and has real-world practical use cases.
  • The paper reads well and is easy to follow.
  • The idea, ie arbitrator gated, is well defined.
Critical weaknesses
  • Missing literature discussion and comparison as baseline: [1],[2],[3],[4]
  • How can one make sure that the blending action formula (eq 1) leads to a valid action? Each a_base or a_rl might be valid actions individually but when combined, they might either be invalid eg by violating the system's limitations or lead to bad outcomes that are not identifiable by the critic ensemble.
  • In line 106, what's pi_beta? It's not defined earlier.
  • Tasks in the experiments section are simple and do not show the full potential and limitations of the proposed method. Also, comparing against just one baseline that is specifically designed for this problem (RL-AR) is limited. More baselines that are designed for this problem should be added to the experiment section.

[1] Zhang et al. "Policy expansion for bridging offline-to-online reinforcement learning." ICLR 2023. [2] Laroche et al. "Safe policy improvement with baseline bootstrapping." ICML 2019. [3] Danesh et al. "Safe domain randomization via uncertainty-aware out-of-distribution detection and policy adaptation." arXiv:2507.06111. [4] Hu et al. "Imitation bootstrapped reinforcement learning." arXiv:2311.02198.

Questions for authors

See above.

1 comment

1. We appreciate these suggestions, which helped us broaden both the literature discussion and the empirical evaluation. We now cite PEX [1] and IBRL [4] in the introduction as approaches that select between a pretrained reference policy and a learner using value estimates. IBRL also incorporates this selection into its bootstrap targets, which connects to our discussion of continuation policies. We additionally cite these approaches in the conclusion when discussing discrete selection as an alternative where action blending may be unsuitable.

SPIBB [2] and UARL [3] share our concern with reliable policy improvement under uncertainty, although their training settings differ. SPIBB learns from a fixed dataset and seeks improvement relative to the data-collecting baseline. UARL refines policies through domain randomization in simulation and uses target-domain data to assess deployment suitability. Neither studies online handover from an existing controller while that controller keeps the system running.

Following your suggestion to broaden the comparisons, we have added CORE-RL, BCF and RPL in Section 6. We prioritized methods designed for online learning alongside an existing baseline controller, using its action proposals without a separate offline pretraining stage. The results are reported in Section 6.1, Table 1 of the revised manuscript.

2. In all our experiments, the action space has independent lower and upper bounds for each input. Equation (1) blends each component with a weight in [0,1][0,1], so it remains between the two proposed values and within those bounds. The blended action therefore respects the actuator limits by construction. For action spaces with coupled or nonconvex input constraints, feasibility would need to be enforced separately. Respecting input bounds does not guarantee satisfaction of state constraints.

For outcomes beyond input feasibility, the arbitrator estimates the return of the blended policy that actually controls the system. Training the gate against this estimate can discourage blends predicted to perform poorly. The actor prior encourages learner actions near the baseline proposals in the replay buffer, and LayerNorm stabilizes critic estimates. These mechanisms limit estimation error, but learned values can still miss unsafe outcomes during training. Even accurate return estimates do not certify constraint satisfaction.

We therefore assess training safety empirically through the failure counts in Section 6.1, Table 1 and the extended ablation in Section 6.2, Table 2. We explicitly acknowledge the limitation in the conclusion (Section 7)

“Our gate reduces violations empirically but does not bound them. As a direction for future work, this method could be extended with formal safety guarantees, for example by using a safety filter.”

Blending action proposals that correspond to different successful strategies can also cause poor outcomes. We have added this limitation to the conclusion (Section 7)

“A second limitation is that blending two acceptable action proposals can lead to a poor or unsafe outcome, particularly with multimodal rewards or dynamics.”

For these settings, the conclusion identifies discrete selection between baseline and learner actions as a possible extension.

3. πβ\pi_\beta denotes the blended policy that executes aβ(s)(s)a_{\beta(s)}(s) at each state ss. It was introduced immediately after Eq. (1) in Section 2. We have reworded the definition to make the notation clearer.

4. The expanded baseline comparison is reported in Section 6.1, Table 1 of the revised manuscript. We have also extended the ablation to all four environments and moved it into the main paper, in Section 6.2, Table 2.

We chose these tasks because they provide existing MPC controllers and interpretable model mismatches, allowing us to study training safety and final performance during handover. They include three benchmarks used in RL-AR, made more challenging here, and a quadrotor task from safe-control-gym. Together, they cover balancing, process control, medical dosing and aerial robotics. Although low-dimensional, they present a challenging online adaptation problem. RL-AR incurs more than one hundred mean training failures on each task.

References

[1] Zhang et al. “Policy expansion for bridging offline-to-online reinforcement learning.” ICLR 2023.

[2] Laroche et al. “Safe policy improvement with baseline bootstrapping.” ICML 2019.

[3] Danesh et al. “Safe domain randomization via uncertainty-aware out-of-distribution detection and policy adaptation.” arXiv:2507.06111.

[4] Hu et al. “Imitation bootstrapped reinforcement learning.” arXiv:2311.02198.

Reviewer_upSxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies online controller adaptation, where a fixed baseline controller remains in operation while an RL policy is learned and gradually takes over control. The authors identify two problems with existing handover methods such as RL-AR: the learner’s critic evaluates the learner rather than the blended controller actually being executed, and its value estimates can suffer from extrapolation error because the learner’s actions are initially poorly represented in the collected data. They propose arbitrator-gated handover, which uses a separate critic to evaluate the blended policy, together with a pessimistic critic ensemble, LayerNorm, and a prior that regularizes the learner toward the baseline. Across four continuous-control environments, the method substantially reduces constraint violations during training while producing final standalone policies that outperform the baseline controller.

Critical strengths

Strengths. The paper studies an important and practically motivated setting and identifies a compelling failure mode in existing approaches to controller handover. In particular, I found the distinction between the value function required to optimize the learner and the value function required to judge the executed blended controller to be both insightful and convincingly motivated. The theoretical counterexample nicely demonstrates that reusing the learner critic can misdirect handover even with exact value estimates, providing a clear motivation for the proposed arbitrator. I also appreciated that the remaining design choices are tied to the second identified challenge of value-estimation error, resulting in an overall coherent method. Empirically, the results are strong across all four environments, with substantial reductions in failures while maintaining or improving performance. Importantly, the authors separately evaluate the final learner without assistance from the baseline, ruling out the trivial explanation that improved safety simply results from retaining baseline control. Finally, the experiments are generally careful and transparent, with multiple seeds, per-seed results, ablations, and detailed implementation information.

Critical weaknesses

Missing comparison/discussion of adaptive MPC. The paper motivates the proposed approach primarily through mismatch between the nominal model underlying the MPC and the true system. However, an established alternative is to adapt the dynamics model online while retaining MPC, rather than learning a separate model-free policy and handing control over to it. For example, prior work* on online dynamics learning for MPC continually updates the model during deployment and explicitly demonstrates adaptation to disturbances/model errors, including on quadrotor systems. This alternative seems particularly relevant here since several experimental perturbations are fixed parameter changes or constant unmodeled forces. I would therefore appreciate either a comparison against a representative adaptive/learning-based MPC method or a clearer discussion of when policy adaptation via RL handover is preferable to online model adaptation. Even if the nominal model is not learned, one could e.g. train a diff-model online... The lack of this comparison is not critical in the sense that it requires rejection, however a couple of pretty straightforward baselines are missing (e.g. the also the mentioned BCF or CHEQ), so the overall empirical evaluation is a bit narrow.

*(See e.g. Online Dynamics Learning for Predictive Control with an Application to Aerial Robots, CoRL 2022 / PMLR 2023)

More critically, the current ablation does not fully isolate the contribution of the arbitrator and thus doesnt cleanly establish that the new arbitrator is responsible for a substantial part of the empirical gains (which appears to be to me a central piece of the paper). In particular, we'd need to see RL-AR + LayerNorm + actor prior while retaining the original learner-critic-based gate, compared against the full method. This seems important because LayerNorm alone already reduces e.g. CSTR training failures from 184.4 to 1.8, suggesting that critic stabilization explains almost all of the improvement. The sequential ablation shows that each component can contribute in the chosen ordering, but does not establish that the independent arbitrator remains necessary once the learner critic is stabilized and the actor is regularized toward the baseline. Such a comparison would more directly connect the central theoretical motivation, namely the continuation-policy mismatch of critic reuse, to the empirical gains.

Another issue i'd like to have at least briefly discussed: In some environments, creating a convex blend of actions from different policies may be inherently unsafe due to multi-modal reward / dynamics functions, i.e. baseline predicts -1, learner +1, both earn a decent reward, blend at 0 however is terrible... It seems to not be a practical issue in the environments observed, but generally it appears to me like a potential hazard.

Questions for authors
  1. would you agree that the main improvements simply come from adding LayerNorm?

  2. Would it be possible to add the requested additional ablation?

  3. Would it be possible to add further relevant baselines as mentioned in weaknesses? / Do you have specific reasons not to consider those?

  4. Do you see the same potential issue with multi-modal rewards / dynamics?

1 comment

We thank you for your thorough review and raising several interesting points for discussion, which have helped us sharpen our arguments.

1–2. The extended ablation shows additional safety gains from the arbitrator after LayerNorm and the actor prior are included. We have added RL-AR + LayerNorm + actor prior with the original gate based on the learner critic. The ablation now covers all four environments and appears in Section 6.2, Table 2 of the main paper.

Adding the arbitrator to LN + Prior reduces mean training failures from 39.0 to 1.3 on CartPole, 10.2 to zero on BiGlucose, and 107.8 to zero on Quadrotor2D. CSTR already incurs very few training failures without the arbitrator. The full method also produces learners with no observed standalone failures on all four tasks.

LayerNorm and the actor prior contribute to controlling estimation error. This new comparison supports the empirical importance of continuation-policy alignment.

3. Regarding Weakness 1, we agree that online model adaptation is an established alternative, particularly for the fixed parameter changes and constant forces considered here. The expanded introduction now cites the suggested paper, Online Dynamics Learning for Predictive Control with an Application to Aerial Robots (CoRL 2022 / PMLR 2023). In the introduction, we now state

“Where the model behind the controller is accessible, the model itself can be adapted online (Jiahao et al., 2023).”

We also clarify the requirements on the baseline controller

“Our method is agnostic to the type of baseline controller. It relies only on the baseline action proposals and does not require access to the underlying model or controller implementation.”

Our empirical comparisons therefore focus on other policy adaptation methods that are agnostic to the type of baseline controller. Your comment helped us clarify these use cases and the practical strengths of the approach.

In response to Question 3, we have expanded the evaluation to include BCF, CORE-RL and residual policy learning in Section 6. These cover uncertainty-based policy fusion, control regularization and additive policy corrections, respectively. We also evaluate the adaptive TD-error variant of CORE-RL in Appendix G.2, Table 10. The results are reported in Section 6.1, Table 1 of the revised manuscript.

4. We agree that two acceptable action proposals do not guarantee that their blend is acceptable, as your example illustrates. This could happen, for example, in obstacle avoidance. Two policies may safely steer around opposite sides of an obstacle, while their blend steers toward it.

We did not observe this issue in our experiments. All four environments are regulation or tracking tasks with costs that penalize deviations from a target or reference trajectory. The actor prior encourages the learner to stay close to the baseline action proposals stored in the replay buffer, whose early transitions are dominated by baseline control. This may favor agreement on the direction of correction, with differences mainly in magnitude. In that case, the blend lies between nearby action proposals. Since the gate is trained to improve the estimated return of the blended action, it may also discourage blends that perform poorly. Such blends can still occur over the course of training.

For tasks where the learner needs to depart substantially from the baseline actions, discrete selection between their proposals may be more suitable than blending. Your comment points to an interesting avenue for future work, which we also include in the conclusion (Section 7)

“A second limitation is that blending two acceptable action proposals can lead to a poor or unsafe outcome, particularly with multimodal rewards or dynamics. […] Extending the arbitrator to evaluate such a handover is a direction for future work.”

Reviewer_tHpPOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

They are looking at novel ways to determine when to switch from a baseline control to a reinforcement learner through the use of arbitrator judge. The baseline control is the preexisting control. They bring up two problems: first that the transfer is based off the expected value of the blend from the RL and the control, and second, the RL error is corrected only as the gate transfer to it.

They contribute meaningfully by identifying these challenges, proposing a solution using an arbitrator-gated handover, and demonstrate effectiveness on several experiments.

Critical strengths
  1. They have identified two flaws to existing systems, and provide a proof for the first flaw, and use existing literature to back the second.
  2. They methodically go through the weaknesses and provide a careful solution that addresses these
  3. The paper reads well and uses space effectively.
  4. They compare their solution to existing methods and show that theirs outperforms the others, while providing adequate nuance
  5. They state that the code will be publicly available and results will be published so others can contribute to further work.
  6. Their work makes the system significantly safer, and they claim by 1-3 orders of magnitude.
Critical weaknesses

None

Questions for authors

What is the computational cost, inference time, and training speed? Second, how does this compare to the other methods and are they significant in when considering to use the proposed approach?

1 comment

Thank you for your questions about computational cost and practical applicability.

We have broadened the comparison to include CORE-RL, BCF and residual policy learning alongside RL-AR and SAC. We have also added the computational analysis below to Appendix G.3 (Computational Cost), Tables 11 and 12.

AGH adds two arbitrator critics to RL-AR. The representative parameter counts below exclude target networks and vary by less than 3% across tasks.

MethodTrained networksParameters
SACActor and two critics204k
RL-ARActor, gate and two critics209k
RPLActor and two critics205k
CORE-RLActor and two critics204k
BCFTen actors and twenty critics2.04M
AGHActor, gate and four critics348k

Although AGH trains approximately two-thirds more parameters than RL-AR, the MPC solve accounts for most of the elapsed training time in our experiments. AGH, RL-AR, RPL, CORE-RL and BCF all compute an MPC action at each training interaction. The actor prior and arbitrator reuse these baseline action proposals, requiring no additional MPC solves. This computational balance depends on the baseline controller. With a cheaper controller, neural-network updates could account for a larger share of training time.

All methods receive the same interaction budget. The following measurements report wall-clock training time per collected transition, including action selection, system interaction and learning updates. Values are medians across ten seeds, in milliseconds.

MethodCartPoleCSTRBiGlucoseQuadrotor2D
SAC3334
RL-AR28508946
RPL21508844
CORE-RL18498643
BCF386510162
AGH24549149

The recorded AGH medians differ from RL-AR by at most 4 ms on each task. BCF has the highest recorded times, reflecting the additional updates to its ten-member ensemble.

During handover, AGH and RL-AR both evaluate the MPC, learner actor and gate. Their network contribution takes approximately 0.2 ms per action, and the arbitrator is not evaluated for action selection. The corresponding network contributions are approximately 0.1 ms for RPL and CORE-RL, and 0.4 ms for BCF. Each is added to the MPC solve, which takes approximately 20–95 ms depending on the task.

When deployed standalone, AGH requires only one actor forward pass, taking approximately 0.2 ms per action. SAC and standalone RL-AR have the same actor-only inference cost. The evaluated RPL, CORE-RL and BCF configurations retain the MPC at deployment and therefore continue to incur its solve time.

The practical benefit of AGH is few training failures alongside a competent standalone learner. AGH trains more parameters than SAC, RL-AR, RPL and CORE-RL, while remaining smaller than BCF. Its standalone evaluations show no observed failures and higher return than the MPC across all four tasks. The additional training work relative to methods with fewer trained parameters can therefore be worthwhile when constraint violations are costly and deployment requires fast action selection.

Reviewer_wH1QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The setting is online controller adaptation. A fixed baseline controller (MPC) keeps the system running while an RL learner (SAC) trains live, and a gate blends the two actions and gradually hands control to the learner. The paper argues that the existing approach (RL-AR) gates using the learner's own critic, which is wrong for two reasons. (1) It estimates the value of the learner acting alone, not the blend that actually runs, and Prop. 1 shows this can flip the sign of the gate update even with an exact critic. (2) The learner basically never acts alone during training, so that critic is extrapolating, like in offline RL. The fix is a separate "arbitrator" critic that bootstraps on the blended action, used as an ensemble with a lower confidence bound, plus LayerNorm in the critics and a gate-weighted BC-style prior pulling the actor toward the baseline. On CartPole, CSTR, BiGlucose and Quadrotor2D with model mismatch (10 seeds), training failures drop from hundreds to roughly zero, and final standalone learners beat both MPC and RL-AR.

Critical strengths
  1. The diagnosis is clear and correct. Gating on V of the learner when the decision is about V of the blend is a real mismatch, and the small counterexample in Prop. 1 makes it concrete. The fix (bootstrap on the executed blended action) is the obvious right thing once it's pointed out, and that's a good sign.
  2. The results are strong and reasonably well done: 10 seeds, CIs, per-seed failure counts in the appendix, and honest notes such as CartPole's 7.6 failures coming from one seed. Going from 100s of constraint violations to ~0 during training is exactly what this setting cares about.
Critical weaknesses
  1. The only gated baseline is RL-AR. The intro lists several other gating schemes (TD-error based, Cheng et al. 2019; policy ensemble disagreement, Rana et al. 2023; critic ensembles, Cramer et al. 2025; JSRL), plus residual RL, and none are compared. Since the paper's claim is "gating on the learner's critic is the problem", it would be much stronger to show the arbitrator also beats gates that don't use the learner's critic value directly.
  2. The ablation (Appendix F, only CSTR and Quadrotor2D) makes the attribution murky. LayerNorm on the learner's critic alone takes CSTR failures from 184 to 1.8 per seed and Quadrotor from 561 to 33, so most of the safety gain comes from a known trick. The arbitrator by itself mostly keeps the gate shut (β ≈ 0.78), and it's the actor prior that actually gets the handover to happen. So the headline component, the arbitrator, looks less important than the framing suggests, and RL-AR + LayerNorm + actor prior (no arbitrator) seems like a missing row. I'd also like the ablation to be in the main paper and to cover all four tasks.
Questions for authors
  • Can you add RL-AR + LayerNorm + actor prior, without the arbitrator, on all four tasks? That's the cleanest test of what the arbitrator itself adds.
  • Can you compare against at least one of the other gating approaches (e.g. Cramer et al. 2025 or the TD-error gate)?
  • The arbitrator estimates Q of the blend, but the blend changes every time β is updated. How stale does it get, and does the update rate of β vs the arbitrator matter?
  • How sensitive are the results to κ (the LCB coefficient), the ensemble size and λ (prior weight)? Were they tuned per task?
  • All four tasks are low-dimensional sims with a constant bias-type mismatch. Any sense of how this behaves with time-varying drift, which is part of the motivation?
1 comment

We thank you for your thoughtful comments and feedback which helped us to demonstrate the utility of the arbitrator critic, and to provide more detail on other components of the method. We address your questions below:

1. Thank you for this suggestion. This is a valuable ablation to include. We have added the requested RL-AR + LayerNorm + actor prior configuration, retaining the original gate based on the learner critic. We have extended the ablation to all four environments and moved it into the main paper, in Table 2, also above in the main comment.

This also addresses your second listed weakness. With LN + Prior already present, adding the arbitrator reduces mean training failures from 39.0 to 1.3 on CartPole. They fall from 10.2 to zero on BiGlucose and from 107.8 to zero on Quadrotor2D. CSTR already incurs very few failures without the arbitrator. The full method also produces learners with no observed standalone failures on all four tasks.

The extended ablation shows that its safety contribution remains substantial after critic stabilization and actor regularization are included.

2. We agree that RL-AR alone was too narrow a comparison. We have added CORE-RL, BCF and residual policy learning. CORE-RL includes the adaptive TD-error gate you mention (Cheng et al., 2019). We report tuned fixed weights in the main comparison and evaluate the adaptive variant in Appendix G.2, Table 10. BCF fuses baseline and learner action distributions according to their relative uncertainty (Rana et al., 2023). Residual policy learning adds the learner to the baseline without a gate (Silver et al., 2018). The results are reported in Section 6.1, Table 1 of the revised manuscript.

3. Thank you for this question. We have added a short clarification to Section 5.1 and made the recomputation of bootstrap actions explicit in Appendix A, Algorithm 1.

The arbitrator evaluates a moving policy because both the gate and learner change during training. SAC critics also track a changing policy.

Section 5.1 now states

“Transitions are stored as (s,a,abase,r,s′,abase′,d)(s,a,a_{\mathrm{base}},r,s',a_{\mathrm{base}}',d), with abase′a_{\mathrm{base}}' the action the baseline proposes at s′s', taken from the following step. The weight β(s′)\beta(s') and the action a~′\tilde a' are recomputed from the current gate and learner at every update.”

Bootstrap actions therefore reflect the current blend (Eqs. 10–11). The remaining lag is in the arbitrator and its target networks.

The relative update rates matter because the arbitrator must track the changing blend. We use one arbitrator update followed by one gate update per environment step. The learner policy uses an update period of two environment steps. We retain the critic and gate learning rates from the RL-AR setup. These are 10−310^{-3} for the arbitrator and 5×10−65\times10^{-6} for the gate. The smaller gate learning rate encourages slower gate changes while the arbitrator learns. This schedule is fixed across all four tasks.

4. Thank you for raising these questions. The extended ablation helped us simplify the main method. With LayerNorm and the actor prior included, the ensemble provided limited, task-dependent benefits in standalone return. Twin critics matched or reduced mean training-failure counts compared with the ensemble on all four tasks. Both variants produced learners with no observed standalone failures. The ensemble also requires more critic networks and introduces additional hyperparameters. We therefore use a twin-critic arbitrator in the main method. We retain the ensemble comparison and settings in Appendix G.1, Tables 8 and 9. In those experiments, N=5N=5 and κ=1\kappa=1 were fixed across all tasks.

The actor-prior weight λ\lambda is tuned per task. We swept 0.1,0.3,1,10\\{0.1,0.3,1,10\\} on all four tasks using five seeds per candidate. Selection considered the tradeoff between training failures and standalone return. We now include this sensitivity study in Appendix G.4, Table 13. The results show task-dependent sensitivity and support the selected weights as a favorable balance between training safety and standalone performance.

5. Our method targets adaptation after drift has degraded baseline performance. A controller may match the system at commissioning and lose performance through wear. We consider drift that is slow relative to an adaptation run. We have clarified the intended scenario in the introduction, which now states

“Adaptation is started once the mismatch shows, at commissioning or when performance has dropped.”

The corresponding assumption is stated explicitly in Section 2

“The mismatch between the system and the model behind the baseline is taken as fixed during training.”

The experiments represent this setting. If the dynamics continue to change during adaptation, both the learner and arbitrator would need to track those changes. Studying their ability to track time-varying mismatch within an adaptation run is a direction for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Significant Extension of Ablation and Baselines

We thank the reviewers for their thoughtful comments and suggestions. These helped us strengthen the empirical evaluation and clarify the method.

1. Extended ablation. We have expanded the ablation with additional component combinations, extended it to all 4 environments, and moved it into the main paper, in Section 6.2, Table 2. These additions include the requested RL-AR + LayerNorm + actor prior configuration, retaining the original gate based on the learner critic. The comparison below highlights this configuration alongside RL-AR and full AGH, reporting training failures and standalone learner return. Values are mean ± standard deviation over ten seeds.

TaskConfigurationTraining failuresStandalone learner return
CartPoleRL-AR127.7±82.4-1.84±1.48
LN + Prior (no arbitrator)39.0±41.1-0.78±0.82
Full AGH1.3±4.1-0.32±0.19
CSTRRL-AR184.4±159.4-1.69±1.84
LN + Prior (no arbitrator)0.3±0.9-0.27±0.28
Full AGH0.2±0.6-0.17±0.14
BiGlucoseRL-AR202.6±179.1-64.16±186
LN + Prior (no arbitrator)10.2±18.8-4.41±4.07
Full AGH0.0±0.0-3.13±0.88
Quadrotor2DRL-AR561.1±646.9-0.91±0.96
LN + Prior (no arbitrator)107.8±316.9-0.36±0.85
Full AGH0.0±0.0-0.13±0.05

The extended ablation shows that the arbitrator remains a core component for training safety after critic stabilization and actor regularization are included. Its addition substantially reduces mean training failures on CartPole, BiGlucose and Quadrotor2D, while CSTR already incurs very few failures with LN + Prior. It also improves mean standalone learner return on all four tasks, with no observed standalone failures for the full method.

When extending the ablation study, we found that the full method with a twin-critic arbitrator matched or reduced mean training-failure counts compared with the ensemble across all four tasks. We therefore use the twin-critic version as the main method. This reduces computational requirements and the number of hyperparameters to tune. We have moved the ensemble comparison and its settings to Appendix G.1, Tables 8 and 9.

2. Broader comparisons. We have added CORE-RL, BCF and RPL to the evaluation. CORE-RL blends baseline and learner actions through control regularization. We report tuned fixed weights in the main comparison and evaluate its adaptive TD-error variant in Appendix G.2, Table 10. BCF fuses baseline and learner action distributions according to their relative uncertainty. RPL adds a learned correction to the baseline action.

AGH offers the strongest overall balance between training safety and final performance among the compared methods (Table 1 of the revised manuscript). It achieves the lowest training-failure count on every task, including ties, and the highest final-policy return on 3/4 tasks. RPL and CORE-RL achieve higher return on Quadrotor2D.

Table 1. revised manuscript (summary) Training and final-policy results, mean ± standard deviation over ten seeds. The final-policy columns evaluate the deployed policy over twelve episodes.

TaskMethodTraining failuresFinal-policy returnFinal-policy failures
CartPoleRL-AR127.7±82.4-1.839±1.4760.0±0.0
RPL42.5±51.7-0.656±0.8330.0±0.0
CORE-RL27.9±21.8-1.966±0.2200.0±0.0
BCF599.7±112.6-120.915±101.8136.5±3.1
AGH (ours)1.3±4.1-0.322±0.1930.0±0.0
CSTRRL-AR184.4±159.4-1.691±1.8400.0±0.0
RPL67.4±59.8-0.973±1.8430.0±0.0
CORE-RL307.0±373.1-2.543±3.9290.0±0.0
BCF451.2±91.7-0.597±0.0850.0±0.0
AGH (ours)0.2±0.6-0.173±0.1350.0±0.0
BiGlucoseRL-AR202.6±179.1-64.164±186.2431.2±3.8
RPL31.0±26.1-12.142±7.4510.0±0.0
CORE-RL0.0±0.0-4.821±0.3610.0±0.0
BCF237.0±158.8-677.674±588.9467.4±5.6
AGH (ours)0.0±0.0-3.127±0.8790.0±0.0
Quadrotor2DRL-AR561.1±646.9-0.909±0.9596.4±5.7
RPL24.1±19.3-0.034±0.0040.0±0.0
CORE-RL0.0±0.0-0.038±0.0010.0±0.0
BCF26.6±10.7-0.284±0.0200.0±0.0
AGH (ours)0.0±0.0-0.129±0.0490.0±0.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.