acceptodds
Under review as a conference paper at ICLR 2027

RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation

Abstract

Dexterous manipulation requires robots to leverage existing datasets for rapid skill acquisition while maintaining safety during real-world deployment to prevent damage to the robot or its surroundings. Offline-to-online safe reinforcement learning (O2O safe RL) provides a promising solution for this problem by leveraging prior experience for online improvement under safety constraints, thereby mitigating the sample inefficiency and exploration risk of learning safe policies from scratch. However, existing O2O safe RL approaches do not explicitly account for the safety budget already consumed when making decisions within an episode. Excessive cost incurred early can therefore leave little room for subsequent exploration and policy improvement. To address this limitation, we propose Residual-Budget Guide Arbitration (RBGA), an algorithm that explicitly tracks the remaining episode safety budget and uses it to adaptively select between offline guidance and the evolving online policy. At each step, RBGA compares the cost-to-go estimates of the guide and online actions against the remaining budget, enabling step-wise action arbitration throughout the episode. We provide theoretical analysis characterizing the safety implications of residual-budget-based selection and its effect on guide-assisted exploration. Extensive experiments demonstrate that RBGA enables effective online policy improvement while maintaining strong safety performance.

Primary area:
reinforcement learning
Keywords:
Dexterous manipulation, Offline2Online safe reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WZJTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Residual-Budget Guide Arbitration (RBGA), a safe offline-to-online RL framework for dexterous manipulation. The key idea is to explicitly track the remaining safety budget within each episode when using an offline policy to guide safe online learning. RBGA uses a gradually decreasing guidance horizon, tracks the remaining episode cost budget, and uses a learned cost-to-go estimator to evaluate candidate actions from both policies. A step-wise arbitration mechanism selects actions based on their predicted costs, the remaining budget, and a relative cost margin. Experiments on dexterous manipulation tasks and a real-world cube-grasping demonstration show that RBGA can improve online policy learning while maintaining safety performance compared with several O2O RL baselines.

Critical strengths
  1. Reasonable and pragmatic safety-aware arbitration mechanism. The paper addresses an important issue in O2O safe RL by explicitly considering the remaining episode safety budget. The proposed action arbitration mechanism is intuitively reasonable and can be integrated into existing methods easily. It can well address the safety concerns in applications.

  2. Direct empirical evaluation with real-world experiments. RBGA is evaluated on dexterous manipulation tasks against multiple baselines, showing promising reward, cost, and success rate. The ablation provides (somewhat) evidence supporting the effectiveness of the residual-budget arbitration method. The real-world experiment is a plus.

  3. Good compatibility with existing safe RL methods. RBGA is compatible with both SAC-Lag and DDPG-Lag, which is useful for practical applications.

Critical weaknesses
  1. Limited methodological novelty, although a good engineering framework. RBGA combines several existing techniques, including guide horizon scheduling from JSRL, action arbitration from PEX, and budget-aware decision-making from standard constrained/safe RL. This study does not introduce a new learning or algorithmic principle.

  2. Redundant theory and some flaws. The satisfaction of the safety constraint is not always valid in the current formulation. Also, the derived cost bound does not consider the benefits of residual-budget filtering. Additionally, some propositions have weak connections with the central claim of the proposed method. Details are provided in the questions below.

Questions for authors

This paper addresses an interesting issue in the safe RL application. The paper is well organized and easy to follow. In general, the motivation and the proposed method are both intuitively reasonable. However, I have the following concerns.

Q1. I am curious about the Eq.(10) in proposition 4.3. The result bounds the additional violation of the expected cost constraint, but the formula does not even include the residual budget btb_t. This proposition seems naive; so what is its role in this paper? Please explain.

Q2. And why do we need Proposition 4.5? The reward performance bound follows from the performance difference lemma about state coverage and the optimal policy's remaining advantage. However, these assumptions do not appear to have connections with the proposed residual-budget arbitration or guide horizon scheduling. Does RBGA indeed improve the coverage coefficient or reduce the advantage bounds compared with guide-based baselines? Is there any empirical evidence about that? Otherwise, what additional information does Proposition 4.5 provide? This is confusing for the readers.

Q3. I think that there is an inconsistency between the remaining budget dd and the remaining budget bound on the cost Q-value. Eqs. (4) and (5) are misleading. Although Eq.(6) addresses this re-scaling coefficient correctly, I suggest integrating it into Eq.(4). Otherwise, the readers have a wrong impression in the beginning.

Q4. Is the cost-to-go estimator indeed consistent with the finite-horizon theoretical formulation? The cost-to-go depends on the remaining episode horizon. However, the auxiliary cost estimator does not have an explicit time input. Particularly, how can the time-independent critic estimate finite-horizon cost-to-go with γ=1\gamma = 1? Please explain.

Q5. About the experimental results: I am curious about how safety budgets are consumed throughout the episodes. I think the consumption curve can provide more direct evidence about the effectiveness of the proposed remaining budget trackig mechanism. If the authors have the results at hand, please add.

10 comments
AuthorsOpenReview ↗

Answer to Reviewer WZJT [part 1]

We are grateful to the reviewer for the thoughtful comments and constructive suggestions.

[W1] Methodological novelty of RBGA.

Thank you for reviewing our work. We emphasize that, to the best of our knowledge, we are the first to introduce trajectory-level budget awareness into O2O safe RL and the first to apply O2O safe RL to dexterous manipulation. JSRL controls the guide horizon, while PEX selects between offline and online actions according to estimated values. Our contribution is to use the remaining episode cost budget to determine when to retain offline guidance during online learning. RBGA compares the estimated cost-to-go of the guide and online actions against the remaining budget. Thus, action selection accounts for both the cost already incurred and the estimated future cost within the same episode. This allows RBGA to retain useful offline guidance while limiting its additional cost.

[W2]/[Q1] Safety constraints and the role of residual-budget arbitration in the cost bound.

Thank you for your comments.

1. Satisfaction of the safety constraint. In Appendix C.3, Proposition C.1 proves a sufficient condition for RBGA to satisfy the expected cumulative cost constraint, including policy updates within an episode. In particular, Eq. (36) guarantees that the expected episode cost does not exceed the cost limit dd when its right-hand side is at most dd. The main experiments in Figure 1 show RBGA reducing episode costs toward the cost limit while improving task performance, and the quantitative results in Table 10 above in this response show lower violation rates and higher success rates at most evaluated training stages. The additional manipulation tasks in Appendix B.3 and the real-world deployment in Appendix B.5 further support the safety performance and practical applicability of RBGA across different settings.

2. The role of residual-budget arbitration in the cost bound. As you noted, accounting for the residual budget btb_t in the cost bound is important. We explicitly include btb_t in the derivation. The constraint imposed by the residual budget on action selection is given in Eq. (29) in Appendix C.2, while Eq. (10) gives the episode cost bound after further bounding this term. Specifically, Eq. (29) bounds the difference in cost-to-go between the selected guide action and the online action: \gamma^t\left[Q_C^{\pi_o}(s_t,a_t^g)-Q_C^{\pi_o}(s_t,a_t^o)\right]\le\gamma^t\left(2\varepsilon_C+\min\left\\{\zeta,\\,[b_t-\widehat q_t^o]_+\right\\}\right). The additional cost of the guide action is thus bounded using both the comparison margin and the budget remaining after accounting for the online candidate's estimated discounted cost. To obtain an upper bound for the entire episode, the derivation uses \min\left\\{\zeta,\\,[b_t-\widehat q_t^o]_+\right\\}\le\zeta. Therefore, btb_t no longer appears explicitly in Eq. (10).

Proposition 4.3 bounds the additional violation of the expected cost constraint under the composite policy π\pi relative to the online policy πo\pi_o. The final term in Eq. (10) shows the effects of the comparison margin ζ\zeta, cost estimation error εC\varepsilon_C, and maximum guide horizon HiH_i. Under the conditions of the proposition, with other factors fixed, this term does not increase as HiH_i is reduced and equals zero when Hi=0H_i=0 and the guide is removed. This provides a quantitative explanation for gradually reducing the maximum guide horizon to limit the additional cost of offline guidance.

AuthorsOpenReview ↗

Answer to Reviewer WZJT [part 2]

[Q2] The role of Proposition 4.5 and its connection to RBGA.

We thank the reviewer for this question.

Proposition 4.5 connects the state distributions induced by RBGA roll-ins to the reward performance of the learned online policy. Here, μt\mu_t is the state distribution induced by the complete RBGA selection rule. The residual budget and guide horizon scheduling affect this distribution through action selection. The state coverage condition therefore applies to the states visited under residual-budget arbitration, while the policy advantage condition measures the remaining positive reward advantage of the optimal feasible policy under this distribution.

Under Assumption 4.4, Proposition 4.5 applies the performance difference lemma to the state coverage and policy advantage conditions to obtain the reward gap bound in Eq. (14): $ J_R(\pi^\star)-J_R(\pi_o)\le D_\infty\sum_{t=0}^{H-1}\gamma^t\varepsilon_{R,t}.$ The coverage coefficient D∞D_\infty and reward advantage bounds εR,t\varepsilon_{R,t} are the quantities used in this conditional performance analysis. Directly comparing D∞D_\infty and εR,t\varepsilon_{R,t} requires the state distribution of the optimal feasible policy π⋆\pi^\star and its reward advantage relative to the online policy. These quantities are not directly available in the continuous, high-dimensional dexterous manipulation tasks considered here. We therefore evaluate overall learning performance using reward and success rate. As shown in Figure 1 in the main text, RBGA achieves strong reward and success rate performance across the four tasks while maintaining episode costs close to the cost limit.

The cost analysis characterizes the additional cost of offline guidance. Proposition 4.5 complements this analysis by specifying the state coverage and policy advantage conditions under which the online policy has a bounded reward gap. Once the maximum guide horizon decreases to zero, the composite policy becomes the online policy. The same reward bound then applies to the online policy acting alone, provided that Assumption 4.4 holds.

[Q3] Consistent discount scaling for the remaining budget and cost-to-go.

Thank you for this valuable suggestion.

We have incorporated the discount scaling directly into Eq. (4) in Section 4.1 of the main text, defining the remaining budget as: $ b_t=\frac{d-\sum_{j=0}^{t-1}\gamma^j c_j}{\gamma^t}, b_0=d. $ With this definition, the remaining budget and the cost-to-go in Eq. (5) use the same discount scale and can be compared directly in Eq. (6). We have also updated the notation for the candidate cost estimates, the budget update in Algorithm 1, and the corresponding expressions in Section 4.2 and Appendices C.2 and C.3. This reformulation is equivalent to the original formulation and does not change action selection or the theoretical cost bound.

[Q4] Consistency between the cost-to-go estimator and the finite-horizon formulation.

Thank you for your feedback.

Eq. (5) defines the cost-to-go from the current time step to the end of the episode, whose full notation is QC,tπo(s,a)Q_{C,t}^{\pi_o}(s,a). Section 4.1 (page 5, lines 235-237) states that the time index on finite-horizon value and advantage functions is omitted for notational simplicity when the corresponding episode step is clear from context. Thus, the quantity defined in Eq. (5) remains dependent on the remaining episode horizon.

The auxiliary network uses state and action as inputs to approximate this time-dependent quantity. It is trained using the TD target in Eq. (19) of Appendix A.4: $ y^{\mathrm{aux}}=c+\gamma(1-\delta)\widehat Q_C^{\pi_o}(s',a'), a'\sim\pi_o(\cdot\mid s'). $ Here, δ\delta is the episode termination indicator. At a terminal transition, the target contains only the current cost, consistent with the terminal condition VCπo(sH)=0V_C^{\pi_o}(s_H)=0 in Appendix C.1. Therefore, γ=1\gamma=1 in our experiments corresponds to the undiscounted cumulative cost until episode termination, QC,tπo(st,at)=Eπo[∑j=tH−1cj,|,st,at]Q_{C,t}^{\pi_o}(s_t,a_t)=\mathbb E_{\pi_o}\left[\sum_{j=t}^{H-1}c_j\\,\middle|\\,s_t,a_t\right]. This quantity is well-defined for the finite episode horizon.

Assumption 4.1 characterizes the prediction error relative to the true finite-horizon cost-to-go at the corresponding time step. For a fixed online policy and cost estimator, it assumes that both candidate actions satisfy the following bound at each step t<min⁡h,Hit < \min\\{h,H_i\\}: $ \left|\widehat Q_C^{\pi_o}(s_t,a)-Q_{C,t}^{\pi_o}(s_t,a)\right|\le\varepsilon_C, a\in\{a_t^g,a_t^o\}. $ The subsequent cost bounds are derived under this assumption and explicitly account for the effect of prediction error through εC\varepsilon_C.

AuthorsOpenReview ↗

Answer to Reviewer WZJT [part 3]

[Q5] Budget consumption within an episode.

Thank you for your suggestion. We have added within-episode budget consumption results in Appendix B.4 (Figure 6) of the revised manuscript. We evaluate policies at the 10k, 20k, and 30k training stages on the four tasks, using three training seeds and 100 complete episodes per condition and seed. We compare three methods. Specifically, Residual budget uses the remaining episode budget for action arbitration, Fixed budget uses a constant threshold of 30 instead of the remaining budget, and No arbitration directly executes the guide action within the guide window. All other settings follow the main experiments. Residual-budget arbitration retains more budget at later episode steps and reduces budget overshoot compared with the other two methods. Please refer to Appendix B.4 of the revised manuscript for details.

Reviewer_afoCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses offline-to-online safe reinforcement learning through Residual-Budget Guide Arbitration (RBGA). The method first learns a guide policy from offline data, then uses a gradually decreasing guide horizon to limit its participation during online training. Within the guidance period, RBGA samples actions from both the guide and online policies and selects between them based on estimated cost-to-go and the remaining episode cost budget. The online policy is updated using the resulting experience collected in a replay buffer. Experimental results demonstrate improvements over the evaluated baselines on the studied tasks.

Critical strengths
  • The paper provides theoretical analysis to support the proposed method.

  • The experiments demonstrate improvements over the evaluated baselines on the studied tasks.

Critical weaknesses
  • There is no clear guarantee that the learned policy satisfies the cumulative cost constraint, particularly the CMDP constraint, which is important in safe RL.

  • The method is largely heuristic, with many components relying on heuristic designs, such as the residual budget and step-wise arbitration. This makes it difficult to extend the method to other applications.

  • The main idea is not particularly novel, as many previous works have studied reusing offline demonstration data to facilitate online safe RL. For example, Recovery RL [1] also uses a recovery policy to generate backup actions.

  • The method lacks comparisons with related or state-of-the-art approaches and is primarily compared with Lagrangian variants of standard RL methods. Furthermore, the environments used for evaluation are not standard safe RL environments.

[1] Thananjeyan, Brijen, et al. "Recovery rl: Safe reinforcement learning with learned recovery zones." IEEE Robotics and Automation Letters 6.3 (2021): 4915-4922.

Questions for authors

Q1. How are the offline demonstration data collected? Are there any specific requirements for these data?

Q2. Could you compare the method with existing O2O safe RL methods or evaluate it on standard safe RL environments, such as Safety-Gymnasium?

Q3. Can the online policy learn efficiently from the states and actions provided by the guide alone? Why not use the guide policy as a prior for the online policy?

8 comments
AuthorsOpenReview ↗

Answer to Reviewer afoC [part 1]

We sincerely thank the reviewer for these helpful comments.

[W1] Clarify the safety guarantee for the learned policy under the CMDP cumulative cost constraint.

We appreciate the reviewer's feedback. We formulate our optimization constraint within the constrained Markov decision process (CMDP) framework, following established safe RL work such as Constrained Policy Optimization (CPO) [1] and Recovery RL [2]. CPO constrains the expected discounted cumulative cost, while Recovery RL adopts a CMDP formulation with binary constraint costs. Our safety constraint follows the same modeling principle: maximize expected return subject to an upper bound on expected cumulative cost. We incorporate this constraint directly into online policy learning through SAC-Lag. A cost critic and an adaptive Lagrange multiplier are used to optimize task reward while constraining cumulative cost. Residual-budget arbitration further addresses safety when the offline guide participates. Under the assumptions of Proposition 4.3, the additional expected cost violation introduced by RBGA relative to the online policy has an explicit upper bound determined by the cost-estimation error, comparison margin, and guide horizon. With the other terms fixed, reducing the guide horizon does not increase this additional bound. This term vanishes when the guide is fully removed, after which the online policy executes independently. Together, SAC-Lag incorporates the cumulative cost constraint into online policy learning, while residual-budget arbitration controls the additional expected cost violation introduced by offline guidance. We also evaluate the final online policy independently using three training seeds and 100 episodes per seed. Table 8 reports mean episode cost, which can be compared with the cost limit, together with the episode violation rate. These measurements assess the learned policy's constraint satisfaction in practice and quantify how often individual episodes exceed the budget.

[1] Achiam, Joshua, et al. "Constrained policy optimization." International conference on machine learning. Pmlr, 2017.

[2] Thananjeyan, Brijen, et al. "Recovery rl: Safe reinforcement learning with learned recovery zones." IEEE Robotics and Automation Letters 6.3 (2021): 4915-4922.

AuthorsOpenReview ↗

Answer to Reviewer afoC [part 2]

Table 8. Final policy performance and episode violation rates across four tasks.

Results are averaged across three training seeds, with each seed's policy at the final training stage evaluated over 100 episodes. A violation is defined as an episode cost exceeding 30.

TaskMethodRewardMean episode costSuccess rate (%)Violation rate (%)
FingerRBGA19.10 ± 1.3830.07 ± 0.0575.67 ± 9.5021.33 ± 3.61
FingerJSRL-Random-Lag18.74 ± 0.0430.65 ± 0.0249.33 ± 1.6429.00 ± 1.53
FingerJSRL-Curr-Lag18.65 ± 10.9429.92 ± 0.0963.33 ± 381.0134.00 ± 14.18
FingerPEX-Lag9.27 ± 0.0129.92 ± 0.0113.33 ± 0.2610.67 ± 4.36
FingerSO2-Lag8.64 ± 24.4444.07 ± 929.4511.67 ± 397.1372.33 ± 19.63
FingerDirect6.51 ± 2.5730.39 ± 0.370.00 ± 0.0041.00 ± 23.16
FingerScratch6.47 ± 12.3131.02 ± 3.0811.00 ± 368.9336.33 ± 27.73
JointRBGA118.68 ± 16.6629.67 ± 1.1581.67 ± 117.6727.67 ± 17.04
JointJSRL-Random-Lag103.84 ± 1.7031.49 ± 0.0252.33 ± 0.6234.33 ± 2.65
JointJSRL-Curr-Lag105.39 ± 0.6629.66 ± 0.0363.00 ± 204.4353.67 ± 2.65
JointPEX-Lag110.55 ± 14.3837.32 ± 6.7256.67 ± 202.9439.67 ± 12.50
JointSO2-Lag115.42 ± 4.8429.68 ± 2.1481.00 ± 33.5820.00 ± 12.66
JointDirect37.24 ± 2034.1427.49 ± 4.9017.33 ± 901.3323.33 ± 16.50
JointScratch33.71 ± 23.4729.96 ± 0.252.67 ± 6.5248.33 ± 7.23
ArmFingerRBGA21.53 ± 3.3729.51 ± 0.1482.33 ± 10.0230.33 ± 26.69
ArmFingerJSRL-Random-Lag14.20 ± 0.2430.60 ± 0.0134.67 ± 3.8331.00 ± 0.58
ArmFingerJSRL-Curr-Lag20.51 ± 1.3429.93 ± 0.0574.33 ± 20.1626.00 ± 6.56
ArmFingerPEX-Lag16.85 ± 0.2233.94 ± 0.0147.00 ± 15.5328.00 ± 9.54
ArmFingerSO2-Lag20.29 ± 1.1325.67 ± 1.8778.00 ± 29.0241.67 ± 42.67
ArmFingerDirect4.40 ± 48.7418.24 ± 116.776.67 ± 133.3322.00 ± 35.22
ArmFingerScratch0.60 ± 0.0130.18 ± 0.070.00 ± 0.0028.00 ± 36.10
ArmJointRBGA128.84 ± 17.1929.22 ± 0.0184.00 ± 10.0322.33 ± 22.14
ArmJointJSRL-Random-Lag102.08 ± 54.1338.84 ± 36.4533.67 ± 195.4129.67 ± 13.58
ArmJointJSRL-Curr-Lag0.62 ± 0.0857.40 ± 3.660.00 ± 0.00100.00 ± 0.00
ArmJointPEX-Lag123.34 ± 28.9630.12 ± 1.2975.00 ± 78.7210.33 ± 12.34
ArmJointSO2-Lag125.25 ± 95.7829.81 ± 60.9680.33 ± 20.284.33 ± 11.24
ArmJointDirect25.45 ± 501.0829.23 ± 0.815.33 ± 80.9734.67 ± 22.85
ArmJointScratch22.65 ± 208.9830.13 ± 0.224.67 ± 65.3329.00 ± 11.27
AuthorsOpenReview ↗

Answer to Reviewer afoC [part 3]

[W2] Heuristic design and extension to other applications.

We are grateful for the reviewer's comments. We emphasize that RBGA is not a heuristic method. At a fixed training stage, RBGA's arbitration can be represented as a Markov decision rule on an augmented state consisting of the environment state, remaining budget, current time step, and sampled guide horizon. The remaining budget summarizes the past cost relevant to the constraint, so arbitration does not require the full trajectory history. The key designs are motivated by the cumulative cost constraint and do not rely on switching rules specific to a particular task. Specifically, the remaining budget is computed by subtracting the cumulative cost already incurred from the total budget, allowing actual budget consumption to inform action selection. At each step, arbitration combines this budget with estimates of the candidate actions' future cumulative costs to decide whether to retain offline guidance or execute the online action. These two components therefore determine how the guide policy is used under a cumulative cost constraint, without manually specifying when to switch at different task stages. Although arbitration is performed at each step, it considers future cumulative cost rather than only the immediate cost of the current action. For application to other tasks, RBGA's arbitration rule depends on the cost signal, remaining budget, and cost estimates of candidate actions. For a new task with the same form of cumulative cost constraint, one can define the corresponding cost signal and budget, while retaining the same budget update and action arbitration rules. The switching logic therefore does not need to be redesigned for each task.

[W3] Novelty and differences from Recovery RL.

We thank the reviewer for these valuable comments. We acknowledge that using offline experience to support safe online learning has been studied in prior work. We emphasize that, to the best of our knowledge, we are the first to introduce trajectory-level budget awareness into O2O safe RL and to apply O2O safe RL to dexterous manipulation. RBGA differs substantially from Recovery RL in the following respects.

  1. Different roles of the offline policies. Recovery RL uses offline data containing constraint violations to learn a safety critic and a recovery policy, which aims to reduce the risk of future constraint violations. RBGA reuses the task knowledge of an offline guide policy to support online exploration. The guide policy is not required to satisfy the safety constraints of the online phase in advance. Whether its action is executed is determined by residual-budget arbitration.
  2. Different criteria for action selection. Recovery RL compares the future violation risk estimated by its safety critic with a predefined threshold to determine whether to activate the recovery policy. RBGA explicitly tracks the cumulative cost already incurred in the current episode and updates the remaining budget accordingly. It then compares the estimated future cumulative costs of the guide and online actions with the remaining budget, while also accounting for the cost difference between the two candidates. Thus, even with the same current state and candidate actions, different past cost consumption can lead to different arbitration decisions.
  3. Different online learning objectives and guidance mechanisms. Recovery RL's task policy aims to maximize task reward, while the recovery policy handles safety interventions. RBGA's online policy uses SAC-Lag to maximize task reward subject to the cumulative cost constraint. The guide horizon gradually decreases during training, allowing the online policy to assume more control until it executes the task independently.
Reviewer_NKffOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RBGA for offline-to-online safe RL in dexterous manipulation. The method tracks the remaining episode cost budget and uses estimated cost-to-go to select between an offline guide and an online policy. It also gradually reduces guide participation during training. The paper provides theoretical bounds and evaluates the method on simulated manipulation tasks, with a real-world grasping demonstration in the appendix.

Critical strengths

The motivation makes sense. Accounting for the cost already incurred within an episode is a reasonable way to improve action selection under a safety budget.

The method is relatively simple and shows promising performance compared with the evaluated baselines, balancing task success and average episode cost.

The paper includes ablations, experiments with a second online learner, and additional manipulation tasks. These help support the usefulness of the approach.

Critical weaknesses

I am not fully convinced how significant the residual-budget component is. In Figure 3, the final performance is often similar without it, and the main benefit seems to be earlier cost reduction. Quantifying cumulative safety violations during training would help show how important this improvement is.

The ablation does not clearly establish whether tracking the remaining budget is better than simpler cost-based action selection. A comparison using the same guide schedule and cost estimator, but without budget-dependent selection, would make the contribution clearer.

The evaluation is still limited. The four main tasks correspond to two manipulation settings with different joint constraints. The additional tasks help, but robustness to different budgets, guide quality, and distribution shifts is not systematically studied.

The real-world experiment is a qualitative demonstration of a simulation-trained policy. There are no quantitative safety measurements or baseline comparisons, and it does not evaluate online adaptation on hardware.

The theoretical results have proofs, but their practical implications seem limited. The cost bound depends on the online policy’s safety and an assumed estimation-error bound. The reward bound also relies on coverage and policy-advantage assumptions that the learning procedure is not shown to achieve.

Average episode cost near the threshold does not show how frequently individual episodes violate the budget or how severe those violations are. These measurements would better support the safety claims.

Questions for authors
  1. What exactly is removed in the “without residual budget” ablation in Figure 3? Does it keep the same cost-based action comparison and guide schedule?

  2. How does RBGA compare with a simpler selector that chooses the action with lower estimated cost-to-go, using the same guide schedule? This would help clarify the benefit of tracking the remaining budget.

  3. Can you quantify the reduction in cumulative safety violations during training, including warm-up? Figure 3 suggests the main benefit is earlier cost reduction, but it is hard to judge its magnitude from the curves.

  4. Are the theoretical cost bounds informative under the experimental settings? Can you estimate the cost-prediction error empirically and discuss what safety guarantee the resulting bound provides?

10 comments
AuthorsOpenReview ↗

Answer to Reviewer NKff [part 1]

Thank you for your thoughtful feedback!

[W1]/[Q3] Quantify the reduction in cumulative safety violations during training

We thank the reviewer for raising this question. To quantify the contribution of residual-budget arbitration, we evaluated full RBGA and the variant without this module at different training stages across four tasks. For each task and stage, we used three training seeds and evaluated 100 episodes per seed, reporting mean episode cost, violation rate, and success rate. A violation is defined as an episode cost exceeding the cost limit. The experimental results in the table below show that full RBGA achieves lower violation rates and higher success rates at most evaluated stages. These findings indicate that the benefits of residual-budget arbitration extend beyond early cost reduction to lower violation rates and better task performance of the online policy at later training stages.

Table 1. Comparison of RBGA and RBGA without residual-budget arbitration (w/o RB) across training stages on four tasks.

TaskIterRBGAw/o RB
Reward ↑Cost ↓Violations (%) ↓Success (%) ↑Reward ↑Cost ↓Violations (%) ↓Success (%) ↑
Finger0k13.93 ± 0.0252.97 ± 0.2199.67 ± 0.580.67 ± 0.3314.24 ± 0.2355.72 ± 0.9398.67 ± 1.530.33 ± 0.58
Finger20k15.78 ± 0.1529.83 ± 0.1326.33 ± 11.0664.67 ± 7.0516.02 ± 0.6129.38 ± 0.4843.33 ± 1.5362.33 ± 3.51
Finger40k17.12 ± 2.8629.90 ± 0.0533.00 ± 4.5873.00 ± 36.2417.94 ± 2.4128.67 ± 1.4037.00 ± 12.1772.33 ± 7.37
Finger100k19.10 ± 1.3830.07 ± 0.0521.33 ± 3.6175.67 ± 9.5016.55 ± 5.0231.35 ± 2.1657.00 ± 20.4256.33 ± 25.54
Joint0k76.48 ± 0.0840.93 ± 0.3496.67 ± 2.0818.00 ± 0.8778.83 ± 1.2649.69 ± 0.2496.67 ± 1.531.33 ± 1.53
Joint20k100.91 ± 18.5629.98 ± 0.8440.33 ± 4.9371.67 ± 1.59104.87 ± 2.8729.37 ± 3.4539.33 ± 16.2976.00 ± 12.29
Joint40k100.03 ± 241.5229.89 ± 0.4035.33 ± 9.0277.67 ± 129.15115.47 ± 4.8429.86 ± 4.1337.67 ± 16.2976.67 ± 11.93
Joint100k118.68 ± 16.6629.67 ± 1.1527.67 ± 17.0481.67 ± 117.67109.33 ± 7.3528.50 ± 3.0937.33 ± 8.0867.67 ± 2.89
ArmFinger0k11.09 ± 0.1548.32 ± 1.1577.00 ± 4.361.67 ± 1.4011.06 ± 0.7756.12 ± 1.8191.33 ± 1.530.00 ± 0.00
ArmFinger20k16.25 ± 2.5429.92 ± 0.0335.67 ± 15.9560.00 ± 63.7414.33 ± 1.1629.52 ± 1.7742.00 ± 11.5346.00 ± 7.21
ArmFinger40k17.88 ± 0.6129.88 ± 0.0643.67 ± 23.0369.00 ± 13.2917.80 ± 2.3129.21 ± 0.7645.67 ± 3.7971.67 ± 5.51
ArmFinger100k21.53 ± 3.3729.51 ± 0.1430.33 ± 26.6982.33 ± 10.0216.77 ± 1.4631.44 ± 4.7057.33 ± 23.8055.67 ± 15.31
ArmJoint0k76.13 ± 5.9547.41 ± 0.1194.67 ± 1.5310.33 ± 2.2682.68 ± 1.2362.23 ± 1.2795.00 ± 3.460.00 ± 0.00
ArmJoint20k115.52 ± 12.5930.20 ± 1.0248.67 ± 16.0775.00 ± 13.77111.15 ± 7.9328.47 ± 1.6836.67 ± 6.0371.33 ± 2.52
ArmJoint40k117.03 ± 19.6129.61 ± 0.9739.67 ± 25.1174.00 ± 27.2387.51 ± 21.7430.02 ± 4.9242.67 ± 29.1450.67 ± 21.55
ArmJoint100k128.84 ± 17.1929.22 ± 0.0122.33 ± 22.1484.00 ± 10.03111.56 ± 18.7228.32 ± 1.3633.00 ± 4.0084.33 ± 10.97

[Q1] Clarify the ablation setup: cost-based action comparison and the guide schedule.

We thank the reviewer for the opportunity to clarify the ablation setup. The ablation retains the same guide schedule but removes the budget-aware action arbitration module, including the auxiliary cost critics and cost-based comparison between guide and online actions. It therefore executes the guide action directly within the scheduled guide window and the online action otherwise. The original SAC-Lag cost critics and Lagrangian constraint mechanism remain unchanged. We have also clarified this ablation setup in the revised manuscript. Please see Appendix B.1 for details.

AuthorsOpenReview ↗

Answer to Reviewer NKff [part 2]

[W2]/[Q2] Clarify the benefit of tracking the remaining budget over cost-based action selection.

We thank the reviewer for suggesting this comparison. To address these two comments, we kept the same guide schedule, cost estimator architecture, and training procedure, and replaced budget-dependent selection with a simpler selector (Min-cost) that chooses the candidate action with lower estimated cost-to-go. We compared this selector with RBGA on Joint and Finger. For each task and training stage, we used three training seeds and evaluated 100 episodes per seed, reporting mean reward, cost, and success rate. The experimental results are shown in Table 2 below. Compared with simply selecting the action with lower estimated cost-to-go, our method can achieve better task performance while satisfying the safety constraint. These results suggest that tracking the remaining budget can help the online policy make more effective use of the offline guide, improving learning efficiency and task performance.

Table 2. Comparison of RBGA and Min-cost action selection on Joint and Finger across training stages.

TaskIterationsMethodRewardCostSuccess rate (%)
Joint0RBGA76.48 ± 0.0840.93 ± 0.3418.00 ± 0.87
Joint0Min-cost77.00 ± 6.8349.46 ± 1.810.67 ± 0.58
Joint5000RBGA77.05 ± 2.9330.00 ± 0.4161.33 ± 2.80
Joint5000Min-cost48.86 ± 4.4631.02 ± 1.1719.00 ± 1.51
Joint10000RBGA95.56 ± 0.6429.61 ± 0.3771.33 ± 2.29
Joint10000Min-cost91.90 ± 4.7730.41 ± 0.3259.67 ± 6.24
Joint20000RBGA100.91 ± 18.5629.98 ± 0.8471.67 ± 1.59
Joint20000Min-cost105.30 ± 1.6730.70 ± 0.7469.33 ± 4.31
Joint30000RBGA109.24 ± 3.4130.24 ± 0.4175.00 ± 3.25
Joint30000Min-cost110.01 ± 2.8530.85 ± 1.1871.33 ± 11.92
Joint40000RBGA100.03 ± 241.5229.89 ± 0.4077.67 ± 129.15
Joint40000Min-cost110.29 ± 1.2229.64 ± 0.3965.00 ± 7.85
Joint100000RBGA118.68 ± 16.6629.67 ± 1.1581.67 ± 117.67
Joint100000Min-cost114.12 ± 2.7830.11 ± 0.0867.67 ± 8.61
Finger0RBGA13.93 ± 0.0252.97 ± 0.210.67 ± 0.33
Finger0Min-cost13.61 ± 0.3256.09 ± 0.590.00 ± 0.00
Finger5000RBGA12.29 ± 0.2732.50 ± 0.4943.67 ± 1.57
Finger5000Min-cost4.28 ± 0.3731.75 ± 0.740.67 ± 0.15
Finger10000RBGA14.55 ± 0.4429.78 ± 0.4354.67 ± 3.12
Finger10000Min-cost13.64 ± 0.3130.26 ± 0.2746.00 ± 0.68
Finger20000RBGA15.78 ± 0.1529.83 ± 0.1364.67 ± 7.05
Finger20000Min-cost16.76 ± 0.3030.12 ± 0.3868.00 ± 1.85
Finger30000RBGA17.94 ± 1.0930.03 ± 0.3468.33 ± 5.03
Finger30000Min-cost16.68 ± 1.0529.88 ± 0.1767.67 ± 4.39
Finger40000RBGA17.12 ± 2.8629.90 ± 0.0573.00 ± 36.24
Finger40000Min-cost16.90 ± 1.2529.64 ± 0.6572.67 ± 6.24
Finger100000RBGA19.10 ± 1.3830.07 ± 0.0575.67 ± 9.50
Finger100000Min-cost18.81 ± 1.4729.95 ± 0.2774.67 ± 3.49
AuthorsOpenReview ↗

Answer to Reviewer NKff [part 3]

[W3] Evaluate robustness to different budgets, guide quality, and distribution shifts.

We thank the reviewer for highlighting the need for a more systematic evaluation. We added experiments on Finger and Joint with different cost limits and guide policies, as detailed below.

(1) Different budgets. We evaluated RBGA with cost limits of 20, 30, and 40 on Finger and Joint. Only the cost limit was changed. All other settings were identical to those in the main experiments. The experimental results are shown in Table 3 below. RBGA maintains strong task performance while satisfying the safety constraints. These results indicate that our method remains effective across different cost limits.

Table 3. Robustness to different cost limits.

RBGA performance on Finger and Joint with cost limits of 20, 30, and 40. All other settings are identical to the main experiments. Results are averaged across three training seeds, with each seed's policy at the final training stage evaluated over 100 episodes.

TaskCost limitRewardCostSuccess rate (%)
Finger20.0020.15 ± 0.8519.92 ± 1.8485.67 ± 5.25
Finger30.0019.10 ± 1.3830.07 ± 0.0575.67 ± 9.50
Finger40.0016.73 ± 21.9540.84 ± 8.1177.00 ± 1.74
Joint20.00116.77 ± 37.8119.11 ± 8.8577.33 ± 12.99
Joint30.00118.68 ± 16.6629.67 ± 1.1581.67 ± 117.67
Joint40.00117.47 ± 13.2937.59 ± 10.4268.33 ± 7.01

(2) Guide quality. We evaluated RBGA with three guide policies on Finger and Joint: the original guide policy, guide A, and guide B. The original guide has strong task performance but high cost. Guide A has weaker task performance and high cost, while guide B has strong task performance and approximately satisfies the constraint on mean cost. Only the guide policy was changed. All other settings were identical to those in the main experiments. We used three training seeds and evaluated 100 episodes per seed using each seed's policy at the final training stage. The experimental results are shown in Table 4 below. Across guide policies of different quality, our method maintains strong task performance while satisfying the safety constraints. These results indicate that our method can adapt to guide policies of different quality.

(3) Distribution shifts. Guide policies of different quality induce different state distributions, so the guide-quality experiment also demonstrates our method's robustness to these distribution shifts.

Table 4. Robustness to different guide policies.

Final RBGA performance at 100k iterations with the original guide policy and alternatives A and B. All other settings are identical to the main experiments. Results are averaged across three training seeds, with each seed's policy at the final training stage evaluated over 100 episodes.

TaskIterationsGuide policyRewardCostSuccess rate (%)
Finger100kOriginal19.10 ± 1.3830.07 ± 0.0575.67 ± 9.50
Finger100kA19.28 ± 1.5128.05 ± 1.5673.67 ± 2.52
Finger100kB19.90 ± 0.6830.20 ± 1.2981.00 ± 7.94
Joint100kOriginal118.68 ± 16.6629.67 ± 1.1581.67 ± 117.67
Joint100kA116.22 ± 1.5429.57 ± 4.6863.00 ± 7.94
Joint100kB113.83 ± 2.9032.22 ± 2.5561.67 ± 21.08

[W4] Quantify real-world safety, add baseline comparisons, and assess online adaptation on hardware.

We thank the reviewer for highlighting this problem. We conducted an additional real-world comparison on the Cube Grasping task using the Linker Hand L20 dexterous hand. We trained a Cube Grasping policy with SAC and compared its joint angles during grasping with those of RBGA. The experimental results are shown in Table 5 below. SAC exhibited pronounced joint-angle deviations in the index and middle fingers, particularly at the MCP pitch and PIP joints, whereas RBGA avoided these deviations. These results provide quantitative evidence supporting the safety of our policy on real hardware.

Table 5. Real-world joint-angle comparison between RBGA and SAC.

Joint angles recorded during Cube Grasping on the Linker Hand L20 dexterous hand. Bold SAC values highlight the four joints with the largest differences from RBGA.

JointRBGASAC
thumb_cmc_roll0.440.46
thumb_cmc_yaw0.990.96
thumb_cmc_pitch0.410.44
thumb_mcp0.540.57
index_mcp_roll-0.08-0.08
index_mcp_pitch0.65-0.15
index_pip1.040.20
middle_mcp_roll0.100.10
middle_mcp_pitch0.80-0.15
middle_pip1.050.20
ring_mcp_roll0.120.11
ring_mcp_pitch0.780.81
ring_pip1.071.12
pinky_mcp_roll-0.14-0.12
pinky_mcp_pitch0.840.81
pinky_pip1.071.11

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.