- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Residual-Budget Guide Arbitration (RBGA), a safe offline-to-online RL framework for dexterous manipulation. The key idea is to explicitly track the remaining safety budget within each episode when using an offline policy to guide safe online learning. RBGA uses a gradually decreasing guidance horizon, tracks the remaining episode cost budget, and uses a learned cost-to-go estimator to evaluate candidate actions from both policies. A step-wise arbitration mechanism selects actions based on their predicted costs, the remaining budget, and a relative cost margin. Experiments on dexterous manipulation tasks and a real-world cube-grasping demonstration show that RBGA can improve online policy learning while maintaining safety performance compared with several O2O RL baselines.
Critical strengths
-
Reasonable and pragmatic safety-aware arbitration mechanism. The paper addresses an important issue in O2O safe RL by explicitly considering the remaining episode safety budget. The proposed action arbitration mechanism is intuitively reasonable and can be integrated into existing methods easily. It can well address the safety concerns in applications.
-
Direct empirical evaluation with real-world experiments. RBGA is evaluated on dexterous manipulation tasks against multiple baselines, showing promising reward, cost, and success rate. The ablation provides (somewhat) evidence supporting the effectiveness of the residual-budget arbitration method. The real-world experiment is a plus.
-
Good compatibility with existing safe RL methods. RBGA is compatible with both SAC-Lag and DDPG-Lag, which is useful for practical applications.
Critical weaknesses
-
Limited methodological novelty, although a good engineering framework. RBGA combines several existing techniques, including guide horizon scheduling from JSRL, action arbitration from PEX, and budget-aware decision-making from standard constrained/safe RL. This study does not introduce a new learning or algorithmic principle.
-
Redundant theory and some flaws. The satisfaction of the safety constraint is not always valid in the current formulation. Also, the derived cost bound does not consider the benefits of residual-budget filtering. Additionally, some propositions have weak connections with the central claim of the proposed method. Details are provided in the questions below.
Questions for authors
This paper addresses an interesting issue in the safe RL application. The paper is well organized and easy to follow. In general, the motivation and the proposed method are both intuitively reasonable. However, I have the following concerns.
Q1. I am curious about the Eq.(10) in proposition 4.3. The result bounds the additional violation of the expected cost constraint, but the formula does not even include the residual budget . This proposition seems naive; so what is its role in this paper? Please explain.
Q2. And why do we need Proposition 4.5? The reward performance bound follows from the performance difference lemma about state coverage and the optimal policy's remaining advantage. However, these assumptions do not appear to have connections with the proposed residual-budget arbitration or guide horizon scheduling. Does RBGA indeed improve the coverage coefficient or reduce the advantage bounds compared with guide-based baselines? Is there any empirical evidence about that? Otherwise, what additional information does Proposition 4.5 provide? This is confusing for the readers.
Q3. I think that there is an inconsistency between the remaining budget and the remaining budget bound on the cost Q-value. Eqs. (4) and (5) are misleading. Although Eq.(6) addresses this re-scaling coefficient correctly, I suggest integrating it into Eq.(4). Otherwise, the readers have a wrong impression in the beginning.
Q4. Is the cost-to-go estimator indeed consistent with the finite-horizon theoretical formulation? The cost-to-go depends on the remaining episode horizon. However, the auxiliary cost estimator does not have an explicit time input. Particularly, how can the time-independent critic estimate finite-horizon cost-to-go with ? Please explain.
Q5. About the experimental results: I am curious about how safety budgets are consumed throughout the episodes. I think the consumption curve can provide more direct evidence about the effectiveness of the proposed remaining budget trackig mechanism. If the authors have the results at hand, please add.
10 comments
Answer to Reviewer WZJT [part 2]
[Q2] The role of Proposition 4.5 and its connection to RBGA.
We thank the reviewer for this question.
Proposition 4.5 connects the state distributions induced by RBGA roll-ins to the reward performance of the learned online policy. Here, is the state distribution induced by the complete RBGA selection rule. The residual budget and guide horizon scheduling affect this distribution through action selection. The state coverage condition therefore applies to the states visited under residual-budget arbitration, while the policy advantage condition measures the remaining positive reward advantage of the optimal feasible policy under this distribution.
Under Assumption 4.4, Proposition 4.5 applies the performance difference lemma to the state coverage and policy advantage conditions to obtain the reward gap bound in Eq. (14): $ J_R(\pi^\star)-J_R(\pi_o)\le D_\infty\sum_{t=0}^{H-1}\gamma^t\varepsilon_{R,t}.$ The coverage coefficient and reward advantage bounds are the quantities used in this conditional performance analysis. Directly comparing and requires the state distribution of the optimal feasible policy and its reward advantage relative to the online policy. These quantities are not directly available in the continuous, high-dimensional dexterous manipulation tasks considered here. We therefore evaluate overall learning performance using reward and success rate. As shown in Figure 1 in the main text, RBGA achieves strong reward and success rate performance across the four tasks while maintaining episode costs close to the cost limit.
The cost analysis characterizes the additional cost of offline guidance. Proposition 4.5 complements this analysis by specifying the state coverage and policy advantage conditions under which the online policy has a bounded reward gap. Once the maximum guide horizon decreases to zero, the composite policy becomes the online policy. The same reward bound then applies to the online policy acting alone, provided that Assumption 4.4 holds.
[Q3] Consistent discount scaling for the remaining budget and cost-to-go.
Thank you for this valuable suggestion.
We have incorporated the discount scaling directly into Eq. (4) in Section 4.1 of the main text, defining the remaining budget as: $ b_t=\frac{d-\sum_{j=0}^{t-1}\gamma^j c_j}{\gamma^t}, b_0=d. $ With this definition, the remaining budget and the cost-to-go in Eq. (5) use the same discount scale and can be compared directly in Eq. (6). We have also updated the notation for the candidate cost estimates, the budget update in Algorithm 1, and the corresponding expressions in Section 4.2 and Appendices C.2 and C.3. This reformulation is equivalent to the original formulation and does not change action selection or the theoretical cost bound.
[Q4] Consistency between the cost-to-go estimator and the finite-horizon formulation.
Thank you for your feedback.
Eq. (5) defines the cost-to-go from the current time step to the end of the episode, whose full notation is . Section 4.1 (page 5, lines 235-237) states that the time index on finite-horizon value and advantage functions is omitted for notational simplicity when the corresponding episode step is clear from context. Thus, the quantity defined in Eq. (5) remains dependent on the remaining episode horizon.
The auxiliary network uses state and action as inputs to approximate this time-dependent quantity. It is trained using the TD target in Eq. (19) of Appendix A.4: $ y^{\mathrm{aux}}=c+\gamma(1-\delta)\widehat Q_C^{\pi_o}(s',a'), a'\sim\pi_o(\cdot\mid s'). $ Here, is the episode termination indicator. At a terminal transition, the target contains only the current cost, consistent with the terminal condition in Appendix C.1. Therefore, in our experiments corresponds to the undiscounted cumulative cost until episode termination, . This quantity is well-defined for the finite episode horizon.
Assumption 4.1 characterizes the prediction error relative to the true finite-horizon cost-to-go at the corresponding time step. For a fixed online policy and cost estimator, it assumes that both candidate actions satisfy the following bound at each step : $ \left|\widehat Q_C^{\pi_o}(s_t,a)-Q_{C,t}^{\pi_o}(s_t,a)\right|\le\varepsilon_C, a\in\{a_t^g,a_t^o\}. $ The subsequent cost bounds are derived under this assumption and explicitly account for the effect of prediction error through .
Answer to Reviewer WZJT [part 3]
[Q5] Budget consumption within an episode.
Thank you for your suggestion. We have added within-episode budget consumption results in Appendix B.4 (Figure 6) of the revised manuscript. We evaluate policies at the 10k, 20k, and 30k training stages on the four tasks, using three training seeds and 100 complete episodes per condition and seed. We compare three methods. Specifically, Residual budget uses the remaining episode budget for action arbitration, Fixed budget uses a constant threshold of 30 instead of the remaining budget, and No arbitration directly executes the guide action within the guide window. All other settings follow the main experiments. Residual-budget arbitration retains more budget at later episode steps and reduces budget overshoot compared with the other two methods. Please refer to Appendix B.4 of the revised manuscript for details.
Answer to Reviewer WZJT [part 1]
We are grateful to the reviewer for the thoughtful comments and constructive suggestions.
[W1] Methodological novelty of RBGA.
Thank you for reviewing our work. We emphasize that, to the best of our knowledge, we are the first to introduce trajectory-level budget awareness into O2O safe RL and the first to apply O2O safe RL to dexterous manipulation. JSRL controls the guide horizon, while PEX selects between offline and online actions according to estimated values. Our contribution is to use the remaining episode cost budget to determine when to retain offline guidance during online learning. RBGA compares the estimated cost-to-go of the guide and online actions against the remaining budget. Thus, action selection accounts for both the cost already incurred and the estimated future cost within the same episode. This allows RBGA to retain useful offline guidance while limiting its additional cost.
[W2]/[Q1] Safety constraints and the role of residual-budget arbitration in the cost bound.
Thank you for your comments.
1. Satisfaction of the safety constraint. In Appendix C.3, Proposition C.1 proves a sufficient condition for RBGA to satisfy the expected cumulative cost constraint, including policy updates within an episode. In particular, Eq. (36) guarantees that the expected episode cost does not exceed the cost limit d when its right-hand side is at most d. The main experiments in Figure 1 show RBGA reducing episode costs toward the cost limit while improving task performance, and the quantitative results in Table 10 above in this response show lower violation rates and higher success rates at most evaluated training stages. The additional manipulation tasks in Appendix B.3 and the real-world deployment in Appendix B.5 further support the safety performance and practical applicability of RBGA across different settings.
2. The role of residual-budget arbitration in the cost bound. As you noted, accounting for the residual budget bt in the cost bound is important. We explicitly include bt in the derivation. The constraint imposed by the residual budget on action selection is given in Eq. (29) in Appendix C.2, while Eq. (10) gives the episode cost bound after further bounding this term. Specifically, Eq. (29) bounds the difference in cost-to-go between the selected guide action and the online action: \gamma^t\left[Q_C^{\pi_o}(s_t,a_t^g)-Q_C^{\pi_o}(s_t,a_t^o)\right]\le\gamma^t\left(2\varepsilon_C+\min\left\\{\zeta,\\,[b_t-\widehat q_t^o]_+\right\\}\right). The additional cost of the guide action is thus bounded using both the comparison margin and the budget remaining after accounting for the online candidate's estimated discounted cost. To obtain an upper bound for the entire episode, the derivation uses \min\left\\{\zeta,\\,[b_t-\widehat q_t^o]_+\right\\}\le\zeta. Therefore, bt no longer appears explicitly in Eq. (10).
Proposition 4.3 bounds the additional violation of the expected cost constraint under the composite policy π relative to the online policy πo. The final term in Eq. (10) shows the effects of the comparison margin ζ, cost estimation error εC, and maximum guide horizon Hi. Under the conditions of the proposition, with other factors fixed, this term does not increase as Hi is reduced and equals zero when Hi=0 and the guide is removed. This provides a quantitative explanation for gradually reducing the maximum guide horizon to limit the additional cost of offline guidance.