- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SafeOT builds a graph from collected trajectories and solves a constrained flow problem to price safety budgets. It adds these prices to standard Lagrangian updates, reducing training violations in many tests with one or more safety budgets.
Critical strengths
-
SafeOT changes how safety penalties are computed while keeping the underlying policy update largely intact. It can handle several budgets together, and the measured overhead is small in the default setup.
-
The Safety-Gymnasium results show consistent reductions in training violations against TRPO-Lag. Additional seeds, budget settings, controller ablations, and final-policy evaluations make the evidence more convincing. I particularly appreciate the reporting of cases where the method does not help.
Critical weaknesses
It is still unclear how much the flow problem helps. In Table 37, the simple reactive switch has a median violation ratio of 0.91 compared with default SafeOT. This alone does not mean the switch is better overall: Table 27 shows that unclipped SafeOT gets higher returns with similar violations on the robot tasks. I would like to see the switch included in the same return–violation comparison as the main baselines. This would make it easier to judge what the proposed prices add.
Questions for authors
-
When the reactive switch is included in the main return–violation comparison, where does SafeOT provide a clear advantage? Does that advantage become more consistent as the number of budgets increases?
-
How much of the Bullet improvement remains when TRPO-Lag uses exactly the same cost-feedback inputs and first-update handling as SafeOT?
4 comments
Rebuttal by Authors (2/2)
Q1. SafeOT’s advantage over the reactive switch as the number of budgets increases.
AQ1. Thank you for this important question. W1 already compares SafeOT and the switch on the main benchmarks. To examine whether the advantage becomes more consistent with more budgets, we additionally varied the budget configuration on four MuJoCo robots. We grouped joint costs into one budget, two budgets, or one budget per joint:
Table R2. Return–safety comparison across budget configurations. Rankings and dominance compare the two methods using seed-mean outcomes. Excess-cost ratios are relative to the switch; joint wins/losses count robot-level comparisons.**
Budget configuration Method Joint wins / losses ↑ / ↓ Pareto robots ↑ Median excess-cost ratio ↓ Mean return rank ↓ One budget — observed SafeOT with reactive floor 3 / 0 4/4 0.763× 1.25 Reactive switch 0 / 3 1/4 1.000× 1.75 Two budgets SafeOT with reactive floor 2 / 0 3/3 0.924× 1.33 Reactive switch 0 / 2 1/3 1.000× 1.67 Per-joint budgets SafeOT with reactive floor 2 / 0 4/4 0.875× 1.50 Reactive switch 0 / 2 2/4 1.000× 1.50
With one budget, SafeOT achieves lower mean excess cost and higher mean final return on three of the four robots. With per-joint budgets, it lowers mean excess cost on all four, while the switch achieves similar mean return. These results identify a joint advantage in the first setting and a return–violation trade-off in the latter, rather than an advantage that consistently strengthens with more budgets.Flow pricing jointly accounts for the reward–cost trade-offs across budgets. More budgets do not, by themselves, guarantee a larger benefit. Flow pricing accounts for how individual budgets constrain the proposed reward improvement. Its benefit therefore depends on the underlying reward–cost trade-offs, not simply on the number of budgets.
Q2. Comparing SafeOT and TRPO-Lag with matched cost feedback.
AQ2. Thank you for identifying this important comparison. We modified native TRPO-Lag to match SafeOT’s multiplier initialization and 50-episode cost-feedback window, including the first update, while retaining its native policy-update implementation. We evaluated all methods on the six Bullet-Safety-Gym tasks with ten seeds:
Table R3. Matched-feedback comparison. Hypervolume uses the fixed normalization and reference point of the original Bullet comparison. CV ratios are task-level medians relative to default SafeOT. Joint wins/losses, Pareto membership and return ranks are computed among these three methods.
Method Mean HV ↑ Joint wins / losses ↑ / ↓ Pareto tasks ↑ Median CV ratio ↓ Mean return rank ↓ Original TRPO-Lag 0.049 0 / 6 2/6 3.17× 2.00 TRPO-Lag with matched feedback and initialization 0.173 4 / 1 5/6 2.21× 1.87 SafeOT 0.288 3 / 0 6/6 1.00× 2.13
The results show that 68.6% of SafeOT’s original CV advantage remains after matching feedback and initialization, with lower mean CV on all six tasks. SafeOT also achieve higher hypervolume, zero joint losses and Pareto membership on every task, with similar return ranks (1.87 versus 2.13). Together, these findings would support an improved return–safety trade-off beyond differences in feedback and initialization.
We hope these experiments answer the reviewer's questions, and we would be glad to provide further results during the discussion.
Thank you for your additional information and response. These response addressed my concerns. And I will keep my positive score.
Official Comment by Authors
Dear Reviewer HtrG,
We are truly grateful for your follow-up and for maintaining your positive assessment of our work.
It is immensely rewarding to know that our responses have addressed your concerns about return–safety trade-offs with the reactive switch, performance across budget configurations, and matched feedback with TRPO-Lag.
Thank you once again for this constructive scientific dialogue and your continued support. Your feedback has been invaluable in improving our work.
Sincerely,
The Authors
Rebuttal by Authors (1/2)
We sincerely thank Reviewer HtrG for the thoughtful and constructive feedback. Your suggestions to compare directly with the reactive switch, examine the effect of the number of budgets, and align the cost-feedback inputs and first-update handling on Bullet-Safety-Gym have helped us clarify the contribution of flow pricing. In response, we provide additional experiments and detailed, point-by-point replies below.We believe these additions further reinforce the contributions of our work.
W1. Return–safety comparison with the reactive switch.