acceptodds
Under review as a conference paper at ICLR 2027

SafeOT: A Unified Safety-Price View of Safe Reinforcement Learning

Abstract

Safe reinforcement learning (safe RL) is commonly formulated in constrained Markov decision processes (CMDPs) with explicit constraints on expected safety costs, or optimized through scalarized penalty and Lagrangian objectives. Although both address the same reward–safety trade-off, they are usually treated as distinct families of methods rather than within a single operational framework, and common multi-constraint extensions either tune one Lagrange multiplier per cost or rely on scalarized penalties, which require per-constraint weighting and are prone to oscillation as the number of costs grows. This makes it difficult to jointly represent and control multiple safety constraints with distinct cost functions and bounds. We propose SafeOT, which recasts safe policy optimization as a budget- constrained optimal-flow problem over the transition graph of a CMDP. This representation places penalty-based and constraint-based safe RL on one spectrum and handles all constraints in a single projection. Coupling this projection with a PPO-like per-iteration policy update yields SafeOT, a GPU-efficient algorithm whose projection takes under 0.2 seconds per update. On 16 Safety-Gymnasium and Bullet-Safety-Gym tasks, SafeOT reduces cumulative constraint violation relative to TRPO-Lagrangian by 63% and 70%, respectively, lowering it on every task with comparable return on Safety-Gymnasium. With six to eight per-joint budgets on MuJoCo robots, its unclipped variant also incurs less violation than Lagrangian and PID updates.

Primary area:
reinforcement learning
Keywords:
safe reinforcement learning, constrained Markov decision processes, optimal transport, occupancy measures, Lagrangian methods, trust-region policy optimization, multi-constraint safety, VLA alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HtrGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SafeOT builds a graph from collected trajectories and solves a constrained flow problem to price safety budgets. It adds these prices to standard Lagrangian updates, reducing training violations in many tests with one or more safety budgets.

Critical strengths
  1. SafeOT changes how safety penalties are computed while keeping the underlying policy update largely intact. It can handle several budgets together, and the measured overhead is small in the default setup.

  2. The Safety-Gymnasium results show consistent reductions in training violations against TRPO-Lag. Additional seeds, budget settings, controller ablations, and final-policy evaluations make the evidence more convincing. I particularly appreciate the reporting of cases where the method does not help.

Critical weaknesses

It is still unclear how much the flow problem helps. In Table 37, the simple reactive switch has a median violation ratio of 0.91 compared with default SafeOT. This alone does not mean the switch is better overall: Table 27 shows that unclipped SafeOT gets higher returns with similar violations on the robot tasks. I would like to see the switch included in the same return–violation comparison as the main baselines. This would make it easier to judge what the proposed prices add.

Questions for authors
  1. When the reactive switch is included in the main return–violation comparison, where does SafeOT provide a clear advantage? Does that advantage become more consistent as the number of budgets increases?

  2. How much of the Bullet improvement remains when TRPO-Lag uses exactly the same cost-feedback inputs and first-update handling as SafeOT?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank Reviewer HtrG for the thoughtful and constructive feedback. Your suggestions to compare directly with the reactive switch, examine the effect of the number of budgets, and align the cost-feedback inputs and first-update handling on Bullet-Safety-Gym have helped us clarify the contribution of flow pricing. In response, we provide additional experiments and detailed, point-by-point replies below.We believe these additions further reinforce the contributions of our work.

W1. Return–safety comparison with the reactive switch.

AW1. Thank you for this helpful suggestion. We agree that the switch should be evaluated alongside the main baselines using both training violation and final-policy return. We therefore compare the switch with SafeOT on ten Safety-Gymnasium tasks and six Bullet-Safety-Gym tasks:

Table R1. Return–safety comparison over 10 seeds. Dominance and return ranks use the eight standard baselines plus the two displayed methods. CV ratios are relative to submitted default SafeOT

BenchmarkMethodMean HV ↑Joint wins / losses ↑ / ↓Pareto tasks ↑Median CV ratio ↓Mean return rank ↓
Safety-Gymnasium (10 tasks)SafeOT with reactive floor0.597649 / 010/100.303×4.80
Reactive switch0.554136 / 73/100.430×5.30
Bullet-Safety-Gym (6 tasks)SafeOT with reactive floor0.32021 / 34/60.85×5.50
Reactive switch0.30017 / 72/60.95×5.33

On Safety-Gymnasium, SafeOT lowers mean CV on all ten tasks and improves mean final return on seven. The average relative CV reduction is 21.7%, accompanied by higher mean hypervolume and a similar mean return rank. These development results support an improved return–safety trade-off beyond the reactive switch.These results support the additional value of flow-derived pricing within the shared reactive framework. Our responses to Reviewer zAdc also provide complementary evidence: W4 explains how the prices measure the marginal value of relaxing each budget, and W7 verifies policy safety while accounting for realization and estimation errors. We will include these comparisons in Sections 5.2–5.3 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q1. SafeOT’s advantage over the reactive switch as the number of budgets increases.

AQ1. Thank you for this important question. W1 already compares SafeOT and the switch on the main benchmarks. To examine whether the advantage becomes more consistent with more budgets, we additionally varied the budget configuration on four MuJoCo robots. We grouped joint costs into one budget, two budgets, or one budget per joint:

Table R2. Return–safety comparison across budget configurations. Rankings and dominance compare the two methods using seed-mean outcomes. Excess-cost ratios are relative to the switch; joint wins/losses count robot-level comparisons.**

Budget configurationMethodJoint wins / losses ↑ / ↓Pareto robots ↑Median excess-cost ratio ↓Mean return rank ↓
One budget — observedSafeOT with reactive floor3 / 04/40.763×1.25
Reactive switch0 / 31/41.000×1.75
Two budgetsSafeOT with reactive floor2 / 03/30.924×1.33
Reactive switch0 / 21/31.000×1.67
Per-joint budgetsSafeOT with reactive floor2 / 04/40.875×1.50
Reactive switch0 / 22/41.000×1.50

With one budget, SafeOT achieves lower mean excess cost and higher mean final return on three of the four robots. With per-joint budgets, it lowers mean excess cost on all four, while the switch achieves similar mean return. These results identify a joint advantage in the first setting and a return–violation trade-off in the latter, rather than an advantage that consistently strengthens with more budgets.Flow pricing jointly accounts for the reward–cost trade-offs across budgets. More budgets do not, by themselves, guarantee a larger benefit. Flow pricing accounts for how individual budgets constrain the proposed reward improvement. Its benefit therefore depends on the underlying reward–cost trade-offs, not simply on the number of budgets.

Q2. Comparing SafeOT and TRPO-Lag with matched cost feedback.

AQ2. Thank you for identifying this important comparison. We modified native TRPO-Lag to match SafeOT’s multiplier initialization and 50-episode cost-feedback window, including the first update, while retaining its native policy-update implementation. We evaluated all methods on the six Bullet-Safety-Gym tasks with ten seeds:

Table R3. Matched-feedback comparison. Hypervolume uses the fixed normalization and reference point of the original Bullet comparison. CV ratios are task-level medians relative to default SafeOT. Joint wins/losses, Pareto membership and return ranks are computed among these three methods.

MethodMean HV ↑Joint wins / losses ↑ / ↓Pareto tasks ↑Median CV ratio ↓Mean return rank ↓
Original TRPO-Lag0.0490 / 62/63.17×2.00
TRPO-Lag with matched feedback and initialization0.1734 / 15/62.21×1.87
SafeOT0.2883 / 06/61.00×2.13

The results show that 68.6% of SafeOT’s original CV advantage remains after matching feedback and initialization, with lower mean CV on all six tasks. SafeOT also achieve higher hypervolume, zero joint losses and Pareto membership on every task, with similar return ranks (1.87 versus 2.13). Together, these findings would support an improved return–safety trade-off beyond differences in feedback and initialization.

We hope these experiments answer the reviewer's questions, and we would be glad to provide further results during the discussion.

Thank you for your additional information and response. These response addressed my concerns. And I will keep my positive score.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer HtrG,

We are truly grateful for your follow-up and for maintaining your positive assessment of our work.

It is immensely rewarding to know that our responses have addressed your concerns about return–safety trade-offs with the reactive switch, performance across budget configurations, and matched feedback with TRPO-Lag.

Thank you once again for this constructive scientific dialogue and your continued support. Your feedback has been invaluable in improving our work.

Sincerely,

The Authors

Reviewer_3C5fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SafeOT, a new way to compute the Lagrange multipliers in safe RL. It builds a small graph from the collected transitions, solves a constrained flow problem on it, and uses the dual variables as "safety prices" that can rise before a violation happens. These prices are added to the usual Lagrangian multiplier in a PPO/TRPO update. Experiments on Safety-Gymnasium, Bullet-Safety-Gym, and multi-constraint MuJoCo tasks show lower constraint violation than Lagrangian baselines.

Critical strengths
  1. The new safety price can increase before the policy violates the constraint, while a normal Lagrangian multiplier only increases after violations happen. Since SafeOT only changes the multiplier of TRPO-Lag and keeps everything else the same, the lower constraint violation in experiments is a fair and convincing comparison.

  2. SafeOT handles many constraints at once in one optimization problem.

Critical weaknesses

It is hard to say that SafeOT performs better than all baselines. For example, sometimes CPO, PCPO, and the PID Lagrangian methods have lower constraint violation, and the Lagrangian methods and CUP have higher return.

Questions for authors
  1. Why does Table 2 use the default setting but Table 3 use the unclipped setting? Does the default setting also work with multiple constraints?

  2. Section 4.3 says the policy may not be able to follow the target flow. How often does this happen in practice, and does the safety price become misleading in that case?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank Reviewer 3C5f for the positive and constructive feedback. Your suggestions to clarify the return–safety comparisons, evaluate the default setting with multiple constraints, and examine the effect of imperfect target-flow tracking have helped us strengthen the paper. In response, we provide additional experiments and analysis, with detailed, point-by-point replies below. We believe these additions further reinforce the contributions of our work.

W1. Regarding the return–safety trade-off compared with the baselines..

AW1. Thank you for raising this important point. Our evaluation considers both training safety and task return. As defined in Section 5.1:

Lower CV indicates less training violation, but can come at the expense of return. The submitted paper therefore evaluates CV and final-policy return jointly using hypervolume (Section 5.1).Hypervolume measures the area dominated by each method’s normalized return–CV point, with larger values indicating a better joint trade-off. The ten-seed results are summarized below.

Table R1. Mean return–violation hypervolume across tasks.

BenchmarkSafeOT ↑Best of eight standard baselines ↑
Safety-Gymnasium0.4880.435 (CPO)
Bullet-Safety-Gym0.2850.275 (CUP)

These results clearly show that default SafeOT has higher mean hypervolume than each of the eight standard baselines on both suites, supporting our central empirical contribution of an improved return–safety trade-off.

Q1. Regarding the default and unclipped settings in Tables 2 and 3.

AQ1. Thank you for pointing out this difference. The default setting also works with multiple constraints. Table 2 tests whether a single default configuration performs well across tasks without retuning. For the multi-budget experiments in Table 3, we remove the cap to preserve the different penalty strengths assigned by the flow optimization. Otherwise, distinct prices above 1 would all be reduced to the same value. We compare both settings against the original baselines below.

Table R2. Default and unclipped SafeOT on the four multi-budget robots of Table 3. Ten paired training seeds per robot. Relative changes are computed from each robot’s seed means and averaged equally across robots.

ComparisonPairs with lower excess cost ↑Mean excess-cost reduction ↑Mean relative final-return gain ↑
SafeOT (default) vs TRPO-Lag40/4070.6%+26.4%
SafeOT (default) vs TRPO-PID32/4038.8%+20.5%
SafeOT (unclipped) vs TRPO-Lag40/4075.9%+20.4%
SafeOT (unclipped) vs TRPO-PID36/4046.5%+14.4%

These results clearly show that both settings reduce excess cost and improve final return on average relative to both baselines. Default SafeOT already provides substantial multi-budget benefits, while the unclipped setting achieves larger average excess-cost reductions. We will include both settings in the revised Table 3, retaining per-robot results for excess cost, satisfied budgets and final return.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2. Regarding target-flow tracking and the usefulness of safety prices.

AQ2. Thank you for this sharp and accurate observation. SafeOT uses flow-derived prices to weight cost penalties in the actual policy update. The policy can therefore benefit from these prices without exactly reproducing the target flow.

How often does this occur? We examined three fixed stages on ten Safety-Gymnasium tasks with ten seeds each. For the configuration, 89.3% of the 300 sampled updates moved along the target direction, completing 32.0% of the requested directional progress on average. Thus, the policy usually moves in the intended direction, but can realizes only part of the proposed change.

Does pricing still help when tracking is limited? We tested this directly on four multi-budget robots with ten fresh seeds. Having tested pricing against feedback alone, we next compare it with the reactive switch to determine whether flow optimization improves the return–safety trade-off beyond reactive control.

Table R3. Effect of flow pricing when policies do not fully follow the target flow. A setting is dominated when another achieves no greater violation and no lower return, with at least one strict improvement.

RobotCV reduction at matched strength ↑Switch settings dominated by SafeOT ↑SafeOT settings dominated by switch ↓Final-return gain at matched violation ↑
HalfCheetah7.6%6/60/6+1.9%
Ant21.3%5/60/6+35.3%
Hopper14.8%3/63/6+0.2%
Walker2d21.3%3/61/6+6.5%
Overall16.3%17/244/24+11.0%

These results show that flow pricing reduces subsequent violations and improves return relative, even with limited tracking. The comparison with feedback alone establishes the contribution of the price, while the comparison with the switch demonstrates the additional value of computing the price through flow optimization.Thus, the flow’s reward–cost information can guide the changes the policy can make, even when the complete target flow is out of reach.

We would be grateful to know whether these answers address your questions.

Thank you for your response. I will keep my positive score.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 3C5f,

Thank you for your response and feedback.

We are pleased that our response addressed your concerns. We sincerely appreciate you maintaining your score for acceptance and are grateful for your support.

Thank you once again for your time and constructive engagement. Your feedback has been invaluable in improving our work.

Sincerely,

The Authors

Reviewer_zAdcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SafeOT constructs an empirical transition graph and solves an entropically regularized flow problem subject to multiple safety budgets. Its dual variables provide safety prices, which are clipped, smoothed and added to conventional Lagrangian feedback before an otherwise unchanged PPO/TRPO update. The theory characterizes the graph projection and limiting price rules, and decomposes policy-cost violations into target infeasibility, flow-realization error and cost-estimation error.

Validation covers 16 primary benchmark tasks, additional seeds, a four-budget sweep, harder navigation tasks, multi-budget MuJoCo experiments and hard-contact studies. Relative to default TRPO-Lagrangian, the paper reports 63% and 70% reductions in cumulative violation on Safety-Gymnasium and Bullet-Safety-Gym, respectively, with favorable reported return–violation trade-offs on the former (Table 2). Its unclipped variant also improves several multi-budget comparisons against Lagrangian and PID methods (Table 3). These results support practical promise, but do not establish policy-level safety guarantees.

Critical strengths

The paper offers extensive, candid evaluation, low computational overhead and promising return–safety trade-offs.

Critical weaknesses

The central claim, that the flow projection supplies useful anticipatory prices, is not established for learned policies. SafeOT clearly beats a reactive switch only with exact tabular models (Appendices G.1–G.2), and its margin shrinks sharply as estimation noise grows (G.1). With learned policies, a switch whose price ignores the projection has lower cumulative violation on seven of ten Safety-Gymnasium tasks (median ratio 0.91; Table 37) and five of six harder tasks (Table 19), and on the robots SafeOT misses the paper’s own criterion for improving on it (G.4); none of this appears in the main text. With c = 1, SafeOT adds a bounded term in [0, 1] to the TRPO-Lagrangian multiplier, and no learned-policy ablation shows why a flow optimization is needed to set it. PID-Lagrangian, CPO and PCPO have similar or lower median violation, usually at lower return, and the paper’s own joint decision against TRPO-PID on Bullet-Safety-Gym is negative (Appendix C.2). Tuning TRPO-Lagrangian by the same pre-registered rule narrows the gap (2.75× to 1.59×); with both tuned, total training cost is about equal (Appendix D.6).

The theory concerns a graph surrogate whose flows need not be realizable by a policy, and Lemma 2 leaves realization and estimation errors uncontrolled. The hard-contact results rely on a separate executor, and the authors’ controls attribute the contact reduction to critic-guided selection rather than pricing (Appendices I.2, I.4).

On the four level-2 navigation tasks, CPO and both PID methods violate less, and SafeOT is dominated on both Button2 tasks (Table 19).

Novelty beyond established components is insufficiently justified, and decisive qualifications are dispersed across 64 pages. Comparisons against comparably tuned reactive controllers with learned policies are needed.

Questions for authors

None

10 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

We sincerely thank Reviewer zAdc for the detailed and critical feedback. We recognize the concerns raised regarding the added value of the flow projection, the baseline comparisons, the scope of the theory, and novelty and presentation. We have taken these points very seriously and, in response, have conducted five extensive new sets of experiments to address every weakness raised. This includes:

  1. Anticipatory pricing and the contribution of the bounded price term (W1, W4).
  2. Robustness to model noise and comparison with the reactive switch under matched tuning (W2, W3).
  3. New comparisons with comparably tuned baselines(W5, W6).
  4. Policy safety analysis and experiments on hard-contact tasks (W7, W8).
  5. Novelty and a more focused presentation (W9).

We believe these significant additions substantially strengthen our paper and directly address the reviewer's concerns, which we detail below.

W1. Regarding anticipatory prices "not established for learned policies".

A1. Thank you for this central question. With learned policies, SafeOT assigns prices before violations occur, identifying budgets at elevated risk. Separate ablations show that adding flow pricing reduces training violations. To validate this, we conduct an extensive ablation study on the four multi-budget robots over ten seeds and compare SafeOT with the reactive switch and TRPO-Lag. The results are summarized below and have been added to the Sections 5.3 of the revised manuscript..

Table R1. Anticipatory pricing and violation on the four multi-budget robots (ten seeds).

Robot (K)Onsets priced ahead ↑ (SafeOT / switch)Violated within three epochs (priced early ↑ / all met)Excess cost, ×10³ ↓ (SafeOT / TRPO-Lag)
HalfCheetah (6)22% / 0%44% / 12%15.6 / 26.0
Ant (8)13% / 0%28% / 12%0.95 / 12.0
Hopper (3)11% / 0%50% / 20%1.25 / 4.90
Walker2d (6)6% / 0%26% / 6%0.41 / 1.67

These results clearly show that SafeOT prices 6–22% of violation onsets one epoch ahead and the budgets it prices early are violated within three epochs 2.3–4.2 times as often as all met budgets. Adding its price to TRPO-Lag removes 40–92% of the excess cost, with higher final return on three of the four robots task. Thus, with learned policies, the projection prices the budgets at risk before they are violated, and its price substantially lowers violation.

W2. Robustness to estimation noise and performance with learned policies.

A2. Thank you for this important question. Estimation noise narrows SafeOT’s advantage over the switch, but SafeOT still achieves lower violation in most tested instances. We also extend the comparison to learned policies. To validate this, we added synthetic noise to the model estimates. At each noise level, we evaluated 20 grid CMDPs under two budget settings each, giving 40 test cases.The results are summarized below and have been added to the revised PDF.

Table R2. SafeOT versus the switch under synthetic estimation noise. Each noise level includes 40 test cases.

Relative noise level rCases where SafeOT has lower cumulative violation ↑
0 (exact model)40/40
0.0540/40
0.15 (registered test level)40/40
0.4528/40

These results clearly show that SafeOT violates less than the switch in every cell up to the registered test level, and still wins 28 of 40 cells at three times that level. We also compare SafeOT and the switch using learned policies on the four multi-budget MuJoCo tasks in Table R3 below:

Table R3. SafeOT versus the switch on multi-budget robots. Six matched price strengths and 40 seeds per strength. Changes are relative to the switch.

Robot (K)Violation reduction ↑Win rate on violation ↑Relative return gain ↑
HalfCheetah (6)+7.6%89.6%−0.20%
Ant (8)+21.3%75.4%+2.28%
Hopper (3)+14.8%65.8%−0.91%
Walker2d (6)+21.3%65.0%−2.16%
Overall+16.3%74.0%−0.25%

Averaged over 40 seeds, SafeOT lowers violation on all four robots. The overall reduction is 16.3%, while final return changes by only −0.25% . These results extend the evidence beyond exact tabular models: SafeOT reduces violations in learned policies while maintaining a similar average return. Thus, SafeOT's advantage over the switch extends from exact models to noisy estimates and multi-budget learned policies.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

W3. Regarding the reactive switch’s lower violation on learned-policy benchmarks.

A3. Thank you for this sharp and accurate observation. The original comparison tests two pricing rules: the switch directly penalizes observed budget excess, whereas SafeOT derives its price from the proposed flow. It therefore changes both the information used for pricing and the response to current violations.

To examine what flow optimization adds beyond reactive control, we retain the same reactive response in both controllers. Revised SafeOT uses the switch price as a floor: its price is the larger of the flow-derived price and the reactive price. This preserves the response to observed violations while allowing the proposed flow’s reward–cost information to guide additional pricing.

Table R4. SafeOT with a reactive price floor versus the switch. The completed development comparison uses ten seeds per task. Both retain the same feedback and smoothing.Final returns are normalized and averaged equally across tasks.

BenchmarkMean relative CV reduction ↑Tasks with lower SafeOT mean CV ↑Tasks with higher SafeOT mean final return ↑Mean normalized final return ↑ (SafeOT / switch)
Ten Safety-Gymnasium tasks — earlier pilot21.7%10/107/100.666 / 0.636
Six harder tasks8.0%4/63/60.49 / 0.50

In this development comparison, SafeOT has lower mean CV on all ten tasks, averaging a 21.7% reduction, together with higher mean normalized final return (0.666 versus 0.636). The harder-task results show 8.0% lower CV on average, with slightly lower return (0.49 versus 0.50).

Together with the multi-budget robots results in W2, these findings support combining reactive control with flow-derived pricing: preserving a direct response to observed violations while improving the aggregate return–safety trade-off. We will present this evidence in Sections 5.2–5.3 of the revised manuscript.

W4. The contribution of flow optimization to policy learning with a bounded term in [0, 1].

A4. Thank you for this important question. Flow optimization uses the proposed reward-improving transitions and budget constraints to determine the cost penalties in each policy update. By Proposition 1, each price is a budget dual variable: it measures the marginal value of relaxing that budget in the flow optimization. The prices therefore reflect how strongly each budget constrains the proposed improvement. The c = 1 cap limits their added influence on the policy update. We examine their practical contribution through learned-policy ablations on all ten Safety-Gymnasium tasks:

Table R5. Effect of the pricing rule on training violation (five seeds per task). Values are median task-level CV ratios relative to default SafeOT; lower is better.

MethodRelative CV ↓
SafeOT1.0×
TRPO-Lagrangian (flow price removed)2.8×
SafeOT with LP pricing (ε = 0)1.8×

On all ten tasks, adding the flow price reduces mean CV relative to TRPO-Lagrangian, and entropic pricing outperforms the LP limit. The comparisons in W2–W3 further support improved aggregate return–safety trade-offs over the switch for the revised configuration with a reactive price floor.Together, these comparisons evaluate both the contribution of the price to actual policy updates and its value relative to a simpler pricing rule.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

W5. Comparisons with PID-Lagrangian, CPO and PCPO, and the joint test in Appendix C.2.

A5. Thank you for this important question. SafeOT can further improve on PID-based controllers method, it combines flow-derived prices with observed-cost feedback, which can be integral or PID as described in submitted Appendix J. W4 examines flow pricing with integral feedback.The comparison with PID methods raises a further question: does flow pricing still help when the underlying controller uses PID feedback? We address this using the PID-feedback to evaluates the additional value of flow pricing on this stronger controller. The results are summarized below and have been added to Section 5.2 of the revised manuscript.

Table R6. SafeOT versus PID-Lagrangianp[3], CPO[1] and PCPO[2], with ten seeds per task. CV ratios are task-level medians relative to SafeOT. Return ranks and hypervolume use the eight standard baselines.

Task groupMetricSafeOT - PIDCPPO-PIDTRPO-PIDCPOPCPO
Ten Safety-Gymnasium tasksMedian CV ratio ↓1.00×1.73×1.18×1.45×0.98×
Mean return rank ↓5.406.805.905.608.30
Mean hypervolume ↑0.5120.2770.4240.4350.230
Six harder tasksMedian CV ratio ↓1.00×1.95×1.25×1.67×1.64×
Mean return rank ↓5.336.507.836.177.33
Mean hypervolume ↑0.3490.2410.1520.3470.159

SafeOT achieves the best mean return rank and highest mean hypervolume among these methods in both groups. SafeOT also has lower mean CV than CPO and PID baselines on all four level-2 tasks. On Button2 tasks, it also achieves higher mean return than CPO and TRPO-PID.

For the Bullet-Safety-Gym concern, we extended the C.2 evaluation to ten seeds. Also to test whether flow pricing provides additional benefits when the controller already uses PID feedback. SafeOT + PID achieves lower mean CV than TRPO-PID (115.1 vs 118.7; 0.97×), together with a positive mean normalized return difference of +0.01.

W6. Comparisons with comparably tuned reactive controllers.

A6. Thank you for this excellent suggestion to further disentangle the effects of our method. We tested whether SafeOT’s advantage remains after baseline tuning by the pre-registered rule in Appendix D.6. The comparisons are summarized below and will be included in Section 5.2 of the revised manuscript.

Table R7. Performance against reactive baselines after tuning. Ten Safety-Gymnasium tasks, ten seeds per task. Each row reports its own matched training cohort. Return differences are SafeOT minus the baseline.

Reactive baselineBaseline mean CV ↓SafeOT mean CV ↓CV reduction ↑Normalized return difference (%) ↑
TRPO-Lag42.2725.2340.3%+7.2%
PPO-Lag129.5923.7781.7%−0.1%
TRPO-PID26.0021.0219.1%+4.7%
CPPO-PID36.7620.7243.6%+20.3%

These results clearly show that SafeOT reduces mean CV by 19–82% against the four baselines after tuning. Regarding total training cost, CV counts only budget exceedances. Similar total cost can therefore coexist with lower CV: SafeOT’s improvement concerns the portion exceeding the safety budget.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response

We sincerely thank the reviewers for recognizing our extensive evaluation and promising return–safety trade-offs (zAdc), anticipatory pricing and joint handling of multiple constraints (3C5f), and low overhead and consistent violation reductions (HtrG). Our responses address the following points:

  • Flow pricing contributes to learned-policy performance (zAdc, HtrG). SafeOT jointly derives budget prices from the proposed policy update. Our experiments demonstrate anticipatory pricing with learned policies, while separate ablations establish the contribution of flow pricing and entropic regularization to reducing training violations.

  • The benefits persist against stronger reactive baselines (zAdc, 3C5f, HtrG). We added comparisons with tuned Lagrangian and PID controllers. These controls support the additional value of flow-derived prices.

  • Flow pricing improves the return–safety trade-off over the switch (zAdc, HtrG). We retains the switch price as a floor. The development comparisons show lower training violation and improved return–safety performance.

  • The evidence covers noisy estimates and multiple budgets (zAdc, 3C5f, HtrG). SafeOT retains a violation advantage under estimation noise. Both default and unclipped settings benefit multi-budget robot learning.

  • Useful pricing does not require exact target-flow realization (zAdc, 3C5f). Our new experiments show that flow pricing reduces violations and improves the return–safety trade-off even when policies do not fully realize the target flow. We also certify trained policies’ expected-cost feasibility under the evaluation distribution by accounting for realization and estimation errors.

We have submitted the revised manuscript incorporating the new results and clarifications discussed above. We thank the reviewers again for helping us strengthen the evidence for SafeOT and clarify when and how flow-derived pricing improves safe policy learning.

AuthorsOpenReview ↗

Author Final Remarks

We are grateful to the reviewers for their rigorous engagement, which has been instrumental in strengthening our manuscript.

We are pleased the discussion led to a clear consensus on the paper's merits and summarize the key strengths below:

  • Novel Solution: Our approach was recognized for jointly addressing anticipation and multiple constraints. Reviewer 3C5f highlighted that safety prices “can increase before the policy violates the constraint” and that SafeOT “handles many constraints at once in one optimization problem.”

  • Efficient Algorithm: Reviewers appreciated that SafeOT modifies safety penalties while “keeping the underlying policy update largely intact” (HtrG). Its “low computational overhead” (zAdc) further supports practical integration into existing policy-learning algorithms.

  • Comprehensive Experiments: The results were recognized for “consistent reductions in training violations against TRPO-Lag” on Safety-Gymnasium (HtrG) and “promising return–safety trade-offs” (zAdc). Reviewer 3C5f also described the comparison with backbone as “fair and convincing.”

We also want to acknowledge the initial concerns, as addressing them directly has improved our paper:

  1. Return–Safety Comparisons. These concerns were addressed through additional comparisons and controlled ablations, clarifying the return–safety trade-offs, effectiveness under multiple constraints, and fairness of baseline comparisons.

  2. Value of Flow Pricing. Randomized-allocation ablations, matched return–violation comparisons, and switch controls with the same early threshold provided complementary evidence for the added value of flow pricing.

  3. Policy-Level Bounds. We clarified target-flow tracking and final-policy certification, and added a retrospective bound on expected budget excess during training.

We will incorporate the additional experiments and clarifications into the final manuscript, strengthening the presentation of the unified flow formulation, the controlled comparisons, and the scope of the policy-level analysis.

Finally, we are encouraged by the reviewers’ recognition of our work’s potential. We believe SafeOT contributes both a practical algorithm for learning under multiple safety constraints and a unified perspective connecting penalty-based and constraint-based safe RL through flow-derived safety prices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.