acceptodds
Under review as a conference paper at ICLR 2027

Do We Need Cost Critics? Local Cost Relabeling for Offline Safe Reinforcement Learning

Abstract

Offline safe reinforcement learning commonly relies on learned cost-to-go functions, feasibility models, or adaptive penalty mechanisms to enforce cumulative-cost constraints. We ask how much of this machinery is necessary when safety violations are directly observed in the offline data. We study a simple alternative, RELABEL(COST), that retains the full dataset, replaces the reward on positive-cost transitions by a fixed negative value derived from the reward scale, and then trains a behavior-regularized offline RL policy. The method uses no cost critic, no feasibility estimator, and no safety-specific penalty tuning. In backbone-matched continuous-control experiments, RELABEL(COST), consistently satisfies the safety budget and achieves near-zero cost on most tasks while remaining competitive with, and often outperforming, adaptive additive penalties and cost-critic-based relabeling. These comparisons also reveal an important role for strong behavior regularization, with local cost supervision identifying undesirable transitions while regularization limits policy improvement to regions supported by the offline data. Our analysis characterizes when fixed local penalties align with zero-cost policy optimization, explains how delayed cost events propagate through ordinary reward value learning, and shows that safe behavior can in principle be recovered even when complete trajectories in the dataset are infeasible. Together, the results suggest that direct local cost supervision combined with strong offline regularization can provide a simple and robust alternative to explicit cumulative-cost modeling in offline safe RL.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, safe reinforcement learning, constrained reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WUYtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies strict zero-cost offline reinforcement learning and proposes RELABEL(COST). For each offline transition, it keeps the original reward when the observed cost is zero and replaces the reward by a fixed negative value when the cost is positive. It then trains a behavior-regularized offline RL algorithm, ReBRAC, on the relabeled data, without a cost critic, feasibility model, or Lagrange multiplier. The paper argues that ordinary reward Bellman backups propagate the penalty from delayed cost events to earlier decisions. It provides bounds relating the relabeled objective to the number of positive-cost events, an event-price characterization for exact recovery of the optimal zero-cost policy, delayed-cost analyses, and experiments on 19 DSRL tasks.

Critical strengths
  1. The paper asks a meaningful question for offline safe RL: whether direct transition-level cost labels can sometimes replace an explicitly learned long-horizon cost model. Retaining useful portions of globally infeasible trajectories is a reasonable offline-data perspective.

  2. The experiments include useful controls. In particular, comparing no relabeling, additive fixed penalties, adaptive penalties, and cost-critic relabeling helps isolate the role of the proposed signal. The results suggest that fixed penalties plus strong behavior regularization can be effective on the evaluated tasks.

Critical weaknesses

The proposed method is essentially the classical fixed reward-penalty approach, expressed using a local positive-cost indicator. Strict zero-cost RL is a legitimate and useful problem setting, but restricting the objective does not by itself make the fixed-penalty solution new. In the paper's notation, the relabeled objective is Aπ−PnπA^\pi-Pn^\pi: it assigns the same price to every positive-cost event and then relies on the ordinary reward critic to propagate that price to earlier actions. The latter is standard Bellman credit assignment, while the former is exactly the fixed penalty mechanism used before the development of adaptive safe-RL methods. The event-price theorem formalizes the familiar sufficient condition P>ρ0P>\rho_0, where ρ0\rho_0 is unknown and depends on the task and policy class; choosing PD=max⁡i∣ri∣P_D=\max_i |r_i| neither estimates ρ0\rho_0 nor guarantees that it is large enough. More importantly, if the deployment budget is positive, a fixed price for all cost events cannot express that spending a small amount of budget may be worthwhile for a large reward gain, and it cannot adapt the reward-cost weighting to state, remaining budget, cost magnitude, or local uncertainty. These are precisely the practical limitations that motivated Lagrangian, constrained policy optimization, and state-dependent safety weighting in safe RL. The paper's theory therefore establishes when an old penalty method works under its assumptions, but does not resolve the reason fixed penalties are difficult to use. The empirical evidence does not fill this gap: the additive fixed-penalty ablation obtains similarly strong safety and return, so the results are consistent with a sufficiently large classical penalty plus behavior regularization being responsible for the gains. At most, the paper shows that this simple baseline is effective on benchmarks with favorable local cost coverage; it does not establish a new mechanism or a general replacement for adaptive cost modeling.

Questions for authors

What is the precise scientific contribution beyond a classical fixed-penalty baseline? Please explain why PD=max⁡i∣ri∣P_D=\max_i |r_i| should be considered a principled solution to the task-dependent penalty-selection problem when the exact-recovery condition depends on the unknown ρ0\rho_0. More importantly, can the authors provide a comparison against an otherwise identical fixed-penalty method and evaluate tasks where the optimal feasible policy deliberately spends part of a positive budget, where cost magnitudes differ, or where safety information is delayed and state-dependent? Such evidence is needed to show that local relabeling contributes beyond the fixed penalty itself and that eliminating a cost critic addresses a limitation of existing safe-RL methods rather than simply avoiding it by changing the objective.

5 comments
AuthorsOpenReview ↗

Response to reviewer WUYt (Weaknesses)

Thank you for the detailed review. We first address the two weaknesses you identified.

W1. The theory and the difficulty of using fixed penalties.

A fixed price can express the reward–cost tradeoff described in the review. The difficulty is choosing a price that yields a policy within the desired budget. Under our replacement objective, a policy π2\pi_2 with more expected cost events than π1\pi_1 has higher relabeled return precisely when

Aπ2−Aπ1>P(nπ2−nπ1),A^{\pi_2}-A^{\pi_1}>P\bigl(n^{\pi_2}-n^{\pi_1}\bigr),

where AπA^\pi is expected reward collected on zero-cost transitions and nπn^\pi is the expected number of cost events. The additional events are worthwhile whenever the additional retained reward outweighs their charge.

The benchmark constraint is on expected total cost. For a feasible finite-horizon CMDP with finite state and action spaces, optimizing over all randomized policies, linear-programming duality gives a scalar λB\lambda_B for which a budget-optimal policy maximizes JRπ−λBJCπJ_R^\pi-\lambda_B J_C^\pi (Altman, 1999). A state-dependent multiplier is therefore not necessary to represent an optimum of this expected-cost problem. Lagrangian methods search for the appropriate scalar multiplier. Finding that multiplier and selecting a feasible optimizer, which can require randomization among tied policies, is the calibration problem. It is distinct from an inability of fixed prices to express reward–cost tradeoffs.

Our results demonstrate worthwhile budget spending under replacement itself. In the one-event instance of Appendix F.5, the default penalty selects the policy that spends the full budget and attains the constrained optimum. On the four Goal tasks, a quarter of the default penalty increases goal completion by about two to five times, with task-level mean costs using 2%–37% of the budget. Each of the twelve training seeds remains within budget in mean evaluation cost (Tables 11 and 12). In our response to Q3 part (b) below, we give the exact comparison and the learned-policy results.

For the zero-cost target, Theorem 2 establishes an exact threshold. Every P>ρ0P>\rho_0 makes exact optimization of the relabeled objective recover the best zero-cost policies, whereas for 0≤P<ρ00\le P<\rho_0 a positive-cost policy beats the zero-cost optimum. Thus, the penalty does not have to be calibrated to the budget for this target. Every value above ρ0\rho_0 gives the same optimum. Proposition 6 bounds this threshold under an explicit model of sustained violations, giving a regime in which the reward-scale default suffices. The training target is stricter than the budget, and the evaluated problem is unchanged. Every method is measured on the original reward against the same benchmark budgets.

W2. What the successful additive baseline establishes.

The contribution is to identify when separate cumulative-cost estimation is unnecessary and why broader cost labels can reduce performance. Both rules use observed cost events and the same behavior-regularized learner. Their comparable performance shows that this finding extends across two local penalty rules. In Table 1, each of the eight baselines exceeds the budget on 7 to 15 of the 19 tasks, and the fixed penalty exceeds it on none.

The additional comparisons show that the choice of safety information matters. Broader labels reduce return, and restricting a learned label to observed cost events restores feasibility. The theory identifies a loss from behavior-dependent labeling even with an exact cost critic and identical zero-cost and budget-constrained optima. Together, these results go beyond establishing that a large penalty works; our response to your Q1 (see below) further clarifies this and provides the evidence.

AuthorsOpenReview ↗

Response to reviewer WUYt (Questions 1 and 2)

Next, we address the first two questions you pose.

Q1. What is the precise scientific contribution?

We establish which cost information is sufficient for offline policy improvement in the studied regimes, what the dataset must retain, and why adding broader cost labels can make learning worse.

First, the comparisons hold the learner fixed. With ReBRAC, data, hyperparameters, and seeds unchanged, removing cost relabeling produces mean costs above ten times the budget on all ten ablation tasks. Local penalties satisfy every matched task's budget, while the tested adaptive and cost-critic alternatives do not consistently improve return or feasibility (Figure 1, Tables 3 and 4). Relabel(Cost) also satisfies all 19 main-table mean-cost budgets.

Second, complete feasible trajectories are unnecessary in these datasets. Relabel(Cost) remains within budget on all seven Bullet tasks after every trajectory with cost at most 10 is removed. Removing the positive-cost transitions instead leaves only 6% of evaluation episodes within budget (Section 5.3). This separates the need for observed violations from the availability of complete safe demonstrations.

Third, richer relabeling signals do not improve the seven-task comparison (Table 16).

CODAC (Conservative Offline Distributional Actor Critic, Ma et al., NeurIPS 2021) learns a conservative distribution over future returns. Our CODAC-based relabeler uses a distributional cost critic and thresholds its predicted mean cost plus one standard deviation.

Relabeling signalTasks within mean-cost budgetMean normalized return
Observed cost (ours)7/70.62
Behavior-policy cost Q-function6/70.36
Trajectory classifier5/70.27
Suffix after the budget is exceeded3/70.40
CODAC cost critic2/70.46
CODAC restricted to observed cost7/70.52

CODAC's unrestricted mask covers more than 80% of transitions on five tasks late in training. Restricting its labels to observed cost events restores feasibility on all seven tasks and improves mean return (Table 17).

Appendix F.3 identifies a mechanism that can produce such a loss. A behavior-policy cost critic can assign a high cost-to-go to an action because the behavior subsequently chooses an unsafe continuation, even when a safe continuation is covered. In the example, with full coverage, exact planning, and an exact behavior cost critic, critic-based relabeling earns 0.75 while local relabeling earns 2. Both the zero-cost optimum and the budget-constrained optimum are 2. The loss therefore arises from the relabeling rule, without estimation error or a more conservative safety target. This is a concrete limitation of using behavior-dependent cost-to-go labels as offline reward supervision.

Q2. Why is PD=max⁡i∣ri∣P_{\mathcal D}=\max_i|r_i| a principled choice when ρ0\rho_0 is unknown?

For a positive budget, the appropriate penalty depends on the task, as the Goal sweeps illustrate. PDP_{\mathcal D} is a default for the zero-cost target, where every P>ρ0P>\rho_0 suffices under exact optimization. We use this default for three reasons.

  • Reward scale. −PD-P_{\mathcal D} is no greater than any reward in the dataset, and multiplying rewards by a positive constant rescales the penalty by the same constant. The default requires no safety-multiplier tuning.

  • A sharp sufficient condition. Under the deterministic repeated-segment assumptions of Appendix F.4, every covered failing segment incurs cost on at least mm of its LL transitions. Proposition 6 gives

    ρ0≤[LΔrm−r+]+,\rho_0\le\left[ \frac{L\Delta_r}{m} - r_+ \right]_+,

where Δr=r+−r−\Delta_r = r_+ - r_-. This bound is attained within the model. For nonnegative rewards, when the dataset contains their positive maximum r+r_+, we have PD=r+P_{\mathcal D}=r_+. Corollary 2 then guarantees PD>ρ0P_{\mathcal D}>\rho_0 whenever 2mr+>LΔr2mr_+>L\Delta_r. In particular, m>L/2m>L/2 suffices. Sustained violations can therefore make the reward-scale default sufficient without estimating the exact event price.

  • Empirical robustness. Half the default remains within budget on all seven Bullet tasks, and one tenth remains within budget on six of seven, using task-level mean evaluation costs (Table 14). Halving the penalty also increases mean normalized return from 0.62 to 0.66 (Table 5).

The segment model gives a sufficient regime; the support on the benchmarks is empirical. The default can lie below ρ0\rho_0 when a brief violation permits a long reward gain. In our response to Q3(b) in the next message, we give two fully solved examples showing when the chosen penalty is sufficient to recover the best zero-cost policy and when it is too small.

AuthorsOpenReview ↗

Response to reviewer WUYt (Question 3)

Q3. The requested comparisons and regimes.

(a) An otherwise identical fixed-penalty method. Figure 1 and Tables 3 and 4 provide this control. They use the same backbone, data, hyperparameters, and seeds, with only the reward rule changed to r−λPDcr-\lambda P_{\mathcal D}c. At the default scale, both rules satisfy every matched task's budget and their normalized returns differ by at most 0.05. The expanded seven-task sweep gives the following feasibility counts.

MultiplierReplacement tasks within budgetAddition tasks within budget
17/77/7
0.57/74/7
0.256/71/7
0.16/70/7

Counts use mean evaluation costs (Table 14). Replacement removes the original reward on costly transitions as well as imposing the negative value, so equal numerical multipliers do not mean equal effective charges. When each rule uses its best tested feasible multiplier, returns differ by at most 0.01 on six tasks; replacement is higher by 0.06 on AntCircle (Table 15). The observed advantage is a wider feasible range of tested multipliers. The comparison uses a common budget, without requiring identical achieved costs.

(b) A task whose optimal policy spends budget. The exact CMDP family in Appendix F.5 has a three-step instance with budget 1 and both paths covered by the dataset. The safe path earns 0.2 per step at no cost. The risky path earns 1 per step with unit cost on its last mm steps, so PD=1P_{\mathcal D}=1. The table reports original returns and expected costs, allowing randomized policies.

Cost events on risky branchBudget-optimal (return, cost)Replacement at PDP_{\mathcal D}Addition at PDP_{\mathcal D}
m=1m=1(3,1)(3,1)(3,1)(3,1)(3,1)(3,1)
m=2m=2(1.8,1)(1.8,1)(0.6,0)(0.6,0)(3,2)(3,2)

For m=1m=1, ρ0=1.4>PD\rho_0=1.4>P_{\mathcal D}. Both rules choose the risky path, which spends the full budget and attains the constrained optimum. For m=2m=2, ρ0=0.2<PD\rho_0=0.2<P_{\mathcal D}, so replacement chooses the safe path. Addition chooses the risky path because its zero-cost threshold is ρ0add=1.2>PD\rho_0^{\mathrm{add}}=1.2>P_{\mathcal D}, and its cost of 2 violates the budget. The constrained optimum instead mixes the two paths equally. Thus a fixed penalty can spend budget, but it does not automatically select the policy appropriate to a given budget.

The Goal sweeps provide complementary evidence from learned policies (Tables 11 and 12). At multiplier 0.25, return is positive and goal completion improves on all four tasks. All twelve training seeds remain within budget in their mean evaluation cost. At multiplier 0.1, return improves further, but both CarGoal tasks exceed their budgets. The exact example supplies the known optimum; the Goal experiments measure the achieved return–cost tradeoff.

(c) Different cost magnitudes. On nine MetaDrive datasets with fractional costs, replacement satisfies all nine mean-cost budgets with positive normalized return on every dataset; magnitude-weighted addition at the same scale satisfies one (Table 20). This compares event-based and magnitude-weighted penalties on nonbinary-cost tasks at the reported scale. Replacement treats positive-cost events equally, while the additive comparator retains their magnitude information.

(d) Delayed and state-dependent safety information. We perturb training labels and evaluate with the true costs. Both a uniform five-step delay and a state-dependent ten-step delay affecting about 45% of events leave all four tested tasks within budget (Table 21). AntVelocity loses the most safety margin. Its cost/budget ratio rises from 0.04 to 0.81 under the state-dependent delay, while normalized return remains 0.99. Its median cost-event duration is only two steps. These experiments test whether local supervision remains effective when the recorded labels are displaced in time.

Reviewer_HwD3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Summary

This paper proposes a simple approach to offline safe reinforcement learning, which is relabelling the costs in the safe control setting. It replaces the reward of each positive-cost transition with a fixed negative value derived from the dataset’s reward scale, retains all transitions, and trains a ReBRAC policy on the modified dataset.

The method requires neither a separate cost critic nor a deployment budget during training. The authors provide theoretical conditions under which local penalties favor zero-cost policies and explain how ordinary reward learning propagates delayed safety signals. By evaluating over a wide range of experiments, the authors have also shown the empirical gain of this proposed method.

Critical strengths

Here in this paper I find these strengths that convinces me towards accepting this paper.

Strengths

  1. The paper establishes a useful and simple baseline. This method is easy to deploy and is robust across different tasks in BulletGym and in SafetyGym. Relabel(Cost) satisfies the reported mean-cost budgets on all 19 tasks and achieves the highest feasible return on nine. Comparisons using the same ReBRAC backbone help separate the effect of cost relabeling from the underlying offline RL algorithm. The finding that fixed additive penalties also perform well is informative and is acknowledged by the authors.

  2. The theoretical claims is solid and has further improved the soundness of the method. The theorems in section 4 is well-written and fully demonstrates why the proposed Relabelling methods could help improve safety.

  3. The analysis and ablations help explain when the approach can work. The paper distinguishes zero-cost optimization from budget-constrained optimization and explains why a local penalty can still influence earlier decisions through Bellman backups. The trajectory-filtering experiments also provide evidence that complete feasible demonstrations are not always necessary when useful transition-level information remains available.

Critical weaknesses

Despite the above strengths, I would like to point out the following weaknesses in this work.

Weaknesses

  1. Mismatch between the theory and the practical algorithm, as part of my concerns is that the theory may provide limited guarantees for the practical algorithm. Theorem 1 assumes that the learned policy approximately matches a zero-cost comparator under the population relabeled objective, but the error term absorbs optimization, estimation, and distribution-shift errors without bounding them. Theorem 2 requires a penalty larger than the unknown event price, while the implemented penalty depends only on the maximum absolute reward in the dataset. The sufficient condition in Appendix B.4 applies to a restricted delayed-cost construction. Furthermore, the main analysis uses undiscounted returns, whereas ReBRAC uses a discount factor of 0.99 and additional regularization.

  2. The cost of conservative behavior is insufficiently examined. The method ignores the deployment budget and penalizes every positive-cost event equally, favoring near-zero-cost behavior even when some cost is permitted. Table 1 shows very low normalized returns on all four Goal tasks: 0.03, −0.01, −0.04, and 0.00. On CarGoal1 and PointGoal1, BC-Safe achieves higher returns while remaining within budget. These results weaken the claim that performance remains competitive outside the tasks where the method wins. However, the main relabeling and penalty ablations exclude the Goal tasks, leaving this failure mode unexplained. Comparisons across cost budgets and diagnostics of goal completion would help distinguish useful conservatism from poor task performance.

Questions for authors

I would appreciate it if the authors could further respond to the following questions:

Questions

  1. Can the bound in Theorem 1 be shown to be nonvacuous for any evaluated task using the implemented penalty?

  2. Since fixed additive penalties achieve similar performance on the matched tasks, what advantage does reward replacement provide when the two methods are compared at matched safety levels rather than the same numerical penalty multiplier?

5 comments
AuthorsOpenReview ↗

Response to reviewer HwD3 (Weakness 1)

Thank you for your careful review and for your positive assessment. We first address the two weaknesses (in the first two messages) that you identified and then answer the two questions (in the third message).

W1. The connection between the theory and the implemented algorithm.

Learning error and regularization. We now make the learning-error term quantitative under explicit coverage assumptions. The resulting bound shows how sample size, optimization accuracy, and behavior regularization affect safety. Proposition 4 establishes this for a finite-model learner whose dataset contains mm independent samples at each covered state–action–time pair and whose policies use only covered actions. With probability at least 1−δ1-\delta, a planner that optimizes the empirical regularized objective to tolerance ϵopt\epsilon_{\rm opt} satisfies the population comparison required by Theorem 1 with

ϵeff=2Em(P)+ϵopt+β[ΩD(π0)−ΩD(π^)]+.\epsilon_{\rm eff}=2E_m(P)+\epsilon_{\rm opt}+\beta[\Omega_{\mathcal D}(\pi_0)-\Omega_{\mathcal D}(\widehat\pi)]_+.

These terms account for estimation, optimization, and the empirical regularizer used by the planner. Proposition 4 gives the constants in Em(P)E_m(P) explicitly. For fixed horizon, reward bounds, model size, and confidence, Em(P)=O((1+P)/m)E_m(P)=O((1+P)/\sqrt m). If the planner maximizes the empirical objective exactly, ϵopt=0\epsilon_{\rm opt}=0. The estimation bound holds uniformly over covered Markov policies, so changes in policy occupancy within the covered model are controlled. It also holds simultaneously for all P≥0P\ge0, allowing the penalty to be chosen from the same data. Substituting ϵeff\epsilon_{\rm eff} into Theorem 1 gives a safety bound with explicit dependence on coverage, optimization, and regularization. In particular, the dependence of Em(P)E_m(P) on PP shows why increasing the penalty alone need not improve the learned-policy guarantee.

The implemented penalty. The theory connects the implemented penalty to the event-price threshold under explicit structural conditions. Proposition 6 broadens Appendix B.4 from a single terminal cost event to at least mm cost events at arbitrary positions within a failing segment of length LL. In the deterministic repeated-segment model, it gives the sharp bound

ρ0≤[LΔrm−r+]+,\rho_0\le\left[\frac{L\Delta_r}{m}-r_+\right]_+,

where Δr=r+−r−.\Delta_r=r_+-r_-. For nonnegative rewards, if the dataset contains their positive maximum r+r_+, then PD=r+P_{\mathcal D}=r_+. Corollary 2 guarantees PD>ρ0P_{\mathcal D}>\rho_0 when 2mr+>LΔr2mr_+>L\Delta_r. In particular, m>L/2m>L/2 suffices even when r−=0r_-=0, a case not covered by the sufficient condition for PDP_{\mathcal D} in Appendix B.4. The condition quantifies when the repeated charges for a sustained violation outweigh its reward advantage. Thus the extension connects the practical reward scale to violation duration, while retaining the repeated-segment assumptions.

The penalty sweep provides complementary evidence for the implemented scale. Half the default remains within budget on all seven Bullet tasks, and one tenth remains within budget on six of seven, using mean evaluation costs (Table 14).

Discounting. We have extended the guarantees to the discounted objective used in training. Proposition 3 applies to every 0<γ≤10<\gamma\le1, including ReBRAC's γ=0.99\gamma=0.99. The zero-cost feasible set is unchanged, and the optimal reward and event price are defined for the discounted objective. To relate the result to the undiscounted evaluation budget, we use

nπ^≤γ−(H−1)nγπ^.n^{\widehat\pi}\le\gamma^{-(H-1)}n_\gamma^{\widehat\pi}.

At γ=0.99\gamma=0.99, the conversion factor is approximately 7.47.4 for H=200H=200 and 2.3×1042.3\times10^4 for H=1000H=1000. Thus the discounted extension can give a weak undiscounted guarantee for long horizons. The remaining question of a numerical bound for the evaluated ReBRAC policies is addressed in our response to your Question 1.

AuthorsOpenReview ↗

Response to reviewer HwD3 (Weakness 2)

W2. Goal-task performance and the cost of conservatism.

This is a fair concern. The Goal tasks are the main exception to the method's competitive returns. On the fifteen non-Goal tasks, our default has the highest feasible return on eight and trails the best feasible baseline by at most 0.120.12 on the other seven (Table 1). The Goal sweeps show that excessive penalty strength contributes to the poor performance there. With a common multiplier of 0.250.25 and the original budget B=10B=10, return and goal completion improve on all four tasks, and all twelve training seeds remain within budget in mean evaluation cost (Tables 11 and 12).

TaskReturn at defaultReturn at 0.25Cost/budget at 0.25Goals per episode, default → 0.25
CarGoal10.030.150.092.08 → 3.91
CarGoal2−0.010.110.370.59 → 1.69
PointGoal1−0.040.080.020.39 → 1.91
PointGoal20.000.130.190.80 → 1.48

Proposition 7 identifies a mechanism for this conservatism. Replacement removes the original reward on each cost-bearing transition as well as imposing the penalty, so useful progress involving permitted cost can become unattractive under the relabeled objective. On CarGoal1, a quarter of the default adds about 1.81.8 goals per episode for approximately 0.50.5 additional cost. Mean cost rises only from 0.40.4 to 0.90.9, well below the budget of 1010.

The smaller multiplier 0.10.1 improves return further. Both PointGoal tasks remain within budget for every training seed, reaching returns 0.270.27 and 0.280.28. On PointGoal1, this exceeds BC-Safe's return of 0.220.22 while using less of the budget, with C/B=0.27C/B=0.27 compared with 0.910.91. PointGoal2 uses 0.77B0.77B. The two CarGoal tasks exceed the budget at this multiplier. BC-Safe remains better on CarGoal1 at B=10B=10, with return 0.200.20 compared with 0.150.15 for replacement at 0.250.25.

At the larger budget B=20B=20, the policies trained with multiplier 0.10.1 satisfy all four task budgets in mean evaluation cost. In particular, the CarGoal1 and CarGoal2 policies obtain returns 0.510.51 and 0.220.22 with mean costs approximately 17.417.4 and 13.013.0. We evaluate the same trained policies against the larger budget.

The Goal ablations also compare relabeling rules with the same backbone (Table 13). Replacement at 0.250.25 and the adaptive penalty both average return 0.120.12, but replacement satisfies all four task budgets while the adaptive penalty satisfies two. Cost-critic relabeling satisfies all four with mean return −0.02-0.02. Without relabeling, return is higher but every task exceeds its budget. These results establish that substantially better goal completion and return are compatible with the deployment budgets. They do not identify the optimal zero-cost return.

AuthorsOpenReview ↗

Response to reviewer HwD3 (Questions)

Q1. Is Theorem 1 numerically nonvacuous on an evaluated task at the implemented penalty?

For the continuous-control tasks, we have not established a numerical bound below the trivial maximum. We can establish a nonvacuous bound at the implemented penalty in the exact finite construction of Appendix F.5, where all the required quantities can be computed.

The first bound in the revised Theorem 1 does not require the event price. The revision uses the fact that only zero-cost transitions retain their original rewards to replace the submitted denominator P+r−P+r_- with P+r+P+r_+. At P=PDP=P_{\mathcal D}, the revised bound is

nπ^≤Hr+−JRπ0+ϵRLPD+r+.n^{\widehat\pi}\le\frac{Hr_+-J_R^{\pi_0}+\epsilon_{\rm RL}}{P_{\mathcal D}+r_+}.

In the three-step example, the safe branch earns 0.20.2 per step with no cost. The risky branch earns 11 per step and incurs unit cost on its final two steps. Both branches are covered by the dataset, so PD=r+=1P_{\mathcal D}=r_+=1 and JRπ0=0.6J_R^{\pi_0}=0.6. We can compute their values exactly from the deterministic data. A planner that maximizes the relabeled return therefore has ϵRL=0\epsilon_{\rm RL}=0. The submitted bound gives (3−0.6)/(1+0.2)=2(3-0.6)/(1+0.2)=2, the largest possible event count in this instance. The revised first bound gives

nπ^≤3−0.62=1.2<2.n^{\widehat\pi}\le\frac{3-0.6}{2}=1.2<2.

The second bound in the revised Theorem 1 is nπ^≤ϵRL/0.8n^{\widehat\pi}\le\epsilon_{\rm RL}/0.8, since ρ0=0.2\rho_0=0.2. It certifies zero cost for an exact optimizer and expected cost at most 0.50.5 for optimization error 0.40.4, below the budget B=1B=1. This bound is attained. A policy choosing the risky branch with probability 1/41/4 has relabeled return 0.20.2, exactly 0.40.4 below the optimum, and expected cost 0.50.5 (Appendix F.5).

For the continuous-control tasks, the missing step is to establish valid reward bounds and justify the comparator return and learning error numerically. Proposition 4 provides one set of sufficient learning assumptions, but we have not verified them for ReBRAC.

Q2. What does replacement add when safety levels are matched?

At equal or closely matched achieved mean costs, the two rules obtain comparable returns. The practical advantage demonstrated by the sweep is that replacement remains feasible over a wider range of penalty scales.

Table 5 contains eleven same-task replacement–addition pairs whose mean episode costs differ by at most 0.10.1. Addition has higher return in five pairs, replacement in four, and two tie. Every normalized-return difference is at most 0.030.03. The complete set of these closely matched comparisons therefore supports comparable returns.

In the expanded seven-task comparison, we also select each rule's highest-return tested multiplier that satisfies the common budget B=5B=5. The returns differ by at most 0.010.01 on six tasks, with replacement higher by 0.060.06 on AntCircle (Table 15). This comparison uses the same deployment budget, although the selected policies need not achieve identical costs.

The difference in penalty sensitivity is substantial. At half the default, replacement satisfies all seven task budgets and addition satisfies four. At one tenth, replacement satisfies six and addition satisfies none (Table 14). On these binary-cost tasks, replacement assigns a costly transition reward −λPD-\lambda P_{\mathcal D}, while addition assigns r−λPDr-\lambda P_{\mathcal D}. For positive rr, replacement imposes a larger reward reduction at the same numerical multiplier.

Proposition 7 quantifies the resulting difference in the penalty required for zero-cost optimality. For nonnegative rewards, ρ0≤ρ0add≤ρ0+r+\rho_0\le\rho_0^{\mathrm{add}}\le\rho_0+r_+. In the example in Q1, ρ0=0.2<PD=1<ρ0add=1.2\rho_0=0.2<P_{\mathcal D}=1<\rho_0^{\mathrm{add}}=1.2. Replacement selects the zero-cost branch, while addition selects the branch with cost 22 and violates budget 11. This explains the difference in penalty thresholds, while the benchmark comparisons above establish comparable returns at matched safety.

Reviewer_CabKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether explicit cost critics are necessary for offline safe reinforcement learning when safety violations are directly observed in the offline dataset. The authors propose RELABEL(COST), a simple approach that retains all transitions but replaces the reward of each transition with positive observed cost by a fixed negative value. A standard behavior-regularized offline RL algorithm, ReBRAC, is then trained on the transformed dataset without learning a cumulative-cost critic, feasibility model, Lagrange multiplier, or using the deployment budget during training.

They provide theoretical analysis explaining when local cost relabeling can control the expected number of cost events and recover an optimal zero-cost policy. It also studies how penalties associated with delayed cost events can influence earlier decisions through ordinary Bellman credit assignment, and argues that complete feasible trajectories are not necessary when useful local transition-level information remains covered by the dataset. The role of behavior regularization is further analyzed as a mechanism for preventing policy improvement toward unsupported actions.

Empirically, RELABEL(COST) satisfies the safety budget on all evaluated tasks and achieves surprisingly strong performance in the main results. The controlled ablations are also convincing and well designed.

Critical strengths
  1. A simple method exposes an interesting conceptual point about offline safe RL.

The strongest aspect of the paper is the empirical and theoretical evidence that an explicit cumulative-cost critic may be unnecessary in some offline safe RL regimes. The separation between where safety supervision enters (observed local cost events) and how its consequences propagate temporally (the ordinary reward critic) is conceptually useful.

  1. The controlled experiments provide direct evidence for the proposed mechanism.

The backbone-matched ablations are particularly valuable. With ReBRAC fixed, no relabeling violates the safety budget, whereas direct relabeling produces safe policies across the evaluated tasks. The experiments removing all feasible trajectories while retaining local transition information further support the paper’s central argument. Conversely, removing positive-cost transitions severely degrades safety, providing complementary evidence that observed violation events carry important supervision. The combination of these experiments makes the empirical argument considerably stronger than simply reporting improved benchmark performance against unrelated baselines.

Critical weaknesses
  1. Limited methodological novelty.

While the question investigated in this paper is interesting and potentially important, the proposed method itself offers limited methodological novelty. RELABEL(COST) essentially replaces the reward of transitions with observed positive cost by a fixed negative value and then applies an existing behavior-regularized offline RL algorithm (ReBRAC). Reward penalization for unsafe transitions is not new. In fact, the ablation showing that a simple fixed additive penalty achieves very similar performance further raises the question of whether reward replacement itself constitutes a sufficiently novel methodological contribution.

  1. The optimized objective differs from the standard budget-constrained Safe RL objective.

The objective induced by RELABEL(COST) is not equivalent to the conventional Safe RL objective of maximizing cumulative reward subject to a cumulative-cost budget. With a sufficiently large replacement penalty, the proposed method strongly favors policies that avoid positive-cost events altogether, whereas the standard constrained objective permits a policy to incur nonzero cost as long as the expected cumulative cost remains within the prescribed budget. Although the paper acknowledges this distinction theoretically, I consider it an important limitation because the proposed method addresses a somewhat different optimization objective from the general Safe RL problem it is compared against.

Questions for authors
  1. How large is the empirical gap between the best zero-cost policy and the best policy satisfying the nonzero deployment budget?
  2. How dependent is the result on ReBRAC? How does the method perform when the same relabeling strategy is combined with another behavior-regularized offline RL backbone, such as TD3+BC?
  3. How sensitive is the method to very sparse or strongly delayed cost signals in practice?
2 comments
AuthorsOpenReview ↗

Response to reviewer CabK (Weaknesses)

Thank you for your careful review and for your positive assessment. We first address the two weaknesses (in this message) and then the three questions in order (in the next message).

W1. Limited methodological novelty.

The penalty mechanism is simple and established. Our contribution is the empirical and theoretical evidence for when cumulative-cost estimation is unnecessary and how useful transitions from infeasible trajectories can support safe learning.

With ReBRAC held fixed, removing cost relabeling violates the budget on every ablation task. Both local penalty rules satisfy all the matched task budgets, while the tested adaptive and cost-critic alternatives do not consistently improve return or feasibility (Figure 1, Tables 3 and 4). Relabel(Cost) also remains within budget on all seven Bullet tasks after every trajectory with cost at most 10 is removed. Removing the positive-cost transitions instead produces policies that exceed the budget on all seven tasks (Section 5.3).

The new comparisons also show that more elaborate cost labels can make learning worse. CODAC (Conservative Offline Distributional Actor Critic, Ma et al., NeurIPS 2021) learns a conservative distribution over future returns. Our CODAC-based relabeler uses a distributional cost critic and thresholds its predicted mean cost plus one standard deviation. Restricting the CODAC mask to observed cost events improves feasibility from two of seven tasks to all seven (Table 16). Appendix F.3 explains one way this can happen even with an exact behavior-policy cost critic. The critic can assign a high cost-to-go because the recorded behavior chooses an unsafe continuation, although a safe continuation is covered. In the example, critic-based relabeling earns 0.75 and local relabeling earns 2, with zero cost under both policies. The loss comes from the labeling rule without any cost-estimation error.

The additive baseline's success strengthens the finding that a separate cumulative-cost critic is unnecessary for these results. Replacement has a practical advantage in its tolerance to smaller penalty multipliers. At half the default, it satisfies all seven Bullet budgets, compared with four for addition. At one tenth, replacement satisfies six and addition satisfies none (Table 14). Equal multipliers impose different reward reductions under the two rules. When each rule uses its highest-return tested feasible multiplier, returns differ by at most 0.01 on six tasks, with replacement higher by 0.06 on AntCircle (Table 15).

W2. The objective differs from the budget-constrained objective.

The objectives are not equivalent. We use a conservative surrogate for the budget-constrained problem. The comparison remains meaningful because every method is evaluated on the original reward under the same deployment budget, with higher return preferred among feasible policies. The limitation is that the conservative target can sacrifice reward when spending some budget is beneficial. Proposition 1 bounds this loss.

AuthorsOpenReview ↗

Response to reviewer CabK (Questions)

Q1. The empirical gap between the best zero-cost and budget-feasible policies.

The gap between the two benchmark optima is unknown. Among the learned policies in Table 1, our default has the highest feasible return on eight of the fifteen non-Goal tasks and trails the best feasible baseline by at most 0.12 in normalized return on the remaining seven. The larger shortfalls occur on the Goal tasks, where reducing the penalty recovers much of the lost return while maintaining feasibility.

For the Goal tasks, the following table compares the default with the highest-return feasible choice among the tested replacement multipliers at B=10B=10 (Table 11).

TaskReturn at defaultReturn at selected multiplierSelected multiplierCost/budget
CarGoal10.030.150.250.09
CarGoal2−0.010.110.250.37
PointGoal1−0.040.270.10.27
PointGoal20.000.280.10.77

All twelve training seeds at these selected multipliers remain within budget in mean evaluation cost. The return deficits relative to the best feasible Table 1 baseline shrink to 0.05 on CarGoal1 and 0.01 on CarGoal2. On PointGoal1, return exceeds BC-Safe's 0.22 while using less of the budget, with C/B=0.27C/B=0.27 compared with 0.91. No other Table 1 method is feasible on PointGoal2. At the common multiplier 0.25, all four tasks also improve goal completion, by approximately two to five times (Table 12).

Q2. Dependence on ReBRAC.

The result carries over to TD3+BC, although ReBRAC gives higher return and more consistent feasibility. With default TD3+BC hyperparameters and 200,000 training steps, the same relabeling satisfies seven of eight Bullet mean-cost budgets and obtains mean normalized return 0.57. ReBRAC satisfies all eight at return 0.66 (Table 18). The only task-level violation is on DroneRun, where one of the three TD3+BC seeds exceeds the budget.

Q3. Sparse or strongly delayed cost signals.

The method remains within budget under the tested label delays and partial label removal, with the largest loss of safety margin on the task with the shortest cost events. We alter only the training labels and evaluate using the original costs. A uniform five-step delay and a state-dependent ten-step delay affecting about 45% of events both leave all four tested tasks within budget in mean evaluation cost (Table 21).

AntVelocity is the most sensitive. Its mean cost/budget ratio rises from 0.04 to 0.76 and 0.81 under the two delays, while normalized return remains 0.99. Its median cost event lasts two steps, compared with 9, 14, and 26 steps on the other three tasks. For a two-step event, a five-step shift moves both labels past the original event. Events longer than five steps retain some overlap between the shifted labels and the original event.

To test missing supervision, we randomly remove half the positive-cost labels on AntVelocity. Return remains 0.99 and the cost/budget ratio is 0.15.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response and Summary of Updates

We thank the reviewers for their careful reading. The paper asks what safety information is needed for effective offline policy improvement and when broader cost labels can reduce performance. We combine controlled experiments and theory to show when observed cost events support safe learning without a separate cumulative-cost critic. All methods are evaluated on the original reward under the same benchmark budgets.

The submitted experiments isolate the contribution of cost supervision within a behavior-regularized learner. With ReBRAC held fixed, removing cost relabeling violates the budget on every ablation task. Both fixed local penalty rules perform well, while the tested adaptive and cost-critic alternatives do not consistently improve the results. On the seven Bullet tasks, policies remain within budget after all budget-feasible trajectories are removed. Removing those transitions instead sharply reduces safety (Figure 1, Section 5.3).

The additional experiments address three questions raised by the reviews.

  • Richer cost signals. On seven Bullet tasks, observed-cost relabeling achieves the highest mean return and satisfies every mean-cost budget. Classifier, behavior cost-Q, suffix, and CODAC relabeling all have lower mean returns and violate at least one task's budget. Restricting CODAC labels to observed cost events restores feasibility on all seven tasks (Tables 16 and 17). CODAC is the conservative distributional actor-critic method of Ma et al. (2021), adapted here to estimate cost distributions for relabeling.
  • Goal-task performance. A common penalty multiplier of 0.25 gives positive return and more goals reached on all four tasks. All twelve training seeds remain within budget in their mean evaluation cost (Tables 11 and 12).
  • Replacement versus addition. Across eleven comparisons of replacement and addition on the same task at closely matched mean costs, normalized returns differ by at most 0.03 (Table 5). Replacement remains feasible over a wider range of tested penalty scales. At half the penalty used in the main experiments, it satisfies all seven Bullet budgets, compared with four for addition (Table 14).

The revised theory sharpens Theorem 1 and makes the learning-error term explicit for a finite covered model, accounting for estimation, optimization, and regularization (Appendix F.2). We also extend the analysis to discounted returns (Appendix F.1). The new results explain why trajectory-dependent labels can suppress useful safe continuations even with an exact cost critic, and give conditions under which the reward-scale penalty suffices for zero-cost optimization (Appendices F.3 and F.4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.