- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies strict zero-cost offline reinforcement learning and proposes RELABEL(COST). For each offline transition, it keeps the original reward when the observed cost is zero and replaces the reward by a fixed negative value when the cost is positive. It then trains a behavior-regularized offline RL algorithm, ReBRAC, on the relabeled data, without a cost critic, feasibility model, or Lagrange multiplier. The paper argues that ordinary reward Bellman backups propagate the penalty from delayed cost events to earlier decisions. It provides bounds relating the relabeled objective to the number of positive-cost events, an event-price characterization for exact recovery of the optimal zero-cost policy, delayed-cost analyses, and experiments on 19 DSRL tasks.
Critical strengths
-
The paper asks a meaningful question for offline safe RL: whether direct transition-level cost labels can sometimes replace an explicitly learned long-horizon cost model. Retaining useful portions of globally infeasible trajectories is a reasonable offline-data perspective.
-
The experiments include useful controls. In particular, comparing no relabeling, additive fixed penalties, adaptive penalties, and cost-critic relabeling helps isolate the role of the proposed signal. The results suggest that fixed penalties plus strong behavior regularization can be effective on the evaluated tasks.
Critical weaknesses
The proposed method is essentially the classical fixed reward-penalty approach, expressed using a local positive-cost indicator. Strict zero-cost RL is a legitimate and useful problem setting, but restricting the objective does not by itself make the fixed-penalty solution new. In the paper's notation, the relabeled objective is : it assigns the same price to every positive-cost event and then relies on the ordinary reward critic to propagate that price to earlier actions. The latter is standard Bellman credit assignment, while the former is exactly the fixed penalty mechanism used before the development of adaptive safe-RL methods. The event-price theorem formalizes the familiar sufficient condition , where is unknown and depends on the task and policy class; choosing neither estimates nor guarantees that it is large enough. More importantly, if the deployment budget is positive, a fixed price for all cost events cannot express that spending a small amount of budget may be worthwhile for a large reward gain, and it cannot adapt the reward-cost weighting to state, remaining budget, cost magnitude, or local uncertainty. These are precisely the practical limitations that motivated Lagrangian, constrained policy optimization, and state-dependent safety weighting in safe RL. The paper's theory therefore establishes when an old penalty method works under its assumptions, but does not resolve the reason fixed penalties are difficult to use. The empirical evidence does not fill this gap: the additive fixed-penalty ablation obtains similarly strong safety and return, so the results are consistent with a sufficiently large classical penalty plus behavior regularization being responsible for the gains. At most, the paper shows that this simple baseline is effective on benchmarks with favorable local cost coverage; it does not establish a new mechanism or a general replacement for adaptive cost modeling.
Questions for authors
What is the precise scientific contribution beyond a classical fixed-penalty baseline? Please explain why should be considered a principled solution to the task-dependent penalty-selection problem when the exact-recovery condition depends on the unknown . More importantly, can the authors provide a comparison against an otherwise identical fixed-penalty method and evaluate tasks where the optimal feasible policy deliberately spends part of a positive budget, where cost magnitudes differ, or where safety information is delayed and state-dependent? Such evidence is needed to show that local relabeling contributes beyond the fixed penalty itself and that eliminating a cost critic addresses a limitation of existing safe-RL methods rather than simply avoiding it by changing the objective.
5 comments
Response to reviewer WUYt (Questions 1 and 2)
Next, we address the first two questions you pose.
Q1. What is the precise scientific contribution?
We establish which cost information is sufficient for offline policy improvement in the studied regimes, what the dataset must retain, and why adding broader cost labels can make learning worse.
First, the comparisons hold the learner fixed. With ReBRAC, data, hyperparameters, and seeds unchanged, removing cost relabeling produces mean costs above ten times the budget on all ten ablation tasks. Local penalties satisfy every matched task's budget, while the tested adaptive and cost-critic alternatives do not consistently improve return or feasibility (Figure 1, Tables 3 and 4). Relabel(Cost) also satisfies all 19 main-table mean-cost budgets.
Second, complete feasible trajectories are unnecessary in these datasets. Relabel(Cost) remains within budget on all seven Bullet tasks after every trajectory with cost at most 10 is removed. Removing the positive-cost transitions instead leaves only 6% of evaluation episodes within budget (Section 5.3). This separates the need for observed violations from the availability of complete safe demonstrations.
Third, richer relabeling signals do not improve the seven-task comparison (Table 16).
CODAC (Conservative Offline Distributional Actor Critic, Ma et al., NeurIPS 2021) learns a conservative distribution over future returns. Our CODAC-based relabeler uses a distributional cost critic and thresholds its predicted mean cost plus one standard deviation.
| Relabeling signal | Tasks within mean-cost budget | Mean normalized return |
|---|---|---|
| Observed cost (ours) | 7/7 | 0.62 |
| Behavior-policy cost Q-function | 6/7 | 0.36 |
| Trajectory classifier | 5/7 | 0.27 |
| Suffix after the budget is exceeded | 3/7 | 0.40 |
| CODAC cost critic | 2/7 | 0.46 |
| CODAC restricted to observed cost | 7/7 | 0.52 |
CODAC's unrestricted mask covers more than 80% of transitions on five tasks late in training. Restricting its labels to observed cost events restores feasibility on all seven tasks and improves mean return (Table 17).
Appendix F.3 identifies a mechanism that can produce such a loss. A behavior-policy cost critic can assign a high cost-to-go to an action because the behavior subsequently chooses an unsafe continuation, even when a safe continuation is covered. In the example, with full coverage, exact planning, and an exact behavior cost critic, critic-based relabeling earns 0.75 while local relabeling earns 2. Both the zero-cost optimum and the budget-constrained optimum are 2. The loss therefore arises from the relabeling rule, without estimation error or a more conservative safety target. This is a concrete limitation of using behavior-dependent cost-to-go labels as offline reward supervision.
Q2. Why is a principled choice when is unknown?
For a positive budget, the appropriate penalty depends on the task, as the Goal sweeps illustrate. is a default for the zero-cost target, where every suffices under exact optimization. We use this default for three reasons.
-
Reward scale. is no greater than any reward in the dataset, and multiplying rewards by a positive constant rescales the penalty by the same constant. The default requires no safety-multiplier tuning.
-
A sharp sufficient condition. Under the deterministic repeated-segment assumptions of Appendix F.4, every covered failing segment incurs cost on at least of its transitions. Proposition 6 gives
where . This bound is attained within the model. For nonnegative rewards, when the dataset contains their positive maximum , we have . Corollary 2 then guarantees whenever . In particular, suffices. Sustained violations can therefore make the reward-scale default sufficient without estimating the exact event price.
- Empirical robustness. Half the default remains within budget on all seven Bullet tasks, and one tenth remains within budget on six of seven, using task-level mean evaluation costs (Table 14). Halving the penalty also increases mean normalized return from 0.62 to 0.66 (Table 5).
The segment model gives a sufficient regime; the support on the benchmarks is empirical. The default can lie below when a brief violation permits a long reward gain. In our response to Q3(b) in the next message, we give two fully solved examples showing when the chosen penalty is sufficient to recover the best zero-cost policy and when it is too small.
Response to reviewer WUYt (Question 3)
Q3. The requested comparisons and regimes.
(a) An otherwise identical fixed-penalty method. Figure 1 and Tables 3 and 4 provide this control. They use the same backbone, data, hyperparameters, and seeds, with only the reward rule changed to . At the default scale, both rules satisfy every matched task's budget and their normalized returns differ by at most 0.05. The expanded seven-task sweep gives the following feasibility counts.
| Multiplier | Replacement tasks within budget | Addition tasks within budget |
|---|---|---|
| 1 | 7/7 | 7/7 |
| 0.5 | 7/7 | 4/7 |
| 0.25 | 6/7 | 1/7 |
| 0.1 | 6/7 | 0/7 |
Counts use mean evaluation costs (Table 14). Replacement removes the original reward on costly transitions as well as imposing the negative value, so equal numerical multipliers do not mean equal effective charges. When each rule uses its best tested feasible multiplier, returns differ by at most 0.01 on six tasks; replacement is higher by 0.06 on AntCircle (Table 15). The observed advantage is a wider feasible range of tested multipliers. The comparison uses a common budget, without requiring identical achieved costs.
(b) A task whose optimal policy spends budget. The exact CMDP family in Appendix F.5 has a three-step instance with budget 1 and both paths covered by the dataset. The safe path earns 0.2 per step at no cost. The risky path earns 1 per step with unit cost on its last steps, so . The table reports original returns and expected costs, allowing randomized policies.
| Cost events on risky branch | Budget-optimal (return, cost) | Replacement at | Addition at |
|---|---|---|---|
For , . Both rules choose the risky path, which spends the full budget and attains the constrained optimum. For , , so replacement chooses the safe path. Addition chooses the risky path because its zero-cost threshold is , and its cost of 2 violates the budget. The constrained optimum instead mixes the two paths equally. Thus a fixed penalty can spend budget, but it does not automatically select the policy appropriate to a given budget.
The Goal sweeps provide complementary evidence from learned policies (Tables 11 and 12). At multiplier 0.25, return is positive and goal completion improves on all four tasks. All twelve training seeds remain within budget in their mean evaluation cost. At multiplier 0.1, return improves further, but both CarGoal tasks exceed their budgets. The exact example supplies the known optimum; the Goal experiments measure the achieved return–cost tradeoff.
(c) Different cost magnitudes. On nine MetaDrive datasets with fractional costs, replacement satisfies all nine mean-cost budgets with positive normalized return on every dataset; magnitude-weighted addition at the same scale satisfies one (Table 20). This compares event-based and magnitude-weighted penalties on nonbinary-cost tasks at the reported scale. Replacement treats positive-cost events equally, while the additive comparator retains their magnitude information.
(d) Delayed and state-dependent safety information. We perturb training labels and evaluate with the true costs. Both a uniform five-step delay and a state-dependent ten-step delay affecting about 45% of events leave all four tested tasks within budget (Table 21). AntVelocity loses the most safety margin. Its cost/budget ratio rises from 0.04 to 0.81 under the state-dependent delay, while normalized return remains 0.99. Its median cost-event duration is only two steps. These experiments test whether local supervision remains effective when the recorded labels are displaced in time.
Response to reviewer WUYt (Weaknesses)
Thank you for the detailed review. We first address the two weaknesses you identified.
W1. The theory and the difficulty of using fixed penalties.
A fixed price can express the reward–cost tradeoff described in the review. The difficulty is choosing a price that yields a policy within the desired budget. Under our replacement objective, a policy π2 with more expected cost events than π1 has higher relabeled return precisely when
Aπ2−Aπ1>P(nπ2−nπ1),
where Aπ is expected reward collected on zero-cost transitions and nπ is the expected number of cost events. The additional events are worthwhile whenever the additional retained reward outweighs their charge.
The benchmark constraint is on expected total cost. For a feasible finite-horizon CMDP with finite state and action spaces, optimizing over all randomized policies, linear-programming duality gives a scalar λB for which a budget-optimal policy maximizes JRπ−λBJCπ (Altman, 1999). A state-dependent multiplier is therefore not necessary to represent an optimum of this expected-cost problem. Lagrangian methods search for the appropriate scalar multiplier. Finding that multiplier and selecting a feasible optimizer, which can require randomization among tied policies, is the calibration problem. It is distinct from an inability of fixed prices to express reward–cost tradeoffs.
Our results demonstrate worthwhile budget spending under replacement itself. In the one-event instance of Appendix F.5, the default penalty selects the policy that spends the full budget and attains the constrained optimum. On the four Goal tasks, a quarter of the default penalty increases goal completion by about two to five times, with task-level mean costs using 2%–37% of the budget. Each of the twelve training seeds remains within budget in mean evaluation cost (Tables 11 and 12). In our response to Q3 part (b) below, we give the exact comparison and the learned-policy results.
For the zero-cost target, Theorem 2 establishes an exact threshold. Every P>ρ0 makes exact optimization of the relabeled objective recover the best zero-cost policies, whereas for 0≤P<ρ0 a positive-cost policy beats the zero-cost optimum. Thus, the penalty does not have to be calibrated to the budget for this target. Every value above ρ0 gives the same optimum. Proposition 6 bounds this threshold under an explicit model of sustained violations, giving a regime in which the reward-scale default suffices. The training target is stricter than the budget, and the evaluated problem is unchanged. Every method is measured on the original reward against the same benchmark budgets.
W2. What the successful additive baseline establishes.
The contribution is to identify when separate cumulative-cost estimation is unnecessary and why broader cost labels can reduce performance. Both rules use observed cost events and the same behavior-regularized learner. Their comparable performance shows that this finding extends across two local penalty rules. In Table 1, each of the eight baselines exceeds the budget on 7 to 15 of the 19 tasks, and the fixed penalty exceeds it on none.
The additional comparisons show that the choice of safety information matters. Broader labels reduce return, and restricting a learned label to observed cost events restores feasibility. The theory identifies a loss from behavior-dependent labeling even with an exact cost critic and identical zero-cost and budget-constrained optima. Together, these results go beyond establishing that a large penalty works; our response to your Q1 (see below) further clarifies this and provides the evidence.