acceptodds
Under review as a conference paper at ICLR 2027

Primal-only Safe Policy Improvement at Test-time

Abstract

Safe reinforcement learning typically synthesizes a cost constraint into policy parameters during training, and the deployed policy does not re-check the constraint at the state being visited. We ask whether the constraint can instead be imposed at test time on sampled candidate actions. An action constructed at decision time can be checked against the current cost critic at the visited state, without waiting for a policy or a dual variable to update. We propose safe test-time extraction of policies (STEP), which draws candidates from a behavior prior and refines each with a primal-only switch. Specifically, a candidate follows the reward gradient while its estimated cost is below the threshold and receives a cost correction otherwise. The gate evaluates each candidate as a single deviation followed by the behavior policy, and safety under repeated STEP execution is assessed empirically. On DSRL tasks, the average cost of STEP over seeds stays within the budget on , and it attains the highest return among agents within the budget on . In online fine-tuning on five tasks, STEP maintains or slightly improves return while reducing cost, without an initial performance drop.

Primary area:
reinforcement learning
Keywords:
Safety, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MAUaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces STEP which is a safe offline RL method that aims to improve actions at test time while keeping their estimated costs within a given budget. Instead of learning a policy with safety constraints built into its training objective, STEP learns a behaviour policy using flow matching, along with reward and cost critics trained using SARSA. During deployment, it samples several candidate actions and refines them by switching between reward improvement and cost correction based on their estimated costs. It also restricts how far actions can move from their initial samples and uses an uncertainty-aware cost filter to select the final action.

The authors provide theoretical bounds on how refinement affects reward and cost, along with an analysis of repeated execution. They evaluate STEP on 28 tasks from three DSRL benchmarks, reporting that it meets the average cost budget on 26 tasks and achieves the highest feasible return on 16. They also study online fine-tuning on five tasks, where STEP maintains or improves returns while reducing costs.

Critical strengths
  1. Practical and simple approach: The paper explores an interesting way to handle safety constraints at test time rather than during policy training. The primal-only switching mechanism is straightforward and could be useful in offline RL, where retraining a policy is not always feasible.

  2. Promising experimental results: The evaluation covers 28 tasks across three benchmarks, with STEP meeting the average cost budget on 26 tasks and achieving the highest feasible return on 16. The online fine-tuning results are also encouraging, showing improved cost control without sacrificing much return.

Critical weaknesses
  1. Limited evidence for the contribution of primal-only refinement: While the overall results are promising, 11 of the 28 tasks use K=0, where STEP reduces to constrained best-of-N sampling. Although Appendix F.3 provides aggregate comparisons with other refinement methods, it is difficult to assess how much the proposed switching mechanism contributes to the main results under the task-specific configurations.

  2. Unclear hyperparameter selection: STEP uses task-specific values of K, ε, and κ (Table 4), and the ablations show that these choices affect the reward-cost trade-off. Although the authors discuss heuristic tuning, the procedure used to select the final configurations is not clearly explained. This makes it difficult to assess the robustness of the reported improvements.

Questions for authors
  1. Contribution of refinement: How much does primal-only refinement improve over constrained best-of-N? Could the authors provide per-task comparisons, given that 11 tasks use K=0?

  2. Hyperparameter selection: How were the task-specific values of K, ε, and κ selected? Was a separate validation procedure used?

  3. Safety under repeated execution: How often does STEP fail to find a feasible candidate, and how does this affect the actual cost during deployment?

6 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/2)

We thank the reviewer for the careful review. We provide a response to the three questions below, and the revised manuscript includes each revision colored in blue.


W1 & Q1. Contribution of primal-only refinement over constrained best-of-NN

Thank you for this question. Table 4 uses K=0K=0 on 13 of the 28 tasks, where STEP reduces to constrained best-of-NN. For the other 15 tasks, we compare STEP with constrained best-of-NN below. For a fair comparison, we tuned best-of-NN over its deployment configurations with the same rule in DSRL benchmark: 1) when mean cost ≤ 1, then the highest return, and 2) when mean cost >1, then the lowest cost.

Both use the same pre-trained prior, critics, candidate budget, and evaluation episodes. Note that the table below reports reward and cost as return/cost.

TaskSelected KKSTEP with K=0K=0 (best configuration)STEP with selected KK
CarPush130.18 / 0.410.27 / 0.74
CarCircle130.31 / 2.750.15 / 0.80
CarCircle210.44 / 6.720.09 / 0.96
AntVel10.76 / 0.200.85 / 0.32
HalfCheetahVel10.87 / 0.010.95 / 0.14
BallRun10.28 / 2.530.25 / 1.05
BallCircle10.43 / 2.690.26 / 0.97
CarRun50.76 / 0.980.77 / 0.89
DroneCircle50.42 / 0.610.42 / 0.74
AntRun10.62 / 1.520.59 / 0.98
easysparse10.52 / 0.730.53 / 0.94
easymean10.55 / 0.780.55 / 0.83
mediumsparse10.78 / 0.660.79 / 0.87
hardsparse50.41 / 0.960.43 / 0.84
hardmean50.39 / 0.750.41 / 0.95
Average (15 tasks)0.51 / 1.490.49 / 0.80
Average (10 tasks in the budget for both)0.56 / 0.610.60 / 0.73

Best-of-NN satisfies the constraint in 10 of the 15 tasks and STEP in 14. For all tasks, Best-of-NN satisfies the constraint in 22 out of 28 and STEP in 26. On the 10 tasks where both satisfy the constraint, the return of STEP is the same or higher on every task. On the other 5 tasks, best-of-NN, even its lowest-cost configuration, is 1.5∼6.71.5 \sim 6.7 times the budget, and refinement decreases the cost in 39 of the 40 seeds. Therefore, the higher return of best-of-NN on these tasks is not feasible under the DSRL criterion.

Related to Q3. The effect of refinement depends on the gate. On MetaDrive, where the admitted set is often not empty, refinement also takes reward steps, and the return is the same or higher on all five tasks. On Safety-Gymnasium, where it is almost always empty, about 99%99\% of the refinement steps are cost corrections. These let STEP satisfy the constraint with a less conservative configuration than best-of-NN, which gives a higher return (e.g., CarPush1, AntVel, and HalfCheetahVel). With the same configuration and K=0K=0, STEP satisfies the constraint on only 5 of these 15 tasks, so the cost correction of refinement, not the configuration alone, lets STEP satisfy the constraint on these tasks.

With K=0K = 0, STEP is a sample-based approximation of the closed-form solution of the offline problem, and Table 4 uses it on 13 tasks. With K≥1K \ge 1, an action can move from its prior sample by at most ϵ\epsilon, and the same cost gate examines each refined action. Refinement contributes because best-of-NN can only reselect prior samples, which are rarely admitted where the data is unsafe, while the primal-only switch corrects a violating sample from the first step. With the same configuration, this turns 9 of the 10 tasks where best-of-NN violates the constraint into tasks that satisfy it, whereas primal-dual refinement with the same K≤5K \le 5 violates the constraint on average for every dual step size, since its multiplier cannot grow in so few steps (Appendix F.3).

Revision.

  • We provided the table above and the proportion of reward steps for each task in Appendix F.8.
AuthorsOpenReview ↗

Response to reviewer's concerns (2/2)

W2 & Q2. Hyperparameter selection

We thank the reviewer for this important point, and we apologize that the paper did not present the procedure clearly.

Procedure. We used the common procedure of offline RL, where papers usually select hyperparameters with evaluation [5]. We also used the convention of offline safe RL and the DSRL benchmark to compare methods [1] (See W1 & Q1). Based on this, we selected the deployment knobs on the 8-seed mean of the 20 evaluation episodes per seed in Table 1. The grid is K∈{0,1,3,5}K \in \{0,1,3,5\}, ϵ∈{0.25,0.5}\epsilon \in \{0.25, 0.5\}, and κ∈{0,1,3}\kappa \in \{0,1,3\}. We change one knob at a time from the defaults (Table 3) and use it with each value of KK. The slack η\eta is fixed to −0.1-0.1 on all tasks and is not part of the selection.

After selecting parameters, we run them on additional episodes to obtain the values below.

Return / CostTasks with mean cost ≤ 1
STEP, selection (Table 1)0.41 / 0.7426/28
STEP, held-out (30 ep.)0.41 / 0.6927/28
STEP, one setting (K=1K=1, ϵ=0.5\epsilon=0.5, κ=1\kappa=1)0.23 / 0.6224/28
FISOR0.41 / 1.1123/28

On the held-out episodes, STEP keeps the same return and satisfies the constraint on 27 of the 28 tasks. The per-task selection holds on episodes that were not used for selection. For the one setting, we selected the setting of the grid that satisfies the constraint on the most tasks, then the one with the highest return. With this setting, STEP satisfies the constraint on more tasks than FISOR, but its return is lower.

Revision.

  • We stated the selection procedure in Appendix E.

Q3. Safety under repeated execution

Thank you for this question. The gate examines the candidates sampled from the prior and their refinements, which stay within ϵ\epsilon of these samples. On Safety-Gymnasium, the prior trains on behavior data with a cost 2∼182 \sim 18 times the budget. Thus, on eight of the eleven tasks, the admitted set is not empty on approximately 1% of the decisions or less. On Bullet, it is not empty on 0∼78%0\sim78\% of the decisions, and on MetaDrive, on 14∼81%14\sim81\%.

When STEP admits no candidate, it uses the candidate with the lowest pessimistic cost in Eq. (10). The cost critic learns with SARSA from behavior data with a cost that is more than the budget. Thus, it estimates the cost-to-go of the behavior policy, and its value is almost always more than dd, also for safe actions. The table below gives the 10 tasks where the admitted set is empty on approximately 99%99\% of the decisions or more (all 28 tasks in Appendix F.7).

TaskAdmitted set not emptyCost (Table 1)
CarButton10.0%0.92
CarButton20.0%1.08
CarPush10.4%0.74
CarGoal10.4%0.56
CarCircle10.0%0.80
CarCircle20.0%0.96
AntVel1.0%0.32
HalfCheetahVel0.0%0.14
AntCircle0.5%0.16
DroneCircle0.0%0.74

On these 10 tasks, STEP satisfies the constraint on 9. Where refinement is used, the fallback of Eq. (10) selects among candidates that the cost branch has already corrected, and this brings the cost into the budget where best-of-NN cannot (CarCircle1: 2.75 → 0.80, CarCircle2: 6.72 → 0.96; see W1 & Q1).

If we can select the data, we can train the prior on safe data or on data generated to satisfy the constraint [6]. Then the prior can give safe candidates at the first step, as safe behavior priors do in safe RL [7, 8]. We expect that the admitted set will then be non-empty much more often.

Revision.

  • We added a statement about the second branch of Eq. (10) in Appendix C.
  • We provided the proportion of decisions with an admitted candidate for all 28 tasks in Appendix F.7.

Remarks. Appendix E states the selection procedure, and we will add a discussion of how the hyperparameters were selected to the appendix. A single configuration for all tasks (K=1K=1, ϵ=0.5\epsilon=0.5, κ=1\kappa=1) already satisfies the constraint on more tasks than every baseline. With per-task tuning of the deployment knobs, which the main text reports, the return rises from 0.23 to 0.41 and STEP satisfies the constraint on 26/28 tasks, on the same pre-trained model and without retraining; this result also holds on held-out episodes.


Closing. We believe our response above answers your questions and would appreciate it if you would consider increasing your score. Please let us know if you have any further questions, we would be very happy to answer.

We would like to kindly remind the reviewer that the discussion phase is approaching its deadline.

Once again, we deeply appreciate the insightful comments and suggestions. We hope our clarification and additional empirical studies could address the concerns raised by the reviewer. Should there be any leftover questions, please let us know, and we will make every effort to address them during the subsequent discussion period.

Reviewer_xNP7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author proposed an algorithm STEP, that separates behavior modeling and value estimation from the choice of actions at deployment. It trains a flow-matching behavior prior and reward and cost critics. At deployment, it samples actions from the prior and refines them within a neighborhood of each original sample.

It takes normalized reward-gradient steps when predicted cost is low enough, and minimum-norm corrections to the linearized cost constraint otherwise. It keeps the original actions in the candidate pool, then chooses the highest-reward candidate that passes an ensemble-based cost check. If no candidate passes, it chooses the one with the lowest predicted cost.

Experiments cover 28 offline safe-RL tasks, with mean normalized cost within budget on 26 tasks, as well as online fine-tuning and ablations. The practical appeal is that the deployment rule can be made more or less conservative without retraining the behavior prior.

Critical strengths
  • The method makes constraint handling at deployment more flexible. The pretrained prior and critics can stay fixed while the extraction rule changes the safety–return tradeoff. Keeping the original candidates also gives a simple but valid guarantee, refinement cannot reduce the best admitted estimated reward compared with the same unrefined candidate set.
  1. The offline experiments cover 28 tasks. The appendix reports standard deviations, refinement sweeps, alternative updates, and critic-training ablations. The tables support the stated 26/28 count for mean cost within budget. Restoring FISOR's original architecture when the shared architecture hurt its performance is also a reasonable step toward a fair comparison.
Critical weaknesses
  1. The benefit of refinement is not clearly separated from sampling, selection, and tuning. Table 4 uses K=0 on 13 of 28 tasks, so those configurations reduce to constrained best-of-N. The aggregate ablations are useful, but they do not show the paired gain on each task over K=0 with the same prior, critics, uncertainty gate, and candidate budget. The settings for \epsilon and \kappa also vary by task. The paper should explain which data or seeds were used to choose them and whether final evaluation was kept separate. Without this information, it is hard to judge the contribution of the switching refinement itself.

  2. Some improvement and safety claims are stronger than the analysis supports. On p. 6, the paper claims that smoothness makes every normalized reward step improve estimated value. For Q_r (a)=-a^2, starting at a=0.01 with step length \alpha=0.05 moves to -0.04 and lowers value from -0.0001 to -0.0016, even with zero cost and ample slack. This support does not imply an arbitrary divergence bound, and a correction to a linearized constraint need not restore nonlinear feasibility. The valid local bounds concern Q^μ after one deviation; they do not certify the deployed policy's episodic budget, as Appendix B acknowledges. The experiments also need to distinguish a mean below budget from reliable budget satisfaction: CarCircle2 reports normalized cost 0.96±1.51.

Questions for authors
  1. Table 4 selects K = 0 on 13 of 28 tasks. For the tasks that use refinement, can you report paired results against K = 0 with the same trained prior, critics, uncertainty gate, and candidate budget? Please use existing paired runs where available and include deployment time. This would show how much of the reported gain comes from the new refinement step.

  2. How were K, \epsilon, and \kappa selected for each task? Please identify the validation data or seeds, the selection criterion, and whether any final evaluation results were used to make that choice. Were the baselines given comparable tuning budgets? This information is needed to interpret the main comparison.

  3. Can you report the fraction of seeds and episodes that exceed the budget, together with how often no candidate passes the cost check? Please also explain how the discounted threshold in Table 2 relates to the reported episodic budget when costs vary over time. These answers would clarify what the reported 26/28 mean-feasibility result establishes.

  4. For the smooth reward Q(a) = −a², a normalized step of length 0.05 from a = 0.01 moves to −0.04 and lowers the value, even with zero cost. Is a step-size condition or line search missing from the p. 6 statement or implementation? Please also clarify whether the p. 5 correction guarantees only linearized feasibility, and whether the support restriction is intended as a heuristic rather than a guarantee of the divergence constraint.

7 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/4)

We sincerely thank the reviewer for the careful and constructive review, and in particular for the concrete counterexample. We have colored the revised part of the manuscript in blue.


W1 & Q1. Effect of refinement compared with sampling and selection

Thank you for this question. Table 4 uses K=0K=0 on 13 of the 28 tasks, where STEP reduces to constrained best-of-NN. For the other 15 tasks, we report paired results against K=0K=0 with same configuration and best configuration. For a best configuration of best-of-NN, we tuned it with the same rule in DSRL benchmark: 1) when mean cost ≤ 1, then the highest return, and 2) when mean cost >1, then the lowest cost.

Table reports normalized return / cost.

TaskSelected KKSTEP with selected KKSTEP with K=0K=0 (same config with selected K)Best K=0K=0 configuration
CarPush130.27 / 0.740.28 / 1.380.18 / 0.41
CarCircle130.15 / 0.800.43 / 5.630.31 / 2.75
CarCircle210.09 / 0.960.48 / 8.030.44 / 6.72
AntVel10.85 / 0.320.97 / 3.250.76 / 0.20
HalfCheetahVel10.95 / 0.141.01 / 16.20.87 / 0.01
BallRun10.25 / 1.050.29 / 2.860.28 / 2.53
BallCircle10.26 / 0.970.46 / 3.460.43 / 2.69
CarRun50.77 / 0.890.75 / 1.050.76 / 0.98
DroneCircle50.42 / 0.740.56 / 3.570.42 / 0.61
AntRun10.59 / 0.980.68 / 2.950.62 / 1.52
easysparse10.53 / 0.940.52 / 0.730.52 / 0.73
easymean10.55 / 0.830.55 / 0.780.55 / 0.78
mediumsparse10.79 / 0.870.78 / 0.750.78 / 0.66
hardsparse50.43 / 0.840.38 / 0.610.41 / 0.96
hardmean50.41 / 0.950.34 / 0.440.39 / 0.75
Average (15 tasks)0.49 / 0.800.57 / 3.440.51 / 1.49

With the same gate, K=0K=0 is in the budget on 5 of the 15 tasks, the best K=0K=0 configuration on 10, and STEP on 14 (over all 28 tasks, 22 for the best K=0K=0 configuration and 26 for STEP). In either column, wherever K=0K=0 has a higher return than STEP, its cost is above the budget, and wherever both are in the budget, the return of STEP is the same or higher.

The same-gate column isolates refinement. It lowers the cost on all ten Safety-Gymnasium and Bullet tasks; on Safety-Gymnasium, about 99% of the refinement steps are cost corrections. On MetaDrive, where K=0K=0 is already in the budget, reward steps raise the return on four of the five tasks and keep it on the fifth, while still satisfying the constraint.

Deployment time. With K=0K=0, STEP takes about 1.3 ms per action, and each refinement step adds about 0.35 ms, so K=5K=5 takes about 3 ms (Appendix F.4).

Revision.

  • We added the table above and the proportion of reward steps for each task in Appendix F.8.
AuthorsOpenReview ↗

Response to reviewer's concerns (2/4)

W1 & Q2-1. Selection of KK, ϵ\epsilon, and κ\kappa

We thank the reviewer for this important point, and we apologize that the paper did not give the procedure clearly.

Procedure. We used the common procedure of offline RL, where papers usually select hyperparameters with evaluation [5]. We also used the convention of offline safe RL and of DSRL benchmark to compare methods [1] (See W1 & Q1). Based on this, we selected the deployment knobs on the 8-seed mean of the 20 evaluation episodes per seed of Table 1. The grid is K∈{0,1,3,5}K \in \{0,1,3,5\}, ϵ∈{0.25,0.5}\epsilon \in \{0.25, 0.5\}, and κ∈{0,1,3}\kappa \in \{0,1,3\}. We change one knob at a time from the defaults (Table 3) and use it with each value of KK. The slack η\eta is fixed to −0.1-0.1 on all tasks and is not part of the selection.

After selecting parameters, we run them on additional episodes to obtain the values below.

Return / CostTasks with mean cost ≤ 1
STEP, selection (Table 1)0.41 / 0.7426/28
STEP, held-out (30 ep.)0.41 / 0.6927/28
STEP, one setting (K=1K=1, ϵ=0.5\epsilon=0.5, κ=1\kappa=1)0.23 / 0.6224/28
FISOR0.41 / 1.1123/28

On the held-out episodes, STEP keeps the same return and satisfies the constraint on 27 of the 28 tasks. The per-task selection holds on episodes that were not used for selection. For the one setting, we selected the setting of the grid that satisfies the constraint on the most tasks, then the one with the highest return. With this setting, STEP satisfies the constraint on more tasks than FISOR, but its return is lower.

Tuning of the baselines. The baselines did not get a per-task tuning budget comparable to the knobs of STEP. The knobs of STEP act at deployment on one checkpoint, whereas the return–cost trade-off of each baseline is fixed at training time, so per-task tuning of a baseline would require retraining on each task. The one-setting row above is the comparison with equal tuning. Each baseline uses the hyperparameters from its paper or official implementation, which target the same DSRL tasks and budgets (Appendix D.2). For FISOR, the shared architecture satisfies the constraint on only 3 of the 11 Safety-Gymnasium tasks, so we use its original configuration (8 of 11; Appendix F.5), as the reviewer notes.

Revision.

  • We stated the selection procedure in Appendix E.
AuthorsOpenReview ↗

Response to reviewer's concerns (3/4)

Q3. Seeds and episodes with a cost more than the budget, the empty candidate set, and the discounted threshold

Thank you for this question. We provide each answer over three sub-questions below.

Seeds and episodes. Table 1 uses the DSRL protocol [1]. It gives the normalized cost as the mean of 8 seeds × 20 episodes. A method satisfies the constraint on a task when this mean is 1 or less. We agree that a mean does not show reliable satisfaction, and safe RL also uses other criteria [3, 4]. For each method, the table below gives the number of tasks and of runs that satisfy the constraint (Appendix F.6).

MethodTasks with mean cost ≤ 1Runs with cost ≤ 1 (of 224)
Safe-BC15/28136 (61%)
CPQ15/28130 (58%)
SafeFQL9/28101 (45%)
FISOR23/28171 (76%)
STEP26/28190 (85%)

STEP satisfies the constraint on the most tasks and in the most runs. Following the reviewer's suggestion, we checked the per-seed results of CarCircle2. The eight seeds are 0.31,0.23,0.20,0.28,0.32,0.97,0.510.31, 0.23, 0.20, 0.28, 0.32, 0.97, 0.51, and 4.914.91. Namely, seven seeds satisfy the constraint, and one is far above the limit.

How often is the feasible set empty? The admitted set is non-empty on about 1% of the decisions or less on eight of the eleven Safety-Gymnasium tasks, on 0∼78%0\sim78\% on Bullet-Safety-Gym, and on 14∼81%14\sim81\% on MetaDrive. On Safety-Gymnasium, the behavior data costs 2∼182\sim18 times the budget, so the predicted cost of any action followed by the behavior policy is above dd. STEP then selects the candidate with the lowest pessimistic cost. This is a one-step cost improvement over the behavior policy, not a certificate of the budget, as Appendix C of the revised PDF now states. Remark 4.1 does not apply to these decisions. On the 10 tasks where this happens on about 99%99\% of the decisions or more, STEP satisfies the constraint on 9.

Discounted threshold. Table 2 sets d=B(1−γcH)/((1−γc)H)d = B(1-\gamma_c^H)/((1-\gamma_c)H) with γc=0.99\gamma_c = 0.99, the discounted cost of spending the budget at the constant rate B/HB/H. For a trajectory with total cost BB, the discounted cost lies between γcH−1B\gamma_c^{H-1}B and BB, so Qc≤dQ_c \le d is exact for a constant rate, too strict for front-loaded cost, and too weak for back-loaded cost. Since the effective horizon of γc\gamma_c is about 100 steps and STEP repeats the check at every state, the check asks that the predicted discounted cost over roughly the next 100 steps, for one deviation followed by the behavior policy, stays at the budget rate. This is a heuristic link rather than a bound on the episode cost, and it applies only when a candidate passes the check.

We agree with the reviewer that Remarks 4.1 and 4.2 bound QμQ^\mu after one deviation. They do not make sure that the deployed policy stays in the episode budget. The submission gives this statement in the abstract ("safety under repeated STEP execution is assessed empirically") and in Section 4.3. Appendix B extends the remarks to repeated execution, but only with more assumptions. Namely, the relation to the episode-level constraint is empirical. The table above gives this data with the episode costs without a discount.

Revision.

  • We added the table above in Appendix F.6.
  • We added the proportion of decisions with an admitted candidate for each task in Appendix F.7.
  • We stated the second branch of Eq. (10) in Appendix C and repeated the execution in Section 4.3.
Reviewer_qQiqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STEP, a test-time action refinement method for safe offline reinforcement learning. STEP learns a behavior prior and reward/cost critics from offline data. At deployment, it samples several actions, improves reward when the estimated cost is below a threshold, and reduces cost when the threshold is exceeded. It then selects the highest-reward action that passes a pessimistic cost check.

On 28 DSRL tasks, STEP stays within the average cost budget on 26 tasks and achieves the highest return among budget-satisfying methods on 16 tasks. The paper also evaluates online fine-tuning and provides several ablations.

Critical strengths
  1. Useful test-time formulation. Applying the safety constraint directly to candidate actions at the current state is simple and practically meaningful. The primal-only switch avoids waiting for a dual variable to adapt.

  2. Broad experimental evaluation. The experiments cover 28 tasks and eight seeds, together with online fine-tuning, ablations, and runtime measurements. The results suggest that STEP can improve the empirical reward–cost balance.

Critical weaknesses
  1. The safety claim needs clearer limits. The paper describes the gate as checking a single action or single deviation, while the main result counts a task as safe using cost averaged over episodes and seeds. This is not the same as an episode-level safety guarantee. STEP also fails on CarButton2 and BallRun, and several successful tasks are very close to the budget boundary, but the paper does not analyze these cases.

  2. The evaluation of the reward–cost trade-off is incomplete. Section 5.3 uses task-specific values of KK, ϵ\epsilon, κ\kappa, and η\eta, but the main text does not explain how these values were selected. Figure 4 reports aggregated sensitivity rather than task-level trade-off curves. This makes it difficult to know whether the comparison is affected by task-specific tuning.

Questions for authors
  1. How were the task-specific values of KK, ϵ\epsilon, κ\kappa, and η\eta selected? Was test performance used? Please provide either one global setting or a clear validation-only selection procedure, and explain whether the baselines received comparable tuning.

  2. Why does STEP fail on CarButton2 and BallRun? Please report how often the feasible candidate set is empty, how well predicted cost matches realized cost, and how many individual seeds or episodes violate the budget. It would also be useful to report safety margins rather than only the 26/28 count.

  3. What exact safety claim is intended? The method checks actions locally, but the experiments report average episode cost. Please clarify this difference and provide reward–cost comparisons at matched cost or matched return. For example, on MetaDrive, STEP changes return from 0.56 to 0.57 relative to FISOR while increasing cost from 0.51 to 0.84.

6 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/3)

We sincerely appreciate the time and effort the reviewer dedicated to reviewing our work. This evaluation and constructive feedback are extremely valuable to us. Thank you for helping us further refine and strengthen the paper.


W1-1. Limits of the safety claim

The original manuscript stated these limits in three places below, but we agree that they were not prominent enough in the main text.

  • Abstract: the gate "evaluates each candidate as a single deviation followed by the behavior policy, and safety under repeated STEP execution is assessed empirically".
  • Section 4.3: the two remarks "bound single-deviation values".
  • Appendix B, after Proposition B.1: "the per-state gate alone does not certify Jc(πSTEP)≤BJ_c(\pi_{\text{STEP}}) \le B".

We would like to also clarify that our manuscript makes two statements.

  • Local. For an admitted action, the cost of this action followed by the behavior policy exceeds dd by at most the critic error plus a term linear in the refinement radius (Remark 4.1).
  • Empirical. The mean episode cost over all evaluation episodes and seeds is within the budget. This is the empirical counterpart of the CMDP constraint E[∑tct]≤B\mathbb{E}[\sum_t c_t] \le B [2], and the DSRL protocol [1] uses the same quantity for every baseline.

We do not make a claim for each episode.

Although several offline safe RL works follow the same convention, we agree that a mean does not show the result of each seed. The table below provides the number of tasks and of runs (28 tasks × 8 seeds) that satisfy the constraint.

MethodTasks with mean cost ≤ 1Runs with cost ≤ 1 (of 224)
Safe-BC15/28136 (61%)
CPQ15/28130 (58%)
SafeFQL9/28101 (45%)
FISOR23/28171 (76%)
STEP26/28190 (85%)

STEP satisfies the constraint on the most tasks and in the most runs. Additionally, we check that the worst task of STEP is near the limit (1.081.08 on CarButton2), whereas the baselines are far above it on the tasks where they violate the constraint.

Revision.

  • We moved the statement of Appendix B into Section 4.3.
  • In Table 1, we stated that "safe" means "mean cost within budget".
AuthorsOpenReview ↗

Response to reviewer's concerns (2/3)

W1-2 & Q2. Failure cases, near-boundary tasks, and the feasible set

Thank you for these requests. In addition to the 2020 episode evaluation of Table 1, we additionally run the evaluations for selected settings of each task on 3030 additional episodes per seed with new environment seeds, and we record the gate at each decision. These 3030 episodes were not used to select the knobs (see W2 & Q1).

TaskKKCost: selection (20 ep.)Cost: held-out (30 ep.)Seeds ≤ 1 (held-out)
CarButton100.920.896/8
CarButton201.080.707/8
CarGoal100.560.618/8
CarCircle130.800.777/8
CarCircle210.960.348/8
AntVel10.320.398/8
BallRun11.051.282/8
CarRun50.890.817/8
DroneCircle50.740.797/8
On the held-out episodes, BallRun is the only task in the table above that violates the constraint.

Failure cases. CarButton2 is above the budget on the 20 selection episodes (1.08) and within it on the 30 held-out episodes (0.70). The 20-episode estimate is noisy near the limit, and we keep CarButton2 as a failure in Table 1. BallRun is the only task where the single-deviation check is too weak: the gate admits a candidate on 42%42\% of the decisions, but 17%17\% of the admitted actions have a measured cost-to-go above dd. On the held-out episodes, its cost is 1.28, against 3.09 for FISOR.

How often is the feasible set empty? The admitted set is non-empty on about 1% of the decisions or less on eight of the eleven Safety-Gymnasium tasks, on 0∼78%0\sim78\% on Bullet-Safety-Gym, and on 14∼81%14\sim81\% on MetaDrive. On Safety-Gymnasium, the behavior data costs 2∼182\sim18 times the budget, so the predicted cost of any action followed by the behavior policy is above dd. STEP then selects the candidate with the lowest pessimistic cost. This is a one-step cost improvement over the behavior policy, not a certificate of the budget, as Appendix C of the revised PDF now states. Remark 4.1 does not apply to these decisions. On the 10 tasks where this happens on about 99%99\% of the decisions or more, STEP satisfies the constraint on 9.

Safety margins. We agree that the count of tasks alone is not sufficient. Table 1 gives the mean cost of each method on each task, i.e., its margin to the limit, and the table in W1-1 gives the proportion of runs that satisfy the constraint. For STEP, the held-out results above give the same quantities on episodes that were not used for selection.

Revision.

  • We stated in Appendix C that the second branch of Eq. (10) is a one-step cost improvement, not a certificate of the budget.
  • We added the proportion of decisions with an admitted candidate for each task in Appendix F.7.

Q3. Intended safety claim and matched comparisons

Thank you for this question. The claim is the one in W1-1: the check is local and single-deviation, and the experiments show that the mean episode cost satisfies the constraint.

Since STEP provides a return-cost curve from the same pre-trained model, we compare it with FISOR at matched cost. For each task, we take the best STEP return among the grid settings whose mean cost is at most that of FISOR. We compare on the 16 tasks where FISOR satisfies the constraint and STEP has a grid setting with a cost at or below that of FISOR.

TestbedTasks comparedFISOR return / costSTEP at matched costSTEP higher on
Safety-Gymnasium50.26 / 0.500.14 / 0.392/5
Bullet-Safety-Gym20.32 / 0.380.36 / 0.251/2
MetaDrive90.56 / 0.510.46 / 0.430/9

At matched cost, STEP has the higher return on 3 of the 16 tasks. On MetaDrive, we agree with the reviewer that FISOR has a higher return at the same cost. The main difference between the two methods is on the tasks where a comparison at matched cost is not possible: on CarGoal1, CarCircle1, CarCircle2, BallRun, and DroneRun, the cost of FISOR is 1.2∼71.2\sim7 times the budget, and STEP satisfies the constraint on four of them (BallRun at 1.05).

Revision.

  • We added the return-cost curve of STEP and the result of FISOR for each task in Appendix F.9.
AuthorsOpenReview ↗

Response to reviewer's concerns (3/3)

W2 & Q1. Selection procedure for the deployment knobs

We apologize that the main body did not state the selection procedure. We clarify it below and add an evaluation on episodes that were not used for selection.

Procedure. We train STEP with one procedure on all 28 tasks. The values for each task are deployment knobs: they act on the same pre-trained model, and no retraining is necessary. We selected them with the DSRL criterion [1] on the 8-seed mean of the 20 evaluation episodes per seed of Table 1: among the settings with mean cost ≤ 1, the one with the highest return. The grid is K∈{0,1,3,5}K \in \{0,1,3,5\}, ϵ∈{0.25,0.5}\epsilon \in \{0.25, 0.5\}, and κ∈{0,1,3}\kappa \in \{0,1,3\}. We change one knob at a time from the defaults (Table 3) and use it with each value of KK. The slack η\eta is fixed to −0.1-0.1 on all tasks and is not part of the selection.

After selecting parameters, we run them on additional episodes to obtain the values below.

Return / CostTasks with mean cost ≤ 1
STEP, selection (Table 1)0.41 / 0.7426/28
STEP, held-out (30 ep.)0.41 / 0.6927/28
STEP, one setting (K=1K=1, ϵ=0.5\epsilon=0.5, κ=1\kappa=1)0.23 / 0.6224/28
FISOR0.41 / 1.1123/28

On the held-out episodes, STEP has the same return as in Table 1 and a slightly lower cost. It satisfies the constraint on 27 of the 28 tasks, and BallRun is the only violation (see W1-2 & Q2). Therefore, the result of the per-task selection holds on episodes that were not used for selection.

With one setting for all tasks (third row), STEP satisfies the constraint on more tasks than FISOR. However, its return is lower. The per-task selection recovers the return from the same pre-trained model, and the held-out row shows that this return remains on episodes that were not used for selection. Appendix A notes that we tune the knobs as a heuristic and that a stopping rule for refinement is an open question.

Baselines. The manuscript provides the configuration of each baseline (Appendix C, D.2, and F.5).

  • Same problem setting. All methods use the same data normalization, action space, seeds, and evaluation episodes. All methods except FISOR also share the architecture and training hyperparameters (Appendix C, Table 2).
  • Hyperparameters of each method. The problem setting (e.g., DSRL tasks, budgets, and normalization) is the same as in the papers of these methods. Each baseline uses the hyperparameters from its paper or from the implementation that its paper provides (Appendix D.2).
  • FISOR. With the shared architecture, FISOR satisfies the constraint on only 3 of the 11 Safety-Gymnasium tasks. With the configuration from its paper, it satisfies the constraint on 8 of 11, so Table 1 reports the latter (Appendix C, D.2, and F.5, Table 6).
  • Tuning across tasks. Each baseline uses one configuration on all tasks, as in its paper, and its return–cost trade-off is fixed at training time. The one-setting row above gives STEP under the same condition.

Trade-off curves for each task. We agree that Figure 4 shows only the mean of many tasks. Thus, we made a figure with one panel for each task. Each panel shows each grid setting in the return–cost plane, the frontier, and all baselines. We also compared STEP with K=0K=0 on each task (Reviewer xNP7 W1 & Q1).

Revision.

  • We added the figure for each task in Appendix F.9, and compared STEP with K=0K=0 in Appendix F.8.
  • We stated the selection procedure and the fixed η\eta in Appendix E and Section 5.

We thank you once again for your feedback and hope our responses adequately address your concerns, we are happy to answer any other open questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Comment

We express our gratitude to AC for managing the review process and all three reviewers for their insightful feedback. We are pleased to present the updates we have made in response to valuable suggestions, as detailed below. In the revised manuscript, we color all changes in blue.

  • Safety at the seed and episode level. For each method, we give the tasks and runs that satisfy the constraint, and we report how often the gate admits a candidate (qQiq W1-1, W1-2 & Q2 xNP7 Q3 MAUa Q3).
  • Hyperparameter selection and held-out evaluation. We state the selection procedure and the grid. On 30 additional episodes per seed that were not used for selection, STEP keeps the same return and satisfies the constraint on 27 of the 28 tasks. We also report one setting for all tasks (qQiq W2 & Q1 xNP7 W1 & Q2-1 MAUa W2 & Q2).
  • Trade-off curves for each task. We also show the return-cost curve for each task, and we compare STEP with FISOR at the same cost (qQiq W2 & Q1, Q3).
  • Refinement and constrained best-of-NN. We also compare refinement with K=0K=0 on each task, for the same configuration and for the best configuration. We give the proportion of reward steps and the deployment time for each KK (xNP7 W1 & Q1 MAUa W1 & Q1).
  • Narrowed statements. We narrowed the statements about the scope of the safety claim, the reward step, the cost correction, the support restriction, and the fallback branch. (qQiq W1-1 xNP7 Q3, W2 & Q4 MAUa Q3).

Evaluation protocol. Some questions are about how we measure safety. We would like to clarify the evaluation protocol. Table 1 uses the standard protocol of the DSRL benchmark [1], and other offline safe RL methods, for example, FISOR [3], use it too. The protocol divides the episode cost by the budget and calculates the mean for all evaluation episodes and seeds. A method satisfies the constraint on a task when this mean value is 1 or less. Among the methods that satisfy the constraint, the method with the highest normalized return is the best.

The mean value is the empirical equivalent of the CMDP constraint E[∑tct]≤B\mathbb{E}[\sum_t c_t] \le B [2]. The paper uses the usual setting for offline safe RL, and the evaluation of each baseline uses the same setting.

We agree that a mean does not show the result of each seed, so the individual responses also give the runs that satisfy the constraint and a held-out evaluation.


One structure for offline and online, and the function of KK. STEP applies the constraint at test time, not in the policy objective. The training losses have no offline-only term. The prior uses flow matching, and the critics use SARSA. Thus, it is not necessary to change the objective between offline training and online fine-tuning. We also do not need to recalibrate the value estimates. This is a property of the structure, and it is correct for all values of KK.

In this structure, KK controls how far the action can go from the samples of the prior.

  • K=0K=0. With K=0K=0, STEP is constrained best-of-NN. This is a sample-based approximation of the closed-form solution of the offline problem for the admitted samples (Section 5.3). It can only select from the samples of the prior. Table 4 selects K=0K=0 on 13 of the 28 tasks.
  • K≥1K \ge 1. A refined action can be different from each prior sample by a maximum of ϵ\epsilon. In online fine-tuning, these actions go into the buffer, and STEP trains the prior on them with the same loss. Therefore, the prior can cover more actions after each update (Section 5.2).

We have left individual comments to address each reviewer’s questions and concerns and to clarify statements that were unclear. Please review each individual comment for more details!

Below are the references we use in all our responses.

References

[1] Z. Liu, et al. Datasets and benchmarks for offline safe reinforcement learning. arxiv 2023.

[2] E. Altman. Constrained Markov decision processes. Routledge 2021.

[3] Y. Zheng, et al. Safe offline reinforcement learning with feasibility-guided diffusion model. ICLR 2024.

[4] Y. Chow, et al. Risk-constrained reinforcement learning with percentile risk criteria. JMLR 2018.

[5] T. L. Paine, et al. Hyperparameter selection for offline reinforcement learning. arXiv 2020.

[6] Y. Yao, et al. OASIS: Conditional distribution shaping for offline safe reinforcement learning. NeurIPS 2024.

[7] D. Slack, et al. SAFER: Data-efficient and safe reinforcement learning via skill acquisition. arXiv 2022.

[8] J. Li, et al. Guided online distillation: Promoting safe reinforcement learning by offline demonstration. ICRA 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.