- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces STEP which is a safe offline RL method that aims to improve actions at test time while keeping their estimated costs within a given budget. Instead of learning a policy with safety constraints built into its training objective, STEP learns a behaviour policy using flow matching, along with reward and cost critics trained using SARSA. During deployment, it samples several candidate actions and refines them by switching between reward improvement and cost correction based on their estimated costs. It also restricts how far actions can move from their initial samples and uses an uncertainty-aware cost filter to select the final action.
The authors provide theoretical bounds on how refinement affects reward and cost, along with an analysis of repeated execution. They evaluate STEP on 28 tasks from three DSRL benchmarks, reporting that it meets the average cost budget on 26 tasks and achieves the highest feasible return on 16. They also study online fine-tuning on five tasks, where STEP maintains or improves returns while reducing costs.
Critical strengths
-
Practical and simple approach: The paper explores an interesting way to handle safety constraints at test time rather than during policy training. The primal-only switching mechanism is straightforward and could be useful in offline RL, where retraining a policy is not always feasible.
-
Promising experimental results: The evaluation covers 28 tasks across three benchmarks, with STEP meeting the average cost budget on 26 tasks and achieving the highest feasible return on 16. The online fine-tuning results are also encouraging, showing improved cost control without sacrificing much return.
Critical weaknesses
-
Limited evidence for the contribution of primal-only refinement: While the overall results are promising, 11 of the 28 tasks use K=0, where STEP reduces to constrained best-of-N sampling. Although Appendix F.3 provides aggregate comparisons with other refinement methods, it is difficult to assess how much the proposed switching mechanism contributes to the main results under the task-specific configurations.
-
Unclear hyperparameter selection: STEP uses task-specific values of K, ε, and κ (Table 4), and the ablations show that these choices affect the reward-cost trade-off. Although the authors discuss heuristic tuning, the procedure used to select the final configurations is not clearly explained. This makes it difficult to assess the robustness of the reported improvements.
Questions for authors
-
Contribution of refinement: How much does primal-only refinement improve over constrained best-of-N? Could the authors provide per-task comparisons, given that 11 tasks use K=0?
-
Hyperparameter selection: How were the task-specific values of K, ε, and κ selected? Was a separate validation procedure used?
-
Safety under repeated execution: How often does STEP fail to find a feasible candidate, and how does this affect the actual cost during deployment?
6 comments
Response to reviewer's concerns (2/2)
W2 & Q2. Hyperparameter selection
We thank the reviewer for this important point, and we apologize that the paper did not present the procedure clearly.
Procedure. We used the common procedure of offline RL, where papers usually select hyperparameters with evaluation [5]. We also used the convention of offline safe RL and the DSRL benchmark to compare methods [1] (See W1 & Q1). Based on this, we selected the deployment knobs on the 8-seed mean of the 20 evaluation episodes per seed in Table 1. The grid is , , and . We change one knob at a time from the defaults (Table 3) and use it with each value of . The slack is fixed to on all tasks and is not part of the selection.
After selecting parameters, we run them on additional episodes to obtain the values below.
| Return / Cost | Tasks with mean cost ≤ 1 | |
|---|---|---|
| STEP, selection (Table 1) | 0.41 / 0.74 | 26/28 |
| STEP, held-out (30 ep.) | 0.41 / 0.69 | 27/28 |
| STEP, one setting (, , ) | 0.23 / 0.62 | 24/28 |
| FISOR | 0.41 / 1.11 | 23/28 |
On the held-out episodes, STEP keeps the same return and satisfies the constraint on 27 of the 28 tasks. The per-task selection holds on episodes that were not used for selection. For the one setting, we selected the setting of the grid that satisfies the constraint on the most tasks, then the one with the highest return. With this setting, STEP satisfies the constraint on more tasks than FISOR, but its return is lower.
Revision.
- We stated the selection procedure in Appendix E.
Q3. Safety under repeated execution
Thank you for this question. The gate examines the candidates sampled from the prior and their refinements, which stay within of these samples. On Safety-Gymnasium, the prior trains on behavior data with a cost times the budget. Thus, on eight of the eleven tasks, the admitted set is not empty on approximately 1% of the decisions or less. On Bullet, it is not empty on of the decisions, and on MetaDrive, on .
When STEP admits no candidate, it uses the candidate with the lowest pessimistic cost in Eq. (10). The cost critic learns with SARSA from behavior data with a cost that is more than the budget. Thus, it estimates the cost-to-go of the behavior policy, and its value is almost always more than , also for safe actions. The table below gives the 10 tasks where the admitted set is empty on approximately of the decisions or more (all 28 tasks in Appendix F.7).
| Task | Admitted set not empty | Cost (Table 1) |
|---|---|---|
| CarButton1 | 0.0% | 0.92 |
| CarButton2 | 0.0% | 1.08 |
| CarPush1 | 0.4% | 0.74 |
| CarGoal1 | 0.4% | 0.56 |
| CarCircle1 | 0.0% | 0.80 |
| CarCircle2 | 0.0% | 0.96 |
| AntVel | 1.0% | 0.32 |
| HalfCheetahVel | 0.0% | 0.14 |
| AntCircle | 0.5% | 0.16 |
| DroneCircle | 0.0% | 0.74 |
On these 10 tasks, STEP satisfies the constraint on 9. Where refinement is used, the fallback of Eq. (10) selects among candidates that the cost branch has already corrected, and this brings the cost into the budget where best-of- cannot (CarCircle1: 2.75 → 0.80, CarCircle2: 6.72 → 0.96; see W1 & Q1).
If we can select the data, we can train the prior on safe data or on data generated to satisfy the constraint [6]. Then the prior can give safe candidates at the first step, as safe behavior priors do in safe RL [7, 8]. We expect that the admitted set will then be non-empty much more often.
Revision.
- We added a statement about the second branch of Eq. (10) in Appendix C.
- We provided the proportion of decisions with an admitted candidate for all 28 tasks in Appendix F.7.
Remarks. Appendix E states the selection procedure, and we will add a discussion of how the hyperparameters were selected to the appendix. A single configuration for all tasks (, , ) already satisfies the constraint on more tasks than every baseline. With per-task tuning of the deployment knobs, which the main text reports, the return rises from 0.23 to 0.41 and STEP satisfies the constraint on 26/28 tasks, on the same pre-trained model and without retraining; this result also holds on held-out episodes.
Closing. We believe our response above answers your questions and would appreciate it if you would consider increasing your score. Please let us know if you have any further questions, we would be very happy to answer.
We would like to kindly remind the reviewer that the discussion phase is approaching its deadline.
Once again, we deeply appreciate the insightful comments and suggestions. We hope our clarification and additional empirical studies could address the concerns raised by the reviewer. Should there be any leftover questions, please let us know, and we will make every effort to address them during the subsequent discussion period.
Response to reviewer's concerns (1/2)
We thank the reviewer for the careful review. We provide a response to the three questions below, and the revised manuscript includes each revision colored in blue.
Thank you for this question. Table 4 uses K=0 on 13 of the 28 tasks, where STEP reduces to constrained best-of-N. For the other 15 tasks, we compare STEP with constrained best-of-N below. For a fair comparison, we tuned best-of-N over its deployment configurations with the same rule in DSRL benchmark: 1) when mean cost ≤ 1, then the highest return, and 2) when mean cost >1, then the lowest cost.
Both use the same pre-trained prior, critics, candidate budget, and evaluation episodes. Note that the table below reports reward and cost as return/cost.
Best-of-N satisfies the constraint in 10 of the 15 tasks and STEP in 14. For all tasks, Best-of-N satisfies the constraint in 22 out of 28 and STEP in 26. On the 10 tasks where both satisfy the constraint, the return of STEP is the same or higher on every task. On the other 5 tasks, best-of-N, even its lowest-cost configuration, is 1.5∼6.7 times the budget, and refinement decreases the cost in 39 of the 40 seeds. Therefore, the higher return of best-of-N on these tasks is not feasible under the DSRL criterion.
Related to
Q3. The effect of refinement depends on the gate. On MetaDrive, where the admitted set is often not empty, refinement also takes reward steps, and the return is the same or higher on all five tasks. On Safety-Gymnasium, where it is almost always empty, about 99% of the refinement steps are cost corrections. These let STEP satisfy the constraint with a less conservative configuration than best-of-N, which gives a higher return (e.g., CarPush1, AntVel, and HalfCheetahVel). With the same configuration and K=0, STEP satisfies the constraint on only 5 of these 15 tasks, so the cost correction of refinement, not the configuration alone, lets STEP satisfy the constraint on these tasks.With K=0, STEP is a sample-based approximation of the closed-form solution of the offline problem, and Table 4 uses it on 13 tasks. With K≥1, an action can move from its prior sample by at most ϵ, and the same cost gate examines each refined action. Refinement contributes because best-of-N can only reselect prior samples, which are rarely admitted where the data is unsafe, while the primal-only switch corrects a violating sample from the first step. With the same configuration, this turns 9 of the 10 tasks where best-of-N violates the constraint into tasks that satisfy it, whereas primal-dual refinement with the same K≤5 violates the constraint on average for every dual step size, since its multiplier cannot grow in so few steps (Appendix F.3).
Revision.