acceptodds
Under review as a conference paper at ICLR 2027

Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data

Abstract

Learning constraint-satisfying policies from offline data without risky online interaction is crucial for safety-critical decision making. Conventional methods typically learn cost value functions from abundant unsafe samples to define safety boundaries and penalize violations. However, in high-stakes scenarios, risky trial-and-error is infeasible, yielding datasets with few or no unsafe samples. Under this limitation, existing approaches often treat all data as uniformly safe, overlooking safe-but-infeasible states-states that currently satisfy constraints but inevitably violate them within a few steps-leading to deployment failures. Drawing inspiration from the concept of knowledge-data integration, we leverage large language models (LLMs) to incorporate natural language knowledge into the policy to address this challenge. Specifically, we propose PROCO, a model-based offline safe reinforcement learning (RL) framework tailored to datasets largely free of violations. PROCO first learns a dynamics model from offline data and constructs a conservative cost function by grounding natural-language knowledge of unsafe states in LLMs, enabling risk estimation even without observed violations. Using the cost function and learned model, PROCO performs model-based rollouts to synthesize diverse counterfactual unsafe samples, supporting reliable feasibility identification and feasibility-guided policy learning. Across a range of Safety-Gymnasium tasks with exclusively safe or minimally risky training data, PROCO integrates seamlessly with a variety of offline safe RL algorithms and consistently demonstrates reduced constraint violations and improved safety performance compared to both the original methods and other behavior cloning baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Reinforcement Learning, Safe Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N6tFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Get a mathematical constraint function from an LLM agent
  2. Learn a dynamics model with the limited offline dataset
  3. Rely on PROCO to generate enough rollouts using this to generate unsafe (or close to unsafe) samples without needing to actually crash
  4. Pair with your favorite offline safe RL algorithm
Critical strengths

This is a cool idea and the logic is straightforward, though the use case is fairly narrow. Refreshingly, the paper is written quite well. The results and ablation study are nice.

Critical weaknesses

My main concern is that it seems like a big ask to rely on an LLM to derive a conservative cost function, then learn a dynamics model with the limited offline (and presumably only safe) dataset, and then rely on PROCO to generate enough rollouts using this to generate unsafe (or close to unsafe) samples. There is a fairly limited setting wherein this would work directly (pretty simple notions of safety, pretty simple dynamics that can be learned accurately from limited safe data). The papers own up to this with the assumptions that are made and the discussion section. I would like a summary of the analysis of the experiments and ablations section that would provide sort of a guide on when this would work well, when it would work even better (e.g. given a valid dynamic model), and when it would not work well.

I submitted the above to Claude and asked it to point out anything I missed. Here are some comments it gave back that I thought were relevant after checking:

  • The reward drop is large, and there’s no trivial-policy baseline. FISOR PROCO averages 0.14 reward versus 0.50 for FISOR and 0.36 for BC (Table 1). Several tasks are near zero: CarButton1 −0.03, CarButton2 0.00, PointButton1 0.01, CarPush2 0.01. A policy that stands still or barely moves would also look “safe” here. It’s worth asking for a do-nothing or minimal-motion baseline, or at least asking the authors to show that PROCO isn’t converging toward one. The headline “over 400% / 1011% safety improvement” is also never defined, so you might ask how it’s computed.
  • “No unsafe samples” isn’t quite the setting tested. Every experiment uses the 100-sample D_unsafe to validate the LLM cost function. The baselines don’t appear to get these samples, or the natural-language constraint. The fairer comparisons would be FISOR, LSPC, and CAPS trained with those 100 unsafe samples, and the baselines run on data relabeled with the LLM cost function. The “W/o Model” ablation partially covers the second for FISOR (avg cost 1.27), but not for the others.
  • The rollout horizon is H = 1. This directly supports the reviewer’s point about limited settings. The “model-based rollouts to simulate future evolution” are one-step lookaheads in practice (Appendix C.3, Table 2). H = 5 badly hurts navigation tasks (avg cost 4.41, Table 10). Most of the multi-step reasoning therefore comes from the conservative cost margin plus value propagation over dataset transitions, not from the model.
  • Assumption 3.5 bounds KL divergence, but Lemma A.2 (from Janner et al.) is applied as if it bounds TV. Going from KL to TV needs Pinsker’s inequality, which gives a √δ dependence rather than δ.

I then asked it to generate questions based on these points that I checked, which I then modified and put in the question section.

Questions for authors

Highest Priority:

  • Can you add a trivial policy baseline and report the reward and cost? Could you also show evidence that PROCO’s policies are completing meaningful task progress rather than converging toward inaction, for instance trajectory visualizations or task-specific progress metrics on the near-zero-reward tasks?
  • How are the reported safety improvements (e.g., “over 400%,” “1011%”) computed? Please be explicit
  • Could you report (a) FISOR, LSPC, and CAPS trained with the same 100 unsafe samples added to D, and (b) all three baselines trained on D relabeled with the final LLM-generated cost function, without model rollouts? Relatedly, how should the “no unsafe samples” framing be read, given that D_unsafe is used in every experiment?

Medium Priority:

  • How much of the multi-step feasibility reasoning described in Section 3.1 is due to the model, versus the conservative cost margin combined with value propagation over dataset transitions? Can you characterize when longer horizons help, for example as a function of model error? Can you also reconcile the H = 1 setting with the analysis in Theorems 3.4 and 3.7, which is framed in terms of a horizon H* to reach unsafe states?
  • Can you expand/clarify these points on where PROCO works well and where it fails in the discussion section? I would love an honest picture of when I should use PROCO and when I should not.

Lower Priority:

  • Can you clarify which divergence you rely on (KL vs TV) in Assumption 3.5/Lemma A.2?
9 comments

Thank you for your kind reviews and constructive suggestions. We have revised the manuscript to address your comments and marked all additions in blue. In the Conclusion, we now discuss the conditions under which PROCO is most applicable and the limitations that arise when those conditions are not met. We also added experiments and discussion in five appendix sections:

  • Appendix E.4 examines how providing specific examples of misclassified unsafe observations affects performance and the number of reflection rounds.
  • Appendix E.5 evaluates reflection using conservatism as the sole source of feedback when no unsafe samples are available.
  • Appendix E.6 compares cost functions tuned by the LLM with those using parameters chosen by a human.
  • Appendix E.9 examines how effectively PROCO uses cost information by comparing direct training on unsafe samples, relabeling with the generated cost function, and relabeling combined with model rollouts.
  • Appendix E.10 compares PROCO with a trivial policy that always outputs a zero action and reports the mean absolute action values on CarButton1 and CarButton2.

We appreciate your feedback and hope that these revisions address your concerns.

Q1 Discussion about when should and when should not use PROCO

Thank you for encouraging us to clarify when PROCO is most effective. We have revised the Conclusion to describe the three conditions under which PROCO is best suited and the limitations that arise when these conditions do not hold.

  1. The cost function can be expressed in code and evaluated from the available observations. The observations must contain all the information the code needs to compute the cost.

  2. An accurate dynamics model is available or can be learned reliably. When the dynamics are harder to learn, extending the rollout horizon can substantially lower reward and increase cost, as shown in Figure 3(b). When model error is small, Figure 3(c) shows that a moderate increase in the rollout horizon can make feasibility estimates more conservative, with little effect on overall performance.

  3. A small number of unsafe samples are available for reflection. Both our original ablation and the new experiments in Appendix E.5 show that PROCO performs best when it can use unsafe samples during reflection. Without these samples, reflection relies on conservatism alone. The resulting policy can be overly conservative on some tasks and too aggressive on others, leading to a modest decline in overall performance. Even in this setting, PROCO still substantially improves safety over the FISOR baseline.

Q2 Comparison with a trivial policy

Thank you for suggesting this comparison. We added a Trivial Policy baseline that outputs a zero action at every step. As the table below shows, this policy satisfies the safety constraints on navigation tasks but consistently earns zero return. On velocity tasks, it typically earns only the survival reward. PROCO achieves higher returns while staying within the prescribed safety limits.

TaskTrivial PolicyFISOR+PROCO
r↑ c↓r↑ c↓
PB10.00 0.00, 0.00 0.000.01 0.01, 1.15 0.56
PB20.00 0.00, 0.00 0.000.08 0.02, 2.76 0.88
PG10.00 0.00, 0.00 0.000.35 0.00, 0.96 0.44
PG20.00 0.00, 0.00 0.000.08 0.05, 0.38 0.43
PP10.00 0.00, 0.00 0.000.17 0.00, 0.86 0.53
PP20.00 0.00, 0.00 0.000.12 0.02, 0.30 0.18
CB10.00 0.00, 0.00 0.00-0.03 0.04, 0.60 0.23
CB20.00 0.00, 0.00 0.000.00 0.00, 0.99 0.21
CG10.00 0.00, 0.00 0.000.13 0.02, 0.02 0.03
CG20.00 0.00, 0.00 0.000.03 0.02, 0.52 0.50
CP10.00 0.00, 0.00 0.000.16 0.02, 0.93 0.86
CP20.00 0.00, 0.00 0.000.01 0.00, 0.04 0.06
SwimmerV1-0.03 0.00, 0.00 0.000.02 0.02, 0.12 0.12
HopperV10.05 0.00, 0.00 0.000.11 0.01, 0.07 0.10
HalfCheetahV1-0.02 0.00, 0.00 0.000.48 0.01, 0.00 0.00
Walker2dV10.02 0.00, 0.00 0.000.09 0.03, 0.58 0.09
AntV10.33 0.00, 0.00 0.000.52 0.04, 0.00 0.00
Average0.02 0.000.14 0.60

We also examined the actions taken by PROCO in CarButton1 and CarButton2, where its returns were relatively low. As shown in the table below, the mean absolute action value across 1,000 observations per environment was 0.62 in CarButton1 and 0.69 in CarButton2, compared with 0.00 for the Trivial Policy. These values confirm that the policy takes nonzero actions in both environments. The low returns reflect how these tasks reward progress toward the goal. Moving away from the goal incurs negative rewards that can offset the rewards earned by moving closer, leaving little net reward over an episode.

absolute action meanCarButton1CarButton2
PROCO+FISOR0.620.69
Trivial Policy0.000.00

Q3 Advantage over simply ways of incorporating cost information

PROCO’s model-based cost generation adds complexity, but it makes more effective use of the available cost information. To quantify this benefit, we compared three ways of using the same 100 unsafe samples with each of three base algorithms (LSPC, CAPS, and FISOR). Only 100 Unsafe adds the samples that PROCO uses for reflection directly to the training set. W/o Model uses the generated cost function to relabel the existing dataset. PROCO also applies the cost function to transitions generated through model rollouts.

As the table below shows, adding the 100 unsafe samples directly yielded little safety improvement for any of the three algorithms, and the resulting policies still violated the safety constraints by a substantial margin. Relabeling the existing data with the generated cost function substantially improved safety over Only 100 Unsafe. Model rollouts reduced normalized cost by a further 0.4–0.7 compared with relabeling alone. This comparison quantifies the benefit of applying the same cost function to transitions beyond those in the original dataset. We have added these results and the accompanying discussion to Appendix E.9.

TaskLSPC
Only 100 UnsafeW/o ModelPROCO
r↑ c↓r↑ c↓r↑ c↓
PB10.14 0.06, 3.95 0.410.05 0.09, 3.06 3.430.09 0.04, 1.28 0.32
PB20.27 0.03, 6.70 0.670.19 0.05, 6.51 3.380.14 0.08, 3.02 2.30
PG10.60 0.10, 3.42 0.270.10 0.03, 0.14 0.130.21 0.07, 0.23 0.10
PG20.60 0.04, 7.62 1.420.08 0.06, 0.88 0.500.11 0.03, 0.45 0.27
PP10.16 0.06, 1.73 0.340.10 0.08, 1.75 0.820.16 0.01, 2.09 1.38
PP20.14 0.05, 2.54 0.890.14 0.01, 1.88 0.330.15 0.04, 1.79 0.62
CB1-0.15 0.06, 4.34 1.05-0.06 0.01, 2.09 2.01-0.03 0.02, 1.92 2.43
CB2-0.13 0.15, 3.06 1.30-0.10 0.07, 1.80 1.46-0.11 0.07, 1.88 0.63
CG10.40 0.09, 1.53 0.430.24 0.12, 0.64 0.300.20 0.06, 0.68 0.31
CG20.26 0.04, 3.34 0.320.07 0.05, 1.01 0.530.18 0.06, 0.52 0.34
CP10.20 0.05, 1.47 0.240.14 0.04, 0.79 0.650.10 0.02, 0.55 0.74
CP20.13 0.06, 3.78 2.290.07 0.03, 1.55 0.460.12 0.01, 1.30 0.63
SwimmerV10.65 0.07, 14.62 4.380.17 0.25, 4.85 6.860.05 0.06, 0.58 0.59
HopperV10.19 0.15, 2.28 1.270.22 0.23, 1.06 0.750.17 0.03, 5.86 4.20
HalfCheetahV11.04 0.01, 47.42 8.950.27 0.03, 0.00 0.000.41 0.05, 0.00 0.00
Walker2dV10.80 0.00, 0.05 0.020.74 0.03, 5.42 0.560.50 0.08, 2.08 1.31
AntV10.99 0.00, 9.22 1.270.78 0.01, 0.00 0.000.81 0.01, 0.16 0.21
Average0.37 6.890.19 1.970.19 1.44
TaskCAPS
Only 100 UnsafeW/o ModelPROCO
r↑ c↓r↑ c↓r↑ c↓
PB10.14 0.09, 4.48 1.94-0.02 0.02, 1.51 0.45-0.03 0.02, 1.13 0.34
PB20.25 0.07, 7.32 0.490.11 0.05, 2.17 1.070.02 0.02, 0.63 0.18
PG10.57 0.00, 3.14 0.420.03 0.03, 0.03 0.050.02 0.05, 0.00 0.00
PG20.45 0.10, 7.31 0.870.12 0.06, 1.48 0.600.05 0.03, 0.66 0.41
PP10.23 0.05, 2.59 1.030.08 0.01, 0.30 0.330.11 0.02, 0.26 0.33
PP20.12 0.07, 2.31 1.100.09 0.04, 2.31 1.360.09 0.03, 1.61 1.82
CB10.03 0.05, 4.58 1.45-0.08 0.09, 2.22 1.17-0.02 0.01, 2.02 1.30
CB2-0.09 0.05, 4.43 1.23-0.13 0.07, 1.19 0.59-0.09 0.04, 0.34 0.23
CG10.40 0.06, 2.10 0.560.14 0.04, 0.66 0.060.14 0.11, 0.14 0.10
CG20.29 0.04, 3.88 0.120.08 0.06, 0.81 0.580.07 0.03, 0.30 0.26
CP10.19 0.02, 1.55 1.070.11 0.01, 0.54 0.550.16 0.03, 0.20 0.09
CP20.10 0.03, 2.42 0.160.01 0.01, 1.75 0.240.03 0.03, 1.50 0.57
SwimmerV10.38 0.16, 3.16 2.040.08 0.05, 0.42 0.360.03 0.03, 0.08 0.12
HopperV10.52 0.01, 8.05 1.460.32 0.07, 0.67 0.830.26 0.03, 0.06 0.04
HalfCheetahV10.98 0.00, 6.39 2.730.01 0.01, 0.00 0.000.00 0.01, 0.00 0.00
Walker2dV10.79 0.00, 0.17 0.190.53 0.09, 2.35 1.470.53 0.09, 2.83 0.78
AntV10.99 0.00, 10.13 2.440.52 0.01, 0.00 0.000.50 0.06, 0.00 0.00
Average0.37 4.350.11 1.080.11 0.69
Reviewer_sx58OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PROCO, a model-based offline safe RL framework for learning policies from datasets with few or no unsafe samples. PROCO uses an LLM to generate conservative cost functions from natural-language safety constraints and a learned dynamics model to generate synthetic unsafe transitions. These transitions are used for feasibility estimation and policy learning. Experiments across 17 Safety-Gymnasium tasks show substantial reductions in constraint violations when PROCO is integrated with existing offline safe RL algorithms.

Critical strengths
  • Learning safe policies from datasets with limited unsafe samples is an important challenge for safety-critical applications.
  • PROCO can be incorporated into several existing offline safe RL algorithms without fundamentally modifying their policy learning procedures.
  • Experiments across 17 tasks and three offline safe RL methods demonstrate substantial reductions in constraint violations.
Critical weaknesses
  • The LLM receives explicit safety constraints and detailed descriptions of observation dimensions. The generated cost functions largely implement more conservative thresholds on known safety conditions. It is unclear what the LLM contributes beyond manually adjusting these thresholds, and comparisons with equivalent non-LLM conservative cost functions are missing.
  • PROCO relabels a prescribed fraction (10–30%) of originally safe samples as unsafe and uses a default rollout horizon of only one step. Consequently, improvements may primarily arise from injecting conservative cost signals and augmenting near-boundary transitions rather than identifying infeasible states. The experiments do not sufficiently distinguish these effects.
  • Existing offline safe RL algorithms are trained on datasets with no observed violations, leaving them without informative cost signals, whereas PROCO receives explicit safety knowledge and generated cost labels. Although this demonstrates the benefit of additional safety information, it does not establish the advantage of the proposed mechanism over simpler ways of incorporating the same information.
  • The main experiments use 100 unsafe samples per task to validate LLM-generated cost functions, despite emphasizing settings with no unsafe data. Furthermore, the procedure for constructing safe-only datasets is insufficiently specified, making it difficult to determine how near-boundary and potentially infeasible transitions are represented.
Questions for authors
  • How does PROCO compare against manually specified conservative cost functions with equivalent safety margins and the same model-based augmentation?

  • Given that the default rollout horizon is H=1H=1, what evidence demonstrates that PROCO identifies dynamically infeasible states rather than simply penalizing states near conservative safety boundaries?

  • How exactly are the safe-only datasets constructed? Are trajectories truncated before their first constraint violation, or are individual unsafe transitions filtered out?

  • How does PROCO perform when no unsafe samples are available, including for validation and refinement of the generated cost function?

6 comments

Thank you for your insightful reviews. We have revised the manuscript to address your comments and marked all additions in blue. In the Conclusion, we now discuss the conditions under which PROCO is most applicable and the limitations that arise when those conditions are not met. We also added experiments and discussion in five appendix sections:

  • Appendix E.4 examines how providing specific examples of misclassified unsafe observations affects performance and the number of reflection rounds.
  • Appendix E.5 evaluates reflection using conservatism as the sole source of feedback when no unsafe samples are available.
  • Appendix E.6 compares cost functions tuned by the LLM with those using parameters chosen by a human.
  • Appendix E.9 examines how effectively PROCO uses cost information by comparing direct training on unsafe samples, relabeling with the generated cost function, and relabeling combined with model rollouts.
  • Appendix E.10 compares PROCO with a trivial policy that always outputs a zero action and reports the mean absolute action values on CarButton1 and CarButton2.

We appreciate your feedback and hope that these revisions address your concerns.

Q1 About human-generated cost function

In PROCO, the LLM translates domain knowledge about costs into executable cost functions and automatically tunes their coefficients for each task and dataset. We agree that human experts can write and tune these functions themselves. Our motivation for using an LLM is to reduce the manual effort this requires, a goal shared by many applications of AI and LLMs. We explain this choice in the first paragraph of the Limitations section, quoted below:

In this work, although LLMs are employed as the primary approach for cost function generation, our core objective is not to rely on LLMs per se. Rather, we view LLMs as a knowledge processing tool that transforms natural-language safety specifications into a cost function-an intermediate representation that can be effectively aligned with data. Consequently, alternative approaches that serve a similar role can also be naturally used. For instance, human experts could directly process and translate safety descriptions into a cost function, which can then be iteratively refined based on feedback from the data. Compared to manual efforts, the use of LLMs offers a higher degree of automation and reduced human labor, while a potential drawback lies in the additional computational and monetary costs associated with LLM inference.

We added a Human baseline to evaluate the benefit of using the LLM to tune cost function parameters for each task. To limit manual tuning, we chose one fixed setting for each task family. For all navigation tasks, we set the safety distance to twice the collision distance (2 × 0.3 = 0.6). For all velocity tasks, we used a speed buffer of 0.15.

As the table below shows, the Human baseline was generally more conservative than PROCO on navigation tasks. It met the safety constraint on one additional task but earned lower returns on several tasks where both methods met the constraint. These results show that tuning the cost function to each task can preserve reward while satisfying the safety constraint. We have added these results and the accompanying discussion to Appendix E.6.

TaskFISOR+PROCO HumanFISOR+PROCO
r↑ c↓r↑ c↓
PB10.01 0.02, 0.88 0.620.01 0.01, 1.15 0.56
PB20.07 0.00, 1.88 0.310.08 0.02, 2.76 0.88
PG10.23 0.02, 0.52 0.500.35 0.00, 0.96 0.44
PG20.06 0.01, 0.59 0.170.08 0.05, 0.38 0.43
PP10.12 0.02, 0.91 1.100.17 0.00, 0.86 0.53
PP20.08 0.04, 0.50 0.330.12 0.02, 0.30 0.18
CB1-0.03 0.03, 0.55 0.13-0.03 0.04, 0.60 0.23
CB2-0.04 0.03, 0.92 0.100.00 0.00, 0.99 0.21
CG10.13 0.04, 0.00 0.000.13 0.02, 0.02 0.03
CG20.01 0.01, 0.08 0.110.03 0.02, 0.52 0.50
CP10.14 0.02, 0.31 0.100.16 0.02, 0.93 0.86
CP2-0.01 0.00, 0.00 0.000.01 0.00, 0.04 0.06
SwimmerV10.01 0.00, 0.00 0.000.02 0.02, 0.12 0.12
HopperV10.05 0.04, 0.00 0.000.11 0.01, 0.07 0.10
HalfCheetahV10.60 0.01, 0.00 0.000.48 0.01, 0.00 0.00
Walker2dV10.10 0.01, 0.85 0.140.09 0.03, 0.58 0.09
AntV10.51 0.02, 0.00 0.000.52 0.04, 0.00 0.00
Average0.12, 0.470.14 0.60

Q2 Can PROCO identify infeasible states

Our results indicate that PROCO improves safety when paired with feasibility-based algorithms because it helps them identify infeasible states more accurately. Three experiments show why proximity to a boundary alone cannot explain these gains.

  • Figure 2(c) shows how the feasibility values learned with PROCO vary with the car’s velocity. The car travels clockwise along an arc, with boundaries on its left and right. When the car moves rapidly toward either boundary, the feasibility value is higher and the region classified as infeasible extends farther from the boundary, indicating that the car needs to slow down earlier. When the car moves rapidly in the vertical direction, parallel to the boundaries, the infeasible region is largely confined to the boundary itself. A penalty based solely on distance to the boundary would assign the same penalty at a given distance regardless of the car’s direction of travel.
  • Figure 4(b) compares the signs of the feasibility values learned with PROCO against those from FISOR Oracle, which has access to a large number of real unsafe samples. The signs agreed in at least 90% of cases in each of the three environments. This agreement shows that PROCO’s feasible and infeasible classifications closely match the oracle’s.
  • Appendix E.8 tests how much safety improves when soft-constraint algorithms use PROCO’s generated unsafe samples directly to penalize states near the boundary. Adding these samples improved safety for all three algorithms tested: CPQ, BCQ-L, and COptiDICE. Before augmentation, COptiDICE’s normalized cost was more than 5 points lower than BCQ-L’s. After augmentation, the gap narrowed to about 0.1 points. COptiDICE with PROCO achieved an average normalized cost of 2.95, still 2.35 points above the 0.60 achieved by FISOR with PROCO. These results show that the generated unsafe samples can improve safety through penalties alone, while the larger gains observed with FISOR support the value of using those samples to learn which states are infeasible.

Q3 Advantage over simply ways of incorporating cost information

PROCO’s model-based cost generation adds complexity, but it makes more effective use of the available cost information. To quantify this benefit, we compared three ways of using the same 100 unsafe samples with each of three base algorithms (LSPC, CAPS, and FISOR). Only 100 Unsafe adds the samples that PROCO uses for reflection directly to the training set. W/o Model uses the generated cost function to relabel the existing dataset. PROCO also applies the cost function to transitions generated through model rollouts.

As the table below shows, adding the 100 unsafe samples directly yielded little safety improvement for any of the three algorithms, and the resulting policies still violated the safety constraints by a substantial margin. Relabeling the existing data with the generated cost function substantially improved safety over Only 100 Unsafe. Model rollouts reduced normalized cost by a further 0.4–0.7 compared with relabeling alone. This comparison quantifies the benefit of applying the same cost function to transitions beyond those in the original dataset. We have added these results and the accompanying discussion to Appendix E.9.

TaskLSPC
Only 100 UnsafeW/o ModelPROCO
r↑ c↓r↑ c↓r↑ c↓
PB10.14 0.06, 3.95 0.410.05 0.09, 3.06 3.430.09 0.04, 1.28 0.32
PB20.27 0.03, 6.70 0.670.19 0.05, 6.51 3.380.14 0.08, 3.02 2.30
PG10.60 0.10, 3.42 0.270.10 0.03, 0.14 0.130.21 0.07, 0.23 0.10
PG20.60 0.04, 7.62 1.420.08 0.06, 0.88 0.500.11 0.03, 0.45 0.27
PP10.16 0.06, 1.73 0.340.10 0.08, 1.75 0.820.16 0.01, 2.09 1.38
PP20.14 0.05, 2.54 0.890.14 0.01, 1.88 0.330.15 0.04, 1.79 0.62
CB1-0.15 0.06, 4.34 1.05-0.06 0.01, 2.09 2.01-0.03 0.02, 1.92 2.43
CB2-0.13 0.15, 3.06 1.30-0.10 0.07, 1.80 1.46-0.11 0.07, 1.88 0.63
CG10.40 0.09, 1.53 0.430.24 0.12, 0.64 0.300.20 0.06, 0.68 0.31
CG20.26 0.04, 3.34 0.320.07 0.05, 1.01 0.530.18 0.06, 0.52 0.34
CP10.20 0.05, 1.47 0.240.14 0.04, 0.79 0.650.10 0.02, 0.55 0.74
CP20.13 0.06, 3.78 2.290.07 0.03, 1.55 0.460.12 0.01, 1.30 0.63
SwimmerV10.65 0.07, 14.62 4.380.17 0.25, 4.85 6.860.05 0.06, 0.58 0.59
HopperV10.19 0.15, 2.28 1.270.22 0.23, 1.06 0.750.17 0.03, 5.86 4.20
HalfCheetahV11.04 0.01, 47.42 8.950.27 0.03, 0.00 0.000.41 0.05, 0.00 0.00
Walker2dV10.80 0.00, 0.05 0.020.74 0.03, 5.42 0.560.50 0.08, 2.08 1.31
AntV10.99 0.00, 9.22 1.270.78 0.01, 0.00 0.000.81 0.01, 0.16 0.21
Average0.37 6.890.19 1.970.19 1.44
TaskCAPS
Only 100 UnsafeW/o ModelPROCO
r↑ c↓r↑ c↓r↑ c↓
PB10.14 0.09, 4.48 1.94-0.02 0.02, 1.51 0.45-0.03 0.02, 1.13 0.34
PB20.25 0.07, 7.32 0.490.11 0.05, 2.17 1.070.02 0.02, 0.63 0.18
PG10.57 0.00, 3.14 0.420.03 0.03, 0.03 0.050.02 0.05, 0.00 0.00
PG20.45 0.10, 7.31 0.870.12 0.06, 1.48 0.600.05 0.03, 0.66 0.41
PP10.23 0.05, 2.59 1.030.08 0.01, 0.30 0.330.11 0.02, 0.26 0.33
PP20.12 0.07, 2.31 1.100.09 0.04, 2.31 1.360.09 0.03, 1.61 1.82
CB10.03 0.05, 4.58 1.45-0.08 0.09, 2.22 1.17-0.02 0.01, 2.02 1.30
CB2-0.09 0.05, 4.43 1.23-0.13 0.07, 1.19 0.59-0.09 0.04, 0.34 0.23
CG10.40 0.06, 2.10 0.560.14 0.04, 0.66 0.060.14 0.11, 0.14 0.10
CG20.29 0.04, 3.88 0.120.08 0.06, 0.81 0.580.07 0.03, 0.30 0.26
CP10.19 0.02, 1.55 1.070.11 0.01, 0.54 0.550.16 0.03, 0.20 0.09
CP20.10 0.03, 2.42 0.160.01 0.01, 1.75 0.240.03 0.03, 1.50 0.57
SwimmerV10.38 0.16, 3.16 2.040.08 0.05, 0.42 0.360.03 0.03, 0.08 0.12
HopperV10.52 0.01, 8.05 1.460.32 0.07, 0.67 0.830.26 0.03, 0.06 0.04
HalfCheetahV10.98 0.00, 6.39 2.730.01 0.01, 0.00 0.000.00 0.01, 0.00 0.00
Walker2dV10.79 0.00, 0.17 0.190.53 0.09, 2.35 1.470.53 0.09, 2.83 0.78
AntV10.99 0.00, 10.13 2.440.52 0.01, 0.00 0.000.50 0.06, 0.00 0.00
Average0.37 4.350.11 1.080.11 0.69
TaskFISOR
Only 100 UnsafeW/o ModelPROCO
r↑ c↓r↑ c↓r↑ c↓
PB10.51 0.08, 13.0 0.980.06 0.00, 2.84 0.580.01 0.01, 1.15 0.56
PB20.46 0.02, 15.57 1.270.09 0.03, 4.32 0.280.08 0.02, 2.76 0.88
PG10.71 0.05, 5.78 0.440.54 0.04, 2.41 0.760.35 0.00, 0.96 0.44
PG20.80 0.02, 13.68 2.100.14 0.03, 1.58 0.680.08 0.05, 0.38 0.43
PP10.35 0.02, 3.39 0.600.25 0.02, 1.03 0.320.17 0.00, 0.86 0.53
PP20.32 0.06, 5.47 0.170.11 0.03, 1.21 0.360.12 0.02, 0.30 0.18
CB10.49 0.05, 26.97 4.20-0.02 0.01, 1.24 0.25-0.03 0.04, 0.60 0.23
CB20.55 0.04, 24.94 2.320.00 0.02, 3.22 0.940.00 0.00, 0.99 0.21
CG10.74 0.01, 5.31 0.690.27 0.02, 0.37 0.130.13 0.02, 0.02 0.03
CG20.77 0.00, 15.53 1.380.06 0.03, 0.14 0.120.03 0.02, 0.52 0.50
CP10.43 0.01, 2.71 0.680.24 0.04, 1.03 0.690.16 0.02, 0.93 0.86
CP20.43 0.02, 9.04 2.210.03 0.03, 0.48 0.030.01 0.00, 0.04 0.06
SwimmerV1-0.05 0.00, 5.21 1.470.01 0.01, 0.00 0.000.02 0.02, 0.12 0.12
HopperV10.09 0.02, 3.60 0.340.17 0.05, 0.17 0.120.11 0.01, 0.07 0.10
HalfCheetahV11.03 0.01, 16.48 4.570.51 0.04, 0.00 0.000.48 0.01, 0.00 0.00
Walker2dV10.15 0.01, 5.87 1.310.15 0.03, 1.47 0.500.09 0.03, 0.58 0.09
AntV11.01 0.00, 10.87 0.600.59 0.01, 0.00 0.000.52 0.04, 0.00 0.00
Average0.52 10.790.19 1.270.14 0.60

Q4 How are the safe-only datasets constructed

As described in Section 4.1, we construct our safe-only datasets by removing individual unsafe transitions. To test whether the filtering procedure affects FISOR’s performance, we added a baseline, FISOR safe only dataset 2, that truncates each trajectory immediately before its first positive-cost transition.

As the table below shows, FISOR performed similarly under both filtering procedures. This suggests that the number of unsafe transitions available for training matters more than the choice between these two ways of constructing a safe-only dataset.

TaskFISORFISOR safe only dataset 2
r↑ c↓r↑ c↓
PB10.49 0.12, 11.50 0.910.54 0.05, 11.13 0.46
PB20.46 0.06, 14.69 0.410.49 0.03, 13.23 0.57
PG10.73 0.01, 5.70 0.520.70 0.01, 4.67 0.87
PG20.59 0.03, 17.10 1.010.55 0.05, 14.85 1.85
PP10.34 0.06, 3.70 1.270.33 0.01, 2.57 0.40
PP20.27 0.06, 6.61 1.790.27 0.03, 4.63 0.62
CB10.47 0.03, 28.01 1.780.53 0.06, 28.13 1.49
CB20.57 0.04, 25.96 0.940.52 0.03, 28.23 1.11
CG10.75 0.01, 4.10 0.910.75 0.02, 5.02 0.44
CG20.80 0.02, 14.56 0.200.77 0.01, 17.23 1.73
CP10.42 0.04, 1.91 0.350.42 0.07, 2.40 0.08
CP20.39 0.06, 8.04 1.840.42 0.04, 9.60 1.47
SwimmerV1-0.05 0.01, 4.51 0.56-0.07 0.01, 2.33 0.48
HopperV10.07 0.01, 3.23 0.150.07 0.01, 3.47 0.61
HalfCheetahV11.03 0.01, 16.13 3.451.02 0.00, 14.17 1.86
Walker2dV10.15 0.03, 5.71 0.950.16 0.01, 5.67 0.73
AntV11.01 0.00, 10.47 0.761.01 0.00, 10.83 1.18
Average0.50 10.700.50 10.48
Reviewer_F9JWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the PROCO framework to address offline safe policy learning when unsafe samples are scarce. PROCO first uses an LLM to convert natural-language safety rules into a cost function, requiring the cost function to fully identify unsafe samples while also identifying a small number of safe samples close to dangerous states as infeasible samples. Meanwhile, it trains a dynamics model using the offline dataset, and uses the environmental dynamics model together with the cost function to perform branched rollouts to obtain potentially unsafe samples. These samples are then used to train the reinforcement learning model offline, thereby improving policy safety.

Critical strengths
  1. The research is practically meaningful: unsafe samples are scarce and difficult to obtain, and the PROCO framework can enrich unsafe samples;
  2. The PROCO framework can be used as a plug-in and combined with other safe reinforcement learning algorithms to improve policy safety;
  3. Extensive experiments show that combining the PROCO framework with various offline safe RL algorithms can significantly increase the proportion of safe tasks.
Critical weaknesses
  1. The cost function relies on LLM generation, but the validation-feedback mechanism only asks the LLM to adjust the cost function according to the unsafe-sample identification rate and conservativeness, without reference to specific samples; thus, the resulting cost function may not be reliable;
  2. The paper states that the DunsafeD_{unsafe} dataset is optional, but in practice the validation and correction of the cost function rely entirely on this dataset.
Questions for authors

If $D_ {unsafe} $is unavailable, how should the cost function be validated and corrected?

7 comments

Thank you for your thoughtful reviews. We have revised the manuscript to address your comments and marked all additions in blue. In the Conclusion, we now discuss the conditions under which PROCO is most applicable and the limitations that arise when those conditions are not met. We also added experiments and discussion in five appendix sections:

  • Appendix E.4 examines how providing specific examples of misclassified unsafe observations affects performance and the number of reflection rounds.
  • Appendix E.5 evaluates reflection using conservatism as the sole source of feedback when no unsafe samples are available.
  • Appendix E.6 compares cost functions tuned by the LLM with those using parameters chosen by a human.
  • Appendix E.9 examines how effectively PROCO uses cost information by comparing direct training on unsafe samples, relabeling with the generated cost function, and relabeling combined with model rollouts.
  • Appendix E.10 compares PROCO with a trivial policy that always outputs a zero action and reports the mean absolute action values on CarButton1 and CarButton2.

We appreciate your feedback and hope that these revisions address your concerns.

Q1 Reliability of the LLM generated cost function

Thank you for suggesting that we show the LLM the unsafe samples that its cost function misclassifies. We agree that these examples can provide more precise feedback and help the LLM correct specific errors. To test this idea, we used GPT-5.5 to generate the cost functions and compared our original reflection procedure with Specific Sample Reflection. Whenever the unsafe identification ratio fell below 100%, this variant gave the LLM every unsafe sample that the current cost function had classified as safe, along with the usual feedback on the unsafe identification ratio and conservatism.

As the table below shows, the two procedures achieved similar overall performance. The small differences may reflect variability in the underlying algorithm. In the relatively simple observation spaces we studied, feedback on conservatism and the unsafe identification ratio was sufficient to generate reliable cost functions given enough reflection rounds. Providing the misclassified samples helped the LLM reach comparable performance in fewer rounds.

We agree that feedback on specific errors may be necessary to generate reliable cost function code in more complex settings, particularly in embodied environments with visual observations. We consider this an important direction for future work. We have added the experiment and discussion to Appendix E.4.

TaskSpecific Sample ReflectionFISOR+PROCO
r↑ c↓r↑ c↓
PB10.03 0.04, 1.76 0.420.01 0.01, 1.15 0.56
PB20.07 0.00, 2.37 1.030.08 0.02, 2.76 0.88
PG10.40 0.02, 1.67 0.390.35 0.00, 0.96 0.44
PG20.07 0.02, 0.22 0.220.08 0.05, 0.38 0.43
PP10.18 0.03, 1.15 0.940.17 0.00, 0.86 0.53
PP20.11 0.04, 0.75 0.200.12 0.02, 0.30 0.18
CB1-0.03 0.01, 0.36 0.14-0.03 0.04, 0.60 0.23
CB2-0.03 0.00, 0.90 0.730.00 0.00, 0.99 0.21
CG10.12 0.03, 0.08 0.060.13 0.02, 0.02 0.03
CG20.03 0.00, 0.08 0.110.03 0.02, 0.52 0.50
CP10.17 0.00, 0.71 0.500.16 0.02, 0.93 0.86
CP20.00 0.00, 0.01 0.010.01 0.00, 0.04 0.06
SwimmerV10.02 0.01, 0.00 0.000.02 0.02, 0.12 0.12
HopperV10.12 0.09, 0.02 0.020.11 0.01, 0.07 0.10
HalfCheetahV10.47 0.02, 0.00 0.000.48 0.01, 0.00 0.00
Walker2dV10.11 0.02, 0.99 0.480.09 0.03, 0.58 0.09
AntV10.52 0.01, 0.00 0.000.52 0.04, 0.00 0.00
Average0.14 0.650.14 0.60
Average Reflection Rounds3.474.06

Q2 Performance when no unsafe samples are available

Unsafe samples provide useful feedback during reflection. Even when no unsafe samples are available, the initial LLM-generated cost function provides a fallback that substantially improves safety over FISOR. Reflection can further improve safety using conservatism alone.

To test this, we added W/o Unsafe Reflection alongside the existing W/o Reflection baseline. W/o Reflection uses the cost function as initially generated by the LLM. W/o Unsafe Reflection refines that function using conservatism as its sole reflection objective, with an acceptance range of 20%–30%.

As the table below shows, FISOR+PROCO already substantially outperforms FISOR in safety with reflection disabled. Refining the cost function using conservatism alone yields a further improvement over W/o Reflection. Compared with full reflection, this variant can be overly conservative on some tasks and insufficiently conservative on others. Even so, its safety gains over FISOR show that PROCO remains effective when no unsafe samples are available for reflection. We have added these results and the accompanying discussion to Appendix E.5.

TaskFISORFISOR+PROCO W/o ReflectionFISOR+PROCO W/o Unsafe ReflectionFISOR+PROCO
r↑ c↓r↑ c↓r↑ c↓r↑ c↓
PB10.49 0.12, 11.50 0.910.06 0.03, 3.40 1.700.03 0.04, 1.67 0.970.01 0.01, 1.15 0.56
PB20.46 0.06, 14.69 0.410.07 0.01, 4.55 1.480.07 0.01, 2.90 0.670.08 0.02, 2.76 0.88
PG10.73 0.01, 5.70 0.520.34 0.04, 0.86 0.550.41 0.01, 1.44 0.270.35 0.00, 0.96 0.44
PG20.59 0.03, 17.10 1.010.08 0.02, 0.42 0.130.06 0.01, 0.32 0.100.08 0.05, 0.38 0.43
PP10.34 0.06, 3.70 1.270.20 0.05, 2.17 2.380.16 0.02, 0.91 1.100.17 0.00, 0.86 0.53
PP20.27 0.06, 6.61 1.790.07 0.01, 1.22 0.900.08 0.04, 0.50 0.390.12 0.02, 0.30 0.18
CB10.47 0.03, 28.01 1.78-0.01 0.03, 2.57 1.32-0.07 0.02, 0.61 0.04-0.03 0.04, 0.60 0.23
CB20.57 0.04, 25.96 0.940.00 0.01, 3.21 1.44-0.01 0.01, 1.09 0.650.00 0.00, 0.99 0.21
CG10.75 0.01, 4.10 0.910.16 0.02, 0.16 0.140.12 0.02, 0.00 0.000.13 0.02, 0.02 0.03
CG20.80 0.02, 14.56 0.200.04 0.03, 0.80 0.930.03 0.00, 0.18 0.130.03 0.02, 0.52 0.50
CP10.42 0.04, 1.91 0.350.11 0.04, 1.86 2.430.17 0.00, 1.03 0.020.16 0.02, 0.93 0.86
CP20.39 0.06, 8.04 1.840.03 0.04, 1.19 0.97-0.01 0.00, 0.00 0.000.01 0.00, 0.04 0.06
SwimmerV1-0.05 0.01, 4.51 0.560.00 0.00, 0.23 0.330.02 0.00, 0.02 0.020.02 0.02, 0.12 0.12
HopperV10.07 0.01, 3.23 0.150.13 0.17, 0.00 0.000.13 0.09, 0.15 0.180.11 0.01, 0.07 0.10
HalfCheetahV11.03 0.01, 16.13 3.450.65 0.07, 0.00 0.000.20 0.05, 0.00 0.000.48 0.01, 0.00 0.00
Walker2dV10.15 0.03, 5.71 0.950.18 0.00, 2.09 0.570.22 0.08, 1.10 0.360.09 0.03, 0.58 0.09
AntV11.01 0.00, 10.47 0.760.50 0.18, 0.00 0.000.47 0.03, 0.00 0.000.52 0.04, 0.00 0.00
Average0.50 10.700.15 1.450.12 0.700.14 0.60

Dear Reviewer F9JW,

Thank you very much for your valuable feedback on our work! As the author-reviewer discussion period is closely approaching its end, we would be happy to address any remaining questions or concerns you may have. We sincerely appreciate your time and consideration!

Best regards,

The Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you very much for your valuable feedback on our work! As the author-reviewer discussion period is approaching its end, we would be happy to address any remaining questions or concerns you may have. We sincerely appreciate your time and consideration!

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of rebuttal experiments and revisions for PROCO

Dear Area Chair,

Thank you for the time you have devoted to our submission and for overseeing the discussion. As the discussion draws to a close, one reviewer has yet to follow up on our response. We would therefore like to summarize the strengths highlighted in the reviews and the revisions we made in response to the reviewers’ concerns.

Two reviewers emphasized the practical importance of learning safe policies from offline data when unsafe examples are scarce and difficult to collect. They highlighted the value of PROCO’s central idea of generating additional unsafe transitions that existing offline safe RL algorithms can use directly. They also noted the safety improvements across 17 tasks and three algorithms. A third reviewer found the idea appealing and its logic straightforward.

These positive assessments came with questions about what drives the observed safety gains and how to interpret them. In particular, the reviewers asked how reliably PROCO can construct a cost function from limited feedback, how much model rollouts add beyond relabeling the offline data, and whether low returns on some tasks reflect inactivity. These questions motivated the following three revisions.

  1. Reliability of the generated cost function (Appendices E.4–E.6). We tested how PROCO performs when no unsafe examples are available for reflection. Using conservatism as the sole feedback signal, FISOR+PROCO achieved an average normalized cost of 0.70, compared with 10.70 for FISOR and 0.60 for full PROCO. When unsafe examples were available, showing the LLM the examples it had misclassified reduced the average number of reflection rounds from 4.06 to 3.47 with similar policy performance. We also compared task-specific LLM tuning with hand-set parameters shared within each task family.
  2. Contribution of model rollouts (Appendix E.9). We compared three ways of using the same 100 unsafe examples. With FISOR, adding the examples directly to training yielded an average normalized cost of 10.79. Relabeling the offline data with the generated cost function lowered it to 1.27, and applying the same function to model rollouts lowered it further to 0.60. Rollouts also reduced cost beyond relabeling alone for LSPC and CAPS.
  3. Policy behavior and scope of applicability (Appendix E.10 and Conclusion). To investigate whether low returns reflected inactivity, we added a zero-action baseline. On CarButton1 and CarButton2, where returns were close to zero, PROCO’s mean absolute action values were 0.62 and 0.69, respectively. We also clarified that PROCO is best suited to settings where the safety cost can be computed from the available observations, the dynamics can be modeled reliably, and a small set of unsafe examples is available to refine the cost function.

We hope this summary helps you assess the revised paper. We would be happy to clarify any remaining questions while the discussion is open.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.