- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- Get a mathematical constraint function from an LLM agent
- Learn a dynamics model with the limited offline dataset
- Rely on PROCO to generate enough rollouts using this to generate unsafe (or close to unsafe) samples without needing to actually crash
- Pair with your favorite offline safe RL algorithm
Critical strengths
This is a cool idea and the logic is straightforward, though the use case is fairly narrow. Refreshingly, the paper is written quite well. The results and ablation study are nice.
Critical weaknesses
My main concern is that it seems like a big ask to rely on an LLM to derive a conservative cost function, then learn a dynamics model with the limited offline (and presumably only safe) dataset, and then rely on PROCO to generate enough rollouts using this to generate unsafe (or close to unsafe) samples. There is a fairly limited setting wherein this would work directly (pretty simple notions of safety, pretty simple dynamics that can be learned accurately from limited safe data). The papers own up to this with the assumptions that are made and the discussion section. I would like a summary of the analysis of the experiments and ablations section that would provide sort of a guide on when this would work well, when it would work even better (e.g. given a valid dynamic model), and when it would not work well.
I submitted the above to Claude and asked it to point out anything I missed. Here are some comments it gave back that I thought were relevant after checking:
- The reward drop is large, and there’s no trivial-policy baseline. FISOR PROCO averages 0.14 reward versus 0.50 for FISOR and 0.36 for BC (Table 1). Several tasks are near zero: CarButton1 −0.03, CarButton2 0.00, PointButton1 0.01, CarPush2 0.01. A policy that stands still or barely moves would also look “safe” here. It’s worth asking for a do-nothing or minimal-motion baseline, or at least asking the authors to show that PROCO isn’t converging toward one. The headline “over 400% / 1011% safety improvement” is also never defined, so you might ask how it’s computed.
- “No unsafe samples” isn’t quite the setting tested. Every experiment uses the 100-sample D_unsafe to validate the LLM cost function. The baselines don’t appear to get these samples, or the natural-language constraint. The fairer comparisons would be FISOR, LSPC, and CAPS trained with those 100 unsafe samples, and the baselines run on data relabeled with the LLM cost function. The “W/o Model” ablation partially covers the second for FISOR (avg cost 1.27), but not for the others.
- The rollout horizon is H = 1. This directly supports the reviewer’s point about limited settings. The “model-based rollouts to simulate future evolution” are one-step lookaheads in practice (Appendix C.3, Table 2). H = 5 badly hurts navigation tasks (avg cost 4.41, Table 10). Most of the multi-step reasoning therefore comes from the conservative cost margin plus value propagation over dataset transitions, not from the model.
- Assumption 3.5 bounds KL divergence, but Lemma A.2 (from Janner et al.) is applied as if it bounds TV. Going from KL to TV needs Pinsker’s inequality, which gives a √δ dependence rather than δ.
I then asked it to generate questions based on these points that I checked, which I then modified and put in the question section.
Questions for authors
Highest Priority:
- Can you add a trivial policy baseline and report the reward and cost? Could you also show evidence that PROCO’s policies are completing meaningful task progress rather than converging toward inaction, for instance trajectory visualizations or task-specific progress metrics on the near-zero-reward tasks?
- How are the reported safety improvements (e.g., “over 400%,” “1011%”) computed? Please be explicit
- Could you report (a) FISOR, LSPC, and CAPS trained with the same 100 unsafe samples added to D, and (b) all three baselines trained on D relabeled with the final LLM-generated cost function, without model rollouts? Relatedly, how should the “no unsafe samples” framing be read, given that D_unsafe is used in every experiment?
Medium Priority:
- How much of the multi-step feasibility reasoning described in Section 3.1 is due to the model, versus the conservative cost margin combined with value propagation over dataset transitions? Can you characterize when longer horizons help, for example as a function of model error? Can you also reconcile the H = 1 setting with the analysis in Theorems 3.4 and 3.7, which is framed in terms of a horizon H* to reach unsafe states?
- Can you expand/clarify these points on where PROCO works well and where it fails in the discussion section? I would love an honest picture of when I should use PROCO and when I should not.
Lower Priority:
- Can you clarify which divergence you rely on (KL vs TV) in Assumption 3.5/Lemma A.2?
9 comments
Q2 Comparison with a trivial policy
Thank you for suggesting this comparison. We added a Trivial Policy baseline that outputs a zero action at every step. As the table below shows, this policy satisfies the safety constraints on navigation tasks but consistently earns zero return. On velocity tasks, it typically earns only the survival reward. PROCO achieves higher returns while staying within the prescribed safety limits.
| Task | Trivial Policy | FISOR+PROCO |
|---|---|---|
| r↑ c↓ | r↑ c↓ | |
| PB1 | 0.00 0.00, 0.00 0.00 | 0.01 0.01, 1.15 0.56 |
| PB2 | 0.00 0.00, 0.00 0.00 | 0.08 0.02, 2.76 0.88 |
| PG1 | 0.00 0.00, 0.00 0.00 | 0.35 0.00, 0.96 0.44 |
| PG2 | 0.00 0.00, 0.00 0.00 | 0.08 0.05, 0.38 0.43 |
| PP1 | 0.00 0.00, 0.00 0.00 | 0.17 0.00, 0.86 0.53 |
| PP2 | 0.00 0.00, 0.00 0.00 | 0.12 0.02, 0.30 0.18 |
| CB1 | 0.00 0.00, 0.00 0.00 | -0.03 0.04, 0.60 0.23 |
| CB2 | 0.00 0.00, 0.00 0.00 | 0.00 0.00, 0.99 0.21 |
| CG1 | 0.00 0.00, 0.00 0.00 | 0.13 0.02, 0.02 0.03 |
| CG2 | 0.00 0.00, 0.00 0.00 | 0.03 0.02, 0.52 0.50 |
| CP1 | 0.00 0.00, 0.00 0.00 | 0.16 0.02, 0.93 0.86 |
| CP2 | 0.00 0.00, 0.00 0.00 | 0.01 0.00, 0.04 0.06 |
| SwimmerV1 | -0.03 0.00, 0.00 0.00 | 0.02 0.02, 0.12 0.12 |
| HopperV1 | 0.05 0.00, 0.00 0.00 | 0.11 0.01, 0.07 0.10 |
| HalfCheetahV1 | -0.02 0.00, 0.00 0.00 | 0.48 0.01, 0.00 0.00 |
| Walker2dV1 | 0.02 0.00, 0.00 0.00 | 0.09 0.03, 0.58 0.09 |
| AntV1 | 0.33 0.00, 0.00 0.00 | 0.52 0.04, 0.00 0.00 |
| Average | 0.02 0.00 | 0.14 0.60 |
We also examined the actions taken by PROCO in CarButton1 and CarButton2, where its returns were relatively low. As shown in the table below, the mean absolute action value across 1,000 observations per environment was 0.62 in CarButton1 and 0.69 in CarButton2, compared with 0.00 for the Trivial Policy. These values confirm that the policy takes nonzero actions in both environments. The low returns reflect how these tasks reward progress toward the goal. Moving away from the goal incurs negative rewards that can offset the rewards earned by moving closer, leaving little net reward over an episode.
| absolute action mean | CarButton1 | CarButton2 |
|---|---|---|
| PROCO+FISOR | 0.62 | 0.69 |
| Trivial Policy | 0.00 | 0.00 |
Q3 Advantage over simply ways of incorporating cost information
PROCO’s model-based cost generation adds complexity, but it makes more effective use of the available cost information. To quantify this benefit, we compared three ways of using the same 100 unsafe samples with each of three base algorithms (LSPC, CAPS, and FISOR). Only 100 Unsafe adds the samples that PROCO uses for reflection directly to the training set. W/o Model uses the generated cost function to relabel the existing dataset. PROCO also applies the cost function to transitions generated through model rollouts.
As the table below shows, adding the 100 unsafe samples directly yielded little safety improvement for any of the three algorithms, and the resulting policies still violated the safety constraints by a substantial margin. Relabeling the existing data with the generated cost function substantially improved safety over Only 100 Unsafe. Model rollouts reduced normalized cost by a further 0.4–0.7 compared with relabeling alone. This comparison quantifies the benefit of applying the same cost function to transitions beyond those in the original dataset. We have added these results and the accompanying discussion to Appendix E.9.
| Task | LSPC | ||
|---|---|---|---|
| Only 100 Unsafe | W/o Model | PROCO | |
| r↑ c↓ | r↑ c↓ | r↑ c↓ | |
| PB1 | 0.14 0.06, 3.95 0.41 | 0.05 0.09, 3.06 3.43 | 0.09 0.04, 1.28 0.32 |
| PB2 | 0.27 0.03, 6.70 0.67 | 0.19 0.05, 6.51 3.38 | 0.14 0.08, 3.02 2.30 |
| PG1 | 0.60 0.10, 3.42 0.27 | 0.10 0.03, 0.14 0.13 | 0.21 0.07, 0.23 0.10 |
| PG2 | 0.60 0.04, 7.62 1.42 | 0.08 0.06, 0.88 0.50 | 0.11 0.03, 0.45 0.27 |
| PP1 | 0.16 0.06, 1.73 0.34 | 0.10 0.08, 1.75 0.82 | 0.16 0.01, 2.09 1.38 |
| PP2 | 0.14 0.05, 2.54 0.89 | 0.14 0.01, 1.88 0.33 | 0.15 0.04, 1.79 0.62 |
| CB1 | -0.15 0.06, 4.34 1.05 | -0.06 0.01, 2.09 2.01 | -0.03 0.02, 1.92 2.43 |
| CB2 | -0.13 0.15, 3.06 1.30 | -0.10 0.07, 1.80 1.46 | -0.11 0.07, 1.88 0.63 |
| CG1 | 0.40 0.09, 1.53 0.43 | 0.24 0.12, 0.64 0.30 | 0.20 0.06, 0.68 0.31 |
| CG2 | 0.26 0.04, 3.34 0.32 | 0.07 0.05, 1.01 0.53 | 0.18 0.06, 0.52 0.34 |
| CP1 | 0.20 0.05, 1.47 0.24 | 0.14 0.04, 0.79 0.65 | 0.10 0.02, 0.55 0.74 |
| CP2 | 0.13 0.06, 3.78 2.29 | 0.07 0.03, 1.55 0.46 | 0.12 0.01, 1.30 0.63 |
| SwimmerV1 | 0.65 0.07, 14.62 4.38 | 0.17 0.25, 4.85 6.86 | 0.05 0.06, 0.58 0.59 |
| HopperV1 | 0.19 0.15, 2.28 1.27 | 0.22 0.23, 1.06 0.75 | 0.17 0.03, 5.86 4.20 |
| HalfCheetahV1 | 1.04 0.01, 47.42 8.95 | 0.27 0.03, 0.00 0.00 | 0.41 0.05, 0.00 0.00 |
| Walker2dV1 | 0.80 0.00, 0.05 0.02 | 0.74 0.03, 5.42 0.56 | 0.50 0.08, 2.08 1.31 |
| AntV1 | 0.99 0.00, 9.22 1.27 | 0.78 0.01, 0.00 0.00 | 0.81 0.01, 0.16 0.21 |
| Average | 0.37 6.89 | 0.19 1.97 | 0.19 1.44 |
| Task | CAPS | ||
|---|---|---|---|
| Only 100 Unsafe | W/o Model | PROCO | |
| r↑ c↓ | r↑ c↓ | r↑ c↓ | |
| PB1 | 0.14 0.09, 4.48 1.94 | -0.02 0.02, 1.51 0.45 | -0.03 0.02, 1.13 0.34 |
| PB2 | 0.25 0.07, 7.32 0.49 | 0.11 0.05, 2.17 1.07 | 0.02 0.02, 0.63 0.18 |
| PG1 | 0.57 0.00, 3.14 0.42 | 0.03 0.03, 0.03 0.05 | 0.02 0.05, 0.00 0.00 |
| PG2 | 0.45 0.10, 7.31 0.87 | 0.12 0.06, 1.48 0.60 | 0.05 0.03, 0.66 0.41 |
| PP1 | 0.23 0.05, 2.59 1.03 | 0.08 0.01, 0.30 0.33 | 0.11 0.02, 0.26 0.33 |
| PP2 | 0.12 0.07, 2.31 1.10 | 0.09 0.04, 2.31 1.36 | 0.09 0.03, 1.61 1.82 |
| CB1 | 0.03 0.05, 4.58 1.45 | -0.08 0.09, 2.22 1.17 | -0.02 0.01, 2.02 1.30 |
| CB2 | -0.09 0.05, 4.43 1.23 | -0.13 0.07, 1.19 0.59 | -0.09 0.04, 0.34 0.23 |
| CG1 | 0.40 0.06, 2.10 0.56 | 0.14 0.04, 0.66 0.06 | 0.14 0.11, 0.14 0.10 |
| CG2 | 0.29 0.04, 3.88 0.12 | 0.08 0.06, 0.81 0.58 | 0.07 0.03, 0.30 0.26 |
| CP1 | 0.19 0.02, 1.55 1.07 | 0.11 0.01, 0.54 0.55 | 0.16 0.03, 0.20 0.09 |
| CP2 | 0.10 0.03, 2.42 0.16 | 0.01 0.01, 1.75 0.24 | 0.03 0.03, 1.50 0.57 |
| SwimmerV1 | 0.38 0.16, 3.16 2.04 | 0.08 0.05, 0.42 0.36 | 0.03 0.03, 0.08 0.12 |
| HopperV1 | 0.52 0.01, 8.05 1.46 | 0.32 0.07, 0.67 0.83 | 0.26 0.03, 0.06 0.04 |
| HalfCheetahV1 | 0.98 0.00, 6.39 2.73 | 0.01 0.01, 0.00 0.00 | 0.00 0.01, 0.00 0.00 |
| Walker2dV1 | 0.79 0.00, 0.17 0.19 | 0.53 0.09, 2.35 1.47 | 0.53 0.09, 2.83 0.78 |
| AntV1 | 0.99 0.00, 10.13 2.44 | 0.52 0.01, 0.00 0.00 | 0.50 0.06, 0.00 0.00 |
| Average | 0.37 4.35 | 0.11 1.08 | 0.11 0.69 |
Thank you for your kind reviews and constructive suggestions. We have revised the manuscript to address your comments and marked all additions in blue. In the Conclusion, we now discuss the conditions under which PROCO is most applicable and the limitations that arise when those conditions are not met. We also added experiments and discussion in five appendix sections:
We appreciate your feedback and hope that these revisions address your concerns.
Q1 Discussion about when should and when should not use PROCO
Thank you for encouraging us to clarify when PROCO is most effective. We have revised the Conclusion to describe the three conditions under which PROCO is best suited and the limitations that arise when these conditions do not hold.
The cost function can be expressed in code and evaluated from the available observations. The observations must contain all the information the code needs to compute the cost.
An accurate dynamics model is available or can be learned reliably. When the dynamics are harder to learn, extending the rollout horizon can substantially lower reward and increase cost, as shown in Figure 3(b). When model error is small, Figure 3(c) shows that a moderate increase in the rollout horizon can make feasibility estimates more conservative, with little effect on overall performance.
A small number of unsafe samples are available for reflection. Both our original ablation and the new experiments in Appendix E.5 show that PROCO performs best when it can use unsafe samples during reflection. Without these samples, reflection relies on conservatism alone. The resulting policy can be overly conservative on some tasks and too aggressive on others, leading to a modest decline in overall performance. Even in this setting, PROCO still substantially improves safety over the FISOR baseline.