acceptodds
Under review as a conference paper at ICLR 2027

CRAX: Fast Safe Reinforcement Learning Benchmarking

Abstract

Safety is a core concern for deploying reinforcement learning (RL) agents in real-world domains such as robotics and autonomous driving. While benchmarks have been central to progress in RL, existing 3D physics-based safety benchmarks remain computationally slow, limiting large-scale experimentation and rapid prototyping. To address this gap, we propose **CRAX** (**C**onstrained **R**L **A**ccelerated with JA**X**). Built on top of the MuJoCo XLA (MJX) physics engine, CRAX leverages vectorized operations and hardware acceleration, yielding up to 200x faster training over comparable CPU-based safety benchmarks. The benchmark features eight tasks spanning three difficulty levels and multiple agent morphologies. Evaluating seven popular safe RL methods, we find that none dominates across tasks, and that learning safe policies from pixels remains largely unsolved.

Primary area:
datasets and benchmarks
Keywords:
reinforcement learning, safe RL, safety, benchmark, JAX, 3D, physics, simulation, MuJoCo, XLA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XH8zOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

CRAX is a hardware-accelerated safe reinforcement learning benchmark built on MJX and JAX. It adapts navigation and velocity tasks from Safety-Gymnasium, introduces additional constrained-control tasks, and supports multiple agent morphologies, three difficulty levels, and vector or pixel observations. The paper evaluates seven safe RL baselines alongside unconstrained PPO, with additional experiments on off-policy methods, hyperparameter sensitivity, training-time violations, curriculum learning, and transfer. It reports substantial throughput improvements over Safety-Gymnasium and uses the resulting computational capacity for broader evaluations, generally with ten seeds. The main contribution is a benchmark and software infrastructure rather than a new learning algorithm.

Critical strengths

1.The computational improvement has practical value for SafeRL research. Moving simulation and learning onto the accelerator substantially reduces experimental cost. This enables more seeds, broader hyperparameter sweeps, and larger task coverage, which can improve the reliability of future algorithm comparisons.

2.The benchmark covers several constraint types and includes substantial supporting infrastructure. The new tasks extend beyond navigation, while the evaluation examines both final performance and training-time violations. The accessible anonymous repository provides implementations, documentation, and demonstration assets, making the contribution more useful than a collection of reported scores alone.

Critical weaknesses
  1. The visual-learning evaluation omits directly relevant baselines and provides limited diagnostic insight. Section 5.2 evaluates PPO, PPOLag, P3O, and FOCOPS with a CNN, but does not include methods specifically developed for visual SafeRL, such as LAMBDA, SafeDreamer, or Safe SLAC. Although the omission of model-based methods is acknowledged, it materially limits what can be concluded about the difficulty of the visual benchmark for existing approaches. Moreover, unconstrained PPO also fails on five tasks, so these results do not isolate safety-specific difficulties from general visual-control failures. The poorer allocentric performance is reported without controlled analysis of its cause.

  2. Limited qualitative comparison of learned behaviors. Although the repository includes selected demonstration videos, the paper lacks a systematic qualitative comparison of the policies learned by different baseline algorithms. Reward and cost statistics alone provide limited insight into how the agents satisfy constraints or why particular methods fail. Representative rollouts and behavioral analyses, especially on the newly introduced tasks, would help distinguish effective constrained task completion from overly conservative or inactive behavior and clarify the practical meaning of the reported performance differences.

  3. Limited novelty beyond engineering integration. As AI-assisted coding potentially lowers the effort required for such implementations, I place greater weight on demonstrated technical advances, rigorous validation, and informative experimental findings. The current paper provides limited evidence on these dimensions, particularly in its analysis of visual-learning failures and the behaviors learned across algorithms.

Questions for authors

1.Visual baseline coverage. What motivated the exclusion of established visual SafeRL methods such as LAMBDA, SafeDreamer, and Safe SLAC? Can you provide a representative comparison with a method designed for pixel observations, or delimit the conclusions of Section 5.2 explicitly to the evaluated baselines? Such a comparison would help distinguish benchmark difficulty from limitations of the selected visual learners.

2.Allocentric performance and observation configuration. What evidence explains the poorer allocentric results? In particular, what are the exact camera, frame-history, and actor/critic observation settings used for Figure 5? Table 11 and the current configuration default specify one frame, whereas docs/VISION_TRAINING.md describes a three-frame default. Have you checked whether the viewpoint gap persists when temporal information and image scale are controlled?

5 comments

W1/Q1 Visual baselines

Thank you for this suggestion. Our original pixel study used the same model-free baselines as the rest of the benchmark, so that state-based and pixel-based results are directly comparable. We agree that this leaves open whether failures reflect the benchmark or the chosen learners. Following your suggestion, we therefore added Safe SLAC [1], a method designed specifically for safe RL from pixels, using the hyperparameters from the original paper. Since Safe SLAC was designed for and evaluated on first-person camera observations, we evaluate it on the egocentric camera:

EnvSafe SLACPPOPPO-LagP3OFOCOPS
Point Goal30.9 / 22.928.4 / 108.826.7 / 22.626.4 / 23.017.0 / 22.4
Point Circle119.0 / 15.863.6 / 54.762.7 / 18.259.3 / 34.157.5 / 19.0
Point Push1.2 / 11.631.1 / 101.629.1 / 25.426.2 / 25.35.7 / 22.5
Reacher65.2 / 15.410.5 / 51.88.8 / 32.58.8 / 32.37.9 / 37.8

Safe SLAC achieves the highest reward while satisfying the cost bound on Point Goal, Point Circle, and Reacher. The margin is especially large on Point Circle and on Reacher, where every model-free method, including unconstrained PPO, fails to learn from single frames and exceeds the bound. On both tasks, Safe SLAC even surpasses unconstrained PPO in reward. On Point Push, Safe SLAC remains safe but does not learn to push the block. This matches the original paper, where the authors report that Safe SLAC "fails to make substantial improvement in reward return in PointPush1" (Safety-Gymnasium) while remaining safe, and attribute this to strong partial observability. These results show that, on these tasks, the poor pixel-based performance in Section 5.2 largely reflects the limitations of the evaluated model-free learners rather than the benchmark itself, and that methods designed for pixel observations can make substantial progress. We have added Safe SLAC to Section 5.2 and revised its conclusions to refer explicitly to the evaluated model-free baselines.

[1] Y. Hogewind et al. Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation. ICLR, 2023.

Q2 Allocentric performance and observation config

Configuration for Figure 5. Input: each agent receives a single 64×64 RGB frame, with no frame history. Encoder: a Nature-DQN CNN produces 1024 features per camera. The policy, reward critic, and cost critic all share it, and all three consume only the image. Augmentation: random-shift augmentation with 4-pixel padding (Table 11). Egocentric camera: mounted on the agent. Allocentric camera: selected based on the most favourable position. In Point Goal and Point Push, it follows the agent from above at an angle. In Reacher and Point Circle, it stays fixed over the arena. In Spider Lift, Walker Pathway, and Humanoid Velocity/Height, it follows from the side. Documentation: the three-frame default in docs/VISION_TRAINING.md was outdated. We have aligned the documentation and configuration with the setup used in the paper and added these details to the paper.

Temporal information We trained unconstrained PPO from each camera with one frame, three stacked frames, and a GRU:

EnvCamera1 frame (R / C)3 frames (R / C)GRU (R / C)
Point GoalEgo29.3 / 108.625.1 / 136.70.3 / 274.0
Point GoalAllo13.9 / 121.812.0 / 202.60.3 / 306.5
Point CircleEgo78.6 / 51.57.3 / 86.54.6 / 90.7
Point CircleAllo6.0 / 88.55.0 / 88.95.9 / 89.1
Point PushEgo30.0 / 108.25.6 / 203.4-0.3 / 260.1
Point PushAllo2.4 / 120.60.3 / 187.3-0.2 / 202.8
ReacherEgo11.4 / 45.4172.3 / 41.28.3 / 54.7
ReacherAllo8.3 / 75.08.5 / 52.88.4 / 54.0

On Point Goal and Point Push, egocentric agents outperform allocentric ones both with a single frame and with three frames. On Reacher, temporal history widens the gap: three frames lift egocentric reward from 11.4 to 172.3, while allocentric stays at 8.5. Temporal information therefore does not explain the allocentric deficit. On the locomotion tasks (Walker Pathway, Spider Lift, Humanoid Velocity, Humanoid Height), neither viewpoint learns a competent policy, so there is no meaningful gap to analyze (full table in our response to Reviewer s4hW).

Image resolution. We agree with the reviewer that the pixel results may reflect the observation setup rather than benchmark difficulty. In particular, at 64×64 the agent and hazards occupy only a few pixels in the allocentric view. We therefore retrained PPO at 128×128 with all other settings unchanged.

EnvCamera64×64 R / C128×128 R / C
GoalEgo28.6 / 10318.6 / 132
GoalAllo19.4 / 1250.4 / 434
CircleEgo0.8 / 5590.9 / 58
CircleAllo6.0 / 888.4 / 87
PushEgo36.1 / 10823.1 / 135
PushAllo1.7 / 1090.1 / 12
ReacherEgo10.8 / 4510.1 / 47
ReacherAllo8.3 / 549.0 / 31

Higher resolution does not fix the difficulty. With the allocentric camera, reward stays near zero or falls on every task. On Goal, both reward and cost get worse. On Push, cost drops only because reward falls to zero. Reacher is the only allocentric case that slightly improves. With the egocentric camera, 128×128 helps a lot on Circle but lowers reward on Goal and Push. On Reacher, neither camera benefits. So resolution does not explain the failures. Since learning from pixels alone is hard even for unconstrained RL, we include proprioception combined with an egocentric camera as an additional setting. It outperforms both the pixel-only and the proprioception + lidar settings (see our response to Reviewer s4hW).

W2 Qualitative behaviors

We agree that reward and cost alone do not show how agents satisfy constraints. We have added a section in the appendix with representative rollout frames for the following.

We identify the cases where agents are effective, conservative, or inactive. Reading reward together with how much of the cost budget a policy uses, already separates these cases. Effective constrained policies reach high reward while operating close to the bound. Examples are P3O and FOCOPS on Point Goal Levels 2–3 (reward 56–73, cost ≈24–27 against a bound of 25). Overly conservative policies stay far below the bound at a large reward cost. Examples are FOCOPS on Walker Pathway (reward ~850 vs. ~9,000 for PPOLag/P3O, at cost ≈6) and SACLag on Point Goal and Ant Velocity (costs 5.4 and 0.4). Inactive policies satisfy the constraint by abandoning the task. On Spider Lift Level 3, reward collapses to ≈0 for all constrained methods. As the episode begins, the agent immediately flips over to cause episode termination to avoid the cost of not keeping the designated legs off the ground.

We further identify distinct behaviors per method. Inspecting rollouts shows that methods with similar scores often solve tasks very differently. In Humanoid Height, methods consistently develop distinct gaits under the ceiling constraint. CRPO crawls on all fours. PPOLag, PPOPID, and FOCOPS lie down and drag themselves forward. P3O learns the most effective strategy, running on its feet while bent forward beneath the ceiling hazard. On Point Push, all methods except P3O and FOCOPS often push the block into a hazard zone and then leave it there. Retrieving it would require entering the hazard, so the agent accepts losing the goal reward rather than incurring the cost. On Point Goal (Levels 2–3), with state observations, agents of all methods sometimes try to pass between two solid hazards through a gap too narrow to fit. They repeatedly collide with the hazards, which heavily inflates cost. With proprioception and an egocentric camera, agents learn to escape this trap.

Reviewer_juxoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a high-throughput benchmark for safe RL, namely CRAX, a JAX-accelerated platform for constrained RL. The main idea is to execute environment simulation, policy inference, data collection, and neural network updates on a single GPU, while running a large number of environments in parallel to accelerate safe RL experimentation and evaluation. Using cumulative cost constraints as the safety criterion, the authors conduct experiments with several representative safe RL algorithms, including Lag-based methods, CRPO, and P3O.

Critical strengths

1.In evaluating safe RL algorithms, the paper considers not only cumulative cost constraints but also the extent of constraint violations during training, as reported in the appendix. This provides a more comprehensive evaluation of algorithmic safety.

2.Compared with CPU-based benchmarks, the proposed platform achieves up to a 200× speedup in training, making it more practical to conduct experiments with multiple random seeds.

Critical weaknesses

1.The footnote below Table 1 states that two robotic manipulation suites in Safety-Gymnasium are GPU-accelerated through Safe Isaac Gym. However, Safe Isaac Gym in Safety-Gymnasium actually includes four tasks: FreightFrankaCloseDrawer, FreightFrankaPickAndPlace, ShadowHandCatchOver2UnderarmSafe, and ShadowHandOverSafe. All four tasks satisfy the Safety, GPU, and 3D Phys. criteria in Table 1 and should be explicitly listed. In addition, ReDMan[1] is an open-source simulation platform that provides standardized implementations of safe RL algorithms for Reliable Dexterous Manipulation, which also satisfies these three criteria. Therefore, Table 1 overlooks existing safe RL benchmarks that already meet the Safety, GPU, and 3D Phys. requirements.

2.There are also some concerns regarding the experimental performance of the baseline algorithms. In particular, the implementation of CRPO should account for the relationship between the number of reward-optimizing updates and cost-reducing updates, as discussed in Lemma 9 of the original CRPO paper, which may lead to better performance. However, the paper does not mention whether this detail is considered in its CRPO implementation. This omission may partly explain the relatively poor performance of CRPO reported in the experiments.

[1] Geng Y, Ji J, Chen Y, et al. ReDMan: reliable dexterous manipulation with safe reinforcement learning: Y. Geng et al[J]. Machine Learning, 2025, 114(8): 189.

Questions for authors

1.The claim in the caption of Table 1 that CRAX "uniquely combines a focus on safety with hardware acceleration and 3D physics-based tasks" is not sufficiently justified. For example, ReDMan also supports explicit safety constraints, GPU-based parallel simulation, and 3D physics-based environments. What are the additional contributions of CRAX compared with existing GPU-accelerated safe RL benchmarks?

2.Does the CRPO implementation in this paper consider a separate hyperparameter controlling the ratio of reward-optimizing updates to cost-reducing updates? Accounting for this ratio may potentially improve the performance of CRPO.

3.Figure 6 demonstrates the throughput advantage of CRAX over selected CPU-based tasks in Safety-Gymnasium. However, how does the computational efficiency of CRAX compare with existing GPU-accelerated safe RL environments, such as Safe Isaac Gym within Safety-Gymnasium?


Update: Thank you for your response and explanation. After going through all the responses and comments from other reviewers, I will keep my score for now.

6 comments

W1/Q1 Table 1 positioning (Safe Isaac Gym, ReDMan)

Thank you for pointing this out. We agree that the caption overstates CRAX's uniqueness, and we have revised Table 1.

  • We list Safe Isaac Gym as a separate row, with a footnote noting that it is part of Safety-Gymnasium. It runs on a different simulator (Isaac Gym rather than CPU MuJoCo) and covers a different domain, so a separate row ensures that each row's criteria apply to the same set of tasks. The footnote names its tasks: FreightFrankaCloseDrawer, FreightFrankaPickAndPlace, ShadowHandCatchOver2Underarm and ShadowHandOver, the latter two in Finger and Joint constraint variants.
  • We added ReDMan as a row satisfying the Safety, GPU, and 3D Physics criteria.
  • We added a column, Nav./Loco., indicating whether a benchmark provides continuous-control navigation or locomotion tasks. The three original criteria alone do not show which domains a benchmark covers. The new column makes clear that existing GPU-accelerated safe RL benchmarks (Safe Isaac Gym, ReDMan) target manipulation, whereas existing navigation and locomotion safe RL benchmarks (Safety Gym, Safety-Gymnasium, Bullet-Safety-Gym) are CPU-bound.
  • We revised the caption to: "CRAX is the first GPU-accelerated safe RL benchmark for navigation and locomotion."

Compared with existing GPU-accelerated safe RL benchmarks, CRAX contributes the following:

1. Coverage of the central safe RL domain. Constrained navigation and locomotion remains the standard evaluation setting in safe RL. Beyond the foundational works (CPO, PPO-Lag, PID-Lagrangian, FOCOPS, CRPO), recent methods are still evaluated primarily on Safety Gym and Safety-Gymnasium navigation and velocity tasks, e.g. SafeDreamer [1], CAL [2], ActSafe [3], safety-prioritizing curricula [4], robust safety transfer [5], safety representations [6], and FDPI [7]. All of these run on CPU-bound simulation. Within this domain, CRAX provides (1) eight tasks with three difficulty levels each, (2) eight agent morphologies, and (3) five constraint types: hazard, velocity, height, contact, and boundary. The manipulation benchmarks instead mainly constrain joint ranges or the robot's position relative to objects.

2. End-to-end compilation. Isaac Gym runs physics on the GPU, but the learning loop remains a Python/PyTorch program around the simulator. In CRAX, physics, cost computation, pixel rendering, rollouts, and all policy and critic updates compile into a single JAX program. This allows, for example, vectorizing entire training runs across seeds or hyperparameters on one device.

3. Long-term usability. Both ReDMan and Safe Isaac Gym depend on Isaac Gym, which NVIDIA has deprecated in favor of Isaac Lab and no longer supports. Its closed-source binaries do not run on recent GPU architectures (Blackwell). CRAX builds on MJX, which is open source and actively maintained, and runs on any accelerator supported by JAX/XLA.

4. Familiarity and continuity. The navigation and velocity suites descend directly from Safety-Gymnasium, so existing safe RL familiarity and intuitions carry over. Appendix G validates fidelity with a matched comparison against Safety-Gymnasium + OmniSafe.

We have revised the related-work discussion accordingly.

[1] Huang et al. SafeDreamer: Safe Reinforcement Learning with World Models. ICLR, 2024.
[2] Wu et al. Off-Policy Primal-Dual Safe Reinforcement Learning. ICLR, 2024.
[3] As et al. ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning. ICLR, 2025.
[4] Koprulu et al. Safety-Prioritizing Curricula for Constrained Reinforcement Learning. ICLR, 2025.
[5] Zubia et al. Robust Transfer of Safety-Constrained Reinforcement Learning Agents. ICLR, 2025.
[6] Mani et al. Safety Representations for Safer Policy Learning. ICLR, 2025.
[7] Yang et al. Breaking Safety Paradox with Feasible Dual Policy Iteration. ICLR, 2026.

W2/Q2 CRPO implementation

Thank you for this point. In the original CRPO (Algorithm 1), there is no separate ratio hyperparameter. Each iteration takes a reward step if the estimated cost is within d + η, and a cost step otherwise. The ratio is therefore set by the tolerance η: Lemma 9 gives a condition on α, η and T that guarantees at least T/2 reward iterations. We follow this switching rule with a PPO-clip surrogate instead of NPG, and use η = 0 (Table 10). The paper did not describe the quantity the switch compares with d. We now report it in the appendix, together with the reward-step fractions below. The switch compares d with the mean undiscounted per-step training cost of the last rollout minibatch, scaled by the episode length. The table reports the fraction of reward steps, the final switch estimate, and the evaluation cost of the final policy. It does so for the default optimiser settings and for the tuned settings of Appendix H.1.

EnvVariantReward-step frac (all / last 10%)Switch estimateCost
Reacherdefault0.62 / 0.6221.921.2
Reachertuned0.58 / 0.6024.324.2
Point Goaldefault0.61 / 0.6524.824.8
Point Goaltuned0.64 / 0.6724.124.3
Point Pushdefault0.48 / 0.5125.425.0
Point Pushtuned0.53 / 0.5624.424.4
Point Circledefault0.48 / 0.5125.124.9
Point Circletuned0.50 / 0.5124.824.7
Spider Liftdefault0.62 / 0.6921.16.7
Spider Lifttuned0.45 / 0.5624.89.4
Walker Pathwaydefault0.39 / 0.4825.57.3
Walker Pathwaytuned0.06 / 0.1427.65.2
Humanoid Heightdefault0.81 / 1.004.24.9
Humanoid Heighttuned0.66 / 0.916.24.9
Humanoid Velocitydefault0.80 / 0.4431.929.0
Humanoid Velocitytuned0.48 / 0.4730.328.2

A natural way to use η > 0 without letting the policy exceed the true limit is to run CRPO with tolerance η on the tightened limit d − η. Since d and η enter the update only through the threshold d + η, this gives the threshold (d − η) + η = d, which is exactly the switch in our runs. Our reported CRPO is therefore identical to CRPO with any tolerance η on the limit d − η. On Point and Reacher, it takes reward steps in at least half of the late updates, which is the outcome Lemma 9 is meant to guarantee. A higher share of reward steps requires a threshold above d, which by construction permits a higher cost. To show this trade-off, we raised the tolerance with the limit kept at d = 25 per episode. We used Point Goal, Circle, and Push. η is implemented as a per-step rate times the episode length, so the same three settings give η = 5 / 12.5 / 25 per episode on Goal and Circle (1000-step episodes) and 10 / 25 / 50 on Push (2000 steps), up to three times the limit. Table entries are reward / cost:

Envη = 0η = lowη = midη = high
Point Goal33.2 / 24.8 (25)34.8 / 29.7 (30)34.7 / 37.1 (37.5)35.4 / 49.2 (50)
Point Circle114.5 / 24.7 (25)112.7 / 30.6 (30)112.8 / 37.2 (37.5)113.2 / 68.0 (50)
Point Push17.3 / 25.0 (25)16.1 / 35.1 (35)35.6 / 48.8 (50)60.8 / 67.6 (75)

The final cost settles at the raised threshold d + η. Only Circle at η = low / high overshoots. The switch thus keeps the cost at whatever threshold it is given. Reward does not improve while the extra budget is small. Only Push gains reward at η = mid / high, where the policy incurs more cost. We therefore do not think the reward:cost update ratio explains CRPO’s reward gap to the Lagrangian baselines. At the true limit, the switching rule fixes the ratio. Loosening it trades constraint violation for reward on one task and does not help the other two. On Lift and Walker, where CRPO takes too few reward steps, the cause is the bias of the switch estimate, not the ratio.

Q3 Throughput

Thank you for the suggestion. We measured the simulation throughput of all four Safe Isaac Gym tasks on the same hardware (H100) and compared it with the three CRAX tasks of Figure 6. Values are in thousands of steps per second, measured as environment stepping with random actions:

SuiteTask256 envs1,0244,09616,384
CRAXPoint Goal67.8190.2544.1925.2
CRAXAnt Velocity55.2202.4613.31,062.1
CRAXHumanoid Velocity23.487.2252.6337.6
Safe Isaac GymShadowHandOver_Safe_joint13.655.4328.0612.2
Safe Isaac GymFreightFrankaCloseDrawer20.564.0331.7526.8
Safe Isaac GymFreightFrankaPickAndPlace27.088.9392.6848.2
Safe Isaac GymShadowHandCatchOver2Underarm_Safe_joint11.154.7241.6522.0

Both benchmarks reach throughput of the same order of magnitude, far beyond CPU-based simulation (3K–20K steps per second for Safety-Gymnasium). CRAX is faster on Point Goal and Ant Velocity at every scale, reaching about 1M steps per second. Humanoid Velocity, with the most complex morphology in CRAX (23 DoF), is comparable at small scales and slower at large ones. Since the tasks and morphologies differ, this comparison reflects simulation cost per task rather than a like-for-like engine comparison.

We do not claim a raw speed advantage over GPU-based simulators, nor did we do so in the paper. CRAX's contribution lies in bringing this throughput to safe navigation and locomotion, which existing GPU-accelerated safe RL benchmarks do not cover, and in compiling simulation and learning end-to-end.

Reviewer_s4hWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

CRAX is an MJX/JAX benchmark for constrained reinforcement learning, offering eight task suites, multiple robot morphologies, three difficulty levels, and vector or pixel observations. The authors implement several safe RL baselines and study final performance, training-time violations, hyperparameter sensitivity, curriculum learning, and computational throughput. The main contribution is an accelerated experimental platform that makes larger-scale safe RL studies affordable.

Critical strengths
  1. Practical value of accelerated experimentation. Moving simulation and learning into the same GPU execution pipeline addresses a real bottleneck. The measured throughput improvements on shared task families, together with the implemented baselines, could substantially lower the cost of developing and evaluating safe RL methods.

  2. More extensive evaluation than a throughput demonstration alone. Ten-seed experiments, training curves, safety-budget sweeps, an off-policy comparison, and analyses of curriculum learning and hyperparameter sensitivity provide useful evidence about the benchmark. In particular, reporting violations during training complements the final-policy results.

Critical weaknesses
  1. The paper does not establish that all baselines are evaluated against the same safety objective. Section 3 defines discounted cumulative cost and a constraint for every initial state. Section 5 instead evaluates undiscounted episode sums, while Table 10 states that multiplier and penalty updates use the per-step threshold d/Td/T. These quantities are not generally interchangeable, particularly with task-dependent horizons of 200–2,000 steps and early termination. PPOSauté additionally uses a budget discount of 0.99. Its original formulation concerns an augmented safety state and almost-sure budget satisfaction [1], so its adaptation to the common benchmark objective needs to be explicit. Table 15 shows that changing its budget discount from 0.99 to 0.999 radically changes its apparent safety/performance trade-off. Without the precise budget recursion, normalization, and terminal handling, it is unclear how much of its poor reported safety reflects the algorithm versus a mismatch between its configured constraint and the reported undiscounted metric. This is central to a benchmark intended to compare constraint-handling methods.

  2. Several algorithm-level conclusions are confounded by optimization and observation choices. The main comparison uses shared Brax defaults, despite the paper's own sweeps demonstrating substantial ranking changes. For example, Table 13 raises PPOLag's Point Push reward from 25.3 to 81.2 by changing only the PPO clip from 0.3 to 0.1, with costs of 24.6 and 24.9 respectively; the latter exceeds the default FOCOPS reward of 50.3. This makes the reported winners a comparison of particular configurations rather than a robust characterization of the algorithms. The pixel study has a related attribution problem: Table 11 supplies one RGB frame without temporal history, while the vector input contains velocities and other state information. Section 5.2 reports that even unconstrained PPO fails on five environments. Those failures do not isolate a challenge specific to safe RL from partial observability or a weak visual-control baseline. Both issues limit the scientific conclusions drawn from an otherwise useful implementation.

[1] Sootla et al. Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation. ICML 2022

Questions for authors
  1. Safety-objective clarification: What exact objective does each implementation optimize: discounted episodic cost, undiscounted episodic cost, or average per-step cost? Please give the budget conversion and early-termination handling, especially for PPOSauté, and reconcile these with Section 3 and the common d=25d=25 evaluation rule.

  2. Fair algorithm comparison: What rankings remain after giving each method a comparable tuning budget and evaluating the selected configurations with fresh seeds? How sensitive are the reported safe percentages and wins to uncertainty around the cost threshold, rather than classifying methods solely by their mean cost?

  3. Pixel-control diagnosis: Do the negative results persist with frame stacking or recurrence, or with proprioceptive information matched to the vector baseline? Establishing a competent unconstrained visual baseline would help determine which failures are caused by safety learning itself.

  4. Reported evaluation protocol: Section 5 describes 128 evaluation episodes for the final policy, whereas Appendix H describes reported reward/cost as sums from training rollouts. Which measurements populate Figures 4–5 and Tables 3 and 16, and how are the confidence intervals constructed?

5 comments

W1/Q1 Safety objective

Good catch! The writing is indeed inconsistent on this. In all experiments, we report the undiscounted expected episodic cost, as in other popular Safe RL benchmarks, and we revised Sections 1-3 and 5 accordingly. The discount γ\gamma appears only in the reward and cost critics, as the usual variance-reducing surrogate for GAE advantage estimation. It is not part of the constraint. We now state explicitly in Section 3 that this is a surrogate: its effective horizon (~100 steps) is shorter than our episodes. The constraint itself is enforced by the multiplier and penalty updates, which compare the mean undiscounted cost of episodes completed in each rollout batch against dd. We have also fixed the Table 10 caption, which wrongly referred to a per-step threshold left over from early experiments. We have also corrected the writing for CRPO’s tolerance "per-step" (Table 10) and SACLag’s "divided by T" (Table 11). Episodic cost is summed until termination, so ending an episode early lowers cost. Like any trivial policy (e.g., standing still in Goal/Circle), this is always feasible but earns near-zero reward, so it never wins. Our tasks also make falling unattractive: agents lose the per-step healthy reward and forward-progress terms, and Pathway adds a terminal cost (c_term, Table 5). For this reason, we report reward alongside Safe%.

We also appreciate pointing out the issue with PPOSaute. We re-ran it on all our environments with a cost budget discount of γ=1.0\gamma=1.0, so that it enforces the same undiscounted episodic constraint as the other methods to keep the comparison fair. The safety budget is normalized to z₀ = 1 and updated as zₜ₊₁ = zₜ − cₜ / d. The agent observes zₜ, and once zₜ ≤ 0, its reward is replaced by the violation penalty (−1) until the episode ends. We report the new results below.

EnvOld R / CNew R / C
reacher176.6 / 6.2188.6 / 10.9
goal_point33.9 / 94.029.2 / 18.6
push_point41.7 / 244.326.8 / 23.5
lift_spider11.7 / 49.812.0 / 22.5
circle_point120.2 / 46.082.6 / 6.8
height_humanoid4848 / 17.37756 / 10.8
pathway_walker2d9095 / 17.78322 / 12.3
velocity_humanoid2867 / 480.52662 / 10.4

PPOSaute now successfully adheres to the cost bound of 25, maintaining moderate to high rewards.

W2a/Q2 Tuning confounds rankings

Thanks for raising this concern. Indeed, the fixed shared configuration can favour some methods over others. In the submission, we kept the Brax defaults for all methods to control for the base optimizer and isolate the constraint-handling mechanism. However, our own sweeps show that this choice is not neutral. We therefore gave every method a comparable tuning budget, by extending the sweeps of Appendix H.1 to all environments, and reran our full evaluation suite with the best shared and safety-specific hyperparameters with fresh seeds. The fast throughput of CRAX makes this feasible even within the rebuttal window. For brevity, we present the new Level 1 results of differences in reward and cost in the table below, and have updated our figures & tables and Section 5.1.

AlgorithmReacherPoint_GoalPoint_PushSpider_LiftPoint_CircleHumanoid_HeightWalker_PathwayHumanoid_Velocity
PPO+6% / -52%+7% / +2%+21% / -65%+7% / -3%+3% / +1%+34% / -1%+89% / -5%+93% / -2%
PPOCost+140% / -14%+29% / -91%+9% / -8%+29% / -6%+11% / +4%-9% / -34%-7% / -2%+6% / -8%
PPOLag+94% / +4%+9% / 0%+186% / +43%+50% / -4%+29% / +27%+33% / +39%+84% / -6%+88% / +7%
PPOSaute+7% / +35%+4% / -98%-15% / -96%+3% / -61%-11% / -98%+60% / -52%-13% / -71%-7% / -95%
PPOPID+132% / +6%+11% / 0%+347% / +26%+23% / -3%+30% / +7%-5% / +2%+60% / -3%+84% / -77%
P3O+109% / -7%+12% / -2%+61% / -6%+7% / +8%+31% / +19%+97% / -36%+57% / -3%+25% / -6%
FOCOPS+38% / +2%+1% / -1%+23% / -9%+50% / -5%+48% / +4%0% / -16%+574% / +115%+298% / +116%
CRPO+67% / +14%+11% / -2%+83% / -2%+64% / +4%-7% / 0%+57% / -3%+94% / -29%+85% / -2%

To address what rankings remain, we include a sensitivity analysis by recomputing the summary counting a method as safe only when the full 95% CI of its cost lies below dd (values in parentheses). Note that environments where no method meets this stricter criterion have no winner. We keep the safety criterion aligned with the objective, i.e., the mean episodic cost below dd.

AlgorithmL1 WinsL1 Safe%L2 WinsL2 Safe%L3 WinsL3 Safe%Total WinsTotal Safe%
PPO1 (1)25 (25)1 (1)12 (12)0 (0)0 (0)2 (2)12 (12)
PPOCost0 (1)88 (88)0 (1)38 (38)1 (1)25 (25)1 (3)50 (50)
PPOLag3 (3)75 (50)0 (1)100 (75)0 (0)100 (50)3 (4)92 (58)
PPOPID3 (1)88 (62)0 (0)50 (50)1 (1)88 (62)4 (2)75 (58)
PPOSaute0 (0)100 (100)0 (0)25 (12)0 (0)12 (12)0 (0)46 (42)
P3O0 (1)75 (62)3 (2)62 (38)2 (2)62 (50)5 (5)67 (50)
FOCOPS1 (1)75 (62)2 (2)88 (75)4 (3)100 (88)7 (6)88 (75)
CRPO0 (0)88 (38)2 (1)62 (50)0 (0)38 (38)2 (1)62 (42)

The strongest methods under this classification remain P30 & FOCOPS. The safest method under the 95% CI is no longer PPOLag, but FOCOPS. We agree that it is useful to additionally report the CI-based view, while keeping the mean cost as the safety criterion, as is customary in the safe RL literature.

To quantify how sensitive the summary is to uncertainty in the cost estimates, we bootstrap over seeds (10,000 resamples), reclassifying every method in each resample with the same mean-cost criterion (see table below). The brackets give 95% percentile intervals. P(most wins) is the fraction of resamples in which a method attains the most total wins (ties split equally). Safe% is fairly stable, varying by about one environment per level for most methods. Wins are less stable, since many classifications sit close to the bound. FOCOPS is the most robust method, attaining the most wins in 72% of resamples. P3O matches it on the point estimate, but its wins depend on near-threshold classifications (95% interval [3, 7]), and it attains the most wins in only 15% of resamples. We have added this analysis to Appendix I and revised Section 5.1 to name FOCOPS as the most robust method, with P3O competitive but less reliable.

AlgorithmL1 WinsL1 Safe%L2 WinsL2 Safe%L3 WinsL3 Safe%Total WinsTotal Safe%P(most wins)
PPO1 [0, 2]25 [12, 25]1 [1, 1]12 [12, 12]0 [0, 0]0 [0, 0]2 [1, 3]12 [8, 12]0%
PPOCost0 [0, 1]88 [75, 88]0 [0, 2]38 [38, 50]1 [0, 2]25 [25, 25]1 [1, 4]50 [46, 54]0%
PPOLag3 [1, 5]75 [50, 88]0 [0, 2]100 [75, 100]0 [0, 2]100 [75, 100]3 [1, 6]92 [75, 96]4%
PPOPID3 [1, 5]88 [62, 100]0 [0, 1]50 [50, 75]1 [0, 2]88 [75, 100]4 [2, 6]75 [67, 83]9%
PPOSaute0 [0, 1]100 [100, 100]0 [0, 0]25 [12, 25]0 [0, 1]12 [12, 25]0 [0, 1]46 [42, 50]0%
P3O0 [0, 2]75 [62, 88]3 [0, 3]62 [50, 75]2 [1, 3]62 [50, 62]5 [3, 7]67 [58, 75]15%
FOCOPS1 [0, 3]75 [62, 88]2 [2, 3]88 [75, 100]4 [2, 4]100 [88, 100]7 [4, 9]88 [79, 92]72%
CRPO0 [0, 0]88 [50, 88]2 [1, 3]62 [50, 75]0 [0, 2]38 [38, 62]2 [1, 4]62 [54, 71]0%

W2b/Q3 Pixel observations

Thank you for raising this point. We would first like to point out that the most common setting for safe RL evaluations in the literature has been the state-based setting [1, 2, 3, 4, 5]. Hence, researchers will likely gain the most benefit from running this setting in an accelerated training fashion. Learning from pixels alone is indeed very difficult, and as you correctly pointed out, this is a challenge not necessarily to safe RL, but even for general RL methods. We have adjusted this in the writing. For this setting, the 3 environments are already a good starting point. Following your suggestion, we investigated whether frame stacking or a recurrent layer would help unconstrained PPO perform better. It turns out that both lead to similar or worse performance in most cases:

EnvCamera1 frame (R / C)3 frames (R / C)GRU (R / C)
Point GoalEgo29.3 / 108.625.1 / 136.70.3 / 274.0
Point GoalAllo13.9 / 121.812.0 / 202.60.3 / 306.5
Point CircleEgo78.6 / 51.57.3 / 86.54.6 / 90.7
Point CircleAllo6.0 / 88.55.0 / 88.95.9 / 89.1
Point PushEgo30.0 / 108.25.6 / 203.4-0.3 / 260.1
Point PushAllo2.4 / 120.60.3 / 187.3-0.2 / 202.8
ReacherEgo11.4 / 45.4172.3 / 41.28.3 / 54.7
ReacherAllo8.3 / 75.08.5 / 52.88.4 / 54.0
Walker PathwayEgo205.5 / 5.196.9 / 5.094.7 / 5.0
Walker PathwayAllo201.4 / 5.1100.0 / 5.098.7 / 5.0
Spider LiftEgo0.0 / 113.20.1 / 36.50.1 / 940.3
Spider LiftAllo0.0 / 29.60.0 / 446.20.0 / 222.1
Humanoid VelocityEgo22.9 / 0.03.0 / 0.113.6 / 0.1
Humanoid VelocityAllo24.4 / 0.19.5 / 0.121.2 / 0.1
Humanoid HeightEgo-81.8 / 12.6-50.6 / 2.839.6 / 4.3
Humanoid HeightAllo370.7 / 20.1-55.9 / 2.7-20.0 / 3.6

Learning from pixels alone is challenging even for unconstrained RL and is less common in the safe RL literature, so we do not aim to establish a strong pixel-only baseline. We have softened the corresponding claims and instead focus on the more realistic setting of proprioception combined with an egocentric camera, which is closer to actual robots. A robot would rarely need to act only from camera inputs. It is more plausible to also have access to its proprioceptive state. Users of the benchmark will likely make more use of this type of evaluation. We therefore evaluate this setting on CRAX environments. We process the proprioceptive and egocentric camera observations via their respective encoders and combine the latent vectors with a simple concatenation, to be further processed by a few extra layers. We find that this multi-modality setting greatly surpasses the state-based and pixel-only variants, notably on higher difficulty levels.

Env (level)AlgorithmProprio + lidar (R / C)Pixels only (R / C)Proprio + ego camera (R / C)
Point Push L3CRPO10.1 / 25.212.7 / 23.738.3 / 24.1
Point Push L3P3O13.9 / 25.617.9 / 26.746.3 / 25.3
Point Push L3PPOLag5.8 / 24.98.2 / 24.836.9 / 25.0
Point Goal L3CRPO12.1 / 25.99.5 / 29.717.4 / 22.7
Point Goal L3P3O56.0 / 23.512.3 / 26.672.1 / 26.6
Point Goal L3PPOLag7.0 / 18.813.0 / 24.523.8 / 25.1
Walker Pathway L3CRPO628.0 / 7.3170 / 6.28911 / 23.2
Walker Pathway L3P3O2098.4 / 13.3241 / 7.27020 / 21.3
Walker Pathway L3PPOLag2076.2 / 14.8-30 / 5.06569 / 23.8

These results confirm your intuition. The failures in the pixel-only setting stem from perception rather than from safety itself. Once the agent has access to proprioceptive information, vision becomes an asset rather than a bottleneck, and the constrained methods can make use of the additional information while staying at or near the cost bound. Isolating safety-specific challenges therefore requires a competent observation pipeline, which the proprioceptive + egocentric setting now provides in CRAX. We have included the results and discussion in the paper.

[1] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained Policy Optimization. ICML, 2017.
[2] A. Ray, J. Achiam, and D. Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. arXiv:1910.01708, 2019.
[3] A. Stooke, J. Achiam, and P. Abbeel. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. ICML, 2020.
[4] Y. Zhang, Q. Vuong, and K. Ross. First Order Constrained Optimization in Policy Space. NeurIPS, 2020.
[5] T. Xu, Y. Liang, and G. Lan. CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee. ICML, 2021.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear AC, dear reviewers,

As the discussion period ends tomorrow, we would like to summarize the main changes made during the rebuttal so far.

A clear highlight is the new observation setting for robotic applications (closer to real robots). Based on the comments of s4hW and XH8z, we have added a setting that combines proprioception with camera inputs. On the hardest level, it clearly outperforms the standard proprioception + lidar setting, most commonly used in safe RL evaluations.

New experiments

  • Following s4hW's suggestion, we tuned each algorithm independently, which strengthens the comparison between algorithms. Furthermore, we ran a sensitivity analysis to increase the robustness of such comparisons.
  • Pixel observations experiments: as suggested by XH8z and s4hW, we have investigated the effect of different approaches (frame stacking / GRU) to handle pixel-based observations in an unconstrained baseline to isolate the challenges related to pixel-based and safety-aware learning and included a new pixel-based baseline (Safe SLAC).
  • We reran PPOSauté with its budget matched to the undiscounted episodic constraint. It now satisfies the cost bound on all eight tasks (s4hW).
  • We thoroughly verified the implementation of CRPO (juxo).
  • We compared the throughput of CRAX with Safe Isaac Gym (juxo).

Based on the new experiments, we would like to highlight that running such experiments was only possible thanks to CRAX's high-throughput training, which enables constrained RL research at a previously infeasible scale.

Further improvements

  • We narrowed the novelty claims to ensure a fair representation of prior work that also considers and clarifies the distinction from existing GPU-accelerated safe RL benchmarks. In particular, CRAX focuses on navigation and locomotion, whereas ReDMan and Safe Isaac Gym target manipulation (juxo).
  • We now consistently use the undiscounted episodic cost, and we have clarified our evaluation protocol (s4hW).
  • We have described the qualitative behaviours that emerge across different baselines (XH8z).

We thank all reviewers for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.