- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CRAX is a hardware-accelerated safe reinforcement learning benchmark built on MJX and JAX. It adapts navigation and velocity tasks from Safety-Gymnasium, introduces additional constrained-control tasks, and supports multiple agent morphologies, three difficulty levels, and vector or pixel observations. The paper evaluates seven safe RL baselines alongside unconstrained PPO, with additional experiments on off-policy methods, hyperparameter sensitivity, training-time violations, curriculum learning, and transfer. It reports substantial throughput improvements over Safety-Gymnasium and uses the resulting computational capacity for broader evaluations, generally with ten seeds. The main contribution is a benchmark and software infrastructure rather than a new learning algorithm.
Critical strengths
1.The computational improvement has practical value for SafeRL research. Moving simulation and learning onto the accelerator substantially reduces experimental cost. This enables more seeds, broader hyperparameter sweeps, and larger task coverage, which can improve the reliability of future algorithm comparisons.
2.The benchmark covers several constraint types and includes substantial supporting infrastructure. The new tasks extend beyond navigation, while the evaluation examines both final performance and training-time violations. The accessible anonymous repository provides implementations, documentation, and demonstration assets, making the contribution more useful than a collection of reported scores alone.
Critical weaknesses
-
The visual-learning evaluation omits directly relevant baselines and provides limited diagnostic insight. Section 5.2 evaluates PPO, PPOLag, P3O, and FOCOPS with a CNN, but does not include methods specifically developed for visual SafeRL, such as LAMBDA, SafeDreamer, or Safe SLAC. Although the omission of model-based methods is acknowledged, it materially limits what can be concluded about the difficulty of the visual benchmark for existing approaches. Moreover, unconstrained PPO also fails on five tasks, so these results do not isolate safety-specific difficulties from general visual-control failures. The poorer allocentric performance is reported without controlled analysis of its cause.
-
Limited qualitative comparison of learned behaviors. Although the repository includes selected demonstration videos, the paper lacks a systematic qualitative comparison of the policies learned by different baseline algorithms. Reward and cost statistics alone provide limited insight into how the agents satisfy constraints or why particular methods fail. Representative rollouts and behavioral analyses, especially on the newly introduced tasks, would help distinguish effective constrained task completion from overly conservative or inactive behavior and clarify the practical meaning of the reported performance differences.
-
Limited novelty beyond engineering integration. As AI-assisted coding potentially lowers the effort required for such implementations, I place greater weight on demonstrated technical advances, rigorous validation, and informative experimental findings. The current paper provides limited evidence on these dimensions, particularly in its analysis of visual-learning failures and the behaviors learned across algorithms.
Questions for authors
1.Visual baseline coverage. What motivated the exclusion of established visual SafeRL methods such as LAMBDA, SafeDreamer, and Safe SLAC? Can you provide a representative comparison with a method designed for pixel observations, or delimit the conclusions of Section 5.2 explicitly to the evaluated baselines? Such a comparison would help distinguish benchmark difficulty from limitations of the selected visual learners.
2.Allocentric performance and observation configuration. What evidence explains the poorer allocentric results? In particular, what are the exact camera, frame-history, and actor/critic observation settings used for Figure 5? Table 11 and the current configuration default specify one frame, whereas docs/VISION_TRAINING.md describes a three-frame default. Have you checked whether the viewpoint gap persists when temporal information and image scale are controlled?
5 comments
Q2 Allocentric performance and observation config
Configuration for Figure 5. Input: each agent receives a single 64×64 RGB frame, with no frame history. Encoder: a Nature-DQN CNN produces 1024 features per camera. The policy, reward critic, and cost critic all share it, and all three consume only the image. Augmentation: random-shift augmentation with 4-pixel padding (Table 11). Egocentric camera: mounted on the agent. Allocentric camera: selected based on the most favourable position. In Point Goal and Point Push, it follows the agent from above at an angle. In Reacher and Point Circle, it stays fixed over the arena. In Spider Lift, Walker Pathway, and Humanoid Velocity/Height, it follows from the side. Documentation: the three-frame default in docs/VISION_TRAINING.md was outdated. We have aligned the documentation and configuration with the setup used in the paper and added these details to the paper.
Temporal information We trained unconstrained PPO from each camera with one frame, three stacked frames, and a GRU:
| Env | Camera | 1 frame (R / C) | 3 frames (R / C) | GRU (R / C) |
|---|---|---|---|---|
| Point Goal | Ego | 29.3 / 108.6 | 25.1 / 136.7 | 0.3 / 274.0 |
| Point Goal | Allo | 13.9 / 121.8 | 12.0 / 202.6 | 0.3 / 306.5 |
| Point Circle | Ego | 78.6 / 51.5 | 7.3 / 86.5 | 4.6 / 90.7 |
| Point Circle | Allo | 6.0 / 88.5 | 5.0 / 88.9 | 5.9 / 89.1 |
| Point Push | Ego | 30.0 / 108.2 | 5.6 / 203.4 | -0.3 / 260.1 |
| Point Push | Allo | 2.4 / 120.6 | 0.3 / 187.3 | -0.2 / 202.8 |
| Reacher | Ego | 11.4 / 45.4 | 172.3 / 41.2 | 8.3 / 54.7 |
| Reacher | Allo | 8.3 / 75.0 | 8.5 / 52.8 | 8.4 / 54.0 |
On Point Goal and Point Push, egocentric agents outperform allocentric ones both with a single frame and with three frames. On Reacher, temporal history widens the gap: three frames lift egocentric reward from 11.4 to 172.3, while allocentric stays at 8.5. Temporal information therefore does not explain the allocentric deficit. On the locomotion tasks (Walker Pathway, Spider Lift, Humanoid Velocity, Humanoid Height), neither viewpoint learns a competent policy, so there is no meaningful gap to analyze (full table in our response to Reviewer s4hW).
Image resolution. We agree with the reviewer that the pixel results may reflect the observation setup rather than benchmark difficulty. In particular, at 64×64 the agent and hazards occupy only a few pixels in the allocentric view. We therefore retrained PPO at 128×128 with all other settings unchanged.
| Env | Camera | 64×64 R / C | 128×128 R / C |
|---|---|---|---|
| Goal | Ego | 28.6 / 103 | 18.6 / 132 |
| Goal | Allo | 19.4 / 125 | 0.4 / 434 |
| Circle | Ego | 0.8 / 55 | 90.9 / 58 |
| Circle | Allo | 6.0 / 88 | 8.4 / 87 |
| Push | Ego | 36.1 / 108 | 23.1 / 135 |
| Push | Allo | 1.7 / 109 | 0.1 / 12 |
| Reacher | Ego | 10.8 / 45 | 10.1 / 47 |
| Reacher | Allo | 8.3 / 54 | 9.0 / 31 |
Higher resolution does not fix the difficulty. With the allocentric camera, reward stays near zero or falls on every task. On Goal, both reward and cost get worse. On Push, cost drops only because reward falls to zero. Reacher is the only allocentric case that slightly improves. With the egocentric camera, 128×128 helps a lot on Circle but lowers reward on Goal and Push. On Reacher, neither camera benefits. So resolution does not explain the failures. Since learning from pixels alone is hard even for unconstrained RL, we include proprioception combined with an egocentric camera as an additional setting. It outperforms both the pixel-only and the proprioception + lidar settings (see our response to Reviewer s4hW).
W2 Qualitative behaviors
We agree that reward and cost alone do not show how agents satisfy constraints. We have added a section in the appendix with representative rollout frames for the following.
We identify the cases where agents are effective, conservative, or inactive. Reading reward together with how much of the cost budget a policy uses, already separates these cases. Effective constrained policies reach high reward while operating close to the bound. Examples are P3O and FOCOPS on Point Goal Levels 2–3 (reward 56–73, cost ≈24–27 against a bound of 25). Overly conservative policies stay far below the bound at a large reward cost. Examples are FOCOPS on Walker Pathway (reward ~850 vs. ~9,000 for PPOLag/P3O, at cost ≈6) and SACLag on Point Goal and Ant Velocity (costs 5.4 and 0.4). Inactive policies satisfy the constraint by abandoning the task. On Spider Lift Level 3, reward collapses to ≈0 for all constrained methods. As the episode begins, the agent immediately flips over to cause episode termination to avoid the cost of not keeping the designated legs off the ground.
We further identify distinct behaviors per method. Inspecting rollouts shows that methods with similar scores often solve tasks very differently. In Humanoid Height, methods consistently develop distinct gaits under the ceiling constraint. CRPO crawls on all fours. PPOLag, PPOPID, and FOCOPS lie down and drag themselves forward. P3O learns the most effective strategy, running on its feet while bent forward beneath the ceiling hazard. On Point Push, all methods except P3O and FOCOPS often push the block into a hazard zone and then leave it there. Retrieving it would require entering the hazard, so the agent accepts losing the goal reward rather than incurring the cost. On Point Goal (Levels 2–3), with state observations, agents of all methods sometimes try to pass between two solid hazards through a gap too narrow to fit. They repeatedly collide with the hazards, which heavily inflates cost. With proprioception and an egocentric camera, agents learn to escape this trap.
W1/Q1 Visual baselines
Thank you for this suggestion. Our original pixel study used the same model-free baselines as the rest of the benchmark, so that state-based and pixel-based results are directly comparable. We agree that this leaves open whether failures reflect the benchmark or the chosen learners. Following your suggestion, we therefore added Safe SLAC [1], a method designed specifically for safe RL from pixels, using the hyperparameters from the original paper. Since Safe SLAC was designed for and evaluated on first-person camera observations, we evaluate it on the egocentric camera:
Safe SLAC achieves the highest reward while satisfying the cost bound on Point Goal, Point Circle, and Reacher. The margin is especially large on Point Circle and on Reacher, where every model-free method, including unconstrained PPO, fails to learn from single frames and exceeds the bound. On both tasks, Safe SLAC even surpasses unconstrained PPO in reward. On Point Push, Safe SLAC remains safe but does not learn to push the block. This matches the original paper, where the authors report that Safe SLAC "fails to make substantial improvement in reward return in PointPush1" (Safety-Gymnasium) while remaining safe, and attribute this to strong partial observability. These results show that, on these tasks, the poor pixel-based performance in Section 5.2 largely reflects the limitations of the evaluated model-free learners rather than the benchmark itself, and that methods designed for pixel observations can make substantial progress. We have added Safe SLAC to Section 5.2 and revised its conclusions to refer explicitly to the evaluated model-free baselines.
[1] Y. Hogewind et al. Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation. ICLR, 2023.