acceptodds
Under review as a conference paper at ICLR 2027

RiskNav: Safety-Readable Latent World Models for Crowd Navigation

Abstract

Safe crowd navigation requires anticipating how candidate robot actions affect future human-robot interactions under partial observation. Reactive methods do not explicitly evaluate these futures, whereas predictive planners that optimize multi-agent trajectories online can become costly as crowds grow. Latent world models offer compact prediction, but their representations alone do not provide explicit safety quantities for risk-aware action selection. We present RiskNav, a risk-supervised, action-conditioned latent world model for crowd navigation. For each candidate action sequence, RiskNav rolls out future latent representations conditioned on the observation history and local waypoint. This latent rollout avoids explicitly predicting individual pedestrian trajectories or optimizing multi-agent trajectories online. A risk evaluator predicts static and dynamic safety margins and collision probabilities from each representation, with gradients from the resulting risk loss propagating through the predictor into the encoder. The planner incorporates these predicted safety quantities into a horizon-wise risk term for receding-horizon planning. Across indoor, open, and out-of-distribution settings, RiskNav achieves the highest strict success rates and the fewest robot-responsible contacts among nine baselines, with strict success rates of 0.76, 0.57, and 0.50, respectively. Replanning stays near 42ms as the crowd grows from 1 to 21 pedestrians. Ablations show that RiskNav's advantage comes from coupling safety-supervised latent prediction with risk-aware action selection, as strict success falls from 0.750 to 0.350 without representation shaping and to 0.125 without the planning-time risk term. RiskNav makes action-conditioned future risk directly usable for planning, improving both safety and success in crowd navigation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Crowd Navigation, World Models, Representation Learning, Risk-Aware Planning, Model Predictive Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bzteOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RiskNav is an action-conditioned, risk-supervised latent world model based on a Joint-Embedding Predictive Architecture (JEPA) that maps observation histories and candidate action sequences into a fixed-dimensional latent space to directly predict safety margins and collision probabilities without requiring pixel reconstruction or explicit individual trajectory forecasting. By using risk-loss backpropagation to shape a safety-readable latent space, RiskNav enables real-time Cross-Entropy Method (CEM) receding-horizon planning, achieving state-of-the-art navigation success and low collision rates across dense crowd simulations, out-of-distribution motion dynamics, and zero-shot real-world deployment on a Unitree H1-2 humanoid robot—all while maintaining a near-constant replanning latency of ~42 ms regardless of crowd size.

Critical strengths

(1) A critical strength supporting acceptance is the architecture's ability to maintain a constant computational complexity (O(1)O(1) scaling with respect to pedestrian count) by using a Joint-Embedding Predictive Architecture (JEPA) to roll out action-conditioned latent representations rather than forecasting individual agent trajectories or reconstructing high-dimensional observations. This design directly resolves the key bottleneck of traditional interaction-aware planners, enabling real-time receding-horizon control at ~42 ms latency even in dense crowds while achieving superior safety performance and successful zero-shot transfer to real-world humanoid hardware.

Critical weaknesses

(1) A critical weakness lies in the paper's reliance on a simplified 2D planar observation representation (a 128×128×3128 \times 128 \times 3 bird's-eye view raster map encoding history and static maps), which restricts the method's real-world applicability in unconstrained or non-flat environments. Because the latent world model is trained strictly on this stylized 2D spatial representation, the system remains vulnerable to perception noise, occlusion, 3D terrain variations, and fine-grained agent poses present in complex physical deployments, raising concerns about whether the safety-readable latent space can reliably generalize beyond controlled setups without requiring high-fidelity pre-mapped 2D inputs.

(2) Another major weakness is the heavy dependence on explicit privileged ground-truth state supervision during training to learn the risk evaluator and shape the latent space. Ground-truth distance metrics and collision probabilities require perfect spatial annotations and low-noise tracking data in training environments, which significantly limits the model's capacity to be trained or fine-tuned directly on raw, unannotated real-world sensor logs (such as onboard camera or LiDAR data) without expensive offline annotation pipelines.

Questions for authors

(1) How are the ground-truth risk supervisory signals (e.g., minimum pedestrian clearances and collision labels) constructed across the prediction horizon during offline training—does this rely on future ground-truth trajectory access, and how sensitive is the learned latent representation to perception/tracking noise in these labels?

(2) How does the latent world model handle severe partial observability and occlusions in the BEV raster input, and does the risk evaluator tend to produce overly optimistic safety estimates when pedestrians quickly enter the field of view from unobserved blind spots?

(3) What is the failure rate and computational overhead of the CEM planner when navigating highly constrained or dense dead-end topologies (e.g., narrow bottleneck corridors), where open-loop Gaussian action sampling often struggles to discover feasible collision-free trajectories?

(4) To what extent does performance degrade when deploying zero-shot in environments with non-stationary background dynamic obstacles (such as bicycles, mobility scooters, or pets) whose kinematics and interaction dynamics significantly deviate from the pedestrian models used during training?

Reviewer_5eThOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RiskNav, a framework for crowd navigation that uses an action-conditioned latent world model to predict the consequences of candidate robot actions without explicitly forecasting individual pedestrian trajectories. The main idea is to make these latent predictions useful for safety-aware planning by jointly training a risk evaluator to predict future safety margins and collision probabilities. These predictions are then incorporated into a receding-horizon planner to select actions that balance navigation progress and safety.

The authors evaluate RiskNav against nine baselines in indoor, open-field, and out-of-distribution simulation settings, reporting improved success rates and fewer robot-responsible contacts. They also show that planning latency remains nearly constant as crowd size increases. Ablation studies investigate the contributions of safety-supervised representation learning and risk-aware planning, while experiments on a physical humanoid robot provide initial evidence of zero-shot transfer to real-world navigation.

Critical strengths

(1) The use of safety supervision to guide latent representation learning is an interesting aspect of the paper. Rather than simply predicting future latent states, the method explicitly encourages the model to retain information about safety margins and collision risks that can be used during planning. The ablation results are particularly useful here, as removing either representation shaping or the planning-time risk term leads to a substantial drop in performance. This provides evidence that both components contribute to the overall approach.

(2) The experimental results are encouraging, particularly in terms of navigation performance and computational efficiency. RiskNav outperforms the evaluated baselines in strict success rate and robot-responsible contacts across indoor, open-field, and out-of-distribution settings. The nearly constant replanning latency as crowd size increases is also noteworthy, given the computational challenges of planning in crowded environments. The real-robot experiments, although limited in scope, provide additional evidence that the approach can transfer beyond simulation. These results make the method practically relevant and are an important positive factor in my assessment.

Critical weaknesses

(1) Most of the main experiments use an invisible-robot setting, where pedestrians do not react to the robot's actions. While this is a valid evaluation setting, it does not fully test one of the paper's main motivations, namely the ability to predict how different robot actions affect future human-robot interactions. The additional robot-aware evaluation in the appendix is useful, but the improvement over the strongest baseline (SARL) is not statistically significant. Moreover, the accuracy of counterfactual predictions under robot-responsive pedestrian behavior is not directly evaluated. This makes it difficult to assess how well the proposed approach handles genuinely interactive crowd navigation.

(2) I am not fully convinced that the paper isolates the contribution of the latent world model itself. The ablations show that safety supervision and the planning-time risk term are important, but there is no comparison with a simpler approach that directly predicts future safety margins and collision probabilities from observations and candidate actions, without explicitly learning and rolling out latent states. Such a baseline would help establish whether the proposed formulation offers a meaningful advantage over simpler alternatives. This is particularly important given that several components of the framework build on existing work. Furthermore, the main ablations are conducted on only 40 episodes in a single indoor configuration, which makes it harder to judge how consistently these findings generalize across different settings.

Post-rebuttal update — 19 November 2026

I thank the authors for their detailed response and the additional experiments, which address several of my concerns and strengthen the empirical evaluation. However, my main reservation remains the limited methodological novelty. While the results are encouraging, I am still not fully convinced that the paper offers sufficient new insight beyond combining existing ideas in latent prediction, safety supervision, and model-predictive control. I therefore maintain my initial recommendation.

Questions for authors

(1) The main experiments use pedestrians who do not react to the robot, while Appendix D.1 includes a robot-aware setting. Could the authors clarify how well the model captures changes in pedestrian behavior caused by different robot actions? In particular, it would be useful to know whether counterfactual prediction accuracy has been evaluated in the robot-aware setting, especially during close encounters where these interactions matter most.

(2) The ablation results show that safety supervision and risk-aware planning are important, but it is less clear how much of the improvement comes from the latent world model itself. How would RiskNav compare with a simpler model that directly predicts future safety margins and collision probabilities from observations and candidate actions, using the same supervision and planner? I think this comparison would help clarify the added value of the proposed latent prediction approach.

(3) Could the authors clarify some of the choices made in the baseline comparisons? In particular, Appendix C.2 mentions that SICNav considers only the six nearest pedestrians in the navigation benchmarks, while this restriction is removed for the latency experiment. How sensitive are the results to this choice? I would also be interested in understanding how the differences between retrained and zero-shot baselines affect the comparison, as this would help assess the significance of the reported improvements.

Reviewer_dbBGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RiskNav is an action-conditioned latent world model for crowd navigation. The input is a short history (three frames, 0.2 s apart) of bird's-eye-view rasters with separate channels for static obstacles, pedestrians and the robot with its heading, rendered from simulator states in simulation and from LiDAR tracks on the robot. A ViT encoder and a causal Transformer predictor, following LeWorldModel (JEPA prediction loss plus SIGReg), roll out latents for candidate action sequences conditioned on a local waypoint. A readout head predicts static and dynamic safety margins, collision probabilities, waypoint distance and progress from each predicted latent, trained with simulator labels, and these losses also update the predictor and encoder. At test time CEM selects actions with a navigation cost plus a discounted per-step risk cost in a receding-horizon loop. Evaluation uses indoor and open-field scenes with ORCA pedestrians that ignore the robot, and SFM pedestrians as an OOD case, against nine baselines. RiskNav gets the highest strict success and the fewest robot-responsible contacts, and replanning latency stays near 42 ms from 1 to 21 pedestrians.

Critical strengths
  1. The system is practical. Planning cost does not depend on the number of pedestrians, latency stays flat where SICNav grows to hundreds of milliseconds, and the per-step margins and collision probabilities are interpretable outputs a separate safety layer could use.
  2. The evaluation protocol is careful: nine baselines from three families, 1,200 paired episodes with shared seeds, paired bootstrap intervals, and robot-aware and SFM variants with full metric tables (reach, freeze, minimum separation, time, SPL) in the appendix.
  3. The ablation cleanly separates shaping the representation with the safety loss from using the risk prediction at planning time, and shows both matter. The counterfactual check that branches the simulator to score unexecuted plans (Fig. 3b,c) is a good idea, and the larger error in close encounters is reported honestly.
  4. The paper is clear and easy to follow.
Critical weaknesses
  1. The setting is not convincing. The BEV input is built from detection and tracking (simulator states or LiDAR tracks), so the pedestrians are already tracked trajectories before the model sees them. Rendering them back into rasters and learning a world model on top is a roundabout form of trajectory prediction and planning, and it drops the usual reason for a world model, learning from the robot’s raw egocentric observations under partial observability. The paper never argues why this beats predicting the tracked trajectories directly or modeling the egocentric view. The main benchmark also uses pedestrians that ignore the robot, so the action-dependent interaction that motivates the paper never occurs there.
  2. The paper has limited novelty. The backbone is LeWM, planning is DINO-WM-style CEM, and latent-shaping prediction heads follow TD-MPC. Jointly trained cost heads with CEM planning (SafeDreamer, Huang et al., ICLR 2024), per-step collision probabilities in sampling MPC (BADGR, Kahn et al., RA-L 2021), BEV latent rollouts scored by simulator-supervised collision heads (WoTE, Li et al., ICCV 2025) and safety classifiers on world-model latents (Nakamura et al., RSS 2025) exist and are not cited.

• Huang, W., Ji, J., Xia, C., Zhang, B., and Yang, Y. SafeDreamer: Safe Reinforcement Learning with World Models. ICLR 2024. arXiv:2307.07176. • Kahn, G., Abbeel, P., and Levine, S. BADGR: An Autonomous Self-Supervised Learning-Based Navigation System. IEEE Robotics and Automation Letters, 2021. arXiv:2002.05700. • Li, Y., Wang, Y., Liu, Y., He, J., Fan, L., and Zhang, Z. End-to-End Driving with Online Trajectory Evaluation via BEV World Model. ICCV 2025. arXiv:2504.01941. (WoTE) • Nakamura, K., Peters, L., and Bajcsy, A. Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis. RSS 2025. arXiv:2502.00935.

Questions for authors
  1. Why learn a world model on rendered BEV rasters of already-tracked agents, rather than predicting the tracked trajectories directly or modeling the robot's egocentric observations?
  2. Is the JEPA objective needed, and what does RiskNav add over SafeDreamer, BADGR and WoTE beyond the backbone and dense margin labels? Please ablate the JEPA loss (a purely supervised action-conditioned risk predictor), SIGReg, and margin-only vs probability-only risk cost over several training seeds. Also explain why "w/o Risk cost" freezes on 86% of steps and reaches only a quarter of goals despite the analytic waypoint-tracking term; is the robot pinned against walls, which RC does not count?
  3. On the robot, which radii were used, how many near-contacts occurred, and how does a simple baseline such as DWA do on the same route?

DWA: Dynamic Window Approach(Fox, Burgard, Thrun, IEEE Robotics & Automation Magazine 1997)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.