acceptodds
Withdrawn from ICLR 2027

Controllable Crowd Generation through World-Model Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Crowd simulation plays a central role in robot navigation, autonomous driving, and urban planning. For these applications, realistic simulation requires crowds to adapt their behavior to environmental changes and user objectives. However, existing methods that rely on predefined control settings have limited flexibility in accommodating new user-specified objectives. To address this limitation, we propose Ctrl-CWM, a multi-agent Controllable Crowd World Model that integrates crowd generation and run-time control. Our key idea is to adapt the world-model principle of planning using imagined futures to crowd simulation. To this end, Ctrl-CWM consists of an encoder that learns a representation of human motion dynamics, an actor that proposes pedestrian displacements, a critic that evaluates imagined crowd trajectories, and a planner that selects actions. We first learn human motion dynamics through trajectory prediction on real-world pedestrian videos and then freeze the encoder to preserve them. Using this representation, the actor generates imagined crowd trajectories through repeated state updates, and the planner combines the critic's scores with user costs to select actions. Repeated planning advances the simulated crowd, while additional user costs introduce new control objectives without retraining. We extensively evaluate crowd generation under varied agent arrival conditions and run-time control across avoidance and attraction scenarios. Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, and adapts crowd behaviors to user-specified objectives introduced during simulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Crowd Simulation, Model-Based Planning, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NPkjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Ctrl-CWM, a framework for crowd generation and runtime spatial control through learned pedestrian behavior and sampling-based planning. At inference, a cross-entropy planner can take into account of user-defined spatial costs to support avoidance and attraction without retraining. Experiments show improvements over CrowdES on several generation metrics and higher average avoidance compliance than the evaluated baselines.

Critical strengths
  1. The same trained model can generate crowds and respond to newly introduced avoidance or attraction costs without retraining. This is practically valuable when simulation requirements change during execution.
  2. The critic provides a meaningful control–realism trade-off. Optimizing only the user cost achieves slightly better avoidance compliance, but worse motion metrics. This supports the idea that the learned critic helps keep controlled behavior plausible.
  3. The paper tests several datasets and two arrival mechanisms, with identical arrivals across methods. This helps separate improvements in pedestrian behavior from differences in how pedestrians enter the scene. The gains over CrowdES in trajectory matching and collision rates are useful evidence.
Critical weaknesses
  1. Compliance does not necessarily indicate successful navigation. Avoidance can improve when pedestrians stop outside a region, while attraction can improve through overcrowding. The appendix reports both stopping and increased collisions under stronger control, leaving realistic task completion insufficiently established.
  2. The control comparison uses unmatched intervention timing. Baseline obstacles are present from initialization, whereas Ctrl-CWM receives objectives during simulation. This limits conclusions about its advantage in responding to new objectives.
Questions for authors
  1. Any metrics or results that can help distinguish successful rerouting from stopping? For the avoidance metric, wonder how much the observed stopping behavior affects the results
  2. How much does simultaneous execution deviate from the independently imagined rollouts, particularly at higher crowd densities or stronger control weights?
Reviewer_gzdyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Ctrl-CWM for crowd generation and runtime control. It learns scene and interaction representations through pedestrian trajectory prediction, then trains an actor to generate displacements and a critic to rank imagined trajectories. A CEM planner combines the learned critic with user-defined spatial costs to support avoidance and attraction objectives without retraining. Experiments evaluate crowd generation, spatial control, trajectory prediction, and component ablations.

Critical strengths

Runtime control while preserving realistic pedestrian behavior is a relevant problem. Replaying identical arrivals across methods provides a useful controlled comparison of motion generation.

Critical weaknesses
  1. My concern about generalization is whether a world model trained on playgrounds, parks, or shopping malls can be applied to subway stations, railway stations, or airports without retraining. These environments differ in layout, density, and pedestrian behavior. The trajectory-prediction results are useful, but they do not establish this kind of transfer or show whether the training data provide sufficient behavioral coverage.

  2. Although the method includes an interaction encoder, it is unclear how it handles conflicts between groups of pedestrians, such as opposing flows meeting or several groups competing for limited space. These situations require closely coupled responses between agents. The paper does not sufficiently demonstrate whether its interaction modeling and planning can produce coordinated behavior in such encounters.

  3. The main crowd-generation and control experiments compare only against ORCA and CrowdES. ORCA is a useful classical baseline, but comparison with only one learning-based simulator is insufficient to establish a broad state-of-the-art claim. Comparisons with other relevant crowd-generation and controllable simulation methods would strengthen the evaluation.

  4. The discussion of existing crowd-simulation research could be more thorough, particularly regarding group interactions and behavioral diversity. Although several relevant methods are cited, their differences from the proposed approach are not examined in enough depth or supported by experimental comparisons.

  5. Realistic crowd simulation involves population dynamics, heterogeneous intentions, and diverse social behaviors, beyond avoiding or approaching specified regions. The method includes local goals and interaction features, but the evaluation provides limited evidence that it captures these broader aspects of crowd behavior. In particular, the experiments do not sufficiently test dense, strongly interacting crowds, leaving its effectiveness at larger scales unclear.

Questions for authors

pls address the Weaknesses

Reviewer_ikjBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on the problem of controllable crowd simulation, where pretrained crowd trajectories can be adapted in real time to satisfy user-defined objectives. To achieve this, the authors propose Ctrl-CWM, a world-model-based framework that learns crowd dynamics from trajectory data and performs planning in imagination. A critic evaluates predicted rollouts, which are then combined with user-defined costs in a CEM-style optimization procedure. The framework is evaluated on several crowd-navigation scenarios involving obstacle avoidance and attraction-based control.

The idea of leveraging world models for controllable crowd simulation is interesting and potentially impactful. However, I'm not convinced that the current results demonstrate a clear advancement over prior work on crowd control and steering.

Critical strengths

I found the proposed Ctrl-CWM framework well thought out and rather promising. World models have the potential to offer a compelling paradigm for controllable crowd simulation. The proposed system is also technically reasonable and well executed, enabling integration of trajectory prediction and run-time crowd control within a unified framework.

  • Application of world-model-based planning to crowd simulation

  • Integration of trajectory prediction and run-time crowd control within a unified framework

  • Ability to influence to a certain degree crowd behavior without retraining

Critical weaknesses
  • The tasks highlighted in the paper, particularly obstacle avoidance and attraction-based crowd steering, have been extensively studied across graphics, robotics, and crowd-simulation communities. While some of the showcased results are visually appealing, there has been substantial prior work on controllable crowd simulation and steering, including interactive, planning-based and learning-based approaches such as Interaction Fields, Navigation Fields, the RVO-based Proxy Agents, and CCP. More recently, TRACE supports crowd controlability through diffusion guidance; the authors criticize its low trajetctory sampling rate but this is largely a consequence of incorporating PACER for real-time physics-based character animation (see below). Without discussing and quantitatively comparing (to the degree that publicly available implementations permit) against these approaches, it is difficult to assess the contribution of the proposed work.

  • Although the paper emphasizes real-time controllability, the range of controllable behaviors appears relatively narrow. Because simulated agents share the same learned policy and dynamics model, behavioral diversity appears to be induced primarily through changes in attraction regions and forbidden regions during execution. Many other aspects of crowd behavior remain largely fixed by the learned dynamics, including movement styles and social interactions. As a result, the demonstrated controllability is largely restricted to obstacle avoidance and attraction-based steering. I would have liked to see experiments involving more complex behavioral modifications, such as group splitting and merging, heterogeneous agent populations, or changes in crowd flow structure.

  • The paper does not discuss the interplay between planned trajectories and character animation. Similar to most prior work, Ctrl-CWM operates at the trajectory level decoupling animation from planning, yet the perceived quality of a crowd simulation depends heavily on how the planned trajectories are realized through animation. As such, artifacts like footsliding, oscillations, etc are apparent in the companion video. Prior work such as TRACE & PACE has explicitly considered both trajectory generation and character animation within a unified framework leading to high animation quality at the cost of trajectory fidelity. I am not arguing that Ctrl-CWM should also solve the animation problem, but the authors should discuss this and better place their work within the context of prior works while acknowledging its limitations.

  • I would be cautious about using the term "realistic crowd". To me, establishing realism requires carefully designed perceptual studies involving human participants. Furthermore, the target population matters substantially, as crowd-simulation experts may evaluate realism using criteria that differ from those of non-experts. The metrics used in the paper provide a useful starting point, but they mainly assess statistical agreement with observed crowd trajectories rather than perceived realism. Personally, I did not find the resulting simulations particularly realistic. The main reason was the subpar animation quality that disctracted me from appreciating the behaviors that the simulated crowd was executing (see above).

  • It is unclear how computationaly expensive the proposed planning framework is. The authors should report runtime performance during inference as a function of the number of controllable agents

Questions for authors
  • How many agents are controlled in the results reported in Figure 5.

  • What is the runtime performance of the approach? Can you provide runtime numbers as a function of the # of controllable agents?

  • The metrics reported in Table 1 are not explained in sufficient detail. At minimum, the full metric names and brief descriptions should be included in the main paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.