acceptodds
Withdrawn from ICLR 2027

PADSIM: A Photorealistic Closed-Loop Simulator with Controllable Pedestrians for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Closed-loop simulation is essential for evaluating autonomous driving systems (ADS) in safety-critical urban scenarios. However, existing simulators primarily focus on vehicle-centric behaviors and largely overlook the impact of pedestrian dynamics on decision-making. To address this limitation, we present , a photorealistic closed-loop simulator that enables controllable and interaction-consistent pedestrian behaviors in dynamic environments. PADSIM supports flexible placement and motion control of pedestrians while ensuring physically plausible and collision-free interactions with surrounding agents. At the core of our system is an Interaction-Driven Motion Generation (IDMG) pipeline, which produces scene-aware pedestrian trajectories by explicitly modeling multi-agent interactions and incorporating physical constraints. To maintain visual realism, we further introduce a Global-Local Illumination-Consistent Synthesis (GLICS) strategy that preserves human–scene appearance consistency by leveraging environmental lighting cues without additional optimization or training. By tightly coupling motion generation, rendering, and policy inference in a recurrent loop, PADSIM enables reactive simulation beyond open-loop replay. In addition, we propose a system-level complexity metric to quantify the risk and difficulty of generated scenarios from the ego vehicle’s perspective. Experiments on real-world datasets demonstrate that PADSIM supports realistic augmentation of pedestrian behaviors, facilitates policy-driven scenario evolution, and enables more comprehensive evaluation of autonomous driving systems under safety-critical human–vehicle interactions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, closed-loop simulator, pedestrian behaviors

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wS5XOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents PADSIM, a 3D Gaussian Splatting (3DGS)-based closed-loop simulator for autonomous driving with controllable pedestrian augmentation. The system integrates interaction-aware pedestrian trajectory generation, avatar animation, and illumination-consistent rendering. Specifically, the proposed IDMG pipeline combines social-force-based motion modeling with collision avoidance, while GLICS uses estimated scene illumination to adjust the appearance of inserted pedestrian Gaussians. The authors also introduce three metrics to quantify scenario complexity.

Experiments evaluate pedestrian collision rates against RESCUE, illumination consistency before and after relighting, scenario complexity, and the closed-loop performance of several autonomous driving policies. The authors report improved collision avoidance, better illumination consistency, and increased scenario complexity.

Critical strengths

The paper addresses a relevant problem in autonomous driving simulation. Controllable pedestrian behaviors are important for evaluating driving policies under safety-critical interactions.

The integration of pedestrian motion generation, animated Gaussian avatars, scene rendering, and driving policy inference into a unified simulation framework is a potentially useful engineering contribution. The reported collision-rate improvements over RESCUE also suggest that the interaction-aware motion module can improve pedestrian placement and trajectory feasibility.

Critical weaknesses

1. Insufficient downstream validation and questionable practical value.

The paper does not convincingly demonstrate that PADSIM provides meaningful or reliable autonomous driving evaluation. Table 6 only reports driving performance on two augmented scenarios, without controlled comparisons against the original scenes or other simulation methods. Moreover, Figures 4 and 5 show noticeable appearance discrepancies between inserted pedestrians and the backgrounds. It is unclear whether the observed driving failures result from challenging pedestrian interactions or synthetic visual artifacts. The paper also fails to establish why PADSIM is preferable to existing closed-loop benchmarks such as HABIT, as the inserted pedestrains still remain mesh style.

2. Unclear closed-loop interaction and limited methodological novelty.

Algorithm 1 and Appendix E.4 describe pedestrian motion generation as an offline process, while Algorithm 2 claims online pedestrian updates in response to ego actions. This inconsistency raises doubts about whether pedestrians genuinely react to the driving policy. The real-time claim also lacks end-to-end FPS measurements. Furthermore, the proposed components largely combine existing techniques, including social force models, ORCA, PACER, and conventional illumination models, without sufficiently demonstrating the advantages of their integration.

The manuscript also contains obvious presentation issues. Appendix A begins with the unfinished placeholder "I will add a discussion of related work on relighting here." Additionally, Algorithm 2 incorrectly references Eqs. (26)–(27) for HD-Score computation, although these equations define collision-rate metrics. Such errors raise concerns about the manuscript's technical verification.

Questions for authors

Can the authors evaluate downstream planner performance after reducing the noticeable visual artifacts in the inserted pedestrians? This may require an additional harmonization module or pipeline, potentially leveraging video generation models or APIs. Given the substantial appearance discrepancies in the current visual results, I find it difficult to assess whether the reported planning failures reflect meaningful pedestrian interactions or artifacts introduced by the simulation.

Reviewer_Kmz1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a photorealistic simulator for autonomous driving. The key contribution lies in a controllable motion generation module that generate plausible pedestrian trajectories and motions, and a relighting module to fit the GS human avatars to the environment lighting. Experimental results show the effectiveness of each module, which results in overall more realistic and complex scenarios for testing autonomous driving models.

Critical strengths
  1. The resulting autonomous driving scenarios are realistic both in pedestrian motion and image rendering.
  2. The whole simulator requires significant engineering effort especially for the pedestrian motion generation module.
Critical weaknesses
  1. The method claims it can generate realistic pedestrian motion and behavior, but the pedestrian trajectory is governed by rule-based approaches (social force model and ORCA), and the pedestrian motion is generated by a physics-based low-level motion controller PACER trained from limited data. The resulting behavior and motion may thus lack the diversity and realism compared to real-world pedestrians.
  2. Lack of explanation and motivation for the proposed scenario complexity metrics. Their definitions seem arbitrary without any references to prior works and contain many hyperparameters and weight coefficients. Its unclear whether such a complex metric is useful for evaluating the scenario complexity.
  3. The experiments mainly focus on evaluating the realism and the plausibility of the generated scenarios but whether such a simulation platform is useful for downstream AD model testing is unclear. For example, for the proposed complexity metric, does higher complexity also indicate lower model success rates? For the pedestrian simulation, does more realistic behaviors result in less sim2real gap in the model predictions? The paper should have more analysis in these down-stream policy perspectives.
  4. Lack of novelty. Both the motion generation and the illumination synthesis module are based on some existing methods without specific design/modification for the AD domain.
Questions for authors
  1. Can the authors provide more analysis on the use of the proposed simulator for downstream AD evaluation and show the proposed modules are helpful in this aspect?
  2. Can the authors provide more explanation and motivation on the design choices of the proposed scenario complexity metrics and their relevance to the AD evaluation?
Reviewer_Z9kyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a closed-loop simulation for pedestrian behaviors in dynamic environments to support vehicle-pedestrian interaction research. The core of the system includes a pedestrian motion generation pipeline via multi-agent interaction modeling and a global-local illumination-consistent synthesis to improve visual realism. Pedestrian behavior generation and simulation during dynamic interaction remains as a challenging topic and is much needed for current automate driving and physical AI research.

Critical strengths

None.

Critical weaknesses

The pedestrian behavior model still uses existing social-force and collision-avoidance methods, so its novelty is limited. The paper does not clearly address the known limitations, such as sudden behavior changes, cognitive factors like distraction and limited situation awareness, and the potential effects of contextual conditions. At practical level, the work may be useful, but the paper does not show that this makes the simulated pedestrian behavior more realistic.

Although the paper generates body poses through a trajectory-conditioned physics controller, it does not guarantee the generated motion represents the temporal relationships between pedestrian body cues, behavioral intentions, and subsequent trajectories. Physical feasibility does not ensure that orientation changes, hesitation, or other intention-relevant cues are represented realistically. Kinematics and pose information is usually fused for both human drivers or automated driving algorithms to infer pedestrian intentions, predict trajectories, and make decisions. The current simulation lacks the key capability to simulate realistic human behaviors in that sense.

Questions for authors

The scenario complexity metrics is a arbitrary composition of several existing metrics. I am wondering why this is the best or a good metric. It will be helpful for the authors to explain the intuitions and novelty behind this design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.