acceptodds
Withdrawn from ICLR 2027

SceneFactory-3D: Physical Grounded Counterfactual Simulation for Closed-Loop Traffic Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scalable driving simulators typically execute vehicle commands using prescribed behavioral or kinematic rules, overlooking the physics of tire-road interfaces, thereby limiting their ability to capture how adverse road and environmental conditions alter vehicle execution and propagate through traffic. To address this limitation, we present SceneFactory-3D, a GPU-batched, physics-grounded multi-agent driving simulator. Vehicles execute acceleration and steering commands via suspension- and friction-limited forces evaluated at each wheel-contact point. Spatially varying friction, per-world 3D heightfields, gravity, and rigid contact consistently govern wheel motion and chassis collisions. Per-world terrain isolation and GPU batching enable SceneFactory-3D to run matched physical counterfactuals in parallel: traffic scenario setup and vehicle controllers remain fixed while only the road condition changes, enabling the resulting closed-loop effects to be evaluated across parallel worlds. To demonstrate the advantage of the SceneFactory-3D-enabled counterfactual evaluation, we conduct an empirical study on vehicle controllers' sensitivity to road conditions. We study three learned-policy families on 1,024 matched 12-vehicle worlds per condition, and two classical planners on a shared 32-world subset, across 21 friction and grade conditions. When the friction coefficient drops from 1 to 0.18 due to slippery conditions, the vehicle passage rate drops by 18% and driving risk increases by 29%.

Primary area:
datasets and benchmarks
Keywords:
Physical counterfactual simulation, autonomous driving, closed-loop evaluation, traffic simulation, multi-agent interaction, GPU-parallel simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xt2DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Summary

The paper makes a solid technical contribution: a batched simulator that couples per-wheel tire forces to spatially varying friction and terrain, backed by a careful, reproducible codebase. However, it does not show why simulation at this physical level is needed, since it includes no kinematic or planar comparison and no planning-versus-tracking analysis. The evaluation, a single straight road with each condition in its own batch, does not exercise the features the paper advertises, and qualitative results are absent. I value the engineering and the codebase, but in its current form the experiments and presentation do not support the paper's main claims.

Critical strengths

Strengths

  • A relevant gap and a well-engineered platform. The platform combines GPU batching, per-wheel tire and contact forces, spatially varying friction, and per-world heightfields, which existing large-scale driving simulators lack (Table 2). The implementation is carefully verified: controller-free probes cover all 21 conditions (Table 5), and terrain consistency is checked across many worlds (App. C). The platform scales to 1,024 worlds × 12 vehicles on one GPU (Table 11).
  • Reproducibility and statistical practice above the norm. The release includes a frozen manifest with code, checkpoint, and terrain hashes. The design pairs results by world, reports training-seed variability separately, and keeps the degenerate seed instead of dropping it. Limitations are stated candidly (L466–472).
  • A valuable codebase. Classical and learned controllers share the same route interface, actuation, and physics. A batched simulator with physics in the loop could be useful for both training and evaluating planning and control methods.
Critical weaknesses

Weaknesses

W1. The need for force-level 3D simulation is not demonstrated.

The paper motivates itself by arguing that kinematic or behavioral simulation may overestimate safety (L049–051), and it presents the study as demonstrating the platform's advantage (L024–025). However:

  • No baseline. No experiment compares against kinematic or planar-dynamic execution on the same scenes and controllers.
  • What 3D adds is not shown. The main effects come from friction, which planar friction-limited models already capture, as the paper itself notes (L053–055). The only 3D-specific factor tested, longitudinal grade, barely changed outcomes for most checkpoints (L436). Banking, cited as motivation (L056), is never tested.
  • Planning and tracking are not separated. No controller explicitly accounts for friction limits. The paper also does not report how far executed motion departs from the planned trajectory as friction drops, e.g., for PDM-Closed's kinematic planner with an LQR tracker (L935–938).

I would ask for a kinematic or planar baseline, a friction-aware planner as a positive control, and an analysis of planned versus executed trajectories.

W2. The evaluation is too narrow to support the advertised features.

  • One scenario. All results come from one straight two-lane work-zone merge (L311–315). There are no curves, cross-slope, or intersections, although training already uses several scene families (Table 8).
  • Per-world terrain isolation is not used in the main evaluation. The paper presents it as what enables matched counterfactuals to run in parallel (L020–023). In the main evaluation, however, each condition runs in its own batch where every world has the same terrain (L1047). Batches that mix terrains appear only in the throughput test.
  • The title is not matched. Despite the title, no real-world traffic scene is lifted to 3D; the study uses one procedural road (Table 3).
  • The systems study is limited. Loop time is nearly constant from 64 to 1,024 worlds, which indicates unused headroom rather than measured efficiency (L1239–1241). There is no comparison with other simulators.

I would ask for a multi-scenario evaluation with batches that mix conditions or scenes, and for throughput comparisons on the same hardware.

W3. Qualitative results are missing.

  • Visual evidence consists of schematic figures (Fig. 1B, Fig. 2) and one trajectory plot (Fig. 6).
  • There are no renderings of the simulated worlds, no rollout videos, and no visualization of failure events.
  • Since the model already runs in Isaac Sim (L830–831), such visualizations should be easy to add.
Questions for authors

N/A

Reviewer_EfUsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SceneFactory-3D, a GPU-batched multi-agent driving simulator that incorporates detailed vehicle-road physical interactions, including per-wheel tire forces, spatially varying friction, and 3D terrain. The authors also introduce a matched counterfactual evaluation framework to study how different road conditions affect driving safety while keeping the traffic scenarios and controllers fixed. They evaluate several learning-based and classical controllers on a lane-closure task under different road conditions. The results show that lower friction significantly reduces vehicle passage rates and increases safety risks, while the benefits of terrain-aware training and physical preview vary across training seeds.

Critical strengths
  1. The detailed wheel-level physical modeling is a strength of this work. The authors separately model tire forces and suspension loads at each wheel, allowing road geometry and gravity to affect the available tire friction.

  2. The integration of 3D terrain and spatially varying road conditions into large-scale GPU-batched multi-agent simulation. This provides a useful framework for evaluating how physical road conditions affect driving safety.

Critical weaknesses
  1. The first concern is that it is difficult to see the practical benefits of the proposed improvements compared with existing simulators. The original SceneFactory already supports GPU-based multi-agent simulation with vehicle physics, while CARLA and Chrono also support wheel-level dynamics and 3D road environments. The paper mainly compares the features of these simulators, but lacks direct experimental comparisons to show whether the additional physical modeling leads to different or more useful safety evaluation results. This makes the value of the proposed improvements less convincing.

  2. The evaluation is largely limited to a single straight lane-closure scenario, which may not be sufficient to demonstrate the benefits of 3D terrain modeling. In particular, while lower friction significantly reduces vehicle passage rates, road grades have relatively little effect on most policies. Given that 3D terrain is one of the main contributions, these results do not seem to provide strong evidence of its practical importance. More complex road geometries, such as curved or uneven roads, could better demonstrate the value of this capability.

Questions for authors

My questions are based on the weaknesses discussed above.

  1. Could the authors further clarify the practical advantages of SceneFactory-3D over the original SceneFactory and other existing simulators? In particular, can the authors provide a direct comparison to show how the additional physical modeling leads to different or more meaningful safety evaluation results?

  2. Given that road grades have limited effects on most policies in the current experiments, could the authors explain how the evaluation supports the importance of 3D terrain modeling? Are there scenarios where 3D terrain and wheel-level physical interactions lead to safety outcomes that simpler physical models cannot capture?

Reviewer_aSDMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SceneBatch3D builds a 3D physics simulator that incorporates road–tire surface physics. Based on Isaac Lab, it uses a GPU-batched simulator to generate different road conditions for each scene. The experimental evaluation is limited in scope, using valid passage rate as the main metric, and shows that the learned policy is less sensitive to friction. As a result, the overall contribution remains somewhat unclear.

Critical strengths
  • Good transparency of providing code
  • Adding the physics of road condition does not reduce the throughput of the simulator, which other prior self-play methods can easily adopt
Critical weaknesses
  • What is the main takeaway of this paper? The experiments of Table 1 does not highlight the strength of SceneBatch3D. The author should highlight in what real-world scenarios, the road tire surfaces modeling or randomization will help.
  • For example, whether the problem can be decomposed by high-level decision making and low-level controller, and if this work's goal is to train the policy aware of low-level road conditions, the experiment should highlight what scenarios previous work overlooked
  • Contribution is unclear, Table 1 is confusing, shouldn't higher friction lead to better pass rate?
  • The training setup is also relatively simple, with only 12 vehicles per world (why not randomize number of agents during training?). More importantly, it is unclear whether the goal of this work is to train agents for multi-agent coordination or simply to improve vehicle control under different road conditions.
Questions for authors

Please refer to Weakness session

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.