- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CARLAGEN4D, a framework that converts text prompts into executable CARLA simulation environments by generating road layouts, background assets, and actor trajectories. A compact DSL and compiler feedback support map construction and repair. Authors provide experimental results on a benchmark for text-to-map generation, covering over 250 prompts, and test 7 driving policies on held-out generated maps to demonstrate the downstream application of the proposed method. Additionally, the paper shows that the generated environments could be used as training data for both driving policy training, perception model training and reasoning model training, showing the potential of the proposed framework.
Critical strengths
- The paper is focusing on the useful topic for testing / evaluation for autonomous driving. Generating the road network itself addresses a practical limitation of systems that place scenarios in existing towns. And the experimental results are encouraging, CARLAGEN4D produces valid map artifacts for 98.4% of the 250 prompts and receives a VLM score of 4.54, compared with 4.02 for OmniTester. This capability makes it possible to construct environments around a requested interaction rather than search for a suitable location in a fixed map.
- The ablation studies provide clear evidence for the DSL and compiler feedback. Under a common model and evaluation budget, the complete method produces 39 clean maps out of 50, compared with 12 when generating raw OpenDRIVE and 33 without iterative repair. The DSL also reduces output tokens by 2.6 times and generation time by 2.1 times relative to direct OpenDRIVE generation, demonstrating both the effectiveness and efficiency of the proposed framework.
- The downstream training experiments across different tasks help demonstrate the practical value of the generated scenes. The authors evaluate waypoint prediction, camera to BEV perception, and driving narration and justification. With a matched training frame budget, generated data reduces waypoint error from 7.79 to 3.10 m and improves BEV mIoU from 0.110 to 0.364 on unseen generated maps, while also improving performance on existing CARLA towns. These results suggest that the framework provides useful training coverage beyond its role in scenario generation.
Critical weaknesses
- The distinction between producing a valid map and fully satisfying the prompt needs to be clearer. The main evaluation reports 98.4% valid artifacts, while the separate ablation reports clean completion on 39 of 50 prompts. These results use different criteria and prompt distributions, so they are not directly contradictory. However, returning an earlier map after budget exhaustion may leave requested features incomplete. Similarly, the 69.4% elevation score measures whether any grade is present, rather than whether the requested elevation structure is correct.
- The trajectory comparison largely reflects whether a suitable map is available. CARLAGEN4D executes all 39 scenarios, whereas TTSG executes 12, but 26 of TTSG’s 27 failures occur before simulation begins. This supports the value of generating compatible road networks, but provides less evidence that the trajectories themselves are better. Motion fidelity scores are close, at 3.06 and 2.99, with overlapping confidence intervals.
- The policy benchmark does not fully isolate why some maps are harder. Across 15 generated maps, InterFuser’s driving score decreases from 0.410 on easy layouts to 0.123 on elevated layouts. This is a useful observation, but differences in traffic, route length, appearance, and geometry could also contribute. More controlled comparisons would strengthen the interpretation that elevation is the main source of failure.
- The downstream task is not fully evaluated, in particular, the driving policy training results are evaluated only through open loop prediction. Reducing waypoint error from 7.79 to 3.10 m is encouraging, but it does not establish that the trained policy completes more routes or avoids more collisions. The separate evaluation of 7 released policies demonstrates the difficulty of the generated maps rather than the driving benefits of training on them.
- Minor concern: the photorealistic video examples do not yet demonstrate transfer to real driving data. No quantitative real data evaluation accompanies the translated videos, and Figure 17 shows clear artifacts like pedestrian's appearance is totally wrong. This raises a practical question about whether simulator annotations remain accurate after translation, particularly for dynamic objects.
Questions for authors
- Could the authors clarify how many of the 250 prompts satisfy all requested constraints and route checks? In particular, how are maps returned after budget exhaustion counted in the 98.4% generation success rate?
- Do the authors have existing closed loop results for policies trained with generated data? These would clarify whether the waypoint error reduction from 7.79 to 3.10 m translates into better route completion or fewer collisions.
- Could the authors provide trajectory quality results on the 12 scenarios that both methods successfully execute? This would help separate improvements in trajectory generation from the benefit of having a compatible map.
- Could the authors explain how route length, traffic, and other difficulty factors vary across the 15 evaluation maps? This would clarify whether the observed performance drops can be attributed specifically to road topology.
10 comments
Q2 and W4
Q2/W4 - Closed-loop policy evaluation results - We additionally evaluate the driving-policy checkpoints from Table 3 in closed-loop simulation on CARLA test towns. Adding generated data improves both route completion and collision rates. These results provide evidence that the open-loop waypoint improvements translate to downstream closed-loop driving benefits.
We emphasize that these are small, single-camera policies trained offline at the scale used in Table 3. Their absolute route completion remains low, and we do not claim that generated data alone produces highly competent driving policies. Rather, these results establish that generated data can improve downstream policy performance under our matched data experimental setup. We also report these closed loop metrics in Appendix H.4.2 of the revised paper.
| Training data | Route completion ↑ | Collisions / km ↓ |
|---|---|---|
| Hand-authored only | 0.013 ± 0.003 | 30.3 ± 3.0 |
| + generated (matched size) | 0.116 ± 0.020 | 22.9 ± 3.1 |
| + generated (all data) | 0.127 ± 0.026 | 19.4 ± 0.7 |
| Generated only | 0.072 | 23.5 |
Q3 and W2
Q3/W2 - Trajectory quality on the 12 scenarios both methods execute - We agree that the original comparison is influenced by map compatibility. To isolate trajectory quality, we additionally evaluate both methods on the 12 scenarios that are successfully executed by both. On this common subset, CARLAGen4D achieves higher motion fidelity and lower heading discontinuity but this is within overlapping confidence intervals. Therefore, even though ours is slightly better, we do not claim strictly better trajectory quality when both methods run. The advantage of our method is executability i.e. it runs all 39 prompts against TTSG's 12, because it builds a road network for each prompt instead of searching a stock town for a matching site. Please also find the analysis in Appendix G.3. of the revised paper.
| Metric | Ours | TTSG |
|---|---|---|
| Motion fidelity | 3.21 | 2.99 |
| Collision rate | 0.000 | 0.071 |
| Heading discontinuity (%) | 0.13 | 1.266 |
| Ego-actor collision | 0.000 | 0.077 |
W1 and Q1
Q1/W1 - "Valid map" vs. "satisfies the prompt"; how budget exhausted maps are counted - We agree that a loadable map does not necessarily satisfy all requested constraints. We therefore provide a breakdown of the 250 prompts using progressively stricter criteria. We now report the number of maps that satisfy different constraints and route checks. The reported 98.4% generation success rate measures whether the generated artifact can be loaded by CARLA, not whether the agent fully satisfies the prompt. In particular, 12 maps were returned after budget exhaustion; 7 of these have prompt-adherence scores <= 3, highlighting that artifact validity alone does not guarantee prompt satisfaction. Of the remaining 234 maps for which generation completed cleanly, 210 pass route checks and contain a single connected lane network. Further, 183 achieve both prompt-adherence and, for elevation-related prompts, elevation scores >= 4. We have added this breakdown in Appendix F.3 of the revised paper.