- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ColliGen adapts normal-driving trajectory predictors into collision trajectory generators through two-phase, physics-guided self-training. Each phase alternates between generating pseudo-trajectories and fine-tuning the predictor. Bézier-based perturbations and independent Metropolis–Hastings sampling guide candidate generation, while physical scores encourage collision feasibility and plausible post-impact motion. The method is evaluated using HiVT and QCNet on 4,583 accident scenarios reconstructed from NHTSA records with CARLA/Chrono. Reported displacement errors improve substantially over the compared baselines, particularly after impact.
Critical strengths
-
Meaningful problem and well-motivated adaptation framework. Complete collision trajectories are scarce, and adapting predictors trained on abundant normal-driving data is a useful direction. The Grow–Improve procedure and the separation of pre- and post-collision training provide a coherent approach to progressively learning these rare events.
-
Substantial physical scoring design with encouraging empirical results. The framework combines collision feasibility, post-impact momentum and velocity consistency, and motion regularity. The reported improvements are considerable, and the scoring ablations support the usefulness of combining impact physics with motion regularity.
Critical weaknesses
-
The central physical-plausibility claim lacks direct quantitative validation. Evaluation primarily measures displacement and relative-distance errors, which do not establish collision success, absence of interpenetration, or physically consistent impact responses.
-
The practical utility of the generated data remains unclear. Autonomous-driving training and evaluation motivate the work, but no downstream experiment establishes whether the generated trajectories improve a predictor or driving policy, or provide effective safety tests. Reconstruction accuracy alone does not substantiate these intended uses.
Questions for authors
-
Could the authors directly evaluate the physical validity of the generated trajectories, including collision success rates, interpenetration, and post-impact momentum or impulse consistency?
-
Could the authors demonstrate the utility of the generated data through a focused downstream training or safety-testing experiment, analogous to the policy-training evaluation in TrafficGen [1]? At minimum, a functional demonstration of generated scenarios being used by a driving system would help establish executability and initial utility beyond trajectory visualization.
-
Since CARLA/Chrono already generate the post-impact reference trajectories, what advantages does the learned post-collision generator offer over generating pre-collision trajectories and simulating the subsequent motion?
[1] Feng et al. Trafficgen: Learning to generate diverse and realistic traffic scenarios. ICRA, 2023.
5 comments
Rebuttal by Authors (part 2/3)
W2: The practical utility of the generated data remains unclear. Autonomous-driving training and evaluation motivate the work, but no downstream experiment establishes whether the generated trajectories improve a predictor or driving policy, or provide effective safety tests. Reconstruction accuracy alone does not substantiate these intended uses.
Q2: Could the authors demonstrate the utility of the generated data through a focused downstream training or safety-testing experiment, analogous to the policy-training evaluation in TrafficGen [1]? At minimum, a functional demonstration of generated scenarios being used by a driving system would help establish executability and initial utility beyond trajectory visualization.
Downstream training: collision prediction. We thank the reviewer for the insightful suggestion. To investigate the utility of collision data generated by COLLIGEN for downstream training, we consider the target task of collision prediction as follows.
-
A trajectory model is input with trajectories of two nearby vehicles (not yet collided).
-
The model simulates future trajectories of the input: it predicts a collision if the simulation outcome incurs a collision; otherwise, it predicts the input as normal.
-
Two models are compared: one trained with both normal and collision data generated by COLLIGEN; the other with normal trajectories only.
The key ability to test is whether a model can predict future trajectories that lead to collisions. Possible applications include downsizing trajectory generation models (e.g., quantization) or performing knowledge distillation (KD) to student models. Such small models can be embedded in vehicles to support real-time prediction of short-period future trajectories.
Setup. We conducted an experiment of fine-tuning TrafficGen [1] with collision data, where TrafficGen was initially pretrained on normal driving data. Specifically, we partitioned our generated collision data into training, validation, and test sets at the scenario level, using a split ratio of 0.75:0.20:0.05, respectively. We fine-tuned TrafficGen on the training set, selected a checkpoint using the validation set, and evaluated the pretrained and fine-tuned models on the same test set under identical input conditions.
Next, we compared the original and fine-tuned TrafficGen models. Specifically, we conducted inference on 200 scenarios, comprising 100 normal driving scenarios from Argoverse and 100 collision scenarios from reconstructed ground-truth (GT) data. For the collision scenarios, we provided observations from to seconds before the collision as input and evaluated whether each model could predict the impending collision.
Confusion matrices for collision prediction before and after fine-tuning TrafficGen.
TrafficGen
| Actual \ Predicted | Collision | Normal |
|---|---|---|
| Collision | 61 | 39 |
| Normal | 0 | 100 |
Fine-tuned TrafficGen
| Actual \ Predicted | Collision | Normal |
|---|---|---|
| Collision | 95 | 5 |
| Normal | 2 | 98 |
Results: a significant drop in missed collision detection. Table (TrafficGen confusion matrices) shows the confusion matrix of the collision prediction. The original TrafficGen failed to predict the collision in 39 out of 100 collision scenarios, whereas the fine-tuned TrafficGen missed only 5 cases. These missed detection, or false-negative (FN) cases, are critical because the model failed to predict collision. The reduction in FN ratio is from 39% to 5% thanks to the fine-tuning with the collision data generated by Colligen.
The collision fine-tuning incurs a slight increase in false alarms, or false positive (FP) cases, where a model predicts a collision from otherwise normal trajectories. Specifically, the base model did not predict any collision or 0% FP rate. For the fine-tuned model, there is only a slight increase of the FP rate to 2%. However, not only the rate is quite small, but also the FP cases can be regarded as less critical errors than FN cases in some scenarios.
Conclusion. Our experiments show that fine-tuning TrafficGen with collision data substantially improves its ability to anticipate impending collisions while maintaining high prediction accuracy in normal driving scenarios. This further demonstrates the practical utility of collision data generated by COLLIGEN for downstream collision prediction tasks, beyond improvements in trajectory reconstruction accuracy.
REVISION ACTION. We will add the collision prediction experiments with TrafficGen to Section 3.2.
Rebuttal by Authors (part 3/3)
Q3: Since CARLA/Chrono already generate the post-impact reference trajectories, what advantages does the learned post-collision generator offer over generating pre-collision trajectories and simulating the subsequent motion?
Learning-based vs Rule-based Generators. Thank you for the insightful comment. The biggest difference is that CARLA/Chrono are heavily rule-based simulators; whereas we focus on learning-based generators. The status quo is that learning-based generators have just begun to catch up with the CARLA/Chrono output quality, as shown by our work.
Learning-based simulators are inherently lighter, faster, but tend to be inexact. However, we envision that learning-based generators will eventually dominate over complex rule-based counterparts. This has happened in graphics research: for example, simulating liquid, smoke, human clothing, and hair. Recent interesting examples are world models: Google has recently introduced the 4th generation of Genie [1], which renders world scenes based entirely on a data-driven approach without game or physics engines. While not without limitations, the approach is an important direction for physical AI research.
We observed a similar trend even in our work. Colligen combines the parametric knowledge of the base model with fine-tuning guided by simple physics; yet it is able to generate outputs comparable to CARLA/Chrono, which uses a significantly more complex physics engine. Thus, we believe our work can be a meaningful step towards this research direction.
[1] Bruce, Jake, et al. "Genie: Generative interactive environments." Forty-first international conference on machine learning. 2024.
Rebuttal by Authors (part 1/3)
We sincerely appreciate your insightful and constructive comments. Please let us know if we misunderstood any of your questions. We would be happy to provide corrected responses upon your clarification.
W1: The central physical-plausibility claim lacks direct quantitative validation. Evaluation primarily measures displacement and relative-distance errors, which do not establish collision success, absence of interpenetration, or physically consistent impact responses.
Q1: Could the authors directly evaluate the physical validity of the generated trajectories, including collision success rates, interpenetration, and post-impact momentum or impulse consistency?
Thank you for the suggestion. We report the validity of the following metrics.
Collision rate. This measures how frequently the generated outputs produce collisions. A collision is detected if the two vehicles make contact at least once during the prediction horizon. A vehicle is represented by an oriented rectangle of dimension 4.5m×1.8m.
Absence of interpenetration. For this evaluation, we place a 0.9m×0.9m box at the center of each vehicle, aligned with its direction of travel, and define interpenetration as contact or overlap between these two boxes. The interpenetration cases are counted only from the collision samples. That is, we consider the rate of interpenetration events conditional on the collision events. We report the rate of absence of interpenetration.
ΔV. This measures the prediction accuracy of the change in the velocity magnitude after collision.
Normal ΔV. This measures the momentum consistency as the change in the normal components of velocity after impact. This is associated with the impact response from restitution.
Tangential ΔV. This measures the momentum consistency as the change in the tangential components of velocity after impact, which is associated with the impact response from contact friction.
Yaw rate. This measures the rotational validity after impact. The yaw rate is one of our physical scoring metrics (defined Appendix A.5, Eqs. (25)–(27)).
Additional quantitative evaluation of collision occurrence, velocity-change accuracy, absence of interpenetration, physical consistency of impact responses, and rotational validity. Velocity errors and residuals are reported in m/s, and yaw-rate errors in rad/s.
Table shows the results of the additional evaluation. We observe that COLLIGEN outperforms the baselines across all metrics. One exception is the absence of penetration;in that case, KING performs the best. This is because of the property of the KING method. As shown in the qualitative examples in Fig. 4 and Appendix A.11, vehicles simply come to a halt after a collision in KING. Since the vehicles remain still, the likelihood of penetration also decreases, leading to a high rate of absence-of-interpenetration. However, such vehicle behavior under KING is not necessarily optimal, which can be observed from the performance in other metrics.
REVISION ACTION. We added the results to the main text as Table 2 in the revised paper.