acceptodds
Under review as a conference paper at ICLR 2027

Planner-Oriented Scenario Generation: Uncertainty-Guided Generation of Learnable Long-Tail Scenarios

Abstract

Handling long-tail scenarios is one of the main challenges in autonomous driving. Several recent works have demonstrated that generative models can be a promising way to address this challenge by synthesizing diverse long-tail scenarios. However, they require domain knowledge of what makes a scene critical, such as specific traffic or safety conditions. This domain knowledge is often not available in practice and does not reflect which scenarios the planner has not learned well. Furthermore, these works also need additional components to keep their scenarios learnable. In this paper, we propose Planner-Oriented Scenario Generation (POSG), which generates uncertain and learnable long-tail scenarios based on what the planner has learned without any domain knowledge. We hypothesize that a scene is worth training on if it is uncertain and learnable for the planner, not merely if it is rare or adversarial. To implement our hypothesis, POSG combines the score estimate of the diffusion model with the gradient obtained from the planner's uncertainty. POSG also identifies learnable scenarios by leveraging a generative model that synthesizes the entire scene. The generative model enables us to determine whether the generated trajectories are learnable in the context of the entire scene. Experiments show that scenarios generated by POSG induce a higher planner error than unguided samples from the same generative model, while improving the planner when it is retrained on them.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, scenario generation, generative models, diffusion models, long-tail, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_UMvqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to expand the distribution of driving training data by using the uncertainty of the trajectories predicted by the planner as a guidance signal. The authors build a scenario generator on top of Scenario Dreamer, measure the uncertainty of the output of a frozen planner under dropout inference, and use it to steer the direction of the next denoising step of scene generation. A learnability measure is introduced to counterbalance the uncertainty guidance, so that the generation does not drift toward driving scenarios that are too hard to be solvable. The authors split the nuPlan dataset into out-of-distribution and in-distribution subsets, and on the in-distribution subset they augment the training data distribution with decision-uncertainty-guided synthetic data. Experimental results demonstrate the effectiveness of the proposed method, and the usefulness of the augmented training data is positively correlated both with the direction of the decision uncertainty and with the magnitude of the uncertainty.

Critical strengths

The idea of using the intrinsic uncertainty in the output of an off-the-shelf planner to guide scenario generation is novel, and the implementation is low-cost and scalable.

The experimental design is sound and sufficiently thorough. The results in Tables 1–4 clearly show the advantage of the training data distribution augmented by the proposed method over the unguided one, the gap between the augmented training data distribution and the upper bound of the data distribution (training data that includes the OOD set), the effect of the learnability term, and the advantage of high uncertainty over high certainty as the guidance direction.

The authors present the implementation details of the proposed method clearly and sufficiently.

Critical weaknesses

My concern is that the proposed decision uncertainty may guide data production along the scene distribution rather than along scene difficulty. Uncertainty may arise not only from the difficulty of the driving decision in a scene, but also from road structure alone: for instance, the space of possible driving decisions at an intersection is far larger than on a straight road, so a decision model's output uncertainty in a simple intersection scene may well exceed its output uncertainty in a difficult straight-road scene. Have the authors observed how the decision model's output uncertainty relates to factors such as road structure, ego state, and the states of surrounding agents, and what the road-type distribution of the synthetic data is?

Questions for authors
  1. Please check W1.
  2. Although the public release of UncArms postdates the submission of this paper and therefore does not affect its contribution, I believe that adding a discussion comparing it with UncArms (Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies) would be beneficial.
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer UMvq (1/2)

Thank you for your thoughtful comments and positive assessment. We are pleased that you found the use of the planner's intrinsic uncertainty novel, the implementation low-cost and scalable, the experimental design sound and thorough, and the implementation details clear. We would like to address your comments and questions below:


[W1 & Q1] Uncertainty, road structure and ego state: Thank you for this insightful question. We agree that the uncertainty of a planner's output can rise with road structure alone, since a scene such as an intersection admits more valid maneuvers than a straight road. The variance of the output rises both when a scene admits several valid futures and when the planner has not learned the scene, whereas an input that a network has not learned is characterized better in its representation space [1]. This is why U(x) measures the dispersion of the planner's hidden representations rather than that of its output trajectory (Section 4.1).

To examine whether U(x) still follows road structure or ego state, we compared the 12,500 scenes with the highest uncertainty among 50,000 unguided scenes with the whole pool and with recorded OOD scenes, as shown in the table below. The uncertain scenes have fewer lanes and agents than both the pool and the recorded OOD scenes, so uncertainty does not simply follow complex road structures such as intersections. On the 42,852 held-out recorded scenes, the rank correlations of U(x) with the number of lanes and the number of agents are also weak (−0.14 and −0.24). As for the ego state, the uncertain scenes concentrate at medium speeds of 2 to 10 m/s and rarely reach 10 m/s or more, a range that recorded OOD scenes also cover more often than the pool. Instead of road structure, U(x) follows the planner's error, with a rank correlation of 0.43 with the error of the frozen planner on the held-out recorded scenes and of 0.55 on their OOD group. This means that U(x) tracks what the planner has not learned rather than road structure or ego state alone. Guiding the generation by U(x) therefore leads POSG to the scenes that the planner still needs, rather than merely to complex road layouts.

\text{Scenes} & \text{All 50,000 unguided} & \text{Top 25\\% by U(x)} & \text{Recorded OOD (held-out)} \\\\ \hline \text{Lanes per scene} & 18.5 & 15.7 & 24.6 \\\\ \text{Agents per scene} & 15.1 & 12.1 & 18.5 \\\\ \text{Ego speed below 2 m/s (\\%)} & 40 & 24 & 41 \\\\ \text{Ego speed of 2 to 10 m/s (\\%)} & 38 & 74 & 52 \\\\ \text{Ego speed of 10 m/s or more (\\%)} & 22 & 2 & 7 \end{array}$$ Lanes and agents are means per scene, and ego speed is taken at the start of the scene. Recorded OOD is the OOD group of the 42,852 held-out recorded scenes (14,446 scenes). We added these analyses to Appendix D.3 of the revised paper. We believe this analysis addresses your concern about road structure and ego state. [1] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS, 2018.
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer UMvq (2/2)

[Q2] Comparison with UncArms: Thank you for pointing us to UncArms [2], which we read with great interest. UncArms trains multi-agent driving policies by closed-loop self-play, in which half of the agents are rewarded for raising the MC-dropout uncertainty of their neighbors. Its analysis shows that scenarios with higher uncertainty fail more often, which supports the same premise as POSG, namely that a policy's own uncertainty marks what it has not learned.

We think the two methods are complementary. UncArms exploits uncertainty in closed loop with reactive agents, but its scenarios start from recorded maps and initial states, so it explores new behaviors within recorded scenes. POSG instead generates new scenes, including new road layouts and agent configurations, so it is not limited to recorded scenes. Its generated futures, however, are fixed, so it trains the planner in open loop. A natural next step is to use POSG scenes as starting points for closed-loop training with reactive agents, as in UncArms, which would carry the benefit of POSG into closed loop.

We added the closed-loop extension of POSG to the conclusion (Section 6) as future work. We also cite UncArms as concurrent work in Appendix D.3, for its finding that uncertainty predicts closed-loop failures, and in Appendix D.4, for closed-loop training with reactive agents.


Thank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns.

[2] Sajad Marvi, Julian Schmidt, Alessandro Canevaro, Julian Jordan, and Abhinav Valada. Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies. arXiv:2610.19714, 2026.

Reviewer_9q3BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Planner-Oriented Scenario Generation (POSG), a diffusion-based framework for generating informative and learnable long-tail driving scenarios. Unlike existing methods that rely on manually specified safety-critical conditions, POSG uses the uncertainty of a target planner to identify scenarios that may be beneficial for its training. The method estimates planner uncertainty through the variation of hidden representations under Monte Carlo dropout. This uncertainty signal is incorporated into the diffusion sampling process through gradient-based guidance. To encourage scenario learnability, the framework jointly generates ego and surrounding-agent trajectories and introduces collision and off-lane costs to favor scenarios with feasible ego trajectories. The resulting scenarios are ranked by planner uncertainty and used to augment the planner's training data.

Experiments on the nuPlan dataset show that POSG generates scenarios with higher planner prediction errors than unguided sampling. Retraining with 50,000 generated scenarios reduces the out-of-distribution average displacement error by 18.5% compared with the original planner, outperforming the 14.1% reduction achieved by unguided augmentation. Additional experiments examine the effects of uncertainty guidance and learnability constraints.

Critical strengths

The main strength of this paper is its planner-oriented formulation of long-tail scenario generation. Rather than relying on predefined adversarial objectives or manually specified critical events, POSG identifies informative scenarios based on the planner's own uncertainty. This makes scenario generation more closely aligned with the planner's actual learning needs.

The use of hidden-representation disagreement under Monte Carlo dropout is interesting. It provides a differentiable uncertainty signal that can directly guide the diffusion sampling process. Moreover, combining this signal with trajectory feasibility costs offers a coherent way to balance scenario difficulty and learnability. This formulation is conceptually meaningful and provides a promising direction for more targeted training data augmentation.

Critical weaknesses

One important limitation is that the evaluation relies primarily on open-loop average displacement error, which does not directly demonstrate improvements in closed-loop driving performance or safety. Although the proposed learnability costs reduce collisions and lane departures compared with uncertainty-only guidance, the resulting scenarios are not necessarily safe or learnable in a broader planning sense.

Table 3 shows that 9.2% of POSG scenarios contain a vehicle collision along the generated ego trajectory, compared with 3.9% for unguided samples. Furthermore, Appendix B.3 shows that removing the learnability costs does not produce a distinguishable difference in the retrained planner's open-loop performance. These results suggest that the costs improve the feasibility of generated trajectories, but their contribution to downstream planning performance remains less clear. Closed-loop evaluations using driving-related safety metrics would provide stronger evidence that POSG produces scenarios that are not only challenging but also genuinely beneficial for learning robust driving behavior.

Questions for authors

The generative model is trained on recorded scenarios from all three groups, including those considered out-of-distribution for the planner. To what extent does POSG depend on this broader training distribution? Could the authors evaluate a setting in which some long-tail scenario types are also excluded from the generative model's training data? This would help clarify whether POSG can discover genuinely unseen challenging scenarios or mainly improve the selection of scenarios already represented in the generator's training distribution.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer 9q3B (1/2)

Thank you for your thoughtful review. We are pleased that you found the planner-oriented formulation closely aligned with the planner's actual learning needs, the MC-dropout disagreement an interesting differentiable signal, and its combination with feasibility costs a coherent balance of difficulty and learnability. We would like to address your comments and questions below:


[W1] Closed-loop driving and safety metrics: Thank you for raising this important point. We agree that the open-loop displacement error alone does not show closed-loop driving performance. Following your suggestion, we additionally evaluated the retrained planners in a short closed-loop replay with the off-lane and at-fault collision rates. In this replay, the planner re-plans every second over 4 s on the 29,417 held-out recorded scenes in which the ego moves, while the ego follows the first second of each plan and the other agents follow their recorded trajectories.

POSG and the planner trained with 12,500 recorded non-ID scenes from the training split reached similar rates in this replay, although that planner reaches by far the lowest open-loop OOD error (1.122 ± 0.011 against 1.290 ± 0.011). We therefore believe that the rates reflect the open-loop training data rather than what POSG generates. Both the generated and the recorded scenes have fixed futures, so they cannot teach the planner how other agents react to its own actions, and open-loop gains of such planners are known to carry over little to closed-loop driving [1]. Closed-loop safety in this replay is therefore not the right measure of the contribution of POSG. POSG nevertheless achieves what it is designed for in this open-loop setting. POSG lowers the OOD error from 1.334 to 1.290 ± 0.011, whereas unguided sampling does not (1.333 ± 0.018), and the learnability costs nearly halve the infeasible ego trajectories in the generated scenes themselves.

Because the limit of the closed-loop rates comes from the fixed futures rather than from POSG, extending POSG to closed-loop scenario generation with reactive agents, where the planner learns from its own interactions, is a promising direction. We added the closed-loop results to Appendix D.4, the two results above to Tables 2 and 3 in Section 5.2, and this direction to the conclusion (Section 6) of the revised paper.

[1] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with Misconceptions about Learning-based Vehicle Motion Planning. CoRL, 2023.

AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer 9q3B (2/2)

[W2] Effect of the learnability costs: Thank you for this careful observation. The learnability costs achieve their purpose in the generated scenes themselves, as the table below shows for the 12,500 scenes that each variant adds. Against uncertainty only, the costs halve the vehicle collisions along the generated ego trajectories, from 14.8% to 7.3%, as they did in Table 3 of the submitted version (from 20.5% to 9.2%). They also nearly halve the trajectories that leave the lanes, and the ego trajectories of POSG collide with any agent less often than unguided ones. This means that, as a result of the learnability costs, POSG generates more feasible training data, in which the ego trajectory that the planner learns to imitate avoids collisions and stays within the lanes.

\text{Share of the 12,500 added scenes (\\%)} & \text{Unguided} & \text{Uncertainty only} & \text{POSG} \\\\ \hline \text{Vehicle collision} & 3.9 & 14.8 & 7.3 \\\\ \text{Any-agent collision} & 37.0 & 38.2 & 23.7 \\\\ \text{Off-lane} & 5.1 & 11.9 & 6.5 \end{array}$$ Each row gives the share of the 12,500 added scenes in which the generated ego trajectory collides with a vehicle, collides with any agent, or passes more than 1.5 m from the nearest lane. We acknowledge that, as you pointed out, the generated ego trajectories of POSG still collide with a vehicle a few percentage points more often than unguided ones (7.3% against 3.9%). This is because uncertainty steers scenes toward interactions between vehicles, which are the situations the planner needs to learn, and the costs keep these collisions at about half of uncertainty only. We also acknowledge that the costs change the open-loop error of the retrained planner little. However, this comes from what the open-loop metric measures, namely how closely the planner follows recorded trajectories, not whether the trajectories it was trained on were feasible. Even so, POSG reaches a lower OOD error than uncertainty only (1.290 ± 0.011 against 1.319 ± 0.022). The learnability costs therefore keep the training data valid without any loss in accuracy. We updated Table 3 in Section 5.2 of the revised paper with this table and added the retraining result of uncertainty only to Appendix B.2. We hope this analysis addresses your concern about the effect of the learnability costs. ___ **[Q1] Long-tail scenario types in the training data of the generative model:** Thank you for this insightful question. As for any diffusion model, scenario types that are entirely absent from the training data are theoretically difficult to generate, with or without guidance. Nevertheless, guidance can still lead the generative model to scenes that unguided sampling rarely produces. Because it shifts every reverse step toward scenes on which the planner is uncertain, it can reach rare regions of the learned distribution and extrapolate slightly beyond them. Inspired by your question, we ran additional experiments with a generative model trained on few long-tail scenes. This setting better shows the goal of our contribution, namely to generate long-tail scenarios that the generative model rarely produces without guidance. The generative model of the submitted version was trained on all recorded scenes, 62.1% of which are not ID for the planner, so even its unguided samples were often OOD and already lowered the OOD error by 14.1% (Table 2 of the submitted version). We therefore retrained its diffusion model on all ID scenes together with randomly chosen non-ID scenes, which make up only 5% of its 182,394 training scenes instead of 62.1%. The evaluation on the 42,852 held-out recorded scenes is unchanged, we report the mean and standard deviation over three seeds, and Section 5 of the revised paper now uses this setting for all of its experiments. The setting above lets us test this, because every long-tail type is rare in the training data of its diffusion model, which moves it toward the setting you describe. With this generator, unguided samples no longer lower the OOD error (1.333 ± 0.018 against 1.334 without augmentation), and selecting them by uncertainty helps only a little (1.316 ± 0.007). POSG lowers the error beyond that selection (1.290 ± 0.011), so guidance steers the generator toward scenes that the planner still needs, beyond selecting the scenes that it already produces. ___ Thank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns.
Reviewer_8GpiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Planner-Oriented Scenario Generation (POSG), a planner-oriented long-tail scenario generation method. It combines the score estimate of the diffusion model with the gradient obtained from the planner’s uncertainty. The experimental results demonstrate that scenarios generated by POSG induce a higher planner error than unguided samples, enabling the planner to achieve better performance.

Critical strengths
  1. The application of diffusion models for generating learnable long-tail scenarios based on the context of the entire scene, avoiding relying on domain knowledge of what makes a scene critical.

  2. The gradient of the learnability costs on the ego trajectory is combined with the gradient of the planner’s uncertainty, allowing for the generation of learnable long-tail scenarios without additional components.

Critical weaknesses
  1. This research hypothesizes that a scene is worth training on if it is uncertain and learnable for the planner, and the proposed model is developed based on it. However, the hypothesis lacks sufficient rigor, and the authors should formulate a more scientifically justified one.

  2. The main contribution of this work appears to be the integration of diffusion models with learnable long-tail scenario generation. However, the contribution offers limited novelty compared to existing methods, and the experimental results demonstrate incremental improvements in some metrics.

  3. While the baselines selected are relevant, the comparison could be strengthened by including methods that use diffusion models for scenario generation, such as GALTraj.

Questions for authors
  1. Could the authors discuss more about the first assumption upon which the POSG operation (lines 209-212)? It is not clear that which models within the pipeline work when no user-specified condition is available.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer 8Gpi (1/3)

Thank you for your constructive comments. We are pleased that you found POSG to generate learnable long-tail scenarios from the context of the entire scene without domain knowledge, and to combine the gradients of the learnability costs and the planner's uncertainty without additional components. We would like to address your comments and questions below:

Before we discuss your concerns and questions, we would like to note that the data used to train the generative model was modified to clarify our contribution: POSG can generate long-tail scenarios which are useful for improving the planner without domain knowledge. Specifically, the generative model of the submitted version was trained on all recorded scenes, 62.1% of which are not ID for the planner, so it readily generated OOD scenes even without guidance. However, this made it unclear how much of the gain came from guidance, because its unguided samples alone already lowered the OOD error by 14.1% (Table 2 of the submitted version). To address this, in this response and in the revised paper, we retrained its diffusion model on all ID scenes together with randomly chosen non-ID scenes, which make up only 5% of its 182,394 training scenes. Section 5 of the revised paper reports all of its experiments in this setting, evaluated on the same 42,852 held-out recorded scenes. In this setting, unguided samples alone no longer lower the OOD error, so the gain of POSG does not come from adding generated scenes alone (see our response to W2). This change does not undermine our contribution, because POSG still steers the generation by the planner's own uncertainty and the learnability costs, without domain knowledge. Rather, since the retrained generative model rarely produces OOD scenes without guidance, the change is consistent with both our goal of generating such long-tail scenarios and our experiments comparing POSG against unguided samples.


[W1] Justification of the hypothesis behind POSG: Thank you for this suggestion. We hypothesize that a scene is worth training on when the planner is uncertain about it and can still learn from it, rather than simply when it is rare or adversarial. This hypothesis is based on two observations. First, a scene on which the planner is uncertain is likely to be one that it has not learned well, so the planner may perform poorly in it. Indeed, the planner’s uncertainty is low on scenes that resemble its training distribution and high on scenes that fall outside it. Second, a scene that the planner cannot learn from does not improve it, however uncertain the planner is about it. The planner imitates the generated ego trajectory, so a scene without a safe ego trajectory cannot teach the planner how to act. This is consistent with previous works, which avoid using such unlearnable scenes for training [1, 2].

To verify the uncertainty part, we reversed the sign of the uncertainty term, so that POSG guides the generation toward the scenes on which the planner is certain. The table below shows the OOD and ID errors of the planners trained with each set of generated scenes. Guiding to certain scenes raises the OOD error even above no augmentation, whereas guiding to uncertain scenes lowers it. This shows that the direction of the uncertainty term carries the gain, as the uncertainty part of the hypothesis predicts.

\text{Training data} & \text{OOD ADE}\downarrow & \text{ID ADE}\downarrow \\\\ \hline \text{No augmentation (ID-only, single run)} & 1.334 & 0.587 \\\\ \text{+ Certainty and learnability costs} & 1.396 \pm 0.030 & 0.583 \pm 0.004 \\\\ \text{+ Certainty only} & 1.373 \pm 0.011 & 0.586 \pm 0.004 \\\\ \textbf{+ POSG} & \mathbf{1.290 \pm 0.011} & \mathbf{0.582 \pm 0.004} \end{array}$$ ADE (m) on the 42,852 held-out recorded scenes, mean ± s.d. over three seeds. To verify the learnability part, we removed the learnability costs from POSG, which leaves uncertainty only. The learnability costs halve the generated ego trajectories that collide with a vehicle (from 14.8% to 7.3%) and nearly halve those that leave the lanes (from 11.9% to 6.5%), and POSG reaches a lower OOD error than uncertainty only (1.290 ± 0.011 against 1.319 ± 0.022). This shows that the learnability costs keep the generated scenes learnable without giving up the gain of the uncertainty term, as the learnability part of the hypothesis predicts. We added these results to Tables 3 and 4 in Section 5.2 and to Appendix B.2, and this justification of the hypothesis to Sections 4.1 and 4.2 of the revised paper. We believe these results support the hypothesis behind POSG. [1] Rempe et al. Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior. CVPR, 2022. [2] Hanselmann et al. KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients. ECCV, 2022.
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer 8Gpi (2/3)

[W2] Novelty of POSG and the effect of guidance: Thank you for this comment. The contribution of POSG lies in where the guidance objective comes from, not in guided generation itself. Existing guided scenario generators encode domain knowledge of a predefined critical situation, such as a rule [3], a collision objective [4], or a language description [5]. POSG does not encode which situation is critical, because its objective comes from the planner's own uncertainty, and its learnability costs only check that the generated ego trajectory avoids the surrounding agents and stays within the lanes. It therefore targets what the given planner has not learned, rather than what a designer expects to be critical.

We would like to note that the performance gain of POSG is not incremental. The table below shows its performance gain in terms of OOD and ID errors of the planners. Unguided samples do not lower the OOD error, whereas POSG lowers it from 1.334 to 1.290 without degrading the ID error. This means that the gain comes from POSG, not from adding generated scenes alone. This is because guidance steers the samples toward scenes that the planner has not learned, which unguided samples rarely contain in this setting.

\text{Training data} & \text{OOD ADE}\downarrow & \text{ID ADE}\downarrow \\\\ \hline \text{No augmentation (ID-only, single run)} & 1.334 & 0.587 \\\\ \text{+ Unguided} & 1.333 \pm 0.018 & 0.582 \pm 0.002 \\\\ \textbf{+ POSG} & \mathbf{1.290 \pm 0.011} & \mathbf{0.582 \pm 0.004} \end{array}$$ ADE (m) on the 42,852 held-out recorded scenes, mean ± s.d. over three seeds. Each variant adds 12,500 generated scenes to the ID-only training data. We added these results to Table 2 and Figure 3 in Section 5.2 of the revised paper. We hope this clarifies our contribution. [3] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided Conditional Diffusion for Controllable Traffic Simulation. ICRA, 2023. [4] Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries. ECCV, 2024. [5] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-Guided Traffic Simulation via Scene-Level Diffusion. CoRL, 2023.
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer 8Gpi (3/3)

[W3] Comparison with a diffusion-based scenario generator such as GALTraj: Thank you for this suggestion. Following your suggestion, we looked for a diffusion-based baseline. We could not find a public implementation of GALTraj [6], so we compared POSG with SAFE-SIM [4], another recent diffusion-based scenario generator. SAFE-SIM guides the diffusion model with an objective that steers a chosen adversary toward a collision with the ego vehicle while keeping the other vehicles on their routes and apart from each other. For a fair comparison, we only replaced the guidance objective of POSG with this objective of SAFE-SIM.

The table below shows the OOD and ID errors of the planners trained with each set of generated scenes. POSG reaches a lower OOD error than SAFE-SIM without domain knowledge of what makes a scene critical. This is because SAFE-SIM steers every scene toward one predefined kind of critical event, a collision between an adversary and the ego vehicle, whereas POSG, without such domain knowledge, steers the scenes toward whatever the given planner is uncertain about and therefore generates the data that this planner still needs. This result shows that the planner's own uncertainty is a more useful target for training data than a hand-designed critical event.

\text{Training data} & \text{OOD ADE}\downarrow & \text{ID ADE}\downarrow \\\\ \hline \text{No augmentation (ID-only, single run)} & 1.334 & 0.587 \\\\ \text{+ SAFE-SIM} & 1.329 \pm 0.006 & 0.587 \pm 0.003 \\\\ \textbf{+ POSG} & \mathbf{1.290 \pm 0.011} & \mathbf{0.582 \pm 0.004} \end{array}$$ ADE (m) on the 42,852 held-out recorded scenes, mean ± s.d. over three seeds. SAFE-SIM replaces only the guidance objective of POSG. We added this comparison and the implementation details of SAFE-SIM to Appendix D.1 of the revised paper and summarized the comparison in Section 5.2. We believe this comparison addresses your request for a diffusion-based baseline. ___ **[Q1] Which models of the generation process contribute to the guidance when no user-specified condition is available (lines 209–212):** We apologize for the confusing explanation in these lines. The models that these lines refer to are the two frozen models that the generation process already contains, the planner and the generative model. Without a user-specified condition, the planner's own uncertainty decides which scenes to generate. POSG steers the generation toward scenes on which the given planner is uncertain, so the generated scenes differ for each planner. The generative model generates each scene with its ego trajectory, and the learnability costs on this trajectory also guide the generation, but only to keep the scene learnable, not to choose which scenes to generate. We revised these lines in Section 4 of the revised paper to state directly that POSG does not require a user-specified condition and to name both models. ___ Thank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns. [4] Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries. ECCV, 2024. [6] Daehee Park, Monu Surana, Pranav Desai, Ashish Mehta, Reuben MV John, and Kuk-Jin Yoon. Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model. ICCV, 2025.
Reviewer_dcBfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the task of synthetic data generation for training autonomous driving policies focused on long-tail scenarios which are critical to ensure a safe driving policy but are usually rare and expensive to gather in bulk. The authors propose Planner-Oriented Scenario Generation (POSG), an inference-time method that augments the generative process of a diffusion model trained on scenarios that shifts it towards samples that cause high uncertainty in the policy and are clean samples that avoid collisions with the ego car staying within the lanes. The proposed inference-time change is by using classifier-guidance, where the classifier is a differentiable energy composed of a policy uncertainty term, which measures the policies representation agreements under dropout, and two cost terms that keep the generated ego trajectory within the driving lanes and avoiding collisions. The authors experimentally evaluate their approach by demonstrating that policies trained on POSG-generated data has better performance on OOD samples than the other considered alternatives.

Critical strengths

The main idea of using generative models, such as diffusion models, to generate data that is subsequently used for training policies is an exciting one with a lot of potential. This is especially true in settings such as autonomous driving, where the vast majority of naturally collected data is relatively uninformative, and only a small portion captures scenarios that are particularly valuable for training safe policies, such as near-collisions. Naturally, collecting such data is prohibitively difficult and potentially dangerous, which is where synthetic data generation can be particularly useful. This paper aims to address this challenge by proposing a simple and straightforward training-free method for generating such data. The authors proposal is to generate scenarios where the policy is uncertain in which a reasonable safe outcome exists.

Critical weaknesses

While the proposed method is simple, training-free, and does not require additional networks or specific domain knowledge, the experimental evaluation is a bit lacking. Here are a few examples:

  • Table 1, on which the paper's main argument is based, compares policies trained on in-distribution (ID) data only, ID + 50K unguided synthetic samples, ID + 50K POSG-guided synthetic samples, and all available data. However, the results show that the difference between using unguided and POSG-guided synthetic data is quite small (0.058) and relatively close to the separability threshold of 0.042. Additionally, according to Table 7, this difference disappears entirely when doubling the unguided data to 100K samples. More importantly, according to Appendix A.5, POSG generates roughly 200K samples and selects the 50K with the highest uncertainty, whereas the unguided baseline generates 100K samples and randomly selects 50K. This makes the comparison somewhat unfair and makes it difficult to determine where the improvements are actually coming from, whether from POSG guidance, the uncertainty-based selection, or the uncertainty-based selection on the larger sampling pool. How would training on 50K unguided samples with the highest uncertainty selected from a pool of 200K compare?
  • Table 8 is also a bit confusing. It seems to show that training policies on POSG-generated samples versus uncertainty-only samples leads to almost identical outcomes. As far as I understand, this suggests that ensuring synthetic samples have a feasible solution (what the paper refers to as learnable) has little to no observable impact on the final policy's performance. This raises questions about the importance of this component of the proposed method.
  • The paper only compares against its own ablations. It would have been helpful to include several prior baselines, potentially ones that use domain knowledge or additional value networks, to get a better sense of how this general approach compares against more specialized methods.
  • A timing analysis of the additional overhead introduced by POSG is missing. The differentiable uncertainty estimator seems quite computationally expensive, and using it for classifier guidance is likely to introduce great overhead. Therefore, it would be useful to understand how much compute is required for synthetic data generation and subsequent policy training, and how this compares to the unguided baseline.
Questions for authors
  • Have the authors tried comparing POSG-generated synthetic data against an equal amount of real OOD data from the full training split?
  • The main idea of the paper is to generate synthetic data on which the policy is uncertain. Training on such scenarios, where a feasible and collision-free ego trajectory exists, is reasonable and intuitively makes sense; it helps the policy learn the correct behavior in these uncertain cases. However, this does not seem to cover settings where the policy confidently generates an incorrect or unsafe trajectory. Do the authors have any thoughts on whether POSG could also be useful in such settings?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer dcBf (1/3)

Thank you for your insightful and constructive feedback. We are pleased that you found the idea of generating training data for long-tail scenarios exciting and full of potential, and POSG simple, training-free, and free of additional networks and domain knowledge. Additionally, your comments on the comparison with unguided sampling led us to run additional experiments in a setting where the effect of guidance is easy to see.

Additional experiments to show the effect of guidance more clearly. The paper's generative model was trained on all recorded scenes, 62.1% of which are not ID for the planner, so even its unguided samples were often OOD and already lowered the OOD error by 14.1% (Table 2 of the submitted version). This hides how much of the gain comes from guidance.

We therefore retrained its diffusion model on all ID scenes together with randomly chosen non-ID scenes, which make up only 5% of its 182,394 training scenes, so that long-tail scenes are rare in its training data. The evaluation on the 42,852 held-out recorded scenes is unchanged, and we report the mean and standard deviation over three seeds.

In the revised paper, Section 5 now reports all of its experiments in this setting, and the new Figure 3 compares the OOD and ID errors of POSG with those of unguided sampling, with and without the same selection. As this figure and the responses below show, POSG reaches the lowest OOD error of all variants that add generated scenes, without degrading the ID error.

We would like to address your other comments and questions below:


[W1] Effect of guidance beyond the uncertainty-based selection: Thank you for this important point. Following your suggestion, we applied the same uncertainty-based selection to unguided sampling from a pool of the same size.

The table below shows the OOD and ID errors of the planners trained with POSG and with this unguided variant. POSG lowers the OOD error further than unguided sampling with the same selection, without degrading the ID error. This shows that guidance brings a gain beyond the selection and the pool size.

\text{Training data} & \text{OOD ADE}\downarrow & \text{ID ADE}\downarrow \\\\ \hline \text{No augmentation (ID-only, single run)} & 1.334 & 0.587 \\\\ \text{+ Unguided, same selection} & 1.316 \pm 0.007 & 0.587 \pm 0.002 \\\\ \textbf{+ POSG} & \mathbf{1.290 \pm 0.011} & \mathbf{0.582 \pm 0.004} \end{array}$$ ADE (m) on the 42,852 held-out recorded scenes, mean ± s.d. over three seeds. Both variants keep the 12,500 scenes with the highest uncertainty out of 50,000. We added these results to Table 2 and Figure 3 in Section 5.2 of the revised paper. We hope these results address your concern about the effect of guidance beyond the selection. ___ **[W2] Importance of the learnability costs:** Thank you for this insightful comment. The learnability costs are designed to keep the generated training data valid. Uncertainty alone also cannot prevent unlearnable scenarios from being generated, such as scenes in which no ego trajectory avoids a collision, as Section 4.2 explains. To examine this, we measured how often the generated ego trajectories collide or leave the lanes, as shown in the table below. These results show that the costs halve the generated ego trajectories that collide with a vehicle and nearly halve those that leave the lanes. This means that, as a result of the learnability costs, POSG generates more feasible training data, in which the ego trajectory that the planner learns to imitate avoids collisions and stays within the lanes. $$\begin{array}{lccc} \text{Share of the 12,500 added scenes (\\%)} & \text{Unguided} & \text{Uncertainty only} & \text{POSG} \\\\ \hline \text{Vehicle collision} & 3.9 & 14.8 & 7.3 \\\\ \text{Any-agent collision} & 37.0 & 38.2 & 23.7 \\\\ \text{Off-lane} & 5.1 & 11.9 & 6.5 \end{array}$$ Each row gives the share of the 12,500 added scenes in which the generated ego trajectory collides with a vehicle, collides with any agent, or passes more than 1.5 m from the nearest lane. We acknowledge that, as you pointed out, POSG and uncertainty only reach almost the same ADE in Table 8 of the submitted version. However, this small difference comes from what ADE measures rather than from the costs being unimportant. ADE measures how closely the planner follows recorded trajectories, not whether the trajectories it was trained on were feasible. Even so, POSG reaches a lower OOD error than uncertainty only (1.290 ± 0.011 against 1.319 ± 0.022). The learnability costs therefore keep the training data valid without any loss in accuracy. We updated Table 3 in Section 5.2 of the revised paper with this table and added the retraining result of uncertainty only to Appendix B.2. We believe this analysis addresses your concern about the importance of the learnability costs. ---
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer dcBf (2/3)

[W3] Comparison with prior methods that use domain knowledge or additional networks: Thank you for this suggestion. To address your concern, we compared POSG with SAFE-SIM [1], a recent diffusion-based method that generates safety-critical scenarios with domain knowledge. SAFE-SIM guides the diffusion model with an objective that steers a chosen adversary toward a collision with the ego vehicle while keeping the other vehicles on their routes and apart from each other. For a fair comparison, we only replaced the guidance objective of POSG with this objective of SAFE-SIM.

The table below shows the OOD and ID errors of the planners trained with each set of generated scenes. POSG reaches a lower OOD error than SAFE-SIM without domain knowledge of what makes a scene critical. This is because SAFE-SIM steers every scene toward one predefined kind of critical event, a collision between an adversary and the ego vehicle, whereas POSG, without such domain knowledge, steers the scenes toward whatever the given planner is uncertain about and therefore generates the data that this planner still needs. This result shows that the planner's own uncertainty is a more useful target for training data than a hand-designed critical event.

\text{Training data} & \text{OOD ADE}\downarrow & \text{ID ADE}\downarrow \\\\ \hline \text{No augmentation (ID-only, single run)} & 1.334 & 0.587 \\\\ \text{+ SAFE-SIM} & 1.329 \pm 0.006 & 0.587 \pm 0.003 \\\\ \textbf{+ POSG} & \mathbf{1.290 \pm 0.011} & \mathbf{0.582 \pm 0.004} \end{array}$$ ADE (m) on the 42,852 held-out recorded scenes, mean ± s.d. over three seeds. We added this comparison and the implementation details of SAFE-SIM to Appendix D.1 of the revised paper and summarized the comparison in Section 5.2. We hope this comparison addresses your concern about prior methods. ___ **[W4] Computational overhead of POSG for data generation and planner training:** Thank you for pointing out what we missed in our paper. We acknowledge that the paper does not report the computational overhead of POSG. A guided scene takes about 1.1 s against 0.27 s for an unguided one (batch of 32 on one GPU), because every reverse step runs the frozen planner K = 10 times with dropout. Sampling a pool of 50,000 scenes therefore takes about 15 GPU-hours with POSG and about 4 GPU-hours without guidance. Generation is offline and done once per planner. Planner training takes the same time for every variant, about 45 minutes on one GPU, since each adds 12,500 scenes and trains for the same number of steps. We added this timing analysis to Appendix D.2 of the revised paper. We hope this clarifies the overhead of POSG. [1] Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries. ECCV, 2024. ---
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer dcBf (3/3)

[Q1] Comparison with an equal amount of recorded OOD data from the training split: Thank you for this question. Following your suggestion, we conducted additional experiments that add, instead of generated scenes, the same number of recorded scenes, namely 12,500 non-ID scenes drawn at random from the nuPlan training split.

This planner reaches an OOD error of 1.122 ± 0.011. Because it adds recorded non-ID scenes to the planner's training data, which POSG does not, it serves as an upper bound on what the generated scenes can recover. POSG improves the planner from 1.334 to 1.290, the closest to this upper bound among all variants that add generated scenes, and closes 21% of the gap, against 1% for unguided sampling and 8% for unguided sampling with the same selection. This means that POSG recovers the largest share of the gain that recorded non-ID scenes bring, although it adds none of them to the planner's training data.

The revised paper reports the result of this planner as the row marked † in Table 2.


[Q2] Scenes on which the planner is confidently wrong: This is an insightful question. POSG was designed to target the scenes on which the planner is uncertain, so we did not consider confident errors in this work. The same framework can be extended to them, because every generated scene carries an ego trajectory and the planner's error against it, which Table 1 measures, can be added to the objective as a further term that rewards errors regardless of uncertainty. Together with the learnability costs, which keep that trajectory feasible, this term would also steer the generation toward scenes in which the planner is confidently wrong although a safe trajectory exists.


Thank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their careful evaluations and constructive feedback. Based on your suggestions, we have clarified the hypothesis and the method of POSG, and strengthened its empirical support through additional experiments, baselines and analyses. We summarize the main revisions below.

Paper Revision Summary

All revisions are highlighted in blue in the revised paper.

  • We extended our experiments to show POSG generates long-tail scenarios that a generative model rarely produces without guidance. For this, Section 5 uses a diffusion model trained mainly on ID scenes, so that long-tail scenes are rare among its unguided samples, with details in Section 5.1 and Appendix A.1. (dcBf, 8Gpi, 9q3B)
  • We extended Table 2 and Figure 3 with two variants: unguided sampling with the same uncertainty-based selection and an upper bound that adds the same number of recorded non-ID scenes. (dcBf)
  • We added SAFE-SIM, a recent diffusion-based method for safety-critical scenario generation, as a baseline in Section 5.2 to compare POSG with a method that relies on domain knowledge, with implementation details in Appendix D.1. (dcBf, 8Gpi)
  • We revised the justification of our hypothesis in Sections 4.1 and 4.2 to explain why a scene that the planner is uncertain about but can learn from is worth training on. (8Gpi)
  • We added a timing analysis to Appendix D.2 to compare the overhead of POSG with that of unguided sampling in data generation and planner training. (dcBf)
  • We added an analysis to Appendix D.3 to examine how U(x) relates to road structure, ego state and the planner's error. (UMvq)
  • We added a short closed-loop replay to Appendix D.4 to evaluate the retrained planners in closed loop, and the closed-loop extension of POSG as future work to the conclusion (Section 6). (9q3B, UMvq)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.