- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces "SceneDemand," a framework for the recursive generation of vectorized scenes for autonomous driving simulation. The method primarily addresses two challenges: determining when an existing scene requires expansion, and deciding where to generate new road structures and traffic NPCs. Specifically, this paper proposes a "temporal coverage" metric to assess whether the current map can support the ego-vehicle’s planned trajectory over the entire prediction horizon. When coverage is insufficient, the system triggers scene generation to complete missing road structures and initialize the states of new NPCs. Furthermore, this paper introduces "Counterfactual Direct Planner Guidance" (CDPG), which leverages a planner’s value assessment during the diffusion sampling process to guide the generation of new NPCs. By comparing the planner’s value, the method estimates the impact of these entities on the ego-vehicle’s planning outcomes. This process is further refined by incorporating multiple probing perspectives and scene feasibility constraints.
Critical strengths
- The framework is capable of simultaneously generating missing road structures and new NPCs, while preserving existing elements during subsequent denoising and scene expansion steps. Newly generated NPCs are integrated into downstream behavioral simulations, enabling closed-loop interaction.
- The experimental evaluation goes beyond merely assessing the generated scenes themselves; it examines recursive expansion capabilities, traffic distribution, and closed-loop planning performance. Additionally, the study tests the utility of the generated scenes for training downstream planners.
Critical weaknesses
- The novelty remains unclear. Existing literature has already explored recursive scene generation, continuous expansion, and the use of generated scenes to train downstream planners. The primary distinction in this paper is the use of the ego-vehicle’s future trajectory to identify map gaps and the "temporal coverage" metric to trigger scene expansion. While this design has merit, the paper fails to clearly articulate the substantive improvements it offers over simply incorporating trajectory-based conditioning into existing recursive generation models.
- The specific novel contributions of CDPG require further clarification. The use of gradients from external objectives to guide diffusion sampling has already been explored in works such as Universal Guidance and Diffuser; similarly, SAFE-SIM and DiffScene have investigated the generation of challenging scenarios for autonomous driving planners using diffusion models. Therefore, the primary distinction of CDPG should lie in its specific guidance mechanism, rather than merely the use of a critic to steer a diffusion model toward generating difficult scenarios.
- The role of the counterfactual value difference remains insufficiently clear. CDPG estimates the impact of adding a new NPC by comparing the planner’s value with and without that NPC present. However, based on the gradient formula provided in the paper, when only a single observation perspective is considered and no new NPCs are present, the counterfactual value difference does not alter the gradient direction of the difficulty objective. In this case, it effectively reduces to standard critic-based gradient guidance. Its primary function appears to be assigning varying weights across multiple perspectives.
- The absence of key ablation studies. CDPG incorporates counterfactual value differences, multi-perspective aggregation, and feasibility constraints; however, the paper does not evaluate the individual contribution of each component. For instance, could a similar outcome be achieved by simply averaging the critic gradients across multiple perspectives? How do the generated scenarios and downstream training performance change if the counterfactual comparison or the feasibility constraints are removed? The current experiments demonstrate that the full CDPG pipeline is effective, but they do not establish that each of these specific design elements provides an incremental benefit.
Questions for authors
- Compared to existing diffusion-based adversarial scenario generation methods like SAFE-SIM and DiffScene, what are the specific advantages of CDPG? Could the authors provide component-level ablation studies and compare against a simpler multi-perspective critic-based gradient guidance approach?
- The paper reports that CDPG increases the internal difficulty metric from 0.0141 to 0.0487, while simultaneously reducing the success rate of a fixed planner from 60% to 52%. However, the guidance strength experiments in the appendix show that the success rate is 18.18% at a strength of 0.5, whereas increasing the strength to 2.0 actually raises the success rate to 55.56%. This suggests that increasing guidance strength does not necessarily result in more challenging scenarios. Could the authors explain this phenomenon and further analyze the relationship between internal difficulty metrics, actual closed-loop performance, and downstream training gains?
4 comments
Comment 2 — Counterfactual weighting and component ablations
W3. Does the counterfactual baseline change the gradient?
Your observation is correct and helps clarify the mechanism. At one fixed probe, removing the new NPCs makes the baseline independent of the new-NPC latent , so
=-\nabla_z V_0(o_k^+).$$ **The paired baseline acts through the relative weights across probes.** For the multi-probe objective, $$\nabla_z E_{\mathrm{hard}} =-\sum_k w_k\nabla_z V_0(o_k^+), \qquad w_k=\operatorname{softmax}_k\left(\frac{V_0(o_k^-)-V_0(o_k^+)}{\tau_a}\right).$$ This gives three directly comparable guidance rules: | Variant | Probe weights | |---|---| | Uniform | $w_k=1/K$ | | Unpaired | $w_k\propto\exp[-V_0(o_k^+)/\tau_a]$ | | Full | $w_k\propto\exp[(V_0(o_k^-)-V_0(o_k^+))/\tau_a]$ | Full emphasizes where the newly introduced NPCs cause the greatest value drop relative to that probe's own baseline. For a single probe, or a common baseline across probes, the baseline cancels as you describe. With different baselines, it can change the combined direction by changing the relative probe weights. We will expand Appendix A.3.3 with this interpretation of Eq. (11) and the explicit comparison of the three weighting rules. ### W4 / Q1. Comparison with uniform multi-probe guidance and removal of components Following your helpful suggestion, we added the Uniform comparison and the paired and legality ablations. The multiple-probe variants use the same probes, and the guided component comparisons hold the denoising schedule and candidate-selection procedure fixed. | Variant | Empirical return ↓ | Frozen-P0 success (%) ↓ | Progress (m) ↓ | $E_{\mathrm{hard}}$ ↑ | |---|---:|---:|---:|---:| | Off | 0.243 | 60.00 | 35.50 | 0.0250 | | Single | 0.218 | 57.78 | 34.70 | 0.0340 | | Uniform | 0.196 | 55.56 | 34.00 | 0.0410 | | Unpaired | 0.193 | 55.56 | 33.80 | 0.0420 | | Full | 0.171 | 53.33 | 32.80 | 0.0490 | | No-soft-legality | 0.134 | 50.00 | 32.40 | 0.0570 | Return is the realized discounted reward of the frozen planner on completed generated scenes. Relative to Uniform, **Full reduces return from 0.196 to 0.171 and success from 55.56% to 53.33%**. Relative to Unpaired, return decreases from 0.193 to 0.171. These are the direct comparisons with simpler multi-perspective guidance requested in your review. We also compared the training value of Full, Unpaired, and No-soft-legality under matched downstream training and held-out evaluation: | Training source | Success (%) ↑ | Collision (%) ↓ | Off-route (%) ↓ | Progress (m) ↑ | |---|---:|---:|---:|---:| | Full | **78.33 ± 2.89** | 11.67 ± 2.89 | 5.00 ± 1.67 | **62.80 ± 0.20** | | Unpaired | 76.67 ± 2.89 | 11.67 ± 2.89 | 6.67 ± 1.67 | 62.10 ± 0.66 | | No-soft-legality | 75.00 ± 5.00 | 15.00 ± 5.00 | 5.00 ± 3.33 | 61.30 ± 1.10 | Values are mean ± standard deviation across downstream policy-training seeds. Full improves mean success by 1.66 percentage points over Unpaired and 3.33 points over No-soft-legality, with higher progress in both cases. The soft-legality comparison retains the hard validity checks and isolates the effect of the soft correction. Its retention improves training performance even though removing it produces higher internal hardness and lower frozen-planner success. We will include the generated-scene and downstream training comparisons in new Appendices C.3.1 and C.3.2, respectively.Comment 3 — Guidance strength, actual difficulty, and training gains
Q2(a). Why does increasing guidance strength sometimes make scenarios easier?
Thank you for this insightful question. The non-monotonic behavior you identify in Appendix Table S8 is informative about how the guidance operates. Guidance strength scales a local denoising update; actual planner difficulty emerges from the resulting scene and its subsequent closed-loop interactions.
At each guided reverse step, the critic evaluates the current predicted-clean scene at static probes. The scaled, masked direction then changes the denoising transition. Later denoising steps, decoding, validity checks, NPC behavior, and ego reactions determine the eventual interaction. Increasing the scale therefore changes the generated configuration and subsequent interaction, and can move it away from a configuration that was particularly difficult for the planner.
We added a controlled strength comparison that reports the internal score, actual planner outcomes, and recursive completion together:
| Guidance strength | Empirical return ↓ | Frozen-P0 success (%) ↓ | Progress (m) ↓ | ↑ | Five-commit completion (%) ↑ |
|---|---|---|---|---|---|
| 0 | 0.243 | 60.00 | 35.50 | 0.0250 | 75.00 |
| 0.5 | 0.171 | 53.33 | 32.80 | 0.0490 | 75.00 |
| 1.0 | 0.214 | 57.78 | 31.90 | 0.0516 | 75.00 |
| 2.0 | 0.255 | 61.54 | 34.00 | 0.0475 | 65.00 |
This supplementary sweep uses a different generator checkpoint from Table S8.
The observations directly clarify the relationship:
- The default strength 0.5 gives the lowest empirical return and success rate, while retaining the same five-commit completion rate as guidance off.
- Strength 1.0 gives the highest internal energy and lowest progress, while 0.5 gives lower success and return. The metrics capture different aspects of the interaction: local critic-estimated value degradation, distance traveled before termination, and realized task outcomes.
- Strength 2.0 increases planner success to 61.54% and lowers generation completion to 65%. This reinforces the usefulness of evaluating both planner behavior and recursive continuation when choosing the operating strength.
Q2(b). How do internal hardness and closed-loop difficulty relate to training gains?
The additional training ablation makes this relationship concrete. Removing soft legality increases from 0.0490 to 0.0570 and lowers frozen-planner success from 53.33% to 50.00%. However, the planner trained on those scenarios attains 75.00% held-out success and 61.30 m progress, compared with 78.33% and 62.80 m for Full. Its collision rate is also higher: 15.00% versus 11.67%.
Thus, the training results favor the complete guidance design, which combines planner-targeted interaction pressure with the soft legality correction. We will expand Appendix C.1 with the new strength comparison and its interpretation, and add the training-utility analysis to Appendix C.3.2. Together, these appendix sections will connect the internal score, actual closed-loop outcomes, and downstream training performance to their corresponding measurements.
Thank you again for your detailed comments. We would be happy to clarify any questions about the counterfactual mechanism, component comparisons, or the relationship between guidance strength and planner performance.
Comment 1 — Novelty of recursive extension and CDPG
We sincerely thank you for the detailed review and for recognizing joint lane–NPC generation, preservation of committed content, reactive integration, and downstream planner evaluation. Your comments are helpful in making the methodological distinctions more precise and motivating the additional component and guidance-strength analyses.
W1. What does SceneDemand add beyond trajectory conditioning in a recursive generator?
SceneDemand turns the selected ego trajectory into an explicit, continually updated generation request. Three connected operations implement this:
For example, two trajectories starting at the same ego position can demand different extensions when one turns and the other continues straight. Temporal coverage also distinguishes trajectories that reach the unsupported region at different times. After a commit and simulator transition, the planner selects a new trajectory and the generation request is recomputed.
Submitted Table 2 separates the learned generation condition from the trigger:
The first comparison shows a 28-point completion improvement from intent-conditioned generation. The second shows an additional 8-point improvement from temporal triggering, together with lower starvation. These comparisons identify the practical contributions of the learned generation request and its temporal scheduling. We will include this explanation in a new Appendix A.4, “Method Positioning and the Relationship Between the Components.”
W2 / Q1. Specific guidance mechanism relative to Universal Guidance, Diffuser, SAFE-SIM, and DiffScene
Thank you for highlighting these relevant connections. Universal Guidance establishes guidance with external functions, and Diffuser uses diffusion for guided trajectory planning. SAFE-SIM develops controllable adversarial traffic behavior with diffusion, while DiffScene guides safety-critical scenario generation through adversarial optimization.
CDPG's technical contribution is the following specific construction:
These choices support a particular capability: shaping the NPCs introduced during on-demand road extension using the target planner's existing value representation. The new Uniform and Unpaired comparisons below directly examine the guidance mechanism against simpler multi-probe alternatives. We will also include this mechanism-level comparison with prior work in Appendix A.4.