acceptodds
Under review as a conference paper at ICLR 2027

SceneDemand: Intent-Conditioned Recursive Vector Scene Generation for Autonomous Driving

Abstract

Generating interactive corner cases helps expose weaknesses in autonomous-driving planners and supports simulation-based training. However, existing scene-centric approaches do not explicitly align extension with the planner’s intended trajectory and planning horizon. Scenes sampled without planner feedback may also lack informative interactions with the ego. Furthermore, increased difficulty alone does not guarantee training value. We propose SceneDemand, an intent-conditioned framework for on-demand recursive vector scene generation. Building on the Scenario Dreamer prior, we design a joint lane–agent diffusion generator that completes regions along the selected ego trajectory while preserving previously committed content. A temporal-coverage mechanism triggers extension when the existing map cannot support the full planning horizon, enabling scene growth to adapt to evolving ego intent. We further introduce Counterfactual Direct Planner Guidance (CDPG), which compares frozen-critic values with and without new vehicles at multiple static intent-aligned viewpoints to steer their generation toward planner-specific challenges. On Waymo, intent conditioning increases the recursive extension completion rate from 48% to 76%. Under matched training and evaluation across three policy seeds, planners trained on SceneDemand-CDPG scenes achieve 77.92% mean success on held-out Waymo Log Replay scenes, compared with 75.75% for Scenario Dreamer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Recursive Vector Scene Generation, Counterfactual Planner Guidance, Closed-Loop Policy Learning.

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xpT9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "SceneDemand," a framework for the recursive generation of vectorized scenes for autonomous driving simulation. The method primarily addresses two challenges: determining when an existing scene requires expansion, and deciding where to generate new road structures and traffic NPCs. Specifically, this paper proposes a "temporal coverage" metric to assess whether the current map can support the ego-vehicle’s planned trajectory over the entire prediction horizon. When coverage is insufficient, the system triggers scene generation to complete missing road structures and initialize the states of new NPCs. Furthermore, this paper introduces "Counterfactual Direct Planner Guidance" (CDPG), which leverages a planner’s value assessment during the diffusion sampling process to guide the generation of new NPCs. By comparing the planner’s value, the method estimates the impact of these entities on the ego-vehicle’s planning outcomes. This process is further refined by incorporating multiple probing perspectives and scene feasibility constraints.

Critical strengths
  1. The framework is capable of simultaneously generating missing road structures and new NPCs, while preserving existing elements during subsequent denoising and scene expansion steps. Newly generated NPCs are integrated into downstream behavioral simulations, enabling closed-loop interaction.
  2. The experimental evaluation goes beyond merely assessing the generated scenes themselves; it examines recursive expansion capabilities, traffic distribution, and closed-loop planning performance. Additionally, the study tests the utility of the generated scenes for training downstream planners.
Critical weaknesses
  1. The novelty remains unclear. Existing literature has already explored recursive scene generation, continuous expansion, and the use of generated scenes to train downstream planners. The primary distinction in this paper is the use of the ego-vehicle’s future trajectory to identify map gaps and the "temporal coverage" metric to trigger scene expansion. While this design has merit, the paper fails to clearly articulate the substantive improvements it offers over simply incorporating trajectory-based conditioning into existing recursive generation models.
  2. The specific novel contributions of CDPG require further clarification. The use of gradients from external objectives to guide diffusion sampling has already been explored in works such as Universal Guidance and Diffuser; similarly, SAFE-SIM and DiffScene have investigated the generation of challenging scenarios for autonomous driving planners using diffusion models. Therefore, the primary distinction of CDPG should lie in its specific guidance mechanism, rather than merely the use of a critic to steer a diffusion model toward generating difficult scenarios.
  3. The role of the counterfactual value difference remains insufficiently clear. CDPG estimates the impact of adding a new NPC by comparing the planner’s value with and without that NPC present. However, based on the gradient formula provided in the paper, when only a single observation perspective is considered and no new NPCs are present, the counterfactual value difference does not alter the gradient direction of the difficulty objective. In this case, it effectively reduces to standard critic-based gradient guidance. Its primary function appears to be assigning varying weights across multiple perspectives.
  4. The absence of key ablation studies. CDPG incorporates counterfactual value differences, multi-perspective aggregation, and feasibility constraints; however, the paper does not evaluate the individual contribution of each component. For instance, could a similar outcome be achieved by simply averaging the critic gradients across multiple perspectives? How do the generated scenarios and downstream training performance change if the counterfactual comparison or the feasibility constraints are removed? The current experiments demonstrate that the full CDPG pipeline is effective, but they do not establish that each of these specific design elements provides an incremental benefit.
Questions for authors
  1. Compared to existing diffusion-based adversarial scenario generation methods like SAFE-SIM and DiffScene, what are the specific advantages of CDPG? Could the authors provide component-level ablation studies and compare against a simpler multi-perspective critic-based gradient guidance approach?
  2. The paper reports that CDPG increases the internal difficulty metric from 0.0141 to 0.0487, while simultaneously reducing the success rate of a fixed planner from 60% to 52%. However, the guidance strength experiments in the appendix show that the success rate is 18.18% at a strength of 0.5, whereas increasing the strength to 2.0 actually raises the success rate to 55.56%. This suggests that increasing guidance strength does not necessarily result in more challenging scenarios. Could the authors explain this phenomenon and further analyze the relationship between internal difficulty metrics, actual closed-loop performance, and downstream training gains?
4 comments
AuthorsOpenReview ↗

Comment 1 — Novelty of recursive extension and CDPG

We sincerely thank you for the detailed review and for recognizing joint lane–NPC generation, preservation of committed content, reactive integration, and downstream planner evaluation. Your comments are helpful in making the methodological distinctions more precise and motivating the additional component and guidance-strength analyses.

W1. What does SceneDemand add beyond trajectory conditioning in a recursive generator?

SceneDemand turns the selected ego trajectory into an explicit, continually updated generation request. Three connected operations implement this:

  1. Demand localization: Qt=C(It)∖Ω(Mt)Q_t=\mathcal C(I_t)\setminus\Omega(M_t) identifies the unsupported part of the selected intent corridor.
  2. Demand timing: Tcov(Mt,It)T_{\mathrm{cov}}(M_t,I_t) measures the supported prefix in planner time and triggers extension when support falls below the planning horizon.
  3. Demand-conditioned generation: the learned embeddings of ItI_t, QtQ_t, and TcovT_{\mathrm{cov}} condition the frontier slots of the joint lane–NPC denoiser, as specified in Eqs. (6)–(8).

For example, two trajectories starting at the same ego position can demand different extensions when one turns and the other continues straight. Temporal coverage also distinguishes trajectories that reach the unsupported region at different times. After a commit and simulator transition, the planner selects a new trajectory and the generation request is recomputed.

Submitted Table 2 separates the learned generation condition from the trigger:

Generation conditionTriggerFive-commit completion (%) ↑Map starvation (%) ↓
Without intent-conditioned generationTemporal48.000.09
Intent-conditionedSpatial68.000.15
Intent-conditionedTemporal76.000.05

The first comparison shows a 28-point completion improvement from intent-conditioned generation. The second shows an additional 8-point improvement from temporal triggering, together with lower starvation. These comparisons identify the practical contributions of the learned generation request and its temporal scheduling. We will include this explanation in a new Appendix A.4, “Method Positioning and the Relationship Between the Components.”

W2 / Q1. Specific guidance mechanism relative to Universal Guidance, Diffuser, SAFE-SIM, and DiffScene

Thank you for highlighting these relevant connections. Universal Guidance establishes guidance with external functions, and Diffuser uses diffusion for guided trajectory planning. SAFE-SIM develops controllable adversarial traffic behavior with diffusion, while DiffScene guides safety-critical scenario generation through adversarial optimization.

CDPG's technical contribution is the following specific construction:

  • Planner-derived signal: the value function is taken from the pretrained PPO planner being targeted and remains frozen.
  • Paired marginal scoring: each probe compares an otherwise identical scene with and without the current increment's NPCs, measuring their added value degradation.
  • Intent-aligned aggregation: the same candidate is evaluated from multiple static positions along the selected intent, and log-mean-exp weights the probe-specific drops.
  • Persistent scene integration: the gradient acts on the new-NPC part of the joint lane–agent increment, which is committed into the world and subsequently evolves through reactive simulation.

These choices support a particular capability: shaping the NPCs introduced during on-demand road extension using the target planner's existing value representation. The new Uniform and Unpaired comparisons below directly examine the guidance mechanism against simpler multi-probe alternatives. We will also include this mechanism-level comparison with prior work in Appendix A.4.

AuthorsOpenReview ↗

Comment 2 — Counterfactual weighting and component ablations

W3. Does the counterfactual baseline change the gradient?

Your observation is correct and helps clarify the mechanism. At one fixed probe, removing the new NPCs makes the baseline independent of the new-NPC latent zz, so

=-\nabla_z V_0(o_k^+).$$ **The paired baseline acts through the relative weights across probes.** For the multi-probe objective, $$\nabla_z E_{\mathrm{hard}} =-\sum_k w_k\nabla_z V_0(o_k^+), \qquad w_k=\operatorname{softmax}_k\left(\frac{V_0(o_k^-)-V_0(o_k^+)}{\tau_a}\right).$$ This gives three directly comparable guidance rules: | Variant | Probe weights | |---|---| | Uniform | $w_k=1/K$ | | Unpaired | $w_k\propto\exp[-V_0(o_k^+)/\tau_a]$ | | Full | $w_k\propto\exp[(V_0(o_k^-)-V_0(o_k^+))/\tau_a]$ | Full emphasizes where the newly introduced NPCs cause the greatest value drop relative to that probe's own baseline. For a single probe, or a common baseline across probes, the baseline cancels as you describe. With different baselines, it can change the combined direction by changing the relative probe weights. We will expand Appendix A.3.3 with this interpretation of Eq. (11) and the explicit comparison of the three weighting rules. ### W4 / Q1. Comparison with uniform multi-probe guidance and removal of components Following your helpful suggestion, we added the Uniform comparison and the paired and legality ablations. The multiple-probe variants use the same probes, and the guided component comparisons hold the denoising schedule and candidate-selection procedure fixed. | Variant | Empirical return ↓ | Frozen-P0 success (%) ↓ | Progress (m) ↓ | $E_{\mathrm{hard}}$ ↑ | |---|---:|---:|---:|---:| | Off | 0.243 | 60.00 | 35.50 | 0.0250 | | Single | 0.218 | 57.78 | 34.70 | 0.0340 | | Uniform | 0.196 | 55.56 | 34.00 | 0.0410 | | Unpaired | 0.193 | 55.56 | 33.80 | 0.0420 | | Full | 0.171 | 53.33 | 32.80 | 0.0490 | | No-soft-legality | 0.134 | 50.00 | 32.40 | 0.0570 | Return is the realized discounted reward of the frozen planner on completed generated scenes. Relative to Uniform, **Full reduces return from 0.196 to 0.171 and success from 55.56% to 53.33%**. Relative to Unpaired, return decreases from 0.193 to 0.171. These are the direct comparisons with simpler multi-perspective guidance requested in your review. We also compared the training value of Full, Unpaired, and No-soft-legality under matched downstream training and held-out evaluation: | Training source | Success (%) ↑ | Collision (%) ↓ | Off-route (%) ↓ | Progress (m) ↑ | |---|---:|---:|---:|---:| | Full | **78.33 ± 2.89** | 11.67 ± 2.89 | 5.00 ± 1.67 | **62.80 ± 0.20** | | Unpaired | 76.67 ± 2.89 | 11.67 ± 2.89 | 6.67 ± 1.67 | 62.10 ± 0.66 | | No-soft-legality | 75.00 ± 5.00 | 15.00 ± 5.00 | 5.00 ± 3.33 | 61.30 ± 1.10 | Values are mean ± standard deviation across downstream policy-training seeds. Full improves mean success by 1.66 percentage points over Unpaired and 3.33 points over No-soft-legality, with higher progress in both cases. The soft-legality comparison retains the hard validity checks and isolates the effect of the soft correction. Its retention improves training performance even though removing it produces higher internal hardness and lower frozen-planner success. We will include the generated-scene and downstream training comparisons in new Appendices C.3.1 and C.3.2, respectively.
AuthorsOpenReview ↗

Comment 3 — Guidance strength, actual difficulty, and training gains

Q2(a). Why does increasing guidance strength sometimes make scenarios easier?

Thank you for this insightful question. The non-monotonic behavior you identify in Appendix Table S8 is informative about how the guidance operates. Guidance strength scales a local denoising update; actual planner difficulty emerges from the resulting scene and its subsequent closed-loop interactions.

At each guided reverse step, the critic evaluates the current predicted-clean scene at static probes. The scaled, masked direction then changes the denoising transition. Later denoising steps, decoding, validity checks, NPC behavior, and ego reactions determine the eventual interaction. Increasing the scale therefore changes the generated configuration and subsequent interaction, and can move it away from a configuration that was particularly difficult for the planner.

We added a controlled strength comparison that reports the internal score, actual planner outcomes, and recursive completion together:

Guidance strengthEmpirical return ↓Frozen-P0 success (%) ↓Progress (m) ↓EhardE_{\mathrm{hard}} ↑Five-commit completion (%) ↑
00.24360.0035.500.025075.00
0.50.17153.3332.800.049075.00
1.00.21457.7831.900.051675.00
2.00.25561.5434.000.047565.00

This supplementary sweep uses a different generator checkpoint from Table S8.

The observations directly clarify the relationship:

  1. The default strength 0.5 gives the lowest empirical return and success rate, while retaining the same five-commit completion rate as guidance off.
  2. Strength 1.0 gives the highest internal energy and lowest progress, while 0.5 gives lower success and return. The metrics capture different aspects of the interaction: local critic-estimated value degradation, distance traveled before termination, and realized task outcomes.
  3. Strength 2.0 increases planner success to 61.54% and lowers generation completion to 65%. This reinforces the usefulness of evaluating both planner behavior and recursive continuation when choosing the operating strength.

Q2(b). How do internal hardness and closed-loop difficulty relate to training gains?

The additional training ablation makes this relationship concrete. Removing soft legality increases EhardE_{\mathrm{hard}} from 0.0490 to 0.0570 and lowers frozen-planner success from 53.33% to 50.00%. However, the planner trained on those scenarios attains 75.00% held-out success and 61.30 m progress, compared with 78.33% and 62.80 m for Full. Its collision rate is also higher: 15.00% versus 11.67%.

Thus, the training results favor the complete guidance design, which combines planner-targeted interaction pressure with the soft legality correction. We will expand Appendix C.1 with the new strength comparison and its interpretation, and add the training-utility analysis to Appendix C.3.2. Together, these appendix sections will connect the internal score, actual closed-loop outcomes, and downstream training performance to their corresponding measurements.

Thank you again for your detailed comments. We would be happy to clarify any questions about the counterfactual mechanism, component comparisons, or the relationship between guidance strength and planner performance.

Reviewer_62hzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SceneDemand, an intent-conditioned framework for recursive vector scene generation in autonomous driving. It introduces a joint lane-agent diffusion generator that extends driving scenes based on the ego planner's intended trajectory and temporal coverage. The authors further propose Counterfactual Direct Planner Guidance (CDPG), which uses a frozen planner's value function to guide the generation of challenging NPC interactions. Experiments on Waymo demonstrate improved recursive scene generation and better downstream planning performance through training on the generated scenarios.

Critical strengths
  1. Planner-guided scene generation. The paper introduces an interesting approach that uses ego intent and planner feedback to generate challenging and relevant driving scenarios in vector space.
  2. Comprehensive downstream evaluation. The paper evaluates both recursive scene generation and its impact on downstream planner training, demonstrating improvements through controlled closed-loop experiments.
Critical weaknesses
  1. Insufficient Ablation of CDPG. The paper lacks sufficient ablation studies to justify the individual components of CDPG, making it difficult to assess their respective contributions.
  2. Presentation Quality. Figures (e.g., Fig.1, 2, 3) have a noticeably AI-generated appearance, and some passages (e.g., lines 99 and 355) also appear potentially highly AI-generated. The overall presentation could be improved.
  3. Lack of Computational Efficiency Analysis. The proposed method involves multi-step diffusion sampling and repeated critic evaluations, but the paper does not report the computational overhead or generation latency, leaving its practical efficiency unclear.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Comment 1 — Component-level ablations

We sincerely thank you for recognizing the value of planner-guided vector scene generation and the downstream evaluation. Your suggestions are helpful in strengthening the component analysis, presentation, and computational characterization of SceneDemand.

W1. Insufficient ablation of CDPG

Following your suggestion, we added controlled comparisons for the individual guidance choices. Off disables the guidance gradient; Single uses the current-pose probe; Uniform averages the paired objectives; Unpaired removes the per-probe counterfactual baseline while retaining the multiple probes; No-soft-legality removes the soft legality-gradient correction while retaining hard validity checks. The candidate-selection procedure and guided-step schedule are fixed for the corresponding comparisons.

VariantEmpirical return ↓Frozen-P0 success (%) ↓Progress (m) ↓EhardE_{\mathrm{hard}} ↑
Off0.24360.0035.500.0250
Single0.21857.7834.700.0340
Uniform0.19655.5634.000.0410
Unpaired0.19355.5633.800.0420
Full0.17153.3332.800.0490
No-soft-legality0.13450.0032.400.0570

Return is the realized discounted reward of the frozen planner on completed generated scenes. Full lowers return and planner success relative to Off, Single, Uniform, and Unpaired, providing component-level evidence for the complete guidance design.

We also evaluated how the paired and legality components affect downstream planner training:

Training sourceHeld-out success (%) ↑Collision (%) ↓Progress (m) ↑
Full78.33 ± 2.8911.67 ± 2.8962.80 ± 0.20
Unpaired76.67 ± 2.8911.67 ± 2.8962.10 ± 0.66
No-soft-legality75.00 ± 5.0015.00 ± 5.0061.30 ± 1.10

The results report mean ± standard deviation across downstream policy-training seeds. Full improves mean success by 1.66 percentage points over Unpaired and 3.33 points over No-soft-legality, with higher progress in both comparisons. Together, the two tables explain how the guidance components affect both the generated challenges and their value for planner learning. We will include these results in new Appendices C.3.1, “Component Ablations and Empirical Planner Returns,” and C.3.2, “Downstream Training with Ablated Guidance.”

AuthorsOpenReview ↗

Comment 2 — Presentation and computational efficiency

W2. Presentation of Figures 1–3 and the identified passages

Thank you for these helpful suggestions on presentation. We appreciate your feedback and will further improve the figures and wording to make the technical presentation clearer and more precise.

We will refine the existing figures and text with the following specific changes:

  • Figure 1: make the sequence “planner intent → coverage test → joint generation → commit → simulation” explicit, so the feedback loop is immediately visible.
  • Figure 2: simplify labels and consistently distinguish committed tokens, frontier tokens, and the three conditioning inputs.
  • Figure 3: clarify that the paired observations share the same scene and differ in the presence of the newly generated NPCs, and show how probe weights compose the guidance direction.
  • The passages around lines 99 and 355: use direct wording for the reported training improvement and for the provenance of the comparison values, with self-contained table captions.

These edits target the clarity of the architecture, mathematical mechanism, and experimental interpretation. We will also check notation and terminology consistently across the figures, captions, and main text.

W3. Computational overhead and generation latency

Thank you for requesting this analysis. We measured generation latency and peak allocated GPU memory for Off, Single, and Full under the same sampling configuration.

VariantTime per candidate (s) ↓Mean time per generation event (s) ↓Peak allocated GPU memory (GiB) ↓
Off5.1145.973.16
Single5.5248.614.24
Full5.9351.255.32

Full adds 0.82 s per candidate and 5.28 s per generation event relative to Off, corresponding to 16.0% and 11.5%, respectively. Its peak allocated memory is 5.32 GiB. Relative to Single, Full adds 2.64 s per generation event for the multi-probe formulation.

The method applies guidance at the selected reverse steps {70,55,40}\{70,55,40\} within DDPM-100. Each guidance evaluation uses static probes of the predicted-clean scene, and gradients flow through the frozen critic and decoder to the active new-NPC latents. CtRL-Sim supplies the subsequent reactive NPC behavior. This explains where the added computation occurs and why the overhead is concentrated at the guided denoising steps.

Temporal coverage determines when a generation event is requested: scene extension is invoked when support for the selected intent becomes insufficient. The table reports the cost of that extension operation. We will include this table, its measurement definitions, and the sampling and guidance settings in a new Appendix C.4, “Computational Efficiency.”

Thank you again for your constructive feedback. Please let us know if any aspects of the added ablations, efficiency analysis, or planned presentation improvements would benefit from further clarification.

I thank the authors for the rebuttal. However, it would be more convincing to submit an edited version of the paper, since ICLR allows it. Also, after considering the other reviewers' comments, I will maintain my initial score of 2 weak reject. I suggest the paper go for another round of peer review to: 1) redraw all three pictures first and 2) refine the main text and experiments.

Reviewer_puhrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SceneDemand, a vector scene generation method for self-driving. SceneDemand generates new lanes and NPC states conditioned on the ego plan, and use critic values to guide NPC generation toward scenarios that are challenging for the planner.

Critical strengths

The paper is well-written and informative. Scene generation with planner's feedback in a closed-loop is an important research problem. The idea of using critic to guide genneration is overall interesting.

Critical weaknesses

The novelty seems to be limited. The broad idea of conditioning scenario generation on intents is not new (e.g., SceneDiffuer, and UniGen). Critic guidance has also been explored by prior works like Diffuser and AdvDiffuser.

Questions for authors

A learned critic can be highly biased such that the guidance could be ineffective. For example, the critic can be very inaccurate when scenarios are unseen by the critic during PPO. It would be very helpful to include more experiments such as (a) examining the empirical returns before and after guidance to show the critic prior makes a difference, and (b) testing the sensitivity to critic estimation by using critics trained with random seeds, which would prove that the training gains are consistent regardless of critic bias.

The guidance prior should also be carefully ablated. For example, the legality or single-probe could be compared to full CDPG. These ablations would show each component in CDPG is not a heuristic design but necessary for the training gains.

4 comments
AuthorsOpenReview ↗

Comment 1 — Novelty and empirical effectiveness of critic guidance

We sincerely thank you for your encouraging assessment of the paper's clarity, the importance of planner-feedback-driven generation, and the interest of critic-guided scene generation. Your suggestions are helpful in clarifying the technical contributions and evaluating the guidance through actual planner behavior.

W1. Novelty relative to intent-conditioned generation and prior critic guidance

SceneDemand makes the planner's evolving needs determine both the scene-extension request and the NPC interactions generated within that extension, with planner training as the downstream objective. Thank you for pointing to these relevant works; the distinctions are most clearly explained at the level of their mechanisms.

Intent-conditioned generation. SceneDiffuser unifies scene initialization and rollout, while UniGen jointly models initial agent states and future trajectories conditioned on scene context. SceneDemand uses the planner-selected trajectory to define a generation request: its unsupported corridor specifies where to extend, and its remaining temporal support determines when extension is needed. The intent, unsupported-region query, and temporal-coverage embeddings jointly condition the frontier slots of the lane–NPC generator in Eqs. (6)–(8). As the planner changes its selected trajectory, this request changes with it. Submitted Table 2 shows the resulting benefit: intent-conditioned generation raises five-commit completion from 48% to 76%, and temporal triggering raises it from 68% to 76% relative to a spatial trigger.

Planner-value guidance. Diffuser uses guided diffusion for trajectory planning, and AdvDiffuser combines a traffic-trajectory diffusion prior with a learned guide model for adversarial generation. CDPG directly reuses the value function of a pretrained PPO planner. At each intent-aligned probe, it compares the same predicted-clean scene with and without the current increment's NPCs. Log-mean-exp then emphasizes the largest paired value drops, and the masked guidance update acts on the active new-NPC latents. This construction targets the marginal impact of newly introduced traffic along the planner's selected intent.

These two mechanisms connect demand-aligned road extension with planner-targeted NPC generation in one persistent scene. We will include these distinctions in a new Appendix A.4, “Method Positioning and the Relationship Between the Components.” The additional comparisons below examine the contribution of the specific guidance choices.

Q(a). Do empirical returns change after guidance?

Thank you for this useful suggestion. We added actual closed-loop returns using the frozen planner and the simulator reward, G=∑tγtrtG=\sum_t\gamma^t r_t.

Generation variantEmpirical return ↓Frozen-P0 success (%) ↓Progress (m) ↓
Guidance off0.24360.0035.50
Full CDPG0.17153.3332.80

These are evaluations on completed generated scenes with the same frozen planner and NPC behavior settings. CDPG reduces empirical return by 29.6%, success by 6.67 percentage points, and progress by 2.70 m. The measured behavioral changes provide the empirical-return evidence you requested and show that guidance produces more challenging interactions for the target planner. We will include this comparison in a new Appendix C.3.1, “Component Ablations and Empirical Planner Returns.”

AuthorsOpenReview ↗

Comment 2 — Critic seeds and component-level evidence

Q(b). Reliability of the pretrained critic and sensitivity to critic initialization

Thank you for this helpful suggestion. We agree that the practical value of critic guidance should be assessed through realized closed-loop outcomes and downstream learning performance, especially when generated scenarios differ from the critic’s pretraining distribution.

CDPG uses the publicly released pretrained PPO checkpoint for planner P0P_0 and reuses its value function, with the parameters frozen during generation. Our objective is to generate useful challenges for this given planner, which also provides the common initialization for downstream training.

The paper evaluates this usefulness at two levels. First, Table 3 measures actual closed-loop behavior under the same NPC behavior model and tilt: moving from Base to CDPG reduces P0P_0’s success rate from 60% to 52% and route progress from 35.92 m to 32.79 m. These behavioral outcomes provide evidence beyond the internal critic score EhardE_{\mathrm{hard}}. Second, Table 4 evaluates downstream policies on 500 held-out Waymo Log Replay scenes under matched training budgets. Across three policy-training seeds, CDPG increases mean success from 75.97±2.25%75.97\pm2.25\% to 77.92±1.98%77.92\pm1.98\%, and mean progress from 61.26±1.6761.26\pm1.67 m to 62.82±0.2062.82\pm0.20 m. Together, these results support the usefulness of the generated challenges for the evaluated pretrained planner.

Your suggestion to evaluate independently trained critics is valuable for understanding sensitivity to pretraining initialization. Our experiments use the publicly released pretrained weights. Given the computational resources available to us and the time remaining in the rebuttal period, we are unable to complete the additional critic-training runs needed for this comparison. The three seeds reported in our study concern downstream policy training from the same P0P_0. We therefore assess the usefulness of the fixed pretrained critic through the realized closed-loop outcomes and downstream learning results reported above. We appreciate this suggestion and will discuss sensitivity to critic initialization as an important direction for further evaluation in Appendix D.1.

We will include this clarification in Appendix D.1 and specify the distinction between downstream training seeds and critic-pretraining seeds in Appendix B.5.

W2. Component-ablation question. What do multiple probes and legality contribute?

Following your helpful suggestion, we compared Single, Uniform, Unpaired, and No-soft-legality with Full. Single uses the current-pose probe; Uniform averages the paired probe objectives; Unpaired retains the same multiple probes but aggregates their negative present-NPC values; No-soft-legality removes the soft legality correction while retaining hard validity checks.

VariantEmpirical return ↓Frozen-P0 success (%) ↓Collision (%)Progress (m) ↓
Single0.21857.7811.1134.70
Uniform0.19655.5611.1134.00
Unpaired0.19355.5613.3333.80
Full0.17153.3313.3332.80
No-soft-legality0.13450.0016.6732.40

The empirical return and success rate are lower with Full than with Single. This supports evaluating interactions from multiple intent-aligned positions, where a frontier NPC's effect can become relevant as the ego moves along its selected trajectory. The Uniform and Unpaired comparisons further separate the aggregation and paired-weighting choices.

We then evaluated the training consequences of the paired and legality terms under matched downstream training and held-out evaluation:

Training sourceSuccess (%) ↑Collision (%) ↓Off-route (%) ↓Timeout (%) ↓Progress (m) ↑
Full78.33 ± 2.8911.67 ± 2.895.00 ± 1.675.00 ± 1.6762.80 ± 0.20
Unpaired76.67 ± 2.8911.67 ± 2.896.67 ± 1.675.00 ± 1.6762.10 ± 0.66
No-soft-legality75.00 ± 5.0015.00 ± 5.005.00 ± 3.335.00 ± 3.3361.30 ± 1.10

Values are mean ± standard deviation across downstream policy-training seeds. Full achieves higher mean success and progress than both training ablations. Relative to Unpaired, it improves success by 1.66 percentage points and progress by 0.70 m. Relative to No-soft-legality, it improves success by 3.33 points and progress by 1.50 m, with a 3.33-point reduction in collision rate.

The latter comparison directly explains the role of legality: its removal makes generated scenes more difficult for the frozen planner, whereas retaining it yields more useful training scenarios according to held-out success and progress. We will include the generated-scene comparisons in Appendix C.3.1 and the downstream training results in a new Appendix C.3.2, “Downstream Training with Ablated Guidance.”

Reviewer_puhrOpenReview ↗

Response to Authors

We thank the author for conducting new experiments and adding clarifications.

I have a minor concern left because the clarifications do not seem to establish clear differences to prior intent-conditioned generation or critic guidance methods to support the novelty claims.

However, most of my concerns have been addressed. I raised my score to 3 since the paper is technically sound.

Thank you very much for taking the time to consider our response and reassess our work. We sincerely appreciate your decision to raise the score and your recognition of the paper’s technical soundness. We understand that our explanation of novelty still needs improvement, and we will revise the manuscript to clarify the specific differences from prior work and acknowledge its contributions more explicitly.

Reviewer_NeWwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SceneDemand, which performs on-demand recursive extension on top of a vector scene diffusion prior. The method dilates the planner-selected future trajectory I_t into an intent corridor, and triggers joint lane–agent inpainting when the longest intent prefix still supported by the committed map, T_cov, falls below the planning horizon; committed content is clamped at every denoising step and each increment is committed atomically before the simulator advances. CDPG, in turn, reconstructs the predicted-clean scene at reverse-diffusion steps, queries the frozen planner's value function at K static intent-aligned probes with paired observations that retain or remove the newly introduced NPCs, and applies guidance after aggregating the largest value drops with log-mean-exp. On Waymo, intent conditioning raises the five-commit completion rate from 48% to 76%, and under matched training the planner trained on CDPG scenes reaches 77.92% success.

Critical strengths

Defining the region to be completed by intent, and triggering inpainting by how many planning steps remain supported, is reasonable: it aligns when and where to extend with the planner's own time coordinate, and it removes the dependence on a fixed spatial boundary or an external control signal. The commit semantics are equally worth noting: committed content is clamped at every denoising step and each increment is atomic and non-overwriting, so later generation operates on an accumulated scene instead of reconstructing a local one, which is what makes the long rollouts in the appendix possible rather than a chain of independent reconstructions.

Critical weaknesses

My concern is that the two contributions have no necessary relation to each other, and that neither is of high value on its own.

1.The trigger and CDPG sit at two different layers. The former is masked inpainting plus a corridor containment test, and does not change the generator's structure; the latter only modifies the noise prediction at reverse steps and is independent of the trigger, the commit, and CtRL-Sim. Their only coupling is the shared input I_t, not a shared mechanism, and Tables 2 and 3 never vary the two sides together. They are two components that stand on their own.

2.Relative to Base, CDPG is an overall A/B over five changes (multiple probes, log-mean-exp, the paired baseline, legality subtraction, and the guided-step set), so no attribution is possible; the paired baseline is the only relatively new part among them, and the authors themselves note in A.3.3 that its contribution is not isolated. Moreover, the softmax in Eq. 10 is invariant to a common shift, so this baseline changes the guidance direction only when V0(o−) differs across probes, and the magnitude of that difference is never reported. Please add a paired versus unpaired comparison at the same K.

Questions for authors

Please check weakness.

3 comments
AuthorsOpenReview ↗

Comment 1 — Relationship and value of the two contributions

We sincerely thank you for your thoughtful review and for recognizing the value of intent-aligned extension, persistent committed content, and atomic scene updates. Your comments are helpful in sharpening the connection between our two contributions and motivating a more detailed examination of CDPG.

W1. How are intent-conditioned extension and CDPG connected?

Thank you for encouraging us to explain this relationship more precisely. Both components serve the same objective: generating coherent and informative driving scenes for planner training and evaluation. Intent-conditioned extension provides road support aligned with the planner's needs, while CDPG shapes the NPC interactions within those extensions to improve their training value.

This is the motivation stated in the submitted Introduction (line 81):

“Intent thus defines coverage demand, while CDPG targets interaction demand.”

The Conclusion (lines 480–482) expresses the same connection:

“Together, these components address both the need for continued scene coverage and the need for informative interactions.”

The two components operate on the road and interaction aspects of the same training environment. A useful generated scene must allow the planner to continue along its selected trajectory and expose it to interactions from which it can learn. The first component determines where road support and accompanying NPC states are needed; CDPG uses the pretrained planner's value function to shape the impact of the newly introduced NPCs. Their relationship is implemented through the joint scene increment and the subsequent planning loop:

  1. Generate the road context needed by the planner. The selected trajectory defines the unsupported region QtQ_t and temporal-coverage trigger. Intent, the unsupported region, and temporal support condition the joint lane–NPC generator through Eqs. (6)–(8).
  2. Generate informative NPC interactions in that context. CDPG evaluates the predicted-clean increment at intent-aligned probes and guides its new-NPC component. The road and NPC states are committed together into the persistent scene.
  3. Use the resulting scene for continued interaction and learning. The planner responds to the committed road and traffic, CtRL-Sim advances the NPCs, and the updated planner intent determines the next generation request. The generated environments supply the experiences used in downstream PPO training.

For example, when the selected intent turns beyond the available map, the generator completes the road needed for that turn. CDPG shapes the newly introduced NPCs according to their impact on the planner along this intent. The resulting scene supports both executing the turn and learning to handle the associated traffic interaction. Algorithm S1 describes how these increments participate in the continuing planning–generation loop.

The submitted results follow this logic. In Table 2, intent-conditioned generation increases five-commit completion from 48% to 76%; temporal triggering increases completion from 68% to 76% relative to a spatial trigger and reduces map starvation from 0.15% to 0.05%. These results establish the value of generating road support according to the planner's evolving needs.

With this scene-generation framework in place, CDPG shapes the generated NPC interactions: frozen-planner success decreases from 60% to 52% in Table 3. The corresponding training comparison in Table 4 then shows that CDPG-generated environments improve held-out planner success from 75.97% to 77.92% and progress from 61.26 m to 62.82 m. Thus, the two contributions are connected by a common training objective and by the generated road–NPC environment through which that objective is realized. We will include this explanation in a new Appendix A.4, “Method Positioning and the Relationship Between the Components.”

AuthorsOpenReview ↗

Comment 2 — CDPG attribution and the paired baseline

W2. Which CDPG components contribute, and what does the paired baseline change?

Thank you for this precise and helpful observation. The paired baseline calibrates the relative weights of the intent-aligned probes. Let zz denote the active new-NPC latent and let bk=V0(ok−)b_k=V_0(o_k^-) and vk(z)=V0(ok+)v_k(z)=V_0(o_k^+). The hardness gradient is

=-\sum_{k=1}^{K}w_k\nabla_z v_k(z), \qquad w_k=\frac{\exp((b_k-v_k)/\tau_a)}{\sum_j\exp((b_j-v_j)/\tau_a)}.$$ Your common-shift observation is correct. A baseline shared by all probes cancels, and for $K=1$ the weight is one. With probe-specific baselines, the relative weight satisfies $$\log\frac{w_k}{w_j} =\frac{(b_k-b_j)-(v_k-v_j)}{\tau_a}.$$ Thus, the contribution of the baseline to the relative log-weight is exactly $(b_k-b_j)/\tau_a$. This makes the role of cross-probe baseline differences explicit: the weighting prioritizes the value degradation associated with adding NPCs at each probe, accounting for differences in the probes' initial values. We additionally quantified these differences at the diagnostic states. The mean within-state range of the counterfactual probe baselines was 0.533, and the mean pairwise absolute baseline difference was 0.311. At the same latent states, the mean cosine similarity between the Full and Unpaired latent gradients was 0.863. These measurements complement the same-$K$ ablation by showing that the probe baselines vary in practice and that the resulting guidance directions are broadly aligned but not identical. Following your suggestion, we added **paired versus unpaired guidance at the same $K$**, together with the other component comparisons: - **Off:** disables the CDPG gradient update while retaining the candidate-selection procedure. - **Single:** uses the current-pose probe. - **Uniform:** replaces log-mean-exp with the arithmetic mean of the paired drops, yielding uniform probe weights. - **Unpaired:** uses log-mean-exp of $-V_0(o_k^+)$ with the same probes as Full. - **No-soft-legality:** removes the soft legality-gradient term while retaining the hard validity checks. The guided-step set is held fixed across the guided component variants. | Variant | Empirical return ↓ | Frozen-P0 success (%) ↓ | Progress (m) ↓ | $E_{\mathrm{hard}}$ ↑ | |---|---:|---:|---:|---:| | Off | 0.243 | 60.00 | 35.50 | 0.0250 | | Single | 0.218 | 57.78 | 34.70 | 0.0340 | | Uniform | 0.196 | 55.56 | 34.00 | 0.0410 | | Unpaired | 0.193 | 55.56 | 33.80 | 0.0420 | | Full | 0.171 | 53.33 | 32.80 | 0.0490 | | No-soft-legality | 0.134 | 50.00 | 32.40 | 0.0570 | Empirical return is the realized discounted reward of the frozen planner on completed generated scenes. Candidate scoring uses the same full paired energy across variants. **Full lowers empirical return relative to Single, Uniform, and Unpaired.** The same-$K$ comparison is particularly relevant to your question: Full reduces return from 0.193 to 0.171 and planner success from 55.56% to 53.33% relative to Unpaired. We also assessed the training utility of the paired and legality components: | Training source | Held-out success (%) ↑ | Collision (%) ↓ | Progress (m) ↑ | |---|---:|---:|---:| | Full | **78.33 ± 2.89** | 11.67 ± 2.89 | **62.80 ± 0.20** | | Unpaired | 76.67 ± 2.89 | 11.67 ± 2.89 | 62.10 ± 0.66 | | No-soft-legality | 75.00 ± 5.00 | 15.00 ± 5.00 | 61.30 ± 1.10 | The entries report mean ± standard deviation across downstream policy-training seeds under matched settings. Full improves mean held-out success by **1.66 percentage points over Unpaired** and **3.33 points over No-soft-legality**. The legality comparison is informative: removing the soft constraint increases frozen-planner difficulty, while retaining it produces better training outcomes. These experiments directly connect the component choices to both generated challenges and planner learning. We will add the component comparisons to new Appendices C.3.1, “Component Ablations and Empirical Planner Returns,” and C.3.2, “Downstream Training with Ablated Guidance,” and expand the weight-ratio interpretation in Appendix A.3.3.

Thank you again for your thoughtful feedback. Please let us know if you have any questions about the relationship between the two contributions or the paired-versus-unpaired comparison. We would be happy to provide further clarification.

Comments on the paper

1 comment
Area_Chair_ZW1wOpenReview ↗

Discussion phase

Dear Reviewers, the authors have posted their rebuttal. Please review their responses and engage in the discussion.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.