acceptodds
Withdrawn from ICLR 2027

Bench2Drive-R: Retrieval Based Online Video Generation for Reactive Autonomous Driving Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce Bench2Drive-R, an interactive and controllable retrieval based online video generation pipeline for closed-loop reactive driving evaluation. Unlike existing video generative models for AD which generate multiple frames all at once, the proposed designs are tailored for interactive simulation, where sensor rendering and behavior rollout are decoupled by applying a separate behavioral controller to simulate the reactions of surrounding agents. As a result, the generative model could focus on image fidelity, control adherence, and spatial-temporal coherence. For temporal consistency, due to the stepwise interaction nature of simulation, we design a noise modulating temporal encoder with Gaussian blurring to encourage long-horizon autoregressive rollout of image sequences without deteriorating distribution shifts. For spatial consistency, a retrieval mechanism, which takes the spatially nearest images as references, is introduced to ensure scene-level rendering fidelity. The spatial relations between target and reference are explicitly modeled with 3D relative position encodings. The potential over-reliance of reference images is mitigated with hierarchical sampling and classifier-free guidance. We compare the generation quality with existing AD generative models and show its superiority in the online driving setting. We further integrate it into nuPlan and evaluate the generative qualities with closed-loop simulation results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous Driving, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hrTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Bench2Drive-R, a retrieval-based interactive online video generation pipeline tailored for closed-loop reactive evaluation of end-to-end autonomous driving (E2E-AD) models. The key insight is to decouple sensor rendering from behavior rollout: a rule-based behavioral controller (built on nuPlan's IDM + LQR) propagates the world state and produces layout controls, while a generative renderer autoregressively produces multi-view camera images. Experiments on nuScenes (open-loop) show improvements. The framework is integrated into nuPlan, where closed-loop evaluations on 14 filtered scenarios (10 clips each) yield higher VAD R-CLS and perception scores compared to log-replay and a no-reference/no-previous-frame baseline.

Critical strengths

Rather than proposing a new diffusion architecture, the paper carefully diagnoses two concrete failure modes introduced by the interactive simulation setting (autoregressive collapse and reference-image train-val mismatch), and proposes correspondingly targeted fixes. Tab. 4 in particular shows a convincing and practically relevant degradation curve across 0.5s–2s rollout horizons, with the proposed modulation nearly halving FID at 2s. The hierarchical reference sampling ablation and case study (Fig. 7) are also compelling.

Critical weaknesses
  1. Closed-loop evaluation is underpowered and its baselines are too weak to support the main claim. The paper's central contribution claim is enabling closed-loop reactive simulation, yet the only closed-loop result (Tab. 5) is on 140 manually filtered clips in a dataset that the authors admit has sensor coverage for only 10% of scenarios. The two baselines are (a) log-replay, which by construction cannot react to ego behavior, and (b) an ablated version of Bench2Drive-R itself. Missing comparisons to any of the 3D reconstruction–based simulation methods the authors cite in the related-work section, or even a stronger non-learned baseline (e.g., retrieved-nearest-frame with interpolated appearance and warped layout), mean we cannot tell whether the generative approach is actually preferable for closed-loop assessment. This directly undermines confidence in the main thesis.

  2. Limited algorithmic novelty and unclear dependence on offline street-view coverage. Each individual module—noise-modulated conditioning, reference retrieval with cross-attention, 3D positional encodings, classifier-free guidance—is a relatively straightforward adaptation of known techniques, and the paper's contribution is therefore primarily at the systems level. That would be acceptable on its own, but the method's core premise (retrieving the two nearest logged frames) implicitly requires dense offline coverage of every scene the agent may visit. There is no characterization of performance as a function of reference-frame sparsity, and no ablation or discussion of behavior in regions with little or no database coverage. Given that log-deviation robustness is already a motivation, this is a notable gap.

Questions for authors
  1. Could the authors clarify why the closed-loop comparison in Tab. 5 is restricted to only 14 scenarios × 10 clips each (i.e., the sensor-data–covered 10% subset), and whether it would be feasible to add a comparison against at least one reconstruction-based simulator (e.g., the cited SURGE or TAG-style methods) or a nearest-frame warping baseline on the same subset? Even a small-scale addendum would substantially strengthen the main claim.
  2. The retrieval scheme requires (and upper-bounds performance by) offline street-view density. What happens when the evaluated E2E-AD agent deviates more than 15 m from the logged trajectory, or traverses a segment with sparse logged frames? Is there an empirical characterization of graceful degradation (e.g., FID vs. distance-to-nearest-reference curve) or a graceful fallback (e.g., disabling retrieval and running a pure layout-to-image path)?
  3. The paper attributes the observed VAD score drop-to-zero in closed loop to imitation-learning limitations of VAD itself (citing Cheng et al. 2023), rather than to rendering artifacts. Could the authors provide a direct diagnostic. For example, reporting the same VAD closed-loop scores under ground-truth camera images fed into the same nuPlan behavioral loop, so the reader can isolate planner failure vs. renderer-induced failure?
Reviewer_wAvpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bench2Drive-R, an autoregressive diffusion renderer for reactive autonomous-driving simulation. It combines previous-frame conditioning, noise modulation and Gaussian blur, object/map controls, and retrieval of nearby logged images with 3D positional encoding; the renderer is integrated with an LQR/kinematic-bicycle ego model and IDM agents. The reported nuScenes results are promising, while the closed-loop study is too limited to establish the claimed general-purpose online simulator.

Critical strengths

S1. Clear simulation-oriented decomposition. Section 3.1 and Figure 2 sensibly separate behavior rollout from sensor rendering. This permits the renderer to condition on updated boxes, maps, ego state, and cameras rather than learning all dynamics implicitly in pixels.

S2. Strong reported open-loop results. Table 1 reports FID 10.95 and FVD 101, compared with 15.50/103 for Panacea+ and 16.20/218.1 for MagicDrive. It also reports strong generated-image perception results, including BEVFormer NDS 34.70 and StreamPETR NDS 40.23.

Critical weaknesses

W1. Closed-loop validation is too limited for the central claim. Section 4.2 evaluates ten clips from each of 14 Val14 scenarios. Table5 compares only against log replay and a “No Ref & No Prev” variant. Only VAD is tested, although the appendix documents frequent planner failures. The full method improves R-CLS from 28.5 6 to 30.49, but no per-scenario distributions, confidence intervals, or repeated seeds are reported. More clips, multiple planners(https://huggingface.co/spaces/AGC2024-P/e2e-driving-navtest), more renderer baselines(https://github.com/hyzhou404/HUGSIM) are needed.

W2. The paper does not state the maximum rollout length of the closed-loop evaluation, or the corresponding visual quality. For the simulation framework on nuPlan, please also report how many frames are generated in each interaction with the policy model, and how long each interaction takes.

W3. Novelty and generalization claims are overstated. Previous-frame conditioning, ControlNet, CFG, reference attention, 3D positional encoding, and corrupted teacher forcing are established ideas. Figure 10 demonstrates some control extrapolation, but not broad environmental or geographic generalization.

Questions for authors

Q1. How does retrieval handle parallel roads and large lateral deviations?

Q2. What are the renderer latency, throughput, peak memory?

Reviewer_5v1EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Bench2Drive-R, a retrieval-based autoregressive image generation framework for closed-loop autonomous driving simulation. It separates traffic behavior simulation from sensor rendering, using nuPlan to update world states and a diffusion model to generate multi-view camera images conditioned on those states. Its main components include previous-frame conditioning with noise modulation and Gaussian blurring to mitigate autoregressive error accumulation, and retrieval of nearby logged images with 3D positional encoding to maintain scene consistency. Hierarchical reference sampling and reference classifier-free guidance further address robustness when simulated trajectories deviate from recorded ones. The framework is evaluated on nuScenes through image/video generation metrics, downstream perception, and open-loop planning, and on nuPlan through closed-loop evaluation with VAD. Comparisons, ablations, and qualitative examples report improvements in generation quality, control adherence, and downstream task performance over the evaluated baselines.

Critical strengths
  1. Well-motivated and practical framework. Separating traffic behavior simulation from sensor rendering provides a clear way to integrate generative models into nuPlan, enabling reactive closed-loop evaluation with explicit world states.

  2. Evaluation beyond visual quality. The experiments assess downstream perception and planning alongside generation quality. Improvements across multiple perception models and component ablations support the effectiveness of temporal conditioning and reference-image retrieval.

Critical weaknesses
  1. Limited methodological novelty and insufficient differentiation from relevant prior work. The proposed framework primarily integrates established techniques, including ControlNet conditioning, 3D positional encoding, and classifier-free guidance. More specifically, corrupting historical latent frames with Gaussian noise and conditioning on the noise level to mitigate autoregressive drift closely follows GameNGen, which the paper cites. At the system level, DriveArena already combines traffic simulation with autoregressive generative rendering for closed-loop driving evaluation, yet is neither discussed nor compared against. The paper therefore needs to establish more clearly the technical advance provided by its retrieval-based design. Quantitative ablations of hierarchical sampling, reference CFG, and 3D positional encoding are also missing, making it difficult to assess whether the remaining contributions constitute a substantial methodological advance.

  2. Insufficient evidence for reliable closed-loop evaluation. The central application is supported by experiments with only one driving policy, VAD, on 140 selected clips, compared against log replay and a renderer without reference or previous-frame conditioning. The R-CLS improvement from 28.56 to 30.49 is reported without uncertainty estimates, and a higher driving score alone does not establish that the simulator provides a more faithful assessment of driving capability. Furthermore, the autoregressive stability ablation in Table 4 extends only to two seconds, providing limited support for the claimed long-horizon robustness. These limitations leave the framework’s reliability under sustained interaction and deviations from logged trajectories insufficiently established.

Questions for authors
  1. Novelty and comparison with prior work. What substantive advances distinguish the proposed method from GameNGen’s noise augmentation and DriveArena’s closed-loop generative simulation? Please provide quantitative ablations of hierarchical sampling, reference CFG, and 3D positional encoding using the same backbone and reference inputs. A comparison with a directly relevant generative renderer, with training data and reference-image access clearly specified, would help establish the contribution.

  2. Reliability of closed-loop evaluation. How can improvements in VAD’s driving score be attributed to greater simulation fidelity? Please report uncertainty estimates and results with an additional driving policy or controlled failure scenarios. Please also extend the stability evaluation beyond two seconds to full simulation rollouts, measuring generation quality and control adherence as functions of rollout duration and deviation from logged trajectories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.