acceptodds
Withdrawn from ICLR 2027

Video Diffusion Rendering for 3D Asset Insertions in Driving Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The growing availability of reusable 3D assets and reconstructed real-world scenes is opening new opportunities for scalable and controllable driving simulation. A key capability is asset insertion, where external objects are placed into existing scenes to create diverse and interactive scenarios. However, geometrically rendered insertions often remain visually implausible, suffering from appearance mismatch and limited scene interaction, especially when assets originate from different scenes. While frame-wise harmonization can improve visual realism, it often introduces temporal inconsistency in video outputs. We address this naive-to-real rendering problem with a one-step video diffusion harmonizer that maps naive insertion renderings to photorealistic and temporally consistent driving videos. The harmonizer is obtained by fine-tuning a pretrained video diffusion model, with temporally agnostic latent injection introduced to recover spatial details lost during 3D VAE decoding. To effectively train the harmonizer, we construct DriveInsert-3K, an offline paired video dataset built with a two-stage alignment procedure to provide clean naive-insertion/real-video supervision. At inference time, our model takes a naive insertion video from an upstream insertion pipeline and produces a harmonized video in a single denoising step. Experiments show that our approach improves visual realism and temporal consistency across diverse driving scenes and inserted assets while enabling efficient single-step inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Asset Insertion, One-Step Video Diffusion, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mSpdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This submission looks at photorealistic 3D asset insertion in the context of driving simulators. Although being able to insert, replace, and manipulate 3D assets such as cars into a scene unlocks many valuable simulation use cases, it can be challenging to do so in a photorealistic manner (naively inserted assets will have different lighting than the rest of the scene, the materials don't fully match, etc). R3D2 [1] uses a 1-step image diffusion model to harmonize inserted assets with the rest of the scene, but suffers from temporal consistency issues such as flickering and appearance variations since the diffusion model denoises each frame independently. This submission instead proposes to LoRA-finetune a video diffusion model (CogVideoX) into a single-step harmonizer.

One challenge is that video models use highly compressed VAEs (CogVideoX's does 8x spatial and 4x temporal compression) that can cause renderings to lose fine detail and alter the original scene structure. As a solution, the paper proposes a "temporally agnostic latent injection" strategy where each frame is also independently encoded with a separate 2D VAE encoder. The resulting intermediate features are cached and merged into the 3D VAE decoder's last two upsampling blocks via "cross-modal fusion blocks." The authors train the fusion blocks and LoRA-finetune the 3D VAE decoder to handle this additional 2D signal with reconstruction loss ("Reconstruction-LoRA") before finetuning the rest of the video model.

The authors then LoRA-finetune the diffusion transformer and the VAE decoder with a second LoRA ("Harmonization-LoRA") with pixel-space losses (LPIPS loss and Gram). To supervise training, the authors curate a paired video dataset (DriveInsert-3K) from Waymo Open Dataset videos. Each pair consists of a clip with a naively inserted asset (the authors take care to geometrically align the asset as much as possible with its real-world counterpart) and the original real video.

The method is evaluated on a validation set that is not clearly defined (my guess is held-out scenes from DriveInsert-3K) against a "naive insertion" baseline and R3D2. The authors' method provides the best PSNR, LPIPS, DISTS, and vFID scores on full-resolution frames, although the PSNR gap reduces notably when R3D2 is retrained on DriveInsert-3K (these findings are relegated to the appendix). When evaluating actor-centric crops, the method has slightly worse vFID than the original R3D2. Temporal consistency is measured via flow-warping error, where the method is worse than naive insertion but better than R3D2 in both full-resolution videos and asset crops. The authors also share videos in the supplement where their method exhibits nicer results than the baselines (and less flickering than R3D2) along with scene editing capabilities (such as adjusting the orientation of a car).

References:

  • [1] William Ljungbergh et al. R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation. CVPRW 2026
Critical strengths

I agree with the premise that asset manipulation makes driving simulators far more useful, and that naive asset insertion is a challenge. I also agree that diffusion models are a promising solution and that temporal consistency is important. An efficient video diffusion method seems like a natural approach.

If released, DriveInsert-3K could also be a valuable asset to the community if properly contextualized relative to prior datasets such as that released by DiffusionHarmonizer [1] and the R3D2 dataset (R3D3) that is also derived from Waymo. The authors should explicitly confirm that they can commit to the release if the submission is accepted, as this would be a big factor in potentially recommending acceptance depending on the resolution of the concerns below.

Last but not least, I found this paper easy to read and I very much appreciate that the authors did not over-complicate things, which I do not take for granted.

References:

  • [1] Yuxuan Zhang et al. DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer. CVPR 2026.
Critical weaknesses

I am concerned about this paper's positioning and novelty relative to prior work, especially DiffusionHarmonizer [1] (CVPR 2026) which is highly relevant and not cited. DiffusionHarmonizer addresses a near-identical problem (temporally consistent harmonization of inserted assets), has public code and model checkpoints, and is an essential baseline to include in the paper to merit acceptance. The set of baselines used for experiments (just R3D2 and naive insertion in the main paper) is generally very thin - DiffusionRenderer [2], ChatSim [3], and GeoSim [4] all handle object insertion, and the first two have public code (the paper cites none of these). More broadly, there are many video harmonization works such as Huang et al. [5], Lu et al. [6], Harmonizer [7], and VHTT [8] that a video harmonizer-themed submission should be citing.

I also have serious reservations about the experimental protocol beyond the inadequate number of baselines:

  • How the evaluation set is constructed is not explicit in the paper. This should be explicitly disclosed and the authors should confirm that there is no leakage of scenes between the training and evaluation sets.
  • It is implied that it comes from the same DriveInsert-3K dataset used to train the method. From that perspective, having the main paper experiments compare to an R3D2 checkpoint trained on a different dataset is unfair. The appendix shows how the gap narrows when R3D2 is trained on the same data as the paper's method - at the very least, this stronger baseline should be what's included in the paper.
  • The evaluation section would be much stronger if evaluated on additional datasets such as the released DiffusionHarmonizer dataset to show proof of generalization.
  • A big reason why prior work uses image diffusion models instead of video models is because the latter are much slower and memory hungry. From this perspective, the claim that this paper's method renders at higher FPS than the R3D2 image-based pipeline is surprising, even when accounting for the higher temporal compression factor of the 3D VAE (both methods are single-step). The R3D2 number matches the R3D2-BIG configuration of their paper's Table 1 (the smaller variant clocks 13.36 FPS compared to this paper's 4.54), which raises the question of whether the timing protocols are matched. This paper uses 512x768 frames for training and validation (and doesn't mention if inferencing uses different hardware than the H200s used to train), whereas R3D2 renders 1080x1920 frames on a 5090. This would imply that R3D2 is processing 5.3x more pixels per frame on a different GPU.
  • Assuming the video model is using bidirectional attention (and not distilling into a causal auto-regressive model like CausVid [9]), the method can only generate a certain number of frames at a time. The paper mentions validating on 9-frame clips, but denoising in 9-frame chunks in more realistic settings could cause a lot of flickering at chunk boundaries. The supplemental videos don't seem to exhibit this, so it's unclear if the inferencing is actually denoising 9 frames at a time (and the flickering isn't happening), or if these videos are generated with different settings.
  • Fusing the 2D VAE features into the 3D VAE is one of the core technical contributions of the paper, but this is not properly ablated - the appendix does not show any metrics like warping error to measure flickering and FPS to quantify the overhead of the additional encoding. It's also unclear whether Table 2 measures full-image or asset-centric regions and if the other VAE variants are finetuned on the same data as the paper's method.
  • As another potentially core contribution of the paper, DriveInsert-3K's alignment strategy should also be ablated (how much does this alignment matter for training the model? The appendix implies that it helps R3D2's performance but that's a much more confounded experiment than running the paper's method with/without the alignment).

References:

  • [1] Yuxuan Zhang et al. DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer. CVPR 2026.
  • [2] Ruofan Liang et al. DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR 2025.
  • [3] Yuxi Wei et al. Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents. CVPR 2024.
  • [4] Yun Chen et al. GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving. CVPR 2021.
  • [5] Hao-Zhi Huang et al. Temporally Coherent Video Harmonization Using Adversarial Networks. TIP 2020.
  • [6] Xinyuan Lu et al. Deep Video Harmonization with Color Mapping Consistency. IJCAI 2022.
  • [7] Zhanghan Ke et al. Harmonizer: Learning to Perform White-Box Image and Video Harmonization. ECCV 2022.
  • [8] Zonghui Guo et al. Video Harmonization with Triplet Spatio-Temporal Variation Patterns. CVPR 2024.
  • [9] Tianwei Yin et al. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. CVPR 2025.
Questions for authors
  • Can you provide additional baselines in your experiments? DiffusionHarmonizer is the most essential, but more baselines can only strengthen the paper.
  • Can you clarify how your evaluation set is constructed, and provide results on different datasets than the one used to train the model, such as DiffusionHarmonizer's dataset (which would provide a fairer comparison for non-DiffusionHarmonizer baselines like R3D2), or R3D2's rotation, cross-scene, PandaSet and text-to-3D tests?
  • Can you confirm that the FPS comparison between this method and R3D2 is apples-to-apples, or provide a fairer comparison? The default non-big R3D2 variant should also be benchmarked, and per-frame latency, throughput, and peak memory should all be measured.
  • Can you similarly measure the speed/memory overhead of the 2D VAE fusion strategy, and clarify how the other variants are trained?
  • How are videos longer than 9 frames handled? What is the warp error for longer videos, including across chunk boundaries?
  • Can you report the full set of Table 1 metrics (including warp error) for the DriveInsert-3K-trained R3D2?
  • Can you ablate the importance of the 3D-to-2D alignment by training your method without it?
  • Can you explicitly commit to a public dataset release at this time?
  • In Demo-SequentialComparison.mp4, I observed that the method's results become more synthetic-looking as the inserted vehicles recede and become smaller. Could you provide some insight as to why that is the case?
Reviewer_7u1NOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a one-step video diffusion harmonizer for 3D asset insertions in driving scenes. It adapts CogVideoX with frame-wise 2D feature injection into its 3D VAE decoder and separate reconstruction and harmonization adapters. It also introduces DriveInsert-3K, containing 3,550 aligned rendering–real-video pairs. Experiments compare rendering quality and temporal consistency with R3D2 and evaluate the proposed VAE adaptation.

Critical strengths

The paper addresses a practical limitation of frame-wise asset harmonization. The video model improves actor-crop reconstruction and perceptual quality over R3D2, while reducing warping error from 1.119 to 0.874.

The decoder adaptation has useful experimental support. Adding VAE adaptation improves final-output PSNR from 27.89 to 28.60 and vFID from 0.206 to 0.178, whereas direct 3D skip connections degrade performance. The aligned dataset also benefits R3D2, suggesting value beyond the proposed model.

Critical weaknesses

The matched data comparison is incomplete. Retraining R3D2 on DriveInsert-3K raises its PSNR from 27.61 to 28.31, close to the proposed method’s 28.60. However, this matched-data baseline lacks quantitative temporal and actor-crop results, which are central to the contribution. The paper also does not specify evaluation-set size or whether training and evaluation are separated by driving sequence.

The evidence for geometry-preserving insertion remains limited. Quantitative evaluation focuses on reinsertion against the original video, while repositioning and replacement are demonstrated qualitatively. Reconstruction metrics do not establish that edited asset shape, pose, and identity survive harmonization. Evaluation is also restricted to nine-frame clips, leaving longer-term consistency untested.

The related-work discussion could better cover controllable driving-scene rendering, including FlexDrive and ReCamDriving. Discussing their approaches to trajectory flexibility and geometry-guided synthesis would help clarify where the proposed asset harmonization method fits within this broader literature.

Questions for authors

Can you specify the evaluation-set size and sequence-level split, and report all Table 1 metrics for R3D2 retrained on DriveInsert-3K?

Can you quantify asset pose, shape, and identity preservation under repositioning or replacement, and evaluate temporal consistency beyond nine-frame clips?

Reviewer_GB6mOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses photorealistic rendering of 3D assets inserted into reconstructed driving scenes. Given a "naive" insertion video rendered by a geometry-based pipeline, it fine-tunes CogVideoX-1.5 with LoRA in to a one-step video harmonizer (fixed timestep t=199, fixed prompt, LPIPS + Gram loss). To recover spatial detail lost by the causal 3D VAE, it injects per-frame features from a pretrained 2D VAE encoder into the late, frame-aligned layers of the 3D VAE decoder ("temporally agnostic latent injection"), trained in a separate VAE-adaptation stage. The authors also build DriveInsert-3K, a set of 3,550 paired clips from Waymo. Each pair replaces a real vehicle with a generated asset, aligned by a 3D similarity transform followed by a global 2D bounding-box-based affine correction. Validation consists of a comparison against naive rende ring and R3D2 on PSNR/SSIM/LPIPS/DISTS/warp error/vFID (actor-centric crops and full frames), qualitative figures, and ablations that appear only in the appendix.

Critical strengths
  1. A practically relevant problem with a useful paired dataset. Harmonizing inserted assets is a real bottleneck for controllable driving simulation. The proposed 3D-to-2D alignment yields cleaner naive/real p airs than prior reinsertion data: Appendix Table 3 shows that it improves even R3D2 (PSNR 27.61 → 28.31, vFID 0.206 → 0.193). If released, DriveInsert-3K could benefit the community.
  2. Careful visual presentation. The figures are well made and the qualitative examples (repositioning, 180° reversal, over-exposed scenes, diverse vehicle types) clearly communicate the intended use case.
Critical weaknesses
  1. Limited technical novelty and insight. The method is a straightforward adaptation of existing one-step diffusion fixing to video. The single-step recipe and losses follow Difix3D+ [1]; the paper itself states that it follows [1] for the fixed timestep t=199. One-step restoration by fine-tuning a pretrained CogVideoX model was already explored in DOVE [2]. The paper does not analyze the limitations of existing approaches for this task beyond frame-wise flickering, and I did not gain new understanding of the problem from it.

  2. Insufficient and unfair main comparison. Table 1 compares against a single baseline, R3D2. Its numbers (27.61 / 0.072 / 0.206) correspond to R3D2 not trained on DriveInsert-3K (Appendix Table 3). When R3D2 is trained on the same data, it reaches 28.31 / 0.072 / 0.193, close to the proposed method (28.60 / 0.069 / 0.178). The gain attributable to the method itself is therefore small. In addition, the main text contains no ablation study; all ablations are in the appendix.

References

[1] J. Z. Wu et al. Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models. CVPR 2025. arXiv:2503.01774.

[2] Z. Chen et al. DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution. NeurIPS 2025. arXiv:2505.16239.

Questions for authors
  1. Please report the full Table 1 with R3D2 trained on DriveInsert-3K. With the training data controlled, what is the gain attributable to the method itself?
  2. Beyond adopting the one-step recipe of Difix3D+ and a pretrained video model as in DOVE, what limitation specific to asset-insertion harmonization does the method address, and what evidence shows that the proposed latent injection is what resolves it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.