- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This submission looks at photorealistic 3D asset insertion in the context of driving simulators. Although being able to insert, replace, and manipulate 3D assets such as cars into a scene unlocks many valuable simulation use cases, it can be challenging to do so in a photorealistic manner (naively inserted assets will have different lighting than the rest of the scene, the materials don't fully match, etc). R3D2 [1] uses a 1-step image diffusion model to harmonize inserted assets with the rest of the scene, but suffers from temporal consistency issues such as flickering and appearance variations since the diffusion model denoises each frame independently. This submission instead proposes to LoRA-finetune a video diffusion model (CogVideoX) into a single-step harmonizer.
One challenge is that video models use highly compressed VAEs (CogVideoX's does 8x spatial and 4x temporal compression) that can cause renderings to lose fine detail and alter the original scene structure. As a solution, the paper proposes a "temporally agnostic latent injection" strategy where each frame is also independently encoded with a separate 2D VAE encoder. The resulting intermediate features are cached and merged into the 3D VAE decoder's last two upsampling blocks via "cross-modal fusion blocks." The authors train the fusion blocks and LoRA-finetune the 3D VAE decoder to handle this additional 2D signal with reconstruction loss ("Reconstruction-LoRA") before finetuning the rest of the video model.
The authors then LoRA-finetune the diffusion transformer and the VAE decoder with a second LoRA ("Harmonization-LoRA") with pixel-space losses (LPIPS loss and Gram). To supervise training, the authors curate a paired video dataset (DriveInsert-3K) from Waymo Open Dataset videos. Each pair consists of a clip with a naively inserted asset (the authors take care to geometrically align the asset as much as possible with its real-world counterpart) and the original real video.
The method is evaluated on a validation set that is not clearly defined (my guess is held-out scenes from DriveInsert-3K) against a "naive insertion" baseline and R3D2. The authors' method provides the best PSNR, LPIPS, DISTS, and vFID scores on full-resolution frames, although the PSNR gap reduces notably when R3D2 is retrained on DriveInsert-3K (these findings are relegated to the appendix). When evaluating actor-centric crops, the method has slightly worse vFID than the original R3D2. Temporal consistency is measured via flow-warping error, where the method is worse than naive insertion but better than R3D2 in both full-resolution videos and asset crops. The authors also share videos in the supplement where their method exhibits nicer results than the baselines (and less flickering than R3D2) along with scene editing capabilities (such as adjusting the orientation of a car).
References:
- [1] William Ljungbergh et al. R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation. CVPRW 2026
Critical strengths
I agree with the premise that asset manipulation makes driving simulators far more useful, and that naive asset insertion is a challenge. I also agree that diffusion models are a promising solution and that temporal consistency is important. An efficient video diffusion method seems like a natural approach.
If released, DriveInsert-3K could also be a valuable asset to the community if properly contextualized relative to prior datasets such as that released by DiffusionHarmonizer [1] and the R3D2 dataset (R3D3) that is also derived from Waymo. The authors should explicitly confirm that they can commit to the release if the submission is accepted, as this would be a big factor in potentially recommending acceptance depending on the resolution of the concerns below.
Last but not least, I found this paper easy to read and I very much appreciate that the authors did not over-complicate things, which I do not take for granted.
References:
- [1] Yuxuan Zhang et al. DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer. CVPR 2026.
Critical weaknesses
I am concerned about this paper's positioning and novelty relative to prior work, especially DiffusionHarmonizer [1] (CVPR 2026) which is highly relevant and not cited. DiffusionHarmonizer addresses a near-identical problem (temporally consistent harmonization of inserted assets), has public code and model checkpoints, and is an essential baseline to include in the paper to merit acceptance. The set of baselines used for experiments (just R3D2 and naive insertion in the main paper) is generally very thin - DiffusionRenderer [2], ChatSim [3], and GeoSim [4] all handle object insertion, and the first two have public code (the paper cites none of these). More broadly, there are many video harmonization works such as Huang et al. [5], Lu et al. [6], Harmonizer [7], and VHTT [8] that a video harmonizer-themed submission should be citing.
I also have serious reservations about the experimental protocol beyond the inadequate number of baselines:
- How the evaluation set is constructed is not explicit in the paper. This should be explicitly disclosed and the authors should confirm that there is no leakage of scenes between the training and evaluation sets.
- It is implied that it comes from the same DriveInsert-3K dataset used to train the method. From that perspective, having the main paper experiments compare to an R3D2 checkpoint trained on a different dataset is unfair. The appendix shows how the gap narrows when R3D2 is trained on the same data as the paper's method - at the very least, this stronger baseline should be what's included in the paper.
- The evaluation section would be much stronger if evaluated on additional datasets such as the released DiffusionHarmonizer dataset to show proof of generalization.
- A big reason why prior work uses image diffusion models instead of video models is because the latter are much slower and memory hungry. From this perspective, the claim that this paper's method renders at higher FPS than the R3D2 image-based pipeline is surprising, even when accounting for the higher temporal compression factor of the 3D VAE (both methods are single-step). The R3D2 number matches the R3D2-BIG configuration of their paper's Table 1 (the smaller variant clocks 13.36 FPS compared to this paper's 4.54), which raises the question of whether the timing protocols are matched. This paper uses 512x768 frames for training and validation (and doesn't mention if inferencing uses different hardware than the H200s used to train), whereas R3D2 renders 1080x1920 frames on a 5090. This would imply that R3D2 is processing 5.3x more pixels per frame on a different GPU.
- Assuming the video model is using bidirectional attention (and not distilling into a causal auto-regressive model like CausVid [9]), the method can only generate a certain number of frames at a time. The paper mentions validating on 9-frame clips, but denoising in 9-frame chunks in more realistic settings could cause a lot of flickering at chunk boundaries. The supplemental videos don't seem to exhibit this, so it's unclear if the inferencing is actually denoising 9 frames at a time (and the flickering isn't happening), or if these videos are generated with different settings.
- Fusing the 2D VAE features into the 3D VAE is one of the core technical contributions of the paper, but this is not properly ablated - the appendix does not show any metrics like warping error to measure flickering and FPS to quantify the overhead of the additional encoding. It's also unclear whether Table 2 measures full-image or asset-centric regions and if the other VAE variants are finetuned on the same data as the paper's method.
- As another potentially core contribution of the paper, DriveInsert-3K's alignment strategy should also be ablated (how much does this alignment matter for training the model? The appendix implies that it helps R3D2's performance but that's a much more confounded experiment than running the paper's method with/without the alignment).
References:
- [1] Yuxuan Zhang et al. DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer. CVPR 2026.
- [2] Ruofan Liang et al. DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR 2025.
- [3] Yuxi Wei et al. Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents. CVPR 2024.
- [4] Yun Chen et al. GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving. CVPR 2021.
- [5] Hao-Zhi Huang et al. Temporally Coherent Video Harmonization Using Adversarial Networks. TIP 2020.
- [6] Xinyuan Lu et al. Deep Video Harmonization with Color Mapping Consistency. IJCAI 2022.
- [7] Zhanghan Ke et al. Harmonizer: Learning to Perform White-Box Image and Video Harmonization. ECCV 2022.
- [8] Zonghui Guo et al. Video Harmonization with Triplet Spatio-Temporal Variation Patterns. CVPR 2024.
- [9] Tianwei Yin et al. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. CVPR 2025.
Questions for authors
- Can you provide additional baselines in your experiments? DiffusionHarmonizer is the most essential, but more baselines can only strengthen the paper.
- Can you clarify how your evaluation set is constructed, and provide results on different datasets than the one used to train the model, such as DiffusionHarmonizer's dataset (which would provide a fairer comparison for non-DiffusionHarmonizer baselines like R3D2), or R3D2's rotation, cross-scene, PandaSet and text-to-3D tests?
- Can you confirm that the FPS comparison between this method and R3D2 is apples-to-apples, or provide a fairer comparison? The default non-big R3D2 variant should also be benchmarked, and per-frame latency, throughput, and peak memory should all be measured.
- Can you similarly measure the speed/memory overhead of the 2D VAE fusion strategy, and clarify how the other variants are trained?
- How are videos longer than 9 frames handled? What is the warp error for longer videos, including across chunk boundaries?
- Can you report the full set of Table 1 metrics (including warp error) for the DriveInsert-3K-trained R3D2?
- Can you ablate the importance of the 3D-to-2D alignment by training your method without it?
- Can you explicitly commit to a public dataset release at this time?
- In Demo-SequentialComparison.mp4, I observed that the method's results become more synthetic-looking as the inserted vehicles recede and become smaller. Could you provide some insight as to why that is the case?