- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PIH, a physical-intrinsic harmonization framework for improving the visual quality of editable neural reconstructions of driving scenes. Unlike conventional RGB-based harmonization approaches, PIH represents scenes using physical attributes, including surface normals, depth, albedo, roughness, and illumination. This representation supports compositional scene editing, including virtual-object insertion and lighting modifications, without requiring insertion-specific paired RGB training examples.
The authors develop a single-step generative harmonizer using latent flow matching and perceptual endpoint supervision. Temporal coherence is addressed through training with a mixture of RGB-grounded and self-generated histories.
Experiments on Waymo, nuPlan, and PandaSet demonstrate improvements in perceptual realism and reconstruction quality, particularly on Waymo. A human preference study reports that PIH is preferred over DiffusionHarmonizer in 77.41% of comparisons. The paper additionally demonstrates zero-shot object insertion and illumination control.
Critical strengths
-
A meaningful reformulation of neural reconstruction harmonization using physical scene attributes. The proposed physical-intrinsic representation provides a compelling alternative to directly correcting artifacts in RGB space. By explicitly conditioning generation on geometry, material properties, and illumination, the method enables different scene-editing operations to be handled through a common representation rather than requiring separate paired correction datasets. The ability to perform virtual-asset insertion without insertion-specific training pairs is particularly valuable for scalable driving-scene simulation. This represents a practically meaningful contribution beyond conventional image enhancement.
-
Strong perceptual improvements supported by multiple evaluation settings. The paper evaluates original and novel camera trajectories, with and without virtual-object insertion, across three driving datasets. The results show substantial improvements in several perceptual metrics, especially on Waymo, and the human preference study provides additional evidence that the outputs are visually convincing. The combination of latent flow matching, perceptual endpoint supervision, and mixed temporal histories is also supported by targeted ablations. These results demonstrate that the proposed components contribute to reconstruction quality rather than serving merely as architectural additions.
Critical weaknesses
-
The evaluation does not sufficiently establish physical and temporal consistency under scene editing. Although PIH is motivated by physically grounded scene representation, the experimental evaluation primarily measures perceptual realism and feature-level structural similarity. These metrics do not directly establish whether inserted objects preserve their intended geometry, position, motion, or illumination consistency across frames. This distinction is particularly important for driving simulation, where visually plausible but geometrically inconsistent observations could compromise the validity of downstream simulation experiments. Furthermore, temporal generalization is not uniformly supported across datasets. On novel trajectories, PIH achieves worse FVD than Difix3D on both nuPlan (824.59 versus 614.08) and PandaSet (705.82 versus 542.85). The human preference evaluation is based on sampled individual frames rather than video-level temporal judgments. Consequently, the evidence for robust temporal consistency remains weaker than the paper's motivation would require.
-
The practical efficiency advantage for online simulation is not fully demonstrated. The paper emphasizes single-step generation as a key requirement for online simulation. However, the reported inference time is approximately 0.6 seconds per frame on a single H100 GPU, corresponding to roughly 1.7 frames per second. Although this reduces the number of generative steps, it does not establish suitability for real-time driving simulation. Moreover, the paper does not provide a sufficiently detailed end-to-end latency comparison against the strongest relevant harmonization baselines under matched hardware and resolution settings. Since efficient online simulation is a central motivation, the distinction between single-step generation and actual deployment throughput is consequential to the overall assessment.
Questions for authors
-
Could the authors provide a quantitative evaluation of edit fidelity and temporal consistency, particularly for inserted vehicles? For example, measuring object displacement, segmentation consistency, or geometric alignment across consecutive frames would help establish whether the model preserves the simulator-defined scene rather than merely producing perceptually plausible outputs. This would also help explain the relatively weak cross-dataset FVD results under novel trajectories.
-
How does the complete inference latency of PIH compare with DiffusionHarmonizer and Difix3D under identical hardware and resolution settings? Given the reported runtime of approximately 0.6 seconds per frame on an H100, what simulation throughput is practically achievable, and which components dominate the computational cost?
6 comments
W1.1 Editing Fidelity (Q1)
Following HorizonForge, we add Vehicle Identity Matching Score (VIMS) and Background Alignment Score (BAS) to evaluate whether the inserted vehicle retains its visual identity and whether the surrounding scene remains consistent after harmonization (Tab. 2).
| Method | VIMS ↑ | BAS ↑ |
|---|---|---|
| DiffusionHarmonizer | 0.9665 | 0.9141 |
| Difix3D+ | 0.9676 | 0.9290 |
| PIH (Ours) | 0.9698 | 0.9491 |
PIH achieves the best results on both metrics, particularly in background preservation (0.9491 vs. 0.9290 for Difix3D+ and 0.9141 for DiffusionHarmonizer).
Importantly, vehicle insertion is performed in the underlying 3D scene: the asset geometry, placement, and motion are specified before rendering the physical-intrinsic conditions. The harmonizer then generates the corresponding RGB observation. This separates the simulator-defined scene configuration from the generative appearance refinement process.
While VIMS and BAS do not directly measure pixel-accurate geometric alignment, they provide additional quantitative evidence that PIH preserves inserted-object identity and background content rather than merely improving perceptual realism.
W1.2 Temporal Consistency (Q1)
We further conduct VBench evaluations across three datasets and four editing settings, including original trajectories, vehicle insertion, shifted trajectories, and their combination (Sec. 4.4, Tab. 4).
We evaluate three temporal dimensions: Subject Consistency, Background Consistency, and Motion Smoothness.
| Method | Subject ↑ | Background ↑ | Motion ↑ | Mean ↑ |
|---|---|---|---|---|
| Original 3DGS | 89.41 | 93.74 | 98.46 | 93.87 |
| Difix3D+ | 89.18 | 92.33 | 97.42 | 92.97 |
| DiffusionHarmonizer | 89.93 | 93.42 | 98.75 | 94.03 |
| PIH (Ours) | 90.33 | 93.12 | 98.17 | 93.87 |
PIH achieves the highest Subject Consistency, while maintaining overall temporal consistency close to DiffusionHarmonizer (93.87 vs. 94.03). Moreover, PIH consistently outperforms Difix3D+ across all three temporal dimensions.
These results are particularly relevant to the reviewer's concern that strong single-frame realism may not translate into stable video generation.
To investigate the specific novel-trajectory settings, we separately examine the OOD Shift and Insert+Shift tasks. Averaged equally across nuPlan and PandaSet, PIH achieves higher Subject Consistency than DiffusionHarmonizer (89.84 vs. 89.39), while maintaining a comparable mean temporal score (93.52 vs. 93.65).
Summary
We have added VIMS/BAS for editing fidelity (Tab. 2) and VBench evaluations across three datasets and four editing settings (Tab. 4) to assess temporal consistency. We also provide detailed H100 component-level profiling (Appendix F.1, Tabs. 14–15), demonstrating a 1.60× latency reduction and identifying the remaining computational bottlenecks.