acceptodds
Under review as a conference paper at ICLR 2027

PIH: Physical-intrinsic Harmonizer for Editable Neural Reconstruction

Abstract

Editable neural driving scenes are essential for scalable simulation, yet neural reconstructions often produce severe artifacts under novel viewpoints and virtual-asset insertion. Existing harmonizers operating in RGB space learn to correct these heterogeneous failure patterns, making extension to new editing operations data-intensive. We reformulate the task as physical-intrinsic harmonization, using imperfect depth, normals, albedo, roughness, and illumination derived from the reconstructed scene as a compositional interface. This interface enables test-time edits by composing physical factors, eliminating the need for insertion-specific paired RGB correction data. To make this formulation practical for online simulation, we propose PIH, a one-step harmonizer trained directly with latent flow matching on the predicted velocity and perceptual supervision on its decoded clean endpoint. Mixed RGB-grounded and self-rollout histories provide temporal context during training to ensure temporal consistency and mitigate exposure bias. Human preference experiments across Waymo, nuPlan, and PandaSet show that PIH is preferred to the prior method in 77.41% of comparisons while using fewer paired training frames. PIH additionally enables illumination control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Generative Models, Neural Rendering, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_iCdROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PIH, a physical-intrinsic harmonization framework for improving the visual quality of editable neural reconstructions of driving scenes. Unlike conventional RGB-based harmonization approaches, PIH represents scenes using physical attributes, including surface normals, depth, albedo, roughness, and illumination. This representation supports compositional scene editing, including virtual-object insertion and lighting modifications, without requiring insertion-specific paired RGB training examples.

The authors develop a single-step generative harmonizer using latent flow matching and perceptual endpoint supervision. Temporal coherence is addressed through training with a mixture of RGB-grounded and self-generated histories.

Experiments on Waymo, nuPlan, and PandaSet demonstrate improvements in perceptual realism and reconstruction quality, particularly on Waymo. A human preference study reports that PIH is preferred over DiffusionHarmonizer in 77.41% of comparisons. The paper additionally demonstrates zero-shot object insertion and illumination control.

Critical strengths
  1. A meaningful reformulation of neural reconstruction harmonization using physical scene attributes. The proposed physical-intrinsic representation provides a compelling alternative to directly correcting artifacts in RGB space. By explicitly conditioning generation on geometry, material properties, and illumination, the method enables different scene-editing operations to be handled through a common representation rather than requiring separate paired correction datasets. The ability to perform virtual-asset insertion without insertion-specific training pairs is particularly valuable for scalable driving-scene simulation. This represents a practically meaningful contribution beyond conventional image enhancement.

  2. Strong perceptual improvements supported by multiple evaluation settings. The paper evaluates original and novel camera trajectories, with and without virtual-object insertion, across three driving datasets. The results show substantial improvements in several perceptual metrics, especially on Waymo, and the human preference study provides additional evidence that the outputs are visually convincing. The combination of latent flow matching, perceptual endpoint supervision, and mixed temporal histories is also supported by targeted ablations. These results demonstrate that the proposed components contribute to reconstruction quality rather than serving merely as architectural additions.

Critical weaknesses
  1. The evaluation does not sufficiently establish physical and temporal consistency under scene editing. Although PIH is motivated by physically grounded scene representation, the experimental evaluation primarily measures perceptual realism and feature-level structural similarity. These metrics do not directly establish whether inserted objects preserve their intended geometry, position, motion, or illumination consistency across frames. This distinction is particularly important for driving simulation, where visually plausible but geometrically inconsistent observations could compromise the validity of downstream simulation experiments. Furthermore, temporal generalization is not uniformly supported across datasets. On novel trajectories, PIH achieves worse FVD than Difix3D on both nuPlan (824.59 versus 614.08) and PandaSet (705.82 versus 542.85). The human preference evaluation is based on sampled individual frames rather than video-level temporal judgments. Consequently, the evidence for robust temporal consistency remains weaker than the paper's motivation would require.

  2. The practical efficiency advantage for online simulation is not fully demonstrated. The paper emphasizes single-step generation as a key requirement for online simulation. However, the reported inference time is approximately 0.6 seconds per frame on a single H100 GPU, corresponding to roughly 1.7 frames per second. Although this reduces the number of generative steps, it does not establish suitability for real-time driving simulation. Moreover, the paper does not provide a sufficiently detailed end-to-end latency comparison against the strongest relevant harmonization baselines under matched hardware and resolution settings. Since efficient online simulation is a central motivation, the distinction between single-step generation and actual deployment throughput is consequential to the overall assessment.

Questions for authors
  1. Could the authors provide a quantitative evaluation of edit fidelity and temporal consistency, particularly for inserted vehicles? For example, measuring object displacement, segmentation consistency, or geometric alignment across consecutive frames would help establish whether the model preserves the simulator-defined scene rather than merely producing perceptually plausible outputs. This would also help explain the relatively weak cross-dataset FVD results under novel trajectories.

  2. How does the complete inference latency of PIH compare with DiffusionHarmonizer and Difix3D under identical hardware and resolution settings? Given the reported runtime of approximately 0.6 seconds per frame on an H100, what simulation throughput is practically achievable, and which components dominate the computational cost?

6 comments

We have added VIMS/BAS for editing fidelity (Tab. 2) and VBench evaluations across three datasets and four editing settings (Tab. 4) to assess temporal consistency. We also provide detailed H100 component-level profiling (Appendix F.1, Tabs. 14–15), demonstrating a 1.60× latency reduction and identifying the remaining computational bottlenecks.

AuthorsOpenReview ↗

W1.1 Editing Fidelity (Q1)

Following HorizonForge, we add Vehicle Identity Matching Score (VIMS) and Background Alignment Score (BAS) to evaluate whether the inserted vehicle retains its visual identity and whether the surrounding scene remains consistent after harmonization (Tab. 2).

MethodVIMS ↑BAS ↑
DiffusionHarmonizer0.96650.9141
Difix3D+0.96760.9290
PIH (Ours)0.96980.9491

PIH achieves the best results on both metrics, particularly in background preservation (0.9491 vs. 0.9290 for Difix3D+ and 0.9141 for DiffusionHarmonizer).

Importantly, vehicle insertion is performed in the underlying 3D scene: the asset geometry, placement, and motion are specified before rendering the physical-intrinsic conditions. The harmonizer then generates the corresponding RGB observation. This separates the simulator-defined scene configuration from the generative appearance refinement process.

While VIMS and BAS do not directly measure pixel-accurate geometric alignment, they provide additional quantitative evidence that PIH preserves inserted-object identity and background content rather than merely improving perceptual realism.

AuthorsOpenReview ↗

W1.2 Temporal Consistency (Q1)

We further conduct VBench evaluations across three datasets and four editing settings, including original trajectories, vehicle insertion, shifted trajectories, and their combination (Sec. 4.4, Tab. 4).

We evaluate three temporal dimensions: Subject Consistency, Background Consistency, and Motion Smoothness.

MethodSubject ↑Background ↑Motion ↑Mean ↑
Original 3DGS89.4193.7498.4693.87
Difix3D+89.1892.3397.4292.97
DiffusionHarmonizer89.9393.4298.7594.03
PIH (Ours)90.3393.1298.1793.87

PIH achieves the highest Subject Consistency, while maintaining overall temporal consistency close to DiffusionHarmonizer (93.87 vs. 94.03). Moreover, PIH consistently outperforms Difix3D+ across all three temporal dimensions.

These results are particularly relevant to the reviewer's concern that strong single-frame realism may not translate into stable video generation.

To investigate the specific novel-trajectory settings, we separately examine the OOD Shift and Insert+Shift tasks. Averaged equally across nuPlan and PandaSet, PIH achieves higher Subject Consistency than DiffusionHarmonizer (89.84 vs. 89.39), while maintaining a comparable mean temporal score (93.52 vs. 93.65).

Reviewer_3H22OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PIH, a one-step neural harmonizer for editable driving scenes. Instead of directly enhancing RGB renderings, PIH conditions a video diffusion model on imperfect G-buffers extracted from the inverse rendering 3D reconstruction, an HDR environment map, and previous frames. The method first trains the newly added conditioning layers are first optimized with latent flow matching, then conducts one-step training that combines latent flow matching with perceptual endpoint losses. During training, temporal histories are drawn from both 3DGS RGB renderings and the model’s own rollouts. The authors claim that PIH enables zero-shot virtual asset insertion and illumination control without insertion-specific paired training data. Experiments demonstrate improvements over DiffusionHarmonizer, and favorable results in user preferences.

Critical strengths

1. Well-motivated editing interface. Using physical scene properties as an interface for harmonization is a promising direction. It could be a unified way to support asset insertion and illumination control, capabilities that RGB-only harmonizers do not directly offer.

2. Strong results. PIH outperforms all baselines across the metrics reported in Tables 1 and 2. Both the VLM-based and human evaluations also favor PIH over DiffusionHarmonizer.

3. Informative analysis of design choices. Figure 6 provides useful insights into why existing approaches to single-step rendering, including endpoint regression, albedo initialization, and Diffusion Forcing, struggle with imperfect intrinsic estimates.

Critical weaknesses

1. Ambiguity on the source of the improvement

The core claim is that the proposed screen-space intrinsic interface, rather than RGB, makes harmonization effective, but this factor is not isolated in the evaluations.

  • 1.1) Missing controlled baseline: The baselines use their official checkpoints (Appendix E.1), whereas PIH is trained on the same domain as the WOD test set. Consistently, the gap is large on WOD but mostly disappears or reverses on nuPlan/PandaSet (Table 3), where Difix3D [2] is better in LPIPS and FVD in most settings. Rather than retraining the baselines, an RGB-input variant of PIH with the same pipeline, data, and recipe would directly support whether the gains come from the interface or from domain-matched training.
  • 1.2) Role of each G-buffer: In Appendix F.2, the authors claim that "the model is not sensitive to normal and roughness," suggesting that the model may mostly rely on the albedo, which likely contains baked shading (Fig. 6(c)), and the RGB history. This reviewer would like to see a per-buffer ablation on the evaluation.

Overall, it is a little bit unclear whether the gains come from the proposed formulation or from domain-matched training and RGB anchoring.

2. Modest technical novelty

The pipeline is largely assembled from existing work: InvRGB+L [3] for reconstruction, LuxDiT [4] for lighting estimation, DiffusionRenderer [5] for the network and the pseudo-G-buffer data, and DiffusionHarmonizer [1] for the task setting and evaluation protocol.

  • The network largely follows the forward renderer of DiffusionRenderer, which is trained on real videos with estimated G-buffers (Sec. 4.1).
  • Albedo-guided insertion with a VAE-concatenated video DiT that harmonizes shading and shadows was explored in AlbedoEdit [6], and ZeroComp [7] for images.
  • Single-step enhancement of reconstruction artifacts was shown in Difix3D+ [2] and FixAnything [10]. The one-step objective is PFM [8] plus a latent flow-matching term, where the benefits seems a little marginal (LPIPS/FID 0.3104/14.82 vs. 0.3134/14.79 without it, Table 6). Self-rollout training is essentially a concept from the Self Forcing [9].

3. Insufficient support for the claims and evaluation

  • Insertion: The inserted vehicle uses its RGB texture as albedo. This conflicts with the authors' observation where baked illumination in albedo harms robustness to lighting changes (Fig. 6(c)).
  • Illumination control: Only qualitative results are shown (Fig. 4). A quantitative evaluation is possible, e.g., on synthetic scenes, or by re-estimating the lighting from the outputs with LuxDiT.
  • Missing baselines: Difix3D+ is missing from Table 2 and the preference evaluation, although it outperforms PIH on several metrics in Table 3; a comparison with FixAnything [10] would also be informative, as it is an enhanced version of Difix3D+ and the code is already available. Plus, the temporal consistency is not assessed in the preference evaluation.
  • Generalization: While the target task is focusing on driving simulations, this reviewer feels the problem formulation itself is not specific to driving ("Editable Neural Reconstruction"). Results or a discussion on other scenes (e.g., Mip-NeRF 360, DL3DV, or in-the-wild videos with estimated G-buffers) would further support the validity of proposed method.
  • Others: No ablation without the DiffusionRenderer pseudo-G-buffer pairs is given for the data-efficiency claim, and the runtime (0.6 s/frame on an H100) seems too slow for the claimed online simulation.

[1] Zhang et al., DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer, CVPR 2026

[2] Wu et al., Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models, CVPR 2025

[3] Chen et al., InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modeling, ICCV 2025

[4] Liang et al., LuxDiT: Lighting Estimation with Video Diffusion Transformer, NeurIPS 2025

[5] Liang et al., DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models, CVPR 2025

[6] Zhou et al., AlbedoEdit: Unified Instance-Level Video Editing with Albedo Guidance, arXiv 2026

[7] Zhang et al., ZeroComp: Zero-Shot Object Compositing from Image Intrinsics via Diffusion, WACV 2025

[8] Zhao et al., Perceptual Flow Matching for Few-Step Generative Modeling, arXiv 2026

[9] Huang et al., Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, NeurIPS 2025

[10] Vuong et al., FixAnything: 3D-Consistent Rendering Refinement via Video Generative Priors, ECCV 2026

Questions for authors
  1. What does the RGB history contain in the insertion and relighting experiments (Figs. 3 and 4, Table 2)? Does it consist of 3DGS renderings without the inserted asset and under the original illumination? How is the history initialized for the first K frames in the self-rollout setting?

  2. What do the reference-based metrics (LPIPS) in Table 2 evaluate, given that the captured reference frames do not contain the inserted vehicles?

  3. How will an RGB-input variant of PIH trained with the same pipeline, data, and training recipe, replacing the G-buffers with the 3DGS RGB rendering work? This will better disentangle the benefits of using intrinsic cues than RGB renderings (Weaknesses 1.1 and 1.2).

  4. An ablation that removes or corrupts each G-buffers individually would be helpful for understanding the role of each modality (Weakness 1.3).

  5. A quantitative evaluation of illumination control (Weakness 3).

  6. Comparisons with Difix3D and FixAnything in the insertion setting and preference evaluations (Weakness 3).

12 comments

We have strengthened the paper with controlled RGB and G-buffer ablations (Tabs. 6–7), additional insertion baselines and editing-fidelity metrics (Tab. 2), temporal evaluations (Tab. 4), and detailed inference profiling (Appendix F.1, Tabs. 14–15). We further clarify our technical distinctions from prior work (Appendix C) and the scope of our claims regarding data efficiency, relighting, and runtime.

AuthorsOpenReview ↗

W1.1 Matched RGB Conditioning (Q3)

Following the reviewer's suggestion, we train an RGB-conditioned counterpart under the same setting. We conduct this comparison with both Cosmos-Predict2.5 and U-Net (Fig. 5 and Tab. 6).

Interestingly, the results reveal a distinction between reconstruction fidelity and perceptual harmonization under scene editing.

On original trajectories, PIH achieves slightly better reconstruction metrics than RGB conditioning. However, under vehicle insertion, RGB conditioning obtains better reference-based metrics, despite producing less convincing vehicle–scene harmonization in the qualitative comparisons.

To investigate this discrepancy, we additionally evaluate HarmonyIQA[1], its vehicle-focused crop variant, TOPIQ-NR[2], and video-level human preference. PIH consistently achieves higher no-reference perceptual quality scores across both backbones. More importantly, 69.3% of non-tie human judgments prefer PIH over RGB conditioning when viewing the corresponding videos.

This observation is also supported by Fig. 5, where RGB-conditioned outputs can exhibit locally inconsistent vehicle appearance and illumination, despite their better reference reconstruction metrics.

We emphasize that this does not invalidate reconstruction metrics, which remain useful for assessing fidelity. Rather, these results suggest that reconstruction fidelity alone is insufficient to characterize the perceptual quality of counterfactual edits, where an exact captured counterpart may be unavailable.

Crucially, these matched experiments directly isolate the effect of the conditioning representation from backbone and training-data differences. Together with the qualitative and human evaluations, these findings support the value of physical-intrinsic conditioning as a compositional interface for editable neural reconstruction. We provide a detailed discussion of this trade-off in Sec. 4.5.

[1]Harmonyiqa: Pioneering benchmark and model for image harmonization quality assessment ICME'25

[2] Topiq: A top-down approach from semantics to distortions for image quality assessment TIP'24

AuthorsOpenReview ↗

W1.2 Contribution of Individual G-buffers (Q4)

We conduct a leave-one-channel-out ablation using the U-Net on Waymo vehicle insertion (Tab. 7).

InputLPIPS ↓FID ↓DINO-S ↓BAS ↑
Full G-buffer0.363129.802.3970.9230
w/o Depth0.369832.002.4380.9185
w/o Normal0.370330.352.4400.9189
w/o Roughness0.402935.022.8000.8975

Removing roughness causes the largest degradation (+10.96% LPIPS, +17.52% FID, −2.77% BAS), while removing depth or normals also reduces reconstruction fidelity and background preservation, although to a smaller extent.

These results directly address our earlier insufficiently qualified observation regarding channel sensitivity. They demonstrate that the network uses information beyond albedo, rather than simply ignoring the additional physical channels.

Reviewer_vij4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • PIH addresses harmonization in editable neural reconstructions of driving scenes. Rather than correcting an imperfect RGB rendering from 3D Gaussian Splatting, it generates images conditioned on the reconstruction’s G-buffers—normals, depth, albedo, and roughness—together with an HDR environment map. Vehicle insertion and relighting are expressed as changes to these inputs, without requiring paired training data specific to each edit.
  • The model builds on Cosmos-Predict2.5, with expanded patch embeddings for the G-buffer latents and a U-Net environment-map encoder connected through cross-attention.
  • Training has two stages. The first trains the new layers using latent flow matching. The second updates all parameters using latent flow matching together with LPIPS/P-DINO losses on the decoded endpoint. Training batches mix individual images, histories grounded in 3DGS RGB renders, and self-rollout histories.
  • The training set contains 200 WOD scenes and fewer than 100k pairs, including pseudo-G-buffers produced by DiffusionRenderer. Evaluation covers 55 WOD, 39 PandaSet, and 6 nuPlan scenes, using both original trajectories and trajectories shifted by 3 m, with and without zero-shot vehicle insertion.
  • The strongest quantitative results are on WOD: on the original trajectory, FID improves from 19.87 to 9.12 and FVD from 261 to 157 compared with DiffusionHarmonizer. Results on nuPlan and PandaSet are more mixed. PIH is preferred over DiffusionHarmonizer in 77.41% of human judgments. Relighting is evaluated qualitatively.
Critical strengths
  • Conditioning on intrinsic buffers gives insertion and relighting a common representation: an edit changes the geometry, appearance buffers, or lighting supplied to the model. This seems useful for neural simulators and avoids collecting new paired data for each edit type. The supplementary examples support the practical potential, including stable 300-frame nuPlan rollouts and plausible night relighting with emissive tail-lights.
  • I also found the experiments supporting the one-step design informative. The NFE study in Figure 5 and the objective ablations in Tables 6–7 help explain the choices between latent and pixel-space flow matching, endpoint prediction, and LPIPS versus P-DINO.
Critical weaknesses
  • The experiments do not establish how much of the improvement comes from the intrinsic representation. The key missing comparison is an RGB-conditioned version trained with the same backbone, data, history scheme, and one-step training procedure. Without that control, it is difficult to separate the value of the proposed interface from the other changes introduced by PIH. The training conditions also differ substantially across methods. The baselines use official checkpoints trained on other degradations, whereas PIH is trained on the degradation used at test time: front-camera InvRGB+L reconstructions stopped at 5k iterations. PIH additionally benefits from Cosmos, DiffusionRenderer pseudo-G-buffers, LuxDiT, and LiDAR-aided InvRGB+L. These are reasonable components to use, but they make the claim about requiring fewer than one-third of the paired frames difficult to interpret as a controlled comparison of data efficiency. I am also unsure how strongly the model relies on the physical interpretation of its inputs. Appendix F.2 states that it “is not sensitive to normal and roughness,” and Figure 6c shows that the albedo contains baked-in shading. A channel-wise ablation would help establish which buffers actually matter.
  • The evaluation leaves some important claims unresolved. The results outside WOD are less consistent. Difix3D achieves better LPIPS on both nuPlan and PandaSet, and better FVD on PandaSet’s original trajectory and on both datasets’ novel trajectories. These results need more discussion given the emphasis on temporal consistency. The human study also provides limited evidence for temporal quality because participants judge individual frames. The insertion metrics need clarification as well. If the reference image does not contain the inserted vehicle, reference-based metrics may penalize a successful edit. The paper should explain exactly what reference is used.
Questions for authors
  • Can you train an RGB-conditioned counterpart using the same backbone, training data, history scheme, and Stage I/II procedure? Replacing the G-buffers with 3DGS RGB, and optionally testing RGB plus G-buffers, would help isolate the contribution of the intrinsic interface.
  • A quantitative relighting evaluation on synthetic driving scenes with known lighting would be valuable. I would also suggest discussing the relationship to LightSim (NeurIPS 2023), ChatSim (CVPR 2024), Cosmos-Transfer1 (2025), StreetCrafter (CVPR 2025), and FreeVS (ICLR 2025).
9 comments

We have added matched RGB baselines (Fig. 5, Tab. 6), individual G-buffer ablations (Tab. 7), VBench temporal evaluations (Tab. 4), and a video-level human preference study (Tab. 5). We also clarify the insertion evaluation protocol, refine our data-efficiency claims and expand the discussion of relighting limitations and related work (Appendix C.1).

AuthorsOpenReview ↗

W1-1 RGB ablation (Q1):

Following the reviewer's suggestion, we train an RGB-conditioned counterpart under the same setting. We conduct this comparison with both Cosmos-Predict2.5 and U-Net (Fig. 5 and Tab. 6).

Interestingly, the results reveal a distinction between reconstruction fidelity and perceptual harmonization under scene editing.

On original trajectories, PIH achieves slightly better reconstruction metrics than RGB conditioning. However, under vehicle insertion, RGB conditioning obtains better reference-based metrics, despite producing less convincing vehicle–scene harmonization in the qualitative comparisons.

To investigate this discrepancy, we additionally evaluate HarmonyIQA[1], its vehicle-focused crop variant, TOPIQ-NR[2], and video-level human preference. PIH consistently achieves higher no-reference perceptual quality scores across both backbones. More importantly, 69.3% of non-tie human judgments prefer PIH over RGB conditioning when viewing the corresponding videos.

This observation is also supported by Fig. 5, where RGB-conditioned outputs can exhibit locally inconsistent vehicle appearance and illumination, despite their better reference reconstruction metrics.

We emphasize that this does not invalidate reconstruction metrics, which remain useful for assessing fidelity. Rather, these results suggest that reconstruction fidelity alone is insufficient to characterize the perceptual quality of counterfactual edits, where an exact captured counterpart may be unavailable.

Crucially, these matched experiments directly isolate the effect of the conditioning representation from backbone and training-data differences. Together with the qualitative and human evaluations, these findings support the value of physical-intrinsic conditioning as a compositional interface for editable neural reconstruction. We provide a detailed discussion of this trade-off in Sec. 4.5.

[1]Harmonyiqa: Pioneering benchmark and model for image harmonization quality assessment ICME'25

[2] Topiq: A top-down approach from semantics to distortions for image quality assessment TIP'24

AuthorsOpenReview ↗

W1-2 Data efficiency claim:

Our intended contribution is not merely a reduction in training-pair counts, but also a reduction in data-curation and engineering effort for new editing operations.

Conventional RGB harmonization pipelines often require constructing degraded/clean pairs that cover the relevant reconstruction artifacts and editing conditions. In contrast, PIH obtains supervision from existing scene reconstructions and automatically estimated pseudo label, without requiring additional paired RGB targets for vehicle insertion.

Although making pseudo gbuff label introduces computational overhead, it is an automated forward-inference process rather than the construction and curation of edit-specific paired reconstruction datasets.

We therefore clarify that the comparison of fewer than 100K pairs against 350K pairs is descriptive, rather than a controlled demonstration of superior sample efficiency. The key advantage is that the physical-intrinsic interface enables new compositions at inference time without collecting paired examples for each editing operation.

Reviewer_kZM2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Physical-Intrinsic Harmonizer, a one-step harmonizer designed to resolve rendering artifacts in edited neural driving scenes. The method conditions a pretrained generative model on imperfect scene G-buffers and a temporal history. Trained by flow matching and perceptual endpoint supervision, PIH bypasses the need for insertion-specific paired training data. PIH achieves the best reported metrics among the evaluated baselines on Waymo and the lowest FID on nuPlan and PandaSet, while other cross-dataset results are mixed. A single-frame user study reports a 77.41% preference over DiffusionHarmonizer.

Critical strengths
  1. The paper addresses a well-recognized bottleneck in driving simulation: while neural reconstructions perform well on recorded trajectories, visual quality noticeably degrades under novel viewpoint shifts or vehicle insertions. Improving these edited observations is of practical importance for testing autonomous driving systems in unrecorded scenarios. The proposed framework handles scene edits through geometry, material, and lighting conditioning and enables vehicle insertion without insertion-specific paired supervision, potentially reducing the data-curation burden.

  2. The method demonstrates clear quantitative improvements on the Waymo dataset. Notably, it reduces the original-trajectory FID from 19.87 to 9.12 compared to DiffusionHarmonizer. The single-frame user study across datasets further reports a 77.41% preference over DiffusionHarmonizer, indicating that these gains translate to meaningful perceptual improvements.

Critical weaknesses

While the results are promising, my main concern is that the ablations, analysis, and comparisons are insufficient to support the main claims. Specifically:

  1. The contribution of the physical-intrinsic conditioning is not sufficiently isolated. Although Tables 4–7 examine losses and history sources, the main baseline comparisons simultaneously involve different generative models and training data, making it difficult to cleanly attribute the performance gains to the proposed representation. Furthermore, Figure 6 provides only limited qualitative evidence for additional design choices, lacking matched, side-by-side comparisons between the final PIH outputs and the alternatives across diverse scenes and lighting conditions. These gaps limit confidence in the claimed advantages of the specific formulation.

  2. Although the DINO-S metric assesses overall structural similarity, the reported metrics and single-frame preference study provide limited evidence of adherence to specific editing conditions (e.g., precise vehicle placement, road geometry preservation, and accurate lighting responses). This distinction is critical for driving simulation, where improved realism must not compromise the intended scene configuration. Therefore, the claims regarding controllability are less convincingly supported than the demonstrated gains in visual quality.

Questions for authors
  1. Could the authors provide a controlled ablation comparing standard RGB conditioning against the proposed physical-intrinsic conditioning? This comparison should use the identical backbone, training dataset, and temporal context to properly isolate the contribution of the intrinsic inputs.

  2. Could the authors provide side-by-side comparisons of the final PIH model and the alternative designs using identical inputs across a wider variety of scenes and lighting conditions? The current examples do not clearly illustrate the specific failure modes of the alternative designs or how the final PIH configuration addresses them. Incorporating matched close-ups, explicit labels, and brief annotations of the key visual differences would make this evaluation much more conclusive.

  3. To better substantiate the editing accuracy, could the authors provide a targeted evaluation of condition adherence? For example, one or more of the following: quantitative or structured qualitative analysis regarding vehicle-mask alignment, lane geometry preservation, and shadow consistency under controlled lighting changes would strengthen the paper.

7 comments

We have substantially strengthened the evaluation by adding controlled RGB-conditioning baselines across two backbones (Fig. 5, Tab. 6), quantitative and qualitative analyses of alternative designs (Fig. 8, Tab. 10), and VIMS/BAS metrics for editing fidelity (Tab. 2). We also identify a gap in existing evaluation metrics and introduce no-reference perceptual evaluations (Tab. 6) and a video-level human preference study (Tab. 5) to better assess harmonization under scene editing.

AuthorsOpenReview ↗

W1-1 RGB ablation (Q1)

Following the reviewer's suggestion, we train an RGB-conditioned counterpart under the same setting. We conduct this comparison with both Cosmos-Predict2.5 and U-Net (Fig. 5 and Tab. 6).

Interestingly, the results reveal a distinction between reconstruction fidelity and perceptual harmonization under scene editing.

On original trajectories, PIH achieves slightly better reconstruction metrics than RGB conditioning. However, under vehicle insertion, RGB conditioning obtains better reference-based metrics, despite producing less convincing vehicle–scene harmonization in the qualitative comparisons.

To investigate this discrepancy, we additionally evaluate HarmonyIQA[1], its vehicle-focused crop variant, TOPIQ-NR[2], and video-level human preference. PIH consistently achieves higher no-reference perceptual quality scores across both backbones. More importantly, 69.3% of non-tie human judgments prefer PIH over RGB conditioning when viewing the corresponding videos.

This observation is also supported by Fig. 5, where RGB-conditioned outputs can exhibit locally inconsistent vehicle appearance and illumination, despite their better reference reconstruction metrics.

We emphasize that this does not invalidate reconstruction metrics, which remain useful for assessing fidelity. Rather, these results suggest that reconstruction fidelity alone is insufficient to characterize the perceptual quality of counterfactual edits, where an exact captured counterpart may be unavailable.

Crucially, these matched experiments directly isolate the effect of the conditioning representation from backbone and training-data differences. Together with the qualitative and human evaluations, these findings support the value of physical-intrinsic conditioning as a compositional interface for editable neural reconstruction. We provide a detailed discussion of this trade-off in Sec. 4.5.

[1]Harmonyiqa: Pioneering benchmark and model for image harmonization quality assessment ICME'25

[2] Topiq: A top-down approach from semantics to distortions for image quality assessment TIP'24

AuthorsOpenReview ↗

W1-2 Quantity albation on design choice (Q2)

We present 3 different design choices in the paper, since all these alternatives has noticeable failures and some of them is hard to quantify, the numerical reults is less informative:

  1. Direct regression and objective design. Direct endpoint regression can produce noticable blurry outputs under imperfect intrinsic conditioning. We additionally compare latent flow matching, pixel-space flow matching, and pixel-space endpoint regression quantitatively. Latent flow matching achieves substantially better temporal fidelity (FVD 215.16 vs. 259.70/258.24 for the pixel-space alternatives), supporting our objective design.
  2. RenderFlow-inspired initialization. We investigate albedo initialization as an alternative to noise-based generation. However, imperfect reconstructed albedo may contain baked-in illumination, reducing robustness to lighting changes (Fig. 8b–c). A direct quantitative relighting comparison is also challenging without paired ground-truth observations under counterfactual illumination.
  3. Diffusion Forcing. We investigate its adaptation to causal attention and observe progressive error accumulation over successive chunks (Fig. 8d). This behavior reflects noticiable long-horizon exposure bias rather than an isolated single-frame failure, motivating our training strategy.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We sincerely thank all four reviewers for their constructive feedback. We have substantially revised the manuscript with new controlled experiments, additional evaluations, and expanded technical discussions. The major revisions are summarized below, grouped by the reviewers' shared concerns. All the modifications are highlighted in BLUE.

1. Contribution of Physical-Intrinsic Conditioning (KZM2, vij4, 3H22).

We introduce matched RGB-conditioning baselines under identical training settings across two backbones (Sec. 4.5, Fig. 5, Tab. 6), together with individual G-buffer channel ablations (Tab. 7). The controlled comparisons reveal an important distinction between reference reconstruction fidelity and perceptual harmonization, with PIH demonstrating superior perceptual quality under scene editing, further supported by our video-level human preference study (Tab. 5). These findings provide direct evidence for the value of physical-intrinsic conditioning as a compositional editing interface. The channel ablations further confirm the contribution of physical-intrinsic inputs beyond albedo alone.

2. Editing Fidelity and Temporal Consistency (KZM2, vij4, 3H22, iCdR).

We add VIMS and BAS (Tab. 2) following HorizonForge to evaluate editing fidelity, where PIH achieves the best vehicle identity preservation and background alignment. We additionally conduct VBench evaluations across three datasets and four editing settings (Sec. 4.4, Tab. 4). PIH achieves the highest Subject Consistency while maintaining temporal quality comparable to DiffusionHarmonizer, including under cross-domain novel-trajectory settings. Our new full-video human preference study (Tab. 5) further complements the original frame-level evaluation.

3. Inference Efficiency (3H22, iCdR).

We provide detailed H100 profiling, optimization results and comparsion with counterpart (Appendix F.1, Tabs. 14–15). The optimized implementation achieves a substantial measured latency reduction without retraining. We identified that future acceleration can be achieved by token compression and engineering, while single-step is a MUST path toward this.

We believe these revisions provide substantially stronger evidence for the effectiveness of physical-intrinsic harmonization and clarify the contribution, evaluation, and practical scope of PIH.

AuthorsOpenReview ↗

Final Discussion Update

As the rebuttal period comes to a close, we sincerely thank all reviewers for their constructive feedback and engagement throughout the discussion.

We are encouraged that reviewers kZM2 and iCdR have explicitly acknowledged that their main concerns were adequately addressed and have both raised their ratings to Weak Accept. In particular, kZM2 recognized that the matched RGB-conditioning experiments substantially addressed the concern regarding the contribution of physical-intrinsic conditioning, while iCdR acknowledged the additional evaluations and clarifications.

We appreciate the remaining feedback and will incorporate these improvements into the final manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.