- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
Background and Problem: Neural rendering methods (e.g., NeRF and 3DGS) often suffer from artifacts and missing textures in dynamic scenes. Existing diffusion-based methods rely on clean supervision or generic priors, limiting their adaptability to scene-specific degradation.
-
Key Contributions: The paper proposes UG-Diff, an uncertainty-guided diffusion framework that eliminates the need for clean supervision. It introduces uncertainty mixing for adaptive feature restoration and multi-view condition injection to leverage complementary geometric and texture information across views.
-
Experimental Evaluation: Experiments on DF3DV, On-the-go, and RobustNeRF compare UG-Diff with existing methods, including Difix3D, GenFusion, and GeoQuery, demonstrating its reconstruction capabilities and potential as a general-purpose plug-in.
Critical strengths
-
Unified and Elegant Weakly Supervised Modeling: The method treats uncertainty as a continuous conditioning variable, enabling adaptive modeling of static structures and dynamic occluders during training, while removing occluders at inference. This design avoids hard-threshold segmentation and additional clean data, offering an effective and streamlined solution.
-
Strong Plug-and-Play Capability and Generalizability: The approach works across different 3D representations, including NeRF and 3DGS, and can be integrated into existing diffusion methods such as Difix3D and GenFusion, demonstrating its practical value and versatility.
Critical weaknesses
-
Strong Dependence on Upstream Uncertainty Estimation: The framework relies heavily on the accuracy of residual uncertainty estimates from the rendering backbone. In scenes with extensive occlusion, weak textures, or reflective surfaces, inaccurate uncertainty maps may lead to hallucinated content or residual artifacts.
-
Sensitivity to Reference View Selection and Cross-View Alignment: Ablation studies show that restoration quality is highly sensitive to reference-view overlap. The reliance on 2D cross-attention without explicit geometric constraints, such as epipolar or depth-reprojection constraints, may cause performance degradation or introduce incorrect textures under large viewpoint changes or limited shared visibility.
Questions for authors
-
Reference View Selection: In the main experiments, are reference views selected automatically based on camera-pose overlap or manually using heuristics? How robust is the method when the camera follows a one-way trajectory with minimal view overlap?
-
Inference Latency and Computational Overhead: How much additional GPU memory and inference time are required by multi-view feature concatenation and cross-attention? How does UG-Diff's end-to-end throughput compare with the single-step baseline Difix3D?
-
Theoretical Motivation for the Perturbation Type: Checkerboard noise significantly outperforms Gaussian and uniform noise in the ablation studies. Could the authors provide further explanations from geometric or frequency-domain perspectives?
4 comments
Response RAqw (Part 2/2)
Q2. Inference Latency and Computational Overhead
We report model latency, peak GPU memory, and end-to-end performance for Difix3D-ref and UG-Diff with four references. With the additional reference views, UG-Diff takes 228.3 ms per image for model inference, compared with 186.2 ms for Difix3D, while peak GPU memory remains comparable (4.58 vs. 4.63 GiB(ours)). End-to-end latency is 442.1 vs. 347.1 ms per image, corresponding to throughputs of 2.26 and 2.88 images/s, respectively. These measurements quantify the overall computational cost of the two reference configurations.
Q3. Theoretical Motivation for the Perturbation Type
Thank you for raising this interpretability question. Our ablations empirically show that checkerboard perturbations outperform Gaussian and uniform noise. We hypothesize that this relates to the diffusion prior: single-step models such as Difix3D+ treat artifact-corrupted renderings as intermediate noisy states in the denoising process. Gaussian or uniform perturbations may therefore be less distinguishable from the corruption already present in the rendering. Checkerboard perturbations instead introduce a structured, high-frequency periodic signal, which may help the model distinguish rendering artifacts from distractor-related variations. This provides a possible frequency-domain explanation for the observed advantage.
Thank you to the authors for the response and the computational overhead benchmarks.
While the latency/memory numbers and frequency-domain explanation address my questions on efficiency and perturbation design, my core concerns regarding robustness (W1, W2/Q1) remain insufficiently resolved:
- The rebuttal relies heavily on selective qualitative visualizations rather than systematic quantitative stress-tests on challenging subsets (e.g., low-texture, reflective, or wide-baseline scenes). A few selected examples are prone to cherry-picking and do not convincingly establish overall reliability, especially given the failure modes already acknowledged in Section 6 and Appendix A.5.
- Using completely unrelated scenes does not accurately reflect low-overlap camera trajectories, where similar local textures in non-overlapping regions can mislead unconstrained 2D cross-attention into copying wrong textures.
Given the lack of rigorous quantitative validation on these failure-prone regimes, I find the evidence insufficient to raise my score and will maintain my rating of Weak Reject.
Response RAqw
Thank you for the follow-up and for acknowledging our responses on efficiency and perturbation design.
Regarding W1, we respectfully disagree with characterizing the evidence as cherry-picked. The cited visualizations illustrate imperfect uncertainty estimates already present in our evaluations, while the quantitative results cover the full test sets under the standard evaluation protocol. Together, they support the practical effectiveness of uncertainty-conditioned learning. We acknowledge that dedicated challenging-subset evaluations would further characterize its robustness.
Constructing reliable low-texture, reflective, and wide-baseline subsets, together with controlled same-scene low-overlap reference sets, requires additional curation and validation that we cannot complete within the rebuttal period. We also recognize the distinction you raise between unrelated-scene references and same-scene low-overlap references. Our experiment examines restoration when useful cross-view reference information is unavailable.
We appreciate your careful assessment and the specific directions for further evaluation.
Response RAqw (Part 1/2)
We thank the reviewer for the thoughtful feedback and are particularly pleased that you found our weakly supervised formulation elegant. Your concerns help us clarify how the method handles imperfect uncertainty estimates and reference information. Below, we provide supporting evidence, report computational costs, and discuss the intuition behind our perturbation design.
Thank you for this insight. The upstream backbone can indeed produce inaccurate uncertainty maps in scenes with extensive occlusions, weak textures, or reflective surfaces. Our core contribution is to use distractor-aware uncertainty as a continuous condition for weakly supervised learning, without requiring it to serve as a pixel-accurate distractor mask.
The last two rows of Fig. 5 show that the model reproduces the supervised umbrella/person and bus during distractor fitting even though the uncertainty responses do not precisely cover these objects. Conversely, Fig. 8 shows that high uncertainty responses on static structures do not automatically induce hallucinated distractors. These examples indicate that the learned behavior accommodates local mismatches between uncertainty responses and object regions. Fig. 3 further illustrates the conditional mapping itself: varying uncertainty for the same rendered input progressively shifts the output from static restoration toward distractor fitting. Together, these observations support the model’s ability to learn coherent conditional behavior from imperfect uncertainty estimates. Table 4 further shows that continuous uncertainty outperforms binary masking across all reported metrics, supporting the value of preserving graded uncertainty information for restoration. These results support the effectiveness of retaining graded uncertainty information for learning restoration under the evaluated backbone estimates.
Thank you for raising these concerns about reference-view selection and alignment, which we also considered in our work. Explicit cross-view alignment can provide more precise restoration guidance, but it relies on multi-view geometric consistency, which is not always guaranteed in distractor-corrupted observations. We reuse the source views selected by MU-GeNeRF to reconstruct the target rendering. Selection is automatic, using a heuristic score based on camera rotations and translations (R,T).
We use reference views to provide scene-level appearance and complementary restoration cues without enforcing explicit geometric correspondences (which can be unreliable in the presence of transient distractors). To preserve restoration capability when these reference cues are unreliable or unavailable,we retain the per-view attention output Xsv alongside the multi-view output Xmv. This blending preserves a dedicated pathway for the target view’s own features, allowing restoration to draw on both the target rendering and reference information.
Our evaluations on DF3DV, On-the-go, and RobustNeRF use sparse, discrete scene observations rather than continuous one-way camera trajectories. To address the concern about minimal overlap, we evaluate with references from unrelated scenes as a proxy for the extreme case where no useful cross-view overlap is available(As shown in A.7). Even under these conditions, UG-Diff better preserves the target structure than Difix-style Attn., supporting the role of Xsv in maintaining restoration based on the target rendering itself. These results characterize the restoration capability retained without useful reference evidence, with qualitative comparisons provided in the appendix.