acceptodds
Under review as a conference paper at ICLR 2027

S2D-NeRF: Screen and Space Decoupling for Distractor-Free Novel View Synthesis

Abstract

Neural Radiance Fields (NeRFs) achieve photorealistic novel view synthesis, but assume that the scene is consistently observed across all views. In real-world captures, transient objects and the shadows they cast violate this assumption and lead to floaters and ghosting artifacts. Existing methods often identify distractors using semantic priors or heuristic rules, making the separation sensitive to segmentation quality or hand-designed assignment criteria. Instead, we find that cross-view consistency itself provides a strong cue for separating distractors, and our representation is designed to turn this cue into an effective inductive bias. We therefore propose SD-NeRF, which models the static scene with a shared 3D field and distractors with per-image 2D fields directly defined in pixel coordinates. This asymmetric representation drives their separation during joint optimization from the first iteration. Experiments on the NeRF On-the-go, RobustNeRF, and Photo Tourism datasets show competitive or superior rendering quality without semantic priors or heuristic rules. Our code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Distractor-free reconstruction, Unconstrained photo collections, Novel view synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rPmGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S2DNeRF for reconstructing clean static scenes from multi-view images containing transient distractors. The method represents static content using a cross-view shared 3D radiance field and distractors using per image 2D feature fields, jointly optimized through alpha compositing. The authors refer to the different sharing structures of these representations as the Asymmetric Representation Prior (ARP). Combined with nearly transparent opacity initialization and three regularization terms, the method separates static and transient content without pretrained semantic models or hard masks. Experiments demonstrate its effectiveness.

Critical strengths
  1. Method is simple and easy to understand.The shared 3D field and per image 2D fields are jointly trained under a unified compositing objective, without external semantic models or predefined distractor masks. Figs. 1 and 4 demonstrate the ability to handle distractors such as cast shadows. 2.Experiments with ablations supporting the core design.The method achieves the best or second-best results in 17 of 18 metrics in Table 1. Table 4 shows that adding only the 2D distractor field improves the average PSNR from 17.267 to 21.657, while the full method further improves it to 22.748. The appendix also studies initialization and representation capacity.
Critical weaknesses
  1. Fig. 3 shows when trained alone, the static field fits static regions faster than distractor regions, but it does not directly demonstrate how the two branches achieve the correct decomposition during joint optimization. Table 7 further shows that opacity initialization significantly affects the decomposition. This matters because ARP is a central explanatory contribution of the paper.
  2. Sec. 2.2 acknowledges that HybridGS also combines a shared 3D representation with per image 2D representations, making the representation and optimization strategies important differences. The per-pixel DeSplat experiment in Appendix B.4 is useful, but it only covers Spot and uses different initialization and training configurations. It shows that simply increasing Gaussian coverage does not improve performance, but does not fully isolate the advantages of neural field representations themselves.
Questions for authors
  1. Could the authors provide the evolution of average distractor opacity or branch contributions in static and distractor regions during joint optimization? Could the authors also clarify whether the claims about ARP should be conditioned on suitable initialization and training configurations?
  2. Appendix B.4 explains that the initial Gaussian opacity is set to 0.1 because of the pruning threshold. Does this constraint still apply to the variants with pruning disabled? If these variants use an initial pixel opacity closer to that of the proposed method, do they still perform significantly worse?
  3. Although S2DNeRF addresses how to separate static scenes from distracting content on its own, I think this is a NeRF task that was solved long ago? For example, in the dynamic NeRF task, it was already possible to reconstruct clean designated regions via masks, e.g., "Detachable Novel Views Synthesis of Dynamic Scenes Using Distribution-Driven Neural Radiance Fields" and "DynIBaR: Neural Dynamic Image-Based Rendering.", did authors try them on this task?
4 comments

RE: R3-W1 (first part) & R3-Q1 (first part): Branch evolution during joint optimization.

We revised Fig. 3. Its upper part visualizes the two branches and the distractor opacity during joint optimization on BabyYoda over all 25k iterations, and its lower part plots the average αd\alpha_d on static and distractor regions (left) and the PSNR of the static field against the clean images with joint optimization (middle) and with the static field trained alone (right). As training proceeds, αd\alpha_d approaches 1 on distractor regions and 0 on static regions (0.96 and 0.017 at 25k iterations). With joint optimization, the static field reaches 35.4 dB, 12.8 dB higher than when it is trained alone (22.6 dB). When trained alone (bottom right), the static field fits cross-view-consistent content before fitting distractors. Its PSNR against the clean images first rises to 23.8 dB at 1.5k iterations as it fits the static regions, then drops as it overfits the distractors. This fitting order supports the premise behind ARP that, during joint training, the per-image 2D fields can fit distractors before the shared 3D field does.

Figure 3

Figure 3: Decomposition during joint optimization on BabyYoda (Sabour et al., 2023).

RE: R3-W1 (second part) & R3-Q1 (second part): Role of initialization. Yes, we claim that ARP provides an inductive bias which, together with suitable initialization and joint optimization, leads to the desired decomposition in the evaluated settings. We now state this condition in the Abstract, Introduction, Sec. 3.2, and Conclusion, and discuss it in Appendix D.

Initializing αd\alpha_d near zero to favor the static field is a deliberate part of our optimization strategy. This choice reduces the risk that the per-image 2D fields explain the entire image early in training, leading to a more stable decomposition. Appendix B.2 evaluates the effectiveness of this initialization strategy. Additionally, α0=0.5\alpha_0=0.5 weights the two branches equally. On five of the six scenes in Table 7, α0=0.5\alpha_0=0.5 stays within 0.15,dB of the default α0=0.018\alpha_0=0.018, so the gains cannot be attributed solely to the near-zero initialization.

Table 7: Initial distractor opacity α0\alpha_0 setting, averaged over the six NeRF On-the-go scenes. Every other setting follows Table 6. The best value in each column is in bold.

α0\alpha_0PSNR↑SSIM↑LPIPS↓
0.0180.01822.750.7770.144
0.5000.50021.810.7570.163
0.9930.99315.080.5410.486

RE: R3-W2: Representation versus the conditions for decomposition. To isolate the representation, we swap our neural distractor branch for a Gaussian one. We swap our per-view hash fields for per-view Gaussians placed at every pixel with the same losses, training strategy, and a closely matched initial pixel opacity. The two reach comparable PSNR, but the 2D hash field reaches better SSIM and LPIPS and is more memory efficient (see Table 11). Thank you for suggesting this experiment.

Table 11: Average results on the six NeRF On-the-go scenes.

MethodBranch StaticDistractorPSNR↑SSIM↑LPIPS↓Train time (min)Train mem. (GB)FPSRender mem. (GB)
Ours (default)Zip-NeRF2D hash field22.750.7780.14239.74.331.44.01
Ours3DGS2D hash field22.220.7610.15512.41.634930.25
Ours3DGSGaussians, per pixel22.400.7560.16013.15.065371.56
DeSplat3DGSGaussians, densified22.080.7450.15717.01.765330.20
HybridGS3DGS2D Gaussians22.660.7570.208Code not released5810.62

RE: R3-Q2: Opacity initialization without pruning. With pruning disabled, we reran both per-pixel variants on Spot with near-zero initial Gaussian opacity. The frozen-geometry variant improved from 20.72 to 22.23 dB in PSNR. Since Gaussians placed at every pixel can fit the image from the start, an initialization favoring the 3D field keeps the static content from being fitted by the distractor branch, consistent with our observation on α0\alpha_0. We add these results to Appendix B.4 (Table 9).

Table 9: Per-pixel initialization of the DeSplat distractor Gaussians on Spot. All DeSplat rows are trained and evaluated with the official DeSplat code (LPIPS with AlexNet). “Init” and “Final” are the numbers of distractor Gaussians per view before and after training (mean over views). “Transparent” is the fraction of training pixels whose accumulated distractor opacity is below 0.050.05 after training. “Opacity” is the initial opacity of each distractor Gaussian.

InitializationDensificationOpacityInitFinalPSNR↑SSIM↑LPIPS↓Transparent
Random (default)✓0.1100010001397139724.730.7860.14741.4%
Random✓0.110,00010{,}0003170317024.690.7890.14240.6%
Per-pixel✓0.1182,780182{,}78028,17128{,}17122.400.7120.30425.9%
Per-pixel0.1182,780182{,}780182,780182{,}78020.580.6630.3803.2%
Per-pixel, frozen geometry0.1182,780182{,}780182,780182{,}78020.720.6930.2680.0%
Per-pixel0.006182,780182{,}780182,780182{,}78022.820.7280.24912.5%
Per-pixel, frozen geometry0.006182,780182{,}780182,780182{,}78022.230.7180.25118.1%

RE: R3-Q3: Comparison with dynamic NeRF.

DetRF and DynIBaR address a different task. As shown in Table A, these methods rely on temporal correspondences between consecutive frames to estimate optical flow and synthesize novel views of dynamic scenes, while our goal is to recover a clean static scene from unordered images, where different distractors may appear or disappear across views without reliable temporal correspondences.

Table A: Input and output of our method, DetRF, and DynIBaR.

MethodInputOutput
OursUnordered imagesStatic scene NVS
DetRF and DynIBaRVideoDynamic scene NVS

Thanks for the detailed rebuttal and the additional exp. The revised Fig. 3 and Table 7 clarify how the decomposition emerges during joint optimization and the role of initialization. Table 11 and the new Table 9 results isolate the effect of the representation. The distinction from dynamic NeRF methods is also clear now. My concerns have been addressed, and I will maintain my rating of Weak Accept.

Thank you for taking the time to review our rebuttal and the additional experiments. We’re glad they addressed your concerns.

Reviewer_NeJ5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies distractor-free novel view synthesis. The static scene is represented by a shared 3D Zip-NeRF field, while distractors are represented by per-image 2D fields defined directly in pixel coordinates.The two are alpha-composited and jointly optimized. The authors propose an "Asymmetric Representation Prior" (ARP), arguing that the asymmetry in how the two branches are shared causes cross-view-consistent content to be assigned to the 3D field and view-inconsistent content to the per-image 2D fields, without semantic priors or residual thresholds. They add three regularizers: binary entropy on distractor opacity, duplicate-color suppression, and near-camera occlusion suppression. The method is validated on NeRF On-the-go, RobustNeRF, and Photo Tourism with PSNR/SSIM/LPIPS, the appendix includes component ablations, per-image hash capacity, initial distractor opacity, and a per-pixel Gaussian control based on DeSplat.

Critical strengths
  1. The per-image 2D absorption field is a simple and effective design. It absorbs both genuine distractors and view-inconsistent illumination and reflections. And it could transfer to other backbones.

  2. The experiments are broad. The paper reports per-scene results on three datasets; Appendix B.3 ablates the per-image hash capacity, and B.4 replaces DeSplat's distractor Gaussians with a per-pixel version and shows a 2.3–4.2 dB drop.

Critical weaknesses
  1. The paper does not test an alternative backbone. Table 4 shows that adding the per-image field alone (row 0) raises PSNR from 17.267 to 21.657 dB (+4.39 dB), while the three regularizers together add only +1.09 dB (22.748). That is, about 80% of the gain comes from the 2D field itself. The authors also note that this field absorbs view-inconsistent illumination and reflections, so its role is closer to a per-image residual/appearance field like DeSplat or HybridGS, yet the paper still presents ARP as the main contribution. Since the absorption branch is representation-agnostic, the natural question is why the static backbone must remain Zip-NeRF. 3DGS is the dominant backbone in this area, and the two closest baselines (DeSplat and HybridGS) are both 3DGS methods, while the paper's argument does not depend on a neural field. B.4 cannot answer this: it replaces only DeSplat's distractor branch and keeps the 3DGS static backbone and its densification/pruning rules, which is the opposite swap. A controlled experiment with a 3DGS static backbone and the paper's per-image 2D field would directly show whether this 2D absorption field is a transferable contribution.

  2. More comparisons with methods from other technical routes are needed. The paper does not compare against recent distractor-free methods that report on the same benchmarks, such as AsymGS (NeurIPS 2025), RobustSplat (ICCV 2025), and SpotLessSplats (ACM TOG / SIGGRAPH 2025), so a cross-route comparison is missing.

Questions for authors
  1. Please compare with AsymGS, RobustSplat, and other recent methods from different technical routes.
  2. Could you run a controlled experiment that replaces the static backbone with 3DGS while keeping your per-image 2D field and the same composite loss?
  3. How does the method behave on a purely static dataset (e.g., mip-NeRF 360)? Does it hurt purely static reconstruction?
6 comments

RE: R2-W1 (first part): Role of ARP. We agree that most of the improvement in Table 4 comes from introducing the per-image 2D field, and this is our main contribution. ARP is not a separate module or loss competing with that field: it describes the inductive bias created by the asymmetric sharing structure - a shared 3D representation coupled with per-image 2D representations. The 2D field-only ablation therefore evaluates this architecture under our initialization and joint-training setup.

RE: R2-W1 (second part) & R2-Q1: 3DGS backbone. We acknowledge that combining a shared 3D representation and per-image representations has precedents in DeSplat and HybridGS. Our contribution is a compact 2D field together with an optimization design that enables high-quality distractor-free reconstruction. The field predicts color and opacity at every pixel and is jointly optimized with the static branch without distractor masks or thresholding. Suitable opacity initialization helps prevent per-image fields from absorbing the entire image. Our empirical analysis complements this practical design by examining how initialization, capacity, and optimization affect the decomposition. Replacing the 2D hash field with per-pixel Gaussians increases the per-image parameters by approximately 24 times, while the quality stays comparable (22.4 dB and 22.22 dB).

We replace the Zip-NeRF static backbone with 3DGS while keeping the same per-image 2D fields and compositing loss (Table 11). The 3DGS and Zip-NeRF backbones achieve comparable quality (22.22 dB and 22.75 dB). These results support the portability of the decomposition design and the compactness of our field. We use Zip-NeRF as the default backbone because it achieves the highest rendering quality among the variants we tested. We will add the 3DGS experiment to the revised manuscript to show that ARP also applies to a Gaussian-based scene representation. Thank you for suggesting this experiment.

Table 11: Average results on the six NeRF On-the-go scenes.

MethodBranch StaticDistractorPSNR↑SSIM↑LPIPS↓Train time (min)Train mem. (GB)FPSRender mem. (GB)
Ours3DGSGaussians, per pixel22.400.7560.16013.15.065371.56
Ours3DGS2D hash field22.220.7610.15512.41.634930.25
DeSplat3DGSGaussians, densified22.080.7450.15717.01.765330.20
Ours (default)Zip-NeRF2D hash field22.750.7780.14239.74.331.44.01

RE: R2-W2 & R2-Q1: Comparisons with prior work. We add SpotLessSplats, RobustSplat, and AsymGS to the comparisons on both NeRF On-the-go (Table 1) and RobustNeRF (Table 2). Averaged over all scenes, our method achieves the best PSNR, SSIM, and LPIPS among these methods on both datasets (Tables 1 and 2), with the same LPIPS as RobustSplat on NeRF On-the-go.

Table 1: Comparison with SpotLessSplats, RobustSplat, and AsymGS on the NeRF On-the-go dataset, averaged over the six scenes of Table 1 in the paper. The best result in each column is in bold.

MethodPSNR↑SSIM↑LPIPS↓
SpotLessSplats (Sabour et al., 2025)21.890.7370.183
RobustSplat (Fu et al., 2025)22.530.7750.142
AsymGS (GS-GS) (Li et al., 2025)22.440.7470.218
AsymGS (EMA-GS) (Li et al., 2025)21.790.7370.227
Ours22.750.7780.142

Table 2: Comparison with SpotLessSplats, RobustSplat, and AsymGS on the RobustNeRF dataset, averaged over the five scenes of Table 2 in the paper. The best result in each column is in bold.

MethodPSNR↑SSIM↑LPIPS↓
SpotLessSplats (Sabour et al., 2025)30.120.9140.082
RobustSplat (Fu et al., 2025)31.340.9300.066
AsymGS (GS-GS) (Li et al., 2025)31.260.9340.074
AsymGS (EMA-GS) (Li et al., 2025)30.560.9300.078
Ours31.380.9380.046

RE: R2-Q3: Purely static scenes. On the mip-NeRF 360 scenes, the mean distractor opacity decays during training on every scene (to 0.013–0.062), so the per-image fields largely withdraw when there is little to separate. However, these scenes may still contain view-inconsistent content: over-exposed or shadowed regions, and specular reflections, as shown in Fig. 11. The per-image fields absorb part of this content so that our static field keeps a consistent appearance across views, while Zip-NeRF fits all observed content into its single field. Our PSNR is therefore 1.59 dB lower on average. Similarly, DeSplat is 1.99 dB lower than Zip-NeRF on average (Table 10). We add these results to Appendix B.5.

Figure 11

Figure 11: A held-out view of Stump from the mip-NeRF 360 dataset. Error difference: per-pixel squared error of ours minus that of Zip-NeRF (red: ours higher; blue: ours lower; white: equal).

Table 10: Results on the purely static mip-NeRF 360 dataset. DeSplat is trained with its official code and default settings on the same images and split.

SceneZip-NeRF PSNR↑SSIM↑LPIPS↓Ours PSNR↑SSIM↑LPIPS↓DeSplat PSNR↑SSIM↑LPIPS↓
Bicycle26.790.8300.12025.740.8220.11925.850.8310.104
Bonsai35.060.9760.01632.120.9670.02431.400.9610.027
Counter29.880.9320.04726.310.9060.07228.180.9210.055
Garden29.710.9150.04828.570.9130.05228.470.9170.042
Kitchen34.610.9750.01433.420.9730.01631.220.9630.022
Room33.270.9640.02631.360.9550.03231.540.9560.033
Stump27.100.8220.12025.720.8040.13025.790.7870.130
Flowers23.510.7300.15322.700.7100.17121.880.7000.182
Treehill24.730.7170.20924.480.7210.19622.460.6980.193
Mean29.410.8730.08427.820.8630.09027.420.8590.088

Thank you for the additional experiments and clarifications. The rebuttal addresses my main experimental concerns: the authors add a 3DGS-backbone experiment, include comparisons with SpotLessSplats, RobustSplat, and AsymGS, and report results on mip-NeRF 360. These results make the compact per-image 2D hash field and its integration with the static branch a convincing empirical contribution. The 3DGS experiment also supports the claim that the design is not fundamentally tied to Zip-NeRF.

However, I remain unconvinced that ARP should be presented as a novel conceptual principle. The central premise of ARP—that static content is cross-view consistent and can be represented by a shared scene model, while transient or view-inconsistent content is better handled by view-specific mechanisms—is already a common basis in prior work. In particular, DualSplat explicitly describes transient objects as violating multi-view consistency, uses their sparse and inconsistent visibility to identify them, and further relies on feature and reconstruction consistency during mask refinement. Related ideas also appear in NeRF On-the-go and RobustSplat. The implementation in S2D-NeRF is different and useful, but the underlying observation is not new. I therefore suggest that the authors soften the novelty claim around ARP and present it primarily as an interpretation or explanation of the proposed asymmetric architecture. The main novelty should instead be positioned around the compact per-image 2D hash field, its initialization and optimization, and the resulting reconstruction quality.

One additional issue concerns the LPIPS comparison. LPIPS values depend substantially on the feature network used. The authors should verify that the AsymGS numbers and their own numbers are computed with the same LPIPS backbone and evaluation protocol. In particular, the relatively high LPIPS reported for AsymGS may be caused by using the repository default VGG network during reproduction, whereas other results may use AlexNet.

I appreciate the authors’ additional experiments, particularly the evaluation on purely static scenes. The observed degradation helps clarify the method’s scope, but I do not consider it a major concern: in practice, whether to use transient handling can be decided based on the scene.

Overall, the compact per-image 2D hash field and its empirical results are sufficient for me to support acceptance. I encourage the authors to temper the novelty claim around ARP and verify that LPIPS is evaluated with a consistent feature network. Thank you for the time and effort invested in addressing my questions.

Looking forward to your further reply.

Reviewer_zPniOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents S2D-NeRF, a framework for distractor-free novel view synthesis from unconstrained image collections. It represents the static scene using a shared 3D radiance field and transient distractors using per-image 2D neural fields. The authors introduce an Asymmetric Representation Prior (ARP), arguing that the different learning efficiencies of the two representations encourage view-consistent content to be captured by the shared 3D field and image-specific content by the 2D fields. Additional regularization is introduced to improve the decomposition. The method is evaluated on NeRF On-the-go, RobustNeRF, and Photo Tourism, achieving competitive or superior rendering quality compared with several NeRF- and 3DGS-based approaches.

Critical strengths
  1. The combination of a shared 3D static field and per-image 2D distractor fields provides an intuitive way to exploit cross-view consistency without relying on external semantic segmentation models. The resulting framework is relatively straightforward and integrates naturally with the Zip-NeRF backbone.

  2. The method achieves strong rendering quality on NeRF On-the-go, RobustNeRF, and Photo Tourism. The comparisons with both NeRF- and 3DGS-based methods, together with the ablation studies, provide useful empirical evidence of its effectiveness, particularly in removing transient distractors and their shadows.

Critical weaknesses

Insufficient justification for the role of the Asymmetric Representation Prior (ARP).

The paper argues that ARP resolves the ambiguity of static–transient decomposition through the different learning efficiencies of shared 3D and per-image 2D fields. However, this does not guarantee a correct decomposition, as both branches can represent static content. More importantly, the alpha-compositing formulation directly affects the gradient allocation between the two branches. The default initialization (α₀ = 0.018) strongly favors the static field during early optimization, while Appendix B.2 shows that a high initial distractor opacity can cause the static field to fail completely. Although the authors acknowledge this sensitivity, the experiments do not sufficiently disentangle the effects of representation asymmetry from initialization-induced optimization bias. Consequently, the claim that ARP itself resolves the decomposition ambiguity is stronger than the presented evidence supports.

Questions for authors

Q1. Temporal variations in Photo Tourism and related work.

The Photo Tourism dataset contains images captured at different times, potentially involving substantial changes in illumination, appearance, and scene structure. Although S2D-NeRF uses per-image appearance embeddings, it remains unclear how its cross-view consistency assumption distinguishes transient distractors from persistent or long-term scene changes. Could the authors clarify this limitation and its impact on reconstruction quality?

Related works, including CR-NeRF [1], SWAG [2], and Wild-GS [3], have investigated unconstrained photo collections with appearance variations and transient occlusions. A discussion of these works and their differences from S2D-NeRF would help better position the proposed approach.

Q2. Reconstruction efficiency and comparison with 3DGS-based methods.

Although the paper compares rendering quality against DeSplat and HybridGS, it does not provide a systematic efficiency comparison. Could the authors report training time, rendering speed (FPS), and GPU memory consumption under comparable settings? Such results would help clarify the practical quality–efficiency trade-offs between S2D-NeRF and existing 3DGS-based methods.

References

[1] Yang, Y., et al. Cross-Ray Neural Radiance Fields for Novel-View Synthesis from Unconstrained Image Collections. ICCV, 2023.

[2] Dahmani, H., et al. SWAG: Splatting in the Wild Images with Appearance-Conditioned Gaussians. ECCV, 2024.

[3] Xu, J., Mei, Y., and Patel, V. M. Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections. NeurIPS, 2024.

4 comments

RE: R1-W1 (first part): Ambiguity between the two branches.

We agree that the per-image 2D fields can also represent static content, but ARP biases the optimization toward the desired decomposition. For a static structure observed in multiple views, the shared 3D field can use gradients from several views in the same batch to update one set of parameters. Each per-image 2D field receives gradients only from its own image and must fit the structure independently. Combining gradients across views can help the shared 3D field fit cross-view-consistent content faster than the per-image 2D fields during joint training.

Fig. 3 illustrates this process. When trained alone (bottom right), the static field fits cross-view-consistent content before fitting distractors. This fitting order supports the premise behind ARP that, during joint training, the per-image 2D fields can fit distractors before the shared 3D field does. During joint optimization, the 2D fields first absorb part of the static content, but as the static field improves, the photometric loss lowers the mean αd\alpha_d on static pixels to 0.10 by 5k iterations, and the regularizers then reduce it to 0.02, while αd\alpha_d on distractor pixels approaches 1. The static field finally reaches a PSNR 12.8 dB higher than when trained alone, which indicates that the distractors are absorbed by the 2D fields without compromising the cross-view consistency of the static field. This analysis is reflected in the discussion of the revised Fig. 3 (Sec. 3.2) and of the regularizers (Sec. 3.3).

Figure 3

Figure 3: Decomposition during joint optimization on BabyYoda (Sabour et al., 2023).

RE: R1-W1 (second part): Role of initialization.

We claim that ARP provides an inductive bias which, together with suitable initialization and joint optimization, leads to the desired decomposition in the evaluated settings. Initializing αd\alpha_d near zero to favor the static field is a deliberate part of our optimization strategy. This choice reduces the risk that the per-image 2D fields explain the entire image early in training, leading to a more stable decomposition. Appendix B.2 evaluates this strategy and shows that a high initial distractor opacity can cause the static field to fail, which is consistent with our motivation for initially favoring the static field and supports this initialization strategy.

Table 7: Initial distractor opacity α0\alpha_0 setting, averaged over the six NeRF On-the-go scenes. Every other setting follows Table 6. The best value in each column is in bold.

α0\alpha_0PSNR↑SSIM↑LPIPS↓
0.0180.01822.750.7770.144
0.5000.50021.810.7570.163
0.9930.99315.080.5410.486

RE: R1-Q1 (first part): Persistent and long-term scene changes.

The method does not infer a canonical historical state or distinguish changes by their duration. It favors content that can be explained consistently across the available views. As shown in the Sacré-Cœur example (Appendix C.3, Fig. 14), the scene contains 830 training images, and the banners on the portico columns, an example of a long-term scene change, are present in 171 of them, i.e., 22% of the 764 training images in which the columns are visible. The distractor fields absorb the banners, and the static field shows the columns without the banners.

Figure 14

Figure 14: Banners on the portico columns of Sacre Coeur.

RE: R1-Q1 (second part): Differences from In-the-Wild NVS Methods. CR-NeRF and Wild-GS predict a per-view mask to down-weight distractor pixels in the reconstruction loss, and both add a regularizer that penalizes lowering these weights. However, distractor pixels whose weights remain nonzero can push the static field toward the distractor colors. Our per-image 2D field predicts a color Cd\mathbf{C}_d and an opacity αd\alpha_d at every pixel. Our optimization target is the composite color (Eq. 7), so even when αd<1\alpha_d < 1 in distractor regions, the static field can keep the background color while the composite matches the observed color. The key to constraining this freedom lies in ARP, the initialization, and the training strategy.

SWAG uses the same set of 3D Gaussians to represent static content and transient distractors, so it needs to optimize the 3D positions of the Gaussians representing distractors. However, transient distractors can appear or disappear across views and lack cross-view correspondences. We only need to explain their 2D appearance in each training image, without estimating their 3D positions. Based on this prior, we model distractors on the image plane of each image in which they appear and composite them with the static field’s rendering.

In Table 3, our method achieves a higher PSNR than CR-NeRF and SWAG on all three scenes, and a higher SSIM than CR-NeRF on all three. We add these works to Related Work, and CR-NeRF and SWAG to Table 3 (Wild-GS has no public code).

Table 3: Comparison with SWAG and CR-NeRF on the Photo Tourism dataset (rows of Table 3 in the paper). The best result in each column is in bold.

MethodBrandenburg Gate PSNR↑SSIM↑LPIPS↓Sacre Coeur PSNR↑SSIM↑LPIPS↓Trevi Fountain PSNR↑SSIM↑LPIPS↓
SWAG (Dahmani et al., 2024)26.330.9290.13921.160.8600.18523.100.8150.208
CR-NeRF (Yang et al., 2023b)26.530.9000.10622.070.8230.15221.480.7120.207
Ours27.420.9340.11423.260.8760.15223.380.7640.245

RE: R1-Q2: Efficiency. Table 11 in Appendix D compares training time, rendering FPS, and GPU memory usage. Our Zip-NeRF variant takes longer to train and render than the 3DGS baselines and uses more GPU memory. With a 3DGS backbone, our method trains in less time and uses less training memory than DeSplat, while achieving a higher SSIM and lower LPIPS than both DeSplat and HybridGS.

Table 11: Average results on the six NeRF On-the-go scenes.

MethodBranch StaticDistractorPSNR↑SSIM↑LPIPS↓Train time (min)Train mem. (GB)FPSRender mem. (GB)
DeSplat3DGSGaussians, densified22.080.7450.15717.01.765330.20
HybridGS3DGS2D Gaussians22.660.7570.208Code not released5810.62
Ours (default)Zip-NeRF2D hash field22.750.7780.14239.74.331.44.01
Ours3DGS2D hash field22.220.7610.15512.41.634930.25

Dear Reviewer,

How do you think of authors' rebuttal. Could you give some response?

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Results

Thank you for your time and thoughtful review. We have uploaded a revised manuscript and additional videos. As the discussion deadline is approaching, we would appreciate it if you could take a look at our responses when you have a chance. We would be glad to answer any remaining questions.

Comments on the paper

1 comment

We sincerely thank the reviewers for their careful reading of our paper and for their thoughtful feedback. We appreciate the opportunity to clarify our work and address each point below, using W for weaknesses and Q for questions.

We’ve re-uploaded the revised manuscript PDF with all changes marked in blue. We’re continuing to revise the manuscript in response to the latest reviewer feedback. We’ve also uploaded videos showing more qualitative comparisons of our method with RobustSplat, SpotLessSplats, and AsymGS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.