- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes S2DNeRF for reconstructing clean static scenes from multi-view images containing transient distractors. The method represents static content using a cross-view shared 3D radiance field and distractors using per image 2D feature fields, jointly optimized through alpha compositing. The authors refer to the different sharing structures of these representations as the Asymmetric Representation Prior (ARP). Combined with nearly transparent opacity initialization and three regularization terms, the method separates static and transient content without pretrained semantic models or hard masks. Experiments demonstrate its effectiveness.
Critical strengths
- Method is simple and easy to understand.The shared 3D field and per image 2D fields are jointly trained under a unified compositing objective, without external semantic models or predefined distractor masks. Figs. 1 and 4 demonstrate the ability to handle distractors such as cast shadows. 2.Experiments with ablations supporting the core design.The method achieves the best or second-best results in 17 of 18 metrics in Table 1. Table 4 shows that adding only the 2D distractor field improves the average PSNR from 17.267 to 21.657, while the full method further improves it to 22.748. The appendix also studies initialization and representation capacity.
Critical weaknesses
- Fig. 3 shows when trained alone, the static field fits static regions faster than distractor regions, but it does not directly demonstrate how the two branches achieve the correct decomposition during joint optimization. Table 7 further shows that opacity initialization significantly affects the decomposition. This matters because ARP is a central explanatory contribution of the paper.
- Sec. 2.2 acknowledges that HybridGS also combines a shared 3D representation with per image 2D representations, making the representation and optimization strategies important differences. The per-pixel DeSplat experiment in Appendix B.4 is useful, but it only covers Spot and uses different initialization and training configurations. It shows that simply increasing Gaussian coverage does not improve performance, but does not fully isolate the advantages of neural field representations themselves.
Questions for authors
- Could the authors provide the evolution of average distractor opacity or branch contributions in static and distractor regions during joint optimization? Could the authors also clarify whether the claims about ARP should be conditioned on suitable initialization and training configurations?
- Appendix B.4 explains that the initial Gaussian opacity is set to 0.1 because of the pruning threshold. Does this constraint still apply to the variants with pruning disabled? If these variants use an initial pixel opacity closer to that of the proposed method, do they still perform significantly worse?
- Although S2DNeRF addresses how to separate static scenes from distracting content on its own, I think this is a NeRF task that was solved long ago? For example, in the dynamic NeRF task, it was already possible to reconstruct clean designated regions via masks, e.g., "Detachable Novel Views Synthesis of Dynamic Scenes Using Distribution-Driven Neural Radiance Fields" and "DynIBaR: Neural Dynamic Image-Based Rendering.", did authors try them on this task?
4 comments
RE: R3-Q2: Opacity initialization without pruning. With pruning disabled, we reran both per-pixel variants on Spot with near-zero initial Gaussian opacity. The frozen-geometry variant improved from 20.72 to 22.23 dB in PSNR. Since Gaussians placed at every pixel can fit the image from the start, an initialization favoring the 3D field keeps the static content from being fitted by the distractor branch, consistent with our observation on . We add these results to Appendix B.4 (Table 9).
Table 9: Per-pixel initialization of the DeSplat distractor Gaussians on Spot. All DeSplat rows are trained and evaluated with the official DeSplat code (LPIPS with AlexNet). “Init” and “Final” are the numbers of distractor Gaussians per view before and after training (mean over views). “Transparent” is the fraction of training pixels whose accumulated distractor opacity is below after training. “Opacity” is the initial opacity of each distractor Gaussian.
| Initialization | Densification | Opacity | Init | Final | PSNR↑ | SSIM↑ | LPIPS↓ | Transparent |
|---|---|---|---|---|---|---|---|---|
| Random (default) | ✓ | 0.1 | 24.73 | 0.786 | 0.147 | 41.4% | ||
| Random | ✓ | 0.1 | 24.69 | 0.789 | 0.142 | 40.6% | ||
| Per-pixel | ✓ | 0.1 | 22.40 | 0.712 | 0.304 | 25.9% | ||
| Per-pixel | 0.1 | 20.58 | 0.663 | 0.380 | 3.2% | |||
| Per-pixel, frozen geometry | 0.1 | 20.72 | 0.693 | 0.268 | 0.0% | |||
| Per-pixel | 0.006 | 22.82 | 0.728 | 0.249 | 12.5% | |||
| Per-pixel, frozen geometry | 0.006 | 22.23 | 0.718 | 0.251 | 18.1% |
RE: R3-Q3: Comparison with dynamic NeRF.
DetRF and DynIBaR address a different task. As shown in Table A, these methods rely on temporal correspondences between consecutive frames to estimate optical flow and synthesize novel views of dynamic scenes, while our goal is to recover a clean static scene from unordered images, where different distractors may appear or disappear across views without reliable temporal correspondences.
Table A: Input and output of our method, DetRF, and DynIBaR.
| Method | Input | Output |
|---|---|---|
| Ours | Unordered images | Static scene NVS |
| DetRF and DynIBaR | Video | Dynamic scene NVS |
Thanks for the detailed rebuttal and the additional exp. The revised Fig. 3 and Table 7 clarify how the decomposition emerges during joint optimization and the role of initialization. Table 11 and the new Table 9 results isolate the effect of the representation. The distinction from dynamic NeRF methods is also clear now. My concerns have been addressed, and I will maintain my rating of Weak Accept.
Thank you for taking the time to review our rebuttal and the additional experiments. We’re glad they addressed your concerns.
RE: R3-W1 (first part) & R3-Q1 (first part): Branch evolution during joint optimization.
We revised Fig. 3. Its upper part visualizes the two branches and the distractor opacity during joint optimization on BabyYoda over all 25k iterations, and its lower part plots the average αd on static and distractor regions (left) and the PSNR of the static field against the clean images with joint optimization (middle) and with the static field trained alone (right). As training proceeds, αd approaches 1 on distractor regions and 0 on static regions (0.96 and 0.017 at 25k iterations). With joint optimization, the static field reaches 35.4 dB, 12.8 dB higher than when it is trained alone (22.6 dB). When trained alone (bottom right), the static field fits cross-view-consistent content before fitting distractors. Its PSNR against the clean images first rises to 23.8 dB at 1.5k iterations as it fits the static regions, then drops as it overfits the distractors. This fitting order supports the premise behind ARP that, during joint training, the per-image 2D fields can fit distractors before the shared 3D field does.
Figure 3
Figure 3: Decomposition during joint optimization on BabyYoda (Sabour et al., 2023).
RE: R3-W1 (second part) & R3-Q1 (second part): Role of initialization. Yes, we claim that ARP provides an inductive bias which, together with suitable initialization and joint optimization, leads to the desired decomposition in the evaluated settings. We now state this condition in the Abstract, Introduction, Sec. 3.2, and Conclusion, and discuss it in Appendix D.
Initializing αd near zero to favor the static field is a deliberate part of our optimization strategy. This choice reduces the risk that the per-image 2D fields explain the entire image early in training, leading to a more stable decomposition. Appendix B.2 evaluates the effectiveness of this initialization strategy. Additionally, α0=0.5 weights the two branches equally. On five of the six scenes in Table 7, α0=0.5 stays within 0.15,dB of the default α0=0.018, so the gains cannot be attributed solely to the near-zero initialization.
Table 7: Initial distractor opacity α0 setting, averaged over the six NeRF On-the-go scenes. Every other setting follows Table 6. The best value in each column is in bold.
RE: R3-W2: Representation versus the conditions for decomposition. To isolate the representation, we swap our neural distractor branch for a Gaussian one. We swap our per-view hash fields for per-view Gaussians placed at every pixel with the same losses, training strategy, and a closely matched initial pixel opacity. The two reach comparable PSNR, but the 2D hash field reaches better SSIM and LPIPS and is more memory efficient (see Table 11). Thank you for suggesting this experiment.
Table 11: Average results on the six NeRF On-the-go scenes.