- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
- In novel-view synthesis, refraction causes strongly view-dependent appearance. Reconstructing objects enclosed inside refractive objects remains challenging for existing methods.
- The method reconstructs a scene in three stages: (1) a simple reconstruction using 2DGS-based surfels without refraction; (2) initializing an SDF from masks of the refractive object and refining the SDF and index of refraction from images; and (3) reconstructing enclosed objects with surfels.
- The method achieves high image fidelity on existing refractive-scene datasets and on the paper's new dataset containing enclosed objects.
Critical strengths
- The paper points out that real-world scenarios such as fish in a tank can contain objects inside refractive objects, while methods that treat refraction without reconstructing enclosed objects do not address this setting.
- It proposes a differentiable rendering and learning pipeline that handles changes in ray direction caused by refraction and reflection, together with occlusion by enclosed objects.
Critical weaknesses
-
Real refractive objects can have substantially more complicated structures. A glass bottle has thickness, can contain air and a liquid with different refractive indices, and may itself be colored. This paper instead considers a single transparent medium with opaque enclosed objects and a uniform refractive index throughout its interior. The colored-absorption extension does not address the multiple material layers of these containers. Why are reflection paths from enclosed objects or the background not explicitly traced, and what happens when these reflections matter? Why are multiple transparent objects or nested layers, such as liquid in a glass, not handled, and how would the method behave in those cases? The introduction's examples at line 28, “fish in a tank or specimens in a glass jar,” seem to be situations the method cannot actually handle.
-
The experimental scope is very limited, even including the appendix. The main paper focuses on existing datasets, a new synthetic dataset for the target task, and ablations. Many other experiments could be considered: reconstructed geometry, failure cases, spatially varying refractive indices, a wider range of real scenes for the target task, rough surfaces such as water, or scenes with many enclosed objects. (These are illustrative possibilities, not a claim that every suggested experiment is necessarily useful or required.) Broader experiments are needed to demonstrate the method's advantages and generalization.
Questions for authors
-
The method requires masks for objects that cause refraction and reflection. These effects are tracked only within a limited region and are not handled elsewhere. Obtaining the masks is also a practical problem: synthetic masks are easy to obtain, but real foreground masks are difficult, and manual segmentation requires substantial effort. Automation with SAM 2 does not eliminate this problem, since existing video segmentation models often lose tracking under occlusion. Consequently, even with a single foreground object, the method is limited to carefully arranged laboratory settings in which the object remains clearly visible from all views throughout a full rotation.
-
The foreground SDF is initialized using only masks. Masks constrain the observed silhouettes but cannot recover concave structures that do not appear in those silhouettes. Is the method therefore limited to simple objects without such hidden concavities, for which mask-based initialization is already sufficient despite subsequent refinement? The curvature regularization in Eq. 9 also favors smooth surfaces, reinforcing my concern about recovering these structures.
-
In Figure 1, Stage 2 appears to receive image supervision without enclosed objects, which is confusing. The refinement discussion around line 287 and the supplied code establish that the original images containing enclosed objects are used without identifying or excluding the affected pixels. Enclosed objects could therefore influence the reconstructed geometry: does the surface retain the intended shape, or do spurious layers appear? Such results should be shown.
-
A previously unaddressed setting does not by itself make a paper good or justify acceptance. The paper should establish a practically needed problem that others have tried but struggled to solve, introduce an idea that addresses it, and demonstrate through varied results and analysis how that idea succeeds. Instead, this paper uses several assumptions to reduce the problem to an easier, smaller subset, then works around it through a sequence of stages built from existing techniques. The evidence remains concentrated on the main experiments and does not sufficiently show whether the model works in varied situations.
-
The reason for separating the first and second stages is unclear. The input masks could be used instead of rendered masks, and it may be possible to learn only the SDF or to learn the surfel and SDF representations together. Jointly training all three stages could make geometry unstable because of ambiguity, but the need to separate the first two stages is less apparent.
-
At the transition from Stage 2 to Stage 3, all foreground surfels are reintroduced, including those representing the refractive surface already modeled by the SDF rather than only enclosed objects. How are these refractive-surface surfels handled? Are opacity reset and the subsequent optimization and pruning expected to make them disappear?
Response to Reviewer egTR
We thank the reviewer for the insightful and constructive comments.
C1. Assumptions (W1, W2, Q2)
Please refer to our Response A2 to Reviewer aJLJ.
C2. Relying on Masks (Q1)
As discussed in our Response A2 to Reviewer aJLJ, existing methods also rely on masks or scene priors, and Appendix A.5 shows that our method is robust to imperfect masks.
C3. Stage 2 Optimization (Q3)
Stage 2 uses the original input images for supervision, following the practice of Eikonal Field.
C4. Significance of the Problem (Q4)
Refractive scenes, such as specimens embedded in resin, are common in the real world. Our method outperforms previous methods on scenes both with and without enclosed objects.
C5. SDF Initialization from Rendered Masks (Q5)
Directly fitting the SDF to input masks cannot distinguish occluded regions from regions outside the refractive object.
C6. Surfel Optimization in Stage 3 (Q6)
The surfel optimization relies on multi-view image supervision, pruning, and opacity reset.