acceptodds
Withdrawn from ICLR 2027

RefracSurfel: Novel View Synthesis of Refractive Scenes with Enclosed Objects via Gaussian Surfel Ray Tracing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent novel view synthesis methods can reconstruct refractive scenes but still struggle with those containing enclosed objects: they either omit enclosed objects or rely on simplified optical models that limit rendering fidelity. To address this, our key idea is to use separate representations for the refractive interface and opaque surfaces: a neural signed distance field and ray-traceable Gaussian surfels, respectively. Our optimization proceeds in three stages: the first stage fits the surfels to the input images and masks while learning a per-surfel probability to separate the foreground from the background; the second stage initializes a signed distance field from the foreground surfels and refines it through differentiable ray tracing; the final stage optimizes surfels representing the enclosed objects using the recovered refracted ray paths. The effectiveness of our approach is demonstrated on various datasets of refractive scenes, comparing favorably against state-of-the-art techniques in terms of rendering quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Transparent Object, 3DGS, Refraction Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_egTROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  • In novel-view synthesis, refraction causes strongly view-dependent appearance. Reconstructing objects enclosed inside refractive objects remains challenging for existing methods.
  • The method reconstructs a scene in three stages: (1) a simple reconstruction using 2DGS-based surfels without refraction; (2) initializing an SDF from masks of the refractive object and refining the SDF and index of refraction from images; and (3) reconstructing enclosed objects with surfels.
  • The method achieves high image fidelity on existing refractive-scene datasets and on the paper's new dataset containing enclosed objects.
Critical strengths
  • The paper points out that real-world scenarios such as fish in a tank can contain objects inside refractive objects, while methods that treat refraction without reconstructing enclosed objects do not address this setting.
  • It proposes a differentiable rendering and learning pipeline that handles changes in ray direction caused by refraction and reflection, together with occlusion by enclosed objects.
Critical weaknesses
  1. Real refractive objects can have substantially more complicated structures. A glass bottle has thickness, can contain air and a liquid with different refractive indices, and may itself be colored. This paper instead considers a single transparent medium with opaque enclosed objects and a uniform refractive index throughout its interior. The colored-absorption extension does not address the multiple material layers of these containers. Why are reflection paths from enclosed objects or the background not explicitly traced, and what happens when these reflections matter? Why are multiple transparent objects or nested layers, such as liquid in a glass, not handled, and how would the method behave in those cases? The introduction's examples at line 28, “fish in a tank or specimens in a glass jar,” seem to be situations the method cannot actually handle.

  2. The experimental scope is very limited, even including the appendix. The main paper focuses on existing datasets, a new synthetic dataset for the target task, and ablations. Many other experiments could be considered: reconstructed geometry, failure cases, spatially varying refractive indices, a wider range of real scenes for the target task, rough surfaces such as water, or scenes with many enclosed objects. (These are illustrative possibilities, not a claim that every suggested experiment is necessarily useful or required.) Broader experiments are needed to demonstrate the method's advantages and generalization.

Questions for authors
  1. The method requires masks for objects that cause refraction and reflection. These effects are tracked only within a limited region and are not handled elsewhere. Obtaining the masks is also a practical problem: synthetic masks are easy to obtain, but real foreground masks are difficult, and manual segmentation requires substantial effort. Automation with SAM 2 does not eliminate this problem, since existing video segmentation models often lose tracking under occlusion. Consequently, even with a single foreground object, the method is limited to carefully arranged laboratory settings in which the object remains clearly visible from all views throughout a full rotation.

  2. The foreground SDF is initialized using only masks. Masks constrain the observed silhouettes but cannot recover concave structures that do not appear in those silhouettes. Is the method therefore limited to simple objects without such hidden concavities, for which mask-based initialization is already sufficient despite subsequent refinement? The curvature regularization in Eq. 9 also favors smooth surfaces, reinforcing my concern about recovering these structures.

  3. In Figure 1, Stage 2 appears to receive image supervision without enclosed objects, which is confusing. The refinement discussion around line 287 and the supplied code establish that the original images containing enclosed objects are used without identifying or excluding the affected pixels. Enclosed objects could therefore influence the reconstructed geometry: does the surface retain the intended shape, or do spurious layers appear? Such results should be shown.

  4. A previously unaddressed setting does not by itself make a paper good or justify acceptance. The paper should establish a practically needed problem that others have tried but struggled to solve, introduce an idea that addresses it, and demonstrate through varied results and analysis how that idea succeeds. Instead, this paper uses several assumptions to reduce the problem to an easier, smaller subset, then works around it through a sequence of stages built from existing techniques. The evidence remains concentrated on the main experiments and does not sufficiently show whether the model works in varied situations.

  5. The reason for separating the first and second stages is unclear. The input masks could be used instead of rendered masks, and it may be possible to learn only the SDF or to learn the surfel and SDF representations together. Jointly training all three stages could make geometry unstable because of ambiguity, but the need to separate the first two stages is less apparent.

  6. At the transition from Stage 2 to Stage 3, all foreground surfels are reintroduced, including those representing the refractive surface already modeled by the SDF rather than only enclosed objects. How are these refractive-surface surfels handled? Are opacity reset and the subsequent optimization and pruning expected to make them disappear?

1 comment
AuthorsOpenReview ↗

Response to Reviewer egTR

We thank the reviewer for the insightful and constructive comments.

C1. Assumptions (W1, W2, Q2)

Please refer to our Response A2 to Reviewer aJLJ.

C2. Relying on Masks (Q1)

As discussed in our Response A2 to Reviewer aJLJ, existing methods also rely on masks or scene priors, and Appendix A.5 shows that our method is robust to imperfect masks.

C3. Stage 2 Optimization (Q3)

Stage 2 uses the original input images for supervision, following the practice of Eikonal Field.

C4. Significance of the Problem (Q4)

Refractive scenes, such as specimens embedded in resin, are common in the real world. Our method outperforms previous methods on scenes both with and without enclosed objects.

C5. SDF Initialization from Rendered Masks (Q5)

Directly fitting the SDF to input masks cannot distinguish occluded regions from regions outside the refractive object.

C6. Surfel Optimization in Stage 3 (Q6)

The surfel optimization relies on multi-view image supervision, pruning, and opacity reset.

Reviewer_su4sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a reconstruction algorithm for scenes containing a refractive object, potentially enclosing opaque objects, from multiview images and foreground masks. The scene components -- the refractive object, the enclosed objects, and the background -- are represented separately, and their respective representations are obtained through a multi-stage optimization pipeline. In the first stage, a masked radiance field is optimized using a Gaussian surfel representation. In the second stage, a neural SDF of the refractive object is optimized using SDF-constrained volume rendering with synthetic masks as supervision. In the third stage, both the SDF and a scalar index of refraction are optimized using photometric supervision. Finally, the enclosed objects are optimized using a surfel-based representation. The results are evaluated on a reasonable collection of synthetic and real-world datasets, demonstrating state-of-the-art performance.

Critical strengths

I appreciate the problem setting, which gives rise to challenges that most existing approaches are not equipped to address properly. Advances in this area are clearly needed, and this paper contributes to such progress.

The presented approach quite clearly establishes a new state of the art. Both qualitative and quantitative results (please also see my corresponding comments in the Weaknesses section) convincingly demonstrate that the method outperforms existing approaches, often by a significant margin.

Reproducibility is supported by the availability of the source code, although I did not run it myself.

Overall, the paper is well written (please also see my corresponding comments in the Weaknesses section) and therefore relatively easy to follow.

Critical weaknesses

I am not sure whether the distribution of content between the main paper and the appendix is optimal and reader-friendly. In particular, one could argue that Sections 3.2 and 3.3 might be unnecessary in the main paper, since the detailed equations (e.g., those describing how certain intersections are differentiated) are never actually built upon beyond being mentioned. On the other hand, certain implementation details in the appendix might be more useful in the main paper.

I believe Sec. 4.1 could more explicitly acknowledge that what is happening here is essentially radiance field reconstruction, i.e., optimization based on an emission-absorption model. I am surprised that this reconstruction does not give rise to floaters, since the light transport assumptions are clearly violated. My suspicion is that this problem might, in part, be mitigated by the fact that, although the considered objects are not necessarily globally convex, they appear to exhibit relatively simple geometry from the observed viewpoints, avoiding pronounced concavities or complex self-occlusions.

I am not completely convinced that the gradients w.r.t. the IOR in Sec. 4.2 can provide useful information. In particular, the photometric loss propagated through ray parameters inherently leads to a highly non-convex energy. This is why, for example, Eikonal Fields incorporated a multi-scale routine to counteract this effect. I wonder how stable IOR reconstruction is and how close the estimated IOR is to the ground truth in synthetic scenes.

A more general comment concerns the evaluation protocol. Reporting novel-view synthesis quality using established image quality metrics is inherently problematic in this setting. My claim is that the quality of the refractive flow should be among the actual quantities of interest, which is not captured by such metrics. This is evidenced by the comparatively high scores of 3DGS in this setting. As a remedy, some baselines consider user studies for this problem, which was not done here. Further, I would encourage the authors to consider measuring image quality only for pixels representing the refractive object.

Minor points:

  • The last sentence in Sec. 4.1 seems better placed at the appropriate point in Sec. 4.2.
  • The SDF-to-density mapping (L258) should cite the original ideas presented in "Volume Rendering of Neural Implicit Surfaces" by Yariv et al. and "NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction" by Wang et al. (both NeurIPS 2021).
  • It is only explained in Sec. 6 that the authors consider the IOR to be constant. This assumption should be communicated transparently early on.
Questions for authors

How important is ray tracing in Sec. 4.1? Would the mentioned discrepancies actually matter?

1 comment
AuthorsOpenReview ↗

Response to Reviewer su4s

We thank the reviewer for the insightful and constructive comments.

B1. Stage 1 in Sec. 4.1 (W1, Q)

Training Stage 1 with ray tracing outperforms splatting, as shown in Tab. R1.

B2. IoR Optimization (W2)

The IoR optimization is trainable because the full pipeline is differentiable. The recovered IoR is robust to its initialization, as shown in Tab. R2.

B3. Evaluation Protocol (W3)

We report PSNR on the refractive region, as shown in Tab. R3, and our method outperforms all baselines on all three datasets.

Table R1. Ablation of the renderer used in the first stage on the Eikonal Field dataset. Ray tracing is our default. Average gives the equal-weight mean over the four scenes. Better results are highlighted in bold.

Ray TracingSplatting
PSNR↑SSIM↑LPIPS↓PSNR↑SSIM↑LPIPS↓
Ball24.290.8750.06523.150.8300.092
Glass24.790.8630.08323.680.8200.103
Pen23.370.8230.11021.710.7570.146
WineGlass23.130.8580.13221.390.7970.184
Average23.900.8550.09722.480.8010.131

Table R2. IoR recovered from different initial values on the TNSR dataset. Mean abs. error denotes the mean absolute IoR error over Optical Ball, Ellipsoid, and Bottle, calculated from unrounded final values.

SceneGT IoRInit. 0.8Init. 1.0Init. 1.4Init. 1.6Init. 1.8Init. 2.0Init. 2.2
Optical Ball1.451.45051.45051.45061.45051.45051.45051.4505
Ellipsoid1.201.20051.20051.20051.20051.20051.20051.2005
Bottle1.151.15051.15081.15081.15071.15071.15081.1506
Mean abs. error–0.00050.00060.00060.00060.00060.00060.0005

Table R3. Foreground PSNR (dB), computed only on the refractive regions. Bold marks the best result in each row.

DatasetOurs3DGSEikonal FieldTNSRNU-NeRFTransparentGS
TNSR21.0215.2813.7317.2514.6414.47
Eikonal Field19.3318.1617.8915.5217.7311.48
Our Synthetic22.9720.1717.0316.4718.8915.67
Reviewer_aJLJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for the task of novel view synthesis for scenes with a refractive object that contains an opaque object, given posed images and refractive object masks. The approach first models the scene with a Gaussian mixture model (surfels) under the standard straight-ray assumption and a photometric loss, while also fitting the foreground probability of each Gaussian with a mask loss - this fits the geometry and appearance of the opaque background well, the geometry of the refractive object roughly, and the appearance of the refractive object poorly. Then an SDF is fit to the foreground Gaussians, before being refined (alongside the refractive index) by differentiating through the refractions and reflections. Finally, any enclosed object is modelled with a further set of Gaussians, with everything else fixed. This is a challenging problem, because photometric signals are local, but geometry changes can lead to wildly different ray--background intersection locations. The method is tested on four real and seven synthetic scenes, half of which contain enclosed opaque objects. Several state-of-the-art are compared, including 3DGS, NU-NeRF, Eikonal Fields, TNSR, and TransparentGS, and the proposed method outperforms these approaches.

Critical strengths
  1. Motivation. Modelling scenes that have opaque objects inside refractive media may seem niche at first, but this is something that we experience whenever we reach for a ball in the water and fail to grasp it, fooled by the refraction. The authors give examples of this, including reconstructing fish in a tank and specimens in a jar. I expect that this would be of relevance for science (e.g., for monitoring the growth of fish in an aquarium) and for robotics (e.g., for grasping objects underwater).

  2. Approach. The three-stage pipeline is well-designed, reducing the ambiguity of joint optimization, supported by the ablation study.

  3. Results. The quantitative and qualitative results are strong, outperforming all compared methods, and look visually appealing.

  4. Clarity. The paper was well-written and clearly presented, making it easy to understand the approach. Moreover, the writing was quite thoughtful, making sound arguments and clearly demonstrating deep technical understanding.

Critical weaknesses
  1. Contribution. It appears that the core method is TNSR (Deng et al, 2024) with some minor updates, which is not acknowledged in the text. TNSR fits the scene with an SDF using NeuS (updated here to fitting Gaussians and then the SDF), then refines the SDF parameters by backpropagating the photometric error through the refraction and reflection rays defined by Snell's Law (the core contribution; slightly updated here to also refine the refractive index). This mirrors Sections 4.1 and 4.2 of the paper. The differences between the approaches are not well-established in the paper. They appear to be (a) IoR backprop; (b) modelling two reflections rather than one; (c) using a Gaussian scene representation rather than NeuS; and (d) separately modelling the enclosed object with Gaussian surfels. The first two are minor (albeit sensible) alterations of TNSR that follow directly from its setup; (c) updates TNSR by integrating an existing SOTA approach 2DGS; and (d) again uses the standard existing technique of 2DGS.

  2. Missing comparison and related work. Opti-NeuS [A] is extremely closely related, addressing the same problem of opaque objects inside transparent ones, and is state-of-the-art. Moreover, it has fewer assumptions than the proposed method, not requiring masks and being able to handle non-convex objects (eg, the cow shape). It needs to be discussed in the related work section and compared in the experiments.

  3. Assumptions. The paper makes several significant assumptions, some implicit, that limit applicability without justifying them or explaining why they are necessary.

    1. Single refractive medium. This means that the approach cannot model water in a glass, despite this setup being in the experiment section (presumably very thin glass), or the motivating example of an aquarium. This compares unfavorably to several works in the existing literature discussed in the related work section.
    2. Refraction and reflection at entry point and first internal surface hit, and up to three total internal reflection bounces. This seems reasonable at first, but implicitly excludes non-convex refractive objects, since re-entry is not modelled.
    3. Refractive object masks. Assuming that these are available and are accurate limits applicability. Showing the effect of relaxing this assumption (eg by estimating the masks and incurring some prediction errors) would be advisable.
    4. The initialize and refine approach can only handle refractive objects that are roughly the same shape as their visual hull from the masks, a hidden assumption. This is because the refinement step cannot make significant changes to the geometry, given the (local) photometric loss alone - this is the same limitation of TNSR.
  4. Evaluation.

    1. The reconstructed geometry is not evaluated (e.g., the depth map error), despite this being an output of the method, via the SDF. It is important to know if good visual fidelity is masking poor geometric quality.
    2. The 11 scenes in the evaluation are all very simple, especially compared to the non-convex and multi-material shapes in other datasets (see, eg, next point) and the quantity of scenes is too small to establish robust performance.
    3. "We do not include additional datasets such as 3DReflecNet (Liang et al., 2026) and RefRef (Yin et al., 2025), as they contain refractive scenes without enclosed objects" [L345]. This is untrue: the RefRef dataset contains many scenes with enclosed opaque objects, and is far larger than the datasets tested here (150 synthetic, 60 real) allowing for a more robust and fair evaluation.
  5. Claim: "refraction breaks multi-view photometric consistency" [L031]. This is not true, multi-view photometric consistency is preserved, but the rays are no longer straight.

  6. Missing methodological details. Equations for the light paths (eg Snell's Law, total internal reflection) are undefined, despite this being central to the approach.

[A] Yang et al., Opti-NeuS: Neural Reconstruction for Dual-Layered  Transparent and Opaque Objects, CVPR 2026

Questions for authors
  1. What distinguishes the proposed approach from the existing literature, especially TNSR?

  2. How does Opti-NeuS compare to the proposed approach?

  3. How does the approach compare with respect to geometric errors?

  4. Could the authors clarify their reasoning for excluding the RefRef dataset from their evaluation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer aJLJ

We thank the reviewer for the insightful and constructive comments.

A1. Comparison with Opti-NeuS (W2/Q2)

Opti-NeuS is not designed for novel view synthesis, and no implementation of Opti-NeuS is available.

A2. Clearer Assumptions (W3)

Single refractive medium. We assume a homogeneous refractive object with a single IoR, which we also use to approximate thin-walled liquid containers.

Non-convex objects. We agree that our silhouette-based initialization cannot handle non-convex objects, and we will state this in the limitations.

Masks. Existing methods also rely on additional input or scene priors: object masks (NeRRF, NEMTO, DiffTrans), a user-annotated 3D bounding box (Eikonal Field, its Sec. 4.1), a unit-sphere assumption (NU-NeRF, its Sec. 3.1), or the largest connected component (Opti-NeuS, its supplementary Alg. 1).

A3. Geometry Evaluation (W4)

While the SDF recovers the refractive interface, the goal of our method is novel view synthesis rather than geometry reconstruction, just like Eikonal Field.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.