acceptodds
Under review as a conference paper at ICLR 2027

Differentiable Shadow Fields: Ray-Calibrated Visibility for Inverse Rendering

Abstract

Visibility is fundamental to rendering and inverse rendering (IR), where occlusion must be repeatedly evaluated as lighting, geometry, or scene parameters change. Ray tracing provides accurate visibility, but requires repeated geometric intersection tests and can become costly under large numbers of queries. Precomputed methods such as shadow maps are efficient, but light-dependent, update-sensitive, and not naturally differentiable with respect to light positions. We introduce Differentiable Shadow Fields (DSF), a ray-calibrated, light-independent visibility representation for IR. DSF amortizes scene geometry into a shared scalar field and evaluates visibility along finite segments between surface points and arbitrary light endpoints. Starting from an unsigned distance field, we calibrate the representation and its differentiable query function using ray-traced visibility supervision. This allows DSF to approximate ray-traced visibility while avoiding source-mesh intersection tests at inference time. Once constructed, DSF supports arbitrary light queries without rebuilding and remains differentiable with respect to light positions. Experiments show visibility and shadow accuracy close to ray tracing with substantially lower query cost, and demonstrate its use in shadow-driven light localization, inverse-rendering pipelines, and compositional scene editing

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visibility representation, inverse rendering, differentiable rendering, ray tracing, distance fields, light localization, scene editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KZvgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DSF (Differentiable Shadow Fields), which is a dense grid that is optimized to represent visibility queries for ray-geometry intersections. The grid can be queried efficiently, and the authors show that it improves performance on inverse rendering baselines.

Critical strengths

The concept of fitting a visibility field to account for the specific ray-queries is novel, and the proposed framework integrates as a general module with other IR frameworks.

Critical weaknesses

Major:

  • Each DSF must be fit per-scene, which is very expensive (several minutes, as reported in Tab 1). In addition, DSF requires geometry and fits a dense voxel grid - it is therefore unclear how it will scale to large scenes (a lot of compute / storage would be wasted in free space). The current framework uses a resolution of 1024^3, which in float32 already is 4GB (as per Tab.30). The implications here are not sufficiently discussed, especially since most results are shown on small, single- or few-object scenes. More realistic applications/scenes like Crytek Sponza or Bistro or the ZipNeRF scenes with more clutter, and an analysis of grid resolution and potential artifacts, would have made a more compelling paper.

  • Also, DSF wants to integrate with existing IR frameworks, but the end-to-end-improvement is not sufficiently demonstrated. Fig.9 and 10 show only minor visual improvements, sometimes DSF even introduces small artifacts (on mustard / plate bottom silhouette, Fig.9), while increasing the rendering time of the methods (e.g., IRGS and LumiGauss, A.4.2). Similarly, DSF-Adam achieves the same success rate as Density in Tab.13, and while it admittedly reduces the median error by 2x, the total time is almost 4x slower.

  • Regarding the gradients: differentiability is one of the main selling points of the paper, it is even included in the paper title. Yet, at no point are the actual gradients visualized or numerically validated. The CMA-vs-Adam comparison does not isolate the practical benefit of the proposed endpoint gradients either: CMA is run with more objective evaluations (1,200 vs. 300 Adam steps) and higher time budget (roughly 2.9x longer). This shows that the DSF objective is useful, but not that the gradients are particularly useful. To show the latter, I would want to see optimization-success-vs-walltime convergence curves. Also, CMA-ES is a gradient-free optimizer, including a BVH+CMA version of this comparison would show how much DSF contributes to the optimization success.

  • Another issue is that the paper mixes claims from different versions of DSF: the high-throughput results mentioned as headline claims of the paper are achieved with the texture-version, whereas the differentiability, one of the main motivations, is achieved with the significantly slower differentiable version. This conflates things and makes the paper harder to parse, since the experiments do not establish a single implementation that simultaneously provides the advertised throughput and differentiability.


Minor:

  • The teaser figure does not show the GT (exact intersection / GPU-BVH), which makes it hard to estimate the absolute quality of the shown results.
  • Regarding the GPU-BVH baseline: it is implemented as a software CUDA closest-hit implementation, not hardware-accelerated any-hit shadow tracing (A.2.1), so conclusions w.r.t. actual production-grade GPU baselines are limited. For visibility, I would image any-hit to be sufficient, since the ray can only be blocked once anyway, so closest-hit is unnecessarily restrictive for this baseline.

Writing:

The writing is subpar, and hard to understand. First of all, the introduction does not clearly define the problem, and makes the reader work unnecessarily hard. The math in the introduction (L.71) arrives before the problem, keep the concept abstract here. The concept of “finite segment" (line connecting hitpoint and light) should be explained earlier, for readers unfamiliar with the field. In general, the writing uses many redundant and bloated phrases, which makes the content hard to understand and unfortunately often sound very AI-generated. Examples:

L.68–69: “We therefore calibrate the field directly through the final visibility query, allowing it to adapt to the evaluator rather than preserving exact metric distance.” Similarly, L. 88–90: “DSF jointly calibrates a shared spatial field and its endpoint-conditioned query against ray-traced visibility, optimizing the representation for finite-segment visibility rather than metric distance reconstruction.”

Why not simply say the much clearer “we optimize grid values and query parameters against ray-traced visibility, and allow them to depart from geometric distances”?


Related work: The following two references are relevant and should be cited:

  • Neural Bounding, Liu et al., SIGGRAPH 2024. Relevance: learn a conservative bounding network around shapes to avoid expensive ray-geometry queries.
  • Sphere Carving: Bounding Volumes for Signed Distance Fields, Schott et al., SIGGRAPH 2025. Relevance: construct bounding volume for signed distance fields via SDF queries.
Questions for authors
  • In the video, why are there artifacts in the shadows? E.g., around the 0:13 - 0:15 minute mark, and also very visible around 1:25 - 1:40. Also, why is the discretization in the shadow so visible (around 1:07min) on the lower right hand side of the silhouette? I thought the calibration had the job of removing such artifacts.
  • How does the method scale to bigger scenes, eg Crytek Sponza or Bistro? What are the limitations in terms of memory? Is the RTX6000Pro necessary, what are the peak VRAM requirements? Adding roughly 1B params (1024^3 dense grid) to inverse rendering methods feels expensive.
  • I get that we’re using a dense grid for throughput efficiency, but comparing to a neural field is an experiment/ablation that’s a key missing ingredient. Using a neural field would allow better space management (almost no capacity wasted on empty regions), finer query granularity, compression/compactness, etc, and would help establish whether the dense grid is really necessary. Note that this is different from the Hash-MLP baseline, which directly predicts visibility (like NIF).
Reviewer_Yc9MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Differentiable Shadow Fields (DSF), which represent the visibility between a surface point and a light endpoint and can be used for inverse rendering applications. The proposed method first initializes an unsigned distance field on a grid and samples finite segments along the ray between the surface point and the endpoint. To correct the errors introduced by UDF interpolation inaccuracy and finite sampling, the method designs a calibration process that optimizes trainable offset parameters for the grid values and several weighting and threshold parameters to compute a soft, differentiable visibility value. The paper shows that their proposed DSF supports efficient computation of endpoint gradients and modular composition from multiple independently constructed fields.

Critical strengths
  1. The overall design is reasonable and well validated by ablation studies. The calibration idea is shown to be effective in mitigating the error caused by inaccurate UDF interpolation and finite sampling.
  2. The support for endpoint gradient computation and modular composition looks promising.
Critical weaknesses
  1. The method requires maintaining a dense 3D grid data structure, which is extremely memory-intensive. The default 102431024^3 grid resolution is already very large, and the authors do not report the memory usage. In addition, quality is very sensitive to grid resolution; thin structures and high-frequency detail can be easily missed when discretized on a grid, even at the highest tractable resolution in large-scale scenes.
  2. The UDF representation cannot distinguish between rays passing through the surface (collision) and rays passing tangent to the surface (non-collision), since both would contain segments with small UDF values decreasing to near 0 and then increasing. Eq. 3's computation may consider a tangent ray as an invisible ray, which is incorrect.
  3. The method requires the scene geometry to be known to construct the UDF field, so its inverse rendering application is limited to scenarios where the underlying geometry is known. More practical inverse rendering settings, e.g., inverse rendering from images only, may not be supported.
Questions for authors
  1. The method requires a finite-segment sampling along the ray. Is the visibility output sensitive to the placement of the segments? Do different segment sampling schemes produce consistent visibility outputs?
  2. The method claims that they choose UDF over SDF to support open surfaces (L187-189), but they do not include any experiments to demonstrate this advantage. Meanwhile, Critical Weakness #2 above points out a significant issue with UDF. The choice of UDF is not well justified.
Reviewer_YC35OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents differentiable shadow fields (DSF), which represents visibility between a surface point and a light endpoint. The proposed method stores unsigned distances to scene geometry on a discrete 3D grid shared across light configurations. The DSF and its differentiable query function are calibrated using ray-traced visibility supervision, which supports queries on the field to approximate ray-traced visibility without traversing the source mesh. Experiments evaluate visibility accuracy, query cost, differentiable light optimization, reuse across inverse rendering systems, and modular visibility composition.

Critical strengths
  1. The idea of ray-calibrated finite-segment visibility is interesting. The proposed method directly calibrates the spatial field through the endpoint-conditioned query against ray-traced visibility rather than preserving exact metric distance.

  2. The proposed method supports visibility prediction, light position optimization, reuse across different inverse rendering methods, and object editing through rigid composition and local field updates.

Critical weaknesses
  1. The DSF takes a very long time to prepare, and the non-rigid objects require updates, which restricts the application in the scenes with deformable objects.

  2. The proposed method can only be used to render shadows or ambient occlusion, which restricts its realism because full light transport cannot be simulated.

Questions for authors
  1. In the light position optimization experiment, why not compare with physically-based differentiable renderers such as Mitsuba 3 and PSDR?

  2. Can the proposed DSF be applied to full light transport solutions such as path tracing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.