- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to store appearance as 2D Gaussians on the mesh surface combined with environment maps (one env map per convex decomposition part). A KAN decoder evaluates the reflection equation for far-field specular. The proposed approach runs at interactive rates.
Spatially-varying surface appearance is commonly used in games and offline rendering, where the BSDF parameters are stored in texture maps. It can be evaluated using path tracing, which now runs at real-time rates with few samples per pixels on modern GPUs. It can also be approximated with techniques like split-sum shading, leveraging prefilted environment maps (the cited approach [Karis 2013]). There are also multiple approaches which store neural BSDFs in small networks (MLPs), like NeuMIP (https://cseweb.ucsd.edu/~viscomp/projects/NeuMIP/), and Real-Time Neural Appearance Models https://research.nvidia.com/labs/rtr/neural_appearance_models/ which can be evaluated at interactive rates.
I lack a comparison against a split-sum shading baseline, for example, nvdiffrec https://nvlabs.github.io/nvdiffrec/ with known geometry, which to me seems like the closest baseline.
I am currently leaning negative due to limited scope, limited novelty and missing evaluations.
Critical strengths
An efficient method for NVS of specular appearance in a rasterization-based pipeline with known geometry. It improves over 2DGS, particularly on specular examples.
Critical weaknesses
I would suggest comparing against a split-sum shading baseline, for example, nvdiffrec https://nvlabs.github.io/nvdiffrec/ with known geometry or follow-ups. Split-sum shading seems like the closest baseline here, and handles specular appearance. Perhaps:
- Split-sum shading, as applied in nvdiffrec
- Split-sum shading, with pre-convolved env maps per convex partition. compared against the proposed appearance representation.
Representing appearance at the surface has been the standard in triangle-based rendering for decades, and is used offline rendering (path tracing with BSDFs) for visual effects. While NeRF and 3DGS approaches instead use volumetric primitives for ease of optimization, I unfortunately do not see an analysis of the benefits of surface-based appearance as a contribution here.
Given the extensive related work in this field, the scope is too small in my opinion. The geometry and camera poses are given. Splitting into convex parts seems cumbersome, and unclear how that scales to larger scenes. Only NVS evaluated, not relighting, and shadows are not accounted for.
Questions for authors
The paper mentions a fixed budget of N=10^6 Gaussians per mesh. How are they distributed over the triangles? Allocated based on triangle surface area?
"Per-face texturing such as Ptex (Burley & Lacewell, 2008) removes the UV atlas, but as a static texture it still cannot capture high-frequency, view-dependent reflection." I disagree with this statement. Ptex has been used in offline visual effects for high quality rendering, and is typically applied to store BSDF parameters (albedo, roughness, ...) in texture maps and is compatible with path tracing supporting strongly specular materials like mirrors/metals. Perhaps rephrase.
It seems that there is only one lookup into the envmap in the reflection direction. What about materials with varying roughness? Are the envmaps pre-convolved/blurred to account for this? It seems that this is done in the KA-network: "the IDE is SH(ωr) attenuated by a learned per-point roughness, so a rougher surface queries the environment over a wider cone — a prefiltered, glossy lookup rather than a perfect mirror", but is the env lookup still sharp, or is the env map also provided on SH format? This could be clarified.
Recent work proposes using kernels on on the mesh surface to store texture detail, that could be cited: Heat Kernel Textures (HKT) https://circle-group.github.io/research/HeatKernelTextures/
UV Versus 2D GMM
Fundamental Difference to UV textures
A UV texture and our 2D Gaussian mixture are different representations, and their difference is one of capabilities rather than quality, which a matched-budget PSNR comparison (Table in the next section) does not capture. A UV texture is defined through a global parameterization: the mesh must be unwrapped into an atlas, and the texture’s resolution and layout are tied to it. Our mixture has no global parameterization: each element is anchored to a mesh face in barycentric coordinates and is individually addressable. This gives capabilities a UV texture structurally lacks:
Conversely, UV textures benefit from decades of hardware and tool support (hardware filtering, mipmaps, block compression). The two are therefore complementary: our representation is suited to automatically baked, element-wise assets where unwrapping is a bottleneck, such as reconstructed or frequently edited meshes.
PSNR Difference
To isolate the representation, we compare our 2D mixture with a UV texture with no decoder or environment map, since a strong decoder with positional encoding can absorb appearance and mask representation differences. Both store diffuse appearance only, without mipmaps or SSAA. 1M 2D disks versus 1730 * 1730 texels to maintain approximate the same parameters. The theoretical prediction would be our proposed 2D-GMM are anti-aliasing kernel that will doing great on complex scene. To seperate the representation difference, we only train UV-texture and 2D-GMM with no additional anti-aliasing such as mipmaps and SSAA . The UV-texture training using the backbone provided by nvidiffrec: https://nvlabs.github.io/nvdiffrec/. The test dataset is nerfSynthetic