acceptodds
Under review as a conference paper at ICLR 2027

Automatic Neural Baking

Abstract

Baking precomputes an object's appearance into a texture on a mesh, but baked formats cannot represent sharp view-dependent reflection. Neural and splat representations can, yet they place appearance in the space around the object rather than on its surface, so there is nothing to bake. We identify two independent obstructions to the natural fix, binding splats to the faces of a known mesh, and prove both are fundamental rather than incidental. First, coplanar primitives admit no compositing order that is both view-stable and geometrically meaningful, and no cheap, per-primitive perturbation of the depth, such as a normal offset, can repair this. Second, reflection about a fixed normal is a rotation of the view sphere, so any per-primitive view-dependent color model is band-limited at a fragment regardless of how many primitives are used: adding primitives adds no angular bandwidth. We resolve both with Automatic Neural Baking, a construction that works rather than the only one that could. An on-manifold kernel field, 2D Gaussian disks anchored to mesh faces and blended by an order-free partition of unity, removes the ordering obstruction; factoring view dependence out of the elements into a shared environment map and a FastKAN decoder removes the locality bound. The result composes directly with the traditional rendering pipeline, baking splat-quality, real-time appearance onto a mesh.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Graphics Baking, Gaussian Splats, Mesh, Light Field, Environment Map, Texture Map, 2DGS, 3DGS, KAN

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kby6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to store appearance as 2D Gaussians on the mesh surface combined with environment maps (one env map per convex decomposition part). A KAN decoder evaluates the reflection equation for far-field specular. The proposed approach runs at interactive rates.

Spatially-varying surface appearance is commonly used in games and offline rendering, where the BSDF parameters are stored in texture maps. It can be evaluated using path tracing, which now runs at real-time rates with few samples per pixels on modern GPUs. It can also be approximated with techniques like split-sum shading, leveraging prefilted environment maps (the cited approach [Karis 2013]). There are also multiple approaches which store neural BSDFs in small networks (MLPs), like NeuMIP (https://cseweb.ucsd.edu/~viscomp/projects/NeuMIP/), and Real-Time Neural Appearance Models https://research.nvidia.com/labs/rtr/neural_appearance_models/ which can be evaluated at interactive rates.

I lack a comparison against a split-sum shading baseline, for example, nvdiffrec https://nvlabs.github.io/nvdiffrec/ with known geometry, which to me seems like the closest baseline.

I am currently leaning negative due to limited scope, limited novelty and missing evaluations.

Critical strengths

An efficient method for NVS of specular appearance in a rasterization-based pipeline with known geometry. It improves over 2DGS, particularly on specular examples.

Critical weaknesses

I would suggest comparing against a split-sum shading baseline, for example, nvdiffrec https://nvlabs.github.io/nvdiffrec/ with known geometry or follow-ups. Split-sum shading seems like the closest baseline here, and handles specular appearance. Perhaps:

  1. Split-sum shading, as applied in nvdiffrec
  2. Split-sum shading, with pre-convolved env maps per convex partition. compared against the proposed appearance representation.

Representing appearance at the surface has been the standard in triangle-based rendering for decades, and is used offline rendering (path tracing with BSDFs) for visual effects. While NeRF and 3DGS approaches instead use volumetric primitives for ease of optimization, I unfortunately do not see an analysis of the benefits of surface-based appearance as a contribution here.

Given the extensive related work in this field, the scope is too small in my opinion. The geometry and camera poses are given. Splitting into convex parts seems cumbersome, and unclear how that scales to larger scenes. Only NVS evaluated, not relighting, and shadows are not accounted for.

Questions for authors

The paper mentions a fixed budget of N=10^6 Gaussians per mesh. How are they distributed over the triangles? Allocated based on triangle surface area?

"Per-face texturing such as Ptex (Burley & Lacewell, 2008) removes the UV atlas, but as a static texture it still cannot capture high-frequency, view-dependent reflection." I disagree with this statement. Ptex has been used in offline visual effects for high quality rendering, and is typically applied to store BSDF parameters (albedo, roughness, ...) in texture maps and is compatible with path tracing supporting strongly specular materials like mirrors/metals. Perhaps rephrase.

It seems that there is only one lookup into the envmap in the reflection direction. What about materials with varying roughness? Are the envmaps pre-convolved/blurred to account for this? It seems that this is done in the KA-network: "the IDE is SH(ωr) attenuated by a learned per-point roughness, so a rougher surface queries the environment over a wider cone — a prefiltered, glossy lookup rather than a perfect mirror", but is the env lookup still sharp, or is the env map also provided on SH format? This could be clarified.

Recent work proposes using kernels on on the mesh surface to store texture detail, that could be cited: Heat Kernel Textures (HKT) https://circle-group.github.io/research/HeatKernelTextures/

Reviewer_ksPbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for fitting view-dependent appearance to a given mesh from multi-view images. The method represents surface colors and reflectivity with a mesh-anchored 2D Gaussian mixture, using normalized blending to avoid depth-ordering conflicts between overlapping primitives. It further introduces adaptive regional environment maps and a FastKAN-based decoder to model reflections. The reported results show sharp reflections and improved novel-view rendering on NeRF-Synthetic and Shiny-Blender, particularly on highly reflective objects, while maintaining real-time rendering. The main evaluation uses ground-truth geometry and synthetic scenes.

Critical strengths

The reflective-object results are sharp, and fitting them to an existing mesh without a UV atlas is useful. The normalized blend removes primitive-order dependence, and the reported rendering rates demonstrate that this representation can be evaluated in real time.

Critical weaknesses

The methodological novelty appears limited. The method combines a surface-based Gaussian feature field, regional environment maps, and a KAN decoder, but it is unclear what modeling capability this introduces beyond increasing local representation capacity and using a stronger decoder. Without comparisons to simpler representations under the same decoder and comparable capacity, the method appears closer to an engineering refinement than a substantial methodological advance.

The ablations need clarification and additional controls. First, the 35.60 dB result is labeled MLP + AdaMap in Table 3 but MLP + M Token in Appendix Table 6. Please clarify which configuration was used. Second, please compare GMM with UV textures and fixed-index 2DGS using the same environment maps and KAN decoder to test whether the proposed representation is better. Third, please compare adaptive and non-adaptive environment-map allocation with the same number of maps, so that the benefit of adaptive allocation can be distinguished from simply using more maps.

ANB uses GT meshes, whereas Ref-Gaussian optimizes its own geometry. Please compare both methods on the same fixed mesh to evaluate appearance reconstruction. The main tests and reconstructed-mesh appendix are all synthetic. It is unclear how well the regional environment maps handle near-field reflections. Please evaluate the method on captured reflective scenes with nearby objects or light sources.

Questions for authors
  1. The 35.60 dB result is labeled MLP + AdaMap in Table 3 but MLP + M Token in Appendix Table 6. Which configuration is correct? Please also clarify whether the MLP and KAN use the same inputs and other components.

  2. Please compare GMM with UV textures and fixed-index 2DGS using the same environment maps and KAN decoder, with comparable storage and training budgets. Please also compare adaptive and non-adaptive environment-map allocation with the same number of maps.

  3. Please compare with Ref-Gaussian and a newer method such as MaterialRefGS on the same fixed mesh, using the same rendering settings. Report training time and rendering speed on the same hardware.

  4. The evaluation is limited to synthetic scenes. Please add experiments on captured reflective scenes, including nearby objects or light sources, to test near-field reflections.

9 comments
AuthorsOpenReview ↗

UV Versus 2D GMM

Fundamental Difference to UV textures

A UV texture and our 2D Gaussian mixture are different representations, and their difference is one of capabilities rather than quality, which a matched-budget PSNR comparison (Table in the next section) does not capture. A UV texture is defined through a global parameterization: the mesh must be unwrapped into an atlas, and the texture’s resolution and layout are tied to it. Our mixture has no global parameterization: each element is anchored to a mesh face in barycentric coordinates and is individually addressable. This gives capabilities a UV texture structurally lacks:

  • No unwrapping: the representation exists directly on any mesh, with no seams, charts, or packing.
  • Local resolution changes: elements can be added, removed, or merged in one region with only a local neighbor update, while a UV texture requires enlarging charts, repacking, and resampling.
  • Element-wise operations: splat tools such as NanoGS compact the asset directly (Table 4), and individual elements can be edited, deleted, or merged.
  • Robustness to mesh edits: elements on unaffected faces remain valid when part of the mesh changes, while a UV layout may need to be redone.

Conversely, UV textures benefit from decades of hardware and tool support (hardware filtering, mipmaps, block compression). The two are therefore complementary: our representation is suited to automatically baked, element-wise assets where unwrapping is a bottleneck, such as reconstructed or frequently edited meshes.

PSNR Difference

To isolate the representation, we compare our 2D mixture with a UV texture with no decoder or environment map, since a strong decoder with positional encoding can absorb appearance and mask representation differences. Both store diffuse appearance only, without mipmaps or SSAA. 1M 2D disks versus 1730 * 1730 texels to maintain approximate the same parameters. The theoretical prediction would be our proposed 2D-GMM are anti-aliasing kernel that will doing great on complex scene. To seperate the representation difference, we only train UV-texture and 2D-GMM with no additional anti-aliasing such as mipmaps and SSAA . The UV-texture training using the backbone provided by nvidiffrec: https://nvlabs.github.io/nvdiffrec/. The test dataset is nerfSynthetic

Scene2D GMMUV 1730²
chair28.1427.62
drums23.5023.59
ficus27.4127.08
hotdog31.2330.98
lego31.0130.70
materials22.2722.20
mic27.8127.69
ship25.4925.58
Mean27.1126.93
AuthorsOpenReview ↗

Main Contribution

We thank the reviewer for the careful reading. We agree the PSNR gains come mainly from the shading model (Table 3), and we do not claim the 2D mixture improves quality over a UV texture at matched capacity (see the new comparison above). Our contribution is different: we ask whether splats can be bound to a given mesh so that the mesh remains the authoritative geometry while appearance keeps the trainable, element-wise form of splats, and we show that the natural approach fails for two provable reasons that dictate the design.

  1. Ordering (Theorem B). Splats on a surface have tied depths, and no static depth perturbation gives a view-stable, geometry-respecting order. This requires an order-free blend; our partition-of-unity field is one such construction, and it gains 3.7 dB over mesh-bound 2DGS on the same mesh (Table 3).
  2. Locality (Theorem A1). Per-primitive view-dependent color has an error floor at each fragment that adding primitives cannot lower. This requires moving view dependence into a shared representation read per fragment, which is the role of the environment maps and the decoder (Prop. A4).

The components are therefore not independent additions: the order-free surface field follows from Theorem B, and the shared environment and decoder follow from Theorem A1. The result is a splat-style representation that is a true surface field: the geometry is the input mesh by construction and can be refined through the same appearance gradients (App. A.3), while elements remain individually addressable, compactable with splat tools, and free of any UV atlas.

AuthorsOpenReview ↗

Fix-Order Run

In this run, we evaluate all with SSAA equals to 2, indeed, the popping has been solved for both 2D GMM and 2DGS* fix-index order representation. But fixed-index order cannot represent the view dependent information due to the collapse of degree of freedom.

The test dataset is shinyBlender

Representation on fixed mesh (Shiny-Blender, SSAA2)View-stablePSNR ↑
2DGS*, depth-sortedNo (popping)27.70
2DGS*, fixed-index orderYes26.89
2D GMM, order-free blend (ours)Yes31.40
Reviewer_H4eyOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Automatic Neural Baking learns view-dependent appearance on a given mesh. Gaussian disks store diffuse color and reflectivity on the surface and are blended without depth sorting. Adaptive environment maps and a FastKAN decoder produce reflections. The method avoids a global UV map. The experiments cover image quality and a mesh-refinement example.

Critical strengths

The representation is useful. It learns appearance directly on the mesh, avoids a UV atlas, and removes the need to sort overlapping disks.

The results on shiny objects are strong, with 38.04 dB on Shiny-Blender and 76.8 FPS at 800 × 800 on an RTX 4090.

Critical weaknesses

The current ablations do not fully separate the benefit of the surface representation from the environment maps and KAN decoder.

Theorem A1 supports the fixed-order SH case. For SG mixtures, more primitives also mean more lobes, so the same conclusion does not follow directly.

Continuity across triangle edges is not fully explained. Both the neighbor sets and environment-map weights can change between faces.

Questions for authors

Could you test fixed-index 2DGS* with the same environment maps and KAN decoder, as discussed in the appendix?

Could you relate your analysis to "MLP-Splatting" and "When 3D Gaussian Splatting Recovers Real Surfaces", and clarify the main differences in assumptions?

How do you avoid seams between adjacent faces with different neighbor sets or environment-map weights?

5 comments
AuthorsOpenReview ↗

Fix-Order Run

In this run, we evaluate all with SSAA equals to 2, indeed, the popping has been solved for both 2D GMM and 2DGS* fix-index order representation. But fixed-index order cannot represent the view dependent information due to the collapse of degree of freedom.

The test dataset is shinyBlender

Representation on fixed mesh (Shiny-Blender, SSAA2)View-stablePSNR ↑
2DGS*, depth-sortedNo (popping)27.70
2DGS*, fixed-index orderYes26.89
2D GMM, order-free blend (ours)Yes31.40

Thank you for the experiments and the table. This gives me two important pieces of information that are helpful.

  1. 2d gmm are indeed reducing popping and have a very large gain in terms of PSNR.
  2. 2dgs* with fixed order has worse PSNR than depth-sorted 2dgs*.

I think it eliminates my concern. Follow-up: What do you exactly mean by "collapse of degrees of freedom"? If the primitive has SH or an MLP predictor, it can still have view dependency with fixed order. Also, just for clarification, have you used the same environment maps and KAN setting in the experiments?

AuthorsOpenReview ↗

Experiments Setting and Collapse of degrees of freedom

Thank you for the follow-up; we are glad the results address your concern. We clarify both points below.

Experimental setting. No, the fixed-index experiment does not use our environment maps or KAN decoder. To isolate the effect of the compositing order, all three variants use the same minimal setting:

Setting2DGS*, depth-sorted2DGS*, fixed-index2D GMM (ours)
Appearance per primitiveRGB onlyRGB onlyRGB only
Environment mapsNoneNoneNone
DecoderNoneNoneNone
Training30K iterations, same initialization, no densification or pruningsamesame

The only difference between the variants is how primitives are combined at a fragment: depth-sorted compositing, fixed-index compositing, or our order-free partition-of-unity blend.

What we meant by "collapse of degrees of freedom." We agree our wording was imprecise, and your observation is correct: with SH or an MLP per primitive, a fixed order still allows view dependence. Our statement applies to the RGB-only setting above. There, each primitive's color is view-independent, so the only source of view dependence in depth-sorted 2DGS* is the reordering itself: as the view rotates, the front-most primitive at a fragment changes, so the rendered color changes with the view. On reflective scenes the optimizer exploits this, at the cost of popping. A fixed-index order removes the reordering, and with it the only view-dependent mechanism available in this setting, so the rendered appearance becomes strictly view-independent. This explains why removing the popping also lowers PSNR (27.70 → 26.89 dB), as predicted by Corollary A3 (App. A.1).

With per-primitive SH, a fixed order does regain view dependence, but it is then bounded by Theorem A1: whenever the blend weights at a fragment are view-independent (a fixed order or our partition of unity), the result lies in the span of the per-primitive SH functions, and its error has a floor that adding primitives cannot lower. Sorted compositing escapes this floor only through reordering, which is exactly what produces popping (Corollary A3). This is why our method does not attach view dependence to individual primitives at all, and instead reads it from shared environment maps through a per-fragment decoder (Prop. A4).

Fixed-index with our environment maps and KAN. We have not yet run the fixed-index variant with our full shading model. Our analysis predicts that view dependence would then come from the shared environment maps and decoder in both cases, so the comparison would test whether fixed-index compositing of the surface attributes (albedo and reflectivity) matches our order-free blend. We will include this experiment in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Progress

Summary of the discussion

We thank all reviewers for their detailed and constructive feedback. Below we summarize the new experiments, clarifications, and revisions; details are in the individual responses. In the revised PDF, rewritten text is in blue and new content in dark green.

New experiments (all added to the revision)

  • Split-sum baseline (nvdiffrec) on the same ground-truth meshes (Reviewer kby6): 38.04 vs. 31.70 dB mean PSNR on Shiny-Blender at 2× SSAA, better on every scene and metric.
  • Split-sum variants within our pipeline (Reviewer kby6): learned mip blending 37.92 dB, learned spherical beta kernel 37.91 dB, and nvdiffrec's full split-sum model 33.70 dB, vs. 38.04 dB for our decoder. Full split-sum trains about 3.5× faster, at a cost of 4.3 dB.
  • Fixed-index 2DGS (Reviewers H4ey, ksPb): in the same RGB-only setting, fixed-index ordering removes popping but drops to 26.89 dB, vs. 27.70 dB for depth-sorted 2DGS* and 31.40 dB for our order-free blend. Reviewer H4ey indicated that this resolves their concern.
  • Ref-Gaussian on the same fixed mesh (Reviewer ksPb): 30.08 dB vs. 38.04 dB for ours on Shiny-Blender.
  • Real captured scenes with reconstructed geometry (Reviewer ksPb): on the Ref-NeRF real scenes, with a mesh reconstructed by Ref-Gaussian, ours reaches 24.12 dB vs. 23.68 dB for Ref-Gaussian on the same mesh, and 24.61 dB with our mesh refinement (new Sec. 4.3).
  • UV texture vs. our 2D mixture at matched parameter count (Reviewers kby6, ksPb): equivalent quality (26.93 vs. 27.11 dB on NeRF-Synthetic) and speed (101.8 vs. 98.8 FPS). The difference lies in capabilities: no atlas, local resolution changes, and element-wise editing.
  • Demonstrations (Reviewers kby6, ksPb): scene-scale rendering in Blender, relighting by environment-map replacement and rotation (video), and local editing (video).

Clarifications (all reflected in the revision)

  • Setting: this is a baking method, so geometry is given, as in DNR and nvdiffrec with known geometry. No method uses the ground-truth textures of synthetic assets; appearance is always recovered from images.
  • Contribution: quality gains come mainly from the shading model (Table 3); we do not claim a quality gain over UV textures at matched capacity. The design follows from Theorems A1 and B: an order-free surface field that keeps the mesh as authoritative geometry while remaining element-wise and atlas-free.
  • Table 3 vs. Table 6: the 35.60 dB configuration is MLP + AdaMap; "M Token" was a leftover label and Table 6 is relabeled. The MLP and KAN decoders take identical inputs; only the trunk differs.
  • Disk allocation: disks are initialized uniformly over the surface, so each triangle's share is proportional to its area.
  • Rough materials: the environment fetch is a single sharp lookup; glossiness comes from the decoder's roughness-attenuated IDE input, and prefiltered variants give the same quality.
  • Convex decomposition is automatic and takes 30–60 s per asset.
  • Theorem A1 for SG mixtures (Reviewer H4ey): we agree it does not carry over directly and removed that claim; a new remark shows the SG limit is identifiability (at most N_v constrained samples per fragment), consistent with the SG ablation.
  • Continuity across faces (Reviewer H4ey): the edge jump is bounded by the weight of disks gathered by only one face, and by 2h/τ for the environment weights (new appendix section).
  • Fixed-index with full shading: the per-primitive layer then stores only view-independent attributes, exactly the RGB-only setting where fixed-index already loses 4.5 dB, so we did not adopt it.

Related work and wording (revised)

  • Discussed "When 3D Gaussian Splatting Recovers Real Surfaces" and MLP-Splatting; cited Heat Kernel Textures, NeuMIP, Real-Time Neural Appearance Models, and MaterialRefGS.
  • Rephrased the Ptex statement and expanded Scope and limitations (baked illumination, relighting, near-field parallax, large scenes).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.