Inverse Rendering without Training from Scratch: Unsupervised Post-Training of Feedforward 3D Gaussian Models

Despite the great success of feedforward 3D Gaussian models in scene reconstruction, their learned representations cannot provide editable physical information. While optimization-based inverse rendering methods are able to produce these attributes, they are not compatible to feedforward paradigm. To close this gap, we introduce an unsupervised post-training framework that adapts feedforward 3D Gaussian models to inverse rendering without training a new network from scratch. We first analyze the propagation of geometry errors from off-the-shelf Gaussian scaffold produced by feedforward models to PBR maps, and are motivated to develop a post-splat decoding paradigm. Specifically, the features produced by feedforward 3D Gaussian models are aggregated to construct scene memory. Afterwards, a learnable adapter is introduced to generate view-independent material latents conditioned on the scene memory, which are then decoded into PBR parameters after splatting. Experiments across seven backbones demonstrate that our framework achieves state-of-the-art performance against previous approaches with only 1.5%–7.0% additional inference parameters. Moreover, the ablation experiments validate the effectiveness of our post-splat decoding paradigm and scene memory, while qualitative results on real-world scenes illustrate promising relighting and illumination editing results.