F-RNG: Feed-Forward Relightable Neural Gaussians with Large Model Priors

Capturing relightable 3D assets from real-world objects is a widely researched problem. Several per-scene optimization-based methods, based on 3D Gaussian splatting (3DGS), support relighting; however, they usually require dense input views, and their overfitting nature makes it difficult to generalize across scenes. Unlike per-scene optimization methods, generalized feed-forward models can directly reconstruct Gaussians from sparse input views. However, the resulting assets have baked-in illumination and cannot be easily used for relighting. In this paper, we present \method, a feed-forward framework that directly generates relightable 3DGS assets from sparse-view inputs. Training such a model from scratch can require massive data and computing resources, and it is especially challenging to generate relightable assets in a feed-forward manner with acceptable cost. To achieve this, we develop \method upon an existing large reconstruction model (LRM) to extract relightable representations by training only small networks with guidance from an intrinsic decomposition model (IDM), without any fine-tuning or re-training of the large models. Specifically, we aim to improve the precision of the underlying geometry and decompose the light and materials. We first introduce a latent-interpolated fine-grained geometry synthesis to enhance the LRM's geometry representation. Second, we propose a prior-guided relightable appearance distillation to extract relightable neural representations by incorporating IDM priors. Finally, a universal neural renderer enables flexible and high-fidelity relighting under arbitrary novel lights. Once the asset is constructed, relighting under novel lights no longer requires the large models, avoiding the cost from their repetitive inference. Compared to the state-of-the-art LRM-based relighting method, \method achieves $\sim$25$\times$ faster relighting, as well as superior quality ($\sim$+2.0 dB).