Self-supervised Separate Modeling of Lighting and Appearance for Photorealistic Portrait Relighting

Existing portrait relighting methods often rely on a highly ambiguous inverse rendering process to estimate the face geometry, reflectance, and lighting, making them prone to visual artifacts. Moreover, their reliance on synthetic paired training data further leads to performance degradation and insufficient generalizability. In this work, we present a self-supervised portrait relighting framework that allows to produce photorealistic results by separately modeling lighting and appearance. To this end, we design appearance adapter and geometry-aware light modulator to obtain appearance and lighting embeddings that are able to reconstruct the input portrait. To avoid possible entanglement of lighting and appearance, we further refine the separation with a light perturbation based reconstruction optimization. Experiments show that our method outperforms existing methods in visual fidelity and lighting coherence, and generalizes well to diverse real-world and even out-of-domain portraits (e.g., cartoon and animation ).