acceptodds
Under review as a conference paper at ICLR 2027

Self-supervised Separate Modeling of Lighting and Appearance for Photorealistic Portrait Relighting

Abstract

Existing portrait relighting methods often rely on a highly ambiguous inverse rendering process to estimate the face geometry, reflectance, and lighting, making them prone to visual artifacts. Moreover, their reliance on synthetic paired training data further leads to performance degradation and insufficient generalizability. In this work, we present a self-supervised portrait relighting framework that allows to produce photorealistic results by separately modeling lighting and appearance. To this end, we design appearance adapter and geometry-aware light modulator to obtain appearance and lighting embeddings that are able to reconstruct the input portrait. To avoid possible entanglement of lighting and appearance, we further refine the separation with a light perturbation based reconstruction optimization. Experiments show that our method outperforms existing methods in visual fidelity and lighting coherence, and generalizes well to diverse real-world and even out-of-domain portraits (e.g., cartoon and animation ).

Primary area:
generative models
Keywords:
Relighting, Portrait Relighting, Self-Supervised, Disentanglement, Consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nADVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a self-supervised framework for portrait image relighting that separately models lighting and appearance. The experiments show that it outperforms the methods included in the comparison. Furthermore, the framework performs well on out-of-domain portraits.

Critical strengths
  • The self-supervised framework does not require paired training data, making it flexible enough to accommodate different data sources.
  • The proposed appearance adapter and geometry-aware light modulator separately model appearance and lighting, achieving good identity consistency and lighting control.
Critical weaknesses
  • The comparison with previous relighting methods is not convincing to me. In the qualitative comparison, only DiFaReli is specifically designed for 2D portrait relighting. It would be better to include more 2D portrait relighting methods, such as Total Relighting [1], Lux Post Facto [2], and POLAR [3].
  • The relit portraits appear slightly over-smoothed, which makes me question whether the trade-off between avoiding supervised data and visual quality is worthwhile in this self-supervised framework.

I would like to read the other reviewers' comments and the authors' rebuttal. I may raise my score if the authors can address some of my concerns.

[1] Pandey et al., Total Relighting: Learning to Relight Portraits for Background Replacement. SIGGRAPH 2021.
[2] Mei et al., Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset. CVPR 2025.
[3] Chen et al., POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling. CVPR 2026.

Questions for authors
  • Why is DiffusionLight needed to estimate the target HDR map for DiffusionRenderer? I believe the target spherical-harmonic illumination could be directly converted into an HDR map. In addition, DiffusionRenderer results are missing from the qualitative comparison.
5 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the thoughtful and constructive feedback. We address the concerns and questions below.

W1. Comparison with additional 2D portrait relighting methods.

We thank the reviewer for this suggestion. Note, as the three suggested methods (Total Relighting [1], Lux Post Facto [2], POLAR [3]) are neither open-sourced, nor provide quantitative results on public datasets, so a controlled comparison with them is not feasible. We therefore compare with three other 2D portrait relighting methods, DreamLight [4], DPR [5], and GeomConsistentFR [6]. As shown in the table below, our method outperforms all the three methods on every metric and on all three benchmarks. We in Fig. 22 further show that our method can produce superior portrait relighting results than these methods.

MethodCelebA-HQSynth. AllSynth. Foreground
ID↓Light↓DepthError↓FID↓PSNR↑LPIPS↓SSIM↑PSNR↑LPIPS↓SSIM↑
DreamLight0.2522.3860.08238.72412.7610.3320.73115.9010.1460.696
DPR0.1811.4310.04532.68718.2080.1950.85519.6110.1310.873
GeomConsistentFR0.1982.8820.03829.16220.3760.1810.84618.6020.1390.728
Ours0.0791.1240.03721.94623.1760.1490.88420.5700.1180.878

[1] Pandey et al., Total Relighting: Learning to Relight Portraits for Background Replacement. SIGGRAPH 2021.

[2] Mei et al., Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset. CVPR 2025.

[3] Chen et al., POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling. CVPR 2026.

[4] Liu et al., DreamLight: Towards Harmonious and Consistent Image Relighting. NeurIPS 2025.

[5] Zhou et al., Deep Single-Image Portrait Relighting. ICCV 2019.

[6] Hou et al., Face Relighting with Geometrically Consistent Shadows. CVPR 2022.

W2. Slightly over-smoothed visual results.

We thank the reviewer for this careful observation. We found that this issue is led by the DDIM inversion error, rather than our self-supervised framework. Note, as shown in the Fig. 25, replacing DDIM with reversible sampler BELM [7] can help remove the over-smooth issue. We have updated the visual results in paper accordingly.

[7] Wang et al., BELM: Bidirectional Explicit Linear Multi-step Sampler for Exact Inversion in Diffusion Models. NeurIPS 2024.

Q1. Compare with DiffusionRenderer using HDR maps converted from spherical-harmonic lighting.

We have followed the suggestion to conduct the comparison. As shown in the table below, the performance of DiffusionRenderer degrades substantially under this condition, indicating that a direct SH-to-HDR conversion alone is not suitable for DiffusionRenderer. Additionally, Fig. 22 provides the visual comparison for DiffusionRenderer (DR).

MethodCelebA-HQSynth. AllSynth. Foreground
ID↓Light↓DepthError↓FID↓PSNR↑LPIPS↓SSIM↑PSNR↑LPIPS↓SSIM↑
DiffusionRenderer0.4133.0420.09373.42012.4840.4170.60214.2990.1880.517
Ours0.0791.1240.03721.94623.1760.1490.88420.5700.1180.878

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Dear Reviewer nADV,

With only about five hours remaining before the discussion period closes, we sincerely hope you could spare a moment to revisit our responses.

We have worked hard to address your concerns through additional experiments, quantitative analyses, and manuscript revisions. We truly hope that our efforts and additional results have adequately addressed your concerns.

We would greatly appreciate your further consideration and look forward to hearing from you before the discussion closes.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_aNFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised portrait relighting framework that separately models appearance and lighting through geometry-aware conditioning and illumination-perturbation consistency. The method achieves promising results in identity preservation and visual fidelity.

Critical strengths
  1. The method addresses an important problem in portrait relighting: changing illumination while preserving identity and local appearance. The qualitative comparisons demonstrate convincing preservation of facial structure, clothing, and background content, supported by favorable fidelity results on both natural and synthetic portraits.
  2. The staged design is coherent and supported by component ablations. Light-perturbation refinement improves responsiveness to target lighting, while the Detail Adapter helps recover local appearance without substantially compromising lighting control. The qualitative examples make these complementary effects clear.
Critical weaknesses
  1. Limited methodological novelty. The framework integrates several established components, and the distinct contribution of the proposed self-supervised training strategy over existing weakly supervised or consistency-based approaches is not sufficiently demonstrated.

  2. Insufficient evidence of disentanglement. The experiments mainly demonstrate lighting controllability rather than genuine appearance-lighting disentanglement. Evaluating different source images of the same person under varying illumination, while fixing the target lighting, would provide stronger evidence.

  3. Incomplete ablation studies. The individual contributions of the proposed consistency losses remain unclear. It is difficult to determine whether the improvements primarily arise from illumination augmentation or the proposed optimization objectives.

  4. Dependence on external estimators. The method relies on DECA and GeoWizard, potentially inheriting their estimation errors. The use of coarse facial geometry and low-order spherical harmonics may also limit performance under complex illumination. Additionally, using DECA for both lighting conditioning and evaluation raises concerns about estimator-dependent bias.

  5. Insufficient evaluation details. The protocols for Depth Error, identity evaluation, and FID computation require further clarification. A user study would also strengthen the perceptual evaluation.

  6. Limited failure-case analysis. Although limitations are briefly discussed in the appendix, more systematic analysis of challenging lighting conditions and geometry estimation failures would strengthen the paper.

Questions for authors

The paper demonstrates promising empirical results and addresses a practically relevant problem. However, the limited methodological novelty, insufficient evidence of genuine appearance-lighting disentanglement, and incomplete experimental validation prevent me from recommending acceptance at this stage.

I am open to increasing my rating if the authors can convincingly address these concerns during the rebuttal, particularly regarding disentanglement, the contributions of individual consistency objectives, and evaluation reliability.

  1. Can the authors quantitatively demonstrate that the proposed refinement reduces the influence of source illumination on the final relighting results, despite keeping the Appearance Adapter frozen?

  2. Is it computed between the predicted depth maps of the input and relit images (using Depth Anything V2), or against a separate reference depth map?

  3. What are the individual contributions of reconstruction consistency, light-response consistency, and detail-feature consistency? How does the method compare against simple illumination augmentation with an otherwise identical training pipeline?

  4. Could the authors evaluate whether different source images of the same person under varying lighting conditions produce consistent relighting results under a fixed target illumination?

  5. Does the reported inference time include all preprocessing, geometry estimation, feature extraction, inversion, and denoising operations? How sensitive is the method to imperfections introduced by Qwen-Image-Edit during illumination perturbation? In particular, what happens when the generated references contain subtle identity or geometry changes?

5 comments
AuthorsOpenReview ↗

Author Response — Part 1/3

We thank the reviewer for the thoughtful and constructive feedback. We address the concerns and questions below.

W1: Limited novelty as the framework integrates several established components.

Note, though our method utilizes several off-the-shelf models (eg., DECA, GeoWizard, and ArcFace) to work, we do not consider them as our contribution. To clarify, we contribution lies in that: (i) we present a self-supervised framework for photorealistic portrait relighting, eliminating the reliance on synthetic paired training data, (ii) we develop appearance adapter and geometry-aware light modulator to separately model, and (iii) We introduce a light-perturbation based refinement scheme to enhance the robustness of our method to complex illumination. Moreover, as demonstrated in our response to W4, our method is robust to estimation errors of these models, validating the merit of our self-supervised framework.

W2: Comparison with more consistency-based baselines.

In the table below, we compare our method with a consistency-based baseline on CelebAMask-HQ. Since most existing portrait relighting methods rely on paired supervision, a directly applicable baseline is lacking. We therefore construct a baseline inspired by [1]: we add a normal consistency loss to the diffusion model’s self-supervised training objective to enhance its geometric awareness, while removing the explicit geometric guidance from coarse facial geometry and normal maps. As shown, our method clearly outperforms the consistency-based alternative, convincingly demonstrating the advantage of our self-supervised framework.

MethodID↓Light↓DepthError↓
consistency-based baseline0.1361.9560.054
Ours0.0791.1240.037

[1] Liu et al., UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback. NeurIPS 2025.

W3: Further evidence of lighting-appearance disentanglement.

To address your concern, we in Fig. 16 show that our method is able to produce consistent relighting results from images of the same subject captured under different scenes and original lighting conditions, while preserving the subject’s appearance. Besides, we further quantitatively assess the effectiveness of appearance–lighting disentanglement on the synthetic benchmark introduced in our paper. Specifically, with a fixed identity and target illumination, we relight input images under different scenes and original lighting conditions, and then compute the Lighting Stability metric from NeRFFaceLighting [2] to assess the consistency of lighting. As can be seen, our method achieves the best result, showing that our method is effective in appearance–lighting disentanglement.

MetricsDiFareliIC-LightLBMGSHeadRelitDiffusionRendererOurs
Light Stability↓0.1080.1280.1710.1290.1350.105

[2] Jiang et al., NeRFFaceLighting: Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields. ACM TOG 2023.

W4: Dependence on DECA and GeoWizard, and potential bias from using DECA for lighting evaluation.

We show in Figs. 17 and 18 that, thanks to our self-supervised framework and the proposed refinement scheme, our method can robustly produce satisfactory results even though the lighting and facial geometry estimated by DECA and GeoWizard are severely unreliable. On the other hand, we would like to point out that, since our method does not directly use DECA-estimated lighting parameters as conditioning input, using DECA for lighting evaluation does not introduce bias. To validate this, we conduct an additional experiment using an alternative lighting estimator [3]. As shown below, our method still maintains very promising performance when using this estimator, showing that there is no bias for using DECA for lighting evaluation.

MetricsDiFareliIC-LightLBMGSHeadRelitDiffusionRendererOurs
Light_new↓0.1210.1550.2430.1430.2550.128

[3] Zhou et al., Deep Single-Image Portrait Relighting. ICCV 2019.

AuthorsOpenReview ↗

Author Response — Part 2/3

W5: Ablation of consistency losses in Stage2.

The table below presents the ablation studies for the consistency losses on CelebAMask-HQ. We have also provided a qualitative ablation analysis in Fig. 10(b). Compared with the data augmentation baseline (without all the consistency losses), incorporating the three consistency losses yields clear performance improvement. Specifically, the reconstruction consistency loss LRCL_{RC} benefits preserving the overall image details, the lighting-response consistency loss LLRCL_{LRC} enhances the effectiveness of relighting, while the detail feature consistency loss LDFCL_{DFC} contributes to improve the robustness to complex shadows and highlights.

LRCL_{RC}LLRCL_{LRC}LDFCL_{DFC}ID↓Light↓DepthError↓
×××0.1191.2230.040
√××0.0831.2680.038
√√×0.0871.1290.037
√×√0.0901.1790.038
√√√0.0791.1240.037

W6: Performance under complex illumination with coarse facial geometry and low-order spherical harmonics.

In fact, the coarse facial geometry and low-order spherical harmonics estimated from external estimators are sufficient for our method to achieve good performance, namely our method is relatively robust to estimation errors from DECA and GeoWizard. To validate this, we in the table below compare with two baselines, where one replaces low-order spherical harmonics with HDR environment map for illumination representation and the other one with detailed facial geometry. Note, since estimating accurate HDR environment map from a single image remains challenging, we instead construct a mixed training dataset with both ground-truth and pseudo HDR environment map annotations for the HDR-based baseline. As shown by the numeric results and the visual results in Fig. 13, using HDR environment map does not lead to performance improvement, because it will increase the difficulty to learn the lighting features. The baseline with detailed facial geometry produces results comparable to ours with coarse facial geometry, but requires more rendering time. These results validate the robustness of our method to estimation errors from DECA and GeoWizard.

MethodID↓Light↓DepthError↓
HDR-based baseline0.0772.0170.040
Detailed-face-based baseline0.0831.1160.041
Ours0.0791.1240.037

W7: Details for Depth Error, identity evaluation, and FID computation.

Depth Error is calculated by mean absolute error (MAE) between the depth maps of the input and relit output, both estimated by Depth Anything V2. Identity Distance (ID) measures the distance between the ArcFace identity features of the input and relit output. FID is computed by measuring the difference between the distributions of feature vectors for relit images generated by each method and a real reference set of CelebA images to evaluate overall image quality. We will clarify these details in paper.

W8: Strengthen the perceptual evaluation with a user study

In the table below, we present a user study to assess the effectiveness of our method. To do so, we randomly select 100 portraits images from CelebAMask-HQ. For each portrait image, we obtain a randomly sampled target lighting condition and then use different methods to produce the relighting outputs. Next, we we recruit participants via Amazon Mechanical Turk to rate the best out of each group of results from the aspects of lighting (LC) and appearance consistency (AC). Note, to avoid subjective bias, the results are presented in a random order without any method identity information. As shown below, our results are more favored by the participants, manifesting its advantage over others.

MethodUserStudy
LC↑AC↑
DiFareli0.0290.055
IC-Light0.0670.042
LBM0.0440.053
GSHeadRelit0.0080.017
DiffusionRenderer0.0040.002
Ours0.8480.831
AuthorsOpenReview ↗

Author Response — Part 3/3

W9: More failure-case analysis.

We have added more failure cases in Fig. 19.

Q1: Quantitatively demonstrate that the proposed refinement reduces the influence of source illumination.

To quantify the effectiveness of our proposed refinement, we in the table below evaluate lighting stability under two settings: (i) relighting source images of the same identity across different scenes under fixed target illumination on the synthetic benchmark, and (ii) relighting randomly sampled images under fixed target illumination on CelebAMask-HQ. As shown, our proposed refinement consistently improves the light stability over the two settings while keeping the Appearance Adapter frozen, convincingly validating its effectiveness in reducing the influence of source illumination.

metricsLight Stability (CelebA-HQ)↓Light Stability (Synth.)↓
ours0.1350.105
w/o refinement0.1510.123

Q2: Does the reported inference time include all preprocessing?

Actually not, our reported inference time in Table 1 does not include normal estimation and shading rendering. In the table below, we report the inference time including all our preprocessing. Note, to achieve fair comparison, we also include the time cost for IC-Light and LBM to perform foreground segmentation, and the time for DiFareli and GSHeadRelit to perform facial parameter estimation. As shown, our method still achieves a runtime comparable to IC-Light and LBM.

MetricsDiFareliIC-LightLBMGSHeadRelitDiffusionRendererOurs
Time (s)115.253.361.4263.6318.644.55

Q3: Sensitivity to imperfections introduced by Qwen.

As shown by Fig. 15, our method is robust to minor imperfections led by Qwen. Moreover, we further quantitatively demonstrate in the table below that Qwen-perturbed references and the original references produce highly similar results, manifesting our robustness to imperfections introduced by Qwen.

MethodID↓Light↓DepthError↓
Ours (Qwen-perturbed references)0.0811.1910.037
Ours (original references)0.0791.1240.037
Reviewer_VRqEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised framework for portrait relighting that aims to disentangle appearance and illumination without requiring paired relighting data. The method combines pretrained geometry estimation, identity and appearance encoders, a diffusion backbone, and a geometry-aware lighting module. Training is performed in two stages: the first reconstructs input portraits using separate appearance and lighting conditions, while the second introduces illumination-perturbed references and consistency losses to reduce the entanglement between appearance and lighting. The authors report improvements over several existing portrait relighting methods in visual quality and content preservation.

Critical strengths
  • Meaningful motivation for self-supervised portrait relighting.

The paper addresses a practical challenge, as collecting large-scale OLAT data requires specialized equipment and is expensive and difficult to scale. Developing a self-supervised alternative is therefore a worthwhile direction. However, the proposed method is still trained on relatively limited existing portrait datasets rather than leveraging large-scale, diverse in-the-wild data, which somewhat weakens the motivation and potential benefits of self-supervised learning.

  • Considerable system engineering effort.

Integrating numerous pretrained components into a unified self-supervised relighting framework is a nontrivial engineering effort. Despite the complexity of the pipeline, the authors successfully make these components work together and demonstrate reasonable relighting results without relying on paired training data. This practical achievement is commendable, even though the individual components and their integration offer limited methodological novelty.

Critical weaknesses
  • Limited lighting representation and controllability.

The method relies on DECA to estimate coarse facial geometry and low-order spherical harmonics for illumination. Specifically, the lighting representation consists of only nine spherical-harmonics coefficients per color channel, which are further averaged across RGB channels when generating shading hints. Such a low-frequency representation fundamentally limits the ability to represent spatially complex or high-frequency illumination, including localized light sources and sharp shadows. Unlike methods that directly support high-resolution environment maps, the proposed framework cannot provide similarly fine-grained lighting control. Moreover, because the lighting guidance is rendered only from coarse facial geometry, its effectiveness for hair, clothing, accessories, and background regions is questionable. These limitations are not sufficiently analyzed, despite being central to the claimed controllability of the method.

  • The complicated pipeline inherits rather than resolves the limitations of its constituent components.

A fundamental concern is that combining multiple off-the-shelf models does not necessarily overcome their individual limitations. The proposed framework inherits the inaccurate geometry and illumination estimation of DECA and GeoWizard, the limited spatial and physical consistency of diffusion-based generation, and the potential illumination leakage in pretrained appearance representations. Additional adapters and consistency losses are introduced to compensate for these issues, making the framework increasingly complicated without fundamentally addressing their causes. For example, the failure cases in Fig. 11 demonstrate that source shadows can remain outside the facial region, highlighting the limitations of geometry guidance derived from a coarse face model. While the method reports favorable benchmark results, it remains unclear whether the overall system meaningfully advances the underlying relighting capability or simply trades the limitations of individual methods for those of a more complicated combined pipeline.

  • The motivation of self-supervised learning.

The main appeal of self-supervised learning is its potential to leverage large-scale, diverse, and readily available unlabeled data without requiring expensive paired supervision. However, the proposed method does not convincingly exploit this advantage. Instead, it relies on the existing FFHQ dataset and introduces an increasingly complicated pipeline involving numerous off-the-shelf models, multiple adapters, and carefully designed consistency losses to compensate for imperfect geometry, lighting, and appearance estimates. Rather than addressing these challenges through scalable data-driven learning, the method largely attempts to overcome them through additional model engineering. This raises the question of whether the proposed framework follows the right direction for self-supervised relighting, and whether a simpler architecture trained on substantially larger and more diverse in-the-wild portrait datasets could achieve better results with fewer assumptions and dependencies.

Questions for authors
  1. Scalability: Why is the self-supervised framework trained only on FFHQ? Could larger and more diverse datasets reduce the need for such a complicated architecture?

  2. Pipeline complexity: Can the authors compare against a simpler self-supervised diffusion baseline to justify the necessity of the numerous pretrained components and adapters?

  3. Lighting controllability: How does the low-order SH representation handle high-frequency lighting, localized light sources, and sharp shadows?

  4. Estimation robustness: How sensitive is the method to inaccurate geometry and lighting estimates from DECA and GeoWizard?

7 comments
AuthorsOpenReview ↗

Author Response — Part 1/3

We thank the reviewer for the thoughtful and constructive feedback. We address the concerns and questions below.

Q1: Lighting representation and controllability. Why use low-order spherical harmonics rather than high-resolution environment maps?

We thank the reviewer for raising this insightful question. We adopt low-order spherical harmonics (SH) rather than high-resolution environment maps primarily because our goal is to provide an efficient and stable lighting representation for geometry-aware portrait relighting, rather than to reconstruct the full complexity of an arbitrary illumination environment. Low-order SH compactly captures the dominant low-frequency components of illumination, such as overall intensity and broad directional effects, while avoiding the additional complexity and estimation uncertainty associated with high-resolution environment maps, particularly in the single-image setting. By evaluating the SH illumination at each point on the estimated facial geometry using its surface normal, we are able to obtain spatially varying shading cues that can effectively guide geometry-aware portrait relighting.

We agree that low-order SH has inherent limitations in representing high-frequency illumination variations, small localized light sources, and sharp shadow boundaries. However, as demonstrated in Fig. 6, our method can also produce satisfactory lighting effects with high-frequency illumination variations, showing that low-order SH is sufficient for our goal to achieve portrait relighting.

Q2: Lighting guidance is rendered only from coarse facial geometry, its effectiveness for hair, clothing, accessories, and background regions is questionable

Although our lighting guidance is derived solely from coarse facial geometry, our method can also produce coherent lighting changes for hair, clothing, and accessories, such as changes in sunglasses transparency and watch shading, as shown in Fig. 6. This benefits from the diffusion model’s generative priors, which allow our method to infer lighting changes in non-facial regions from facial shading hints.

Q3: Estimation robustness: How sensitive is the method to inaccurate geometry and lighting estimates from DECA and GeoWizard?

We have provided robustness analysis in Figs. 17 and 18, where we compare results produced with inaccurate geometry and lighting estimates and the corresponding GTs. As can be seen, results with inaccurate estimates from DECA and GeoWizard are highly similar to those from GT lighting and geometry, demonstrating that our method is not so sensitive to estimation errors from DECA and GeoWizard.

AuthorsOpenReview ↗

Author Response — Part 2/3

Q4: Potential of the proposed self-supervised framework to leverage large-scale unlabeled data.

We thank the reviewer for this insightful comment. We agree that leveraging large-scale unlabeled data is a major advantage of self-supervised learning. We initially adopted FFHQ because it provides high-quality and relatively consistent portrait images, allowing us to establish a controlled benchmark for investigating self-supervised appearance–lighting disentanglement. However, we agree that evaluating scalability beyond this dataset is important. To address this concern, we expanded the training data by incorporating approximately 30,000 3D virtual portraits and 110,000 randomly sampled frames from the VFHQ portrait video dataset. We then conducted additional experiments and component ablations on this expanded dataset. All models are evaluated on CelebAMask-HQ using the same metrics to ensure consistent comparisons.

The results in the table below suggest that increasing data diversity and scale can improve certain aspects of relighting performance. In particular, the lighting error of our full model decreases from 1.124 on FFHQ to 0.984 on the expanded dataset. However, the identity and depth errors change from 0.079 to 0.095 and from 0.037 to 0.040, respectively. Thus, while the expanded data improves lighting estimation, it does not uniformly improve all evaluation metrics. More importantly, our component ablations show that the key components remain important when training on the expanded dataset. Removing appearance modeling components (AA and DA) leads to difficulties in preserving lighting-independent portrait content, while removing geometry-aware illumination modeling (GALM) and shading hint (SA) substantially degrades lighting control.

NMAAGALMSALPDAFFHQFFHQ+3D+VFHQ
ID↓Light↓DepthError↓ID↓Light↓DepthError↓
××√√××0.4160.9850.1040.4441.1090.098
×√√√××0.1381.3880.0570.1480.9890.068
√×√√××0.2121.2780.0610.1681.1960.055
√√××××0.0365.6500.0300.0524.7630.030
√√√×××0.0542.3030.0360.0751.8770.041
√√√√××0.0911.5690.0380.1091.1960.048
√√√√√×0.1181.1200.0440.1320.9750.049
√√√√√√0.0791.1240.0370.0950.9840.040

Q5: Compare against a simpler self-supervised diffusion baseline to justify the necessity of the numerous pretrained components and adapters?

To investigate whether a simpler self-supervised framework without these pretrained models can achieve comparable relighting performance, we construct a consistency-based baseline and evaluate it on both FFHQ and the expanded dataset. Inspired by [1], we simplify the original framework by removing the explicit geometric guidance from DECA and GeoWizard, retaining only DECA-estimated lighting as the conditioning input, and introducing a normal consistency loss to provide physical feedback during training. As shown, the quantitative results in the table below imply that this simplified baseline achieves substantially lower performance than our full framework, even when trained on the expanded dataset, manifesting the necessity of our employed pretrained components and adapters.

MethodFFHQFFHQ+3D+VFHQ
ID↓Light↓DepthError↓ID↓Light↓DepthError↓
consistency-based baseline0.1361.9560.0540.1801.3890.067
Ours0.0791.1240.0370.0950.9840.040

[1] Liu et al., UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback. NeurIPS 2025.

AuthorsOpenReview ↗

Author Response — Part 3/3

Q6: The underlying relighting capability of our model.

We conduct additional experiment demonstrates that our self-supervised framework and refinement strategy enable the model to learn transferable relighting capabilities. Specifically, we use our model as the base model and fine-tune it with supervision on 10,000 pairs of 3D data. After 80,000 training steps, the model achieves inversion-free relighting. Combined with the existing LCM-LoRA acceleration, the fine-tuned model produces high-quality relighting results for both in-the-wild and out-of-domain portraits in only four sampling steps, as shown in Figs. 8 and 14.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: New Experiments, Clarifications, and Summary of Revisions

We sincerely thank Reviewers VRqE (R1), aNFT (R2), and nADV (R3) for their thoughtful and constructive feedback. We appreciate their recognition of the practical significance of self-supervised portrait relighting (R1, R2), our staged design and the complementary roles of its components (R2), appearance preservation and identity consistency (R2, R3), and lighting control and out-of-domain generalization (R3).

We have carefully addressed each comment and believe we have resolved most of the reviewers’ concerns. In our revised manuscript, we have incorporated the suggested experiments, additional discussions, and relevant updates to further strengthen our work. Below, we summarize our core contributions and the main updates.

Core Contributions of Our Work

  1. We present a self-supervised framework for photorealistic portrait relighting, eliminating reliance on synthetic paired training data.
  2. We develop an Appearance Adapter and a Geometry-Aware Light Modulator to separately model appearance and lighting.
  3. We introduce a light-perturbation-based refinement scheme to improve robustness to complex illumination.

Experimental Updates During Rebuttal

  1. Tables 2 and 9 (R1, R2): Expanded the training data, conducted further component ablations, and compared against a simpler consistency-based self-supervised baseline to validate scalability and the importance of our architecture.

  2. Tables 3–4 and Figure 10 (R2): Added lighting-stability evaluations and individual loss ablations to validate the contributions of refinement and the consistency constraints.

  3. Table 7 and Figure 16 (R2): Evaluated relighting consistency for the same subject across different source lighting conditions and scenes under fixed target illumination, providing further evidence of appearance–lighting separation.

  4. Table 5 and Figures 6 and 13 (R1, R2): Added HDR-lighting and detailed-geometry baseline comparisons, together with lighting-editing results, to examine the effectiveness of our lighting representation and geometric guidance.

  5. Table 8 and Figures 17–18 (R1, R2): Analyzed robustness to estimation errors from DECA and GeoWizard and conducted quantitative evaluation using an alternative lighting estimator.

  6. Table 6 and Figure 15 (R2): Evaluated robustness to minor identity and geometry changes in Qwen-generated references.

  7. Table 10 and Figure 22 (R3): Added quantitative and qualitative comparisons with additional 2D portrait relighting methods.

  8. Figure 25 (R3): Compared DDIM with the reversible BELM sampler and adopted BELM to help address visual over-smoothing.

  9. Figures 8 and 14 (R1): Demonstrated four-step, inversion-free relighting through fine-tuning on a limited amount of paired data and combining the model with LCM-LoRA, showing the transferability of the learned relighting capability.

Updates of in-depth and additional discussions during Rebuttal

  • Introduction and Section 4.3 (R1, R2): Explained lighting controllability and its scope.
  • Section 4.1 (R2): Added details on evaluation metric computation.
  • Appendix D and Figure 19 (R1, R2): Added failure cases and expanded the analysis of limitations.
  • Author responses (R2): Provided a user study and runtime analysis including preprocessing.

We believe these additions and clarifications comprehensively address the reviewers’ concerns and enhance the overall quality of our manuscript. All revisions are highlighted in blue for ease of reference. The manuscript has been updated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.