- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper is on the topic of controllable image relighting. The authors first introduce a new lighting representation for scenes, Lumi Map, that is material-decoupled and more physics-oriented. This allows for more accurate lighting modeling from synthetic data. Then, a two-stage training strategy is designed to mitigate the domain gap between synthetic data and real data training. Together, the approach, called Dior-Light, allows users to control relighting results on real images using hand-drawn strokes. The method is evaluated on synthetic and real datasets with other relighting methods and alternative lighting representations with reasonable performance.
Critical strengths
- The proposed Lumi Map representation for lighting has a good theoretical motivation and practical implementation. It models hard-to-model (inter)reflection and shadow effects in complex scenes, better than irradiance and shading that are commonly used in the inverse graphics work (Figure 2). This representation could be useful not only for relighting but also for other relevant research works that require realistic modeling of lighting.
- I found the two-stage training strategy to bridge the domain gap between synthetic and real data to be useful for inverse problems. This allows researchers to develop advanced models in the synthetic environment and then transfer them to real data. Although the current setting requires real training pairs (source and target), it will be interesting to explore a single real-image setting with a bootstrapped estimated lighting map and some iterative refinement process.
Critical weaknesses
- The stroke is a key control element for the relighting application. Currently, the way of obtaining the strokes in synthetic data is to use free-form stroke masks to sample from the lighting map (Figure 12 in the supplemental material). This is quite different from how an actual user would use strokes to express their relighting need. Also, there will be no actual (target) lighting map to sample from, so the inference-time stroke control will always be suboptimal for the trained model. This is a significant gap. Btw, I had trouble finding the details in the main text about how the strokes are obtained from the Lumi Map and eventually found it in the supplemental material. Such important information should definitely be in the main text.
- There are some weaknesses in the quantitative evaluations. 1) While the full relighting results of the proposed Dior Light outperform other methods, Dior w/o stage 2 has a very similar performance to other lighting representations (esp. Irradiance) in Tables 2 and 3. This undermines the significance of the new representation in practice: maybe modeling the interreflections doesn't matter that much? 2) The quantitative or qualitative comparison can be done with more relighting methods (already mentioned in the related work), even if they are not designed for scribbles. 3) With the two-stage training strategy, it is unclear how much real data is needed to properly tune the model to be more generalizable.
Questions for authors
- For the stroke mechanism, how can one design the strokes in the synthetic environment to fit more closely to a real user's use case?
- Can the authors clarify the performance of Dior w/o stage 2 vs other illumination representations?
- How much real data is needed for the two-stage training strategy to be successful?
4 comments
Response to Reviewer EExJ
W2(3) / Q3. Real-data requirements of Stage 2
We thank the reviewer for this question. Our method does not aim to minimize the amount of real data. To ensure sufficient training, the main model uses all available real data (36K images from MultiIllum and ReNe) and trains Stage 2 for 5,000 steps with a batch size of 16.
To investigate the generalization potential of Stage 2 raised by the reviewer, we conducted a small-scale study of how the amount and source of real data affect generalization under a fixed training budget. Starting from the same Stage-1 model, we trained Stage 2 for 2,000 steps under four settings: 100, 500, and 1,000 MultiIllum (MI) samples, and 1,000 ReNe samples. We evaluated each setting with paired metrics on 300 held-out pairs (150 from ReNe and 150 from MultiIllum), with equal weight for the two benchmark means, and with Elo ratings on the in-the-wild (ITW) test set.
Table R5. Analysis of real-data requirements
| Training setting | PSNR ↑ | SSIM ↑ | LPIPS ↓ | ITW Elo ↑ |
|---|---|---|---|---|
| Stage 1 (synthetic only) | 17.50 | 0.6993 | 0.2620 | 1057.30 |
| + MI 100 | 18.75 | 0.7488 | 0.2328 | 1264.61 |
| + MI 500 | 19.67 | 0.7565 | 0.2252 | 1279.59 |
| + MI 1,000 | 18.84 | 0.7466 | 0.2310 | 1149.50 |
| + ReNe 1,000 | 19.12 | 0.7303 | 0.2317 | 1238.83 |
All settings improve over the Stage-1 model on paired benchmark and in-the-wild benchmark, with ITW Elo gains ranging from 92 to 222 points. We further observe:
-
Under a fixed training budget, generalization does not improve monotonically with the amount of real data. Increasing the number of MI samples from 100 to 500 improves all four metrics, and the 500-sample setting achieves the best results on every metric. Increasing the number further to 1,000 degrades all four metrics relative to the 500-sample setting; in particular, the ITW Elo drops from 1279.59 to 1149.50.
-
The source of real data affects ITW generalization. With the same number of samples (1,000), ReNe achieves a higher ITW Elo than MultiIllum (1238.83 vs. 1149.50) and a higher PSNR (19.12 vs. 18.84). MultiIllum achieves a higher SSIM (0.7466 vs. 0.7303), and the two settings have comparable LPIPS (0.2310 vs. 0.2317).
In summary, Stage 2 is data-efficient: with only 100 real samples, PSNR improves by 1.25 dB and the ITW Elo by more than 200 points over the Stage-1 model. Under a fixed training budget, a moderate amount of real data (500 samples in this study) performs best, and the source of the data further affects ITW generalization. A systematic study of data scaling is beyond the scope of this work. We also appreciate the reviewer's suggestion of bootstrapping lighting estimates from single real images, which could further reduce the need for paired real data. We will explore this direction in future work.
References
[1] Zheng Zeng et al. RGB↔X: Image Decomposition and Synthesis Using Material- and Lighting-aware Diffusion Models. ACM SIGGRAPH Conference Papers, 2024.
[2] Chenjian Gao, Linning Xu, and Tianfan Xue. Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relighting. arXiv:2607.08879, 2026.
[3] Ruofan Liang et al. Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR, 2025.
[4] Chong Zeng et al. DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation. ACM SIGGRAPH Conference Papers, 2024.
[5] Worameth Chinchuthakun et al. DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpainting. arXiv, 2025.
Thanks for providing the additional results and analysis. They look good to me. I don't have any further requests or questions for the authors. I recommend acceptance.
We sincerely thank the reviewer for the positive feedback and for recommending acceptance. Your constructive comments have greatly helped us improve the paper.
Response to Reviewer EExJ
Response to Reviewer EExJ
We thank the reviewer for the thoughtful review and the encouraging comments on the representation and the training strategy.
The primary purpose of our automatic stroke sampling is to train the model to infer full-image illumination from sparse lighting cues, not to replicate the shapes of human-drawn strokes. Nevertheless, we agree that the difference between sampled and hand-drawn strokes should be examined. We therefore manually annotated hand-drawn strokes for 150 samples, 50 from each of the three benchmarks, and compared the performance obtained with hand-drawn and automatically sampled strokes on the same samples. The results show that this difference does not lead to a noticeable performance drop. Despite covering fewer pixels, hand-drawn strokes achieve comparable performance on ReNe and better performance on MultiIllum and SynthBench. A likely reason is that human annotators draw on prior knowledge of lighting structure and place strokes in more informative regions. This demonstrates the robustness of our method to the distribution of user strokes. Qualitative results are shown in Fig. 9 (right) of the revised main text, and quantitative results are reported below.
We select 150 samples from three benchmarks for manual stroke annotation (50 per benchmark). Cover. is the mean fraction of pixels occupied by strokes.
Table R3. Quantitative analysis of auto-sampled and hand-drawn strokes
Although sparse sampling enables the model to infer global illumination from local cues, scribble-driven relighting would further benefit from a human–computer interaction perspective. In particular, studying how users actually draw lighting strokes, including their shapes and colors, could inform the design of training-time stroke sampling. We leave this to future work.
To improve readability, we have moved the key details of the sparse lumi map sampling from Supplementary C.2 to Sec. 4.1 of the main text, while the full implementation details remain in the supplementary material.
All representation variants in Table 2 are trained with Stage 2. This additional training on real data explains why some of these variants, particularly the irradiance variant, outperform Dior w/o Stage 2 on ReNe and MultiIllum. The representation ablations should therefore be compared with the full model. Under the same training setting, Dior with the lumi map achieves stronger performance, as shown in Tables 2 and 3.
In addition to the comparisons with RGB↔X [1] and Lume-Palette [2] already included in Supplementary D.2, we have added comparisons with DiffusionRenderer [3] and DiLightNet [4]. All methods are evaluated using their official implementations. For DiffusionRenderer and DiLightNet, the target lighting is estimated directly from the target image using the third-party DiffusionLight-Turbo estimator [5].
Table R4. Extended results inventory