- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
A video relighting method which uses DiffusionRenderer to extract per-frame PBR gbuffers, DiffusionLight to estimate the lighting (which is refined via inverse rendering), then renders a PBR approximation under novel lighting and refines that estimate via a video2video "fixer" network (a finetuned CogVideoX model).
The PBR approximation is a strong guidance signal and the approach produces convicing results. I also appreciate the training approach of the fixer network, which is trained on a mix of synthetic pairs, real-world unpaired videos and multi-illumination captures.
It is a neat but straightforward approach which combines a set of recent techniques, and inherits their limitations (e.g., roughness and metallic predictions of DiffusionRenderer are not always reliable), but the results are convincing.
I have a few concerns regarding the evaluation protocol, but they can likely easily be addressed in a minor revision, so I am leaning positive.
Critical strengths
Strong results on the video relighting task. Combining a proxy PBR rendering and a video2video uplifting task, makes the uplifting task easier than in e.g., albedo-RGB or X-RGB.
Nice training setup which leverages both paired synthetic data and real-world unpaired videos.
Critical weaknesses
The approach relies on DiffusionRenderer (DR) to extract video G-buffers in order to create the PBR proxy under novel lighting, and is limited by the quality of DRs predictions, which are not always accurate. Furthermore, this intoduces a gbuffer bottleneck in the pipeline, which makes it harder to handle transparent/translucent objects, and shading effects not handles by the PBR shading model. That said, the fixer model can compensate for this.
My other concern is that it looks like UniRelight is not configured correctly. In Fig 10-12 in the supplemental, UniRelight has almost no shadows, or shadows in the wrong direction, so I wonder if this baseline has been badly configured somehow Looking at the project page, synthetic benchmark, third example (rotating probe), the shadows seems to be in the wrong direction, so I supect a probe orientiation mismatch. There is also a significant intensity difference.
Comparing metrics scores on MIT multi-illumination between the UniRelight paper and LightCrafter, there are large differences (see below). This may be due to difference in evaluation settings, but could be clarified (and I would recommend to check for probe rotation and intensity mismatches also here).
PSNR stats in the UniRelight paper (MIT multi-illumination dataset):
UniRelight: 20.76dB DiffusionRenderer: 16.99dB
PSNR stats in this submission (MIT multi-illumination dataset):
UniRelight: 13.64dB DiffusionRenderer: 12.83dB LightCrafter: 20.27dB
Questions for authors
In Table 1, on MIT, PBR (Mitsuba renderings) have PSNR scores of only 13.59dB, but are then uplifted to 20.27dB. Looking at Fig 6 and Fig 14, PBR on MIT looks ok. Are these just easy examples? Is it mainly the black pixels that are the root cause of the low PSNR scores for PBR?
Also in Table 1. UniRelight performs consistently worse than DiffusionRenderer, which is not the case in the UniRelight paper (Table 1), where UniRelight is consistently better. Which version of DiffusionRenderer is used here, the Cosmos or older Stable Video Diffusion version?
It is stated that PCRP (Careaga & Aksoy, 2025) trains a diffusion model on synthetic data which is not correct. That work shows you can train a neural rendering network that can handle any light source type, solely from self-supervision using real images.
As discussed above, I would highly recommend to check the UniRelight setup (probe orientation and intensity) and rerun metrics.
In Fig 3, the probe seems to be badly rotated for DiffusionRenderer (when looking at the left chromeball). Same on the diffuse example on top, the shadow is in another direction (hard to tell what is right, but could at least be worth checking if it is a probe rotation issue also here).
3 comments
Table 1 with the updated UniRelight and DiffusionRenderer row.
| Method | Synthetic PSNR↑ | Synthetic SSIM↑ | Synthetic LPIPS↓ | Synthetic T-CLIP↑ | Synthetic Warp-SSIM↑ | MIT PSNR↑ | MIT SSIM↑ | MIT LPIPS↓ | DL3DV PSNR↑ | DL3DV SSIM↑ | DL3DV LPIPS↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Light-X | 13.54 | 0.5157 | 0.3916 | 0.9853 | 0.9448 | 14.04 | 0.5933 | 0.4381 | 13.76 | 0.3813 | 0.4523 |
| UniRelight (submission) | 18.87 | 0.5475 | 0.3202 | 0.9832 | 0.9584 | 13.64 | 0.5198 | 0.2977 | 11.98 | 0.3354 | 0.5511 |
| UniRelight (updated) | 20.98 | 0.5849 | 0.2920 | 0.9871 | 0.9575 | 15.01 | 0.6015 | 0.2375 | 16.73 | 0.5247 | 0.3401 |
| DiffusionRenderer (submission) | 20.92 | 0.7130 | 0.2867 | 0.9839 | 0.9574 | 12.83 | 0.5581 | 0.3048 | 15.33 | 0.5551 | 0.4244 |
| DiffusionRenderer (updated) | 21.72 | 0.7245 | 0.2879 | 0.9836 | 0.9522 | 12.95 | 0.5590 | 0.2997 | 16.84 | 0.5930 | 0.4222 |
| PCRP-video | 20.78 | 0.5852 | 0.3099 | 0.9838 | 0.9507 | 13.57 | 0.5253 | 0.2890 | 18.57 | 0.5214 | 0.2257 |
| PBR | 19.59 | 0.7111 | 0.3106 | 0.9755 | 0.9223 | 13.06 | 0.3984 | 0.4867 | 17.75 | 0.5082 | 0.4768 |
| LightCrafter | 24.16 | 0.7366 | 0.2096 | 0.9892 | 0.9598 | 15.07 | 0.6159 | 0.2193 | 19.55 | 0.6695 | 0.2030 |
Thanks for the updated results. I am glad to see the corrected probe orientations on UniRelight and DiffusionRenderer in this revision.
Weaknesses
W1. It looks like UniRelight is not configured correctly. In Fig 10-12 in the supplemental, UniRelight has almost no shadows, or shadows in the wrong direction, so I wonder if this baseline has been badly configured somehow Looking at the project page, synthetic benchmark, third example (rotating probe), the shadows seems to be in the wrong direction, so I suspect a probe orientation mismatch. There is also a significant intensity difference.
A: We thank the reviewer for identifying this issue. We rechecked UniRelight’s light-probe configuration and indeed found an environment-map orientation and intensity mismatch. We corrected it using
env_flip=Trueandenv_rot=180and,env_strength=1, and for videos with camera motion, we rotated the target environment map according to each frame’s camera pose. These corrections improve the lighting directions and reduce the excessive darkening observed in the previous results. We also clarify an important evaluation difference: UniRelight applies per-channel least-squares scaling before computing metrics, whereas our evaluation does not apply this scaling to any method. The scores reported in the two papers therefore follow different protocols and are not directly comparable. Using the corrected configuration and our unchanged evaluation protocol, we reran UniRelight with its released code and checkpoint, recomputed all metrics (reported in Table 1 below), and updated the qualitative results in the revision.Questions
Q1. In Table 1, on MIT, PBR (Mitsuba renderings) have PSNR scores of only 13.59dB, but are then uplifted to 20.27dB. Looking at Fig 6 and Fig 14, PBR on MIT looks ok. Are these just easy examples? Is it mainly the black pixels that are the root cause of the low PSNR scores for PBR?
A: We agree that Figs. 6/14 are easy cases but the black border pixels are not the root cause. The low average PBR score is mainly caused by some challenging cases in the evaluation set. Fig. 15 provides challenging MIT examples under intense illumination. In this case, our PBR proxy fails to reproduce large saturated bright regions and strong highlights, because chrome balls only capture incoming light at a local region rather than a true environment map that captures light from infinitely far sources. For the two examples shown, PBR achieves only 11.71 and 10.45 dB.
Challenging MIT Multi-Illumination examples. Refer to revision figure 15 on page 33.
Q2. Also in Table 1. UniRelight performs consistently worse than DiffusionRenderer, which is not the case in the UniRelight paper (Table 1), where UniRelight is consistently better. Which version of DiffusionRenderer is used here, the Cosmos or older Stable Video Diffusion version?
A: Yes. As the appendix runtime table cites, we used the Cosmos (7B) release. However, the issue is found and described as in W1, and with the adjustment, the ranking flips, as shown in Table 1 below.
Q3. It is stated that PCRP (Careaga & Aksoy, 2025) trains a diffusion model on synthetic data which is not correct. That work shows you can train a neural rendering network that can handle any light source type, solely from self-supervision using real images.
A: Thanks for the correction; in the revision, the Sec. 2 sentence read as: “PCRP trains a neural rendering network self-supervised on real photographs to refine PBR renders from monocular inverse rendering, handling arbitrary light sources.”
Q4. As discussed above, I would highly recommend to check the UniRelight setup (probe orientation and intensity) and rerun metrics. A: Done; see W1 and Table 1 below.
Q5. In Fig 3, the probe seems to be badly rotated for DiffusionRenderer (when looking at the left chromeball). Same on the diffuse example on top, the shadow is in another direction (hard to tell what is right, but could at least be worth checking if it is a probe rotation issue also here).
A: Thank you for pointing this out. Our submission used the codebase’s default probe settings (env_flip=True, env_rot=180). After sweeping the probe orientations, we found that a 90° yaw rotation noticeably improves the alignment of shadows and highlights. Therefore, with the new offset, we reran DiffusionRenderer on all evaluation sets and updated all the figures and quantitative results in the revision accordingly (Table 1 below).