- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents LumiForge, a framework for portrait video relighting under explicit environment illumination. The method introduces Inverse Lighting Curation (ILC) to recover illumination and canonical appearance from in-the-wild videos, constructing the LumiWild dataset with approximately one million frames. It further proposes ORBIT, which represents illumination using directional light tokens, and a temporal residual architecture that extends image relighting to video while preserving the image relighting pathway. The experiments report substantial improvements over existing image and video relighting baselines in reconstruction fidelity and temporal metrics.
Critical strengths
The paper addresses an important problem in portrait video relighting, particularly the challenge of combining accurate lighting control with temporal consistency. The proposed Inverse Lighting Curation provides a potentially useful direction for leveraging in-the-wild video data without requiring extensive controlled dynamic capture. The ORBIT representation also offers an interesting approach to preserving directional illumination information during generation. The authors conduct extensive quantitative evaluations and report substantial improvements over existing baselines. The resulting single-frame visual quality is relatively sharp and demonstrates promising image-level relighting capabilities.
Critical weaknesses
-
Poor specular consistency in the provided video results. My primary concern is the noticeable inconsistency of specular reflections in the supplementary video samples. Several examples exhibit seemingly static specular highlights that do not respond correctly to facial motion, creating obvious relighting artifacts. Given the rich OLAT supervision available, I would expect more physically consistent lighting behavior. These artifacts directly undermine the paper's central claim of precise and temporally consistent portrait video relighting.
-
Insufficient video-level evidence to support the claimed improvements. Although the paper reports significant quantitative improvements over existing methods, the qualitative comparisons primarily emphasize individual frames rather than continuous video performance. The supplementary videos demonstrate the proposed method but lack sufficient synchronized comparisons against competing approaches. The image-based results are relatively sharp, but strong single-frame quality alone does not establish superior video relighting, especially when temporal lighting consistency is the central challenge.
-
Discrepancy between quantitative metrics and observed visual quality. The reported PSNR, LPIPS, and temporal metrics suggest substantial improvements, yet the visible specular artifacts raise concerns about whether these measurements adequately reflect perceptual and physical consistency. Modern image-generation and editing systems can already produce convincing individual relit frames, making temporal coherence and consistent light transport the more meaningful challenges. I would expect more targeted video-level evaluations, particularly of dynamic specular reflections, to substantiate the claimed advantages.
Questions for authors
-
How do the authors explain the seemingly static specular highlights observed across multiple supplementary video examples? Are these artifacts related to the temporal architecture or the learned lighting representation?
-
Could the authors provide synchronized video comparisons against the strongest video relighting baselines, particularly under significant facial motion and challenging specular illumination?
-
Given the substantial quantitative improvements reported, how do the authors reconcile these results with the visible temporal artifacts? Could additional evaluations specifically targeting highlight motion and lighting consistency better support the claimed video-relighting performance?
4 comments
Thanks for the author's detailed explanation and the newly reported results. I checked the specular reflection closely and found consistent results as presented in the paper. However, there are shadowing artifacts in some cases in the synchronized video comparisons, e.g., P25_all_25fps. Can the authors elaborate further on this? Such shadowing effects are not visible in P23_all_25fps. Also, how well will the proposed method work on in-the-wild data with a domain gap, e.g., video conference where the cam quality is low and noisy?
We thank the reviewer for re-examining the highlights and for the two additional questions.
Shadowing in P25. The artifact is likely caused by fast head rotation. When a small region changes shape quickly between frames, as the mouth does in P25, features from adjacent frames can leak into the current frame and leave a trailing contour. Motion-aware temporal aggregation would address this directly. We also re-ran inference with a different random seed, and the result no longer shows this artifact; it is provided for the reviewer’s reference at https://anonymous.4open.science/r/lumiforge-material-FE60/comparison/P25_all_25fps_new.mp4.
In-the-wild video conferencing. Following the reviewer’s suggestion, we ran LumiForge, without any fine-tuning, on real webcam recordings from the Microsoft Video Conferencing Dataset (VCD, https://github.com/microsoft/VCD). We selected the 15 talking-head clips with the lowest perceived quality. These clips exhibit sensor noise, low light, blur and low resolution, and include headsets, glasses and hand gestures. Each clip is relit under a new HDR environment (indoor, studio, sun, overcast or sunset/night). The 65-frame results, shown alongside the source video and the target environment, are available at https://anonymous.4open.science/r/lumiforge-material-FE60/vcd/. Despite the domain gap to our training data, LumiForge preserves identity and head motion and follows the target environment.
Thanks for the author's response. I don't have further questions. I'll reconsider my evaluation. Thanks
Response to Reviewer Te4D
We thank the reviewer for the close look at highlight behaviour, which is indeed the most demanding aspect of video relighting. We answer each point in order and add new video evidence.
W1 — Specular consistency in the supplementary videos.
W2 — Synchronized video comparisons.
Within the rebuttal period we prepared additional videos at https://anonymous.4open.science/r/lumiforge-material-FE60/, all 65 frames and frame-synchronized; in the comparisons every method receives the same input and the same target environment through its own interface:
becominglit/): held-out references with the ground-truth video as the first panel, so that highlight motion can be checked against reality: GT, input, LumiForge, IC-Light, Light-X, DiffusionRenderer, Neural Gaffer, RelightVid, and the target HDR map.comparison/), against 11 baselines: IC-Light, Light-X, DiffusionRenderer, RelightVid, Neural Gaffer, UniLumos, TC-Light, Light-A-Video, IllumiCraft, LCVD, ID-V2V+IC-Light.videos/) on wild videos of held-out subjects under new target HDR environments (indoor, sun, sunset/night), each shown with its source video and target environment.W3 — Do the metrics reflect perceptual and physical consistency?
Image-level quality. LumiForge is not only sharp but follows the requested light. In Table 1 it ranks first on both FaceOLAT and BecomingLit in PSNR and LPIPS; on FaceOLAT it reaches 27.28 dB / 0.0262, vs. 18.10 dB for the strongest baseline. Image editors can produce a convincing frame, but not the frame that a specified HDR environment would produce.
Video-level consistency. Table 2 shows the lowest temporal errors against reference videos. We now add highlight-specific metrics on the 20 BecomingLit environment references (65 frames each):
Light-switch is the correlation between highlight maps of the same frame under two different target lights (lower = highlights follow the light). LumiForge is clearly ahead of the HDR-conditioned baselines (Light-X, DiffusionRenderer) on both metrics. For physical correctness of the shading itself, a direction metric (shading gradient after dividing by the flat-lit input) gives 25° error for LumiForge vs. 71°–100° for all baselines on the FaceOLAT pairs of Table 1 (chance: 90°). We will add these metrics to the paper.
Questions.