acceptodds
Under review as a conference paper at ICLR 2027

LumiForge: Inverse Lighting Curation for Precise Portrait Video Relighting

Abstract

Precise portrait video relighting requires a moving face to respond faithfully and consistently to a specified environment illumination, far beyond generic lighting stylization. The fundamental obstacle is supervision: OLAT provides accurate directional light responses, but extending such capture to diverse moving faces demands tightly synchronized high-speed cameras and extremely fast illumination control, making large-scale video supervision impractical. We present LumiForge with two simple principles: recover rather than capture, and resolve rather than compress. First, our Inverse Lighting Curation (ILC) recovers source illumination and canonical flat-lit appearance from in-the-wild videos, turning real performances into direct relighting supervision. Applying ILC to 15,191 VFHQ clips yields LumiWild, an explicitly environment-conditioned portrait video corpus with nearly one million frames. Second, our ORBIT conditioning preserves environment illumination as directional light tokens and jointly models subject identity, rather than collapsing lighting into a global appearance code. Together, scalable recovered supervision and resolved illumination enable high-fidelity, temporally consistent, and precisely controllable portrait video relighting, while our environment-map estimator achieves state-of-the-art performance on our held-out portrait-based illumination benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, video relight

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Te4DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents LumiForge, a framework for portrait video relighting under explicit environment illumination. The method introduces Inverse Lighting Curation (ILC) to recover illumination and canonical appearance from in-the-wild videos, constructing the LumiWild dataset with approximately one million frames. It further proposes ORBIT, which represents illumination using directional light tokens, and a temporal residual architecture that extends image relighting to video while preserving the image relighting pathway. The experiments report substantial improvements over existing image and video relighting baselines in reconstruction fidelity and temporal metrics.

Critical strengths

The paper addresses an important problem in portrait video relighting, particularly the challenge of combining accurate lighting control with temporal consistency. The proposed Inverse Lighting Curation provides a potentially useful direction for leveraging in-the-wild video data without requiring extensive controlled dynamic capture. The ORBIT representation also offers an interesting approach to preserving directional illumination information during generation. The authors conduct extensive quantitative evaluations and report substantial improvements over existing baselines. The resulting single-frame visual quality is relatively sharp and demonstrates promising image-level relighting capabilities.

Critical weaknesses
  1. Poor specular consistency in the provided video results. My primary concern is the noticeable inconsistency of specular reflections in the supplementary video samples. Several examples exhibit seemingly static specular highlights that do not respond correctly to facial motion, creating obvious relighting artifacts. Given the rich OLAT supervision available, I would expect more physically consistent lighting behavior. These artifacts directly undermine the paper's central claim of precise and temporally consistent portrait video relighting.

  2. Insufficient video-level evidence to support the claimed improvements. Although the paper reports significant quantitative improvements over existing methods, the qualitative comparisons primarily emphasize individual frames rather than continuous video performance. The supplementary videos demonstrate the proposed method but lack sufficient synchronized comparisons against competing approaches. The image-based results are relatively sharp, but strong single-frame quality alone does not establish superior video relighting, especially when temporal lighting consistency is the central challenge.

  3. Discrepancy between quantitative metrics and observed visual quality. The reported PSNR, LPIPS, and temporal metrics suggest substantial improvements, yet the visible specular artifacts raise concerns about whether these measurements adequately reflect perceptual and physical consistency. Modern image-generation and editing systems can already produce convincing individual relit frames, making temporal coherence and consistent light transport the more meaningful challenges. I would expect more targeted video-level evaluations, particularly of dynamic specular reflections, to substantiate the claimed advantages.

Questions for authors
  1. How do the authors explain the seemingly static specular highlights observed across multiple supplementary video examples? Are these artifacts related to the temporal architecture or the learned lighting representation?

  2. Could the authors provide synchronized video comparisons against the strongest video relighting baselines, particularly under significant facial motion and challenging specular illumination?

  3. Given the substantial quantitative improvements reported, how do the authors reconcile these results with the visible temporal artifacts? Could additional evaluations specifically targeting highlight motion and lighting consistency better support the claimed video-relighting performance?

4 comments

Response to Reviewer Te4D

We thank the reviewer for the close look at highlight behaviour, which is indeed the most demanding aspect of video relighting. We answer each point in order and add new video evidence.

W1 — Specular consistency in the supplementary videos.

  • Additional material. Within the limited rebuttal period we have prepared further videos (W2), including ground-truth comparisons in which highlight motion can be checked directly.
  • Where the static-looking highlights come from. We measure a soft highlight map (local luminance excess inside the face mask) and its placement relative to the real video (correlation ↑) on 9 VFHQ performances with large head motion: 0.723 for the per-frame image relighter, which has no temporal component, and 0.741 for LumiForge. The temporal modules therefore do not cause the effect (they slightly improve placement); it originates in the image path: FaceOLAT is captured with a static head, so view-dependent highlight motion is never observed in image training, and ORBIT's 331 regions smooth the narrowest specular lobes. Video-domain highlight supervision and finer angular resolution for compact sources are direct remedies, which we will discuss in the limitations.
  • Scope of the claim. Our temporal-consistency claim concerns shading and flicker: with LumiWild, temporal errors drop by 24% on held-out BecomingLit (1BvC-W1a), and LumiForge has the lowest errors in Table 2. Fine highlight dynamics remain open for all current methods, and we will state this as a limitation.

W2 — Synchronized video comparisons.

Within the rebuttal period we prepared additional videos at https://anonymous.4open.science/r/lumiforge-material-FE60/, all 65 frames and frame-synchronized; in the comparisons every method receives the same input and the same target environment through its own interface:

  1. BecomingLit with ground truth (becominglit/): held-out references with the ground-truth video as the first panel, so that highlight motion can be checked against reality: GT, input, LumiForge, IC-Light, Light-X, DiffusionRenderer, Neural Gaffer, RelightVid, and the target HDR map.
  2. Wild clips with large head rotation and strong directional light (comparison/), against 11 baselines: IC-Light, Light-X, DiffusionRenderer, RelightVid, Neural Gaffer, UniLumos, TC-Light, Light-A-Video, IllumiCraft, LCVD, ID-V2V+IC-Light.
  3. New relighting results (videos/) on wild videos of held-out subjects under new target HDR environments (indoor, sun, sunset/night), each shown with its source video and target environment.

W3 — Do the metrics reflect perceptual and physical consistency?

Image-level quality. LumiForge is not only sharp but follows the requested light. In Table 1 it ranks first on both FaceOLAT and BecomingLit in PSNR and LPIPS; on FaceOLAT it reaches 27.28 dB / 0.0262, vs. 18.10 dB for the strongest baseline. Image editors can produce a convincing frame, but not the frame that a specified HDR environment would produce.

Video-level consistency. Table 2 shows the lowest temporal errors against reference videos. We now add highlight-specific metrics on the 20 BecomingLit environment references (65 frames each):

Methodlight-switch ↓ (GT 0.757)highlight placement vs. GT ↑
LumiForge0.8370.510
Light-X0.9340.471
DiffusionRenderer0.9250.417

Light-switch is the correlation between highlight maps of the same frame under two different target lights (lower = highlights follow the light). LumiForge is clearly ahead of the HDR-conditioned baselines (Light-X, DiffusionRenderer) on both metrics. For physical correctness of the shading itself, a direction metric (shading gradient after dividing by the flat-lit input) gives 25° error for LumiForge vs. 71°–100° for all baselines on the FaceOLAT pairs of Table 1 (chance: 90°). We will add these metrics to the paper.

Questions.

  • Q1 (cause of static highlights). They come from the learned lighting path, i.e. static-head OLAT supervision and the angular resolution of the 331 regions, not from the temporal architecture: the per-frame image model shows the same highlight placement (W1).
  • Q2 (synchronized comparisons). Provided at https://anonymous.4open.science/r/lumiforge-material-FE60/: BecomingLit references with GT, wild clips with large head rotation against 11 baselines, and new results on held-out subjects (W2).
  • Q3 (metrics vs. artifacts). PSNR, LPIPS and temporal Δ are dominated by low-frequency shading, which LumiForge reproduces best. We therefore add highlight-targeted metrics (W3), on which LumiForge leads the HDR-conditioned baselines. Highlight dynamics remain an open problem for all current methods, and we will state this in the limitations.

Thanks for the author's detailed explanation and the newly reported results. I checked the specular reflection closely and found consistent results as presented in the paper. However, there are shadowing artifacts in some cases in the synchronized video comparisons, e.g., P25_all_25fps. Can the authors elaborate further on this? Such shadowing effects are not visible in P23_all_25fps. Also, how well will the proposed method work on in-the-wild data with a domain gap, e.g., video conference where the cam quality is low and noisy?

We thank the reviewer for re-examining the highlights and for the two additional questions.

Shadowing in P25. The artifact is likely caused by fast head rotation. When a small region changes shape quickly between frames, as the mouth does in P25, features from adjacent frames can leak into the current frame and leave a trailing contour. Motion-aware temporal aggregation would address this directly. We also re-ran inference with a different random seed, and the result no longer shows this artifact; it is provided for the reviewer’s reference at https://anonymous.4open.science/r/lumiforge-material-FE60/comparison/P25_all_25fps_new.mp4.

In-the-wild video conferencing. Following the reviewer’s suggestion, we ran LumiForge, without any fine-tuning, on real webcam recordings from the Microsoft Video Conferencing Dataset (VCD, https://github.com/microsoft/VCD). We selected the 15 talking-head clips with the lowest perceived quality. These clips exhibit sensor noise, low light, blur and low resolution, and include headsets, glasses and hand gestures. Each clip is relit under a new HDR environment (indoor, studio, sun, overcast or sunset/night). The 65-frame results, shown alongside the source video and the target environment, are available at https://anonymous.4open.science/r/lumiforge-material-FE60/vcd/. Despite the domain gap to our training data, LumiForge preserves identity and head motion and follows the target environment.

Thanks for the author's response. I don't have further questions. I'll reconsider my evaluation. Thanks

Reviewer_1BvCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LumiForge addresses HDR-environment-conditioned portrait video relighting. Inverse Lighting Curation (ILC) uses an environment estimator and a delighting model to turn 15,191 in-the-wild VFHQ clips into (flat-lit video, estimated environment, original video) training triplets (LumiWild). ORBIT encodes the target HDR map as 331 directional light tokens aligned with an OLAT basis, and temporal residual modules adapt a frozen Wan2.1-based image relighter to video. The method reports large gains over ten baselines on FaceOLAT and BecomingLit (8-9 dB PSNR), lower temporal errors, and state-of-the-art environment estimation on a new benchmark.

Critical strengths
  1. "Recover rather than capture" is an appealing and scalable way to obtain relighting supervision from ordinary portrait videos, and the overall system is coherent and candidly described.
  2. The environment estimator is evaluated on subject- and environment-disjoint splits through a fixed OLAT basis, and clearly outperforms LuxDiT (angular error 4.56 -> 2.08, PSNR 25.03 -> 32.19).
Critical weaknesses
  1. The main contributions are not isolated. There is no final-model ablation with and without LumiWild; the only evidence is same-light reconstruction (Table 9), which the paper itself says can be solved by copying input shading. The delighting model, used on every wild video, is never evaluated. ORBIT is never compared with a global lighting embedding; Tables 6-7 differences are within 0.16 dB with 2 seeds. The relighter is also nearly insensitive to global HDR intensity (slope 0.028, App. C), which conflicts with "precise" relighting.

  2. Table 1 does not show a fair comparison. The FaceOLAT GT is rendered from the same OLAT basis used in training, with flat-lit studio inputs matching the training distribution. Most baselines receive only text or a background instead of the HDR map (Table 8), and it is not stated whether test subjects are disjoint from training. There is no quantitative evaluation of changed-light relighting on in-the-wild videos, and Table 1 has no Image/Joint Relighter rows to support the claim that image-level quality is retained.

Questions for authors
  1. Please report the final model trained without LumiWild (0% VFHQ) on Tables 1-2.
  2. Are the FaceOLAT and BecomingLit test subjects held out from all training stages?
  3. Please compare with HDR-conditioned baselines (e.g., DiffusionRenderer) under the same input, and evaluate LumiForge with natural inputs delit by G_del.
  4. How accurate is the delighting model, and which components (image relighter, ORBIT, joint model) are from prior work versus this paper?
2 comments

Response to Reviewer 1BvC (1/2)

We thank the reviewer for the precise questions. We answer each weakness and question in order.

W1(a) — With vs. without LumiWild.

The paper contains this comparison, though we should have labelled it more clearly. The Image Relighter is trained without LumiWild and is applied frame by frame; the Joint Video Relighter is trained with LumiWild. Table 9 compares them (PSNR 20.34 → 25.77, flicker 1.851 → 1.201), and Fig. 6 shows the same models under target environments (top row: image relighter; lower rows: video models).

Because same-light reconstruction can partly be solved by copying input shading, we add a changed-light control where copying cannot help: the input is flat-lit and the target is a different light. On held-out BecomingLit (20 videos × 65 frames, same checkpoint, sampler and script):

Temporal Δ ↓Warped Δ ↓
Without LumiWild (per-frame Image Relighter)1.5281.414
LumiForge (with LumiWild)1.169 (−24%)1.156 (−18%)

The gain holds in 20/20 videos (Wilcoxon p = 2e-6), with per-frame PSNR unchanged (+0.04 dB). LumiWild supplies real motion under an explicit environment, which trains temporal coherence; lighting accuracy comes from OLAT supervision.

Amount of LumiWild data (new). Same protocol, checkpoints from our training run as the LumiWild subset grows (training length also grows, so this is not a fixed-budget ablation):

LumiWild clipsPSNR ↑LPIPS ↓Temporal Δ ↓Warped Δ ↓
0 (per-frame Image Relighter)18.480.1071.5281.414
6418.580.1091.2261.182
51218.580.1111.2061.179
14,158 (LumiForge)18.520.1131.1691.156

Temporal Δ decreases at every step in 20/20 videos, and warped Δ from 512 to 14,158 clips in 20/20, while PSNR stays within 0.1 dB and LPIPS within 0.006: more labelled wild video buys temporal coherence without costing lighting fidelity.

Labels from another estimator. Noisier labels would teach wrong light–appearance pairs and erode the OLAT-learned accuracy. LuxDiT's angular error is 2.2× that of G_env (4.56° vs. 2.08°, Table 3), so its labels are strictly less accurate; consistent with this, DiffusionRenderer, fine-tuned on real videos auto-labelled by its own inverse model, follows the target light poorly (Table 1).

W1(b) — The delighting model. Now evaluated; see Q4.

W1(c) — ORBIT vs. a global embedding.

We trained the same model with the 331 tokens replaced by their mean, i.e. a global lighting code (same initialization, data and budget):

Global code vs. ORBITΔPSNRORBIT better
Held-out multi-light combinations (Table 7 set; 20 × 2 seeds)−1.06 dB40 / 40
Natural HDRIs (6 subjects × 12 HDRIs × 2 seeds)−0.17 dB88 / 144

The gap grows as light concentrates (20 lights −0.46 dB → 2 lights −1.51 dB); it mainly reflects how much light and which colour reach the face (ORBIT is closer to the reference exposure in 40/40 combinations). Since PSNR is a weak measure of direction, we also add a shading-direction metric (albedo removed with the flat-lit input): on the 48 FaceOLAT pairs of Table 1, LumiForge's direction error is 25° (median 15°), vs. 71°–100° for every baseline (chance: 90°).

W1(d) — Global intensity (slope 0.028).

This slope measures the response to multiplying the whole environment by a constant, i.e. exposure, not lighting control. Training targets are auto-exposed (normalized by their 99th percentile), as portrait footage is, so outputs stay well exposed for any HDR scale. Everything that defines the lighting is followed closely:

  • relative intensity between light regions: log-log slope 0.953 (App. C);
  • requested strength of individual light regions: graded and additive (Fig. 9);
  • direction and colour: token edits (Fig. 3), paired warm/cool and colour references (Fig. 5), and the ORBIT control above.

We will define "precise" as precise in direction, colour and relative intensity, with absolute exposure left to the camera, as in photography.

Response to Reviewer 1BvC (2/2)

W2 — Fair comparison in Table 1; wild-video evaluation.

  1. Splits. Test subjects and test HDR environment maps are held out from all training stages, and the test and validation sets are disjoint (Q2). FaceOLAT targets share the light-transport source of our training data, so BecomingLit (a different capture system) and naturally lit inputs (Q3) are the stricter tests; LumiForge also leads there.

  2. Lighting interfaces. HDR-capable methods (DiffusionRenderer, Light-X) receive the same HDR map as LumiForge; others receive an HDRI-derived background and text, the interface their authors intend (Table 8). We re-ran DiffusionRenderer with its default 180° environment rotation compensated (Q3); the ranking is unchanged. While auditing interfaces we also found that our UniLumos background video lacked the per-frame noise-filled subject slot used in the official examples; with the corrected input UniLumos improves from 12.18 / 0.2320 to 12.23 / 0.1441 (FaceOLAT) and from 10.08 / 0.3544 to 14.28 / 0.1921 (BecomingLit), still well below LumiForge, and we will update Table 1 accordingly.

  3. Changed-light wild videos (new). 100 VFHQ videos, randomly sampled: 20 held-out subjects × 5 target environments (indoor, studio, sun, overcast, sunset/night), 65 frames each. Without ground truth, we measure three properties, each with a control:

    • Identity: ArcFace similarity to the source video 0.884 (0.855 counting undetected frames as 0); input vs. source 0.975, different people 0.008.
    • Temporal stability: flow-warped frame change is 0.95× that of the input (2.09 vs. 2.20); a frame-shuffled control gives 17.3.
    • Lighting (round trip): an independent estimator, LuxDiT, applied to our outputs recovers lighting closer to the target than to the subject's 4 other targets: direction error 84.8° vs. 97.7° (p = 2e-4), irradiance correlation 0.12 vs. −0.09 (p = 2e-6). Its absolute error mainly reflects LuxDiT itself: on real FaceOLAT relit photographs it gives 77.6° vs. 94.4°.

    Four of the 100 outputs are too dark and are included in all statistics.

  4. Image / Joint rows. The Image Relighter equals LumiForge at T = 1, so its FaceOLAT entry is identical (27.28 / 0.0262), which is what "image-level quality is retained" means. On BecomingLit, under the changed-light protocol of W1(a), the two differ by only 0.04 dB. The Joint Video Relighter is in Table 2 (warped Δ 1.404 vs. 1.280 for LumiForge, with 7× more trainable parameters). We will add both rows to Table 1.

Q1 — Final model without LumiWild.

Without LumiWild there is no environment-labelled video to train the temporal modules, so the 0%-VFHQ model is the per-frame Image Relighter. Under the changed-light protocol (W1a) it is identical on FaceOLAT and within 0.04 dB on BecomingLit, while LumiWild reduces its temporal errors by 24% / 18%. Under the paper's protocol, Table 2 already lists it (temporal / warped Δ 1.680 / 1.544 vs. 1.283 / 1.280 for LumiForge).

Q2 — Held-out subjects.

Yes. The FaceOLAT and BecomingLit test subjects and the test HDR environment maps are held out from all training stages (OLAT-basis pre-training, environment relighting, G_env, G_del and the temporal modules), and the test and validation sets are disjoint from each other. We will add a per-stage split table.

Q3 — HDR baselines under the same input; natural inputs delit by G_del.

Same input. All methods receive identical flat-lit frames and masks; DiffusionRenderer and Light-X receive our HDR map through their native interfaces. With DiffusionRenderer's environment rotation compensated:

DiffusionRenderer (PSNR / LPIPS)FaceOLATBecomingLit
Table 118.10 / 0.098316.04 / 0.1534
Rotation compensated17.75 / 0.099015.72 / 0.1656

Natural inputs. For each BecomingLit reference with target light T, the input is G_del applied to the same performance captured under a different light S ≠ T, i.e. the full ILC pipeline on real lit video. Relative to studio flat-lit input: PSNR −0.70 dB, LPIPS better by 0.012, warped Δ 1.156 → 1.021. The PSNR drop is smaller than LumiForge's margin over every baseline in Table 1, so LumiForge transfers to naturally lit inputs.

Q4 — Delighting accuracy; provenance.

G_del on held-out BecomingLit subjects (30 videos × 65 frames), against flat-lit ground truth:

Source lightingPSNRSSIMLPIPS
LED tints (10 videos)32.280.9290.014
Natural HDR environments (20 videos)25.010.8810.079

Provenance. All components are introduced and trained in this paper: the image relighter, ORBIT, the identity branch, G_env, G_del, the Joint Video Relighter and the temporal residual modules. Only the Wan2.1-Fun-14B-Control backbone and VAE, DINOv2 and ArcFace are used off the shelf.

Reviewer_KhUmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LumiForge addresses portrait video relighting.

1, LumiForge estimates illumination from real videos and produces a dataset, called LumiWild, which comprises 15k clips.

2, LumiForge proposes to represent illumination with 331 directional tokens and represent id with a separate ID-conditioning branch.

3, LumiForge reports remarkable fidelity gains over prior works.

Critical strengths

1, Strong results against existing baseline, as presented in Fig4, Fig5 and Tab1.

2, The paper validates its method on Wanvideo-14B, rather than only on a 1.3B model.

Critical weaknesses

1, Compared with a prior image-relighting work, IC-light v2, this work is limited on relative small dataset and the dataset domain is concentrated on portait videos with a large head.

2, The idea and method are similar to the prior work (Lux Post Facto, CVPR2025).

3, It’s hard to tell how much the estimated lighting labels actually help the model relight videos under new lighting.

Minor:

The ablation of orbit encoding in Tab6 shows only small improvements.

Questions for authors

1, what's the nolvety compared to Lux Post Facto

2, How could this work be extended to in-the-wild settings

2 comments

Response to Reviewer KhUm (1/2)

We thank the reviewer for recognizing our comparisons and the 14B-scale validation. We answer each point in order.

W1 — Comparison with IC-Light v2: data scale.

  • A different task. IC-Light v2 is trained on >10M images conditioned on text or a background, without any illumination label. It therefore transfers the look of a lighting (style transfer), but cannot render what a specified HDR environment would produce. LumiForge solves the latter: given an HDR map, it renders the corresponding lighting precisely. The consequence is measurable whenever ground truth exists: on FaceOLAT, IC-Light reaches 17.82 dB vs. 27.28 dB for LumiForge (Table 1), and its shading direction is unrelated to the target (direction error 88° vs. 25°, where 90° is chance; new metric, see Minor).
  • The right data for precise control. Precision requires measured light transport, which text- or background-labelled images cannot provide. Our image training uses FaceOLAT (139 subjects × 331 lights × 40 views, 4K), a recent and the largest public OLAT face dataset at its release, rendered under 6,769 HDR environments (indoor, outdoor, studio, night), i.e. about 6.9M relit images. LumiWild adds 15,191 real videos (987,415 frames), each with an explicit HDR environment.

W2 / Q1 — Novelty compared with Lux Post Facto.

Lux Post Facto learns HDR-conditioned relighting only from static light-stage (OLAT) frames. Its in-the-wild videos carry no environment map, so they are used only for reference-based relighting: the model copies the appearance of a lit reference frame onto the video (in the authors' words, "for videos from the motion-rich dataset, we only use the reference-based conditioning"). This has two consequences:

  1. Reference conditioning is appearance transfer. It can reproduce a lighting that some frame already shows, but cannot express an arbitrary environment, so on moving faces Lux never learns environment-conditioned relighting.
  2. Precision and motion are learned from disjoint data. Lighting accuracy comes from static frames, temporal behaviour from unlabelled videos.

LumiForge removes this split. ILC estimates an explicit HDR environment (G_env) and a flat-lit input (G_del) for every wild clip, so wild videos train the same environment-conditioned task as OLAT data: precise control and real motion are learned together.

Because this design relies on the recovered labels, the environment estimator and the delighting model must both be accurate, and we evaluate both: G_env reaches 2.08° angular error vs. 4.56° for LuxDiT (Table 3), and G_del reaches 32.28 dB (LED) / 25.01 dB (natural HDR) against flat-lit ground truth on held-out BecomingLit subjects (new). To our knowledge Lux Post Facto's code and weights are not public, so we compare in the related work.

Response to Reviewer KhUm (2/2)

W3 — How much do the estimated lighting labels help?

The two data sources have different roles. FaceOLAT (~6.9M images) teaches lighting accuracy; LumiWild (~1M frames) supplies what OLAT cannot: real motion (temporal signal) and in-the-wild appearance, each frame paired with an explicit environment.

  • With vs. without LumiWild (BecomingLit, 20 held-out videos × 65 frames, changed light, same script). Without LumiWild the model is the per-frame image relighter. LumiWild reduces the temporal Δ error by 24% (1.528 → 1.169) and the warped Δ error by 18% (1.414 → 1.156), in 20/20 videos (p = 2e-6), with per-frame fidelity unchanged (+0.04 dB).

  • Amount of LumiWild data (new; same protocol; checkpoints from our training run as the LumiWild subset grows, so training length also grows):

    LumiWild clipsPSNR ↑LPIPS ↓Temporal Δ ↓Warped Δ ↓
    0 (per-frame Image Relighter)18.480.1071.5281.414
    6418.580.1091.2261.182
    51218.580.1111.2061.179
    14,158 (LumiForge)18.520.1131.1691.156

    Temporal Δ decreases at every step in 20/20 videos, and warped Δ from 512 to 14,158 clips in 20/20, while PSNR stays within 0.1 dB and LPIPS within 0.006. The labels thus let wild video improve temporal coherence steadily without degrading the lighting accuracy learned from OLAT.

  • Labels from another estimator. Inaccurate labels would teach wrong light–appearance pairs and erode the relighting ability learned from OLAT. LuxDiT's angular error is 2.2× that of G_env (4.56° vs. 2.08°, Table 3), so its labels would be strictly noisier. Consistent with this, DiffusionRenderer, whose forward renderer is fine-tuned on real videos auto-labelled by its own inverse model, follows the target lighting poorly (Table 1; shading direction error 77°). This is why ILC uses a dedicated, evaluated estimator.

  • New lighting on 100 wild videos (20 held-out subjects × 5 environments): identity is preserved (ArcFace 0.884), temporal change stays at 0.95× that of the input, and an independent estimator (LuxDiT) recovers lighting closer to the target than to mismatched environments (p = 2e-4).

Q2 — Extension to in-the-wild settings.

This is already what LumiForge does: G_env estimates the wild video's environment, G_del removes the source lighting, and the relighter renders it under any given HDR map. Evidence:

  • the 100-video wild evaluation above (details in 1BvC-W2);
  • natural inputs: on BecomingLit, feeding G_del applied to the same performance under a different light changes PSNR by only −0.70 dB relative to studio flat-lit input, with better LPIPS (−0.012) and temporal stability (warped Δ 1.156 → 1.021);
  • synchronized 65-frame videos on wild clips with large head motion against 11 baselines, and new results on held-out subjects: https://anonymous.4open.science/r/lumiforge-material-FE60/

Current limits, which we will list: full-body motion, heavy occlusion, and lighting that changes within a clip.

Minor — Small ORBIT gains in Table 6.

Table 6 varies components inside the 331-token encoder, and PSNR is dominated by low-frequency brightness, so it is a weak measure of direction. We add two measurements:

  1. Global-code control. The same model with the 331 tokens replaced by their mean (same initialization, data and budget) loses 1.06 dB on held-out multi-light combinations (40/40) and 0.17 dB on natural HDRIs (88/144); the gap grows as light concentrates (20 lights −0.46 dB → 2 lights −1.51 dB). It mainly reflects how much light and which colour reach the face: ORBIT is closer to the reference exposure in 40/40 combinations and to the reference colour in 98/144 natural HDRIs.
  2. A direction-specific metric. We remove albedo by dividing each output by the flat-lit input and compare the image-plane direction of the resulting shading gradient with that of the ground truth. On the 48 held-out FaceOLAT pairs of Table 1, LumiForge's shading-direction error is 25° (median 15°; shading correlation 0.80), while every baseline lies between 71° and 100° (correlation ≤ 0.36), i.e. close to chance (90°).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.