acceptodds
Under review as a conference paper at ICLR 2027

LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

Abstract

Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Physically-Based Rendering, Generative Modelling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mDxmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A video relighting method which uses DiffusionRenderer to extract per-frame PBR gbuffers, DiffusionLight to estimate the lighting (which is refined via inverse rendering), then renders a PBR approximation under novel lighting and refines that estimate via a video2video "fixer" network (a finetuned CogVideoX model).

The PBR approximation is a strong guidance signal and the approach produces convicing results. I also appreciate the training approach of the fixer network, which is trained on a mix of synthetic pairs, real-world unpaired videos and multi-illumination captures.

It is a neat but straightforward approach which combines a set of recent techniques, and inherits their limitations (e.g., roughness and metallic predictions of DiffusionRenderer are not always reliable), but the results are convincing.

I have a few concerns regarding the evaluation protocol, but they can likely easily be addressed in a minor revision, so I am leaning positive.

Critical strengths

Strong results on the video relighting task. Combining a proxy PBR rendering and a video2video uplifting task, makes the uplifting task easier than in e.g., albedo-RGB or X-RGB.

Nice training setup which leverages both paired synthetic data and real-world unpaired videos.

Critical weaknesses

The approach relies on DiffusionRenderer (DR) to extract video G-buffers in order to create the PBR proxy under novel lighting, and is limited by the quality of DRs predictions, which are not always accurate. Furthermore, this intoduces a gbuffer bottleneck in the pipeline, which makes it harder to handle transparent/translucent objects, and shading effects not handles by the PBR shading model. That said, the fixer model can compensate for this.

My other concern is that it looks like UniRelight is not configured correctly. In Fig 10-12 in the supplemental, UniRelight has almost no shadows, or shadows in the wrong direction, so I wonder if this baseline has been badly configured somehow Looking at the project page, synthetic benchmark, third example (rotating probe), the shadows seems to be in the wrong direction, so I supect a probe orientiation mismatch. There is also a significant intensity difference.

Comparing metrics scores on MIT multi-illumination between the UniRelight paper and LightCrafter, there are large differences (see below). This may be due to difference in evaluation settings, but could be clarified (and I would recommend to check for probe rotation and intensity mismatches also here).

PSNR stats in the UniRelight paper (MIT multi-illumination dataset):

UniRelight: 20.76dB DiffusionRenderer: 16.99dB

PSNR stats in this submission (MIT multi-illumination dataset):

UniRelight: 13.64dB DiffusionRenderer: 12.83dB LightCrafter: 20.27dB

Questions for authors

In Table 1, on MIT, PBR (Mitsuba renderings) have PSNR scores of only 13.59dB, but are then uplifted to 20.27dB. Looking at Fig 6 and Fig 14, PBR on MIT looks ok. Are these just easy examples? Is it mainly the black pixels that are the root cause of the low PSNR scores for PBR?

Also in Table 1. UniRelight performs consistently worse than DiffusionRenderer, which is not the case in the UniRelight paper (Table 1), where UniRelight is consistently better. Which version of DiffusionRenderer is used here, the Cosmos or older Stable Video Diffusion version?

It is stated that PCRP (Careaga & Aksoy, 2025) trains a diffusion model on synthetic data which is not correct. That work shows you can train a neural rendering network that can handle any light source type, solely from self-supervision using real images.

As discussed above, I would highly recommend to check the UniRelight setup (probe orientation and intensity) and rerun metrics.

In Fig 3, the probe seems to be badly rotated for DiffusionRenderer (when looking at the left chromeball). Same on the diffuse example on top, the shadow is in another direction (hard to tell what is right, but could at least be worth checking if it is a probe rotation issue also here).

3 comments

Weaknesses

W1. It looks like UniRelight is not configured correctly. In Fig 10-12 in the supplemental, UniRelight has almost no shadows, or shadows in the wrong direction, so I wonder if this baseline has been badly configured somehow Looking at the project page, synthetic benchmark, third example (rotating probe), the shadows seems to be in the wrong direction, so I suspect a probe orientation mismatch. There is also a significant intensity difference.

A: We thank the reviewer for identifying this issue. We rechecked UniRelight’s light-probe configuration and indeed found an environment-map orientation and intensity mismatch. We corrected it using env_flip=True and env_rot=180 and, env_strength=1, and for videos with camera motion, we rotated the target environment map according to each frame’s camera pose. These corrections improve the lighting directions and reduce the excessive darkening observed in the previous results. We also clarify an important evaluation difference: UniRelight applies per-channel least-squares scaling before computing metrics, whereas our evaluation does not apply this scaling to any method. The scores reported in the two papers therefore follow different protocols and are not directly comparable. Using the corrected configuration and our unchanged evaluation protocol, we reran UniRelight with its released code and checkpoint, recomputed all metrics (reported in Table 1 below), and updated the qualitative results in the revision.

Questions

Q1. In Table 1, on MIT, PBR (Mitsuba renderings) have PSNR scores of only 13.59dB, but are then uplifted to 20.27dB. Looking at Fig 6 and Fig 14, PBR on MIT looks ok. Are these just easy examples? Is it mainly the black pixels that are the root cause of the low PSNR scores for PBR?

A: We agree that Figs. 6/14 are easy cases but the black border pixels are not the root cause. The low average PBR score is mainly caused by some challenging cases in the evaluation set. Fig. 15 provides challenging MIT examples under intense illumination. In this case, our PBR proxy fails to reproduce large saturated bright regions and strong highlights, because chrome balls only capture incoming light at a local region rather than a true environment map that captures light from infinitely far sources. For the two examples shown, PBR achieves only 11.71 and 10.45 dB.

Challenging MIT Multi-Illumination examples. Refer to revision figure 15 on page 33.

Q2. Also in Table 1. UniRelight performs consistently worse than DiffusionRenderer, which is not the case in the UniRelight paper (Table 1), where UniRelight is consistently better. Which version of DiffusionRenderer is used here, the Cosmos or older Stable Video Diffusion version?

A: Yes. As the appendix runtime table cites, we used the Cosmos (7B) release. However, the issue is found and described as in W1, and with the adjustment, the ranking flips, as shown in Table 1 below.

Q3. It is stated that PCRP (Careaga & Aksoy, 2025) trains a diffusion model on synthetic data which is not correct. That work shows you can train a neural rendering network that can handle any light source type, solely from self-supervision using real images.

A: Thanks for the correction; in the revision, the Sec. 2 sentence read as: “PCRP trains a neural rendering network self-supervised on real photographs to refine PBR renders from monocular inverse rendering, handling arbitrary light sources.”

Q4. As discussed above, I would highly recommend to check the UniRelight setup (probe orientation and intensity) and rerun metrics. A: Done; see W1 and Table 1 below.

Q5. In Fig 3, the probe seems to be badly rotated for DiffusionRenderer (when looking at the left chromeball). Same on the diffuse example on top, the shadow is in another direction (hard to tell what is right, but could at least be worth checking if it is a probe rotation issue also here).

A: Thank you for pointing this out. Our submission used the codebase’s default probe settings (env_flip=True, env_rot=180). After sweeping the probe orientations, we found that a 90° yaw rotation noticeably improves the alignment of shadows and highlights. Therefore, with the new offset, we reran DiffusionRenderer on all evaluation sets and updated all the figures and quantitative results in the revision accordingly (Table 1 below).

Table 1 with the updated UniRelight and DiffusionRenderer row.

MethodSynthetic PSNR↑Synthetic SSIM↑Synthetic LPIPS↓Synthetic T-CLIP↑Synthetic Warp-SSIM↑MIT PSNR↑MIT SSIM↑MIT LPIPS↓DL3DV PSNR↑DL3DV SSIM↑DL3DV LPIPS↓
Light-X13.540.51570.39160.98530.944814.040.59330.438113.760.38130.4523
UniRelight (submission)18.870.54750.32020.98320.958413.640.51980.297711.980.33540.5511
UniRelight (updated)20.980.58490.29200.98710.957515.010.60150.237516.730.52470.3401
DiffusionRenderer (submission)20.920.71300.28670.98390.957412.830.55810.304815.330.55510.4244
DiffusionRenderer (updated)21.720.72450.28790.98360.952212.950.55900.299716.840.59300.4222
PCRP-video20.780.58520.30990.98380.950713.570.52530.289018.570.52140.2257
PBR19.590.71110.31060.97550.922313.060.39840.486717.750.50820.4768
LightCrafter24.160.73660.20960.98920.959815.070.61590.219319.550.66950.2030

Thanks for the updated results. I am glad to see the corrected probe orientations on UniRelight and DiffusionRenderer in this revision.

Reviewer_DB83OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LightCrafter formulates video relighting as refinement of a physically rendered proxy. It estimates scene geometry and material properties from the input video, renders a proxy under the target illumination, and uses a fine-tuned CogVideoX model to correct reconstruction and rendering artifacts. The main contributions are the application of this pipeline to video and an artifact-matched training strategy combining synthetic relighting pairs, real-video reconstruction pseudo-pairs, and real multi-illumination captures. Evaluation includes paired synthetic videos, MIT multi-illumination images, a DL3DV round-trip protocol, human preferences, and data-source and reconstruction-error analyses. Overlapping-window noise fusion supports longer sequences, and qualitative examples demonstrate material editing, light insertion, and object insertion.

Critical strengths
  1. The separation between physical control and generative refinement is well motivated. Rendering the target illumination into a frame-aligned proxy provides an explicit control interface and reduces the burden on the diffusion model to infer lighting effects from abstract conditions.
  2. The reported improvements are supported by several complementary experiments. Paired synthetic evaluation, real multi-illumination images, and human preferences show benefits over the unrefined PBR proxy and evaluated baselines. The reconstruction-error analysis also provides useful evidence about which upstream errors the refiner can correct.
Critical weaknesses

The methodological novelty is limited, and the central training contribution is not sufficiently isolated. As acknowledged in the related work, refining PBR renders with diffusion models has already been explored in image relighting and other rendering settings. LightCrafter extends this approach to video using existing reconstruction modules, a standard conditional video diffusion objective, and overlapping-window inference. Its principal distinction therefore appears to lie in the training-data construction. However, the data ablations remove entire sources rather than comparing artifact-matched and clean conditioning under otherwise matched training conditions. The PCRP-Video comparison also changes the supervision construction, making it difficult to attribute the improvement to the proposed formulation rather than differences in supervision. The experiments demonstrate a useful system, but do not clearly establish the methodological advance beyond combining existing components with curated training data.

Questions for authors
  1. Artifact-matched supervision: With the backbone, target videos, training budget, and data volume held fixed, how much improvement comes specifically from training on proxies generated from estimated scene properties rather than ground-truth properties? This would directly test the main training contribution.
  2. Temporal inference: Do the long-form consistency gains remain when compatible baselines receive comparable overlapping-window processing? Conversely, how does LightCrafter perform when its windows are processed independently? Separating boundary and within-window errors would help identify the source of the improvement.
  3. Practical applicability: Appendix reports approximately 15 minutes of processing for a 49-frame clip at 480 × 720 on an A6000. Given this cost and the dependence on reconstructable geometry and materials, which application settings benefit enough from explicit PBR control to justify the pipeline over direct video editing? Can the quality–cost trade-off be quantified against the evaluated alternatives?
4 comments
AuthorsOpenReview ↗

Response to W1 and Q1

Weaknesses

W1. The methodological novelty is limited, and the central training contribution is not sufficiently isolated.

A: Limited novelty. DB83 argues that extending image-based PBR diffusion refiners to video is not novel. While we agree that the extension to video is relatively straightforward, it is nontrivial because maintaining consistency across video frames is challenging; Fig. 4 (original draft) shows many end-to-end video relighting methods struggle with this. The key contribution we show in this paper is the insight that the consistency of the underlying relit PBR reconstruction empowers the consistency of the final relit video from the video refiner. Indeed Table 1 (original draft) shows that PBR already outperforms some prior art. Moreover, we see our main training contribution as our training data pipeline. We ablate how our novel "artifact-matched" data creation recipe improves relighting accuracy in Table 10 below and Figs. R3a, R3b, and R3c. While both refiners improve on the PBR proxy, the artifact-matched model outperforms in PSNR, SSIM, T-CLIP, and Warp-SSIM while LPIPS is around the same. At inference time, a refiner trained with artifact-matched data can more robustly fix both geometry and material errors in the PBR.

Table 10. Ablation of LightCrafter training on artifact-matched proxies vs. GT G-buffer proxies, evaluated on the synthetic test set. Both refiners are fed the same test-time PBR proxy (constructed by predicted intrinsics and geometry properties).

Refiner (fed pred-G-buffer, MegaSAM-mesh proxy)PSNR↑SSIM↑LPIPS↓T-CLIP↑Warp-SSIM↑
none (PBR proxy)19.590.71110.31060.97550.9223
trained on GT G-buffer proxies22.790.68830.20940.98640.9379
trained on artifact-matched proxies24.160.73660.20960.98920.9598

R3a: Geometry error comparison on synthetic data at inference time. Refer to revision figure R3a on page 27.

R3b: Shadow artifact comparison on a real lighthouse video. Refer to revision figure R3b on page 27.

R3c: Material error comparison at inference time. Refer to revision figure R3c on page 27.

Questions

Q1. Artifact-matched supervision: With the backbone, target videos, training budget, and data volume held fixed, how much improvement comes specifically from training on proxies generated from estimated scene properties rather than ground-truth properties? This would directly test the main training contribution.

A: See W1 above.

AuthorsOpenReview ↗

Response to Q2

Q2. Temporal inference: Do the long-form consistency gains remain when compatible baselines receive comparable overlapping-window processing? Conversely, how does LightCrafter perform when its windows are processed independently? Separating boundary and within-window errors would help identify the source of the improvement.

A: LightCrafter outperforms all baselines when all methods are rerun with the overlap fusion method (Table X-b below) or independent window processing (Table X-a below): it has the best PSNR, SSIM, LPIPS and T-CLIP with and without fusion (PSNR +2.2 dB without and +2.4 dB with fusion over the best baseline), and the best Warp-SSIM with fusion. As the reviewer suggests, we separately evaluate temporal errors for frame pairs spanning window boundaries and find even larger performance wins for LightCrafter, both in the independent window processing (Table Y-a) and fusion (Table Y-b) setting. Without fusion, whole-clip Warp-SSIM varies by only about 0.01 across methods, whereas on the boundary pairs, Lightcrafter reaches 0.901 against 0.709–0.858 for the baselines, showing Lightcrafter already produces consistent chunks even before fusion. Figures R-seam (a)–(b) illustrate these boundary inconsistencies without fusion on synthetic and real-world videos, respectively.

Finally, we observe that although applying overlap fusion produces improvement at the window boundaries for every method, it often results in blurry frames for the baselines, which boosts PSNR and SSIM but hurts LPIPS for over all frames. Figure R-fusion (b) shows examples of baselines' ghosting and blurriness in the overlapping frames after fusion. In contrast, LightCrafter improves on every metric with fusion, due to the consistency in our relight PBR renderings.

This is because when the windows agree (ours, PCRP-Video, and to a large extent Light-X), averaging results in smoothing hence improved metrics. When they disagree (UniRelight and DR re-interpret the lighting in every window, generating inconsistent relighting results), the boundaries become smoother after fusion, but the overlap becomes a superposition of two different videos. Therefore, accuracy degrades or barely changes over all frames (LPIPS for both, UniRelight's PSNR, DR's SSIM), whereas T-CLIP stays within 0.004: it compares semantic content of adjacent frames, and a blurred frame is still semantically the same as its neighbour. Figure R-fusion (a) further illustrates the remaining drift in color, shadows, and scene details across fused chunks, alongside the corresponding input frames.

Table X-a. All methods without fusion, over all frames (hard concat of independent windows)

MethodPSNR↑SSIM↑LPIPS↓T-CLIP↑Warp-SSIM↑
Light-X13.530.51540.40440.98520.9495
UniRelight21.350.57790.27750.98350.9568
DiffusionRenderer20.950.72350.27570.98430.9565
PCRP-Video20.400.57890.31300.98210.9480
PBR19.590.71110.31060.97550.9223
LightCrafter23.530.73050.21130.98760.9570

Table X-b. All methods with the overlap fusion method, over all frames (PBR: not applicable)

MethodPSNR↑SSIM↑LPIPS↓T-CLIP↑Warp-SSIM↑
Light-X13.540.51570.39160.98530.9448
UniRelight20.980.58490.29200.98710.9575
DiffusionRenderer21.720.72450.28790.98360.9522
PCRP-Video20.780.58520.30990.98380.9507
LightCrafter24.160.73660.20960.98920.9598

Table Y-a. All methods without fusion, over boundary frames / Table Y-b. All methods with fusion, over boundary frames

MethodY-a Warp-SSIM↑Y-a T-CLIP↑Y-b Warp-SSIM↑Y-b T-CLIP↑
Light-X0.8580.9510.93060.9705
UniRelight0.7990.9440.92560.9659
DiffusionRenderer0.7090.9360.92610.9538
PBR proxy (refiner input)0.8920.954––
LightCrafter0.9010.9710.96800.9891

R-seam (a): Boundary consistency without fusion on synthetic videos. Refer to revision figure R-seam (a) on page 30.

R-seam (b): Boundary consistency without fusion on real-world videos. Refer to revision figure R-seam (b) on page 30.

R-fusion (a): Cross-chunk consistency with fusion on real-world videos. Refer to revision figure R-fusion (a) on page 30.

R-fusion (b): Overlap-region artifacts with fusion on synthetic videos. Refer to revision figure R-fusion (b) on page 31.

AuthorsOpenReview ↗

Response to Q3

Q3. Practical applicability: Appendix reports approximately 15 minutes of processing for a 49-frame clip at 480 × 720 on an A6000. Given this cost and the dependence on reconstructable geometry and materials, which application settings benefit enough from explicit PBR control to justify the pipeline over direct video editing? Can the quality–cost trade-off be quantified against the evaluated alternatives?

A: We argue that the focus of our work is relighting accuracy and consistency, not processing time. With the advancement of monocular video reconstruction and inverse rendering tools, as well as one-step diffusion distillation techniques, our pipeline can be significantly sped up.

Under the same hardware, video length, and output resolution, Light-X takes 5m 22s, DiffusionRenderer takes 12m 28s, UniRelight takes 13m 45s, and PCRP-Video takes 12m 43s per clip, while our full pipeline takes 12m 43s (3m 56s amortized per subsequent edit after inverse rendering). While the runtimes are on the same order of magnitude, our method leads on relighting accuracy and temporal consistency (Table 1 in original paper); the quality–cost trade-off is plotted in the revision figure Quality vs. runtime on page 31. Unlike baselines that rerun the full pipeline for each edit, LightCrafter can amortize the relighting cost by reconstructing (SFM+inverse rendering) each video only once. Subsequent edits, including changing environment maps, adding lights, modifying materials, or inserting objects, require only rendering and refinement, costing ≈ 4 min per edit.

Our PBR-guided approach is particularly suited to offline relighting under a specified environment map, especially for long videos where lighting fidelity and temporal consistency take priority over interactive speed. Movie editing is an ideal application for generating realistic relit videos for studio-captured characters consistent over a long span of frames and scenes. Our method can also be used as a simulator that generates realistic simulated clips for self-driving or robotics applications under different lighting conditions, where photorealism alone is insufficient: shadow direction and location, surface shading, and specular highlights should consistently follow the target illumination.

Quality vs. runtime. Refer to revision figure Quality vs. runtime on page 31.

We thank the authors for comprehensive feedback. My concerns are well-solved. I will raise my rating to WA.

Reviewer_5fZPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LightCrafter formulates video relighting as PBR proxy refinement. Off-the-shelf inverse rendering (DiffusionRenderer, MegaSAM, DiffusionLight) recovers geometry, materials and lighting; Mitsuba renders the scene under the target light; and a fine-tuned CogVideoX-5B refines the artifact-laden PBR video into a realistic one. Training pairs are "artifact-matched", i.e., produced by the same reconstruction stack used at test time, from synthetic Blender scenes, DL3DV pseudo-pairs and MIT multi-illumination pairs. Long videos are handled with noise-space overlap fusion. The method reports state-of-the-art results on a synthetic benchmark, MIT and a DL3DV round-trip protocol, plus a user study and light/material editing applications.

Critical strengths
  1. A clean and controllable formulation. Baking the target light into a physically rendered proxy avoids teaching the diffusion model environment-map semantics, and enables light insertion, material editing and object compositing with a single model.
  2. Useful analyses. The error-source study (Table 5) shows which upstream errors the refiner can fix (geometry much more than materials), and the paper is open about runtime and failure cases.
Critical weaknesses
  1. All quantitative gains are in-distribution. The synthetic test set shares the scene generator and HDR pool with training, and MIT and DL3DV both have training splits, while DiffusionRenderer, UniRelight and Light-X are evaluated zero-shot. Table 2 shows the gains vanish out of distribution: without synthetic data the model scores 15.86 dB on synthetic, below the raw PBR proxy (19.59); without real data it gets 16.48 dB on real data, below PCRP-video (17.97). In addition, the MIT target lighting source is never specified; if it is fitted to the target image via Eq. 1, the MIT evaluation is circular.

  2. Novelty and key claims are not isolated. PBR refinement with diffusion exists (PCRP, DiLightNet, DiffusionHarmonizer, GR3EN); the new part is mainly the artifact-matched data, but there is no control trained on GT G-buffer proxies. The long-video comparison concatenates baseline chunks while the proposed method uses overlap fusion, so Fig. 4 compares inference protocols rather than methods, and Table 1 temporal differences are tiny.

Questions for authors
  1. How is the target environment map obtained for MIT: from the dataset's light probes or by fitting Eq. 1 to the target image? Do all methods receive the same one?
  2. Please add the control trained on GT G-buffer proxies (no artifact matching), and the w/o-MIT ablation promised in the Table 2 caption.
  3. Please re-run the long-video comparison with overlap fusion applied to the baselines, and add at least one benchmark that no method was trained on.
  4. Is the PBR latent dropped in the unconditional CFG branch? Otherwise guidance would have no effect.
5 comments
AuthorsOpenReview ↗

Response to W1 and W2

Weaknesses

W1. All quantitative gains are in-distribution. The synthetic test set shares the scene generator and HDR pool with training, and MIT and DL3DV both have training splits, while DiffusionRenderer, UniRelight and Light-X are evaluated zero-shot. Table 2 shows the gains vanish out of distribution: without synthetic data the model scores 15.86 dB on synthetic, below the raw PBR proxy (19.59); without real data it gets 16.48 dB on real data, below PCRP-video (17.97).

A: Table 2 should not be interpreted to measure our method's out-of-distribution (OOD) generalization. Real world dataset provides realism while synthetic dataset provides relighting pairs to ensure accuracy, helping refiner correct artifacts from PBR. They complement with each other to help train a generalizable model.

While evaluating on out-of-distribution data is ideal, relighting video pairs are cumbersome to collect. We follow the status quo in the relighting literature and evaluate on both synthetic and real data, using standard train and test splits (He et al., 2025). In fact, much prior work even evaluates only on in-domain synthetic data (Jin et al., 2024; Liang et al., 2025).

To measure OOD performance, we also evaluate on an additional real-world benchmark of WildRelight (Wang et al., 2026), which provides true background environment maps at different times of the day captured using a 360 camera rig. No evaluated method, including ours, has seen this data. We use the captured raw HDR environment map as input for all methods and measure relighting accuracy; Table 7 and Fig. R-WildRelight show that LightCrafter is still state-of-the-art and can generalize to OOD data.

Table 7. WildRelight (OOD): captured target illumination. Same processed HDR environment map for every method; no method was trained on this data.

MethodPSNR↑SSIM↑LPIPS↓
Light-X18.040.5530.466
DiffusionRenderer16.630.5540.392
UniRelight18.680.6240.407
Ours (PBR proxy only)18.580.5310.353
Ours20.190.6290.305

W2: In addition, the MIT target lighting source is never specified; if it is fitted to the target image via Eq. 1, the MIT evaluation is circular.

A: Table 1 in the original paper does fit a target environment map to the target ground-truth using PBR (eq. 1), while baselines use the provided chrome ball maps. This is because chrome balls capture incoming light at a local region (so they also contain foreground geometry) rather than a true environment map that captures light from infinitely far sources (which is the representation our model is trained on). Note that prior relighting methods (e.g. PCRP) follow this protocol for evaluation on real-world datasets. That said, we agree this may leak information so we also evaluate a variant of our model that uses the provided chrome balls as the environment map. Table 8 and Figs. R-MIT–R-WildRelight show that LightCrafter (Ours) is still state-of-the-art when using chrome-ball maps despite UniRelight (the closest baseline) was trained on chrome ball maps as input, though there is a performance drop due to the map mismatch.

Table 8. MIT under the same chrome-ball map for every method. No per-image lighting optimization. “Ours” is the chrome-ball variant; “Ours (test-time envmap optimization)” is the submission protocol that fits the map to the target photo (paper Table 1).

MethodPSNR↑SSIM↑LPIPS↓
Light-X14.040.59330.4381
UniRelight15.010.60150.2375
DiffusionRenderer12.950.55900.2997
Ours (PBR proxy only)13.060.39840.4867
Ours15.070.61590.2193
Ours (test-time envmap optimization)20.270.82050.2114

R-MIT. Qualitative comparison on MIT with the same chrome-ball map for every method. Refer to revision figure R-MIT on page 25.

R-WildRelight. OOD real photographs under captured target illumination. Refer to revision figure R-WildRelight on page 25.

AuthorsOpenReview ↗

Response to W3

W3. Novelty and key claims are not isolated.

A: Limited novelty. DB83 argues that extending image-based PBR diffusion refiners to video is not novel. We respectfully disagree. Video relighting is nontrivial because maintaining consistency across video frames is challenging; Fig. 4 (original draft) shows many end-to-end video relighting methods struggle with this. One key insight is that the consistency of the underlying relit PBR reconstruction empowers the consistency of the final relit video from the video refiner. Indeed Table 1 (original draft) shows that PBR already outperforms some prior art. Moreover, we also agree that our training data pipeline can be seen as a core contribution. We ablate how our novel "artifact-matched" data creation recipe improves relighting accuracy in Table 10 below and Figs. R3a, R3b, and R3c. Compared with a baseline trained on synthetic pairs with ground-truth reconstruction inputs, the artifact-matched model clearly outperforms on PSNR, SSIM, T-CLIP, and Warp-SSIM and is comparable on LPIPS. More importantly, because it doesn't rely on ground-truth reconstruction inputs, it can be trained on pairs built from real-world videos, which substantially improves real-world accuracy (Table 2 in original draft).

Table 10. Ablation of LightCrafter training on artifact-matched proxies vs. GT G-buffer proxies, evaluated on the synthetic test set. Both refiners are fed the same test-time PBR proxy (constructed by predicted intrinsics and geometry properties).

Refiner (fed pred-G-buffer, MegaSAM-mesh proxy)PSNR↑SSIM↑LPIPS↓T-CLIP↑Warp-SSIM↑
none (PBR proxy)19.590.71110.31060.97550.9223
trained on GT G-buffer proxies22.790.68830.20940.98640.9379
trained on artifact-matched proxies24.160.73660.20960.98920.9598

R3a: Geometry error comparison on synthetic data at inference time. Refer to revision figure R3a on page 27.

R3b: Shadow artifact comparison on a real lighthouse video. Refer to revision figure R3b on page 27.

R3c: Material error comparison at inference time. Refer to revision figure R3c on page 27.

AuthorsOpenReview ↗

Response to W4 and Q1

W4. The long-video comparison concatenates baseline chunks while the proposed method uses overlap fusion, so Fig. 4 compares inference protocols rather than methods, and Table 1 temporal differences are tiny.

A: LightCrafter outperforms all baselines when all methods are rerun with the overlap fusion method (Table X-b below) or independent window processing (Table X-a below): it has the best PSNR, SSIM, LPIPS and T-CLIP with and without fusion (PSNR +2.2 dB without and +2.4 dB with fusion over the best baseline), and the best Warp-SSIM with fusion. As the reviewer suggests, we separately evaluate temporal errors for frame pairs spanning window boundaries and find even larger performance wins for LightCrafter, both in the independent window processing (Table Y-a) and fusion (Table Y-b) setting. Without fusion, whole-clip Warp-SSIM varies by only about 0.01 across methods, whereas on the boundary pairs, Lightcrafter reaches 0.901 against 0.709–0.858 for the baselines, showing Lightcrafter already produces consistent chunks even before fusion. Figures R-seam (a)–(b) illustrate these boundary inconsistencies without fusion on synthetic and real-world videos, respectively.

Finally, we observe that although applying overlap fusion produces improvement at the window boundaries for every method, it often results in blurry frames for the baselines, which boosts PSNR and SSIM but hurts LPIPS for over all frames. Figure R-fusion (b) shows examples of baselines' ghosting and blurriness in the overlapping frames after fusion. In contrast, LightCrafter improves on every metric with fusion, due to the consistency in our relight PBR renderings.

This is because when the windows agree (ours, PCRP-Video, and to a large extent Light-X), averaging results in smoothing hence improved metrics. When they disagree (UniRelight and DR re-interpret the lighting in every window, generating inconsistent relighting results), the boundaries become smoother after fusion, but the overlap becomes a superposition of two different videos. Therefore, accuracy degrades or barely changes over all frames (LPIPS for both, UniRelight's PSNR, DR's SSIM), whereas T-CLIP stays within 0.004: it compares semantic content of adjacent frames, and a blurred frame is still semantically the same as its neighbour. Figure R-fusion (a) further illustrates the remaining drift in color, shadows, and scene details across fused chunks, alongside the corresponding input frames.

Table X-a. All methods without fusion, over all frames (hard concat of independent windows)

MethodPSNR↑SSIM↑LPIPS↓T-CLIP↑Warp-SSIM↑
Light-X13.530.51540.40440.98520.9495
UniRelight21.350.57790.27750.98350.9568
DiffusionRenderer20.950.72350.27570.98430.9565
PCRP-Video20.400.57890.31300.98210.9480
PBR19.590.71110.31060.97550.9223
LightCrafter23.530.73050.21130.98760.9570

Table X-b. All methods with the overlap fusion method, over all frames (PBR: not applicable)

MethodPSNR↑SSIM↑LPIPS↓T-CLIP↑Warp-SSIM↑
Light-X13.540.51570.39160.98530.9448
UniRelight20.980.58490.29200.98710.9575
DiffusionRenderer21.720.72450.28790.98360.9522
PCRP-Video20.780.58520.30990.98380.9507
LightCrafter24.160.73660.20960.98920.9598

Table Y-a. All methods without fusion, over boundary frames / Table Y-b. All methods with fusion, over boundary frames

MethodY-a Warp-SSIM↑Y-a T-CLIP↑Y-b Warp-SSIM↑Y-b T-CLIP↑
Light-X0.8580.9510.93060.9705
UniRelight0.7990.9440.92560.9659
DiffusionRenderer0.7090.9360.92610.9538
PBR proxy (refiner input)0.8920.954––
LightCrafter0.9010.9710.96800.9891

R-seam (a): Boundary consistency without fusion on synthetic videos. Refer to revision figure R-seam (a) on page 30.

R-seam (b): Boundary consistency without fusion on real-world videos. Refer to revision figure R-seam (b) on page 30.

R-fusion (a): Cross-chunk consistency with fusion on real-world videos. Refer to revision figure R-fusion (a) on page 30.

R-fusion (b): Overlap-region artifacts with fusion on synthetic videos. Refer to revision figure R-fusion (b) on page 31.

Questions

Q1. How is the target environment map obtained for MIT: from the dataset's light probes or by fitting Eq. 1 to the target image? Do all methods receive the same one? A: See W2.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. We have posted point-by-point text responses under each review. For ease of reference, we have also compiled the reviewers’ questions and our responses, together with supporting figures and tables, in the appendix of the revised PDF, starting on page 24. We hope this consolidated presentation makes our responses and additional results easier to review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.