acceptodds
Under review as a conference paper at ICLR 2027

LightShift: Controllable Indoor Relighting via Video Diffusion Priors

Abstract

Controllable image-based indoor relighting aims to render a source image under user-specified illumination conditions while preserving intrinsic scene properties and structural consistency. Previous approaches typically leverage pretrained im- age diffusion models with auxiliary conditioning branches, such as ControlNet, to incorporate estimated geometry (e.g., depth and normals), intrinsic properties (e.g., albedo), and lighting instructions. However, errors in these estimates can propagate to the relit outputs, degrading visual quality and scene consistency. We present LightShift, a controllable indoor relighting framework that leverages pre- trained video diffusion priors without requiring geometric or intrinsic estimates at inference time. Our key idea is in-sequence visual conditioning: we repre- sent the source image, a scribble-based lighting instruction, and the relit output as a unified three-frame sequence. Rather than injecting the lighting instruction through an auxiliary conditioning branch, this formulation uses temporal interac- tions to integrate scene context and spatial lighting control when generating the relit frame. To support training, we introduce a two-stage data synthesis pipeline that constructs relighting pairs from in-the-wild indoor photographs. The first stage generates coarse relighting pairs through intrinsic-based re-rendering, while the second uses the first-stage model to synthesize relit source images paired with real photographic targets. Quantitative and qualitative evaluations, together with a user study, demonstrate the effectiveness of LightShift in controllable indoor relighting, with improved lighting control and scene consistency.

Primary area:
generative models
Keywords:
Indoor Relighting, Controllable Image Editing, Video Diffusion Models, Visual Prompting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bPE6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LightShift, a controllable indoor relighting framework that repurposes a pretrained video diffusion model (Stable Video Diffusion) without requiring test-time geometric or intrinsic estimation. The authors formulate relighting as in-sequence visual conditioning, arranging the source image, a scribble-based lighting instruction, and the relit output as a three-frame sequence, supported by a two-stage synthetic data generation pipeline. The method is validated through benchmark relighting comparisons, user studies, and ablation experiments analyzing video pretraining and conditioning mechanisms.

Critical strengths
  1. Intrinsic-Free Relighting via Video Priors: Repurposing a pretrained video diffusion model eliminates test-time intrinsic or geometry estimation. Directly synthesizing the relit frame avoids cascading decomposition errors typical of inverse rendering pipelines.
  2. Validation of In-Sequence Conditioning: Controlled ablations confirm the necessity of full spatio-temporal video pretraining over spatial-only initialization. Furthermore, in-sequence prompt conditioning substantially outperforms an auxiliary ControlNet baseline.
Critical weaknesses
  1. Coarse Control Space Without Fine-Grained Attributes: Control is restricted to ternary brighten and darken scribbles ({0, 0.5, 1}), without support for light color, intensity. LightShift cannot perform precise color temperature or intensity adjustments, limiting practical utility.
  2. Oracle Benchmark and Undocumented Resolution Locking: Table 1 evaluates only on dense scribble masks derived from ground-truth target differences, acting as an oracle signal rather than realistic user strokes. The paper would be strengthened by a systematic quantitative/qualitative evaluation on sparse, freehand user scribbles or semantically segmented light fixture masks.
  3. Weak Baselines and Protocol Mismatches: The baseline comparisons do not conclusively demonstrate LightShift's superiority. IC-Light is not designed for localized scribble guidance and achieves competitive PSNR primarily by making conservative, minimal illumination edits. Furthermore, the evaluation omits contemporary multimodal image editing models (e.g., Flux, Qwen-Image, Ideogram) that can execute lighting instructions zero-shot, missing a crucial test against modern editing frameworks. Additional qualitative comparisons should be possible, as already shown in related work like LuxRemix and TokenLight.
  4. Omitted Limitations and Missing Literature: The paper lacks a dedicated limitations section discussing failure modes or the potential drift and texture degradation under the iterative relighting shown in Figure 1. In addition, the related work omits foundational frameworks that repurpose video diffusion models for image editing, notably UniReal, as well as 3D-aware generative relighting methods like GR3EN.
Questions for authors

Some complex shading changes, like casting shadows from a window onto a wall or bed as shown in Fig 3 and 6, are predominantly achieved by the provided visual prompts. In the real use case, how should users provide the visual prompts for such complex shading changes? Is there an auxiliary tool to help users provide the visual prompts for such complex shading changes?

Reviewer_sKSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LightShift, a framework for controllable indoor relighting from a single image using scribble-based lighting instructions. The authors claim two main contributions. First, they introduce in-sequence visual conditioning: the source image, the scribble prompt, and the relit output are arranged as a three-frame sequence in a pretrained video diffusion model, so that relighting becomes next-frame prediction without auxiliary conditioning branches or geometric/intrinsic estimates at inference time. Second, they propose a two-stage data synthesis pipeline that builds training pairs from in-the-wild indoor photographs, combining coarse re-rendered pairs (Stage I) with model-generated source images paired with real photographs as targets (Stage II). They also claim improved lighting control and scene consistency over existing relighting methods.

Critical strengths

S1. The in-sequence conditioning design is supported by a controlled ablation. On the same pretrained video backbone, it outperforms cross-attention, latent concatenation, and ControlNet-style injection, which shows the conditioning strategy itself matters.

S2. The framework is simple and practical. It needs only a source image and a scribble at inference, with no intrinsic or geometry estimators. It is trained on modest compute, and the appendix documents the implementation details and evaluation protocol.

Critical weaknesses

W1. The novelty of the core idea is limited. Transferring video generative priors to other tasks has already been shown by prior work the paper cites (RealGeneral, ObjectMover), and the scribble interface and training data construction follow ScribbleLight.

W2. The evaluation does not directly measure the central claims (lighting controllability and scene consistency). Since the scribbles are derived from the target image, PSNR/SSIM/LPIPS largely reflect reconstruction accuracy, and there is no comparison against an identity baseline that simply outputs the input.

W3. The source of the performance gain is not sufficiently isolated. The key motivation, avoiding error propagation from intrinsic estimates, is not supported by any analysis, and there is no direct comparison with a variant conditioned on intrinsic maps using the same video backbone.

W4. The benefit of the two-stage data synthesis is not clearly established. Stage I alone already comes within 0.17 PSNR of the full model, and it is unclear whether the single-stage variants and the combined model are trained at the same data scale.

Questions for authors

Q1 (W2). Could you report results for an identity baseline that outputs the input unchanged? Could you also add a metric that directly measures control accuracy, such as the brightness change within scribbled regions?

Q2 (W3). Could you provide a variant on the same SVD backbone conditioned on intrinsic maps, and results when errors are injected into the intrinsic estimates?

Q3 (W4). Are the single-stage variants in Table 3 trained at the same data scale as the combined model? Could you provide a scale-matched comparison?

(Clarification). Are noise and the denoising loss applied to the first two frames (source and scribble), or only to the third frame?

Reviewer_u24UOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets controllable single-image indoor relighting and argues that existing diffusion approaches (ScribbleLight, LumiNet, RGB↔X) are limited by their dependence on estimated geometry/intrinsics at inference, whose errors propagate into the relit output. LightShift instead fine-tunes Stable Video Diffusion and introduces in-sequence visual conditioning: the source image, a ternary scribble prompt {0, 0.5, 1}, and the relit result are laid out as three consecutive frames X = (x_s, x_v, x_l), with the relit frame predicted as a next-frame task through the pretrained spatio-temporal U-Net — no auxiliary conditioning branch and no intrinsic estimates at inference time. Cross-attention image-embedding injection is disabled; a zero tensor is concatenated with the target latent for dimensional parity.

Training uses a two-stage synthesis pipeline on LSUN-Bedroom (50K + 50K triplets): Stage I re-renders coarse relighting pairs from estimated albedo/shading (Careaga & Aksoy, 2024) and normals (DSINE) under sampled point lights, with scribbles thresholded from the target shading map; Stage II uses the Stage I model to relight real photographs under an inverse scribble, taking the generated image as the source and the original photograph as the target. Evaluation is on 1,584 BigTime pairs from 132 indoor sequences across six brightness transitions, reporting PSNR/SSIM/LPIPS plus a 30-participant 2AFC user study, with ablations on video pretraining, scribble injection, source reconstruction, and the two data stages.

Critical strengths

The in-sequence conditioning comparison is a clean, properly controlled result. Table 2(d)–(g) holds the backbone, data, and training budget fixed and varies only how the scribble enters: CLIP cross-attention (13.97 / 0.61 / 0.31), scribble–noise channel concatenation (13.90 / 0.56 / 0.37), a ControlNet-style auxiliary branch (13.50 / 0.60 / 0.32), versus in-sequence (17.40 / 0.67 / 0.27). This is the one comparison in the paper where the contrast isolates the claimed design choice, and the margin is large. Table 3(a) adds a sensible companion result — removing source-image reconstruction while keeping the source latent as a concatenated condition costs 2.10 dB — which supports the specific claim that denoising the source along the same trajectory matters, not merely that the source is available. Combined with the removal of inference-time intrinsic estimation, this is a contribution I would like to see properly evaluated.

Critical weaknesses
  1. The conditioning signal is computed from the ground-truth target, there is no trivial baseline, and the two capabilities the paper advertises are never measured quantitatively. Appendix A specifies the evaluation prompt as Δ = gray(I_tgt) − gray(I_src), thresholded at 50/255, dilated into strokes, with the sign of Δ setting brighten/darken. So the input condition for every number in Tables 1, 2, 3, and 4 is a thresholded, dilated map of the ground-truth pixel-difference sign, and the output is then scored against that same target with PSNR/SSIM/LPIPS. Giving all methods the same prompt makes the ranking internally consistent but does not remove the leak, and it systematically favors whichever architecture most directly converts the prompt's spatial support into intensity change — which is exactly what in-sequence conditioning is designed to do. Nothing calibrates what these numbers mean: BigTime pairs are time-lapse frames of a static scene, so source and target are pixel-aligned, and the paper never reports the copy-the-source identity baseline, despite SSIM of 0.67 against a pixel-aligned target being low enough that the comparison is not obviously safe. The paper's own data point in this direction: the method it describes as retaining "lighting patterns close to the input" — IC-Light — ranks second on PSNR (16.67 vs 17.40) and best on LPIPS (0.26 vs 0.27). When the least-editing method is near the top on two of three metrics, those metrics are not measuring lighting control. Meanwhile the two settings the paper actually sells — freehand user scribbles (Figs. 5, 9) and the iterative relighting in the teaser (Fig. 1, three successive applications claimed to preserve content and intrinsics) — carry no quantitative evaluation at all. Half the user study's cases are the user-scribble setting, which means a win-rate table is the only evidence for it, and there is no measurement of drift under repeated application anywhere in the paper.

  2. Every cross-method comparison is run outside the baselines' designed interface or through an unvalidated reimplementation, and the non-architectural contribution does not separate from single-run noise. Of four comparison methods, three receive inputs they were not built for, as the paper states: IC-Light gets the scribble "as an environment-lighting cue," RGB↔X "as a new lighting configuration," and LumiNet gets "our scribble map converted into a grayscale image" used as its photographic reference. The paper then diagnoses their failures in terms the adaptation predicts — RGB↔X "introduces artifacts resembling the visual prompt," LumiNet "exhibits texture artifacts resembling the scribble strokes" — and Appendix C concedes the LumiNet numbers "should be interpreted within this adapted scribble-guided setting." That caveat applies with equal force to RGB↔X's 10.79 dB / 0.44 / 0.50 in Table 1, where it is not stated, and to the user study, whose largest margins (RGB↔X at 95% / 95% on consistency and realism) come from the most heavily adapted baseline. The one baseline in its native interface, ScribbleLight, is a reimplementation scoring 12.14 dB with no check against published numbers, and it is fed albedo and normals from the older intrinsic decomposition (Careaga & Aksoy, 2023) while LightShift's own Stage I pipeline uses the newer one (Careaga & Aksoy, 2024) — an asymmetry that directly concerns the paper's central claim about intrinsic-estimation error propagation. On the method side, contribution (2) — the two-stage pipeline — rests on Stage I only 17.23 / 0.64 / 0.31 versus combined 17.40 / 0.67 / 0.27, i.e. +0.17 dB from a single run with no seeds or variance reported anywhere in the paper, with Stage II alone (16.63) worse than Stage I alone on PSNR. Stage II is also circular in a way the paper does not discuss: its training sources I_ref are Stage-I model outputs, so the final model is supervised to map its own generator's outputs back to real photographs, while at test time the source is always a real photograph. That is the same synthetic-input dependence the paper criticizes in intrinsic-conditioned methods, relocated from inference to training, and its test-time cost is never measured.

Questions for authors
  1. Identity baseline. On the same 1,584 BigTime pairs, what are the PSNR, SSIM, and LPIPS of simply copying the source image to the output? The pairs are pixel-aligned time-lapse frames, so this baseline has perfect structure and wrong illumination, and it is the natural calibration for a 17.40 / 0.67 / 0.27 result. Please also report it per transition type (Dark to Bright, Bright to Dark, and so on), since the brightness gap differs sharply across the six types.

  2. A prompt not derived from the target. Every number in Tables 1 through 4 uses a scribble computed as thresholded, dilated sign(gray(I_tgt) - gray(I_src)) at threshold 50/255. Can you report the same quantitative comparison with prompts that do not see the target - for example the freehand user scribbles already collected for the user study, scored against the targets, or prompts produced by a predictor that only sees the source? If the ranking in Table 1 changes, that would be the more informative result to report.

  3. ScribbleLight reimplementation. Does your reimplementation reproduce the numbers reported in the ScribbleLight paper on any setting they published? 12.14 dB on its own task and interface is low enough that I cannot tell whether it reflects the method or the reimplementation. Relatedly: your Stage I pipeline uses Careaga and Aksoy (2024) for intrinsics while the ScribbleLight baseline is fed Careaga and Aksoy (2023) albedo and normals - why the asymmetry, and what are the baseline's numbers with the 2024 estimator?

  4. Isolating "video priors" from capacity and layout. Table 2(a) is from scratch and Table 2(b) is spatial-only initialization, so both conflate video pretraining with pretraining in general; Table 2(c) changes the backbone, the parameter count, and the input layout at once. Can you add an image-diffusion control with matched parameter count and pretrained spatial weights, given the same three-image in-sequence layout (for example as a 1x3 tiled image), so that the only varied factor is whether the temporal layers were pretrained on video? Without it I cannot distinguish "video priors transfer to illumination change" from "more pretrained parameters, plus a conditioning layout that happens to work well."

  5. Stage II circularity and variance. Stage II supervises the model on sources that are Stage I model outputs, while test-time sources are real photographs. What is the measured gap - for instance, the final model's scores on Stage-I-generated sources versus real photographs with matched prompts? And given that the two-stage claim rests on 17.23 versus 17.40 PSNR, can you report multiple seeds or a variance estimate for that comparison?

  6. Iterative relighting. Figure 1 is the teaser claim: content and intrinsic properties are preserved across three successive applications. Is there any quantitative support - for example albedo or normal consistency, or re-measured PSNR/SSIM against the original source, after 1, 2, and 3 rounds? Self-feeding a generative model is where artifacts usually compound, so a number here would matter.

  7. User study protocol. Were presentation order and left/right position randomized, and were participants blind to which output was the proposed method? Can you report per-criterion confidence intervals and the number of judgments per participant (30 cases x 3 baselines x 3 criteria is 270, which is a lot)? Including at least one baseline-versus-baseline pair would let a reader check rater consistency and rule out a general preference for the more visibly edited image.

  8. Resolution. Training and, I assume, evaluation are at 512x256. Were all baselines evaluated at the same resolution, and how does LightShift behave at the resolutions these baselines were designed for?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.