acceptodds
Withdrawn from ICLR 2027

LumiLens: Choreographing Spatially-Varying Light and Lens for Generative Monocular Video Re-rendering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Our perception of the real world is dynamic, jointly shaped by geometry, intrinsic attributes, and lighting. Building upon this vision, we present LumiLens, the first generative framework that internalizes this intricate interplay for monocular video re-rendering that simultaneously controls viewpoint and fine-grained, parametric spatially-varying local illumination. It is driven by two key observations: (1) unlike global relighting which essentially acts as a uniform 2D filter, local relighting is a high-frequency near-field 3D rendering problem where every pixel receives a distinct light vector, demanding geometry-aware spatial reasoning; and (2) consistent with the principles of radiance transport, surface appearance is inherently defined by the coupling of 3D geometry, lighting, and viewing direction, necessitating precise camera control to synthesize reflections that faithfully respond to perspective shifts. Existing works fail to achieve such joint control or suffer from the "Baked-In Reflection", manifesting as structural stretching artifacts and original highlights that fail to respond to viewpoint changes. To bridge this gap, we introduce a Physically-Grounded Dual-Stream Conditioning Mechanism that decouples spatial illumination modeling from intrinsic scene semantics. The Spatial Stream explicitly encodes the spatial illumination distribution of light-geometry interactions via warped depth and parametric light proxies, while the Global Stream injects global lighting conditions and anchors inherent scene structure and dynamic semantics by integrating the source video. To tackle the scarcity of paired training data, we introduce a scalable Omni-View Illumination (OVI) pipeline based on Unreal Engine 5. By employing a Decoupled Light Rendering strategy, we synthesize over 134K physically disentangled video pairs. We validate the effectiveness of our approach through extensive experiments, demonstrating that LumiLens successfully disentangles illumination from geometry. This capability enables not only the precise simultaneous manipulation of lighting and viewpoint but also the flexible, independent control of each dimension across both synthetic and real-world scenarios.

Primary area:
generative models
Keywords:
video generative ai, video editing, deep learning, ai model design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_71phOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a LoRA-adapted Wan2.2-TI2V-5B video DiT that re-renders a monocular video under a new camera trajectory while switching a user-selected in-scene light emitter on/off and changing its colour/intensity (plus a global ambient scalar). The model is trained on a new 134K-clip UE5 synthetic dataset. It claims to be the first joint camera + spatially-varying local-light controller and beats adapted baselines on all three sub-tasks. The engineering is solid and the depth-vs-RGB warping insight is well evidenced, but the comparisons are stacked: the closest parametric-relighting methods are cited and not run, and no metric measures lighting-control accuracy on real video.

Critical strengths
  1. Clear, well-motivated design choice. The paper uses warped depth + occlusion mask instead of warped RGB, so source shading cannot be copied. Table 4 isolates it (16.29 → 21.76 PSNR) and Figs. 14–15 show the ghosting failure modes of RGB warping.
  2. Emitter-bound parametric control is a practical UI and handles light on/off, chromatic changes, multi-light and zero-shot OOD emitters such as monitors/windows.
  3. OVI is a well-engineered synthetic pipeline. Using path-traced stream decomposition with linear compositing (Eq. 8), sequence-level tone mapping, 12 trajectory primitives, procedural placement rules.
  4. Evaluation spans three decomposed tasks with synthetic GT metrics plus real-video proxies.
  5. The limitations of the proposed method is discussed extensively, with no photometric accuracy, no off-screen emitters, no 360° orbits, floaters from bad depth.
Critical weaknesses
  1. Closest parametric-relighting prior work is cited but never compared. RelightMaster (Bian et al., 2025) and LiveLight (Ma et al., 2026) already relight video with point lights of controllable 3D signals, multiple and moving lights, UE5 training data, and a depth-aligned light-image adapter injected into a DiT with LoRA, i.e. nearly the same lighting-conditioning recipe. The paper dismisses them in one sentence (Sec. 2.1: "does not explicitly encode fixture geometry") and excludes them even from Task 2, comparing instead against text-driven global relighting models.
  2. Architectural novelty is a recombination of three cited systems. DWMesh warping is EX-4D, frame-wise source concatenation is ReCamMaster, the compositing recipe is LightLab, the adapter is RelightMaster-like. The only delta is depth-instead-of-RGB warping plus light masks, which is kind of incremental.
  3. Synthetic benchmark is in-domain for LumiLens only. The held-out set comes from the same UE5 pipeline, asset library and renderer as training, although whether the assets used are unseen is not stated.
  4. Train/test gap in the conditioning signal is unaddressed. Training uses UE5 ground-truth depth, visibility and emitter masks. At inference time, it uses DepthCrafter + DA3 + SAM 3, while whether noise or augmentation is applied in training is not stated.
  5. The evaluation set is small, with synthetic held-out ≈ 80 clips and 100 in-the-wild clips without per-source breakdown.
Questions for authors

Besides the above weaknesses, I have the following questions for the authors:

  1. Why are RelightMaster and LiveLight not compared on Task 2? What does the emitter-mask proxy encode that a depth-aligned light image does not, given both are per-frame mask-like maps?
  2. Exactly how were color/intensity targets converted into prompts or reference images for Light-X, V-RGBX, RelightVid and IC-Light? Were outputs resized to a common resolution before computing metrics on PSNR/VBench?
  3. Please provide Task 3 results for Models B/C.
  4. User study: how many pairwise comparisons per participant, what was the "hard to judge" rate, and were participants shown the parametric target or only the baseline's prompt?
Reviewer_cZ5pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This papers proposes LumLens, the first method for fine-grained joint control of spatially-varying local illumination and viewpoint in monocular videos. In addition, the authors synthesize more than 134K video pairs with physically decoupled illumination streams with Unreal Engine. Experiments on both synthetic and real-world datasets verify the effectivness of the proposed method.

Critical strengths
  1. LumiLens is the first method to for fine-grained joint control of spatially-varying local illumination and viewpoint in monocular videos, which unifies NVS, and parametric, spatially-varying local relighting for monocular video within a single generative framework;

  2. LumiLens decouples geometry from appearance instead of directly warping the source RGB video into the spatial stream like what existing methods did.

  3. The authors build a dataset using Unreal Engine, consisting of 134K video pairs with diverse illumination variants to supervise the model.

Critical weaknesses
  1. The LoRA layers and adapters are trained exclusively on synthetic dataset generated by Unreal Engine while real-world footage presents optical complexities such as sensor noise, non-rigid cloth/hair dynamics, motion blur, and lens flare. Could you please conduct more quantitative evaluations on complex, non-ideal real footage (e.g., subsurface scattering, complex volumetric fog, or intense backlighting)?

  2. Inference relies heavily on a multi-stage prerequisite cascade. What would happen if SAM3 or another module produces false/unsatisfatory results, could the model still perform well?

  3. Please provide more inference metrics w.r.t. inference efficiency, e.g., fps, GPU memory etc. I wonder whether the proposed method can be deployed in real-world scenarios.

  4. For in-the-wild evaluation (Tab. 2), the authors employ VBench metrics. However, VBench does not measure whether the generated illumination adheres to user-specified lighting controls. Objective metrics quantifying controllability and shadow alignment in real footage are lacking.

  5. Could you please provide more Light-Off results? The manuscript repeatedly claims to "decouple" illumination from intrinsic scene properties, but Figs. 5~7 are almost ambient-lit or unlit scenes, and the demonstrated edits consist almost exclusively of turning lights on or injecting color into darkness.

Questions for authors
  1. Please provide challenging experiments on input videos that already feature intense directional illumination with hard cast shadows.

  2. Could you please use more objective metrics to quantify controllability and shadow alignment in real-world benchmarks?

  3. Please provide a comprehensive runtime breakdown of each component/key step as well as peak GPU memory consumption.

Reviewer_KzafOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LumiLens, a generative monocular video re-rendering framework that enables joint control of camera viewpoints and spatially varying local illumination. The method introduces a physically motivated dual-stream conditioning mechanism, where the Spatial Stream incorporates warped depth, occlusion masks, and parameterized illumination cues, while the Global Stream uses the source video to preserve scene semantics and appearance. The authors also construct the Omni-View Illumination (OVI) dataset using Unreal Engine 5 to provide paired training data under varying illumination and camera configurations.

Experiments on synthetic and in-the-wild videos demonstrate improvements over existing relighting and camera-controlled video generation methods.

Critical strengths

S1. Meaningful problem formulation. The paper addresses an important and challenging problem of jointly controlling local illumination and camera viewpoints in monocular video re-rendering. The motivation regarding view-dependent lighting effects and baked-in reflections is reasonable and relevant.

S2. Geometry-aware conditioning design. The proposed dual-stream architecture provides a reasonable way to incorporate geometric information for illumination control while preserving the appearance and semantics of the source video.

S3. Promising quantitative results. The method demonstrates substantial improvements in reconstruction quality and perceptual metrics across the evaluated tasks. The synthetic data generation pipeline is also a potentially valuable contribution for training models with controlled lighting and camera variations.

Critical weaknesses

W1. Insufficient evaluation of independent illumination control.

Figures 5 and 6 do not clearly demonstrate independent control over light color, intensity, and ambient illumination. Some results appear overly bright or globally illuminated. In Figure 6, even the visibility of the subject's face changes significantly across results.

It is unclear whether these differences are caused by color changes alone. I suggest varying one parameter at a time while keeping the others fixed.

W2. Limited and potentially unfair baseline comparisons.

Several baselines have different control capabilities. Light-X and RelightVid mainly use text-based illumination control. IC-Light is applied frame by frame, making its weaker temporal consistency expected.

Comparisons with additional relighting methods, such as Light-A-Video [1], LightLab [2], DiffusionRenderer [3], and GR3EN [4], would strengthen the evaluation. In particular, GR3EN employs a video diffusion model for local relighting before 3D distillation. Its intermediate video relighting results could serve as a relevant baseline. Fixed-camera comparisons would help isolate illumination-control performance.

[1] Zhou et al., "Light-A-Video: Training-free Video Relighting via Progressive Light Fusion." ICCV 2025.

[2] Magar et al., "LightLab: Controlling Light Sources in Images with Diffusion Models." SIGGRAPH 2025.

[3] Liang et al., "DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models." CVPR 2025.

[4] Xing et al., "GR3EN: Generative Relighting for 3D Environments." SIGGRAPH 2026.

W3. Insufficient evidence for the necessity of joint modeling.

The paper argues that jointly modeling camera and illumination avoids baked-in reflection artifacts. However, this advantage is not directly demonstrated.

Comparisons with two sequential pipelines would be valuable:

  • Relighting → Camera Control

  • Camera Control → Relighting

Moreover, the baked-in reflection artifacts in Figure 2 are difficult to distinguish. A clearer example would better support the motivation.

W4. Lack of direct illumination accuracy metrics.

The reported metrics mainly measure video quality, temporal consistency, geometry preservation, and camera accuracy.

For example, Chamfer Distance measures geometric preservation rather than relighting accuracy. The paper would benefit from direct evaluations of light color accuracy, shadow alignment, and spatial illumination consistency using synthetic ground truth.

W5. Limited real-world validation.

The synthetic evaluation uses only four unseen scenes. The in-the-wild evaluation lacks ground-truth illumination measurements.

Evaluation on real videos with controlled lighting would better demonstrate real-world generalization.

Questions for authors
  1. Are ambient illumination and intensity fixed across the results in Figures 5 and 6?

  2. Can the authors provide controlled experiments varying one illumination parameter at a time?

  3. How does the method compare with sequential relighting and camera-control pipelines?

  4. Can the authors provide direct quantitative evaluations of illumination accuracy?

Overall Assessment

The paper addresses a meaningful problem and shows promising results. However, the evaluation does not sufficiently support its central claims of precise local illumination control and physically plausible joint camera-light modeling.

More controlled experiments, stronger baseline comparisons, and direct illumination accuracy metrics are needed to validate these claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.