acceptodds
Withdrawn from ICLR 2027

CoRe: Recovery-Driven Head Routing for Real-Time Streaming Video Relighting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

> Interactive video relighting lets users change the illumination of a video while it plays. Streaming generators achieve the required speed and temporal coherence by reusing cached attention keys and values (K/V), but the cache also remembers the old light: after a lighting edit, reused states leak stale shadows and colors into new frames, whereas clearing or rebuilding the cache sacrifices continuity or latency. We observe that this conflict is concentrated in a subset of attention heads, and present **CoRe** (**Co**unterfactual **Re**covery), which routes each head either to persistent video memory or to the current light. Video-memory heads keep the rolling video cache, while light-control heads attend to K/V computed from the current lighting condition and never store them across blocks. CoRe learns how many light-control heads each layer receives, and which ones, through lighting-switch recovery: starting from a cache built under light A, the model must recover the output that the same video would receive under light B, subject to a global head budget. The learned routing is fixed before deployment, so an edit requires neither cache rebuilding nor online head selection. A single interface accepts HDR environment maps, reference images, and text. On minute-scale text-conditioned streams, CoRe reduces OldRes from 0.18 to 0.07 relative to uniform cache reuse at 23.8 FPS end-to-end on a single GPU, and it achieves the lowest LF among compared methods on short fixed-light clips for all three control modalities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
interactive video relighting, streaming video diffusion, memory routing, KV caching, controllable generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QQ8FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRe, a streaming video relighting method that applies lighting edits without rebuilding the attention cache. It routes several heads to keys and values from the current lighting only, while the others keep the history cache. The selection is learned under a fixed head budget with a loss matching post-edit outputs to constant-light references, then frozen. Experiments on the LongLive streaming generator cover fixed-light clips and 45.6 to 60 s streams with one edit. On text-conditioned streams, the residual of the old lighting after an edit falls from 0.18 to 0.07 against the same backbone trained only under constant lighting, at 23.8 instead of 20 FPS.

Critical strengths
  1. The insight is straightforward: Routing each head to video memory or current lighting avoids cache rebuilds. Instead of keeping or clearing the whole cache, the learned heads read only the current lighting, and the other heads keep the video history.
  2. The experimental results show promise. Stale lighting drops sharply while throughput increases. Compared with the same backbone trained only under constant lighting, the old-light residual (OldRes) falls from 0.18 to 0.07 on text-conditioned streams, with throughput rising from 20 to 23.8 FPS (Table 2). On HDR-conditioned streams it falls from 0.19 to 0.05 (Table 11).
Critical weaknesses
  1. The mechanism behind the gain is not isolated. Direct evidence that stale lighting concentrates in a subset of heads is one causal-patching figure (Figure 13) with undefined axis metrics and no error bars or sample size. The HDR-stream ablations (Table 9) lack a variant whose selected heads lose cache access but receive no lighting keys and values, so the source of the gain is unclear.
  2. Training and evaluation details needed for reproduction are missing. The paper reports no optimizer, learning rate, iteration count, batch size, loss weights, or length of the post-edit recovery window, and no number of clips or streams per table. These gaps matter because training has three stages: constant-light pretraining, head search, and training with frozen heads.
Questions for authors

None.

Reviewer_QmZEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses stale illumination in streaming video relighting. Reusing historical KV caches preserves temporal context but also carries appearance information from previous lighting conditions. The paper learns a fixed partition of attention heads into video-memory heads, which retain historical video KV, and light-control heads, which attend exclusively to current-light KV. The partition is optimized under a global head budget using lighting-switch recovery supervision. Validation includes fixed-light relighting, interactive streams, paired multi-illumination images. The reported results indicate improved lighting-switch response and higher throughput than the evaluated cache-reuse and recaching baselines.

Critical strengths
  1. The paper identifies a concrete conflict between temporal memory and editable conditioning. Separating persistent video memory from current-light control is a well-motivated response, and the fixed routing avoids additional head-selection computation at inference.
  2. The ablations provide meaningful evidence for the proposed design. In particular, the routing-by-recovery experiment and the allocation comparisons do support contributions from both recovery supervision and learned head assignment.
Critical weaknesses
  1. The methodological novelty appears incremental relative to prior work on attention-head specialization and KV-cache management.
  2. The demonstrated applicability is limited to streaming video relighting with mid-stream illumination changes. Although practically relevant, this is a specialized setting, and the paper does not establish whether the proposed head partitioning extends to other editing conditions that may interact more strongly with content and motion. This limits the broader significance of the contribution: the current evidence supports a task-specific solution, while its value as a general mechanism for editable streaming generation remains unclear.
Questions for authors
  1. Can the main OldRes and T90 improvements be corroborated using a shared paired relighting target or controlled rendering setup, particularly for spatial shadow and highlight changes?
  2. What are the wall-clock edit-to-visible-response and edit-to-settled-output latencies, including block buffering and condition encoding, under repeated edits across the supported modalities?
  3. Is the proposed routing mechanism specific to relighting, or can it support broader streaming video editing tasks, such as changes in style, material, or object appearance? In particular, what assumptions make it possible to separate persistent content memory from editable conditions at the attention-head level, and which of these assumptions are specific to illumination?
Reviewer_8VqCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new framework for real-time interactive video relighting called CoRe. The method divides self-attention heads into video-memory heads, which use the video KV cache as usual, and light-control heads, which attend directly to the current lighting tokens. The head assignment is learned through a budget-constrained routing procedure via learnable sigmoid gates and a recovery objective based on simulated lighting changes. The experiments show faster adaptation to new lighting with less stale-light leakage, while preserving competitive video quality and efficiency.

Critical strengths
  1. The paper identifies an important failure mode in streaming video relighting: the KV cache preserves temporal information but also retains stale lighting after an edit. CoRe offers a simple solution by assigning some attention heads to video memory and others directly to the current lighting condition.

  2. The final method is practical and efficient. The learned routing is fixed at inference, requires no cache reconstruction or online optimization, and improves recovery after lighting changes while retaining competitive quality and real-time speed.

Critical weaknesses
  1. It is unclear how general the contribution is, i.e. CoRe replaces video KV with lighting KV for a learned subset of heads, but all main evidence comes from a single LongLive-based backbone and one learnt head partition. The paper does not evaluate whether the proposed routing procedure remains effective across different architectures, model scales, cache designs, or training datasets. This makes it difficult to determine whether the observed head specialization reflects a general property of streaming video models or is specific to the chosen setup.
Questions for authors
  1. How sensitive is CoRe to the streaming backbone and cache design? Do you have evidence that the routing procedure remains effective beyond the LongLive-based architecture used in the paper?
  2. How stable (or random) is the learned head partition across random seeds, training subsets, and lighting modalities?
  3. How much of the improvement comes from learning the head allocation rather than simply adding a fixed number of direct light-attention heads? Please compare against random or uniformly distributed allocations with the same budget.
  4. On top of the previous question, do we really need to learn the light heads, what about a simple heuristics-based approach? Will it be worse than the learnt one? For example, we can measure how much each head's KV cache gets affected when lighting changes (which was mentioned in the paper).
  5. What is the total training cost of the full pipeline, including pretraining, soft routing search, and hard routing fine-tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.