CoRe: Recovery-Driven Head Routing for Real-Time Streaming Video Relighting

> Interactive video relighting lets users change the illumination of a video while it plays. Streaming generators achieve the required speed and temporal coherence by reusing cached attention keys and values (K/V), but the cache also remembers the old light: after a lighting edit, reused states leak stale shadows and colors into new frames, whereas clearing or rebuilding the cache sacrifices continuity or latency. We observe that this conflict is concentrated in a subset of attention heads, and present **CoRe** (**Co**unterfactual **Re**covery), which routes each head either to persistent video memory or to the current light. Video-memory heads keep the rolling video cache, while light-control heads attend to K/V computed from the current lighting condition and never store them across blocks. CoRe learns how many light-control heads each layer receives, and which ones, through lighting-switch recovery: starting from a cache built under light A, the model must recover the output that the same video would receive under light B, subject to a global head budget. The learned routing is fixed before deployment, so an edit requires neither cache rebuilding nor online head selection. A single interface accepts HDR environment maps, reference images, and text. On minute-scale text-conditioned streams, CoRe reduces OldRes from 0.18 to 0.07 relative to uniform cache reuse at 23.8 FPS end-to-end on a single GPU, and it achieves the lowest LF among compared methods on short fixed-light clips for all three control modalities.