acceptodds
Under review as a conference paper at ICLR 2027

Streaming Video Editing with Easy Adaptation

11 views

Abstract

In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU without any auxiliary acceleration techniques.

Primary area:
generative models
Keywords:
computer vision, video editing, streaming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_55B9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SVEET, a framework for transferring video editing capabilities learned on a bidirectional diffusion backbone to a frozen autoregressive streaming backbone. The method combines a frame-wise control branch with Orthogonal Decoupled Training (ODT), which constrains control updates using a subspace estimated from bidirectional–causal feature discrepancies. Experiments cover style transfer, video inpainting, and depth-to-video generation. However, gaps in the theoretical derivation and incomplete evaluation of streaming performance limit the strength of the conclusions.

Critical strengths
  1. Coherent architectural design. Frame-wise attention makes the control pathway compatible with causal inference and avoids an additional temporal KV cache for that branch. The division between spatial control and temporal modeling is clearly motivated.
  2. Encouraging controlled experiments. The paper compares multiple streaming baselines and includes a factorial ablation of attention design and ODT. The reported results suggest that both components contribute to editing quality, with additional qualitative evidence of transfer to Self Forcing.
Critical weaknesses
  1. Streaming performance is insufficiently characterized. The reported 15 FPS does not establish low interactive latency without details on chunk size, denoising steps, first-output latency, and the inclusion of condition encoding and VAE processing. Evaluation primarily uses 81-frame clips, leaving sustained streaming behavior, chunk-boundary artifacts, long-term drift, and memory growth insufficiently tested.
  2. The scope of zero-shot transfer needs clearer qualification. ODT uses features collected from the target streaming backbone during calibration. Thus, the method avoids target-backbone parameter updates but still requires target-model access before training. Furthermore, transfer to an additional streaming model is demonstrated mainly qualitatively. The evidence supports a narrower transfer claim than the broad statements about heterogeneous backbone architectures.
  3. Evaluation of editing fidelity could be more direct. VLM scores, CLIP alignment, and general VBench metrics provide useful but incomplete evidence of task success. Inpainting would benefit from explicit preservation measurements outside the mask, while depth-to-video generation requires structural adherence metrics. The user study includes only 10 participants and 15 videos and omits important baselines such as LiveEdit. Uncertainty estimates are also absent.
Questions for authors
  1. Under what exact configuration is 15 FPS measured? Please specify resolution, chunk size, denoising steps, first-output latency, and whether timing includes condition encoding and VAE processing. Are sustained-streaming measurements available?
  2. For transfer to Self Forcing, is the ODT projector estimated using Causal Forcing retained unchanged, or is it recalibrated? Are quantitative results available for this transfer?
  3. Are task-specific preservation and structural-adherence results available, particularly outside-mask fidelity for inpainting and depth consistency for depth-to-video generation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 55B9 (1/2)

We sincerely thank Reviewer 55B9 for the constructive feedback. We appreciate the reviewer’s recognition of SVEET’s architectural design and experiments. We address the concerns point by point fidelity below.

  • (W1/Q1: Runtime Protocol and Long-Term Streaming Performance)

We thank the reviewer for highlighting the distinction between throughput and interactive latency. We have clarified the inference configuration and timing scope and added measurements for longer continuous videos.

Runtime protocol. We use a single H100 with BF16 precision, batch size 1, 832 × 480 resolution, 81 frames, 4 denoising steps, and 3 latent frames per chunk. Full-video timing includes text encoding, denoising, KV-cache updates, and output VAE decoding; for SVEET, it also includes source-video VAE encoding and control computation.

MetricCF w/o controlCF + SVEET
Full-video time ↓6.615 s8.728 s
Full-video throughput ↑12.24 FPS9.28 FPS
Throughput excluding VAE encoding and decoding ↑18.30 FPS15.03 FPS
First latent-chunk time ↓0.394 s0.591 s
Subsequent latent-chunk time ↓0.672 s0.799 s
Peak allocated memory ↓23.34 GiB25.13 GiB

The current inference path decodes the latent chunks together after generation. The first and subsequent latent-chunk times measure latent computation and exclude VAE decoding; they therefore do not represent first-visible-frame or decoded-output chunk latency. We clarify this limitation and the scope of the reported 15 FPS in Appendix F. We have also revised the corresponding descriptions in the abstract and Introduction to clarify the scope of our performance claims.

Sustained generation. We additionally evaluate 20 continuous 1,005-frame videos, corresponding to approximately 62.8 seconds at 16 FPS, using SVEET with the official Causal Forcing + Rolling Forcing long-video implementation.

MethodCSD ↑Motion Smoothness ↑Imaging Quality ↑
SVEET0.77640.97490.7670
LiveEdit0.73540.95450.7109

As quantitative metrics alone cannot fully capture stylization quality and consistency with the source video, we provide additional visual results for long-video editing in Appendix E.1.

On one 1,005-frame video, SVEET used approximately 21.99 GiB of allocated GPU memory during rolling generation on an H100. Across the 87 rolling windows, the mean and median computation times were 1.00 s and 1.04 s, respectively. Peak allocated memory for the complete run was 35.40 GiB. The backbone maintains a fixed-capacity KV cache of 24 latent frames, while the control branch requires no additional cross-block temporal KV cache.

  • (W2/Q2: Scope and Quantitative Validation of Zero-Shot Transfer)

We thank the reviewer for this important clarification. We agree that ODT requires access to the streaming backbone used for calibration, although its parameters remain frozen. We now distinguish this calibration requirement from zero-shot transfer to additional streaming backbones.

For the Self Forcing results in the original Appendix D.1, we retain the control weights trained using the Causal Forcing–estimated ODT projector, without recalibrating the projector or performing additional training on Self Forcing.

Zero-shot transfer (no target-specific recalibration or fine-tuning). To further substantiate transferability, we provide quantitative zero-shot transfer results on Self Forcing, LongLive, and Rolling Forcing using the same 120-video style-transfer test set. These Wan-based models employ distinct streaming formulations and inference mechanisms. The results are summarized below.

Streaming backboneCLIP-T ↑Subject ↑Background ↑Temporal ↑Motion ↑Aesthetic ↑Overall ↑
Causal Forcing0.22870.94470.92980.98190.98980.55500.1123
Self Forcing0.20380.92220.93100.94970.97830.53380.1202
LongLive0.20650.92200.91950.95370.97990.54850.1183
Rolling Forcing0.21210.93360.92850.94580.97600.55970.1278

More visualization results can be found in Fig.13 in Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer 55B9 (2/2)

Target-specific recalibration and control fine-tuning. We additionally evaluate target-specific adaptation, separately from zero-shot transfer. For each target, we recompute the ODT projector from paired features of the frozen bidirectional backbone and the target backbone under its native causal attention. Starting from the CF-trained control weights, we then fine-tune the control LoRA using the target-specific projector, while keeping the backbone parameters frozen.

Streaming backboneCLIP-T ↑Subject ↑Background ↑Temporal ↑Motion ↑Aesthetic ↑Overall ↑
Self Forcing (w/o reca)0.20380.92220.93100.94970.97830.53380.1202
Self Forcing (w/ reca)0.2102 (+0.0064)0.9384 (+0.0162)0.9341 (+0.0031)0.9561 (+0.0064)0.9800 (+0.0017)0.5415 (+0.0077)0.1091 (−0.0111)
LongLive (w/o reca)0.20650.92200.91950.95370.97990.54850.1183
LongLive (w/ reca)0.2138 (+0.0073)0.9365 (+0.0145)0.9282 (+0.0087)0.9592 (+0.0055)0.9808 (+0.0009)0.5586 (+0.0101)0.1131 (−0.0052)
Rolling Forcing (w/o reca)0.21210.93360.92850.94580.97600.55970.1278
Rolling Forcing (w/ reca)0.2008 (−0.0113)0.9448 (+0.0112)0.9324 (+0.0039)0.9449 (−0.0009)0.9755 (−0.0005)0.5609 (+0.0012)0.1417 (+0.0139)

These results support effective zero-shot transfer across multiple streaming backbones, with target-specific recalibration and control fine-tuning yielding improvements in most evaluated metrics.

  • (W3/Q3: Task-Specific Fidelity and Reliability of the Evaluation)

Thanks for these suggestions. Here we provide more results on task-specific fidelity and user study evaluation.

Task-Specific Fidelity. We additionally evaluate outside-mask preservation for inpainting and depth consistency for depth-to-video generation.

For inpainting, we measure MAE and PSNR between generated and source frames outside the mask. For depth-to-video, we estimate the depth of generated frames and compare it with the input depth condition using affine-aligned RMSE and Spearman rank correlation. Results are averaged over the inpainting test set and depth test set respectively. We find that SVEET achieves the best results on these task-specific metrics.

MethodOutside-mask MAE ↓Outside-mask PSNR ↑Depth RMSE ↓Depth Spearman ↑
SDEdit0.121815.410.14930.7097
SDV2V0.098517.350.11230.8505
DayDream + CF0.053723.070.10200.9091
Channel Concat0.087519.570.09030.9198
LiveEdit0.079919.610.08630.9378
Full attention0.056622.610.09480.9223
SVEET (Ours)0.039825.220.07640.9513

User Study. In terms of user study, we conducted an additional user study with 20 participants and 30 videos (10 per task), covering all seven methods compared in the paper. Unlike the original rating-based study, participants were asked to select the best output among the seven methods for each video, yielding 600 selections. SVEET achieved the highest overall preference rate (72.00%), followed by Full-Attn (12.50%) and LiveEdit (7.17%).

To quantify uncertainty, we computed a 95% confidence interval of [62.67%, 80.67%] for SVEET’s overall preference rate using 10,000 crossed bootstrap replicates, resampling participants and videos independently, with videos stratified by task.

We thank the reviewer again for the insightful comments. We hope our clarifications and additional evaluations address the concerns, and we would be happy to discuss any remaining questions.

AuthorsOpenReview ↗

Response Summary for Reviewer 55B9

Thank you for your constructive feedback. We summarize the main additions and clarifications in our responses below:

  1. Runtime and sustained streaming. We clarified the complete inference configuration and distinguished full-pipeline throughput from the reported throughput excluding VAE processing. We also added evaluation on twenty 1,005-frame videos, with comparisons against LiveEdit and additional measurements of memory usage, rolling-window computation time, and cache management.

  2. Scope of zero-shot transfer. We explicitly distinguish access to the calibration backbone from transfer to additional backbones. Quantitative results on Self Forcing, LongLive, and Rolling Forcing use unchanged CF-trained control weights without target-specific recalibration or fine-tuning. Results with target-specific recalibration and control fine-tuning are reported separately.

  3. Task-specific editing fidelity. We added outside-mask MAE and PSNR for inpainting, and affine-aligned depth RMSE and Spearman correlation for depth-to-video generation. SVEET achieves the best results among the compared methods on all four task-specific metrics.

  4. Expanded human evaluation and uncertainty. We conducted an additional preference study with 20 participants and 30 videos covering all seven compared methods, including LiveEdit. SVEET receives 72.00% of the 600 selections, with a 95% crossed-bootstrap confidence interval of [62.67%, 80.67%].

We would appreciate your feedback on these additions and any remaining questions or concerns. Thank you again for your time and consideration.

Reviewer_c6X2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SVEET, a framework that transfers video editing capabilities learned on a frozen bidirectional diffusion backbone to pretrained streaming backbones without additional backbone training. The method combines a temporally independent control branch with Orthogonal Decoupled Training (ODT) to mitigate the feature mismatch between bidirectional and causal models. Experiments on style transfer, video inpainting, and depth-to-video generation report improved editing quality and an inference throughput of 15 FPS on a single H100 GPU.

Critical strengths
  • Practical formulation for transferring editing capabilities to streaming models: The paper addresses a relevant problem: enabling controllable streaming generation while reusing existing bidirectional editing models and pretrained causal backbones. The frame-wise control branch provides a straightforward mechanism for causal-compatible conditioning without introducing an additional temporal KV cache, while avoiding task-specific retraining or distillation of the streaming backbone.

  • Useful experimental coverage and controlled ablations: The evaluation covers three distinct editing tasks, with the proposed method achieving the highest reported VLM editing scores across all three. The factorial ablation in Table 2 separately examines 2D versus 3D attention and the presence versus absence of ODT. Both components improve the reported editing and motion-smoothness scores, providing evidence for their complementary benefits.

Critical weaknesses
  • Overly restrictive temporal conditioning with insufficient justification: The proposed conditional frame independence principle is more restrictive than causal streaming requires: past-only or bounded-history attention would also avoid dependence on unavailable future inputs. By removing cross-frame attention from the control branch, SVEET limits explicit learning of temporal source correspondences and delegates temporal integration to the frozen autoregressive backbone. While this design reduces cache overhead, its implications for edits requiring temporal source reasoning remain unclear. The comparison between full bidirectional attention and frame-wise attention does not establish whether the benefits arise from eliminating future-frame dependencies or from eliminating temporal interactions altogether.

  • Insufficient evaluation of real-time performance and sustained streaming behavior: Although the paper reports 15 FPS on a single H100 GPU, it does not provide a timing protocol, runtime comparisons, or key inference settings such as chunk size and denoising steps. It is therefore unclear which processing stages are included in this measurement, and throughput alone does not establish low input-to-output latency. Furthermore, Appendix C.2 states that all videos are processed as 81-frame clips, without quantitative evaluation on substantially longer continuous streams. The current results therefore provide limited evidence about temporal drift, artifacts at chunk boundaries, and memory usage during extended streaming operation.

Questions for authors
  • Necessity of frame-independent conditioning: Why is complete frame independence preferable to causal or bounded-history temporal attention in the control branch? Could the authors compare these alternatives using the same frozen backbones, ODT, and training protocol, and report both editing quality and computational overhead? This would help distinguish the benefit of removing future-frame dependencies from the consequences of removing temporal source interactions, particularly in cases involving occlusion or temporally ambiguous input conditions.

  • End-to-end latency and comparability of runtime measurements: Under what exact inference settings is the reported 15 FPS achieved? Please specify the resolution, chunk size, denoising steps, and whether the timing includes source encoding, control-branch computation, and VAE decoding. It would also be helpful to confirm whether the runtime measurement uses the same configuration as the quality evaluation. Could the authors report first-output and subsequent chunk latency, together with runtime comparisons against representative baselines on the same hardware and at the same resolution?

  • Temporal stability and resource usage during extended streaming: Could the authors evaluate substantially longer continuous streams beyond the 81-frame clips described in Appendix C.2? In particular, how do editing faithfulness, source preservation, and temporal consistency change as generation proceeds through successive chunks? Please examine artifacts at chunk boundaries and report how memory usage and latency evolve with stream length. Although the frame-wise control branch avoids an additional temporal KV cache, clarification of how the streaming backbone’s existing cache is managed would help establish the scalability of the complete pipeline.

  • Consistency of the reported results: Could the authors verify the entries in Table 1(c), where DayDream+CF and Channel Concat have identical CLIP scores and identical values for all six VBench metrics despite different VLM scores? Please clarify whether these entries are correct or reflect a reporting error.

  • Transparency and reliability of the VLM evaluation protocol: Could the authors provide the complete GPT-4o evaluation prompt and clarify what information the evaluator receives, including the editing instruction, source video, conditioning inputs, and generated output? Please specify the number and sampling strategy of the evaluated frames, whether their temporal order is preserved, and how scores are aggregated. Since VLM scores are central to the editing-quality claims, reporting scoring variability or checking whether the method rankings remain stable across repeated evaluations would strengthen confidence in the conclusions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer c6X2 (1/2)

We sincerely thank Reviewer c6X2 for the thoughtful and detailed feedback. We appreciate the reviewer’s recognition of the practical formulation and controlled experiments. Below, we address the concerns point by point.

  • (W1/Q1: Frame-independent conditioning)

We thank the reviewer for this insightful suggestion. We compare four control-branch attention strategies using the same frozen CF backbone, ODT and training protocol: frame-independent, full bidirectional, past-only causal, and bounded-history attention. Their style-transfer results on the test set are shown below.

Attention strategyCLIP ↑Subject ↑Background ↑Flickering ↑Motion ↑Aesthetic ↑Overall ↑
Frame-independent + ODT0.22870.94470.92980.98190.98980.55500.1123
Full attention + ODT0.21250.93240.91850.95960.98560.54020.1048
Past-only causal + ODT0.21250.92950.93110.94520.97710.54160.1056
Bounded history + ODT0.21290.93070.93140.94570.97640.54650.1062

These results show that frame-independent conditioning performs best on six of seven metrics. While causal and bounded-history attention also satisfy causal constraints, our findings support frame independence as an effective design choice for this transfer setting.

We also measure computational overhead on a single H100 under the same CF inference configuration, using 10 videos with 3 runs per video:

Attention strategyControl time/chunk ↓Peak allocated memory ↓
Frame-independent0.138 s25.13 GiB
Full attention0.157 s25.13 GiB
Past-only causal0.294 s27.94 GiB
Bounded history0.173 s27.94 GiB

During inference, frame-independent attention processes each condition frame independently and requires no additional temporal control KV cache. Past-only attention retains preceding control features, while bounded-history attention restricts visibility to the current and two preceding latent frames. The full attention in this runtime test is bidirectional within each three-latent-frame chunk. Under this configuration, frame-independent attention has the lowest control computation time and shares the lowest peak allocated memory.

  • (W2/Q2: Runtime protocol and end-to-end latency)

We thank the reviewer for highlighting the distinction between throughput and interactive latency. We report additional measurements on a single H100 using BF16, batch size 1, 832 × 480 resolution, 81 frames, 4 denoising steps, and 3 latent frames per chunk.

MetricCF w/o controlCF + SVEET
Full-video time ↓6.615 s8.728 s
Full-video throughput ↑12.24 FPS9.28 FPS
Throughput excluding VAE encoding and decoding ↑18.30 FPS15.03 FPS
First latent-chunk time ↓0.394 s0.591 s
Subsequent latent-chunk time ↓0.672 s0.799 s
Peak allocated memory ↓23.34 GiB25.13 GiB

Full-video timing includes text encoding, denoising, KV-cache updates, and output VAE decoding. For SVEET, it also includes source-video VAE encoding and control computation. The CF full-video result is averaged over 10 videos with 3 runs per video. As a representative editing baseline, LiveEdit has approximately 7.6% lower full-video runtime, but produces substantially less satisfactory editing results, as shown in Section 5. Simultaneously, in the current CF inference path, latent chunks are decoded together after generation. Thus, the latent-chunk timings do not represent first-visible-frame or decoded-output chunk latency. The runtime measurements use the same inference configuration as the corresponding quality evaluation. We distinguish these quantities and clarify the timing scope of the reported 15 FPS in Appendix F. We have also revised the corresponding descriptions in the abstract and Introduction to clarify the scope of our performance claims.

AuthorsOpenReview ↗

Response to Reviewer c6X2 (2/2)

  • (W2/Q3: Long-Stream Stability and Cache Management)

Thanks for this suggestion. We evaluate the long-stream stability on 20 continuous videos of 1,005 frames (62.8 s at 16 FPS), using the official Causal Forcing + Rolling Forcing long-video implementation. We report three complementary metrics: CSD measures the cosine similarity between the style features of each subsequent output frame and the first output frame; Motion Smoothness and Imaging Quality follow VBench.

MethodCSD ↑Motion Smoothness ↑Imaging Quality ↑
SVEET0.77640.97490.7670
LiveEdit0.73540.95450.7109

To examine changes over time, we aggregate CSD in five-second windows, always using the first frame of the entire output as the reference:

TimeSVEET CSD ↑LiveEdit CSD ↑
0–5 s0.91380.9116
10–15 s0.80610.7353
30–35 s0.77760.7117
55–60 s0.74310.6952

We can observe that SVEET achieves consistently higher style consistency, motion smoothness and imaging quality compared with LiveEdit. CSD decreases for both methods. However, CSD is also sensitive to scene-content changes. As quantitative metrics alone cannot fully capture stylization quality and consistency with the source video, we provide additional visual results for long-video editing in Appendix E.1.

Editing faithfulness and source preservation. We measure source–output correspondence using DINO ViT-B/16 cosine similarity, sampling five frame pairs per window. Similarity decreases modestly from 0.7207 at 0–5 s to 0.6460 at 60–62.8 s. Since this metric is sensitive to editing-induced appearance changes, the decrease does not necessarily indicate content drift. We refer the reviewer to the visualizations in Appendix E.1 for a complementary assessment of editing faithfulness and source preservation throughout the extended rollout.

Temporal consistency and chunk boundaries. Source-flow-warped RGB MAE is 0.0476 at nominal chunk boundaries and 0.0440 within chunks. After subtracting the corresponding boundary–interior difference in the source, the adjusted gap is −0.0036. Thus, this diagnostic detects no excess boundary error beyond that attributable to source motion, supporting consistent transitions across successive chunks.

Memory, latency, and cache management. The streaming backbone maintains a fixed-capacity rolling KV cache of 24 latent frames, including three anchor frames. The control branch processes three latent frames locally without an additional cross-block KV cache. On one instrumented 1,005-frame run on a single H100, allocated GPU memory during rolling generation remained approximately 21.99 GiB. Across the 87 rolling windows, the mean and median computation times were 1.00 s and 1.04 s, respectively. The complete run reached a peak allocated memory of 35.40 GiB. These timings measure rolling-window computation rather than decoded-output latency.

  • (Q4: Verification of Table 1c)

We thank the reviewer for pointing this out. After rerunning the CLIP and VBench evaluations for the entire Table 1c, we found that the DayDream+CF outputs were mistakenly used to compute the CLIP and VBench scores for Channel Concat, resulting in identical entries for the two methods. We apologize for this error and have corrected the Channel Concat results in the revised paper. We also verified the remaining CLIP and VBench entries in the table and found no other errors.

  • (Q5: VLM evaluation protocol and reliability )

We thank the reviewer for this suggestion. We clarify the evaluation protocol below.

Each evaluation receives the editing instruction, five generated frames, and five corresponding reference or conditioning frames. For style transfer, the reference is the source video; for inpainting, it is the masked source video; and for depth-to-video generation, it is the depth condition. For each 81-frame clip, we uniformly sample frames 0, 20, 40, 60, and 80. Reference or conditioning frames and generated frames are sampled at the same indices and presented in chronological order.

GPT-4o [gpt-4o-2024-08-06] assigns two separate integer scores from 1 to 10: (i) instruction faithfulness, assessing whether the edit and relevant conditions are followed, and (ii) visual coherence, assessing structural integrity and consistency within and across the sampled frames. We average the two scores for each video, then report the mean across the test set. Each sample was evaluated three times, and the reported scores were obtained by averaging the three evaluations. The method rankings remained unchanged across the three evaluation runs.

The complete evaluation prompt is provided in Appendix G.

We thank the reviewer again for the constructive suggestions. We hope the additional experiments and clarifications address the concerns, and we welcome any further questions or discussion.

AuthorsOpenReview ↗

Response Summary for Reviewer c6X2

Thank you for your thoughtful and detailed feedback. We have provided point-by-point responses and summarize the main additions below:

  1. Frame-independent versus temporal conditioning. We compared frame-independent, full bidirectional, past-only causal, and bounded-history control attention under the same backbone, ODT, and training protocol. Frame-independent conditioning achieves the best results on six of seven reported quality metrics, with the lowest control computation time and joint-lowest peak memory under the tested configuration.

  2. Runtime protocol and timing scope. We specified the hardware, resolution, chunk size, denoising steps, and included processing stages. We now distinguish full-pipeline throughput (9.28 FPS) from throughput excluding VAE processing (15.03 FPS), and latent-chunk computation from decoded-output latency.

  3. Extended streaming evaluation. We evaluated twenty 1,005-frame videos and added analyses of style consistency, source–output correspondence, and chunk-boundary errors. We also documented the fixed-capacity rolling KV cache and reported memory usage and rolling-window computation times from an instrumented long-video run.

  4. Evaluation reliability and reporting correction. We corrected the Channel Concat entries in Table 1(c) after rerunning the evaluations and verified the remaining entries. We also provided the complete GPT-4o evaluation prompt, frame-sampling and aggregation procedures, and three repeated scoring runs, across which the method rankings remained unchanged.

We would be grateful if you could review these additions and let us know whether any concerns remain. We would be happy to provide further clarification before the discussion closes.

Reviewer_bEtMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SVEET, which aims to transfer editing control trained with a bidirectional video diffusion backbone to a pretrained causal streaming backbone without retraining the streaming model. It uses temporally independent source-video encoding and Orthogonal Decoupled Training (ODT) to reduce the mismatch between editing and causal-generation features. The paper reports strong editing quality and 15 FPS inference on a single H100.

Critical strengths
  1. Transferring editing control without retraining streaming backbones could lower the cost of deploying streaming video editors.
  2. Frame-independent control encoding is compatible with causal inference constraints.
  3. ODT is an intuitive attempt to reduce interference between editing and causal dynamics.
  4. The reported visual quality and throughput make the application compelling.
Critical weaknesses
  1. Theorem 2 appears to require additional assumptions. The bound in Eq. (43) does not follow in general from right-projecting ΔW2\Delta W_2 away from the top right-singular space of ΔW1\Delta W_1. A careful examination of the multiplication order, Hessian assumptions, and projection geometry is needed.
  2. Real-time performance needs a more complete protocol. The 15 FPS figure should be accompanied by first-frame and per-chunk latency, generation settings, control-branch overhead, and an explanation of how much performance is inherited from the pretrained streaming backbone.
  3. Cross-backbone transfer evidence could be stronger. More comprehensive quantitative results across distinct causal model architectures are needed to support broad claims of heterogeneous zero-shot transfer.
Questions for authors
  1. What assumption aligns the right-singular projector of ΔW1\Delta W_1 with the spectral subspace of XPXP in Eq. (26)?
  2. Can Eq. (43) and the associated compositionality bound be corrected or restated under explicit assumptions?
  3. What are the first-frame latency, per-chunk latency, and control-specific runtime overhead?
  4. How consistent are results across different streaming backbones, video lengths, and decoding settings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bEtM (1/2)

We sincerely thank Reviewer bEtM for the constructive feedback. We appreciate the reviewer’s recognition of the potential and motivation of our work. We address the concerns point by point below.

(W1/Q1/Q2: Projection Geometry and Compositionality Bound)

We thank the reviewer for identifying the errors in the projection geometry and the associated bounds. We have corrected the projection order and revised both the theorem statements and proofs in Appendix B.

Under the row-feature convention Y=XWY=XW, we now define Pk=UkUk⊤P_k=U_kU_k^{\top}, where UkU_k contains the top-kk left singular vectors of ΔW1\Delta W_1, and project the second update as

ΔW~2=(I−Pk)ΔW2.\widetilde{\Delta W}_2=(I-P_k)\Delta W_2.

Functional interference. We agree that PkP_k, constructed from ΔW1\Delta W_1, is not necessarily aligned with the leading right-singular subspace of XPXP. The revised proof no longer assumes this alignment or replaces ∥XP(I−Pk)∥2\lVert XP(I-P_k)\rVert_2 with σk+1(XP)\sigma_{k+1}(XP). Instead, it states the general bound

∥E1∥2≤∥XP(I−Pk)∥2⋅∥ΔW2∥2.\lVert E_1\rVert_2 \leq \lVert XP(I-P_k)\rVert_2 \cdot \lVert \Delta W_2\rVert_2.

Thus, a small interference bound requires the relevant input component to be well captured by the protected subspace; approximate low rank of ΔW1\Delta W_1 alone is insufficient.

Output compositionality. With the corrected left projection, the matrix-overlap bound becomes

∥ΔW1⊤ΔW~2∥F≤σk+1(ΔW1)⋅∥ΔW~2∥F.\lVert \Delta W_{1}^{\top}\widetilde{\Delta W}_{2}\rVert_{F} \leq \sigma_{k+1}(\Delta W_{1})\cdot\lVert \widetilde{\Delta W}_{2}\rVert_{F}.

We further clarify that this overlap bound alone does not imply a nonlinear compositionality guarantee. The revised theorem establishes a local spectral-tail bound under an explicit Hessian block-diagonality assumption with respect to the projected parameter subspaces. We state this as a sufficient assumption, rather than a consequence of orthogonality.

(W2/Q3: Runtime protocol, latency, and control-branch overhead)

Thanks for highlighting this. SVEET builds on the efficient autoregressive generation of Causal Forcing (CF) and adds a control branch for video editing. We compare CF with and without SVEET on a single H100 using BF16, batch size 1, 832 × 480 resolution, 81 frames, 4 denoising steps, and 3 latent frames per chunk.

MetricCFCF + SVEET
Full-video time ↓6.615 s8.728 s
Full-video throughput ↑12.24 FPS9.28 FPS
Throughput excluding VAE encoding and decoding ↑18.30 FPS15.03 FPS
First latent-chunk time ↓0.394 s0.591 s
Subsequent latent-chunk time ↓0.672 s0.799 s
Peak allocated memory ↓23.34 GiB25.13 GiB

Full-video timing includes text encoding, denoising, KV-cache updates, and output VAE decoding; for SVEET, it also includes source-video encoding and control computation. The runtime evaluation is conducted on 10 videos, with three runs per video.

Following the current CF inference path, latent chunks are decoded together after generation. The chunk times above therefore measure latent computation rather than first-visible-frame or end-to-end chunk latency. We clarify this distinction and explicitly state the timing scope associated with the reported 15 FPS in Appendix F. We have also revised the corresponding descriptions in the abstract and Introduction to clarify the scope of our performance claims.

AuthorsOpenReview ↗

Response to Reviewer bEtM (2/2)

  • (W3/Q4: Cross-backbone transfer and robustness across video lengths and decoding settings)

We thank the reviewer for these valuable questions. Here we provide detailed results of SVEET's performance on cross-backbone transfer, different video lengths and different decoding settings.

Cross-backbone transfer. To further evaluate zero-shot transfer, we apply the CF-trained SVEET control weights to Self Forcing, LongLive, and Rolling Forcing without target-specific recalibration or fine-tuning. We evaluate all backbones on the same 120-video style-transfer test set.

Streaming backboneCLIP-T ↑Subject ↑Background ↑Temporal ↑Motion ↑Aesthetic ↑Overall ↑
Causal Forcing0.22870.94470.92980.98190.98980.55500.1123
Self Forcing0.20380.92220.93100.94970.97830.53380.1202
LongLive0.20650.92200.91950.95370.97990.54850.1183
Rolling Forcing0.21210.93360.92850.94580.97600.55970.1278

These results support the transferability of SVEET across multiple streaming backbones without target-specific training. Additional visual results are provided in Fig. 13 of Appendix E.2; results with target-specific recalibration are also reported in the Appendix E.2.

Different video lengths. We evaluate long-stream stability on 20 continuous videos of 1,005 frames each (62.8 s at 16 FPS). SVEET uses the official Causal Forcing + Rolling Forcing long-video implementation. We report three complementary metrics: CSD, the cosine similarity between the style features of each subsequent output frame and those of the first output frame; Motion Smoothness and Imaging Quality, following VBench. Scores are averaged first within each video and then across the 20 videos.

MethodCSD ↑Motion Smoothness ↑Imaging Quality ↑
SVEET0.77640.97490.7670
LiveEdit0.73540.95450.7109

To examine changes over time, we aggregate CSD over five-second windows, using the first frame of the entire output as the reference throughout:

TimeSVEET CSD ↑LiveEdit CSD ↑
0–5 s0.91380.9116
10–15 s0.80610.7353
30–35 s0.77760.7117
55–60 s0.74310.6952

We find that SVEET achieves higher style consistency, motion smoothness, and imaging quality than LiveEdit. CSD decreases for both methods; however, it can also be affected by changes in scene content and should therefore not be interpreted as a pure measure of style drift. As quantitative metrics alone cannot fully capture stylization quality and consistency with the source video, we provide additional visual results for long-video editing in Appendix E.1.

Different decoding settings. To examine different decoding settings, we additionally evaluate SVEET with CF++ framewise 2-step on the same 120-video style-transfer test set. Unlike the original chunk-wise CF configuration, CF++ generates one latent frame per block and uses four denoising steps for the first block and two steps for subsequent blocks.

ConfigurationCLIP-T ↑Subject ↑Background ↑Temporal ↑Motion ↑Aesthetic ↑Overall ↑
CF chunk-wise 4-step, original0.22870.94470.92980.98190.98980.55500.1123
CF++ framewise 2-step0.22140.92330.92140.97800.98320.53730.1096

We observe that SVEET remains effective under the substantially different CF++ framewise decoding configuration.

We thank the reviewer again for the insightful comments. We hope our revisions and additional results clarify the theoretical analysis, runtime protocol, and transferability of SVEET. We would be happy to discuss any remaining questions.

Thank you for directly acknowledging and correcting the projection errors in the original theoretical analysis. However, the corrected guarantees are narrower than the original claims, and the assumptions needed for small interference are not established empirically. The rebuttal resolves the original mathematical inconsistency in form but leaves questions about the strength and practical relevance of the revised guarantees.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Reviewer Follow-up on Submission 11822

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their careful reading and constructive feedback. We have submitted detailed responses, corrected the theoretical analysis, and conducted additional experiments to address the main concerns. For convenience, we summarize the key revisions and findings below.


1. Corrected Theoretical Analysis and Clarified Assumptions (Appendix B)

We corrected the projection geometry and revised the theorem statements and proofs. The revised analysis explicitly distinguishes the matrix-overlap bound from the local compositionality guarantee and states the conditions required for each. Our follow-up to Reviewer bEtM further discusses the practical motivation for these conditions, while distinguishing spectral evidence supporting the design from empirical verification of the theoretical assumptions.


2. Complete Runtime Protocol and Extended Streaming Evaluation (Appendices E.1 and F)

We specified the inference configuration and clarified that 15.03 FPS excludes VAE encoding and decoding, whereas full-pipeline throughput is 9.28 FPS. We also reported control overhead, memory usage, and latent-chunk computation times, explicitly distinguishing these from decoded-output latency.

To evaluate sustained generation, we added experiments on twenty 1,005-frame videos, together with analyses of source preservation, temporal consistency, chunk-boundary errors, and rolling-cache management.


3. Additional Conditioning Ablations and Quantitative Transfer Results

We compared frame-independent, full bidirectional, past-only causal, and bounded-history control attention under matched training settings. Frame-independent conditioning performs best on six of seven reported quality metrics, with the lowest control computation time under the tested configuration.

We also evaluated transfer to Self Forcing, LongLive, and Rolling Forcing on the same 120-video test set, using unchanged CF-trained control weights without target-specific recalibration or fine-tuning. We distinguish these zero-shot results from additional target-specific adaptation experiments and qualify the transfer scope to these Wan-based streaming backbones.


4. Stronger Task-Specific Evaluation and Reliability Checks

We added outside-mask preservation metrics for inpainting and depth-consistency metrics for depth-to-video generation. SVEET achieves the best results among the compared methods on all four task-specific metrics.

An expanded user study with 20 participants and 30 videos, covering all seven methods, yields a 72.00% preference rate for SVEET, with a 95% confidence interval of [62.67%, 80.67%]. We also clarified the complete VLM evaluation protocol, confirmed stable rankings across three scoring runs, and corrected the reporting error in Table 1(c).


We appreciate Reviewer bEtM’s follow-up and have provided an additional response addressing the remaining theoretical concerns. We have not yet received follow-up comments from Reviewers c6X2 and 55B9.

As the discussion period draws to a close, we would appreciate the Area Chair’s assistance in encouraging further reviewer engagement. We remain available to provide further clarification before the discussion closes.

Thank you again for your time and consideration.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.