- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SVEET, a framework for transferring video editing capabilities learned on a bidirectional diffusion backbone to a frozen autoregressive streaming backbone. The method combines a frame-wise control branch with Orthogonal Decoupled Training (ODT), which constrains control updates using a subspace estimated from bidirectional–causal feature discrepancies. Experiments cover style transfer, video inpainting, and depth-to-video generation. However, gaps in the theoretical derivation and incomplete evaluation of streaming performance limit the strength of the conclusions.
Critical strengths
- Coherent architectural design. Frame-wise attention makes the control pathway compatible with causal inference and avoids an additional temporal KV cache for that branch. The division between spatial control and temporal modeling is clearly motivated.
- Encouraging controlled experiments. The paper compares multiple streaming baselines and includes a factorial ablation of attention design and ODT. The reported results suggest that both components contribute to editing quality, with additional qualitative evidence of transfer to Self Forcing.
Critical weaknesses
- Streaming performance is insufficiently characterized. The reported 15 FPS does not establish low interactive latency without details on chunk size, denoising steps, first-output latency, and the inclusion of condition encoding and VAE processing. Evaluation primarily uses 81-frame clips, leaving sustained streaming behavior, chunk-boundary artifacts, long-term drift, and memory growth insufficiently tested.
- The scope of zero-shot transfer needs clearer qualification. ODT uses features collected from the target streaming backbone during calibration. Thus, the method avoids target-backbone parameter updates but still requires target-model access before training. Furthermore, transfer to an additional streaming model is demonstrated mainly qualitatively. The evidence supports a narrower transfer claim than the broad statements about heterogeneous backbone architectures.
- Evaluation of editing fidelity could be more direct. VLM scores, CLIP alignment, and general VBench metrics provide useful but incomplete evidence of task success. Inpainting would benefit from explicit preservation measurements outside the mask, while depth-to-video generation requires structural adherence metrics. The user study includes only 10 participants and 15 videos and omits important baselines such as LiveEdit. Uncertainty estimates are also absent.
Questions for authors
- Under what exact configuration is 15 FPS measured? Please specify resolution, chunk size, denoising steps, first-output latency, and whether timing includes condition encoding and VAE processing. Are sustained-streaming measurements available?
- For transfer to Self Forcing, is the ODT projector estimated using Causal Forcing retained unchanged, or is it recalibrated? Are quantitative results available for this transfer?
- Are task-specific preservation and structural-adherence results available, particularly outside-mask fidelity for inpainting and depth consistency for depth-to-video generation?
5 comments
Response to Reviewer 55B9 (2/2)
Target-specific recalibration and control fine-tuning. We additionally evaluate target-specific adaptation, separately from zero-shot transfer. For each target, we recompute the ODT projector from paired features of the frozen bidirectional backbone and the target backbone under its native causal attention. Starting from the CF-trained control weights, we then fine-tune the control LoRA using the target-specific projector, while keeping the backbone parameters frozen.
| Streaming backbone | CLIP-T ↑ | Subject ↑ | Background ↑ | Temporal ↑ | Motion ↑ | Aesthetic ↑ | Overall ↑ |
|---|---|---|---|---|---|---|---|
| Self Forcing (w/o reca) | 0.2038 | 0.9222 | 0.9310 | 0.9497 | 0.9783 | 0.5338 | 0.1202 |
| Self Forcing (w/ reca) | 0.2102 (+0.0064) | 0.9384 (+0.0162) | 0.9341 (+0.0031) | 0.9561 (+0.0064) | 0.9800 (+0.0017) | 0.5415 (+0.0077) | 0.1091 (−0.0111) |
| LongLive (w/o reca) | 0.2065 | 0.9220 | 0.9195 | 0.9537 | 0.9799 | 0.5485 | 0.1183 |
| LongLive (w/ reca) | 0.2138 (+0.0073) | 0.9365 (+0.0145) | 0.9282 (+0.0087) | 0.9592 (+0.0055) | 0.9808 (+0.0009) | 0.5586 (+0.0101) | 0.1131 (−0.0052) |
| Rolling Forcing (w/o reca) | 0.2121 | 0.9336 | 0.9285 | 0.9458 | 0.9760 | 0.5597 | 0.1278 |
| Rolling Forcing (w/ reca) | 0.2008 (−0.0113) | 0.9448 (+0.0112) | 0.9324 (+0.0039) | 0.9449 (−0.0009) | 0.9755 (−0.0005) | 0.5609 (+0.0012) | 0.1417 (+0.0139) |
These results support effective zero-shot transfer across multiple streaming backbones, with target-specific recalibration and control fine-tuning yielding improvements in most evaluated metrics.
-
(W3/Q3: Task-Specific Fidelity and Reliability of the Evaluation)
Thanks for these suggestions. Here we provide more results on task-specific fidelity and user study evaluation.
Task-Specific Fidelity. We additionally evaluate outside-mask preservation for inpainting and depth consistency for depth-to-video generation.
For inpainting, we measure MAE and PSNR between generated and source frames outside the mask. For depth-to-video, we estimate the depth of generated frames and compare it with the input depth condition using affine-aligned RMSE and Spearman rank correlation. Results are averaged over the inpainting test set and depth test set respectively. We find that SVEET achieves the best results on these task-specific metrics.
| Method | Outside-mask MAE ↓ | Outside-mask PSNR ↑ | Depth RMSE ↓ | Depth Spearman ↑ |
|---|---|---|---|---|
| SDEdit | 0.1218 | 15.41 | 0.1493 | 0.7097 |
| SDV2V | 0.0985 | 17.35 | 0.1123 | 0.8505 |
| DayDream + CF | 0.0537 | 23.07 | 0.1020 | 0.9091 |
| Channel Concat | 0.0875 | 19.57 | 0.0903 | 0.9198 |
| LiveEdit | 0.0799 | 19.61 | 0.0863 | 0.9378 |
| Full attention | 0.0566 | 22.61 | 0.0948 | 0.9223 |
| SVEET (Ours) | 0.0398 | 25.22 | 0.0764 | 0.9513 |
User Study. In terms of user study, we conducted an additional user study with 20 participants and 30 videos (10 per task), covering all seven methods compared in the paper. Unlike the original rating-based study, participants were asked to select the best output among the seven methods for each video, yielding 600 selections. SVEET achieved the highest overall preference rate (72.00%), followed by Full-Attn (12.50%) and LiveEdit (7.17%).
To quantify uncertainty, we computed a 95% confidence interval of [62.67%, 80.67%] for SVEET’s overall preference rate using 10,000 crossed bootstrap replicates, resampling participants and videos independently, with videos stratified by task.
We thank the reviewer again for the insightful comments. We hope our clarifications and additional evaluations address the concerns, and we would be happy to discuss any remaining questions.
Response Summary for Reviewer 55B9
Thank you for your constructive feedback. We summarize the main additions and clarifications in our responses below:
-
Runtime and sustained streaming. We clarified the complete inference configuration and distinguished full-pipeline throughput from the reported throughput excluding VAE processing. We also added evaluation on twenty 1,005-frame videos, with comparisons against LiveEdit and additional measurements of memory usage, rolling-window computation time, and cache management.
-
Scope of zero-shot transfer. We explicitly distinguish access to the calibration backbone from transfer to additional backbones. Quantitative results on Self Forcing, LongLive, and Rolling Forcing use unchanged CF-trained control weights without target-specific recalibration or fine-tuning. Results with target-specific recalibration and control fine-tuning are reported separately.
-
Task-specific editing fidelity. We added outside-mask MAE and PSNR for inpainting, and affine-aligned depth RMSE and Spearman correlation for depth-to-video generation. SVEET achieves the best results among the compared methods on all four task-specific metrics.
-
Expanded human evaluation and uncertainty. We conducted an additional preference study with 20 participants and 30 videos covering all seven compared methods, including LiveEdit. SVEET receives 72.00% of the 600 selections, with a 95% crossed-bootstrap confidence interval of [62.67%, 80.67%].
We would appreciate your feedback on these additions and any remaining questions or concerns. Thank you again for your time and consideration.
Response to Reviewer 55B9 (1/2)
We sincerely thank Reviewer 55B9 for the constructive feedback. We appreciate the reviewer’s recognition of SVEET’s architectural design and experiments. We address the concerns point by point fidelity below.
We thank the reviewer for highlighting the distinction between throughput and interactive latency. We have clarified the inference configuration and timing scope and added measurements for longer continuous videos.
Runtime protocol. We use a single H100 with BF16 precision, batch size 1, 832 × 480 resolution, 81 frames, 4 denoising steps, and 3 latent frames per chunk. Full-video timing includes text encoding, denoising, KV-cache updates, and output VAE decoding; for SVEET, it also includes source-video VAE encoding and control computation.
The current inference path decodes the latent chunks together after generation. The first and subsequent latent-chunk times measure latent computation and exclude VAE decoding; they therefore do not represent first-visible-frame or decoded-output chunk latency. We clarify this limitation and the scope of the reported 15 FPS in Appendix F. We have also revised the corresponding descriptions in the abstract and Introduction to clarify the scope of our performance claims.
Sustained generation. We additionally evaluate 20 continuous 1,005-frame videos, corresponding to approximately 62.8 seconds at 16 FPS, using SVEET with the official Causal Forcing + Rolling Forcing long-video implementation.
As quantitative metrics alone cannot fully capture stylization quality and consistency with the source video, we provide additional visual results for long-video editing in Appendix E.1.
On one 1,005-frame video, SVEET used approximately 21.99 GiB of allocated GPU memory during rolling generation on an H100. Across the 87 rolling windows, the mean and median computation times were 1.00 s and 1.04 s, respectively. Peak allocated memory for the complete run was 35.40 GiB. The backbone maintains a fixed-capacity KV cache of 24 latent frames, while the control branch requires no additional cross-block temporal KV cache.
We thank the reviewer for this important clarification. We agree that ODT requires access to the streaming backbone used for calibration, although its parameters remain frozen. We now distinguish this calibration requirement from zero-shot transfer to additional streaming backbones.
For the Self Forcing results in the original Appendix D.1, we retain the control weights trained using the Causal Forcing–estimated ODT projector, without recalibrating the projector or performing additional training on Self Forcing.
Zero-shot transfer (no target-specific recalibration or fine-tuning). To further substantiate transferability, we provide quantitative zero-shot transfer results on Self Forcing, LongLive, and Rolling Forcing using the same 120-video style-transfer test set. These Wan-based models employ distinct streaming formulations and inference mechanisms. The results are summarized below.
More visualization results can be found in Fig.13 in Appendix E.2.