acceptodds
Under review as a conference paper at ICLR 2027

EditaLive! Unified Character Video Editing for Live Streaming

11 views

Abstract

Conventional video editing primarily focuses on scene-level content, whereas live streaming places greater emphasis on the human subject. However, directly applying existing video-editing methods to human-centric live streaming remains challenging, as they may introduce facial-expression inconsistencies and typically depend on multiple offline inference steps, making them unsuitable for real-time interaction. We propose EditaLive, a novel framework for real-time streaming character video editing. In detail, we start from a pretrained image animation model (Wan-Animate), which naturally decouples appearance from motion, and repurpose it as the base model for instruction-based human-centric video editing by reference frame editing and video reconstruction via the collected CharEdit-50K dataset. Besides, we adapt the model from offline bidirectional to causal streaming generation, and design an aligned self-rollout distillation strategy that compresses the model into a two-step sampler, where fixed RoPE and align forcing reduce training–inference discrepancies, and first-frame preserved sparse attention filters redundant historical information to mitigate appearance drift. Extensive experiments demonstrate that EditaLive delivers state-of-the-art editing performance with faithful preservation of facial expressions and low-latency real-time streaming inference. The code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Editing, Video Generation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZckWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EditaLive adapts Wan-Animate for instruction-guided streaming character editing. It combines reconstruction-based training on CharEdit-50K, chunk-wise causal adaptation, and two-step self-rollout distillation with Align Forcing, Fixed RoPE, and first-frame-preserved sparse attention. Short- and long-video benchmarks, component ablations, and a 40-participant user study support improvements in editing quality, motion preservation, and long-term stability.

Critical strengths
  1. Effective training formulation. Using real videos as reconstruction targets avoids introducing expression mismatches through synthesized target videos, while enabling appearance editing with explicit body and facial motion control (Sec. 3.1).

  2. Well-supported streaming adaptation. The paper includes the relevant Qwen-Image-Edit + Wan-Animate baseline and meaningful long-video ablations: removing Align Forcing reduces ID-SIM from 0.492 to 0.252 (Table 2), and the user study favors EditaLive across all evaluated criteria (Fig. 12).

Critical weaknesses
  1. The end-to-end real-time claim is not fully established. Timing starts from model-ready conditions at 384 × 672, rather than the 480 × 832 quality-evaluation resolution (App. C.3), without clearly accounting for pose extraction, face localization, or chunk buffering. The standard decoder achieves 14.47 FPS, below the benchmark's 16 FPS input rate; the 16.4 FPS Flash-VAED variant needs a complete pipeline latency measurement.
  2. Stylization training and evaluation need stronger justification. Stylization uses repeated still images as video targets (App. C.1), but the paper does not show through a targeted ablation how this supervision supports motion-preserving style transfer. Pooled ArcFace/SMIRK scores across realistic and stylized edits, together with omitted consistency scores for two baselines, also make category-specific comparisons difficult to interpret; per-category results are needed.
Questions for authors
  1. What are the throughput and capture-to-output latency at 480 × 832 when pose extraction, face localization, and input buffering are included? Can the Flash-VAED configuration sustain a live 16 FPS stream without an increasing backlog?

  2. For stylization samples, what reference image, instruction, and motion signals accompany the repeated-image target? Can you provide a stylization-training ablation and per-category results, with baseline consistency scores reported on the categories they support?

  3. Sec. 3.3 says the KV cache is constructed from the final prediction, but Algorithm 1 reuses the last denoising-pass KV features for later chunks and performs a clean-input cache forward only for the first chunk. Which caching rule is used, and is it identical during training and inference?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZckW (Part 1/3)

We thank Reviewer ZckW for the detailed and insightful comments. We address each question below.

Response to Weakness 1 (RW1) & Response to Question 1 (RQ1): End-to-end throughput, capture-to-output latency at 480 × 832, and sustaining a 16 FPS live stream.

In Table 1 and Appendix C.3, we measured model-side runtime at 384×672384 \times 672 following prior 14B-scale streaming video generation protocols. Following your suggestion, we now benchmark the complete end-to-end pipeline at 480×832480 \times 832 on H100 GPUs (with torch.compile and warm-up), covering input chunk buffering, ViTPose extraction, face localization, motion encoding, two-step DiT denoising, and VAE decoding. Below we report both unthrottled maximum throughput (Max FPS) and steady-state behavior under a continuous 16 FPS16\text{ FPS} simulated camera stream, where the service temporally subsamples buffered frames whenever processing falls behind the 16 FPS16\text{ FPS} arrival rate:

Setup (480×832480 \times 832)#GPUsWan Max FPS ↑\uparrowWan Lat. ↓\downarrowWan Subsamp. ↓\downarrowFlash Max FPS ↑\uparrowFlash Lat. ↓\downarrowFlash Subsamp. ↓\downarrow
Sequential19.353.82s31.8%11.122.89s31.9%
1 Preproc./VAE + 1 DiT214.002.52s14.6%14.522.23s13.8%
1 Preproc./VAE + 2 DiT321.771.75s0%21.951.44s0%

At 480×832480 \times 832, the 11- and 22-GPU setups peak below 16 FPS16\text{ FPS} and stay responsive under a live 16 FPS16\text{ FPS} stream by temporally subsampling incoming frames (31.8%31.8\% to 31.9%31.9\% and 13.8%13.8\% to 14.6%14.6\%, respectively). Pipelining 11 GPU for preprocessing/VAE with 22 GPUs for parallel DiT denoising (33 GPUs total) raises maximum throughput to 21.77 FPS21.77\text{ FPS} (Wan-VAE) and 21.95 FPS21.95\text{ FPS} (Flash-VAED). Under the continuous 16 FPS16\text{ FPS} stream, both 33-GPU setups process all frames at the full 16 FPS16\text{ FPS} rate with zero frame subsampling or dropping and no increasing backlog (steady-state latency slopes of −0.015-0.015 and −0.016 s/min-0.016\text{ s/min}), achieving median capture-to-output latencies of 1.75s1.75\text{s} and 1.44s1.44\text{s}. At 384×672384 \times 672 on a single GPU, even with chunk buffering and fixed-resolution frontend detectors included, Flash-VAED sustains an end-to-end output rate of 15.88 FPS15.88\text{ FPS} (1.89s1.89\text{s} latency, 1.1%1.1\% subsampled). These full-pipeline results have been added to Appendix D.3 (Table 7).

AuthorsOpenReview ↗

Response to Reviewer ZckW (Part 2/3)

RW2 & RQ2: Stylization training inputs, targeted ablation, and per-category evaluation.

Stylization training configuration. In stylization training, we take the original reference frame from the source video and the forward style instruction as conditions, repeat a stylized still image along the temporal dimension to construct a static target video, and extract the accompanying static skeleton and facial motion signals from this target video (clarified in Appendix C.1). These stylization samples are trained jointly with dynamic real-video local-edit samples, which provide supervision for time-varying motion. Since appearance and motion are learned from decoupled conditions, the static targets supervise the style transformation, which combines with dynamic motion signals from the source video at inference.

Stylization-training ablation. Following your suggestion, we conduct a full-scale Stage-1 ablation by replacing all static stylization targets in CharEdit-50K with an equal number of dynamic stylized videos from Ditto-1M under identical settings, using a source frame as reference and extracting character motion from the stylized target video (Appendix D.4, Table 8). Alongside APD and AED, we report Motion Ratio (Medit/MsrcM_{\text{edit}} / M_{\text{src}}, closer to 11 is better), the mean Farneback optical-flow magnitude of the edited video divided by that of the source video (values below 11 indicate damped motion, while values above 11 indicate extra motion or temporal jitter):

SubsetStylization Target (Stage 1)ID-SIM ↑\uparrowAED ↓\downarrowAPD ↓\downarrowTA ↑\uparrowEQ ↑\uparrowBC ↑\uparrowSR ↑\uparrowMR (→1\to 1)
Style (46)Ditto-1M video0.2680.5860.1521.9202.3912.3480.2831.338
Style (46)Static target (Ours)0.2470.6430.1582.3772.7172.7610.5871.103
Local (104)Ditto-1M video0.6940.3970.0982.1512.6122.6630.6541.123
Local (104)Static target (Ours)0.6970.4050.1002.3212.5902.7560.7401.104
All (150)Ditto-1M video0.5630.4550.1152.0802.5442.5670.5401.189
All (150)Static target (Ours)0.5590.4780.1182.3382.6292.7580.6931.104

Our static-target Stage-1 model achieves better TA, EQ, BC, and SR on both Style and All, more than doubling Style SR (0.5870.587 vs. 0.2830.283). While the Ditto-1M variant obtains slightly better ID-SIM and AED/APD on Style, this comes with substantially lower TA and SR. Regarding motion, the static-target model shows no reduction in average estimated motion magnitude on style edits, attaining virtually the same Motion Ratio on Style (1.1031.103) and real-video-supervised Local edits (1.1041.104), with a much smaller mean absolute deviation ∣1−Medit/Msrc∣|1 - M_{\text{edit}} / M_{\text{src}}| on Style than the Ditto-1M variant (0.1340.134 vs. 0.3380.338). By contrast, the Ditto-1M variant's ratio exceeds 11 in all 4646 Style cases (1.081.08 to 2.062.06), indicating consistently higher estimated motion magnitude, potentially including temporal jitter.

AuthorsOpenReview ↗

Response to Reviewer ZckW (Part 3/3)

RW2 & RQ2: Stylization training inputs, targeted ablation, and per-category evaluation (continued).

Per-category results and baseline consistency scores. Below we report the per-category breakdown for our full model and all baselines on CharEdit-Bench-S (added in Appendix D.2, Table 5, along with CharEdit-Bench-L in Table 6). Bold and underline‾\underline{\text{underline}} denote the best and second-best results within each category (with ties sharing the same rank). Consistent with Table 1, LiveEdit* and LucyEdit* do not support global style editing, so their character-consistency metrics on Style are shown in gray and excluded from ranking:

CategoryMethodID-SIM ↑\uparrowAED ↓\downarrowAPD ↓\downarrowTA ↑\uparrowEQ ↑\uparrowBC ↑\uparrowSR ↑\uparrow
Add (41)LiveEdit0.3620.5440.075‾\underline{0.075}2.6342.000‾\underline{2.000}1.4390.220
SANA-Stream.0.3820.5630.3492.5771.8051.4230.122
Stream.V20.0620.8520.3682.3372.0170.6920.000
LucyEdit0.731‾\underline{0.731}0.3770.0572.6021.9272.4150.707
UniVideo0.7480.4830.3082.6991.9272.0000.537
Ditto0.3800.6610.3432.1631.8371.3820.098
Qwen.+Wan.0.5970.4670.1002.907‾\underline{2.907}2.000‾\underline{2.000}2.1140.805‾\underline{0.805}
Ours0.6570.454‾\underline{0.454}0.0982.9112.000‾\underline{2.000}2.195‾\underline{2.195}0.878
Remove (22)LiveEdit0.4800.4140.085‾\underline{0.085}0.1521.8331.7270.000
SANA-Stream.0.5980.4320.2401.2421.5761.8030.136
Stream.V20.0810.7740.4660.1061.5150.2420.000
LucyEdit0.8310.2620.0510.5301.8642.4700.045
UniVideo0.824‾\underline{0.824}0.4090.3562.6522.197‾\underline{2.197}1.9090.500‾\underline{0.500}
Ditto0.6460.385‾\underline{0.385}0.9220.6061.5761.3030.000
Qwen.+Wan.0.6630.4220.1192.8952.197‾\underline{2.197}2.0300.864
Ours0.7010.4110.1062.864‾\underline{2.864}2.3482.121‾\underline{2.121}0.864
Change (41)LiveEdit0.3980.4620.073‾\underline{0.073}2.1632.0411.4960.195
SANA-Stream.0.5570.4580.2601.9591.7241.6260.195
Stream.V20.0510.8320.3991.5931.9430.4960.000
LucyEdit0.769‾\underline{0.769}0.3390.0621.5852.0732.3330.317
UniVideo0.8360.374‾\underline{0.374}0.2812.6182.0492.2200.732
Ditto0.4360.6190.2451.2761.6991.3170.049
Qwen.+Wan.0.6290.4310.1002.959‾\underline{2.959}2.187‾\underline{2.187}2.0980.829‾\underline{0.829}
Ours0.6700.3930.0932.9762.4552.236‾\underline{2.236}0.878
Style (46)LiveEdit*0.354\color{gray}{0.354}0.496\color{gray}{0.496}0.120\color{gray}{0.120}0.4421.9491.6160.000
SANA-Stream.0.1120.9350.5862.3261.6521.5720.174
Stream.V20.0551.0220.6742.0582.0220.9490.022
LucyEdit*0.335\color{gray}{0.335}0.761\color{gray}{0.761}0.234\color{gray}{0.234}0.8841.2681.5430.000
UniVideo0.1251.0130.7612.6452.0071.3660.261
Ditto0.0520.9080.191‾\underline{0.191}2.5942.080‾\underline{2.080}1.6090.457
Qwen.+Wan.0.193‾\underline{0.193}0.760‾\underline{0.760}0.1932.612‾\underline{2.612}2.0652.007‾\underline{2.007}0.630‾\underline{0.630}
Ours0.2750.6760.1842.6452.2212.2280.783

EditaLive achieves the best or tied-best SR across all four categories and, among methods supporting global stylization, achieves the best or tied-best score on every metric in Style. On local edits, V2V baselines such as offline LucyEdit and UniVideo achieve higher source-referenced consistency, and streaming LiveEdit attains low APD, whereas EditaLive offers a favorable balance between consistency and edit success across all edit types at real-time speed. Compared with the offline Qwen.+Wan. cascade, which shares the same decoupled formulation, EditaLive achieves better ID-SIM, AED, APD, and BC across all four categories. A similar trend holds on CharEdit-Bench-L (Appendix D.2, Table 6), where EditaLive ranks first in ID-SIM, AED, EQ, and BC across all four categories (and in APD on three categories).

RQ3: Caching rule in Sec. 3.3 vs. Algorithm 1.

Thank you for pointing this out. Algorithm 1 describes the exact caching rule used in our implementation: only the first generated chunk undergoes an additional t=0t=0 forward pass on its final denoised prediction to initialize the attention sink, whereas every subsequent chunk directly reuses the keys and values from its last denoising pass. This rule is identical across training and inference, and we have updated Sec. 3.3 to match Algorithm 1.

Reviewer_F94VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses streaming character video editing, where a live video stream of a person must be modified in real time according to text instructions while preserving facial expression, body motion, and long-term appearance stability. The proposed EditaLive framework builds on Wan-Animate and introduces three components: (1) an appearance–motion decoupled editing formulation that edits only a reference frame and reconstructs the source video under motion conditions, supported by a new CharEdit-50K dataset; (2) chunk-wise causal streaming adaptation with bidirectional intra-chunk attention; and (3) aligned self-rollout distillation with Align Forcing, Fixed RoPE, and First-frame Preserved Sparse Attention (FPSA). Experiments on CharEdit-Bench-S/L report competitive character consistency, edit quality, and 14.47 FPS with 0.829 s inter-chunk latency.

Critical strengths
  1. The appearance–motion decoupling reframes editing as appearance transformation under explicit motion control. This is consequential because it directly targets the facial-expression mismatch of synthetic video-editing datasets identified as challenge (i) in the Introduction, and the reconstruction-based objective avoids entangling appearance edits with unintended expression changes.
  2. Align Forcing decouples gradient updates from rollout termination so the propagated KV cache matches inference-time construction, while Fixed RoPE assigns role-based positional indices (0–9) rather than absolute positions. The ablation shows that replacing Align Forcing drops ID-SIM from 0.492 to 0.252 and SR from 0.817 to 0.367, and removing Fixed RoPE yields ID-SIM 0.381, substantiating that drift originates from these specific mismatches.
Critical weaknesses
  1. The Introduction asserts that Wang et al. (2026) and Zhao et al. (2026) "remain misaligned in terms of RoPE indexing and KV-cache construction" and "do not explicitly address stability over long generation horizons", but the manuscript provides no direct analysis or measurement of these baselines' RoPE/KV-cache behavior. The quantitative comparison reports SANA-Streaming and StreamDiffusionV2 at very low SR (0.428 and 0.027), yet no evidence is given that these numbers reflect the claimed misalignment rather than, e.g., resolution mismatch or one-step denoising defaults.
  2. Evaluation relies heavily on a VLM judge that is also used in data construction. Text Alignment, Edit Quality, Background Consistency, and Success Rate are all scored by GPT-5.5, and GPT-5.5 is simultaneously used to generate instructions and filter CharEdit-50K samples. This creates a potential circularity between training-data curation and evaluation criteria.
Questions for authors
  1. Is there a direct measurement (e.g., RoPE index range or KV-cache state statistics during long rollouts) comparing EditaLive with Wang et al. (2026) and Zhao et al. (2026) to substantiate the claimed training–inference misalignment in those baselines, rather than inferring it from their SR/FPS?
  2. Since GPT-5.5 is used both to filter CharEdit-50K and to compute TA/EQ/BC/SR, what is the correlation between these VLM scores and the human user-study preferences (Fig.12)?
  3. What is the effect of the static stylized target used for style editing on motion fidelity? An ablation replacing it with per-frame stylized targets, even on a small subset, would clarify whether this approximation limits style-edit quality.
4 comments
AuthorsOpenReview ↗

Response to Reviewer F94V (Part 1/3)

We thank Reviewer F94V for the thorough and constructive feedback. We address each comment below.

Response to Weakness 1 (RW1) & Response to Question 1 (RQ1): Direct analysis of RoPE/KV-cache misalignment and baseline evaluation settings.

Thank you for noting the need for a direct mechanism comparison. We have added a detailed analysis of self-rollout training and streaming inference across EditaLive, LiveEdit, and SANA-Streaming in Appendix C.4 (summarized below) and refined our phrasing in Secs. 1 and 2:

Method (Phase)Temporal RoPE IndicesCached ContextKV-Cache Construction
LiveEdit (train)absolute, increasingall historyt0=0t_0=0 pass on sampled-step x^0\hat{x}_0
LiveEdit (infer)fixed 00 to 88sink + local windowt0=0t_0=0 pass on final x^0\hat{x}_0
SANA-Stream. (train)sink 00 to 44, rest increasing (bounded)sink + local windowt0=0t_0=0 pass on sampled-step x^0\hat{x}_0
SANA-Stream. (infer)sink 00 to 44, rest increasing (unbounded)sink + local windowt0=0t_0=0 pass on final x^0\hat{x}_0
Ours (train & infer)fixed 00 to 99ref + sink + local window1st chunk: t0=0t_0=0 on final x^0\hat{x}_0, later chunks: reuse final-step KV

Training-Inference Mismatch Analysis. (i) Temporal RoPE Indexing: LiveEdit trains with monotonically increasing indices attending to all preceding chunks, but switches during long-video inference to fixed indices 00 to 88 (sink 00 to 22, local window 33 to 55, current chunk 66 to 88) attending only to the sink and local window. SANA-Streaming shares the same rule across phases, but because its longest released training configuration covers only positions 00 to 121121, long rollouts on CharEdit-Bench-L exceed the training horizon (reaching a median max index of 197197 and a peak of 418418). EditaLive instead shares the exact same bounded layout (00 to 99: reference 00, sink 11 to 33, local window 44 to 66, current chunk 77 to 99) across training and inference. (ii) KV-Cache Construction: Following Self Forcing, LiveEdit and SANA-Streaming run a t0=0t_0=0 pass on x^0\hat{x}_0 from a randomly sampled step during DMD training, but on the final prediction during inference. EditaLive completes the full schedule in both phases, running the t0=0t_0=0 pass on chunk 1 and reusing final-step KVs thereafter. Table 2 validates alignment within our framework, and Figs. 10, 16, and 18 illustrate the progressive artifacts and color shifts in baselines over long streams.

Short-Clip SR and Baseline Settings. We agree that short-clip SRs reflect overall editing capability rather than long-horizon misalignment. Every baseline is evaluated in its official configuration, including SANA-Streaming at its native 704×1280704 \times 1280 resolution (Appendix C.2). StreamDiffusionV2 is training-free (built on Wan 2.1 and CausVid), so our misalignment claim does not apply to it. To verify whether its 1-step default bottlenecks its SR, we also tested it with 4 steps on CharEdit-Bench-S (Appendix C.2, Table 3):

SettingID-SIM ↑\uparrowAED ↓\downarrowAPD ↓\downarrowTA ↑\uparrowEQ ↑\uparrowBC ↑\uparrowSR ↑\uparrowPick ↑\uparrowFPS ↑\uparrowLatency ↓\downarrow
1 step (default)0.0600.8850.4851.7171.9240.6510.00719.9220.960.191
4 steps0.0421.0160.6311.8292.1080.3100.00720.246.200.645

Increasing to 4 steps slightly raises TA and EQ but sharply degrades BC and ID-SIM, leaving SR unchanged while cutting FPS by 3.4×3.4\times. We therefore retain its official 1-step setting.

AuthorsOpenReview ↗

Response to Reviewer F94V (Part 2/3)

RW2: Mitigating circularity with an independent VLM judge (Gemini-4-Argon).

Thank you for raising this important point. To decouple evaluation from CharEdit-50K curation, we re-scored all VLM metrics across the entire paper using Gemini-4-Argon, which was not used in instruction generation or sample filtering. On CharEdit-Bench-S (below), EditaLive ranks first in TA, EQ, and SR under both judges (and first in BC under Gemini-4-Argon), with identical top-3 rankings by SR (Kendall's τ=0.71\tau = 0.71). All VLM tables in the revised manuscript now report Gemini-4-Argon scores:

MethodGPT TA ↑\uparrowGPT EQ ↑\uparrowGPT BC ↑\uparrowGPT SR ↑\uparrowGemini TA ↑\uparrowGemini EQ ↑\uparrowGemini BC ↑\uparrowGemini SR ↑\uparrow
LiveEdit1.5421.9491.6400.3931.4691.9711.5510.113
SANA-Stream.2.2001.7771.5820.4282.1361.7021.5800.160
Stream.V21.3161.8690.4730.0271.7171.9240.6510.007
LucyEdit1.6041.7972.1210.3261.4931.7562.133‾\underline{2.133}0.287
UniVideo2.3772.1821.8660.5332.6532.0241.8520.500
Ditto1.5601.7961.1270.1671.8241.8361.4220.180
Qwen.+Wan.2.792‾\underline{2.792}2.544‾\underline{2.544}1.8970.713‾\underline{0.713}2.829‾\underline{2.829}2.100‾\underline{2.100}2.0640.767‾\underline{0.767}
Ours2.7962.6092.024‾\underline{2.024}0.7202.8402.2432.2060.847

RQ2: Correlation between VLM scores and human user-study preferences (Fig. 12).

We evaluate alignment between VLM scores and the per-case human votes in Fig. 12 (40 short and 20 long sets) at two levels: (i) method-level Spearman correlation ρ\rho between SR and mean human preference share across all four criteria, and (ii) case-level pairwise agreement between total human votes and summed VLM scores (TA+EQ+BC\text{TA}+\text{EQ}+\text{BC}) across all method pairs on the same video, excluding ties:

JudgeShort Pairwise AgreementShort ρ\rhoLong Pairwise AgreementLong ρ\rho
GPT-5.571.5%0.8385.4%0.90
Gemini-4-Argon70.9%0.8383.4%0.97

Both VLM judges show strong rank correlation (ρ≥0.83\rho \ge 0.83) and pairwise agreement well above the 50% chance level (70.9%70.9\% to 85.4%85.4\%, permutation test p<0.01p < 0.01). On short-video pairs involving EditaLive, win rates under GPT-5.5 (90%), Gemini-4-Argon (92%), and aggregated human votes (88%) align closely, confirming that EditaLive's advantage is consistent with human judgment rather than judge-specific (added in Appendix D.6).

AuthorsOpenReview ↗

Response to Reviewer F94V (Part 3/3)

RQ3: Effect of static stylized targets on motion fidelity and style-edit quality.

Following your suggestion, we conduct a full-scale Stage-1 ablation (Appendix D.4, Table 8) by replacing all static stylization targets in CharEdit-50K with an equal number of dynamic stylized videos from Ditto-1M under identical settings (using the first source frame as reference and extracting motion from the stylized target video). Alongside APD and AED, we report Motion Ratio (Medit/MsrcM_{\text{edit}}/M_{\text{src}}, closer to 11 is better), the mean Farneback optical-flow magnitude of the edited video divided by that of the source video (values below 11 indicate damped motion, while values above 11 indicate extra motion or temporal jitter):

SubsetStylization TargetID-SIM ↑\uparrowAED ↓\downarrowAPD ↓\downarrowTA ↑\uparrowEQ ↑\uparrowBC ↑\uparrowSR ↑\uparrowMR (→1\to 1)
Style (46)Ditto-1M video0.2680.5860.1521.9202.3912.3480.2831.338
Style (46)Static target (Ours)0.2470.6430.1582.3772.7172.7610.5871.103
Local (104)Ditto-1M video0.6940.3970.0982.1512.6122.6630.6541.123
Local (104)Static target (Ours)0.6970.4050.1002.3212.5902.7560.7401.104
All (150)Ditto-1M video0.5630.4550.1152.0802.5442.5670.5401.189
All (150)Static target (Ours)0.5590.4780.1182.3382.6292.7580.6931.104

Our static-target model achieves better TA, EQ, BC, and SR on both Style and All, more than doubling Style SR (0.5870.587 vs. 0.2830.283). While the Ditto-1M variant obtains slightly better ID-SIM and AED/APD on Style, this comes with substantially lower TA and SR. Regarding motion, the static-target model shows no reduction in average estimated motion magnitude on style edits, attaining virtually the same Motion Ratio on Style (1.1031.103) and real-video-supervised Local edits (1.1041.104), with a much smaller mean absolute deviation ∣1−Medit/Msrc∣|1 - M_{\text{edit}}/M_{\text{src}}| on Style than the Ditto-1M variant (0.1340.134 vs. 0.3380.338). By contrast, the Ditto-1M variant's ratio exceeds 11 in all 4646 Style cases (1.081.08 to 2.062.06), indicating consistently higher estimated motion magnitude, potentially including temporal jitter. Overall, high-quality stylized still images provide cleaner style supervision without impairing motion fidelity.

Dear Reviewer F94V,

Thank you again for your thoughtful suggestions. With the discussion period nearing its end, we would appreciate any further feedback on our responses above. To address your concerns, we added a direct analysis of RoPE/KV-cache behavior, re-evaluated our results with an independent VLM judge, examined alignment with human preferences, and conducted the targeted stylization-training ablation.

If these additions have addressed your main concerns, we would be grateful if you could revisit your overall assessment in light of the additional evidence. Please also let us know if any concerns remain, we would welcome the opportunity to clarify them during the remaining discussion period.

Thank you for your time and consideration.

Reviewer_CyhPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes reference-image-conditioned live-stream video editing. For this problem, the authors propose a data pipeline and a dataset (CharEdit-50K) that generate an edited reference image and treat the unedited video as the target video, and train the editing model to predict the unedited video as the target under the guidance of the edited reference image and a reverse instruction. The authors also distill the model into a two-step streaming model that runs at 14.47 FPS with 0.829 s inter-chunk latency on a single H100.

Critical strengths

The figures, especially the illustrations of the method, are well plotted and easy to follow.

The proposed method ranks first or second on all eight quality metrics in Table 1 (e.g., ID-SIM 0.550, AED 0.499, APD 0.124, SR 0.720), better than several bidirectional models and streaming models.

Critical weaknesses

The problem studied and method proposed are largely explored or similar to prior work, the main contribution is engineering them into a working system.

The method still relies on a reference image being available. In addition, the training data depend on an external image editing model (Qwen-Image-Edit and Nano Banana 2) to produce the edited reference images, so if the method goes to application, it requires the quality and efficiency of that image editing model.

Minor: I understand that only editing a reference image can alleviate the expression mismatch, but it can still happen. Also, in the comparison figure (Fig. 4), the authors do not cite or mention where the selected samples are from to reveal the weakness of existing datasets, making it hard for readers to verify the claim.

Minor: I also think some citations are missing in the authors' work. I am not forcing the authors to cite every recent instruction-based video editing or streaming editing paper, but I think the proposed first-frame preservation and Fixed RoPE are quite similar to common solutions for attention sinks and long video generation.

Questions for authors

If the authors always treat the unedited source video as the target, how can the model learn special edits, like changing to a magic style or adding a dragon, which are usually impossible to appear in an unedited source video?

For Align Forcing, I do not think it is a very different thing compared to Self Forcing: the same DMD loss, the same real and fake score models, and a very similar training curriculum.

The only difference is how to construct the chunk KV cache. Self Forcing stops at a sampled step and takes the current x0 to do a t=0 pass to get a clean KV cache, whereas the authors run all the steps and take the final step's KV cache from the model instead. Technically, the DMD loss is pushing the student generator's output at every step towards x0 no matter which step it is, so I do not think it would make much difference. In Table 2, the authors show that without Align Forcing the performance drops from ID-SIM 0.492 to 0.252 and SR 0.817 to 0.367, which is quite surprising. Could the authors provide an apples-to-apples comparison between Self Forcing and Align Forcing?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CyhP (Part 1/3)

We thank Reviewer CyhP for the constructive feedback. Before addressing each point, we briefly summarize how our formulation decouples the appearance to be edited from the motion to be preserved. The reference frame and text instruction jointly determine the target appearance, while the skeleton sequence and implicit facial representations capture body motion and facial dynamics. In the summary table below, VV denotes the reconstruction target video during training and XX denotes the input video at inference (the edited frame in training is obtained by applying a forward edit to a frame sampled from VV, whereas the source frame at inference is taken directly from XX):

StageAppearance conditionMotion conditionTarget / output
TrainingEdited frame + reverse instructionSkeleton and facial conditions from VVReconstruct VV
InferenceSource frame + user instructionSkeleton and facial conditions from XXEdited XX

Because the edited frame differs in appearance from VV, reconstructing VV requires the model to execute the appearance transformation specified by the reverse instruction.

Response to Weakness 1 (RW1): Core contributions beyond system integration.

Our contributions span formulation, data curation, and streaming distillation: (1) an appearance-motion decoupled training design that enables instruction-guided character editing while avoiding facial-expression mismatches inherited from synthesized video targets, (2) a cost-efficient data construction pipeline (CharEdit-50K) that pairs single-frame editing with real-video reconstruction instead of costly paired-video synthesis, and (3) an aligned self-rollout distillation strategy that resolves training-inference discrepancies in positional encoding and KV-cache construction for efficient, stable long-horizon streaming.

RW2: Role of reference frames and external image editors in deployment.

At inference time, EditaLive needs neither an external image editor nor an extra user-supplied image: the reference frame is taken directly from the input video stream XX (its first frame in our experiments), and appearance editing is driven directly by the text prompt. Off-the-shelf image editors (Qwen-Image-Edit and Nano Banana 2) are used strictly offline to build training data with quality filtering. Compared with the Qwen-Image-Edit + Wan-Animate cascade, EditaLive achieves better expression preservation (Tables 1 and 4) and 11.4×11.4\times higher throughput (Table 1), even when excluding the cascade's one-time image-editing latency.

RW3: Expression shifts in edited reference images and source attribution in Fig. 4.

While image editing can slightly alter the facial expression of the edited reference frame, our decoupled formulation does not require the original and edited reference images to share identical expressions or poses. Because motion signals are extracted directly from the reconstruction target video VV, supervision remains strictly motion-aligned with VV regardless of expression changes in the reference image. Regarding Fig. 4 (left), the examples are from the Ditto dataset, and we have added the explicit citation to the caption.

AuthorsOpenReview ↗

Response to Reviewer CyhP (Part 2/3)

RW4: Missing citations on frame anchoring, attention sinks, and temporal RoPE re-indexing.

Thank you for pointing this out. We agree that first-frame preservation and Fixed RoPE connect to established ideas in long-video generation, including frame anchoring against drift [1], initial-frame attention sinks [2], and bounded temporal RoPE re-indexing [2, 3]. We have added these citations in Sec. 2 (Efficient and Streaming Video Generation) and Sec. 3.3 of the revised manuscript, and clarify below how our designs differ.

Fixed RoPE. Infinity-RoPE [3] and LongLive's infinity mode [2] re-index temporal positions at inference time without training, and LiveEdit uses fixed indices at inference while training with monotonically increasing indices. We apply the same role-based layout (reference sink, attention sink, local window, and current chunk) during both self-rollout training and streaming inference, so the model is never exposed at inference to a positional layout unseen during training.

First-frame preserved sparse attention. Existing KV-cache frame sinks retain initial-frame tokens under dense attention [2]. Under top-KK block-sparse attention, however, these tokens can still be dropped by content-adaptive block selection, which tends to favor the local window. FPSA therefore always includes all key blocks from the first generated frame among the KK selected key blocks for every query block.

[1] Zhang et al. Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models. NeurIPS, 2025.

[2] Yang et al. LongLive: Real-Time Interactive Long Video Generation. ICLR, 2026.

[3] Yesiltepe et al. Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout. CVPR, 2026.

Response to Question 1 (RQ1): Learning edits that rarely appear in unedited real videos (e.g., stylization or fantasy objects).

Our framework does not restrict target videos VV to unedited real footage. In fact, our stylization targets are already synthetic (constructed by repeating a stylized image across frames, as detailed in Appendix C.1). More generally, synthesized or pre-edited videos can serve as targets VV for out-of-domain concepts: for instance, we can use a synthetic video containing a dragon as the target VV and create the reference image by selecting a frame from that video and removing the dragon. This reference image is then paired with an "add a dragon" instruction to train the model to reconstruct VV.

AuthorsOpenReview ↗

Response to Reviewer CyhP (Part 3/3)

RQ2: Apples-to-apples comparison between Self Forcing and Align Forcing.

Thank you for pointing this out. We agree that Align Forcing retains Self Forcing's DMD objective and score models, while focusing on aligning rollout and KV-cache construction between training and inference with fewer student forward passes. Your question prompted us to check the configurations of our Stage-3 ablations, and we found that the learning rate of w/o Align Forcing in our original submission had been mistakenly set to 1×10−51 \times 10^{-5} instead of 2×10−62 \times 10^{-6} (causing late-stage DMD degradation), while all other ablation runs were configured correctly. Below we report the original 1×10−51 \times 10^{-5} run (shown in gray for reference and excluded from comparison) alongside the corrected 2×10−62 \times 10^{-6} Self Forcing run and Align Forcing on CharEdit-Bench-L, holding Stage-2 initialization, training data, update steps, denoising schedule, score models, Fixed RoPE, and FPSA identical (with VLM metrics updated to the independent Gemini-4-Argon judge):

VariantLRStudent NFEs/chunkID-SIM ↑\uparrowAED ↓\downarrowAPD ↓\downarrowTA ↑\uparrowEQ ↑\uparrowBC ↑\uparrowSR ↑\uparrowPick ↑\uparrowCLIP ↑\uparrowDINO ↑\uparrow
Self Forcing (orig. Table 2)\color{gray}{\text{Self Forcing (orig. Table 2)}}1×10−5\color{gray}{1\times 10^{-5}}2.5 / 3.0\color{gray}{2.5\text{ / }3.0}0.252\color{gray}{0.252}0.711\color{gray}{0.711}0.338\color{gray}{0.338}2.722\color{gray}{2.722}2.311\color{gray}{2.311}1.351\color{gray}{1.351}0.250\color{gray}{0.250}20.03\color{gray}{20.03}98.62\color{gray}{98.62}99.07\color{gray}{99.07}
Self Forcing (corrected)2×10−62\times 10^{-6}2.5 / 3.00.4870.5970.1132.6942.3222.3500.70019.5698.5498.92
Align Forcing (Ours)2×10−62\times 10^{-6}2.0 / 2.00.4920.5760.1092.8172.4002.3780.73319.7898.5198.96

Consistent with your intuition, once the learning rate is matched, the gap between Self Forcing and Align Forcing is modest rather than a sharp drop. Even so, Align Forcing improves 99 out of 1010 metrics over long streams, and the updated qualitative comparison in Fig. 7 (right) shows improved eye and mouth expression preservation and sharper hand details (updated in Table 2 and Sec. 4.3). At the same time, Align Forcing reduces student-generator forward evaluations from 2.52.5 to 2.02.0 NFEs per subsequent chunk in training and from 3.03.0 to 2.02.0 in inference by removing the per-chunk t0=0t_0=0 cache pass. Thank you again for this insightful observation, which helped us catch and correct this configuration error and strengthen the paper.

Dear Reviewer CyhP,

Thank you again for your constructive feedback. As the discussion period is coming to a close, we would like to briefly follow up on our responses above. In particular, your comments helped us identify and correct the learning-rate mismatch in the Self Forcing ablation. We have updated the corresponding results and clarified our formulation, training supervision, and connections to prior work in the revised manuscript.

If you have had a chance to review these updates, we would appreciate hearing whether they address your main concerns and whether any points still need clarification. We remain available to respond. Thank you for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers (CyhP, F94V, and ZckW) and the Area Chair for their constructive feedback and insightful suggestions. We have thoroughly revised the manuscript (with changes highlighted in the updated PDF) and conducted additional experiments and analyses to address every comment. Below is a summary of the main updates in the revised manuscript [R1=CyhP, R2=F94V, R3=ZckW]:

[R2W2, R2Q2] Independent VLM Evaluation and Human Preference Alignment (Tables 1, 2, and 4–6, Appendix C.3 and D.6):

To decouple evaluation from CharEdit-50K data curation, we re-evaluated all benchmarks and ablations using an independent VLM judge (Gemini-4-Argon) and added method- and case-level correlation analyses against human user-study votes in Appendix D.6.

[R3W2, R3Q2] Per-Category Breakdowns and Baseline Consistency Scores (Sec. 4.2, Table 1, Appendix D.2, Tables 5 and 6):

We added 4-category breakdowns (Add, Remove, Change, and Style) on both CharEdit-Bench-S and CharEdit-Bench-L, reporting character-consistency scores for all baselines (including LiveEdit and LucyEdit).

[R3W1, R3Q1] End-to-End Streaming Efficiency at 480 × 832 (Sec. 4.2, Appendix D.3, Table 7):

We benchmarked the complete end-to-end streaming pipeline at 480 × 832 (covering input chunk buffering, ViTPose extraction, face localization, motion encoding, two-step DiT denoising, and VAE decoding) across single- and pipelined multi-GPU setups, reporting both unthrottled maximum throughput and steady-state latency under a continuous 16 FPS stream.

[R2Q3, R3W2, R3Q2] Stylization Supervision Details and Full-Scale Ablation (Sec. 4.3, Appendix C.1 and D.4, Table 8):

We clarified the forward training inputs for stylization in Appendix C.1 and added a Stage-1 ablation in Appendix D.4 comparing our static stylization targets against an equal number of dynamic stylized videos from Ditto-1M.

[R1Q2] Corrected Stage-3 Self Forcing Ablation (Sec. 4.3, Table 2, Fig. 7):

Auditing our Stage-3 ablation configurations revealed a learning-rate misconfiguration in w/o Align Forcing (1e-5 instead of 2e-6). We reran w/o Align Forcing at the matched 2e-6 learning rate and updated Table 2, Sec. 4.3, and Fig. 7 (right).

[R1W3, R1W4, R2W1, R2Q1, R3Q3] Training–Inference Mismatch Analysis, Baseline Step Check, and Clarifications (Secs. 1–3, Fig. 4, Appendix C.2 and C.4, Table 3):

We added a direct comparison of temporal RoPE indexing and KV-cache construction across EditaLive, LiveEdit, and SANA-Streaming (Appendix C.4, with refined phrasing in Secs. 1 and 2), added a 4-step evaluation of StreamDiffusionV2 (Appendix C.2, Table 3), aligned the KV-cache description in Sec. 3.3 with Algorithm 1, credited Ditto in Fig. 4, and added discussions and citations on frame anchoring, attention sinks, and temporal RoPE re-indexing in Secs. 2 and 3.3.

AuthorsOpenReview ↗

Final concluding remarks from the authors

Dear AC and Reviewers,

As the discussion period nears its end, we would like to provide a brief summary of the revisions and experiments we have completed in response to your valuable feedback. We hope these updates clarify the strengths of EditaLive and address the concerns raised.

Key improvements made in the revised manuscript include:

  • Independent Validation: We re-scored all VLM metrics using Gemini-4-Argon (an independent judge) and verified strong alignment between VLM scores and human preferences (Appendix D.6).
  • Full-Pipeline Efficiency: We benchmarked the end-to-end streaming pipeline at 480 × 832 on H100s, demonstrating that our 3-GPU configuration sustains a lossless 16 FPS input stream with sub-2s capture-to-output latency (Appendix D.3).
  • Supervision & Ablation: We clarified the stylization training pipeline and added a full-scale Stage-1 ablation demonstrating that static targets maintain higher edit success rates and motion fidelity than dynamic ones (Appendix D.4).
  • Alignment Analysis: We provided a direct mechanism comparison of RoPE indexing and KV-cache construction, illustrating why our role-based Fixed RoPE and Align Forcing are critical for long-term stability (Appendix C.4).

We are very grateful for the thoughtful reviews, which helped us significantly improve the paper's clarity and robustness. We stand ready to provide any further clarifications should they be needed.

Best regards, Authors of Submission #15798

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.