acceptodds
Withdrawn from ICLR 2027

StreamingEffect: Real-Time Human-Centric Video Effect Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video effect generation aims to add expressive, aesthetically coherent, and instruction-following effects to human-centric videos in real time, enabling live applications such as e-commerce streaming, entertainment, and vlogging. Unlike conventional video editing, this task requires not only temporal consistency and visual quality, but also low-latency generation under strict deployment constraints. In this paper, we introduce StreamingEffect, a new framework for real-time human-centric video effect generation. We formulate the task as a streaming video-to-video generation problem and adopt an in-context video editing architecture, which preserves source-video information more faithfully than prior conditioning schemes such as VACE-style designs, leading to stronger consistency and content preservation. To overcome the high cost of bidirectional diffusion-based generation, we further propose a two-stage distillation pipeline that first converts a high-quality bidirectional teacher into a causal autoregressive student, and then distills it from 50-step sampling to 4-step generation. We further introduce keyframe control, which allows high-quality image-edited keyframes to be injected and propagated online for interactive effect enhancement. To support this task, we present VideoEffect-60K, a large-scale dataset of 60K real high-quality videos covering 600 effect categories. Experiments show that our method achieves the first real-time 720P streaming video effect generation on a single H200 GPU while maintaining strong quality and controllability.

Primary area:
generative models
Keywords:
Video Effect Generation, Streaming Video Generation, Diffusion Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_JeF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies human-centric streaming video effect generation, aiming to apply text- or reference-guided effects while preserving the source subject and scene. StreamingEffect builds on Wan2.2-TI2V-5B with an interleaved representation of reference, source, and target latents. A bidirectional teacher is adapted into a causal student, followed by on-policy training with paired-target supervision to reduce sampling to four steps. The paper also introduces VideoEffect-130K, containing 130K paired videos spanning rendered effects and general editing. Evaluation on 2,000 held-out clips compares editing quality against open-source and commercial systems and examines the quality–throughput trade-off across training stages. The final student achieves 14.1 FPS at 720p on a single H200.

Critical strengths
  1. The paper addresses a useful deployment problem in which source preservation and responsiveness matter alongside generation quality. The combination of reference-conditioned editing, causal attention, and few-step generation is coherent. Table 2 shows that on-policy training improves all reported VLM dimensions over directly running the Stage-1 student with four steps, providing evidence that this training stage is useful for the proposed setting.

  2. The paired dataset is another potentially valuable contribution. Its combination of rendered effects and synthetic edits provides substantial task-specific supervision for a problem that is difficult to address with generic video-generation data alone. Together with the reasonably strong teacher results, this makes the work informative even though the methodological contribution is primarily an integration and adaptation of existing techniques.

Critical weaknesses

1. Selective benchmarking in the primary comparison: Table 1 only evaluates the 50-step offline Teacher (0.14 FPS) against baselines, omitting the proposed 4-step streaming model entirely. When cross-referencing Table 2, the student’s reconstruction quality drops sharply (PSNR falls from 19.4 to 16.7, LPIPS degrades from 0.195 to 0.269), falling behind Kiwi-Edit and Kling O1. Omitting the deployed model from the main comparison obscures the actual quality-latency trade-off of the contribution.

2. Unsubstantiated real-time streaming claims: The reported throughput of 14.1 FPS on an enterprise H200 accelerator falls well below the standard 24-30 FPS input rate, meaning input frames will inevitably back up during continuous streaming. Furthermore, operating on 5-latent-frame chunks introduces an algorithmic buffering delay of roughly 17–20 raw frames (~700–800 ms at 24 FPS) before denoising even starts, yet the paper reports neither end-to-end capture-to-display latency nor time-to-first-frame.

3. Uncontrolled throughput jump in Table 2: The 4-step Causal AR student achieves 7.52 FPS, whereas the 4-step StreamingEffect student reaches 14.1 FPS. Because both models execute the exact same number of sampling steps on identical architectures, this near 2 times speedup is unaccounted for. It remains unclear whether this gain comes from unstated inference modifications (such as dropping CFG or applying compiler optimizations) rather than the proposed training method.

4. Fragile evaluation protocol and lack of human studies: The perceptual evaluation relies heavily on frontier VLMs prompted on concatenated 3-column video grids. VLMs frequently overlook high-frequency temporal flickering, boundary tearing, and fine facial distortions across multi-frame downscaled grids. For an interactive human-centric effect task, the complete absence of human perceptual benchmarks or user preference studies leaves the claimed visual quality weakly supported.

5. Lack of empirical validation for key interactive claims: The paper highlights long-stream generation and dynamic on-the-fly reference switching as core capabilities, but provides no quantitative evaluation for either. Without measuring error drift over extended rollouts or latency/artifact spikes when swapping references within the persistent KV-cache, the feasibility of interactive use remains unproven.

Questions for authors

1. Deployment clarification: Does the reported 14.1 FPS include source encoding, denoising, and output decoding? What are the startup and steady-state end-to-end latencies, including chunk buffering, and how is a continuous 24 FPS input handled?

2. Controlled comparison: What explains the throughput increase from 7.52 to 14.1 FPS between the two four-step students? Are CFG, cache management, numerical precision, and inference optimizations identical?

3. Evidence for the deployed system: Do you have evaluations of the final student on streams substantially longer than the training clips and during online reference changes? In particular, how are cached states handled after a reference update, and what evidence supports the claimed absence of drift and flicker?

Reviewer_yP8fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StreamingEffect addresses streaming visual effect generation for human videos. It supports reference image and text control, converts a bidirectional video editing model into a causal model with four sampling steps, and constructs 130K paired videos for training.

Critical strengths

Main Strengths

The framework supports both text and reference image control across accessories, makeup, and stylization, with clear practical applications. The dataset combines rendered effects and synthetic video editing to provide substantial paired supervision.

Table 2 shows that the second training stage improves effect alignment and temporal consistency over directly applying four sampling steps, supporting the benefits of training with previously generated context.

Critical weaknesses

Main Weaknesses

The evaluation videos run at 24 FPS, while the final model achieves only 14.1 FPS on a single H200. This does not demonstrate that the system can continuously keep pace with the input stream. First frame latency, chunk buffering time, and end to end latency including video encoding and decoding are also missing, leaving the central claim of real time interaction insufficiently supported.

The quality advantages in Table 1 come from the offline teacher with 50 sampling steps, while the final streaming model performs substantially worse. Its PSNR drops from 19.4 to 16.7, and its LPIPS of 0.269 is worse than the 0.227 reported for Kiwi Edit. The paper should assess the quality and speed tradeoff using the final deployed model and include comparisons with existing streaming methods.

Quantitative evaluation covers only 97 frames, approximately four seconds. Although Figure 1 illustrates extended operation and effect switching, there is no corresponding evaluation of temporal drift, switching latency, or persistence of previous effects. The paper also does not explain how cached history conditioned on an earlier reference is handled after a reference update, making sustained operation and online control difficult to assess.

Minor Weaknesses

The two models using four sampling steps achieve 7.52 and 14.1 FPS, respectively, without an explanation for the nearly twofold speed difference. The relevant system optimizations and timing scope should be specified.

Questions for authors

Please specify the target input frame rate and report first frame latency and sustained end to end latency, including whether frame dropping or input frame rate reduction is required.

Please include the final streaming model in the main baseline comparison and explain the speed difference between the two models using four sampling steps.

Please provide experiments on long videos and changes to reference images or text during streaming, and explain how cached history is handled when conditions change.

Reviewer_jQE6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces StreamingEffect, a framework for applying visual effects to human-centric video streams using reference images and text prompts. It adapts a bidirectional editing model for causal streaming inference and uses paired supervision to enable four-step sampling. Users can also update the reference image during streaming to change the effect. The authors introduce VideoEffect-130K, containing 130,000 video pairs across approximately 600 categories. Evaluation on 2,000 held-out samples includes comparisons with existing editors and ablations of the training stages. The final model achieves 14.1 FPS at 720p on a single H200 GPU.

Critical strengths
  1. A useful dataset for human-centric video effects. VideoEffect-130K contains 130,000 video pairs across approximately 600 categories, combining rendered effects with synthetic edits. It helps address the shortage of paired training data for this task.
  2. Experimental support for dedicated few-step training. Compared with directly running the causal student at four steps, the final model improves the VLM-based quality score from 7.26 to 7.75. It also achieves 14.1 FPS at 720p, demonstrating a practical benefit of the training pipeline.
Critical weaknesses
  1. The efficiency trade-off and streaming performance remain unclear. Sequence concatenation allows target tokens to attend directly to the source video and may help preserve input content, but it also lengthens the sequence and increases self-attention computation. Moreover, quantitative evaluation covers only 97-frame clips, while claims of long-term stability rely on a few representative frames in Figure 1. This provides limited evidence that the method remains stable over extended video streams.
  2. The automated quality evaluation needs stronger validation. Appendix B takes the median of three ratings from each judge, GPT-4o and Gemini 2.5 Pro, and then averages the two medians. However, the paper does not report each judge’s results or examine differences in scoring scales, ranking agreement, or agreement with human judgments. One judge consistently giving higher scores would not necessarily affect method rankings, but averaging could conceal method-specific preferences or conflicting rankings. Taking medians of repeated ratings does not resolve these systematic biases.
Questions for authors
  1. Could you quantify the quality gains and computational overhead of sequence concatenation, and clarify whether the reported 14.1 FPS includes video encoding and decoding?
  2. Could you report GPT-4o’s and Gemini’s scores separately, assess their agreement on method rankings, and provide any validation against human judgments?
  3. Do the examples labeled “100 s later” and “1000 s later” in Figure 1 come from a single continuous run, and how was stability evaluated over time and across reference-image changes? I would consider raising my score if the authors provide convincing evidence addressing my concerns.
Reviewer_FExQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

StreamingEffect addresses human-centric streaming video editing using a Wan2.2-TI2V-5B backbone. It combines reference-conditioned in-context editing, causal adaptation, and four-step on-policy training with an SNR-weighted reconstruction loss. The paper also introduces VideoEffect-130K, containing 70K rendered-effect pairs and 60K general-editing pairs. The final model achieves 14.1 FPS at 720p on one H200.

Critical strengths
  • The dataset addresses a useful gap: paired supervision for localized human-centric effects, where preserving the source appearance matters as much as rendering the effect.
  • Interleaving source and target chunks provides a straightforward way to retain source information within a causal editing model.
  • Stage 2 improves the four-step student: effect-reference alignment increases from 7.47 to 8.14, and temporal consistency from 7.93 to 8.37.
Critical weaknesses
  • Long-stream stability and interactive switching remain weakly tested. Quantitative evaluation uses approximately four-second clips. Figure 1 shows selected frames at 100/1000 seconds, but does not measure drift after cache eviction or response time after a reference change. Cached states retain information from the previous reference, making the unspecified cache-update policy particularly relevant.
  • The contribution of the training objective is not isolated. Relative to Self Forcing, Eq. (5) uses paired-target regression instead of distribution matching. Table 2 does not separate the effects of on-policy context, SNR weighting, and additional training. A four-step teacher-forced control with the same loss and training budget would make the claimed benefit much clearer.
  • The main comparison does not represent the deployed model. Table 1 reports the teacher, while the streaming student’s LPIPS worsens from 0.195 to 0.269 and Quality falls from 8.41 to 7.75. This may be a reasonable speed–quality tradeoff, but it should be evaluated directly against streaming methods such as StreamV2V, rather than inferred from the teacher’s results.
  • The benchmark provides limited evidence of generalization. All 2,000 test samples come from the rendered-effect subset. The paper does not specify whether source videos, identities, or effect templates overlap across splits. Performance on unseen effects and independently edited reference images remains unclear.
Questions for authors
  • Why does throughput increase from 7.52 to 14.1 FPS when both students use four sampling steps? What inference settings or optimizations differ?
  • How are reference-dependent cached states handled when the reference changes? Please report switching latency and quality over continuous long streams.
  • Does Stage 2 still outperform a teacher-forced baseline under matched loss, training budget, and inference settings?
  • How are dataset splits constructed, and where do the evaluation reference images come from?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.