acceptodds
Under review as a conference paper at ICLR 2027

Real-time Streaming Video Stylization via Rectified Style Flow

Abstract

Diffusion-based image stylization provides strong appearance control but is too slow and temporally unstable for streaming video. Directly training a causal video generator, on the other hand, entangles style, motion, and long-horizon autoregression and makes every new style expensive to support. We introduce RSF, a staged framework for real-time streaming video stylization. Its core is rectified style flow: rather than generating from pure noise, the model transports a partially corrupted source distribution at noise level s to a target-style distribution. We first learn modular style LoRAs on paired source-target video clips. We then freeze a bank of these LoRAs and train the full video DiT with teacher ODE pairs and causal forcing, separating reusable autoregressive dynamics from style-specific parameters. The resulting causal teacher is distilled to a one-step student using distribution matching distillation, followed by generated-history rollout fine-tuning to reduce exposure bias. Finally, a lightweight recovery procedure adapts previously unseen style LoRAs without repeating the complete video training pipeline. At 832x480 resolution, optimized mixed-FP8 inference sustains 102.1 FPS with 78.4 ms input-to-output latency on a single NVIDIA H100 and 36.3 FPS with 220 ms latency on a single NVIDIA RTX 5090. The H100 deployment requires only 4.81 GiB of steady-state resident GPU memory. We evaluate target-style text alignment, source preservation, and temporal stability on a 50-video benchmark under five styles against four public video stylization systems; the strictly paired CLIP-T comparison uses 47 complete videos. We will open-source all training code, model weights, and prepared datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
generative model, style transfer, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AfCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a real-time streaming video stylization framework that combines source-initialized flow matching, style-specific LoRAs, causal teacher forcing, DMD, and rollout fine-tuning. It achieves impressive inference efficiency with reasonable visual quality.

Critical strengths
  1. Impressive inference efficiency, achieving over 100 FPS on an H100.
  2. A practical framework that combines one-step generation with temporally consistent streaming stylization.
  3. The ability to adapt new style LoRAs without repeating the full distillation pipeline is interesting.
Critical weaknesses
  1. Lack of theoretical analysis for Rectified Style Flow. While the paper argues that source initialization shortens the transport path, it lacks a theoretical analysis of how this affects trajectory straightness and distillation efficiency. The authors are encouraged to discuss its relationship to existing source-to-target bridge formulations, such as Bridge Transformers, and clarify the theoretical distinction.
  2. The method is restricted to style-specific LoRAs rather than general video editing. It is somewhat time-consuming to train a separate LoRA for each style.
  3. Most training components, including interleaved teacher forcing, causal adaptation, and DMD, are established techniques. The technical novelty is therefore somewhat limited.
Questions for authors
  1. Could the authors compare against a simpler baseline combining a pretrained causal video model, style LoRA, and SDEdit-style source initialization? It would be helpful to understand whether the proposed multi-stage training pipeline provides substantial improvements over this simpler approach.
  2. Why does multi-style training collapse, and could a unified model support more general editing tasks?
  3. How much data and training time are needed to adapt an unseen style? The recovery experiments report 100 updates but do not clearly specify the dataset size.
  4. How does the proposed recovery strategy compare with including a new style in the full four-stage training pipeline? Is there a noticeable difference in stylization quality, temporal consistency, or training cost?
1 comment

1. What is the theoretical distinction from bridge formulations, and does RSF make one-step distillation easier?

RSF differs from a Brownian bridge in where it introduces randomness. We corrupt the source endpoint, Zs=(1−s)X+sϵZ_s=(1-s)X+s\epsilon, and use a deterministic linear interpolation from that sampled endpoint to its paired style target YY. For geometry-preserving stylization, this lets the student retain source structure rather than regenerate it from noise. Our sweep also shows why retaining more source information is not always better: teacher CLIP-T peaks at s=0.2s=0.2 for American Cartoon and 0.30.3 for Flat Vector Illustration, but at 0.60.6 for Van Gogh and 3D Chibi. For 3D Chibi, increasing ss from 0.2 to 0.6 improves style alignment while reducing source similarity. Thus, the useful starting point depends on what the edit must change. This motivates distilling a source-conditioned correction into one step.

2. Compared with a pretrained causal video model + style LoRA + SDEdit initialization?

Here is the comparison results of SDEdit + Style Lora (the initial training start at s comes from transferred domain) against RSF + Style Lora (the initial training start at s comes from original domain). RSF wins is the actual clips that RSF achieves higher CLIP-T against SDEdit.

StyleSDEdit CLIP-T ↑RSF CLIP-T ↑RSF − SDEditSDEdit DINO-I ↑RSF DINO-I ↑RSF wins
Van Gogh0.20660.2783+0.07180.97250.712947/47
Flat Vector Illustration0.15990.2203+0.06040.97210.787346/47
American Cartoon0.19030.2608+0.07060.97230.714547/47
3D Chibi0.19530.2376+0.04230.97250.846545/47
Overall0.18800.2493+0.06130.97230.7653185/188

The results show that RSF is effective for the degree of stylization.

3. Why train a LoRA per style? Does a unified model necessarily collapse?

No. We chose per-style LoRAs because of a failure mode we observed, not because a unified model must collapse. When we trained the full RSF DiT jointly across styles, changing the requested style often produced much the same blended appearance, even though we sampled the style bins approximately equally. The outputs looked like a visual average of the styles rather than a convincing example of the requested one. Even strong classifier-free guidance did not reliably restore the distinguishing features.

Our hypothesis is that a shared DiT can reduce the flow-matching loss across many styles by first learning the changes they have in common. Updates for features specific to one style may instead compete. A separate LoRA gives each style a small set of parameters dedicated to its own source-to-target flow, without updates from other styles overwriting it. The trade-off is one adapter per style. Our observations motivate that choice under the setting of an 1.3B-level lightweight models, which may change with larger model (since the have more parameters for each specific style).

4. How much data and compute does recovery need, and how does it compare with full retraining?

Given an existing streaming backbone and a Stage-1 LoRA for the new style, our default recovery uses 10 bidirectional teacher trajectories of 81 frames each per style. We freeze both the backbone and the style LoRA and train only a rank-64 residual LoRA for 100 updates.

Using nested sets of 1, 3, and 10 trajectories per style gave nearly identical mean CLIP-T on Pexels-47: 0.2465, 0.2462, and 0.2464, respectively, compared with 0.2361 without recovery. The clearest benefit of using ten rather than one was lower warped LPIPS (0.01973 vs. 0.02065).

For the full-retraining reference, we reused the same four Stage-1 style LoRAs but retrained the full AR teacher and one-step student on the four styles together. This required 1,800 teacher trajectories (450 per style), 12,000 AR updates, and 2,000 DMD outer iterations:

One-step modelCLIP-T ↑DINO-I ↑Warped LPIPS ↓Flow EPE ↓
Existing H1, without recovery0.23610.75050.017490.1966
Existing H1, with recovery0.24640.72600.019730.2039
Retrained H10.25230.65750.024400.2274
Reviewer_1k4FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Rectified Style Flow (RSF), a framework for real-time streaming video stylization. Starting from partially corrupted source latents, RSF combines style-specific LoRAs, full-parameter causal adaptation, one-step distribution matching distillation, and generated-history rollout fine-tuning. An additional residual LoRA adapts previously unseen styles to the distilled backbone. The paper reports high inference throughput and favorable source-relative motion consistency, while acknowledging trade-offs between stylization strength and content preservation.

Critical strengths
  • Practical potential for streaming applications. The reported throughput of 101.73 FPS on an H100 and 36.3 FPS on an RTX 5090 is compelling. One-step inference with bounded historical context addresses an important deployment constraint, provided that the reported quality is maintained under the same optimized configuration.
  • Useful evidence for generated-history adaptation. Results shows that rollout fine-tuning reduces tLPIPS from 0.02776 to 0.0179 and Flow EPE from 0.2396 to 0.2093, with nearly unchanged CLIP-T. This supports the usefulness of the rollout stage, although relative improvement does not establish that the remaining temporal artifacts are negligible.
Critical weaknesses
  1. Temporal Stability: The supplementary popart_long.mp4 exhibits repeated fluctuations in local textures and object details, including intervals where the source changes very little. This weakens the claim of temporally stable streaming stylization. The quantitative evaluation primarily covers the first 81 frames and therefore provides limited evidence for longer-term stability. The authors should explain these artifacts and provide evidence that the final model maintains temporal consistency beyond short clips.
  2. Content and Motion Preservation: The supplementary results reveal failures beyond ordinary appearance changes. In case2.mp4, the hanging glass bells lose or alter their structural details. In case1.mp4, the passing vehicle visible through the window is not preserved, and the right subject’s facial-expression changes are not faithfully reproduced. These observations weaken the claims of preserving source content and motion. Global CLIP-I and DINOv2 similarity may overlook such local failures; the authors should explain how their evaluation captures object disappearance, structural distortion, and expression mismatch.
  3. Novelty of Real-Time Acceleration: The reported real-time system combines established techniques, including one-step DMD, causal generation with bounded KV caching, lightweight video codecs, and FP8/TensorRT optimization. The paper does not sufficiently distinguish its methodological contribution to acceleration from this engineering integration. Comparisons using different backbones, codecs, precisions, and resolutions cannot isolate the source of the speed advantage. The authors should identify and experimentally substantiate the additional benefit of their proposed design under matched conditions.
  4. Validation of Rectified Style Flow: Table 4 evaluates reconstruction rather than the final one-step stylization task. Consequently, it does not establish that the proposed transport path improves the style–fidelity trade-off or makes distillation more effective than conventional partial-noise source editing. The authors should compare these alternatives using the same backbone, training data, and distillation budget, and evaluate the corruption-level sweep on actual stylization. The current evidence does not isolate the contribution of the central RSF formulation.
  5. Style–Dynamics Separation and Generalization: Freezing style LoRAs while updating the underlying backbone does not by itself establish a separation between style and streaming dynamics. The paper lacks a controlled comparison with joint backbone/style adaptation, while unseen-style recovery is demonstrated on only one style. Moreover, recovery improves CLIP-T at the expense of content correspondence and source-relative temporal metrics. The authors should clarify which aspect of compatibility is restored and provide evidence that the proposed modular design offers advantages over simpler adaptation strategies.
  6. Alignment of Quality and Efficiency Results: Appendix A describes the native Wan VAE, whereas the highest-throughput deployment uses TAE-Wan and mixed-FP8 inference. It is unclear whether Table 1’s quality scores correspond to the same checkpoint and deployment configuration as the reported 101.73 FPS. Without this alignment, the simultaneous achievement of the reported quality and speed remains unestablished. The authors should report quality under the optimized backend and isolate the effects of codec replacement and quantization. Especially TAE-Wan has clearly reconstruction quality drop compared with origin WAN vae decoer.
Questions for authors

See weakness.

1 comment

1. Long-horizon flicker and local content failures.

The examples you identified expose two distinct limitations. In popart_long.mp4, local textures fluctuate even where the source changes little. In case1.mp4 and case2.mp4, the passing vehicle, facial expression, and hanging glass details are not faithfully preserved. A high whole-frame embedding similarity cannot rule out any of these failures, and low flow error does not establish that an object remains recognizable. In our matched rollout ablation, Stage 4 rollout finetuning reduces warped LPIPS from 0.0278 to 0.0185 and flow EPE from 0.2396 to 0.2093 with little change in CLIP-T. This supports a relative reduction in flicker (under H1 settings). However, the stylization process and limited history horizon still introduce flicker, we will add a discussion on this limitation in the paper.

2. Which acceleration comes from RSF, and does the fastest configuration retain the reported quality?

Our fastest reported deployment uses H1, TAE, and mixed FP8. The main quality benchmark also uses this configuration.

RSF’s main algorithmic speedup is that the distilled student applies the selected style in one denoising step per new latent, instead of the teacher’s 25 steps (or generally 4 steps from DMD). It also needs only one previous latent (H1) to stabilize the stream: at fixed weights, retaining that latent rather than no history reduces warped LPIPS from 0.0581 to 0.0185. Which makes RSF being lightweight in both TOPS and device memory, and suitable for deployment on customer level device.

3. Does the evidence isolate the benefit of the proposed flow?

Here is the comparison results of SDEdit + Style Lora (the initial training start at s comes from transferred domain) against RSF + Style Lora (the initial training start at s comes from original domain). RSF wins is the actual clips that RSF achieves higher CLIP-T against SDEdit.

StyleSDEdit CLIP-T ↑RSF CLIP-T ↑RSF − SDEditSDEdit DINO-I ↑RSF DINO-I ↑RSF wins
Van Gogh0.20660.2783+0.07180.97250.712947/47
Flat Vector Illustration0.15990.2203+0.06040.97210.787346/47
American Cartoon0.19030.2608+0.07060.97230.714547/47
3D Chibi0.19530.2376+0.04230.97250.846545/47
Overall0.18800.2493+0.06130.97230.7653185/188

The results show that RSF is effective for the degree of stylization.

4. Does freezing the style bank actually separate style from dynamics?

Freezing the style bank does not guarantee that style and dynamics are disentangled. We tested its practical effect by training two AR teachers on the same trajectories, changing only whether the style LoRAs were frozen or updated jointly with the backbone. On test videos, freezing yielded higher CLIP-T for each of the four styles (mean 0.2479 vs. 0.2404). Joint training, however, better preserved source content (DINO-I 0.7504 vs. 0.7236). We therefore interpret freezing as a way to preserve the style control learned in Stage 1, not as evidence that the backbone learns dynamics alone.

5. What does recovery restore on unseen styles?

Recovery partially restores target-style alignment when a new style LoRA is attached to the frozen streaming backbone. For each unseen style (Van Gogh, Flat Vector Illustration, American Cartoon, and 3D Chibi), we kept both the H1 backbone and the style LoRA fixed and trained only a rank-64 recovery adapter for 100 updates on 10 bidirectional teacher trajectories. On the same 47 held-out videos, mean CLIP-T rose from 0.2361 to 0.2464, compared with 0.2493 for the bidirectional teacher. The gain came with a trade-off: mean DINO-I fell from 0.7505 to 0.7260, while warped LPIPS rose from 0.0175 to 0.0197 and flow EPE from 0.1966 to 0.2039. Thus, recovery narrows the average style-alignment gap to the teacher, with a cost of fidelity or temporal stability.

Reviewer_kKXMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

One-step causal video stylization. The model transports a partially noised source (s = 0.4) to the target style instead of starting from pure noise. Four stages: per-style LoRAs on Wan2.1-1.3B, LoRA-aware causal forcing against a bidirectional teacher, DMD distillation to one step, and rollout fine-tuning on self-generated history.

Critical strengths
  1. this paper aims to solve the interesting problem and realtime streamining video stylization

  2. This paper is easy to understand

  3. The proposed method is clear and simple

Critical weaknesses
  1. Novelty is very limited, this work is a simple combination: an SDEdit-style intermediate start, LoRA styles, causal forcing, DMD, rollout fine-tuning. "Rectified style flow" is essentially SDEdit with a rescaled time axis.

  2. The quality seems not clear sota and the evaluation is only conducted on a simple setting.

  3. Stage 2 trains on teacher outputs rather than real stylized video, so teacher errors are baked in.

Questions for authors
  1. Is a single s = 0.4 right for all styles? would a per-style fix the "too close to the input" limitation?
2 comments

1. Is RSF just SDEdit with a rescaled time axis?

Our distinction from SDEdit is what is learned, not where denoising starts. Both methods can start from a partially corrupted source. SDEdit applies an existing denoising field to that input. RSF instead learns a velocity field from paired source and stylized target videos. Let XX be the source latent, YY its stylized target, and Ws=(1−s)X+sϵW_s=(1-s)X+s\epsilon. On the path Zu=(1−u)Y+uWsZ_u=(1-u)Y+uW_s, with native model time σ=su\sigma=su, the supervised velocity is Ws−Ys=ϵ−Y+1−ss(X−Y).\frac{W_s-Y}{s}=\epsilon-Y+\frac{1-s}{s}(X-Y). When Y=XY=X, this is exactly the native partial-noise reconstruction target. For stylization, the additional term depends on the paired appearance change; rescaling time alone does not supply it.Our contribution is carrying this learned style transformation into a bounded-history streaming model that can still use external style LoRAs.

2. Is a single corruption level appropriate for every style?

No. We used a common ss to compare styles under the same operating conditions, not because it is optimal for every style. We now evaluate separately trained style LoRAs at different ss on the same held-out videos, crops, and noise seeds. Lower corruption best preserves source features (e.g. DINO-I); higher corruption gives the strongest average style alignment (e.g. CLIP-T). Our chosen setting gives the lowest source-relative flow error and warped perceptual error. A per-style choice is reasonable, but should be selected on validation data. We would not expect increasing ss alone to resolve failures that require substantial geometric changes.

CLIP-T:

Styles = 0.2s = 0.3s = 0.4s = 0.5s = 0.6
Van Gogh0.279500.281930.278350.282840.28720
Flat Vector Illustration0.220570.224730.220290.220530.21929
American Cartoon0.264560.262010.260840.261490.26356
3D Chibi0.226370.232170.237550.235540.23853
Mean across styles0.247750.250210.249260.250100.25215

DINO-I:

Styles = 0.2s = 0.3s = 0.4s = 0.5s = 0.6
Van Gogh0.691320.692180.712940.711900.68183
Flat Vector Illustration0.792290.773930.787300.761140.74788
American Cartoon0.715060.709010.714520.706810.68915
3D Chibi0.887090.862020.846510.839330.81810
Mean across styles0.771440.759290.765320.754790.73424

3. Are teacher errors inherited, and what quality is lost?

Yes. We use teacher outputs because the causal model should reproduce what the selected LoRA actually generates, rather than alter its appearance to match a different target distribution (since that will cause the model to learn the altered distribution first, instead of learning the causal frames). This also means that a missing object or an incorrect expression in the teacher output is not corrected by design. On identical held-out inputs, mean CLIP-T is 0.2493 for the bidirectional teacher, 0.2480 for the AR teacher, and 0.2543 for the one-step student. From the AR teacher to the student, DINO-I falls from 0.7229 to 0.6275 and warped LPIPS rises from 0.0183 to 0.0274, despite the higher mean CLIP-T. The main cost of distillation on this benchmark is therefore content and temporal fidelity.

Thanks for the response. The reply explains the formulation but does not establish why the formulation represents enough novelty over existing flow-matching and video stylization approaches. For corruption level, the selection can be empirical and for different network seems different s can be the best pick. This limit the generalization ability. The authors also suggest teacher errors are inherit. Based on these concerns, I will remain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.