acceptodds
Under review as a conference paper at ICLR 2027

Efficient Conditional Modeling for Controllable Video Generation

Abstract

Despite substantial recent progress in controllable video generation with various conditioning signals, current frameworks suffer from severe computational inefficiency when applied to video diffusion transformers. The issue becomes particularly acute when the conditioning signal itself is a video, resulting in excessively long contextual sequences. To address this limitation, we propose EasyVideoControl, an efficient framework for conditional video generation with DiTs (diffusion transformers). EasyVideoControl introduces two core components. First, we present the Asymmetric Shared Causal Mixture-of-Transformers structure, which incorporates a frozen copy of the base diffusion transformer that shares self-attention layers with the backbone under a causal attention scheme. By training only low-rank adaptation (LoRA) parameters in this unidirectional conditioning branch, the model maximizes reuse of pretrained knowledge while minimizing training computational overhead. This design enables the backbone to be modulated by the conditional branch without reciprocal dependency, supporting KV Cache for efficient inference. Second, we introduce Reconstructive Conditioning Compression and Recovery, a lightweight encoder–decoder mechanism that compresses and reconstructs conditional tokens. This approach greatly reduces context length while preserving fine-grained conditional details through a reconstruction-based objective. Extensive experiments across multiple conditional video generation tasks show that EasyVideoControl achieves comparable or superior performance while being significantly more efficient than existing video control methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient conditioning, Video generation, Diffusion transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2G6sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors target the cost of conditioning video diffusion transformers on dense video signals: depth, pose, scribbles, or a reference video supplying camera motion, style or effects. EasyVideoControl has two parts. ASC-MoT adds a conditional branch, a frozen copy of the backbone with LoRA, sharing self-attention under a mask that lets video tokens attend to condition tokens but not the reverse; with the branch's timestep modulation set to zero, its keys and values are computed once and cached across denoising steps. RCC compresses the condition tokens 8× using local attention over 2×2×2 groups, rescales their positions onto the video grid, and adds a reconstruction loss. On Wan2.2-TI2V-5B the method adds 5–8% latency over unconditioned generation; on VACE's own Wan2.1-1.3B backbone it runs 4.9× faster at a tenth of the trainable parameters.

Critical strengths
  • The efficiency result is large and controlled. Table 2 retrains the method on VACE's backbone: latency falls from 51.0 s to 10.5 s, trainable parameters from 100% to 10%, and depth MAE from 0.026 to 0.023, while the seven VBench dimensions split three apiece with one tie. Dense conditioning becomes nearly free: 17.7 s against 16.9 s unconditioned for T2V.
  • The component ablations are matched and informative. Causal masking accounts for most of the saving and compression for the rest; the reconstruction loss lowers MAE at every compression ratio; and learned compression beats average pooling on MAE at identical latency.
  • The authors anticipate the backbone-fairness objection themselves, and the limitations section is candid about the 480p, 49-frame scope and the memory wall beyond it.
Critical weaknesses
  • Control adherence is quantified for depth only, although the framework advertises pose, scribble, camera, style and effect control. The semantic I2V half has no adherence metric at all: its VBench scores do not measure whether the reference's motion or style actually transferred.
  • Imaging quality is the method's consistent weak spot: lowest in Table 1, 3.6 points behind VACE on the shared backbone, and below all eight of its own T2V ablation rows.
  • RCC's motivating regime is untested. At the one configuration evaluated it contributes 3.0 s and 2.5 GB, and the paper itself notes that beyond 8× the backbone dominates (l. 796–798), so its value for long, high-resolution video is argued rather than shown.
Questions for authors
  1. Can you measure adherence for at least one further structural control (pose or scribble) and for the semantic I2V tasks, for example camera-trajectory error or a style-similarity score against the reference?
  2. What is the unconditioned Wan2.1-1.3B latency under the Table 2 settings, and do VACE and Ours (1.3B) use the same sampling steps, guidance and hardware?
  3. Can you report RCC's effect at one longer or higher-resolution configuration, together with the training time and memory it saves?
  4. Sec. 3.1.1 places LoRA in the conditional FFN while Fig. 2 places it at the conditional QKV projections. Which is implemented?
  5. Is a single model trained for all control types, and is the VAP benchmark disjoint from the VAP-data training videos?
Reviewer_z8MROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EasyVideoControl, combining asymmetric attention, condition KV caching, and learnable spatiotemporal compression for controllable video generation. Experiments report improved efficiency while maintaining competitive generation quality and controllability.

Critical strengths

The paper combines asymmetric attention, condition KV caching, and learnable spatiotemporal condition compression for structural control and reference-video semantic control, reducing both the number of condition tokens and repeated computation across denoising steps.

Critical weaknesses
  1. The novelty is insufficiently substantiated. Similar implementations of asymmetric attention, cross-step condition KV caching, and coordinate rescaling after compression already appear in EasyControl and OminiControl2. FullDiT2 also applies asymmetric conditioning interactions and KV caching to video generation.
  2. The paper does not sufficiently discuss FlexTraj, Video2LoRA, and FlashMotion, and the experiments lack comparisons with these methods. Although some methods use different backbones, the authors should clarify the scope of comparability and provide controlled comparisons of the mechanisms or evaluations on shared tasks to further support the claimed efficiency and quality advantages.
Questions for authors
  1. The four quality metrics reported for VAP in Table 1 are the same as the results of its CogVideoX-I2V-5B version in its original paper. VAP provides a Wan2.1-I2V-14B version. Have the authors evaluated this version?
  2. The paper describes VACE as requiring full-backbone fine-tuning and reports a trainable-parameter ratio of 100%. However, VACE provides a variant that freezes the backbone and trains only the Context Embedder and Context Blocks. Was this variant considered in the experiments, and which variant do the reported results correspond to?
  3. At the same 8× compression ratio and without reconstruction supervision for either method, RCC shows marginal or inconsistent advantages over average pooling on the remaining metrics. Could the authors provide additional comparison results to demonstrate what information RCC preserves in terms of structural details and temporal consistency, and validate these benefits using corresponding control metrics?
Reviewer_pDtTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the high computational cost of controllable video generation when dense video conditions are concatenated with noisy video tokens and repeatedly processed at every denoising step.

Critical strengths
  1. The proposed architecture directly targets the two main sources of conditioning overhead. Causal separation removes repeated condition encoding, while RCC reduces the number of condition tokens participating in backbone attention.
  2. The efficiency improvements are substantial. EasyVideoControl requires 17.7 seconds for structural T2V and 19.4 seconds for semantic I2V, compared with 51.0 seconds for VACE and 317 seconds for VAP.
  3. The framework supports several condition types within one design.
  4. The ablation study is useful and reasonably broad.
Critical weaknesses
  1. In the case of RCC, the authors reduce the number of tokens by some local attention/aggregation and then pool it to 2x2x2. Why do they not use something like mixture-of-depth (like DiffCR, Zhang et al., CVPR 2025) to let the model select which condition tokens it wants to use? Would that yield better results, or even more compression?
  2. The main comparison table mixes substantially different backbones, training pipelines, and model sizes. EasyVideoControl primarily uses Wan2.2-TI2V-5B, while VACE uses Wan2.1-1.3B and several other methods use older U-Net backbones. The same-backbone comparison is limited to VACE and one depth-control setting. In that comparison, EasyVideoControl is much faster and has slightly better MAE, but it is worse on several VBench quality dimensions.
  3. The evaluation relies on automated metrics and qualitative examples. The paper does not report a human preference study.
  4. Direct control-fidelity evaluation is incomplete. The only explicit structural control metric is depth MAE, even though the method also claims pose and scribble control. Semantic I2V evaluation reports generic video quality metrics, but does not directly measure camera trajectory adherence, style similarity, costume preservation, or effect accuracy.
  5. The authors have only provided very few of their generated results in the supplementary. They have not provided visual comparisons with other baselines/methods.
Questions for authors
  1. Can the authors provide more same-backbone comparisons using identical VAEs, training data, resolution, and sampling settings?
  2. Can the authors report direct control-adherence metrics for pose, scribble, camera motion, style transfer, costume changes, and visual effects? A human preference study evaluating both control fidelity and overall video quality would be especially valuable for semantic controls.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.