- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a three-stage training framework for factor-aware cinematic video generation, enabling fine-grained control over character, scene, action, and camera through structured MLLM representations. Its key technical contribution, Factor Span Replacement, replaces the latent span corresponding to a target factor with that from an edited description, explicitly training the DiT to respond to localized semantic changes. The authors construct a factor-aware dataset with 100K source–target video pairs, balanced across the four factors, and evaluate the proposed framework through experiments and ablation studies.
Critical strengths
- The proposed method is conceptually simple and easy to implement.
- The paper is clear written and easy to follow.
- The provided samples from the supplementary material clearly demonstrate the performance of the method.
Critical weaknesses
-
The main technical contribution of this paper is Factor Span Replacement in Stage 2, whose success relies heavily on the structure of the MLLM. This work uses UniVideo as the backbone, with Qwen2.5-VL as the MLLM. However, the authors do not thoroughly discuss the role of the MLLM or experimentally evaluate their method with different MLLMs, which limits the generality of the proposed approach.
-
The authors conduct ablation studies within each stage, but I am more interested in the individual contributions of each stage and the rationale behind their ordering. For example, after fine-tuning the MLLM in Stage 3 and updating its features, why not further fine-tune the DiT backbone? Conversely, why does the pretrained DiT need to be updated first? These design choices remain unclear and require further justification.
-
The three-stage training procedure appears rather complicated. In particular, how do the authors determine when to proceed to the next stage, given that generation losses are typically noisy and unreliable [1, 2]?
-
The paper lacks sufficient discussion and comparison between MLLM-based video generation methods and conventional generation methods, such as Wan2.1 (VACE) and CogVideo. Are MLLM-based approaches particularly effective at factor-aware control compared with conventional approaches?
-
The factor-aware dataset contains 25K pairs per factor. Even if these pairs are of high quality, they may not cover the full diversity of characters, scenes, actions, and camera movements. Consequently, the model may generalize poorly to unseen combinations or more complex editing scenarios.
References
[1] Table Target Field for Reduced Variance Score Estimation in Diffusion Models.
[2] Variance Reduction of Diffusion Model’s Gradients with Taylor Approximation-Based Control Variate.
Questions for authors
Please see weaknesses.
7 comments
Response to Reviewer fSc4 (Part 2)
(W3) 3. Training Complexity and Stage Transitions
3-a. Capability-driven stages. The stages address three bottlenecks identified through progressive probing: insufficient MLLM feature use, weak factor control, and limited temporal control from sparse inputs. The required capabilities, rather than a universal three-stage schedule, guide training. For example, MiniMax H3 already supports strong MLLM-only generation and can skip Stage I.
3-b. Explicit transition criteria. We evaluate every 1K steps on a fixed, independent validation set, holding inputs, sampling seeds, and inference settings constant. Stages I and III monitor Motion Order and both camera errors, while Stage II monitors editing ∩-Acc. A stage ends when none of its monitored metrics improves by at least 1% relative to its respective previous best for three consecutive evaluations, with improvement meaning higher accuracy or lower error. Stages I and II then advance, while Stage III terminates training. The reported 10K/15K/10K stage lengths were established empirically through repeated UniVideo trials using this criterion, rather than prescribed universally across backbones. Transitions therefore depend on sustained task performance, not fluctuations in generation loss.
(W4) 4. MLLM-Based vs. Conventional Video Generation
4-a. Why use MLLMs? Cinematic control requires understanding what to change, what to preserve, and how factors interact. MLLMs offer a unified interface for interpreting textual instructions and visual references. FAPL turns this representational potential into selective control, rather than assuming that MLLM conditioning alone is sufficient.
Table R12. Comparison of conditioning paradigms.
| Dimension | MLLM-based models | Text-encoder-based models |
|---|---|---|
| Factor representation | Contextual modeling of relationships among cinematic factors | Text semantics, with factor coordination relying more on generator training |
| Instruction understanding | Joint interpretation of text and visual references | Text encoding with separate visual pathways and additional cross-modal integration |
4-b. Comparison with VACE. Under matched text-only inputs and evaluation protocols, FAPL outperforms VACE by 16 percentage points in Motion Order and 19 in editing accuracy, with lower camera errors. Editing uses generated Before/After pairs with identical initial noise within each pair. These results support FAPL’s advantage on the evaluated tasks. The matched UniVideo ablation further shows that factor-span replacement contributes beyond MLLM conditioning itself.
Table R13. Comparison under matched task inputs.
| Method | Semantic conditioning | Motion Order (%) ↑ | Trans. Error ↓ | Rot. Error (rad) ↓ | ∩-Acc (%) ↑ |
|---|---|---|---|---|---|
| VACE (Wan2.1-14B) | Text encoder | 45 | 0.0812 | 1.52 | 39 |
| UniVideo (13B) | MLLM | 48 | 0.0763 | 1.54 | 42 |
| UniVideo (13B) + Ours | MLLM | 61 | 0.0586 | 1.02 | 58 |
Response to Reviewer fSc4 (Part 3)
(W5) 5. Generalization and Scope
5-a. Learning control, not enumerating combinations. The 25K pairs per factor teach what to change and what to preserve, building on pretrained knowledge rather than attempting to cover every cinematic combination.
5-b. Generalization across common cinematic configurations. As detailed in Appendices C and E, source-video groups are split before pair construction, with no training/test source overlap. On this held-out set, span replacement improves editing accuracy from 48% to 57% over matched no-swapping training. These results support transfer to unseen videos across the common cinematic configurations represented in the benchmark, beyond memorizing training pairs.
5-c. Extending complex control. More challenging factor combinations and complex multi-character interactions remain to be systematically evaluated. These limits do not negate the demonstrated contribution: a training mechanism that improves selective control under matched data and training budgets. As discussed in Appendix G, stronger foundation models and interaction-aware supervision offer a path toward broader and more complex control.
Response to Reviewer fSc4 (Part 1)
Thank you for your careful review and constructive questions. We address your concerns with cross-model validation on MiniMax H3, a comparison with VACE, and explicit stage-transition criteria, while clarifying the training rationale and the scope of generalization.
(W1) 1. Cross-Model Generalization of Factor-Span Replacement
1-a. Role of the MLLM. The MLLM provides contextualized representations of character, scene, action, and camera. Factor-span replacement operates after the connector, using structured descriptions to locate intervention spans. It requires identifiable factor spans in the projected sequence, not Qwen2.5-VL-specific architecture or fully disentangled features.
1-b. Validation on MiniMax H3. Since ICLR submission, we have adapted the method to MiniMax H3, which differs from UniVideo in both its MLLM and diffusion backbone (Table R11). Within each model, swapping and no-swapping variants share the same initialization, paired data, and training budget. Editing accuracy improves from 48% to 57% on UniVideo and 58% to 64% on H3, supporting transfer across video backbones and MLLM generations and scales.
Table R11. Factor-span replacement across different MLLM–DiT combinations. Training conditions are matched within each model.
(W2) 2. Stage Contributions and Training Order
2-a. Stage roles and ordering rationale. Stage I teaches the DiT to decode MLLM features. Stage II builds on this interface to learn selective factor control. Stage III then uses the trained, frozen DiT to improve the MLLM and connector’s representations. The progression is first learning how to use representations, then learning what representations to produce, rather than combining three independent techniques.
2-b. Why not update the DiT again after Stage III? MLLM and connector updates are supervised through the fixed DiT, so the new representations directly learn to work with the existing decoder. Their evolution therefore does not inherently require another DiT adaptation phase. Freezing the DiT also focuses optimization on upstream representations. Further alternating updates remain a possible extension.
2-c. Why start with Stage I? Both factor interventions and upstream representation learning require a DiT that can reliably decode MLLM latents. Stage I establishes this prerequisite, avoiding simultaneous optimization of whether the representations contain useful information and whether the generator can decode it.