acceptodds
Under review as a conference paper at ICLR 2027

Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

Abstract

Cinematic video generation requires both temporal inference from limited inputs and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, which replacing the corresponding projected latent span trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate improved factor-aware cinematic instruction following over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.

Primary area:
generative models
Keywords:
Cinematic Video Generation, Predictive Learning, Multimodal Large Language Model, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_fSc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a three-stage training framework for factor-aware cinematic video generation, enabling fine-grained control over character, scene, action, and camera through structured MLLM representations. Its key technical contribution, Factor Span Replacement, replaces the latent span corresponding to a target factor with that from an edited description, explicitly training the DiT to respond to localized semantic changes. The authors construct a factor-aware dataset with 100K source–target video pairs, balanced across the four factors, and evaluate the proposed framework through experiments and ablation studies.

Critical strengths
  1. The proposed method is conceptually simple and easy to implement.
  2. The paper is clear written and easy to follow.
  3. The provided samples from the supplementary material clearly demonstrate the performance of the method.
Critical weaknesses
  1. The main technical contribution of this paper is Factor Span Replacement in Stage 2, whose success relies heavily on the structure of the MLLM. This work uses UniVideo as the backbone, with Qwen2.5-VL as the MLLM. However, the authors do not thoroughly discuss the role of the MLLM or experimentally evaluate their method with different MLLMs, which limits the generality of the proposed approach.

  2. The authors conduct ablation studies within each stage, but I am more interested in the individual contributions of each stage and the rationale behind their ordering. For example, after fine-tuning the MLLM in Stage 3 and updating its features, why not further fine-tune the DiT backbone? Conversely, why does the pretrained DiT need to be updated first? These design choices remain unclear and require further justification.

  3. The three-stage training procedure appears rather complicated. In particular, how do the authors determine when to proceed to the next stage, given that generation losses are typically noisy and unreliable [1, 2]?

  4. The paper lacks sufficient discussion and comparison between MLLM-based video generation methods and conventional generation methods, such as Wan2.1 (VACE) and CogVideo. Are MLLM-based approaches particularly effective at factor-aware control compared with conventional approaches?

  5. The factor-aware dataset contains 25K pairs per factor. Even if these pairs are of high quality, they may not cover the full diversity of characters, scenes, actions, and camera movements. Consequently, the model may generalize poorly to unseen combinations or more complex editing scenarios.

References

[1] Table Target Field for Reduced Variance Score Estimation in Diffusion Models.

[2] Variance Reduction of Diffusion Model’s Gradients with Taylor Approximation-Based Control Variate.

Questions for authors

Please see weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer fSc4 (Part 1)

Thank you for your careful review and constructive questions. We address your concerns with cross-model validation on MiniMax H3, a comparison with VACE, and explicit stage-transition criteria, while clarifying the training rationale and the scope of generalization.

(W1) 1. Cross-Model Generalization of Factor-Span Replacement

1-a. Role of the MLLM. The MLLM provides contextualized representations of character, scene, action, and camera. Factor-span replacement operates after the connector, using structured descriptions to locate intervention spans. It requires identifiable factor spans in the projected sequence, not Qwen2.5-VL-specific architecture or fully disentangled features.

1-b. Validation on MiniMax H3. Since ICLR submission, we have adapted the method to MiniMax H3, which differs from UniVideo in both its MLLM and diffusion backbone (Table R11). Within each model, swapping and no-swapping variants share the same initialization, paired data, and training budget. Editing accuracy improves from 48% to 57% on UniVideo and 58% to 64% on H3, supporting transfer across video backbones and MLLM generations and scales.

Table R11. Factor-span replacement across different MLLM–DiT combinations. Training conditions are matched within each model.

ModelMLLMDiT backboneSpan replacementMotion Order (%) ↑Trans. Error ↓Rot. Error (rad) ↓∩-Acc (%) ↑
UniVideoQwen2.5-VL-7BHunyuanVideo-T2V-13BNo540.06541.3048
UniVideoQwen2.5-VL-7BHunyuanVideo-T2V-13BYes530.06711.3257
MiniMax H3Qwen3-VL-32BH3-Omni-TransformerNo650.05360.9658
MiniMax H3Qwen3-VL-32BH3-Omni-TransformerYes660.05280.9364

(W2) 2. Stage Contributions and Training Order

2-a. Stage roles and ordering rationale. Stage I teaches the DiT to decode MLLM features. Stage II builds on this interface to learn selective factor control. Stage III then uses the trained, frozen DiT to improve the MLLM and connector’s representations. The progression is first learning how to use representations, then learning what representations to produce, rather than combining three independent techniques.

2-b. Why not update the DiT again after Stage III? MLLM and connector updates are supervised through the fixed DiT, so the new representations directly learn to work with the existing decoder. Their evolution therefore does not inherently require another DiT adaptation phase. Freezing the DiT also focuses optimization on upstream representations. Further alternating updates remain a possible extension.

2-c. Why start with Stage I? Both factor interventions and upstream representation learning require a DiT that can reliably decode MLLM latents. Stage I establishes this prerequisite, avoiding simultaneous optimization of whether the representations contain useful information and whether the generator can decode it.

AuthorsOpenReview ↗

Response to Reviewer fSc4 (Part 2)

(W3) 3. Training Complexity and Stage Transitions

3-a. Capability-driven stages. The stages address three bottlenecks identified through progressive probing: insufficient MLLM feature use, weak factor control, and limited temporal control from sparse inputs. The required capabilities, rather than a universal three-stage schedule, guide training. For example, MiniMax H3 already supports strong MLLM-only generation and can skip Stage I.

3-b. Explicit transition criteria. We evaluate every 1K steps on a fixed, independent validation set, holding inputs, sampling seeds, and inference settings constant. Stages I and III monitor Motion Order and both camera errors, while Stage II monitors editing ∩-Acc. A stage ends when none of its monitored metrics improves by at least 1% relative to its respective previous best for three consecutive evaluations, with improvement meaning higher accuracy or lower error. Stages I and II then advance, while Stage III terminates training. The reported 10K/15K/10K stage lengths were established empirically through repeated UniVideo trials using this criterion, rather than prescribed universally across backbones. Transitions therefore depend on sustained task performance, not fluctuations in generation loss.

(W4) 4. MLLM-Based vs. Conventional Video Generation

4-a. Why use MLLMs? Cinematic control requires understanding what to change, what to preserve, and how factors interact. MLLMs offer a unified interface for interpreting textual instructions and visual references. FAPL turns this representational potential into selective control, rather than assuming that MLLM conditioning alone is sufficient.

Table R12. Comparison of conditioning paradigms.

DimensionMLLM-based modelsText-encoder-based models
Factor representationContextual modeling of relationships among cinematic factorsText semantics, with factor coordination relying more on generator training
Instruction understandingJoint interpretation of text and visual referencesText encoding with separate visual pathways and additional cross-modal integration

4-b. Comparison with VACE. Under matched text-only inputs and evaluation protocols, FAPL outperforms VACE by 16 percentage points in Motion Order and 19 in editing accuracy, with lower camera errors. Editing uses generated Before/After pairs with identical initial noise within each pair. These results support FAPL’s advantage on the evaluated tasks. The matched UniVideo ablation further shows that factor-span replacement contributes beyond MLLM conditioning itself.

Table R13. Comparison under matched task inputs.

MethodSemantic conditioningMotion Order (%) ↑Trans. Error ↓Rot. Error (rad) ↓∩-Acc (%) ↑
VACE (Wan2.1-14B)Text encoder450.08121.5239
UniVideo (13B)MLLM480.07631.5442
UniVideo (13B) + OursMLLM610.05861.0258
AuthorsOpenReview ↗

Response to Reviewer fSc4 (Part 3)

(W5) 5. Generalization and Scope

5-a. Learning control, not enumerating combinations. The 25K pairs per factor teach what to change and what to preserve, building on pretrained knowledge rather than attempting to cover every cinematic combination.

5-b. Generalization across common cinematic configurations. As detailed in Appendices C and E, source-video groups are split before pair construction, with no training/test source overlap. On this held-out set, span replacement improves editing accuracy from 48% to 57% over matched no-swapping training. These results support transfer to unseen videos across the common cinematic configurations represented in the benchmark, beyond memorizing training pairs.

5-c. Extending complex control. More challenging factor combinations and complex multi-character interactions remain to be systematically evaluated. These limits do not negate the demonstrated contribution: a training mechanism that improves selective control under matched data and training budgets. As discussed in Appendix G, stronger foundation models and interaction-aware supervision offer a path toward broader and more complex control.

Reviewer_o1MROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Factor-Aware Predictive Learning (FAPL) for cinematic video generation, targeting generation from limited inputs and selective control over character, scene, action, and camera. The method uses three training stages. Stage 1 freezes the MLLM and connector while adapting the DiT with MLLM latents as its only content conditions. Stage 2 trains the DiT using factor-span replacement between paired descriptions, with paired videos supervising the intended change and preservation of other factors. Stage 3 freezes the trained DiT and updates the MLLM and connector through flow-matching supervision, mixing text-only prediction with text-plus-complete-video reconstruction.

The full method is evaluated on UniVideo across text-only generation, factor-aware editing, and multi-shot generation. Stage-wise ablations, additional qualitative backbone comparisons, and a blinded human study provide complementary validation.

Critical strengths
  1. A concrete and empirically supported mechanism for selective control.
    Factor-span replacement connects a localized conditioning intervention to a paired visual change while retaining the remaining conditioning tokens. Importantly, Table 2 compares swapping against training without swapping under matched paired data and optimization budgets, improving editing accuracy from 48% to 57%. This is meaningful evidence that the way paired supervision is used matters beyond simply adding editing examples.

  2. Complementary improvements supported by stage-wise controls.
    The full method improves Motion Order from 48% to 61% and editing accuracy from 42% to 58% over pretrained UniVideo. The Stage 3 comparison against text-only training supports the benefit of including complete-video observations during training for text-only generation. Evaluation on ShotVerse-Bench and human judgments additionally supports improvements in cross-shot consistency and selective editing. The paper appropriately notes that Stage II primarily benefits editing rather than uniformly improving all metrics.

Critical weaknesses
  1. The reported configurations and evaluation aggregation are not sufficiently explicit.
    Section 3.5 and the appendices distinguish MLLM-only inference from inference that additionally uses native conditioning pathways, but Tables 1–2 do not clearly identify the configuration used for each task and ablation. Appendix E also describes both target-change and non-target-preservation questions, while the reported intersection-accuracy formula only explicitly combines yes/no and multiple-choice outcomes. These details materially affect the interpretation and reproducibility of the central selective-control results.

  2. The mechanistic interpretation is stronger than what the current comparisons isolate.
    Removing native pathways and subsequently adapting the generator demonstrates that MLLM-only conditioning can be made effective, but does not by itself establish the degree of MLLM neglect under the original conditioning configuration. Similarly, Stage 3 jointly updates the MLLM and connector, so its gains establish better generation-compatible conditioning rather than independently demonstrating improved general temporal reasoning in the MLLM. These limitations do not negate the practical contribution, but the corresponding claims should be scoped carefully.

Questions for authors

These questions request clarification of the existing implementation and results rather than new large-scale experiments.

  1. Inference configurations:
    Please specify, for each task in Table 1 and each configuration in Table 2, which conditioning pathways are enabled. For tasks with visual references, are references processed through the MLLM, the VAE-reference pathway, or both? Are these settings held fixed across the relevant ablations?

  2. Stage 2 control:
    Please clarify how the no-swapping control in Table 2 uses the paired examples. Does it condition on each complete description and supervise its corresponding video, using the same videos, sampling or directional accounting, and optimization budget as the swapping condition?

  3. Editing-score aggregation:
    How are the target-change question and the three non-target-preservation questions combined into the reported intersection accuracy? Does a case need to pass all factor checks, or are question-level results aggregated differently? Please also identify the evaluator used.

  4. Camera evaluation:
    Please specify the trajectory alignment and scale normalization used for Translation Error, the temporal correspondence used for both camera metrics, and how trajectory-estimation failures are handled.

-------------------Post rebuttal Edit------------------------

I appreciate the author for the detailed response and revisions. My main concerns have been alleviated, and I will maintain my initial positive score accordingly.

5 comments
AuthorsOpenReview ↗

Response to Reviewer o1MR (Part 1)

Thank you for recognizing the selective-control mechanism and the complementary experimental evidence. We clarify the evaluation configurations, mechanistic evidence, and Stage II control below.

(W1,Q1) 1. Inference Configurations

Sec. 3.5 and Appendices D–E describe the inference interfaces and task inputs. Table R4 maps these settings to the reported results. Conditioning pathways remain fixed across all Table 2 ablations.

Table R4. Conditioning pathways used in the quantitative evaluations.

ResultsTaskFAPL / UniVideoOmni-Video 2Visual references
Table 1Sparse-to-videoMM + TNone
Table 1Factor-aware editingMM + TNone
Table 1Multi-shot generationM + VM + T + VShared character and scene references
Table 2All ablationsMN/ANone

M denotes MLLM features through the connector, T native text-encoder features, and V VAE-reference features. For multi-shot generation, visual references enter both the MLLM and VAE pathways. Editing uses text only. Baselines encode the complete edited description, whereas FAPL replaces only the target-factor span after the connector.

(W2) 2. Mechanistic Clarification: Stage I and MLLM Feature Bypass Mitigation

2-a. Clarification. Stage I targets bypass by strengthening the DiT’s decoding of MLLM features, rather than merely accommodating an input format. With the MLLM and connector frozen and alternative pathways removed, the DiT must learn to extract generation-relevant information from fixed MLLM latents. Our full-conditioning experiments show that this learned capability translates into stronger MLLM instruction influence when competing pathways are restored, supporting bypass mitigation in the tested settings.

2-b. Controlled experiment. We compare checkpoints before and after Stage I with all native pathways enabled. The MLLM specifies factor A, while visual references and, where present, the text encoder specify B. Non-target factors, sampling seeds, and inference settings remain fixed. MLLM instruction alignment increases in both models, and Omni-Video 2 shifts from favoring native text to favoring the MLLM instruction. This supports stronger MLLM influence under conflicting full conditioning. The decreasing VAE MSE values do not establish reduced visual-reference reliance.

Table R2. Output alignment under conflicting full conditioning.

ModelStage ICLIP similarity to MLLM text ↑CLIP similarity to native text ↓VAE latent MSE to reference ↓
Omni-Video 2 (14B)Before0.17480.24030.0502
Omni-Video 2 (14B)After0.25730.20690.0201
UniVideo (13B)Before0.2796N/A0.0142
UniVideo (13B)After0.3356N/A0.0086
AuthorsOpenReview ↗

Response to Reviewer o1MR (Part 2)

(W2) 3. Mechanistic Clarification: How Stage III Strengthens Temporal Control

Stage III trains the MLLM and connector to convey temporal relationships through a fixed DiT. Motion Order evaluates action sequencing, not merely action presence. With the DiT frozen, mixed-observation training improves this score from 57% to 61% over text-only training (paper Table 2). The connector-only control reported in our response to MM99 (Table R3) reaches 54%, versus 61% with MLLM updates under matched settings. These results support improved MLLM conditioning for temporally ordered generation, rather than independently establishing broader temporal reasoning abilities.

(Q2) 4. Stage II: No-Swapping Control

Yes. In the no-swapping control, the DiT receives the encoded complete description of video A as its condition and uses video A as the supervision target and the same applies to video B. Each description includes all four factors—character, scene, action, and camera—and is encoded through the frozen MLLM and connector without span replacement.

Both variants use the same video pairs and sampling procedure, with both videos in each pair contributing supervision and matched target counts. They start from the same Stage I checkpoint and update only the DiT for 15K steps, using identical effective batch sizes and optimizer settings. Thus, the comparison changes only whether factor-span replacement is applied to the conditioning representations, while keeping the video supervision and training budget matched.

AuthorsOpenReview ↗

Response to Reviewer o1MR (Part 3)

(Q3) 5. Editing-Score Aggregation

A case succeeds only if the target change and all three non-target preservation checks pass. We use Qwen3-VL-32B as the evaluator. Each check requires both FiVE’s yes/no and multiple-choice judgments to pass.

Let ci,0c_{i,0} denote target-change success and ci,1,ci,2,ci,3c_{i,1},c_{i,2},c_{i,3} denote non-target preservation, with each value binary. The reported score is

=\frac{100}{N}\sum_{i=1}^{N}\prod_{k=0}^{3}c_{i,k}.$$ Thus, this is a **case-level joint success rate**, not an average of question-level accuracies. Any failed check makes the case unsuccessful, with no partial credit. # (Q4) 6. Camera Evaluation **6-a. Alignment and scale.** We recover camera trajectories from generated and reference videos using frozen π³ and express each trajectory relative to its own first frame. We align the entire predicted trajectory to the reference using a single Umeyama Sim(3) transformation, accounting for scale, rotation, and translation. The alignment rotation is also applied to the predicted camera orientations. **6-b. Error definitions and aggregation.** For each matched frame, Translation Error is the Euclidean distance between the aligned predicted and reference camera positions, and Rotation Error is the geodesic angular difference between their orientations, measured in radians. We first average frame errors within each video, then average the resulting video-level scores with equal weight across valid videos. Scale ambiguity is handled through trajectory-level alignment; we do not divide errors by individual reference displacements. **6-c. Temporal correspondence and failures.** For each case, all methods and the reference are evaluated at the same sampled timestamps. Missing inputs, trajectory-estimation failures, failures of the alignment solver, and non-finite scoring results are recorded with their causes and excluded from the error averages. We report per-method valid coverage and additional results on the common subset successfully evaluated for every method.
Reviewer_HUewOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to make MLLM representations more useful for controllable video generation through factor-aware supervision and three-stage training. Overall, I find the approach reasonable and the results encouraging. The existing ablations provide useful support for the method. My main questions concern the necessity of the three-stage training procedure and the influence of training-data quality.

Critical strengths
  1. The paper addresses a timely and interesting problem to improving instruction following and selective control over character, scene, action, and camera in video generation.

  2. The factor-span replacement idea is intuitive and reuses the existing MLLM–DiT architecture without adding a new representation network.

  3. The matched-data, matched-budget comparison with training without factor swapping supports its benefit for editing. The comparison between text-only and mixed-observation training also provides useful evidence for the final stage.

Critical weaknesses
  1. The necessity of the full three-stage procedure is not fully established. The ablations support factor swapping and mixed-observation training, but do not show whether sequentially freezing and adapting the modules is preferable to a simpler training strategy, such as joint fine-tuning. A comparison using the same data and a comparable total training budget would help justify this design.

  2. The approach relies on carefully constructed single-factor video pairs, including synthesized action and camera edits filtered by a VLM. The sensitivity to pair quality is not quantified. I would like to understand how reliably the non-target factors are preserved and how much imperfect pairs affect the gains.

Questions for authors
  1. How necessary are the three separate training stages? Could you compare with joint fine-tuning or a simpler schedule using the same data and a comparable total training budget?

  2. How sensitive is FAPL to the quality of the constructed pairs? A audit of accepted pairs and a controlled comparison with less strictly filtered data would help clarify the reliability of the supervision and the method's robustness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HUew

Thank you for your positive assessment and constructive questions. We address your two main concerns with matched joint-training comparisons, an independent audit of 2,000 video pairs, and controlled pair-quality experiments. These results support the staged design and quantify both the reliability of the supervision and the effect of imperfect pairs.

(W1,Q1) 1. Staged vs. Joint Training

1-a. Rationale. FAPL first trains the DiT to decode MLLM features and support factor control, then updates the MLLM and connector through the frozen DiT. Each phase therefore adapts to a fixed counterpart, allowing focused optimization and separate capability checks. Joint training instead changes representations and their decoder simultaneously, requiring both to adapt to a moving counterpart.

1-b. Controlled comparison. We compare staged training with joint updates of all three modules under the same data and training budget. Joint training retains both mixed-observation and factor-span replacement supervision, with three sampling ratios tested. FAPL uses the original schedule and outperforms every tested joint configuration. Against the best joint configuration, it improves both Motion Order and editing accuracy by 3 percentage points, supporting the value of staged adaptation beyond the supervision signals alone.

Table R8. Staged versus joint training with matched data and training budget on UniVideo (13B).

Training strategyObservation : factor-swap samplingMotion Order (%) ↑Trans. Error ↓Rot. Error (rad) ↓∩-Acc (%) ↑
Joint1:1580.06231.1755
Joint2:1580.06281.1953
Joint1:2560.06411.2355
Staged FAPLApplied in separate stages610.05861.0258

(W2,Q2) 2. Pair Quality: Reliability and Sensitivity

The human audit confirms that most accepted pairs satisfy the intended single-factor constraints, while controlled perturbations show that editing gains persist with imperfect supervision. We provide both reliability and sensitivity evidence below.

2-a. Independent human audit. Seven professional annotators audited 2,000 randomly sampled accepted pairs, with 500 per factor, checking target changes and preservation of all three non-target factors. Overall, 93.8% preserve all non-target factors, and 90.9% satisfy both requirements. The joint success rates for synthesized Action and Camera pairs are 89.0% and 90.4%, respectively.

Table R9. Human audit of accepted training pairs.

FactorAudited pairsTarget change succeedsAll non-target factors preservedBoth requirements satisfied
Character50096.8%94.4%92.2%
Scene50095.8%95.0%92.0%
Action50093.2%92.2%89.0%
Camera50094.8%93.6%90.4%

2-b. Controlled sensitivity analysis. We compare the original Qwen3-VL-32B-Instruct filtering with Qwen3-VL-8B-Instruct under identical acceptance rules, and with datasets replacing 10% or 20% of accepted pairs with rejected candidates within each factor. All four datasets contain 100K pairs, balanced across factors, with unchanged train/test partitions. Training starts from the same Stage I checkpoint and uses identical factor-span replacement, 15K steps, and evaluation settings. Replacement percentages denote rejected-candidate inclusion, not verified error rates. Editing accuracy decreases from 57% to 53.1%, 55.6%, and 51.8%, respectively. All remain above the Stage I checkpoint’s 44%, showing that imperfect supervision reduces, but does not erase, the editing gains.

Table R10. Stage II sensitivity to pair selection and rejected-candidate inclusion.

ConfigurationStage II dataMotion Order (%) ↑Trans. Error ↓Rot. Error (rad) ↓∩-Acc (%) ↑
Stage I checkpointNo Stage II training540.07081.3844.0
8B filtering100K accepted by the 8B filter530.06821.3453.1
10% replacement90K accepted + 10K rejected by the 32B filter530.06781.3355.6
20% replacement80K accepted + 20K rejected by the 32B filter520.06891.3751.8
Original 32B filtering100K accepted by the 32B filter530.06711.3257.0

Thank you for the detailed response and additional experiments. The joint-training comparison and pair-quality analysis address most of my concerns. I remain supportive of acceptance.

Thank you for your thoughtful feedback and continued support for acceptance! We are glad that the additional experiments have addressed most of your concerns, and we sincerely appreciate the time and care you have devoted to reviewing our work. :)

Reviewer_MM99OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Factor-Aware Predictive Learning (FAPL), a three-stage training procedure for MLLM-conditioned video generation. Stage I trains the DiT using MLLM latents as its only content condition. Stage II replaces factor-specific latent spans between paired examples to train selective changes to character, scene, action, or camera. Stage III freezes the DiT and updates the MLLM and connector using text-only and text-plus-complete-video inputs supervised with the same video target. The method is evaluated on text-to-video generation, factor-aware editing, and multi-shot consistency.

Critical strengths
  1. The paper addresses a relevant problem: providing MLLM features to a video generator does not necessarily make those features useful.

  2. The Stage II ablation is convincing. Under the same paired data and optimization budget. This provides evidence that the proposed use of paired data contributes beyond conventional supervised training.

  3. The reported improvements cover complementary capabilities. Relative to UniVideo, FAPL improves Motion Order from 48% to 61% and editing accuracy from 42% to 58%, alongside better camera and cross-shot consistency metrics. The blinded human study also separately evaluates target changes and non-target preservation.

Critical weaknesses
  1. The full-system comparison does not sufficiently isolate the training strategy from additional fine-tuning. FAPL receives 35K training steps on the curated dataset, whereas the main baseline is pretrained UniVideo. The matched Stage II control is useful, but a conventional fine-tuning baseline using the same data and a comparable optimization budget is needed to assess the contribution of the overall procedure.

  2. The interpretation of Stage III is stronger than the current evidence supports. Its objective combines text-only and complete-video conditional flow-matching losses through a frozen generator. The mixed-observation ablation establishes a practical benefit, but does not distinguish improved temporal representations from other effects of multimodal fine-tuning. Since both the MLLM and connector are updated, a connector-only control would clarify whether adapting the MLLM itself is necessary. The supplementary Stage I comparisons demonstrate improved standalone decoding particularly clearly for OmniVideo2-1.3B, but some UniVideo examples already produce recognizable actions before adaptation. These comparisons establish conditioning compatibility more directly than they establish the extent of MLLM feature bypass under the original full-conditioning configuration.

  3. The inference protocol is insufficiently explicit. The paper allows either MLLM-only inference or inference with native conditioning pathways restored, but does not clearly identify the configuration used in the quantitative tables. Editing preservation is measured against each method's generated Before video, so the task assesses consistency under a conditioning intervention rather than directly measuring preservation of an externally supplied source video. The sampling-seed protocol between Before and After generations should also be specified.

  4. Some evaluation details could affect the comparisons. The camera protocol does not specify translation-scale alignment or treatment of trajectory-estimation failures. FaceSim-Arc excludes sequences with fewer than two valid shots without reporting per-method coverage. The human study is useful but contains only eight cases per task. These limitations leave uncertainty about the robustness of some reported gains.

Questions for authors
  1. How does FAPL compare with conventional fine-tuning of UniVideo on the same dataset under a comparable training budget?

  2. How much of the Stage III improvement remains when only the connector is updated? What evidence specifically supports attributing the gains to improved MLLM temporal representations?

  3. Which conditioning pathways are enabled in Tables 1 and 2? Please clarify the editing seed protocol, camera-trajectory alignment and failure handling, and per-method FaceSim-Arc coverage.

7 comments
AuthorsOpenReview ↗

Response to Reviewer MM99 (Part 1)

Thank you for your careful review and constructive suggestions. We address your main concerns through matched fine-tuning and connector-only controls, full-conditioning tests, and expanded robustness evaluations, while clarifying the inference protocols and the scope of our mechanistic claims. These additions help distinguish FAPL’s contributions from extra training and evaluation effects.

(W1,Q1) 1. Gains Beyond Additional Fine-Tuning

We added conventional SFT using the same UniVideo initialization, paired data, and training steps for updating the "MLLM+connector+DiT", without factor-span replacement or staged training. FAPL outperforms this baseline on all four metrics, improving Motion Order by 4 percentage points and editing accuracy by 7, demonstrating gains beyond additional fine-tuning alone.

Table R1. Comparison under matched data and training steps.

MethodAdditional stepsMotion Order (%) ↑Trans. Error ↓Rot. Error (rad) ↓∩-Acc (%) ↑
Pretrained UniVideo0480.07631.5442
UniVideo (13B) + conventional SFT35K570.06481.2151
Univideo (13B) + Ours35K610.05861.0258

(W2,Q3) 2. Assessing MLLM Feature Bypass Under Full Conditioning

2-a. Rationale. Stage I freezes the MLLM and connector, removes alternative content pathways, and trains only the DiT to decode fixed MLLM latents. The resulting conditioning-to-output mapping is retained in the updated DiT, potentially strengthening MLLM influence when native pathways are restored.

2-b. Controlled experiment. We compare checkpoints before and after Stage I with all native pathways enabled. For the same target factor, the MLLM specifies factor A, while visual references and, where present, the text encoder specify B. Non-target factors, sampling seeds, and inference settings remain fixed. MLLM instruction alignment increases in both models, and Omni-Video 2 shifts from favoring native text to favoring the MLLM instruction. This supports stronger MLLM influence under conflicting full conditioning.

Table R2. Output alignment under conflicting full conditioning.

ModelStage ICLIP similarity to MLLM text ↑CLIP similarity to native text ↓VAE latent MSE to reference ↓
Omni-Video 2 (14B)Before0.17480.24030.0502
Omni-Video 2 (14B)After0.25730.20690.0201
UniVideo (13B)Before0.2796N/A0.0142
UniVideo (13B)After0.3356N/A0.0086
AuthorsOpenReview ↗

Response to Reviewer MM99 (Part 2)

(W2,Q2) 3. Stage III: Connector-Only Control

We added a connector-only control starting from the same Stage II checkpoint, with the same training settings. The only difference is whether the MLLM is updated. Connector-only training raises Motion Order from 53% to 54%, whereas full Stage III reaches 61% and achieves lower camera errors. Thus, updating the MLLM provides substantial temporal-generation gains beyond connector adaptation alone.

Table R3. Stage III with and without MLLM updates.

ConfigurationMLLMConnectorMotion Order (%) ↑Trans. Error ↓Rot. Error (rad) ↓∩-Acc (%) ↑
Connector-onlyFrozenUpdated540.06651.3057
Full Stage IIIUpdatedUpdated610.05861.0258

(W2,Q2) 4. Evidence for Improved Temporal Representations

Motion Order evaluates temporal relationships between actions, not merely action presence. With the DiT frozen, mixed-observation training improves this score from 57% to 61% over text-only training. The connector-only comparison further isolates the benefit of MLLM updates (54% vs. 61%). Together, these controls support MLLM representations that better guide action sequencing, rather than an independent claim of improved general temporal reasoning.

(W3) 5. Inference Configurations

5-a. Conditioning pathways. Sec. 3.5 and Appendices D–E describe the inference interfaces and task inputs. Table R4 maps these settings to the reported results. Conditioning pathways remain fixed across all Table 2 ablations.

Table R4. Conditioning pathways used in the quantitative evaluations.

ResultsTaskFAPL / UniVideo (13B)Omni-Video 2 (14B)Visual references
Table 1Sparse-to-videoMM + TNone
Table 1Factor-aware editingMM + TNone
Table 1Multi-shot generationM + VM + T + VShared character and scene references
Table 2All ablationsMN/ANone

M denotes MLLM features through the connector, T native text-encoder features, and V VAE-reference features. For multi-shot generation, visual references enter both the MLLM and VAE pathways.

5-b. Editing preservation. As explicitly stated in Appendix E, “Question Content” (p. 20), and Appendix F (p. 21), preservation compares each method’s generated Before and After videos. The task measures selective change under a conditioning intervention, not reconstruction of an externally supplied source video. Baselines encode the complete edited description, whereas FAPL replaces only the target-factor span after the connector.

5-c. Sampling seeds. For each editing pair, we set the same seed 66 before sampling the initial noise for both Before and After. Within the same model and latent dimensions, this gives identical initial noise, reducing sampling variability in the preservation comparison.

AuthorsOpenReview ↗

Response to Reviewer MM99 (Part 3)

(W4) 6. Evaluation Protocols and Robustness

We detail camera evaluation, report valid-sample coverage, and assess robustness on shared samples and an expanded human study.

6-a. Camera alignment and failure handling. Following Appendix E, we recover trajectories with frozen π3 and express each relative to its first frame. A single Umeyama Sim(3) transformation aligns the predicted trajectory’s scale, rotation, and translation to the reference. Both errors are computed after alignment using frame-index correspondence, with the predicted and reference videos sampled at the same timestamps. Missing inputs and exceptions during loading, reconstruction, or scoring are logged and excluded. No additional confidence or trajectory-degeneracy filtering is applied. FAPL retains the lowest camera errors on the subset successfully evaluated by every method, so the ranking persists when evaluated samples are matched.

Table R5. Camera evaluation over 5,000 cases. Errors are computed on the common valid subset of 4,700 cases successfully evaluated by all four methods.

MethodFailuresFailure rateValid casesCommon-set Trans. Error ↓Common-set Rot. Error (rad) ↓
Omni-Video 2 (1.3B)2034.06%4,7970.10681.86
Omni-Video 2 (14B)1272.54%4,8730.08111.49
UniVideo (13B)1462.92%4,8540.07521.52
FAPL941.88%4,9060.05821.01

6-b. FaceSim-Arc coverage. Appendix E specifies that eligible sequences contain at least two shots with valid face observations and receive equal weight. Table R6 reports coverage and scores on the same 82 sequences valid for all methods. FAPL retains its advantage, scoring 0.529 versus UniVideo’s 0.447, showing that the improvement persists after controlling for sequence eligibility.

Table R6. FaceSim-Arc coverage and common-set comparison across 100 test sequences.

MethodValid sequencesCoverageOriginal FaceSim-ArcCommon-set FaceSim-Arc ↑
Omni-Video 2 (1.3B)8585%0.4120.407
Omni-Video 2 (14B)9090%0.4680.474
UniVideo (13B)8888%0.4510.447
FAPL9494%0.5230.529

6-c. Expanded human evaluation. We increased the study from 8 to 40 cases per task, totaling 120 cases, randomly sampled from the corresponding test sets with 10 editing cases per factor. We retained the blinded, randomized protocol and evaluation criteria from Appendix F. FAPL remains preferred across all three tasks, with preference rates within 0.5 percentage points of the original results. This expands the number of distinct cases rather than merely collecting more ratings on the same examples.

Table R7. Overall preferences in the expanded human study.

TaskCasesFAPL preferredTieUniVideo preferred
Sparse-to-video4056.4%17.1%26.5%
Factor-aware editing4061.5%15.4%23.1%
Multi-shot generation4054.6%19.4%26.0%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.