acceptodds
Withdrawn from ICLR 2027

CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion-based video generative models achieve strong photorealism and motion fluency, yet fall short in professional production, where conditions are abstract, sparse and often mutually conflicting, e.g., storyboard sketches and clay-render clips that specify the creative intent rather than the pixels. We present **CogOmniControl**, a closed-loop reasoning-generation-verification framework for controllable video generation. The reasoner, CogDirector, is a specialized VLM trained via SFT and RFT on authentic production data; it cognizes the creative intent behind sparse and abstract conditions into a dense production plan, whose representations are injected into generator as semantic guidance through in-context conditioning. The same frozen CogDirector serves as the reward model during generator RFT, anchoring training and test-time verification to a consistent interpretation of creative intent. At test time, it further emits an evaluator harness in the same forward pass to select among multiple sampled candidates via VLM judges and specialized tools, i.e., test-time scaling. We theoretically characterize this verification process, deriving bounds on the misranking probability and expected selection regret of best-of-N selection, and showing that input-adaptive evaluation admits tighter guarantees when it reduces effective evaluation noise while preserving the target quality. We also introduce CogReasonBench and CogControlBench built from real professional workflow data with genuine creative intent. Experiments on the two benchmarks show that CogOmniControl surpasses existing open-source models.

Primary area:
generative models
Keywords:
Video Generation, Harness, Vision-Language-Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MEFZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CogOmniControl addresses multimodal control in professional video production, where storyboards, clay renders, and reference images convey intent rather than final pixels. A VLM is trained through SFT and RFT to form CogDirector, which interprets sparse conditions as a production plan and supplies representations to the video generator. The generator is then reinforcement-fine-tuned using rewards from the frozen CogDirector. At test time, CogDirector also selects evaluation tools to rank four candidate videos. The paper provides conditional analyses of misranking and selection regret, and introduces CogReasonBench and the 200-instance CogControlBench for reasoning and generation evaluation. The authors report improved intent following from domain-specific reasoning and further automatic-score gains from adaptive selection. Final evaluation uses an independent VLM, supplemented by human preference studies.

Critical strengths
  1. The application is well motivated. Abstract requirements in storyboards and clay renders are poorly served by pixel matching alone, making intent interpretation a sensible design choice. Table 4 also supports the domain-specific reasoner's contribution, rather than relying only on comparisons between complete systems with different backbones.

  2. Final evaluation includes an independent model and human studies, rather than relying solely on the training reward model. This provides positive evidence for the overall system, particularly its intent following under professional control conditions.

Critical weaknesses
  1. One Avg in Table 3 is not reproducible. Normalizing as described and averaging the fifteen metrics equally gives 0.700 for VACE-Wan2.1, not 0.665. Additional weights or eligible-sample handling need to be specified. This does not change the overall open-source ranking, but materially changes the gain over that baseline.

  2. The evidence for adaptive selection is incomplete. Changing tools and weights may also change the evaluation objective; the increase from 0.731 to 0.748 does not by itself establish the noise reduction assumed by the theory. I would particularly like to know whether both selectors use identical candidates and whether independent human evaluation favors the adaptive selections.

  3. The authors exclude test clips, but training and testing still come from the same production pipeline. Separation of projects, characters, and adjacent shots is unclear (Appendix C.1). The results therefore do not yet distinguish adaptation within a familiar distribution from transferable intent understanding. This contributes to my current weak-reject recommendation.

Questions for authors
  1. Could you provide the exact aggregation formula and per-instance results for Table 3?

  2. Do fixed and adaptive selection use the same four candidates? How do they compare under human evaluation on that shared pool?

  3. Are data split by project or shot sequence, and are results available on an unseen project?

Reviewer_VgtoOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CogOmniController, a framework for controllable video generation that models creative intent through an explicit reasoning process over multimodal control inputs. The method uses a specialized vision-language model to infer structured intermediate reasoning features from inputs such as storyboards and clay renders, and then injects these features into the video generator to guide generation. To support this setting, the paper also introduces new datasets and benchmarks centered on professional animation-production workflows. Experiments report improvements in controllability and output quality over existing baselines, particularly for abstract and multimodal creative-control conditions.

Critical strengths
  1. The paper tackles a practically important and insufficiently studied problem setting: controllable video generation from sparse, abstract, and potentially conflicting production signals (e.g., storyboards and clay-render clips), rather than only from explicit pixel-aligned controls. This is a consequential strength because it addresses a clear mismatch between existing controllable video generation research and real production workflows. By focusing on intent-level control in professional settings, the paper moves beyond standard benchmark optimization and points to a problem formulation with practical relevance.
  2. The overall framework is well integrated and methodologically coherent. In particular, the paper presents a unified reasoning–generation–verification pipeline in which a specialized multimodal model is used to infer intent-level guidance, support generator alignment, and drive test-time candidate selection. This is a clear system-level design rationale and is supported by a reasonably comprehensive empirical study.
Critical weaknesses
  1. The paper does not convincingly establish that the observed improvements are specifically due to the explicit reasoning module, rather than to simpler alternatives such as rewriting the input conditions into richer prompts or plans. This is the most important weakness in my view because the paper’s central claim is that creative-intent reasoning should be modeled as an explicit intermediate variable. However, the current experiments do not include the most relevant counterfactual baselines. In particular, the paper does not test whether similar gains could be achieved by using the same specialized VLM only to convert the multimodal inputs into richer prompts or structured plans, without explicit reasoning-feature injection into the generator. As a result, the current evidence does not yet isolate the contribution of the explicit reasoning design from that of stronger condition reformulation.
  2. The paper provides limited evidence that the proposed reasoning framework generalizes beyond the specific professional workflow distribution on which it is trained and evaluated. This is consequential because both the specialized training data and the newly introduced benchmarks are concentrated on closely related professional-production inputs, especially storyboard- and clay-render-based workflows. While the reported results are promising within this setting, they do not yet show that the framework transfers to materially different forms of abstract control. As a result, the current evaluation does not sufficiently support the broader claim that the proposed reasoning-based approach generalizes beyond the authors’ workflow setting.
Questions for authors
  1. Can the authors provide a direct baseline in which CogDirector is used only for condition expansion/reformulation (e.g., converting the multimodal inputs into a richer prompt or structured plan), without reasoning-feature injection and without DiT finetuning? This would help determine whether the reported gains truly require an explicit reasoning intermediate, rather than arising primarily from stronger prompt construction.
  2. Can the authors provide stronger evidence that the proposed framework generalizes beyond the storyboard/clay-render-centered workflow distribution used in the specialized training and benchmark construction? For example, results on a substantially different abstract-control setting, or a more explicit breakdown between workflow-specific and more general subsets, would materially affect my assessment by clarifying whether the observed gains reflect a general reasoning capability or mainly domain adaptation.
  3. A more fine-grained analysis of CogDirector would strengthen the paper. For instance, disentangling the contributions of its roles as reasoner, generator-side representation provider, reward model, and test-time harness planner.
Reviewer_D1ZVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CogOmniControl, a reasoning-driven framework for controllable video generation under complex multimodal conditions. It introduces CogDirector, a module trained with SFT and RFT to understand creative intent, guide video generation through reasoning embeddings, and provide rewards for generator fine-tuning. At inference, an Input-Adaptive Evaluator Harness dynamically selects and weights evaluators for Best-of-N candidate selection. The paper also introduces two benchmarks and demonstrates improvements in intent understanding and controllable video generation.

Critical strengths
  1. Clear Presentation: The paper is well-written and clearly organized, making the motivation, methodology, and overall framework easy to understand.

  2. Valuable Data Contribution: The paper collects and curates data from real-world professional animation production workflows, including storyboard-to-video and clay-render-to-video pairs.

Critical weaknesses

Lack of Video-Outcome Supervision: The authors argue that using video-level rewards to train CogDirector is prohibitively expensive. However, prior works such as Video-as-Answer [1] have demonstrated the effectiveness of using generated visual outcomes to optimize upstream planning models, bridging the gap between semantic correctness and visual executability. An additional ablation incorporating similar video-level rewards would help assess their effectiveness and potential gains for this task.

Insufficient Agentic Baselines: Comparisons with agentic video generation frameworks and stronger reasoning models (e.g., GPT-5 as the Director) would better establish the effectiveness of CogDirector.

Data Reproducibility: Will all newly collected and annotated datasets be publicly released? Are there copyright or licensing restrictions, particularly for professional production data?

Limited Harness Novelty: Adaptive reward model selection has been explored in prior works, such as LASeR [2] and Reward Model Routing in Alignment [3]. The authors should clarify the novelty of their approach and compare it against simpler alternatives, such as rule-based evaluator selection or task-conditioned reward weighting.

Missing Compute-Efficiency Analysis: The paper lacks a systematic evaluation of the computational overhead introduced by test-time scaling. A quality-versus-compute plot across different Best-of-N settings would better demonstrate its practical efficiency.

References

[1] Junhao Cheng, Liang Hou, Xin Tao, and Jing Liao. Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO. [2] Duy M. H. Nguyen, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits. [3] Xinle Wu and Yao Lu. Reward Model Routing in Alignment.

Questions for authors

Please refer to weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.