acceptodds
Withdrawn from ICLR 2027

Learning to Interact and Direct: Co-Evolving Video Director Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video director agents orchestrate multistage production to realize users' creative visions. Emerging research on their self-evolution advances screenplay planning effectively. However, the mutual reinforcement between directing and user interaction remains largely overlooked. Improved interaction clarifies user intent and informs workflow refinement, while more reliable execution shifts collaboration from troubleshooting to creative decisions. To this end, we introduce LIT (Learning to Interact and DirecT), a framework comprising a director agent, a simulated user, and a meta engineer. The simulated user assigns creative tasks to probe the director's weaknesses, while the meta engineer uses the task traces to revise the director’s interaction and production strategy. Although this approach can address discovered interaction and production issues, it still relies on passive attribution by the meta agent. This limits its ability to adapt to broader user interaction patterns and production techniques. To solve this problem, we equip LIT with a co-evolution paradigm. The meta agent proactively learns external knowledge, thereby improving its ability to discover problems and propose solutions. The meta agent, simulated user, and director together form an evolutionary closed loop, collaboratively enhancing the system's directing and interaction capabilities. Across the Test and OOD Test splits, LIT improves production and Interaction scores by 19.1% and 32.0%, respectively, relative to the initial pipeline. Controlled ablations reveal the correlation between improvements in production and interaction, where the co-evolution mechanism further encourages directing skill and interaction mode exploration. Code will be released.

Primary area:
generative models
Keywords:
video director agent, self evolution, harness engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DWejOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-project self-improvement in interactive video directing and proposes LIT, a framework comprising a director, a simulated user, and a meta-engineer. LIT enables both interaction strategies and video production workflows to be updated across projects. The meta-engineer proposes revisions based on dialogue, execution traces, and completed videos, while improving its own development process through external knowledge retrieval, task scheduling, and revisions to its evaluation criteria.

Critical strengths
  1. The paper distinguishes failures in eliciting user requirements from failures in executing the video production process. Rather than focusing solely on the quality of the final video, it also studies how the collaboration process with users can improve over successive projects. This makes the research question more compelling than simply optimizing the video generation pipeline.

  2. The director’s skills, the simulated user’s task design, and the meta-engineer’s diagnosis and revision processes are all incorporated into a continuous update cycle. This provides a concrete framework for studying whether the improvement process itself can be optimized, rather than merely proposing co-evolution as a conceptual idea.

  3. The experiments that freeze the interaction or production strategy, remove Co-Evo, and vary the base model provide useful evidence for understanding where the improvements come from. These analyses are more informative than simply showing that the full system outperforms its initial version.

Critical weaknesses
  1. The distinction from existing methods is not sufficiently clear. The paper emphasizes that LIT improves user interaction strategies, but the adapted VideoWeaver baseline also has the ability to update its interaction skills. The authors should therefore clarify which specific mechanism differentiates LIT from existing approaches. In particular, it would be useful to isolate the contribution of the simulated user and the meta-engineer, rather than treating their inclusion alone as the source of novelty.

  2. The experiments do not isolate the effect of co-evolution. Removing Co-Evo simultaneously removes external knowledge retrieval and revisions to the evaluation criteria. Therefore, the resulting performance drop does not establish which component is responsible for the observed improvement. Separate ablations of these components would provide a clearer understanding of the contribution of co-evolution itself.

  3. The comparisons do not control for computational and generation costs. The compared methods appear to use different amounts of resources for development, validation, and video production. Consequently, the higher performance of LIT may partly result from a larger computational budget or a greater number of generation opportunities, rather than from the proposed methodology itself. Comparisons under matched resource budgets, or performance–cost curves, would make the experimental conclusions more convincing.

  4. The human study does not evaluate actual human–AI collaboration. Participants only rated completed videos after watching them and reading the associated dialogue; they did not directly interact with the director or participate in revising the generated work. Thus, the study provides evidence that participants prefer the presented outcomes, but it does not establish that LIT leads to a better collaborative experience for actual users. Supporting this claim would require a user study in which participants directly use and interact with the system.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_iDXeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper covers an approach to video generation where a number of agents (a director agent, a simulated user, and a meta engineer) work together to create a video. The simulated user essentially acts as a critic or judge, and the entire system makes use of web searching and self-reflection to improve, which the authors refer to as co-evolving. The authors demonstrate that the same LLM that makes up the three agents also evaluates the output movies higher on a number of dimensions and that humans generally prefer their output videos in a human subject study.

Critical strengths

The inclusion of a number of baselines is a clear strength. In addition, running a human subject study is also a strength.

Critical weaknesses

This paper draft has a number of major weaknesses.

One major weakness is the text. Normally this would not be the first listed weakness but it's relevant in this case. There are minor grammar issues (e.g. "a framework comprises"->"a framework comprised of", "Simulated user assigns"->"The simulated user assigns", etc.). However, the more major concern is repeated text. The most egregious example of this is the "User Study", which is exactly replicated on both lines 371-377 and lines 427-434. This lack of rigorous checking of the text puts the rest of the paper into question.

In terms of the system, this is essentially a fairly standard agentic system setup. The specifics of the agents and what they are doing in the system are novel but this is very minimal technical contribution. I disagree with the authors' choice to frame the self-improvement as evolution. Evolutionary search is a long-standing and still relevant optimization approach for applications like neuroevolution and approaches like MAP-Elites. Calling what is essentially a hill climbing search based on LLM evaluation evolution is not appropriate at an AI venue.

In terms of the evaluation, the use of the same LLM to evaluate and generate is not appropriate. It is not the case that human evaluation and agentic evaluation is the same [1] and there's a clear opportunity for bias using the same LLM for evaluation and generation. The human subject study should address this, but the methodology is unclear. Specifically, how were the output videos picked for the different approaches. Choosing the best videos from one and the worst from others would be a clear threat to validity. It's also unclear why the authors chose the baselines for inclusion in the human subject study they did instead of GPT-6, which was the best performing baseline.

Given the above, there does not appear to be clear technical novelty and the value (due to the issues in the evaluation) is very unclear.

  1. Agnew, William, et al. "The illusion of artificial inclusion." Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 2024.
Questions for authors
  1. How did the authors select the videos for the human subject study?
Reviewer_u86hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LIT, a framework that jointly improves video directing workflows and user interaction strategies across projects. A meta engineer analyzes simulated user interactions and production traces to revise the director, user simulator, and its own evaluation procedures. Experiments on held-out and OOD tasks, component ablations, and a human preference study show improvements in production quality and interaction over the initial pipeline.

Critical strengths
  1. Meaningful problem formulation: the proposed method of jointly improving user interaction and production addresses an important limitation of production-only optimization, where better execution does not necessarily resolve misunderstood creative intent. I like the user centric view of the problem

  2. Promising empirical results: improvements on held-out and OOD tasks, supported by human preferences, suggest that the framework can improve both production quality and intent fulfillment.

Critical weaknesses
  1. Flawed evaluation First of all the headline results rely on an evaluation that is not independent of what LIT optimizes. All main scores come from a Qwen3.8-27B judge which is the same model used for the Director, simulated user, and meta engineer. This could introduce self-preference bias toward outputs planned and prompted by the same backbone. Also the meta engineer's evolved criteria in Section 3.3 closely follows the test rubric items in Appendix E, could the reported gains then partly reflect LIT learning to fit the judge rather than producing better videos? The independent / objective scores in Table 16 do not clearly favor LIT which also makes the large margins in Table 1 harder to interpret. The authors attribute this to the benchmarks' limited discriminability" but the scores do vary noticeably (e.g., CCA ranges 3.25-4.42, IQ 0.59-0.71)?

Similarly the human study which could complement the author's argument is also flawed and confusing. Section 3.2 says "4 pairs of videos" suggesting pairwise comparison whereas Appendix G.5 says they compare groups of three videos? The user study website in Fig 13 shows "1/8" which seems to indicate a participant completed 8 comparisons? The result is also hard to follow, the authors report Kendall's tau as the main result but where is statistical significance test? Table 2 reports only raw win counts and the 60 assessments reuse the same 16 task groups, so they are not independent. Also, tau is computed between rankings of three methods per dimension so it reflects agreement on the ordering of methods rather than on individual videos? Tau only compares the aggregate ordering of three methods and it does not show that the judge and humans agree on the same videos which is what validating a per-video scoring metric requires.

  1. Unclear attribution of gains The experiments do not clearly attribute the gains to the proposed mechanisms. The main contributions of the paper are the mutual reinforcement between interaction and production, and the co-evolution of the meta engineer. However, I am not convinced that the experiments isolate either of them.

The comparison mixes method with resources, for example, budgets differ across methods, several baselines are substantially modified (e.g., GenMAC’s layout control replaced by text instructions), and only LIT has access to the curated reference database. The ablations are also confusing, Table 3 does not report a Production score even though section 3.4 claims a synergistic effect between production and interaction. When I compute it from Table 3 using the paper's own definition in Section 3.2, the gains look nearly additive rather than synergistic... I am not sure which part of the reported numbers support the author's claim of the mutual reinforcement. Also, the "w/o Co-Evo" removes external retrieval and criteria revision together (given Appendix G.1), so I am not sure which of them drives the improvement.

Questions for authors

The questions below largely overlap with the critical weaknesses above. I reframed as questions whose answers could change my assessment.

  1. The judge (Qwen3.8-27B) is the same model used for the Director, simulated user, and meta engineer. Could the authors evaluate the outputs with a judge from a different model family and report the results, to rule out self-preference bias?
  2. The evolved criteria in Section 3.3 (e.g., question necessity, question timing, actor count/position consistency) appear closely related to the test rubric in Appendix E. Could the authors clarify how the meta engineer’s initial criteria were constructed, and whether the test rubric was finalized independently of the evolution runs?
  3. Table 16 does not show a clear advantage for LIT, although its scores vary across methods (e.g., CCA 3.25–4.42, IQ 0.59–0.71). It would be helpful if the authors could discuss this discrepancy in more detail, for instance how their rubric and these benchmarks differ in what they measure.
  4. Could the authors clarify the user study design? Section 3.2 mentions “4 pairs of videos,” Appendix G.5 describes groups of three videos, and Fig. 13 shows “1/8.” It is unclear how many comparisons each participant made and in what format.
  5. Would the authors be able to report statistical significance for the human preferences in Table 2, taking into account that the 60 assessments reuse the same 16 task groups and 15 participants?
  6. Since Kendall’s τ is computed between aggregate rankings of three methods, could the authors also report task-level agreement between human and automatic judgments, for example how often both selected the same video within a group?
  7. Could the authors provide a summary of the resources each method used in the Table 1 setting (e.g., number of productions, tokens, retries, retrieval calls)? A budget-matched comparison with the closest baseline, VideoWeaver, would also be informative.
  8. How much of LIT’s improvement might be attributable to the curated reference database? It would be helpful to see how a baseline such as VideoWeaver performs with access to the same database.
  9. Section 3.4 describes a synergistic effect between production and interaction, but Table 3 does not report a Production score. Computing it with the definition in Section 3.2, the Test production gains appear close to additive. Could the authors point to the results that most directly support mutual reinforcement, as opposed to complementary gains?
  10. The “w/o Co-Evo” setting removes external retrieval and criteria revision together (Appendix G.1). Would it be possible to ablate these two components separately, to clarify their individual contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.