acceptodds
Withdrawn from ICLR 2027

VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic long video generation requires planning, tool orchestration, and cross-clip coordination over a long horizon. Most existing video agents either rely on static, human-crafted workflows, which require substantial manual effort and poorly adapt across tasks, or iteratively refine the output of the current task without persistently distilling execution experience into reusable skills for future tasks.We introduce VideoWeaver, an agent harness and benchmark that evaluates and evolves skills for long video generation. Given a single high-level instruction, an agent dynamically composes foundation skills into its own workflow rather than following a predefined pipeline. We construct a benchmark of 16 task categories and 285 cases, with references spanning text, image, audio, video, and their combinations. We further propose an evidence-grounded agent-as-judge that inspects both the execution trace and the final video to diagnose process and output failures. Based on this feedback, our evolution algorithm progressively refines category-level composition and creator skills, allowing recurring experience to guide dynamically constructed workflows for unseen cases. Experiments show that explicit composition skills improve the generation process over foundation skills alone, while skill evolution further improves output quality and generalizes to unseen cases. Incorporating judge feedback yields additional gains, especially on output metrics, and the agent-as-judge aligns well with human, particularly on process metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Video Generation, Long Video Generation, Video Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KeryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a general-purpose agent harness can turn one high-level instruction into a multi-clip video by composing a fixed set of media tools, and whether that workflow can be rewritten from past runs. VideoWeaver has three parts: a benchmark of 285 cases in 16 categories (119 train / 123 test in 13 categories, plus 43 cases in 3 held-out categories); an agent-as-judge that scores 6 process metrics and 7 output metrics from the execution trace and the final video; and a skill-evolution procedure that rewrites category-level composition skills and creator skills, then merges creator skills for unseen categories. Final comparison uses a three-annotator ranking of nine systems. The judge is used mainly as the rewriting signal. On OpenClaw + Seed2.0, adding a composition skill raises the process average from 0.765 to 0.922 while the output average stays near 0.55. Human ranking then rises from 2.96 to 4.59 with self-evolution and to 5.23 with judge feedback. A handwritten expert skill remains highest, at ranking 5.63 and output average 0.620. Prompt following stays low across harnesses, from 0.25 to 0.46.

Critical strengths
  1. The process/output split is the right comparison, and the paper follows through on it by using human ranking for the final ranking rather than the LLM judge.
  2. The experimental ladder is substantial and the controls are the right ones. Media tools stay fixed while the harness, backbone, and skill condition change, from no composition skill through a generated skill, self-evolution, judge feedback, and an expert skill. The paper reports that the expert skill still wins, and it includes cost and an order ablation. That makes the study usable as a reference point.
Critical weaknesses
  1. The abstract says that skill evolution improves output quality and generalizes, and that judge feedback adds a further gain. The measurements support a narrower claim. On human ranking, most of the gain is self-evolution without the judge, from 2.96 to 4.59. Feedback reaches 5.23, and the expert skill is still ahead at 5.63. The output-average movement is about one to two points (0.552 to 0.567 to 0.582) and is largely Format Requirements, which rises from 0.728 to 0.913 between the composition skill and feedback. Over the same change, audio consistency falls from 0.567 to 0.448, and visual consistency and reference fidelity also fall. Across three disclosure orders in Appendix E, feedback is only +0.019 output average over self-evolution, with standard deviations of 0.019 and 0.042; prompt requirements and plot coherence favor self-evolution. Pairwise transfer is 25.3% win / 58.2% tie / 16.5% loss on test categories and 23.8% / 59.7% / 16.5% on the held-out categories. Brand Promotion has more losses than wins. I do not see a stable effect of the judge, or reliable transfer to unseen categories.

The written objective does not match the procedure. Equations 2–4 are argmax objectives over executed skills. In Algorithm 1, every stored trace comes from the initial composition skill, and a revised skill is not run again before it is frozen. The test numbers are real executions of the final skill text, but the paper does not show that rewriting increased the training objective it writes down. The scores used for rewriting are also the metrics on which humans disagree most.

  1. The community-facing contribution is thinner than the framing. The learned objects are skill texts, but the paper never shows a composition skill or creator skill before and after an edit, so a reusable procedure cannot be distinguished from checks for duration, planning, and successful tool calls. Video, image, speech, and understanding all go through one ByteDance / Volcengine stack, so the harness comparison is tied to that API. Tasks were filtered to about one minute, so this is multi-clip composition, whereas Section 2.1 discusses minute-scale to hour-scale generation. The three held-out categories sit close to the training categories. The reproducibility statement describes the procedure and does not say that cases, traces, judge prompts, or skills will be released. User-submitted reference media also needs a license before this can function as a benchmark. As written, the supported finding is that a rewritten workflow outranks the initial workflow under human ranking, and a human-written workflow ranks higher.
Questions for authors
  • In Equation 1, how many cases are in 𝑋, and what is the inter-annotator agreement on the rankings? Does the gap among Self-Evolution (4.59), Evolution with Feedback (5.23), and Expert (5.63) remain under resampling over cases and annotators?
  • Please show one composition skill and its creator skill before and after evolution, next to the feedback that caused the edits. I want to see whether the edits change shot planning and reference use, or mainly add checks that raise Format Requirements and the process metrics.
  • Does Algorithm 1 match the implementation: revised skills are not re-executed on training cases before being frozen? If they are re-executed, how does the training value of 𝜙 change after each update?
  • Will the cases, reference media, traces, metric prompts, and evolved skills be released, and under what terms can the user-submitted assets be redistributed? My assessment of the benchmark contribution depends on this answer.
  • Table 7 shows large gaps in relative generation time and token use. If video-generation calls or wall-clock time are matched, does the ranking gap among Composition Skill, Self-Evolution, and Evolution with Feedback remain?
Reviewer_Hu7iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VideoWeaver, an agentic framework for evaluating and evolving agentic long video generation. Firstly, a benchmark comprising different modalities is proposed for agentic long video generation. Secondly, an evidence-grounded agent-as-judge evaluation framework is introduced to inspect both the execution trace and generated output to systematically evaluate the video results and refine the agentic skills. Systematic metrics are proposed to evaluate both process and output quality. Extensive experimental results evaluate several harness/model combinations, comparing no-skill, composition-skill, self-evolution, feedback-based evolution, and expert-skill variants.

Critical strengths

The paper studies an important and underexplored issue, which is long-horizon agentic video generation with multimodal references and multiple generation or editing rounds.

The benchmark is more diverse than a single video generation scenario, with various video generation conditions/outputs. The paper also includes process-level metrics, execution traces, and final-output evaluation.

The paper includes useful comparisons across agent harnesses and backbone models, which provides insights in the agentic long video generation scenario.

Critical weaknesses

The overall agentic evaluation and evolving framework lacks technical novelty. I am convinced that building a good agentic framework for long video generation is useful since video generation performance could vary a lot with different agentic flows. However, the main algorithm mostly relies on existing powerful LLMs and commercial video generation tools.

The relationship between process and output metrics is unclear. For example, in Table 4, it seems that Evolution with Feedback should have better process and output metrics than Self-Evolution since it has evidence-grounded feedback, but most process metrics underperform. Moreover, better process metrics do not contribute to better output metrics, which raises questions about whether the process metrics effectively reflect the quality of the generated videos.

Questions for authors

Can the authors clarify the composition of the proposed benchmark, such as the proportion of real-world and animated video tasks? Whether performance differs between these two categories? It is worth investigating which types of scenarios the proposed agentic video generation framework performs better on. Whether performance differs in different modalities inputs, such as t2v, ti2v, t2av, etc? An analysis on the performance across different modalities is encouraged to provide.

Reviewer_BYN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VideoWeaver, a benchmark and agent framework for evaluating and improving skill-based long-video generation. The benchmark contains 285 cases across 16 task categories and requires agents to coordinate video, image, audio, and media-processing tools. The proposed framework uses creator skills to construct category-level composition skills, which specify workflows over foundation tools. These skills are then refined using execution traces, generated outputs, and feedback from an evidence-grounded Agent-as-Judge. The judge evaluates six process metrics and seven output metrics by inspecting traces, intermediate artifacts, metadata, and final videos. Experiments compare different agent harnesses and backbone models, study skill evolution with and without judge feedback, evaluate transfer to held-out categories, and examine human alignment and resource usage.

Critical strengths
  1. The paper studies an important and relatively underexplored problem. Existing video benchmarks mainly evaluate final outputs, whereas VideoWeaver also examines the generation process, including planning, input processing, tool execution, reference usage, and clip merging.

  2. The experimental evaluation is broad and reasonably transparent. The paper evaluates multiple harness–model combinations, compares automatic skill evolution against no-skill and expert-designed workflows, includes held-out-category experiments, validates the automatic judge against human annotations, and reports execution time, tool usage, token consumption, and monetary cost.

Critical weaknesses
  1. Reusable skills are a reasonable way to transfer experience across cases, and the results show modest gains on both test and OOD tasks. However, the paper does not explain why experience must be stored as reusable workflow skills, nor does it compare against replanning, trajectory retrieval, or contextual memory under matched information and compute budgets.

  2. The construction of the 16 task categories is insufficiently specified. The grouping criteria, annotation procedure, category boundaries, and overlaps are unclear, and several categories appear closely related. The paper also does not explain how real-world requests would be routed, especially for multi-category or ambiguous tasks. In addition, learning one shared workflow per category may reduce procedural and output diversity by imposing similar tool choices, shot structures, or narrative patterns. Since diversity is not evaluated and per-case workflow variations are not shown, it is unclear whether these skills act as flexible policies or rigid templates.

Questions for authors
  1. Could you provide the exact prompts and skill templates used by the creator, optimizer, merge, execution, and evaluation agents, together with examples of skills before and after evolution?

  2. Why is experience stored specifically as reusable workflow skills? How would the method compare with on-the-fly replanning, trajectory retrieval, or contextual memory under matched history and compute budgets?

  3. Can you isolate the contribution of creator-skill evolution—for example, by separately freezing the creator or composition skill, and by comparing against direct workflow generation by a standard LLM?

  4. How were the 16 task categories defined, and how would an arbitrary, ambiguous, or multi-category request be routed in practice? Were category overlap and annotation agreement examined?

  5. Does sharing one composition skill across all cases in a category reduce procedural, narrative, or visual diversity? Could you report diversity measurements or examples showing how workflows vary across cases?

  6. How many unique human annotators participated in the study? Could you report ranking-level inter-annotator agreement, confidence intervals, and significance tests for the main human-ranking results?

  7. Do the reported resource statistics include the offline costs of skill creation, evolution, evaluation, and creator merging? If not, what is the complete cost of obtaining and deploying the evolved skills?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.