- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a general-purpose agent harness can turn one high-level instruction into a multi-clip video by composing a fixed set of media tools, and whether that workflow can be rewritten from past runs. VideoWeaver has three parts: a benchmark of 285 cases in 16 categories (119 train / 123 test in 13 categories, plus 43 cases in 3 held-out categories); an agent-as-judge that scores 6 process metrics and 7 output metrics from the execution trace and the final video; and a skill-evolution procedure that rewrites category-level composition skills and creator skills, then merges creator skills for unseen categories. Final comparison uses a three-annotator ranking of nine systems. The judge is used mainly as the rewriting signal. On OpenClaw + Seed2.0, adding a composition skill raises the process average from 0.765 to 0.922 while the output average stays near 0.55. Human ranking then rises from 2.96 to 4.59 with self-evolution and to 5.23 with judge feedback. A handwritten expert skill remains highest, at ranking 5.63 and output average 0.620. Prompt following stays low across harnesses, from 0.25 to 0.46.
Critical strengths
- The process/output split is the right comparison, and the paper follows through on it by using human ranking for the final ranking rather than the LLM judge.
- The experimental ladder is substantial and the controls are the right ones. Media tools stay fixed while the harness, backbone, and skill condition change, from no composition skill through a generated skill, self-evolution, judge feedback, and an expert skill. The paper reports that the expert skill still wins, and it includes cost and an order ablation. That makes the study usable as a reference point.
Critical weaknesses
- The abstract says that skill evolution improves output quality and generalizes, and that judge feedback adds a further gain. The measurements support a narrower claim. On human ranking, most of the gain is self-evolution without the judge, from 2.96 to 4.59. Feedback reaches 5.23, and the expert skill is still ahead at 5.63. The output-average movement is about one to two points (0.552 to 0.567 to 0.582) and is largely Format Requirements, which rises from 0.728 to 0.913 between the composition skill and feedback. Over the same change, audio consistency falls from 0.567 to 0.448, and visual consistency and reference fidelity also fall. Across three disclosure orders in Appendix E, feedback is only +0.019 output average over self-evolution, with standard deviations of 0.019 and 0.042; prompt requirements and plot coherence favor self-evolution. Pairwise transfer is 25.3% win / 58.2% tie / 16.5% loss on test categories and 23.8% / 59.7% / 16.5% on the held-out categories. Brand Promotion has more losses than wins. I do not see a stable effect of the judge, or reliable transfer to unseen categories.
The written objective does not match the procedure. Equations 2–4 are argmax objectives over executed skills. In Algorithm 1, every stored trace comes from the initial composition skill, and a revised skill is not run again before it is frozen. The test numbers are real executions of the final skill text, but the paper does not show that rewriting increased the training objective it writes down. The scores used for rewriting are also the metrics on which humans disagree most.
- The community-facing contribution is thinner than the framing. The learned objects are skill texts, but the paper never shows a composition skill or creator skill before and after an edit, so a reusable procedure cannot be distinguished from checks for duration, planning, and successful tool calls. Video, image, speech, and understanding all go through one ByteDance / Volcengine stack, so the harness comparison is tied to that API. Tasks were filtered to about one minute, so this is multi-clip composition, whereas Section 2.1 discusses minute-scale to hour-scale generation. The three held-out categories sit close to the training categories. The reproducibility statement describes the procedure and does not say that cases, traces, judge prompts, or skills will be released. User-submitted reference media also needs a license before this can function as a benchmark. As written, the supported finding is that a rewritten workflow outranks the initial workflow under human ranking, and a human-written workflow ranks higher.
Questions for authors
- In Equation 1, how many cases are in 𝑋, and what is the inter-annotator agreement on the rankings? Does the gap among Self-Evolution (4.59), Evolution with Feedback (5.23), and Expert (5.63) remain under resampling over cases and annotators?
- Please show one composition skill and its creator skill before and after evolution, next to the feedback that caused the edits. I want to see whether the edits change shot planning and reference use, or mainly add checks that raise Format Requirements and the process metrics.
- Does Algorithm 1 match the implementation: revised skills are not re-executed on training cases before being frozen? If they are re-executed, how does the training value of 𝜙 change after each update?
- Will the cases, reference media, traces, metric prompts, and evolved skills be released, and under what terms can the user-submitted assets be redistributed? My assessment of the benchmark contribution depends on this answer.
- Table 7 shows large gaps in relative generation time and token use. If video-generation calls or wall-clock time are matched, does the ranking gap among Composition Skill, Self-Evolution, and Evolution with Feedback remain?