- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CreatorBench evaluates whether agents can turn structured commercial production briefs and source assets into complete image/video deliverables. It contains 120 tasks, split equally between image and video, across 14 families, including coordinated advertising layouts, precision editing, narrative video, and branching videos with a player. Five domain experts construct weighted requirement checklists. Two VLM judges score each item, followed by evidence gating, defect caps, and deterministic delivery-specification checks; a separate technical-quality/visual-appeal factor modifies the requirement score. Four agent systems and a direct-generation baseline are evaluated with two generation toolsets. The reported strongest score is 59.67/100, and a 20-case human preference study reproduces the five-system ranking. The contribution is a useful delivery-level task suite and a detailed scoring framework, although the current evidence leaves substantial uncertainty about score validity and the reproducibility of the reported comparisons.
Critical strengths
- CreatorBench covers 120 tasks across 14 image/video families, evaluating joint fulfilment of production briefs rather than isolated outputs . Asset, text, temporal, and cross-file requirements make it relevant to practical creative workflows.
- Weighted checks, clause-level evidence, specification gates, and defect caps make the scoring protocol explicit. The blind 20-case human study supports the broad five-system ranking, although it does not establish fine-grained score validity.
Critical weaknesses
- The 15-case author-target calibration and 20-case ranking study by the checklist authors lack independent, held-out item-level or delivery-acceptance validation. Unmeasured judge-order sensitivity further limits interpreting sub-60 scores as evidence that commercial delivery is beyond current capabilities.
- Section 4.2's fixed denominator conflicts with Table 3's variable valid-case denominators, while Appendix A.16 leaves unclear whether retries affected scored outputs or only cost logs. Missing per-case records and pre-specified budgets make system comparisons difficult to reconstruct.
Questions for authors
same as weakness