acceptodds
Withdrawn from ICLR 2027

CreatorBench: Benchmarking Agents on Multimodal Creative Production from Brief to Deliverable

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Content studios turn structured briefs into finished creative deliverables—posters, advertisements, short dramas, interactive videos—through multi-step pipelines under hard specification constraints; a single failure in asset usage, text rendering, consistency, or technical compliance can reject the entire delivery. No existing benchmark takes a real business brief as input, covers both image and video, and evaluates the finished deliverable against that brief. CreatorBench fills this gap with 120 tasks (60 image, 60 video) abstracted from commercial studio workflows and organized into 14 task families. Each task provides a structured brief, source assets, brand facts, delivery specifications, and multi-step constraints; the agent autonomously plans, produces, and delivers final files. Human experts decompose each task into a weighted checklist of verifiable requirements. Scoring centres on per-case requirement fulfilment: a primary and an audit VLM cross-score each checklist item blindly. Deterministic corrections—objective-metric verification, defect caps, and delivery-specification gates—yield the requirement-satisfaction score R; a joint technical-quality and visual-appeal modifier produces the final score. The resulting automated ranking aligns with human Good/Same/Bad pref- erence judgments. We evaluate five systems under two toolsets with different generation backends. The strongest system (Codex) reaches 59.67/100 under Toolset A; no system exceeds 60 and the Baseline scores 17.78/100. These re- sults suggest that reliably turning a business brief into an acceptable deliverable still lies beyond current agent capabilities.

Primary area:
datasets and benchmarks
Keywords:
Creative Agents, Multimodal Creative Production, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RUjnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CreatorBench evaluates whether agents can turn structured commercial production briefs and source assets into complete image/video deliverables. It contains 120 tasks, split equally between image and video, across 14 families, including coordinated advertising layouts, precision editing, narrative video, and branching videos with a player. Five domain experts construct weighted requirement checklists. Two VLM judges score each item, followed by evidence gating, defect caps, and deterministic delivery-specification checks; a separate technical-quality/visual-appeal factor modifies the requirement score. Four agent systems and a direct-generation baseline are evaluated with two generation toolsets. The reported strongest score is 59.67/100, and a 20-case human preference study reproduces the five-system ranking. The contribution is a useful delivery-level task suite and a detailed scoring framework, although the current evidence leaves substantial uncertainty about score validity and the reproducibility of the reported comparisons.

Critical strengths
  1. CreatorBench covers 120 tasks across 14 image/video families, evaluating joint fulfilment of production briefs rather than isolated outputs . Asset, text, temporal, and cross-file requirements make it relevant to practical creative workflows.
  2. Weighted checks, clause-level evidence, specification gates, and defect caps make the scoring protocol explicit. The blind 20-case human study supports the broad five-system ranking, although it does not establish fine-grained score validity.
Critical weaknesses
  1. The 15-case author-target calibration and 20-case ranking study by the checklist authors lack independent, held-out item-level or delivery-acceptance validation. Unmeasured judge-order sensitivity further limits interpreting sub-60 scores as evidence that commercial delivery is beyond current capabilities.
  2. Section 4.2's fixed denominator conflicts with Table 3's variable valid-case denominators, while Appendix A.16 leaves unclear whether retries affected scored outputs or only cost logs. Missing per-case records and pre-specified budgets make system comparisons difficult to reconstruct.
Questions for authors

same as weakness

Reviewer_jUBNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CreatorBench, a benchmark for evaluating agents on end-to-end multimodal creative production. It comprises 120 image and video tasks across 14 categories, emphasizing real-world requirements and final deliverable quality. The evaluation framework combines expert-defined checklists, dual-VLM judgments, and deterministic verification. Experiments with five systems across two toolsets reveal substantial improvements over a No-Agent Baseline, while highlighting limitations in requirement fulfillment.

Critical strengths
  1. The benchmark reflects realistic commercial workflows, evaluating end-to-end creative production rather than isolated generation tasks.
  2. The framework combines requirement satisfaction, perceptual quality, and delivery-specification compliance to assess final deliverables.
Critical weaknesses
  1. The agents use different backbone models and operate without standardized token, tool-call, or time budgets. Moreover, the No-Agent Baseline is restricted to single-turn generation, while agents can iteratively invoke tools and refine outputs. Consequently, the reported gains cannot be clearly attributed to agentic planning rather than differences in model capabilities or computational resources.

  2. Section 4.2 specifies a fixed denominator of 60 tasks per modality, whereas Table 3 uses variable denominators based on valid cases. Section 4.4 reports Qoder and Workbuddy scores of 44.10 and 42.51, inconsistent with Table 3 (43.51 and 42.48). Additionally, task IDs in Figure 1 do not match those in Appendix A.1. These inconsistencies undermine the reliability and reproducibility of the results.

  3. CreatorBench contains only 120 tasks across 14 categories, with substantial imbalances (e.g., 20 Precision Image Editing tasks versus 2 Multi-format Asset Adaptation tasks). This may bias overall rankings toward overrepresented categories and limit generalizability. Furthermore, the lack of performance analysis by task category, difficulty, and deliverable complexity makes it difficult to assess agents' specific strengths and limitations.

Questions for authors

See weaknesses above

Reviewer_Sv8zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CreatorBench, with tasks sourced from commercial visual production workflows, and design corresponding checklists for evaluation. Unlike conventional generation benchmarks, CreatorBench evaluates the complete deliverable instead of single generated outputs.

Critical strengths
  • Traditional multimodal benchmarks typically regard generated content as the final objective, whereas CreatorBench extends the evaluation focus from single-generation capabilities to the end-to-end task completion capabilities of agents, making it more closely aligned with real-world content production workflows.
  • The 120 tasks cover multiple modalities, i.e., image and video.
Critical weaknesses
  • The number of samples across the 14 task families is imbalanced, with the smallest category containing only 2 tasks and the largest containing 20 tasks, which affects the diagnostic capability of the benchmark.
  • The authors should provide a more comprehensive comparison with existing creativity and generation benchmarks, discuss how the focus of this benchmark differs from theirs, and clarify how it differs from simply combining existing benchmarks.
Questions for authors
  • Are 120 tasks really sufficient to evaluate such complex scenarios? Each task ultimately includes only a few configurations, which limits diversity.
  • How can the alignment between this benchmark and real-world scenarios be justified?
  • How do the authors justify the scientific validity of the scoring weights? Are there more heuristic approaches to calculating them?

I would consider raising my score if the major concerns are adequately addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.