CreatorBench: Benchmarking Agents on Multimodal Creative Production from Brief to Deliverable

Content studios turn structured briefs into finished creative deliverables—posters, advertisements, short dramas, interactive videos—through multi-step pipelines under hard specification constraints; a single failure in asset usage, text rendering, consistency, or technical compliance can reject the entire delivery. No existing benchmark takes a real business brief as input, covers both image and video, and evaluates the finished deliverable against that brief. CreatorBench fills this gap with 120 tasks (60 image, 60 video) abstracted from commercial studio workflows and organized into 14 task families. Each task provides a structured brief, source assets, brand facts, delivery specifications, and multi-step constraints; the agent autonomously plans, produces, and delivers final files. Human experts decompose each task into a weighted checklist of verifiable requirements. Scoring centres on per-case requirement fulfilment: a primary and an audit VLM cross-score each checklist item blindly. Deterministic corrections—objective-metric verification, defect caps, and delivery-specification gates—yield the requirement-satisfaction score R; a joint technical-quality and visual-appeal modifier produces the final score. The resulting automated ranking aligns with human Good/Same/Bad pref- erence judgments. We evaluate five systems under two toolsets with different generation backends. The strongest system (Codex) reaches 59.67/100 under Toolset A; no system exceeds 60 and the Baseline scores 17.78/100. These re- sults suggest that reliably turning a business brief into an acceptable deliverable still lies beyond current agent capabilities.