- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DesignerBench, a benchmark for evaluating image generation models in professional design settings. It contains 2,000 instances across seven task families, covering asset creation, extraction, composition, editing, rendering, dissection, and design-language extension. Instead of assigning a single holistic quality score, the benchmark uses sample-specific checklists to assess task completion and preservation constraints. A vision-language model evaluates individual criteria using a four-level rubric, and the scores are aggregated at the dimension and task levels. The authors evaluate 14 image generation models and report substantial performance differences across tasks, with multi-reference composition and structured fidelity remaining particularly challenging. They also conduct a human evaluation to assess agreement with the automated judge.
Critical strengths
-
Practical task coverage. The benchmark covers several design operations that are not well captured by standard text-to-image evaluation, particularly element extraction, multi-reference composition, and design-language extension. I find this broader evaluation setting useful, as a model that generates visually appealing images may still fail to preserve the content or structure required by a design task.
-
Detailed evaluation and empirical analysis. The sample-specific checklists provide more diagnostic information than a single preference score. The evaluation across 14 models is reasonably extensive, and the task-level and dimension-level results reveal meaningful differences between models. In particular, the analysis of composition failures and increasing reference counts is informative. The human evaluation is also a useful step toward validating the proposed scoring procedure.
Critical weaknesses
-
The contribution relative to existing design benchmarks could be better established. Prior benchmarks such as IDEA-Bench have explored professional design tasks, while BizGenEval has investigated fine-grained evaluation of commercial visual content generation. DesignerBench offers broader coverage and a more explicit organization around design workflows, but the paper does not sufficiently demonstrate what additional insights this organization provides beyond combining several related evaluation tasks. Moreover, the experiments evaluate individual operations rather than complete multi-stage workflows. This does not invalidate the benchmark, but it makes the workflow-centric claim somewhat stronger than the current evidence supports.
-
The reliability of the automated evaluation needs stronger validation. The reported human–VLM agreement is encouraging, but the main result is based on correlations between model-level averages. This does not establish that the judge is equally reliable at the individual sample or checklist-criterion level. There are also inconsistencies in the reported human evaluation counts: Section 4.4 mentions 864 judging tasks, Appendix D.7 describes 90 task IDs and 720 outputs, while Table D.4 reports 120 task IDs and 960 outputs. These numbers may refer to different evaluation units, but their relationship is not explained. Finally, the paper does not report sensitivity to the mean–minimum aggregation parameter or results from an independent judge. These omissions make it difficult to assess how stable the reported comparisons are.
Questions for authors
-
My main concern is the contribution of the workflow-centric formulation. Beyond broader task coverage, what distinct diagnostic insights does DesignerBench provide compared with existing design-oriented benchmarks such as IDEA-Bench and BizGenEval? Since the paper emphasizes a workflow-centric formulation, could the authors also provide a representative multi-stage evaluation (e.g., asset creation, composition, and editing) to demonstrate whether evaluating connected design operations reveals additional failures beyond assessing each task independently? I consider this the most important point to clarify, as it directly affects my assessment of the paper's main contribution.
-
The benchmark is intended to reflect real-world professional design workflows, but many of its tasks and instructions are constructed through a VLM-assisted pipeline. How do the authors establish that these tasks and their constraints are representative of actual design practice? Could the authors provide a validation using real designer-authored briefs or professional design projects to demonstrate that the benchmark reflects practical design requirements?
-
How robust are the reported evaluation results to the choice of scoring protocol? In particular, do the main model rankings remain consistent when using a different VLM judge or changing the mean–minimum aggregation parameter?