acceptodds
Withdrawn from ICLR 2027

DesignerBench: Benchmarking Image Generation for Real-World Design Workflows

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image generative models are increasingly used in professional design, yet evaluation for design applications has not matured into a coherent practice attuned to real design workflows. Evaluation of generative systems in design is not a question of overall output preference, but of whether the operational requirements of a given workflow are satisfied. Existing benchmarks, however, remain largely confined to text-to-image synthesis and generic editing, leaving workflow-specific failures largely unexamined. To close this gap, we introduce DesignerBench, a workflow-centric benchmark that evaluates generative models as design collaborators, spanning 2,000 curated instances across seven design task families: asset creation, element extraction, multi-source composition, professional editing, rendering, multi-view dissection, and design language extension, which together reflect the canonical progression of real design projects. Rather than reducing evaluation to a single holistic score, DesignerBench scores each output against a sample-specific, multidimensional checklist of task-specific objectives and preservation constraints, using an evidence-first vision-language judge with a four-level rubric. Our evaluation of 14 state-of-the-art models reveals substantial variation in performance across design tasks, with rankings shifting according to the operation under evaluation. These results challenge the notion of a single best generative model, underscoring the need for workflow-aware evaluation.

Primary area:
datasets and benchmarks
Keywords:
benchmark, quality assessment, graphic design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p8NuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DesignerBench, a benchmark for evaluating image generation models in professional design settings. It contains 2,000 instances across seven task families, covering asset creation, extraction, composition, editing, rendering, dissection, and design-language extension. Instead of assigning a single holistic quality score, the benchmark uses sample-specific checklists to assess task completion and preservation constraints. A vision-language model evaluates individual criteria using a four-level rubric, and the scores are aggregated at the dimension and task levels. The authors evaluate 14 image generation models and report substantial performance differences across tasks, with multi-reference composition and structured fidelity remaining particularly challenging. They also conduct a human evaluation to assess agreement with the automated judge.

Critical strengths
  1. Practical task coverage. The benchmark covers several design operations that are not well captured by standard text-to-image evaluation, particularly element extraction, multi-reference composition, and design-language extension. I find this broader evaluation setting useful, as a model that generates visually appealing images may still fail to preserve the content or structure required by a design task.

  2. Detailed evaluation and empirical analysis. The sample-specific checklists provide more diagnostic information than a single preference score. The evaluation across 14 models is reasonably extensive, and the task-level and dimension-level results reveal meaningful differences between models. In particular, the analysis of composition failures and increasing reference counts is informative. The human evaluation is also a useful step toward validating the proposed scoring procedure.

Critical weaknesses
  1. The contribution relative to existing design benchmarks could be better established. Prior benchmarks such as IDEA-Bench have explored professional design tasks, while BizGenEval has investigated fine-grained evaluation of commercial visual content generation. DesignerBench offers broader coverage and a more explicit organization around design workflows, but the paper does not sufficiently demonstrate what additional insights this organization provides beyond combining several related evaluation tasks. Moreover, the experiments evaluate individual operations rather than complete multi-stage workflows. This does not invalidate the benchmark, but it makes the workflow-centric claim somewhat stronger than the current evidence supports.

  2. The reliability of the automated evaluation needs stronger validation. The reported human–VLM agreement is encouraging, but the main result is based on correlations between model-level averages. This does not establish that the judge is equally reliable at the individual sample or checklist-criterion level. There are also inconsistencies in the reported human evaluation counts: Section 4.4 mentions 864 judging tasks, Appendix D.7 describes 90 task IDs and 720 outputs, while Table D.4 reports 120 task IDs and 960 outputs. These numbers may refer to different evaluation units, but their relationship is not explained. Finally, the paper does not report sensitivity to the mean–minimum aggregation parameter or results from an independent judge. These omissions make it difficult to assess how stable the reported comparisons are.

Questions for authors
  1. My main concern is the contribution of the workflow-centric formulation. Beyond broader task coverage, what distinct diagnostic insights does DesignerBench provide compared with existing design-oriented benchmarks such as IDEA-Bench and BizGenEval? Since the paper emphasizes a workflow-centric formulation, could the authors also provide a representative multi-stage evaluation (e.g., asset creation, composition, and editing) to demonstrate whether evaluating connected design operations reveals additional failures beyond assessing each task independently? I consider this the most important point to clarify, as it directly affects my assessment of the paper's main contribution.

  2. The benchmark is intended to reflect real-world professional design workflows, but many of its tasks and instructions are constructed through a VLM-assisted pipeline. How do the authors establish that these tasks and their constraints are representative of actual design practice? Could the authors provide a validation using real designer-authored briefs or professional design projects to demonstrate that the benchmark reflects practical design requirements?

  3. How robust are the reported evaluation results to the choice of scoring protocol? In particular, do the main model rankings remain consistent when using a different VLM judge or changing the mean–minimum aggregation parameter?

Reviewer_tmKROpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a workflow-centric benchmark comprising 2,000 samples across seven distinct tasks and proposes an evidence-first vision-language judge for evaluation. The authors evaluate 14 state-of-the-art (SOTA) models. Compared with previous research on design-oriented image generation and benchmarks focusing on specific domains, this work presents a more comprehensive and fine-grained benchmark. Its practice-oriented, multi-type, and multidimensional evaluation framework is more closely aligned with the principles and workflows of professional designers, rather than merely assessing the generation quality of image generation systems.

The paper also provides a comprehensive evaluation and comparative analysis of existing SOTA open-source and closed-source models, establishing a useful reference benchmark for future developments in design-oriented image generation.

Critical strengths
  1. The task definitions and categorization scheme introduced in this design benchmark provide valuable references and guidance for future evaluations of design-related image generation.

  2. The paper presents a comprehensive investigation of existing open-source and closed-source models, establishing a solid reference for model selection in future evaluations of design-related tasks.

  3. Compared with conventional evaluation metrics, the proposed evaluation framework leverages language-model-based criteria to assess design quality. This approach provides useful insights for developing more effective language-model-based evaluation benchmarks in the future.

  4. The proposed material collection methodology has considerable potential value for further benchmark analysis, expansion, and development.

Critical weaknesses
  1. The authors should provide a detailed table summarizing the dataset's scale and characteristics, including the sample distributions and attributes of individual tasks and categories. A comparison with existing datasets and benchmarks should also be included.

  2. The authors claim that their proposed dataset is larger than existing datasets, noting that previous datasets generally contain no more than 1,000 instances. However, I do not consider an increase from 1,000 to 2,000 samples to represent an order-of-magnitude improvement, as both remain within the same order of magnitude. Moreover, since the proposed benchmark covers multiple tasks with only 2,000 samples in total, the number of samples allocated to each task is relatively limited, and the distribution across tasks appears imbalanced. Many existing datasets contain approximately 1,000 samples dedicated to a single task. In contrast, some tasks in the proposed benchmark contain only around 200 samples, which is substantially fewer than those in existing task-specific datasets. This raises concerns about the statistical reliability and representativeness of the proposed benchmark.

  3. I noticed that the construction of the benchmark relies heavily on the image generation and recognition capabilities of generative models, rather than on authentic design examples created by professional human designers. This appears to contradict the design principles emphasized in the paper. Furthermore, the instructions and prompts for individual tasks appear to be constructed according to the preferences or tendencies of generative models, which may not fully reflect the principles and practices of professional designers.

  4. The practical contribution of the benchmark is somewhat limited in its current form. Expanding the dataset and incorporating authentic design cases created by human designers would significantly improve its practical relevance, applicability, and value as a reference for future research.

  5. The paper uses GPT-5.4 as the VLM-based evaluation judge, and the corresponding GPT model achieves relatively high scores. It remains unclear whether this is related to potential evaluator bias. The authors should investigate how the evaluation results and model rankings change when using alternative judges, such as Gemini or other multimodal large language models (MLLMs). The current paper lacks sufficient experiments to address this concern.

  6. Although the paper proposes new evaluation criteria to assess design quality more comprehensively, it lacks comparisons with conventional metrics for image generation quality. The authors should incorporate established evaluation metrics as additional references, such as aesthetic quality, layout accuracy, and adherence to instructions in style transfer or editing tasks. Such comparisons would help clarify the advantages and limitations of the proposed evaluation framework.

  7. The authors should provide concrete examples of generated outputs from at least one vision-language or image generation model, together with their corresponding evaluation scores. I could not find a clear visualization linking specific model outputs to their assigned scores, making it difficult to understand how the proposed evaluation criteria operate in practice.

Questions for authors
  1. Inconsistency between Figure 2 and the main text: Figure 2 illustrates a workflow consisting of Phase 1, Phase 2, and Phase 3. However, the corresponding explanations in the main text do not follow the sequence presented in the figure. Could the authors clarify the relationship between these phases and the descriptions in the paper?

  2. Reliability of automated layout and element extraction: The paper mentions using GPT and Qwen to extract layout information and elements from images. If errors occur during this process, are the extracted annotations manually reviewed and corrected by human annotators? Could such errors in the automated annotation pipeline affect the overall quality and reliability of the benchmark dataset?

  3. Generation of scorable criteria: How are the scorable criteria generated? The authors should provide concrete examples illustrating the generation process and how these criteria are applied during evaluation.

  4. Lack of support for visual prompts: The proposed benchmark does not appear to support tasks involving visual prompts, such as subject-driven editing and other visually conditioned design tasks. Why were these tasks excluded, and how could the proposed evaluation framework be extended to accommodate them?

  5. Statistics on text-related editing tasks: I would appreciate it if the authors could provide statistics on the amount of text involved in text-related image editing tasks, including the number of words or text elements per image and their overall distribution across the dataset.

  6. Insufficient analysis of model performance differences: Although the paper introduces a comprehensive benchmark and reports evaluation results for multiple models across different tasks, it provides limited analysis of why certain models perform substantially worse than others on specific tasks. For example, GPT Image 1.5 achieves a score of 0.674 on Dissection Production, which is considerably lower than that of GPT Image 2, while achieving a higher score on the anime-related task. What factors might explain these contrasting results? A more detailed analysis would help readers better understand the capabilities and limitations of different models.

  7. Failure case analysis: The authors should provide more detailed analyses of failure cases, ideally including representative visual examples and explanations of the underlying reasons for failure. This would offer additional insights into the limitations of current image generation models and the challenges posed by the benchmark.

Reviewer_ZozYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to systematically evaluate the capabilities of image generation models in aiding in real-world design workflows.

The paper’s major contributions are: 1) a workflow-centric benchmark that spans 7 typical tasks arising in real-world workflows, which supports comprehensive, fine-grained performance assessment of image generation models across multiple workflow stages; 2) a checklist-based evaluation protocol using VLMs; 3) a thorough empirical study of 14 state-of-the art models on the benchmark.

Critical strengths
  1. Systematic evaluation of image generation model’s performance on real design tasks is an important problem to investigate, and the proposed methods, including the data construction pipeline and evaluation protocol, are well designed.

  2. Findings from the experiments are interesting, and can provide useful guidance for future research to select and improve models for facilitating practical design workflows.

Critical weaknesses

The scale of the benchmark is limited. The current benchmark only contains 2k task instances, which are insufficient to support large-scale testing. According to the data construction pipeline (Section 3.2), the number of instances seems to be mainly restricted by the number of collected design samples. However, existing public design datasets often have about 20k or more samples, which should make it easy to build a much larger benchmark. It is unclear why only 2k instances are included in the current benchmark. To make the benchmark more useful for the community, it is expected to contain at least 10k instances, making it one order of magnitude larger than similar existing datasets (with 1k instances).

Questions for authors

How many design samples are curated in the benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.