acceptodds
Under review as a conference paper at ICLR 2027

InfoDiag-Bench: A Scenario-aware Diagnostic Benchmark for Deliverable Infographic Generation

Abstract

While modern text-to-image (T2I) models produce visually compelling natural images, generating infographics suitable for practical use remains challenging. Such images require accurate rendering of dense text and numerical content, and coherent topological structures, which vary across different tasks. Consequently, infographic evaluation also requires pronounced inter-domain diversity across application scenarios. Furthermore, assessing practical deliverability necessitates distinguishing tolerable deviations from delivery-blocking errors. To address these challenges, we propose InfoDiag-Bench, a scenario-aware infographic benchmark spanning 663 realistic test cases across 14 application scenarios. Central to our evaluation framework is a set of predefined, evidence-grounded error types designed to systematically separate tolerable flaws from delivery-blocking ones. Across 10 open-source and close-source models, we find that the inability to preserve topological relationships among elements is the primary cause of delivery failure. Human evaluation further demonstrates a strong alignment between our automated judgements with human anticipation.

Primary area:
datasets and benchmarks
Keywords:
Infographic Generation, Text-to-Image Generation, Diagnostic Evaluation, Deliverability Assessment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4Zv2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes InfoDiag-Bench, a scenario-aware benchmark for whether a generated infographic is actually deliverable. It has 663 test cases in 14 scenarios grouped into five information-organization categories, built from web articles and reference infographics through a human-in-the-loop pipeline that produces a generation prompt plus structured key information. An MLLM scores each image on scenario-specific fidelity and visual-quality dimensions and reports evidence-grounded issues from a catalog of 45 error types, each with a severity, and an image is deliverable only if no high-severity error is detected. Ten T2I models are compared, with a human study on four dimensions and on deliverability in two scenarios, and a small comparison against the IGenBench protocol.

Critical strengths
  1. Addresses a practical, under-served question (deliverability) with a concrete, reusable artifact (14 scenarios, 45 error types) yielding actionable diagnostic reports.

  2. Sensible data construction provides a model-agnostic view of how deliverability degrades with information density.

  3. The human study improves upon prior zero-tolerance protocols and is transparent about its closed-set limitations.

Critical weaknesses
  1. The severity mechanism lacks practical tolerance. With only 3 of 45 tags as low severity, a single detection rejects the image, causing zero scores driven by unverified false positives without per-tag precision/recall validation.

  2. The taxonomy and judge (Gemini) are derived from and evaluate the same data, risking circularity. A disjoint discovery split and an independent judge are required.

  3. Human validation is too narrow (2/14 scenarios), with agreement rates barely exceeding majority-class baselines and no per-tag error detection validation.

  4. The claim that topology is the main failure cause is unsupported; scenario difficulty better explains performance gaps, and the "Overall" metric pools completed images, biasing results toward easier tasks.

Questions for authors
  1. How does the acceptance rule practically tolerate errors? Please provide severity distributions, per-tag precision/recall, and justify inconsistent blocking tag assignments.

  2. Was error discovery strictly disjoint from evaluation? Please report tag-level judge agreement and the impact of a 50/50 split.

  3. Which results are official given the discrepancies between Table 2 and Table 7? Please clarify the subsets, evaluators, and the "Historical Results" row.

  4. Please provide per-scenario sample sizes, confidence intervals, paired comparison details, and generation settings for closed models.

3 comments

Comment:

W1: The severity mechanism lacks practical tolerance. With only 3 of 45 tags as low severity, a single detection rejects the image, causing zero scores driven by unverified false positives without per-tag precision/recall validation.

Thank you for raising this concern. We clarify the practical tolerance of the severity mechanism in two aspects. We also describe a validation study at the image–error-type-pair level to assess detection accuracy and severity alignment.

  1. The trigger definition for each error type are tolerated. For instance, Table 5 specifies that if the essential categories or metrics remain identifiable and their mapping to the chart is clear, then the absence of axis lines or axis titles, or the misplacement of labels, will not automatically trigger the high-severity error "axis_missing." Similarly, in the definition of “title_text_error”, abbreviation or omitted descriptive words that preserve core meaning shall not trigger this tag. These evidence prove an delivery-blocking error can be triggered only if it compromised the communication of required information or prevented the image from fulfilling its intended use.

  2. The types of errors to be detected depend on the specific task. Candidate error types are derived from the prompt requirements and scenario-specific evaluation rubrics. An individual image is therefore evaluated against the candidate types relevant to its task, rather than against all 45 types in the catalog. Figure 14 shows the candidate types and their assigned severity for each scenario; the number of candidate types ranges from 7 to 19 across the 14 scenarios. Thus, 3/45 can not represent the level of tolerance when evaluating an individual prompt. The tolerance also depends on which tags apply and what evidence is required to trigger them.

  3. Error-type validation against human judgments. We conduct experiments on 299 images randomly sampled from ten generation models, covering at least 20 images per scenario and 2,041 image–error-type pairs in total. Three annotators independently judge whether each assigned error is present and, if present, whether it independently blocks delivery. Table 4(b) reports aggregate detection precision of 76.4%, recall of 85.7%, and F1 of 80.8%. Severity agreement with human blocking judgments is 88.4%, and high-severity blocking-diagnosis precision is 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.

Comment:

W2: The taxonomy and judge (Gemini) are derived from and evaluate the same data, risking circularity. A disjoint discovery split and an independent judge are required.

Thank you for raising this concern. We address it from the following two points.

  1. For different evaluation dataset verification: We validate the effectiveness of our method on IGenBench~\citep{tang2026igenbench} in Section 4.4, whose dataset is different from us and independent of taxonomy. What’s more, we use Doubao Seed 2.0 Pro~\citep{doubaoSeed2Pro} serves as the MLLM evaluator, which is different from the original judger Gemini, for both methods. The results show that our method achieves the Cohen's κ\kappa of 0.828, better than zero-tolerance method IGenBench~\citep{tang2026igenbench} (0.789).

  2. For different judger verification: We reported comparison of different MLLM judgers in Appendix A.3.2. We randomly sample 100 prompts from the evaluation dataset, and generate images by GPT-Image-2, Seedream-5.0 and SenseNova-U1-v3. Then, we report percentage agreement among evaluators on image-level deliverability decisions. The overall agreement rate is 75.10%, indicating that evaluators tend to agree on deliverability judgments for most images. Specifically, we find that the main disagreement is shown on GPT-Image-2 results, GPT5.5 is tend to determined to be undeliverable compared with other evaluators, especially in "text_low_contrast", "card_structure_error" error types.

Comment:

W3: Human validation is too narrow (2/14 scenarios), with agreement rates barely exceeding majority-class baselines and no per-tag error detection validation.

Thank you for raising this important point. We expand the human evaluation to 299 images randomly sampled from ten generation models, covering at least 20 images per scenario and 2,041 image–error-type pairs in total. Three annotators independently judge whether each assigned error is present and, if present, whether it independently blocks delivery. Table 4(b) reports aggregate detection precision of 76.4%, recall of 85.7%, and F1 of 80.8%, together with severity agreement of 88.4% and high-severity blocking-diagnosis precision of 77.0%. This extends diagnostic validation to all 14 scenarios, including Character Relationship. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.

Comment:

W4: The claim that topology is the main failure cause is unsupported; scenario difficulty better explains performance gaps, and the "Overall" metric pools completed images, biasing results toward easier tasks.

Thank you for raising this important point. We address it from the following two points.

  1. We calculate the error type frequency for all error types across the whole benchmark, defined the number of cases in which the error is triggered divided by the number of cases in which that error type is evaluated. Figure 3 exhibit the most 3 high-frequency error types: “relationship edge missing”, “relationship edge error”, and “entity relationship mismatch”, which are all relationship errors. For each of these error types, the error detection rate exceeds 50%. Thus, the topology relationships is the most prominent delivery failure for inforgraphic generations. We have corrected our descriptions in section 4.2, part1.

  2. The imcompletion evaluation comes from API requests rejection to closed-source models—for example, the output images may have been subject to copyright restrictions. Of the 663 test cases, GPT-Image-2 is missing three images (0.45%), one each in Character Relationship (character_relationship_040), PPT (image_0019), and Sequential Operation Guide (seq_ops_005). Seedream 4.5 is missing one image (0.15%) in Knowledge List (knowledge_list_0015). Given that the number of missing entries is extremely small and the occurrences are not concentrated in any single scenario, these test results can be considered representative of the final outcomes. We have fixed “Overall” index to averaging scenario rates instead of pass/(pass+fail) across all the images, which may bring bias caused by the number of images per scene. The rankings and conclusions remain the same as before.

Comment:

Q1: How does the acceptance rule practically tolerate errors? Please provide severity distributions, per-tag precision/recall, and justify inconsistent blocking tag assignments.

  1. Our acceptance rule tolerates deviations through both the triggering conditions and the severity assignments. For example, Table 5 specifies that missing axis lines or unconventional placement do not trigger “axis_missing” when the required categories, scales, and mappings remain clear. The tolerarance of each error can be seen in Table 5. Consequently, the label "high severity" does not imply that every minor deviation within that category will trigger the label..

  2. The catalog contains 42 high-severity and three low-severity types: connector_style_mismatch, icon_missing, and ui_component_detail_deviation. Each scenario uses a subset of 7–19 candidate types; the scenario-specific severity distribution is shown in Figure 14.

  3. We have added a human audit of 299 images from ten generators, covering all 14 scenarios and 2,041 image–error-type pairs. Three annotators, distinct from those who originally assigned severity, independently assessed error presence and whether a present error independently blocks delivery. Table 4(b) reports detection precision of 76.4%, recall of 85.7%, and F1 of 80.8%. The reported pooled severity agreement is 88.4%, and high-severity blocking-diagnosis precision is 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.

Comment:

Q2: Was error discovery strictly disjoint from evaluation? Please report tag-level judge agreement and the impact of a 50/50 split.

Thank you for raising this important point. In our method, the error discovery is not strictly disjoint from the benchmark data while the taxonomy is not applicable solely to our benchmark dataset. We validate this on external evaluation in Section 4.4. To assess the resulting evaluation rules generalize beyond these data, we conducted an external evaluation using 57 prompts sampled from IGenBench, with images generated by three T2I models. On the same 163 valid images, we compared our protocol with IGenBench’s protocol against human deliverability judgments. Both protocols used Doubao Seed 2.0 Pro as the evaluator, providing validation with a different judge from the Gemini model used in our main experiments. Our protocol achieved a Cohen’s kappa of 0.828 against human judgments, compared with 0.789 for IGenBench’s protocol. This demonstrates that our rules retain strong alignment with human delivery decisions on tasks drawn from an external benchmark. The result provides empirical evidence that their usefulness extends beyond the data used for error discovery, addressing the concern that the taxonomy is specific to our own benchmark..

Comment:

Q3: Which results are official given the discrepancies between Table 2 and Table 7? Please clarify the subsets, evaluators, and the "Historical Results" row.

Thank you for raising this important point. Table 2 contains the main benchmark deliverability results for the ten generators, evaluated using Gemini 3.1 Pro Preview. The former Table 7 served a different purpose: it compared evaluator-specific results on sampled outputs from three generators. Its “Historical Results” row referred to the previously computed partial Gemini results.

We have replaced that presentation with Table 7, which reports agreement on matched image-level deliverability decisions among GPT-5.5, Doubao Seed 2.0 Lite, and Gemini 3.1 Pro Preview (same images, same evalution protocol). Table 2 remains the main benchmark result; Table 7 is a supplementary evaluator-consistency analysis.

Comment:

Q4: Please provide per-scenario sample sizes, confidence intervals, paired comparison details, and generation settings for closed models.

Thank you for raising this important point. Appendix A.2.6 now specifies the image-generation settings. GPT-Image-2 uses 1K resolution and quality=high; Gemini-3-Pro and Gemini-3.1-Flash use 2K; Seedream-5.0 uses 1K; and Seedream-4.5 uses 2K. Other parameters are set to their defaults.

Reviewer_b67eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an interpretable evaluation method for image generation models that connects error diagnosis with image-level deliverability assessment, providing a new approach to evaluating the usability of image generation in real-world applications. The authors introduce a small-scale dataset covering 14 application scenarios and 45 predefined, manually designed error types to evaluate the performance and limitations of current state-of-the-art generative models. According to the experimental results, the proposed method achieves higher agreement with human interactive judgments compared to the previous evaluation method, IGenBench. Under the proposed benchmark, the closed-source model GPT-Image-2 demonstrates significant advantages in the vast majority of tasks.

Critical strengths

The problem addressed in this paper has clear practical value. Compared to previous benchmarks and evaluation methods that focus on specific aspects of image quality and prompt effectiveness, this paper focuses on practical issues where errors made by generative models render the generated images undeliverable, and proposes an interpretable evaluation method.

The proposed dataset covers a variety of real-world application scenarios, converting real-world articles and reference images into generation tasks, which provides a certain degree of realism.

The experiments not only report model rankings but also investigate differences across scenarios, prompt length, error distributions, and agreement with human evaluations.

Critical weaknesses

The proposed evaluation method appears to be more of a rule-based aggregation strategy. Moreover, almost all of the manually defined error types are high-severity errors. Severity is not dynamically determined based on the actual delivery requirements of each image, which limits the generalizability of the method.

The paper uses Gemini 3.1 Pro Preview at multiple stages, including prompt generation, key information extraction, and final automated evaluation. Although the authors include manual checks, the framework still exhibits a strong dependence on this model. The reliability of the MLLM judge has not been sufficiently validated.

The reproducibility of the benchmark remains insufficient. The authors do not provide complete configurations sufficient to fully reproduce the experiments, particularly the complete MLLM judge prompts, original key information, per-sample error diagnosis results, generation parameters, and API model invocation settings.

Questions for authors

The authors should fix exactly the same generated images, key information, and error type lists, evaluate them using multiple MLLMs, and report their agreement, the precision/recall of error category detection, and the range of result variations.

What are the design principles behind the error types? Is there a more reasonable way to define severity? Would a larger list of error types be more beneficial for evaluating model deliverability?

5 comments

Comment:

W1: The proposed evaluation method appears to be more of a rule-based aggregation strategy. Moreover, almost all of the manually defined error types are high-severity errors. Severity is not dynamically determined based on the actual delivery requirements of each image, which limits the generalizability of the method.

Thank you for raising this important point. Our method establishes an explicit set of rules for infographic deliverability, grounded in scenario-specific evaluation criteria and the requirements of each generation prompt. These rules assess whether the generated image faithfully communicates the required information. Different scenarios use different candidate error types, while the prompt specifies the content and relationships to be verified in each image. Thus, although severity labels are fixed, their application depends on the actual task requirements and image content.

The predominance of high-severity types does not imply that every deviation causes rejection. Their definitions explicitly allow deviations that preserve the required meaning. For example, Table 5 specifies that missing axis lines, descriptive axis titles, or conventional axis placement do not trigger "axis_missing" when the required categories or scales remain identifiable and their mapping to the chart is clear. Fixed severity labels provide a consistent consequence once an error meets these defined conditions.

Human evaluations on both our own benchmark and IGenBench, an external evaluation set, support the alignment of our method with human decisions on the deliverability of individual images. On our benchmark, Table 4(a) reports average balanced accuracies of 0.851 for Financial Summary and 0.765 for PPT against human deliverability judgments. On the 163 images generated from IGenBench prompts, our method achieved a Cohen’s kappa of 0.828, compared with 0.789 for IGenBench (Section 4.4). Together, these results provide experimental support for our deliverability rules across both evaluation sets.

Comment:

W2: The paper uses Gemini 3.1 Pro Preview at multiple stages, including prompt generation, key information extraction, and final automated evaluation. Although the authors include manual checks, the framework still exhibits a strong dependence on this model. The reliability of the MLLM judge has not been sufficiently validated.

Thank you for raising this concern. We provide three complementary forms of evidence to validate the reliability of the MLLM judge.

  1. Human validation on our benchmark (Section 4.3, Table 4(a)). Three annotators independently evaluated 1,000 images under blinded conditions, covering one evaluation dimension in each of four scenarios. Automated scores achieved a mean Pearson's r of 0.828 with human scores. More directly relevant to deliverability, the average balanced accuracies against human delivery decisions were 0.851 for Financial Summary and 0.765 for PPT. These results validate both scoring and image-level delivery judgments on the assessed subsets. In addition, we have added an experiment--three annotators assessed error presence and independent blocking impact on 2,041 image–error-type pairs from 299 images across 14 scenarios. Table 4(b) reports aggregate detection precision, recall, and F1 of 76.4%, 85.7%, and 80.8%, respectively, severity agreement of 88.4%, and high-severity blocking-diagnosis precision of 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.

  2. Agreement across different judgers (Appendix A.3.2, Table 7). We compared GPT-5.5, Doubao Seed 2.0 Lite, and Gemini on the same generated images using identical key information, candidate error-type lists, and evaluation instructions across all MLLM judges. Three-way agreement on deliverability was 75.10%, with pairwise agreement of 79.59%–87.35%. We also report where disagreements arise: GPT-5.5 more frequently rejects GPT-Image-2 outputs, particularly for text_low_contrast and card_structure_error. This analysis documents both the consistency and the remaining evaluator sensitivity of the protocol.

  3. External validation with a different judge (Section 4.4). Using Doubao Seed 2.0 Pro to evaluate both protocols on the same 163 images generated from IGenBench prompts, our method achieved a Cohen’s kappa of 0.828 against human delivery judgments, compared with 0.789 for IGenBench. Thus, agreement with human judgments also holds when the final evaluator is replaced and the evaluation data come from another benchmark.

These results support the reliability of the judge on the evaluated samples and the use of our protocol with alternative MLLMs. The data generation and annotation process is carried out through a combination of MLLMs and human annotation; the level of human agreement in the evaluation results further confirms the reliability of the MLLM judge.

Comment:

W3: The reproducibility of the benchmark remains insufficient. The authors do not provide complete configurations sufficient to fully reproduce the experiments, particularly the complete MLLM judge prompts, original key information, per-sample error diagnosis results, generation parameters, and API model invocation settings.

Thank you for raising this concern. We provide the error type denifition and triggering condition in Appendix A.2.4 (Table 5), and MLLM judge prompts in Appendix A.2.5 (Figure 29). The generation parameters are added in Appendix A.2.6.

Comment:

Q1: The authors should fix exactly the same generated images, key information, and error type lists, evaluate them using multiple MLLMs, and report their agreement, the precision/recall of error category detection, and the range of result variations.

Thank you for raising this concern. We report the evalution results by different MLLM judgers on same evaluation protocol in Appendix A.3.2.

Appendix A.3.2 and Table 7 report agreement on image-level deliverability decisions for the same images evaluated by GPT-5.5, Doubao Seed 2.0 Lite, and Gemini 3.1 Pro Preview. Three-way agreement is 75.10%, while pairwise agreement ranges from 79.59% to 87.35%. We report that GPT-5.5 more frequently rejects GPT-Image-2 outputs, particularly for text_low_contrast and card_structure_error. Aggregate human–evaluator detection precision and recall are reported in Table 4(b): 76.4% and 85.7%, respectively. In the same human audit, annotators also judged whether a present error independently blocked delivery; severity agreement is 88.4%, and high-severity blocking-diagnosis precision is 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations. Table 7 separately assesses agreement between MLLM judges on image-level deliverability.

Comment:

Q2: What are the design principles behind the error types? Is there a more reasonable way to define severity? Would a larger list of error types be more beneficial for evaluating model deliverability?

Thank you for raising this concern. We answer this question on three points.

  1. Each error type must describe a clearly observable, reusable violation of these requirements, with a distinct definition and explicit triggering conditions. The error type generation prompt for MLLM is shown in Appendix A.2.2, and each error type is grounded in generation prompts, structured key information, and scenario-specific evaluation rubrics.

  2. Severity is defined by whether an error compromises the communication of required information through omission, ambiguity, or inaccuracy, or prevented the image from fulfilling its intended use in the target application scenario. Tolerated deviations are explicitly specified: for example, missing axis lines do not trigger axis_missing when the required categories, scales, and mappings remain clear. Three annotators independently assigned severity labels, achieving 71.1% exact agreement, with final labels determined by majority vote. At the image level, our delivery judgments achieved balanced accuracies of 0.851 and 0.765 in Financial Summary and PPT. On 163 external IGenBench images, agreement with human judgments reached κ = 0.828, compared with 0.789 for IGenBench’s protocol. These results support the current rules is superior to sota delivery rules.

  3. A larger taxonomy is beneficial when it captures distinct delivery-relevant failures that existing types miss. However, prompt-specific instances, and subdivisions without clear diagnostic boundaries can instead introduce redundant or ambiguous diagnoses. Accordingly, our review rules merge synonymous.

Reviewer_8sudOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InfoDiag-Bench, a benchmark for evaluating the practical deliverability of generated infographics. This benchmark contains 664 samples covering 14 applications scenarios, with 78 evaluation dimensions and 45 diagnostic error types. The framework extracts structured key information from generation prompts and uses a multimodal language model to assess information fidelity, visual quality, and specific errors. Each error type receives a predefined severity label, and an image is considered deliverable only when no high-severity error is detected. Experiments evaluate ten image-generation models and identify substantial difficulties with character-relationship diagrams and longer prompts. Human evaluations and a comparison with IGenBench provide initial evidence that the proposed protocol aligns with human judgments.

Critical strengths
  1. The question that this paper investigated is meaningful and practical.
  2. The benchmark covers comprehensive scenarios.
  3. The diagnostic framework is interpretable.
Critical weaknesses
  1. I have one concern about the severity mapping. From my opinion, the current mapping is not convincing. For example, Table 6 classifies table typography deviations and visual-style mismatches as high severity, while missing icons are low severity. Whether these errors block delivery depends on their semantic role and application requirements.
  2. Human evaluation does not cover enough diagnostic claims, such as Character Relationship. Besides, score correlations also do not establish the precision of individual error diagnoses, particularly high-severity false positives that can independently reject an image.
  3. Some conclusions extend beyond the supporting analysis. The detailed evidence for topology as the primary failure source comes mainly from the Character Relationship scenario, rather than a benchmark-wide attribution analysis. Similarly, prompt length is an imperfect proxy for information density. Lower image-level acceptance for longer prompts may partly reflect more opportunities for failure under the conjunctive acceptance rule, even if per-item accuracy remains unchanged.

I am open to discuss during rebuttal and willing to reconsider the score if authors may provide convincing response.

Questions for authors
  1. How were the 99 images selected from the 57-prompt, three-generator experiment? Is the κ improvement statistically reliable?
  2. What proportion of evaluations is incomplete for each generator and scenario, and how closely do different evaluators agree on the same images?
2 comments

Comment:

W1: The severity mapping is not convincing (visual-style mismatches as high severity, while "icon_missing" are low severity), whether the error severity depends on their sematic role and application requirements.

Thank you for raising this important point. We first explain the relationship between the error severity and application scenarios, then confirm the reliability of the severity mapping from two perspective below.

  1. The error severity is determined by its definition and its test scenario requirements. The error severity is the same for all scenarios, while the set of candidate error types varies for each scenario. As elaborated in section3.2.3, the error types for is produced from the t2i prompt annotations, scenario-specific evaluation rubrics and the initial error type pool, so except general types, such as textual errors, graphic errors, and content overflow, the candidate error types for each scenario are different. The "icon_missing" is candidate for only “conference” and "event notice" scenarios, where the text content is the key information to be transmitted while the decorative icons are ignorable, so the severity is low. "visual_style_mismatch" is candidate for "promo poster” and "tour poster" scene, where visual promotion is the key point, so it’s high serverity. The detailed error type-scenarios relationships are illustrated in Table 5 and Figure 14.

  2. Theoretically, an error was considered delivery-blocking (high severity) if it compromised the communication of required information or prevented the image from fulfilling its intended use in the target application scenario, for example causing an omission, ambiguity, or inaccuracy. Based on this rule, three annotators independently annotate the severity using the error-type definitions and images generated by stochastic models. The consistency of three results is high with a exact agreement rate of 71.1% and the severity for disagreement types were determined by majority vote.

  3. Practically, we conducted an additional human evaluation with three annotators who were not involved in the original severity labeling. The study covered 299 images from ten generation models, with at least 20 images per scenario, yielding 2,041 image–error-type pairs for detection assessment. Each annotator independently examined the image and judged whether the assigned error was present and, if present, whether it independently blocked delivery. Among human judgments confirming an error’s presence, the predefined severity agreed with the human blocking decision in 88.4% of cases. These finding directly assesses severity alignment at the image–error-type-pair level by comparing predefined severity with independent human blocking judgments, complementing image-level score correlations.

W2: Some scene human evaluation is missed, such as character relationship and score correlations also do not establish the precision of individual error diagnoses.

We thank the reviewer for highlighting this concern. We address it from the following experiment

We have extended the human evaluation to 299 images from ten generation models, covering all 14 scenarios, including Character Relationship, with at least 20 images per scenario. Three annotators independently assessed whether each assigned error type was present and, if present, whether it independently blocked delivery, yielding 2,041 image–error-type pairs for detection assessment. The audit includes both evaluator-detected and undetected candidate tags, enabling assessment of false positives and missed errors. Against human judgments, the evaluator achieves 76.4% precision, 85.7% recall, and an F1 score of 80.8% (Table 4(b)). Severity agreement with human blocking judgments is 88.4%, and high-severity blocking-diagnosis precision is 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.

Comment:

W3: Conclusion problem: the topology is the primary failure source comes mainly from the Character Relationship scenario, rather than a benchmark-wide attribution analysis. prompt length is an imperfect proxy for information density so the conclusion from this is not accurate.

Thank you for raising this important point. We address it from the following two points.

  1. We calculate the error type frequency for all error types across the whole benchmark, defined as the number of cases in which the error is triggered divided by the number of cases in which that error type is evaluated. Figure 3 exhibit the most 3 high-frequency error types: “relationship edge missing”, “relationship edge error”, and “entity relationship mismatch”, which are all relationship errors. For each of these error types, the error detection rate exceeds 50%. Thus, the topology relationships is the most prominent delivery failure for inforgraphic generations. We have corrected our descriptions in section 4.2, part1.

  2. We agree that prompt length is an imperfect proxy for information density. We have therefore revised Section 4.2 and Table 3 to describe prompt-length groups rather than information-density levels. As illustrated in Table 3, image-level deliverability is generally higher for short prompts than for long prompts, indicating the defect of long-prompt inforpgraphic rendering.

Q1: How were the 99 images selected from the 57-prompt, three-generator experiment? Is the κ improvement statistically reliable?

Three models on 57 prompts produce 171 images theoretically (57 prompts × 3 generators). Among these, 2 cases could not be generated because the text-to-image API rejected the requests on copyright grounds, 6 cases were excluded because the human annotators disagreed on the delivery decision, and 64 had not been evaluated before submission deadline because of intermittent network failures. The original analysis therefore included the 99 images with both a completed evaluation and an agreed-upon human label.

Now we have resolved the API issues and evaluated the 64 remaining images, bringing the analysis set to 163 images. On these test cases, InfoDiag-Bench achieved a Cohen’s kapp of 0.828 against human judgments, compared with 0.789 for IGenBench. The improvement conclusion is the same as the original analysis. We have clarified this point in Section 4.4

Q2: What proportion of evaluations is incomplete for each generator and scenario, and how closely do different evaluators agree on the same images?

The evaluation covers nearly all test cases, with a small number of images unavailable due to rejected API requests to closed-source models—for example, the output images may have been subject to copyright restrictions. Of the 663 test cases, GPT-Image-2 is missing three images (0.45%), one each in Character Relationship (character_relationship_040), PPT (image_0019), and Sequential Operation Guide (seq_ops_005). Seedream 4.5 is missing one image (0.15%) in Knowledge List (knowledge_list_0015). Given that the number of missing entries is extremely small and the occurrences are not concentrated in any single scenario, these test results can be considered representative of the final outcomes.

We reported comparison of different MLLM judgers in Appendix A.3.2. We randomly sample 100 prompts from the evaluation dataset, and generate images by GPT-Image-2, Seedream-5.0 and SenseNova-U1-v3. Then, we report percentage agreement among evaluators on image-level deliverability decisions. The overall agreement rate is 75.10%, indicating that evaluators tend to agree on deliverability judgments for most images. Specifically, we find that the main disagreement is shown on GPT-Image-2 results, GPT5.5 is tend to determined to be undeliverable compared with other evaluators, especially in "text_low_contrast", "card_structure_error" error types.

Comments on the paper

1 comment

Comment:

We gratefully thank the reviewers for their valuable and constructive comments. We are encouraged that they recognize the practical importance of evaluating infographic deliverability (8sud, b67e, and 4Zv2) and the comprehensive real-world scenario coverage (8sud, b67e). We also appreciate their recognition of the interpretability of our diagnostic framework (8sud and b67e) and the value of a reusable benchmark that produces actionable diagnostic reports (4Zv2). We are glad that the analyses beyond model rankings—including scenario differences, error distributions, and human agreement—are acknowledged as strengths (b67e).

We address the concerns and questions in detail in the individual responses. Based on this feedback, we have revised the manuscript and summarized the main changes as follows:

  1. Additional human validation of error detection and blocking judgments (Sec. 4.3 & Table 4(b), to 8sud, b67e, and 4Zv2): Three independent annotators, distinct from those involved in the original severity labeling, assessed 299 images from ten generators across 14 scenarios, yielding 2,041 image–error-type pairs covering all error types. They judged whether the assigned error was present and, if present, whether it independently blocked delivery. The error detection precision, recall, and F1 are 76.4%, 85.7%, and 80.8%, respectively. Severity agreement with human blocking judgments is 88.4%. Furthermore, 77.0% of evaluator-triggered high-severity errors are confirmed as both present and independently blocking. These experiments directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.

  2. Clarification of error-type principles and severity tolerance (Sec. 3.2.3, Table 5 & Fig. 14, to 8sud, b67e, and 4Zv2): We clarify that error types are grounded in generation prompts, structured key information, and scenario-specific rubrics. Severity reflects whether an error compromises required information or the intended application use. We provide explicit triggering conditions, tolerated deviations, and scenario-specific severity distributions, explaining why a high-severity label does not mean that every minor deviation triggers rejection.

  3. Supplementary evaluation with multiple MLLM judges (Appx. A.3.2 & Table 7, to 8sud, b67e, and 4Zv2): We compare GPT-5.5, Doubao Seed 2.0 Lite, and Gemini 3.1 Pro Preview on the same generated images with identical key information, candidate error-type lists, and evaluation instructions. Three-way deliverability agreement is 75.10%, with overall pairwise agreement ranging from 79.59% to 87.35%, documenting both cross-judge consistency and remaining evaluator sensitivity.

  4. Expanded external validation on IGenBench (Sec. 4.4, to 8sud, b67e, and 4Zv2): We expand the comparison to 163 valid images generated from 57 IGenBench prompts. Using Doubao Seed 2.0 Pro for both protocols, our method achieves Cohen’s κ of 0.828, compared with 0.789 for IGenBench against human deliverability judgments. This provides evidence that the rules can generalize to external datasets and a different evaluator.

  5. Clarification of error-frequency and prompt-length analyses (Sec. 4.2, Fig. 3 & Table 3, to 8sud and 4Zv2): We recomputed the error frequency, which is calculated over the cases evaluated for that error type across the whole benchmark, beyond the Character Relationship scenario. We also replace information-density terminology with prompt-length groups to align the interpretation with the variable actually measured.

  6. Expanded documentation for reproducibility (Appx. A.2.2–A.2.6, to b67e and 4Zv2): We provide the error-discovery prompt, human review rules, complete error definitions and mappings, diagnostic issue-reporting instructions, and image-generation settings to clarify how the evaluation procedure is constructed and applied.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.