- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes InfoDiag-Bench, a scenario-aware benchmark for whether a generated infographic is actually deliverable. It has 663 test cases in 14 scenarios grouped into five information-organization categories, built from web articles and reference infographics through a human-in-the-loop pipeline that produces a generation prompt plus structured key information. An MLLM scores each image on scenario-specific fidelity and visual-quality dimensions and reports evidence-grounded issues from a catalog of 45 error types, each with a severity, and an image is deliverable only if no high-severity error is detected. Ten T2I models are compared, with a human study on four dimensions and on deliverability in two scenarios, and a small comparison against the IGenBench protocol.
Critical strengths
-
Addresses a practical, under-served question (deliverability) with a concrete, reusable artifact (14 scenarios, 45 error types) yielding actionable diagnostic reports.
-
Sensible data construction provides a model-agnostic view of how deliverability degrades with information density.
-
The human study improves upon prior zero-tolerance protocols and is transparent about its closed-set limitations.
Critical weaknesses
-
The severity mechanism lacks practical tolerance. With only 3 of 45 tags as low severity, a single detection rejects the image, causing zero scores driven by unverified false positives without per-tag precision/recall validation.
-
The taxonomy and judge (Gemini) are derived from and evaluate the same data, risking circularity. A disjoint discovery split and an independent judge are required.
-
Human validation is too narrow (2/14 scenarios), with agreement rates barely exceeding majority-class baselines and no per-tag error detection validation.
-
The claim that topology is the main failure cause is unsupported; scenario difficulty better explains performance gaps, and the "Overall" metric pools completed images, biasing results toward easier tasks.
Questions for authors
-
How does the acceptance rule practically tolerate errors? Please provide severity distributions, per-tag precision/recall, and justify inconsistent blocking tag assignments.
-
Was error discovery strictly disjoint from evaluation? Please report tag-level judge agreement and the impact of a 50/50 split.
-
Which results are official given the discrepancies between Table 2 and Table 7? Please clarify the subsets, evaluators, and the "Historical Results" row.
-
Please provide per-scenario sample sizes, confidence intervals, paired comparison details, and generation settings for closed models.
3 comments
Comment:
W3: Human validation is too narrow (2/14 scenarios), with agreement rates barely exceeding majority-class baselines and no per-tag error detection validation.
Thank you for raising this important point. We expand the human evaluation to 299 images randomly sampled from ten generation models, covering at least 20 images per scenario and 2,041 image–error-type pairs in total. Three annotators independently judge whether each assigned error is present and, if present, whether it independently blocks delivery. Table 4(b) reports aggregate detection precision of 76.4%, recall of 85.7%, and F1 of 80.8%, together with severity agreement of 88.4% and high-severity blocking-diagnosis precision of 77.0%. This extends diagnostic validation to all 14 scenarios, including Character Relationship. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.
Comment:
W4: The claim that topology is the main failure cause is unsupported; scenario difficulty better explains performance gaps, and the "Overall" metric pools completed images, biasing results toward easier tasks.
Thank you for raising this important point. We address it from the following two points.
-
We calculate the error type frequency for all error types across the whole benchmark, defined the number of cases in which the error is triggered divided by the number of cases in which that error type is evaluated. Figure 3 exhibit the most 3 high-frequency error types: “relationship edge missing”, “relationship edge error”, and “entity relationship mismatch”, which are all relationship errors. For each of these error types, the error detection rate exceeds 50%. Thus, the topology relationships is the most prominent delivery failure for inforgraphic generations. We have corrected our descriptions in section 4.2, part1.
-
The imcompletion evaluation comes from API requests rejection to closed-source models—for example, the output images may have been subject to copyright restrictions. Of the 663 test cases, GPT-Image-2 is missing three images (0.45%), one each in Character Relationship (character_relationship_040), PPT (image_0019), and Sequential Operation Guide (seq_ops_005). Seedream 4.5 is missing one image (0.15%) in Knowledge List (knowledge_list_0015). Given that the number of missing entries is extremely small and the occurrences are not concentrated in any single scenario, these test results can be considered representative of the final outcomes. We have fixed “Overall” index to averaging scenario rates instead of pass/(pass+fail) across all the images, which may bring bias caused by the number of images per scene. The rankings and conclusions remain the same as before.
Comment:
Q1: How does the acceptance rule practically tolerate errors? Please provide severity distributions, per-tag precision/recall, and justify inconsistent blocking tag assignments.
-
Our acceptance rule tolerates deviations through both the triggering conditions and the severity assignments. For example, Table 5 specifies that missing axis lines or unconventional placement do not trigger “axis_missing” when the required categories, scales, and mappings remain clear. The tolerarance of each error can be seen in Table 5. Consequently, the label "high severity" does not imply that every minor deviation within that category will trigger the label..
-
The catalog contains 42 high-severity and three low-severity types: connector_style_mismatch, icon_missing, and ui_component_detail_deviation. Each scenario uses a subset of 7–19 candidate types; the scenario-specific severity distribution is shown in Figure 14.
-
We have added a human audit of 299 images from ten generators, covering all 14 scenarios and 2,041 image–error-type pairs. Three annotators, distinct from those who originally assigned severity, independently assessed error presence and whether a present error independently blocks delivery. Table 4(b) reports detection precision of 76.4%, recall of 85.7%, and F1 of 80.8%. The reported pooled severity agreement is 88.4%, and high-severity blocking-diagnosis precision is 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.
Comment:
Q2: Was error discovery strictly disjoint from evaluation? Please report tag-level judge agreement and the impact of a 50/50 split.
Thank you for raising this important point. In our method, the error discovery is not strictly disjoint from the benchmark data while the taxonomy is not applicable solely to our benchmark dataset. We validate this on external evaluation in Section 4.4. To assess the resulting evaluation rules generalize beyond these data, we conducted an external evaluation using 57 prompts sampled from IGenBench, with images generated by three T2I models. On the same 163 valid images, we compared our protocol with IGenBench’s protocol against human deliverability judgments. Both protocols used Doubao Seed 2.0 Pro as the evaluator, providing validation with a different judge from the Gemini model used in our main experiments. Our protocol achieved a Cohen’s kappa of 0.828 against human judgments, compared with 0.789 for IGenBench’s protocol. This demonstrates that our rules retain strong alignment with human delivery decisions on tasks drawn from an external benchmark. The result provides empirical evidence that their usefulness extends beyond the data used for error discovery, addressing the concern that the taxonomy is specific to our own benchmark..
Comment:
Q3: Which results are official given the discrepancies between Table 2 and Table 7? Please clarify the subsets, evaluators, and the "Historical Results" row.
Thank you for raising this important point. Table 2 contains the main benchmark deliverability results for the ten generators, evaluated using Gemini 3.1 Pro Preview. The former Table 7 served a different purpose: it compared evaluator-specific results on sampled outputs from three generators. Its “Historical Results” row referred to the previously computed partial Gemini results.
We have replaced that presentation with Table 7, which reports agreement on matched image-level deliverability decisions among GPT-5.5, Doubao Seed 2.0 Lite, and Gemini 3.1 Pro Preview (same images, same evalution protocol). Table 2 remains the main benchmark result; Table 7 is a supplementary evaluator-consistency analysis.
Comment:
Q4: Please provide per-scenario sample sizes, confidence intervals, paired comparison details, and generation settings for closed models.
Thank you for raising this important point. Appendix A.2.6 now specifies the image-generation settings. GPT-Image-2 uses 1K resolution and quality=high; Gemini-3-Pro and Gemini-3.1-Flash use 2K; Seedream-5.0 uses 1K; and Seedream-4.5 uses 2K. Other parameters are set to their defaults.
Comment:
W1: The severity mechanism lacks practical tolerance. With only 3 of 45 tags as low severity, a single detection rejects the image, causing zero scores driven by unverified false positives without per-tag precision/recall validation.
Thank you for raising this concern. We clarify the practical tolerance of the severity mechanism in two aspects. We also describe a validation study at the image–error-type-pair level to assess detection accuracy and severity alignment.
The trigger definition for each error type are tolerated. For instance, Table 5 specifies that if the essential categories or metrics remain identifiable and their mapping to the chart is clear, then the absence of axis lines or axis titles, or the misplacement of labels, will not automatically trigger the high-severity error "axis_missing." Similarly, in the definition of “title_text_error”, abbreviation or omitted descriptive words that preserve core meaning shall not trigger this tag. These evidence prove an delivery-blocking error can be triggered only if it compromised the communication of required information or prevented the image from fulfilling its intended use.
The types of errors to be detected depend on the specific task. Candidate error types are derived from the prompt requirements and scenario-specific evaluation rubrics. An individual image is therefore evaluated against the candidate types relevant to its task, rather than against all 45 types in the catalog. Figure 14 shows the candidate types and their assigned severity for each scenario; the number of candidate types ranges from 7 to 19 across the 14 scenarios. Thus, 3/45 can not represent the level of tolerance when evaluating an individual prompt. The tolerance also depends on which tags apply and what evidence is required to trigger them.
Error-type validation against human judgments. We conduct experiments on 299 images randomly sampled from ten generation models, covering at least 20 images per scenario and 2,041 image–error-type pairs in total. Three annotators independently judge whether each assigned error is present and, if present, whether it independently blocks delivery. Table 4(b) reports aggregate detection precision of 76.4%, recall of 85.7%, and F1 of 80.8%. Severity agreement with human blocking judgments is 88.4%, and high-severity blocking-diagnosis precision is 77.0%. These results directly assess diagnostic accuracy and severity alignment at the image–error-type-pair level, complementing image-level score correlations.
Comment:
W2: The taxonomy and judge (Gemini) are derived from and evaluate the same data, risking circularity. A disjoint discovery split and an independent judge are required.
Thank you for raising this concern. We address it from the following two points.
For different evaluation dataset verification: We validate the effectiveness of our method on IGenBench~\citep{tang2026igenbench} in Section 4.4, whose dataset is different from us and independent of taxonomy. What’s more, we use Doubao Seed 2.0 Pro~\citep{doubaoSeed2Pro} serves as the MLLM evaluator, which is different from the original judger Gemini, for both methods. The results show that our method achieves the Cohen's κ of 0.828, better than zero-tolerance method IGenBench~\citep{tang2026igenbench} (0.789).
For different judger verification: We reported comparison of different MLLM judgers in Appendix A.3.2. We randomly sample 100 prompts from the evaluation dataset, and generate images by GPT-Image-2, Seedream-5.0 and SenseNova-U1-v3. Then, we report percentage agreement among evaluators on image-level deliverability decisions. The overall agreement rate is 75.10%, indicating that evaluators tend to agree on deliverability judgments for most images. Specifically, we find that the main disagreement is shown on GPT-Image-2 results, GPT5.5 is tend to determined to be undeliverable compared with other evaluators, especially in "text_low_contrast", "card_structure_error" error types.