- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SciDiagBench, an evaluation benchmark comprising 165 scientific diagrams across 11 types and 10 domains. The evaluation framework leverages Gemini to generate type-specific criteria and weights, incorporates a HPA mechanism, and utilizes a reference-based relative scoring strategy. The authors conduct multi-dimensional evaluations and stability analyses on 4 generation agents.
Critical strengths
- The paper is well-organized and clearly written.
- The carefully curated dataset holds significant potential value for the community.
Critical weaknesses
- The critique of related work is invalid. The authors argue that Item/Atomic-level methods are difficult to generalize and scale. However, Item-level criteria can be fully scaled using LLM generation, and the scale of 165 diagrams in this work does not demonstrate any scalability advantage. Furthermore, evaluating by category is a well-established practice. This paper degrades the granularity from "diagram-specific Items" to "11 broad category templates," which fails to address the inherent uniqueness of individual diagrams or the issue of sub-categories.
- The reliance on reference images limits the practical applicability of the evaluator. The proposed method requires reference images for relative scoring. However, in real-world agent-assisted diagram generation (e.g., text-to-diagram), reference images are often unavailable. Consequently, this method can only be used for static benchmark ranking; it cannot serve as a critic within agent workflows to provide fine-grained feedback, thereby losing its practical value in improving generation capabilities.
- Lack of analysis regarding the impact of reference image quality on validity and consistency. The paper assumes reference images are the gold standard and does not analyze scenarios where reference images are of poor quality. If the reference images contain defects, it may invalidate the effectiveness of the proposed evaluation method.
- Lack of cross-analysis across scientific domains. Although the dataset covers 10 domains, the experiments only report results broken down by diagram type, without providing domain-level evaluation. Analyzing performance across domains could yield insightful conclusions. Moreover, the paper lacks such insightful takeaways, which could have been beneficial for guiding future agent development.
- Category-level criteria generation may lead to inapplicable criteria for specific diagrams. When a VLM generates criteria based on broad categories, the resulting criteria may not hold for specific diagrams within that category, potentially undermining evaluation reliability.
- The core prompts are not reported, resulting in poor reproducibility.
Questions for authors
no question
4 comments
W4: Assume reference images are the gold standard.
We do not treat reference diagrams as an infallible gold standard. Reference-based evaluation is a widely adopted practice in the LLM-as-a-judge paradigm for reducing evaluation error and improving stability [1, 2, 3]. A reference answer, even if it is not the only valid solution, still gives the judge a stable anchor for comparison and thereby reduces judgment variance. We agree that defective references pose an important risk; we have accounted for this risk in our design and mitigate it in two ways:
- Data-level: To ensure high reference quality, we used rigorous manual screening and careful annotation to curate the 165-instance dataset. In addition, the curation methodology established during this process is designed to filter out low-quality reference diagrams during automated dataset expansion.
- Method-level: Our evaluation framework uses the reference as a comparison baseline rather than as the optimal standard, and does not assume that a reference is flawless or always preferable. In §3.3, the judge receives the source paper and caption and scores both diagrams under the same criteria, with anonymized identities and randomized presentation order. The logical and content accuracy criteria require consistency with the source paper; the reference serves as a comparison anchor rather than a fixed quality ceiling.
[2] Du et al. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. 2026 [3] Zhu et al. AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations. 2026
W5: Lack of cross-analysis across scientific domains.
Our current experiments report results by diagram type because scientific diagrams differ in their visual structures, content organization, and communicative purposes, which directly determine the criteria needed to evaluate their quality.
Different diagram types have distinct disciplinary associations, arising naturally from how various fields use diagrams. As a complementary direction, we plan to construct a dataset with scientific domain as the primary sampling dimension, enabling systematic analysis of domain-specific performance and differences across fields.
W6: The core prompts are not reported.
We will open-source the evaluation framework, including the code, full prompt templates, type-specific criteria, and preference statements in the release documentation. The supporting definitions and evaluation guidance are provided in Appendices B (diagram-type definitions), C (evaluation dimensions and criteria), and D (type-specific evaluation weights) of the paper. The scoring prompt and the type-specific criteria for Timeline is provided below.
Scoring prompt. (Note: Engineering constraints for JSON parsing are omitted via [...] for brevity)
In this template, {criteria_list} contains the criteria for the target diagram type, {visual_intent} is the target caption, and {content} is the source paper text. The two diagrams are supplied as images in an anonymized, randomized order.
<system_role>You are a strict, meticulous, and objective scientific diagram evaluation expert. You excel at using specific assessment criteria to deeply compare two scientific diagrams generated for the same visual intent, providing precise scores and clear justifications based on the provided research paper content.</system_role>
<user_prompt>
**Task Background**
There is a scientific diagram generation task, and you need to evaluate two diagrams created for this task. We will assess the diagrams across three dimensions: Structure, Content, and Aesthetics.
**Diagrams to Evaluate**
The two diagrams you need to evaluate are provided as attached images in this message sequence.
- <diagram_1>: The **first** image provided.
- <diagram_2>: The **second** image provided.
**Evaluation Criteria**
Now, you need to evaluate and compare these two diagrams based on the following **evaluation criteria list**, providing comparative analysis and scoring each on a scale of 0-10. Each criterion includes an explanation and a `human_preference_injection` field. **You must deeply understand the specific human aesthetic or functional preference conveyed in each criterion.**
<criteria_list>
"{criteria_list}"
</criteria_list>
<Instruction>
**Your Task**
Please strictly evaluate and compare `<diagram_1>` (the first image) and `<diagram_2>` (the second image) based on **each criterion** in the `<criteria_list>`. Treat this as a strictly blind evaluation; do not make assumptions about the source of either diagram (e.g., which is the original and which is generated). You need to:
1. **Analyze Each Criterion & Human Preference**: Consider how each diagram fulfills the requirements of each criterion given the <visual_intent> and <paper_content>. Please use the human_preference_injection as a primary reference for high-quality performance across the criteria.
2. **Comparative Evaluation**: Analyze how the two diagrams perform on each criterion, referencing their visual elements, the criterion explanation, and the injected human preference.
3. **Score Separately**: Based on your comparative analysis, score each diagram on each criterion (0-10 points).
**Scoring Rules**
For each criterion, score both diagrams on a scale of 0-10 (continuous values):
* 0-2 points: Very poor performance. Almost completely fails to meet the criterion requirements.
* 2-4 points: Poor performance. Minimally meets the criterion requirements with significant deficiencies.
* 4-6 points: Average performance. Basically meets the criterion requirements, neither good nor bad.
* 6-8 points: Good performance. Largely meets the criterion requirements with notable strengths.
* 8-10 points: Excellent/outstanding performance. Fully meets or exceeds the criterion requirements and perfectly embodies the `human_preference_injection`.
[... Strict JSON parsing, escaping, and non-refusal formatting rules omitted ...]
</Instruction>
<output_format>
```json
{
"structure": {
"Visual Organizational Layout": [
{
"criterion": "[criterion text]",
"analysis": "[comparative analysis]",
"diagram_1_score": [0-10],
"diagram_2_score": [0-10]
},
...
],
"Logical Accuracy": [
{
"criterion": "[criterion text]",
"analysis": "[comparative analysis]",
"diagram_1_score": [0-10],
"diagram_2_score": [0-10]
},
...
]
},
"content": {
"Visual Element Representation": [ ... ],
"Content Accuracy": [ ... ]
},
"aesthetics": [ ... ]
}
```
</output_format>
The visual intent (caption) defining the task and the reference research paper content are provided below:
<visual_intent>
"{visual_intent}"
</visual_intent>
<paper_content>
"{content}"
</paper_content>
Now, evaluate the two diagram images. Output ONLY the json ... code block — no other text whatsoever.
</user_prompt>
Timeline criteria. (Note: To stay within the character limit, we show the complete taxonomy, weights, and three representative full criteria below, with some descriptions omitted via [...])
Timeline
**Structure**
**Visual Organizational Layout**
1. Axis Continuity and Directional Flow
[...]
Weight: 0.15
2. Node-to-Axis Referential Clarity
[...]
Weight: 0.2
3. Spatial Distribution and Density Management
[...]
Weight: 0.15
**Logical Accuracy**
4. Strict Chronological Sequencing
Evaluates the absolute adherence to temporal order. All nodes and chronological markers must be placed in the correct historical or developmental sequence. Any structural hallucination that results in temporal inversions, backward jumps (unless explicitly representing a logical feedback loop), or out-of-order placement fundamentally violates the timeline's logical accuracy.
Human preference: Events and explicit temporal markers must follow the chronological order established by the source. The displayed markers should make that order verifiable at the source-supported temporal resolution. Simultaneous events and uncertain or interval-based dates should be represented consistently without inventing a more precise sequence.
Weight: 0.25
5. Temporal Scale and Structural Integrity
[...]
Weight: 0.15
6. Categorical and Hierarchical Mapping
[...]
Weight: 0.1
**Content**
**Visual Element Representation**
7. Iconographic and Typographic Fidelity
[...]
Weight: 0.15
8. Semantic Clarity and Legend Provision
[...]
Weight: 0.15
9. Temporal Marker Legibility
[...]
Weight: 0.1
**Content Accuracy**
10. Chronological and Factual Precision
The most critical accuracy metric for a timeline. Evaluates whether the specific events, milestones, or developmental stages are factually correct and accurately paired with their corresponding temporal markers. It strictly penalizes chronological hallucinations, where an event is erroneously attributed to the wrong year or date, or where the factual assertions contradict the established parameters of the source paper.
Human preference: Each event must be factually correct and paired with the date, interval, or period supported by the source. Preserve stated uncertainty and temporal precision rather than replacing them with unsupported exact dates. Evaluate correctness at the resolution appropriate to the intended timeline, without automatically favoring finer-grained chronology.
Weight: 0.25
11. Milestone Completeness and Relevance
[...]
Weight: 0.2
12. Terminological Accuracy and Consistency
[...]
Weight: 0.15
**Aesthetics**
13. Typographic Styling and Hierarchy
[...]
Weight: 0.35
14. Color Harmony and Functional Encoding
[...]
Weight: 0.3
15. Stylistic Consistency of Graphical Elements
[...]
Weight: 0.2
16. Visual Polish and Noise Reduction
Evaluates the overall refinement of the diagram's presentation layer. A strong timeline avoids superficial, non-functional decorative elements such as heavy drop shadows, exaggerated gradients, or complex background textures that distract from the chronological narrative. Line weights for the axis and connectors should be crisp and proportionate, resulting in a clean, academically rigorous visual polish.
Human preference: Refinement is demonstrated by clean rendering, proportionate line work, and the absence of visual distractions that interfere with the chronological narrative. Remove unnecessary embellishment while retaining information and visual cues needed to interpret events. Neither sparse content nor a particular rendering style is inherently evidence of greater polish.
Weight: 0.15
W1: Item-level criteria can be fully scaled using LLM generation
While LLMs can automatically generate item-level criteria for individual diagrams, relying on per-instance LLM generation introduces three key limitations:
W2: This paper degrades the granularity from "diagram-specific Items" to "11 broad category templates," which fails to address the inherent uniqueness of individual diagrams or the issue of sub-categories
We respectfully disagree that our approach is intended to "degrade the granularity from 'diagram-specific Items' to '11 broad category templates'," since introducing diagram-specific items into the evaluation process was never our goal. As stated in our Abstract and Introduction, the core motivation of this work is that, compared with general images, scientific diagrams are particularly challenging to evaluate, as they span diverse types designed for different communicative purposes. We therefore evaluate diagrams from the perspective of their practical usage scenarios and communicative functions. Categorizing diagrams by function and usage scenario not only reflects real-world needs but also accounts for the distinct evaluation requirements of different diagram types. Moreover, it makes the evaluation criteria more specific and the evaluation process more stable. As for whether sub-categories exist within each type, we agree this is a promising direction worth exploring in this field. However, we do not consider diagram-specific items or diagram-specific metrics to be a sound approach to evaluation, as explained in our response to W1.
Regarding your concern that “category-level criteria generation may lead to inapplicable criteria for specific diagrams,” the human alignment results for Vanilla reported in Table 4 (85.7% directional agreement, Spearman’s ρ=0.733) and those on the additional PaperBanana human annotations collected during the rebuttal (85.6% directional agreement, ρ=0.752) demonstrate that our evaluation is closely aligned with human preferences and is therefore reliable. Indeed, evaluation criteria are inherently designed as scoring standards for a class of samples rather than for a single instance; grounding these shared criteria in each instance's source content is what ensures both applicability and cross-instance comparability.
W3: The reliance on reference images limits the practical applicability of the evaluator.
The calibrated criteria and relative comparison mechanism can support feedback in agent workflows in two ways: