acceptodds
Under review as a conference paper at ICLR 2027

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Abstract

Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation. Scientific diagrams naturally span diverse types, each characterized by distinct visual structures and content organization. Consequently, during peer review and other scientific evaluation, human experts naturally consider type-specific criteria when assessing their quality. However, existing VLM-based evaluation benchmarks fail to reflect the above human evaluation practices for two main reasons: 1) benchmark datasets typically lack a systematic categorization of different diagram types and thus adopt the same generic criteria for all diagrams, and 2) their evaluation criteria are often designed heuristically, without explicit alignment to human evaluation preferences. Motivated by these limitations, we introduce SciDiagBench, which consists of a benchmark dataset that systematically categorizes scientific diagrams into 11 types and a human preference-guided evaluation method. Specifically, SciDiagBench contains 165 scientific diagrams across 10 broad domains, with types including graphical abstracts, technical roadmaps, and illustrative cases. Based on this dataset, SciDiagBench develops an evaluation method that leverages a preference-guided calibration process to incorporate human preferences into the generation of type-specific criteria. Ultimately, SciDiagBench enables more reliable automated evaluation that better aligns with human evaluation across diverse scientific diagrams.

Primary area:
datasets and benchmarks
Keywords:
Scientific Diagram Evaluation, Scientific Diagram Generation, Human-Aligned Evaluation, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hYvbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SciDiagBench, an evaluation benchmark comprising 165 scientific diagrams across 11 types and 10 domains. The evaluation framework leverages Gemini to generate type-specific criteria and weights, incorporates a HPA mechanism, and utilizes a reference-based relative scoring strategy. The authors conduct multi-dimensional evaluations and stability analyses on 4 generation agents.

Critical strengths
  1. The paper is well-organized and clearly written.
  2. The carefully curated dataset holds significant potential value for the community.
Critical weaknesses
  1. The critique of related work is invalid. The authors argue that Item/Atomic-level methods are difficult to generalize and scale. However, Item-level criteria can be fully scaled using LLM generation, and the scale of 165 diagrams in this work does not demonstrate any scalability advantage. Furthermore, evaluating by category is a well-established practice. This paper degrades the granularity from "diagram-specific Items" to "11 broad category templates," which fails to address the inherent uniqueness of individual diagrams or the issue of sub-categories.
  2. The reliance on reference images limits the practical applicability of the evaluator. The proposed method requires reference images for relative scoring. However, in real-world agent-assisted diagram generation (e.g., text-to-diagram), reference images are often unavailable. Consequently, this method can only be used for static benchmark ranking; it cannot serve as a critic within agent workflows to provide fine-grained feedback, thereby losing its practical value in improving generation capabilities.
  3. Lack of analysis regarding the impact of reference image quality on validity and consistency. The paper assumes reference images are the gold standard and does not analyze scenarios where reference images are of poor quality. If the reference images contain defects, it may invalidate the effectiveness of the proposed evaluation method.
  4. Lack of cross-analysis across scientific domains. Although the dataset covers 10 domains, the experiments only report results broken down by diagram type, without providing domain-level evaluation. Analyzing performance across domains could yield insightful conclusions. Moreover, the paper lacks such insightful takeaways, which could have been beneficial for guiding future agent development.
  5. Category-level criteria generation may lead to inapplicable criteria for specific diagrams. When a VLM generates criteria based on broad categories, the resulting criteria may not hold for specific diagrams within that category, potentially undermining evaluation reliability.
  6. The core prompts are not reported, resulting in poor reproducibility.
Questions for authors

no question

4 comments

W1: Item-level criteria can be fully scaled using LLM generation

While LLMs can automatically generate item-level criteria for individual diagrams, relying on per-instance LLM generation introduces three key limitations:

  1. Reliability and Calibration Cost: Purely LLM-generated, uncalibrated criteria often diverge from human expert judgments (as shown in Table 4, removing Human Preference Alignment drops the Directional Agreement Rate from 85.7% to 76.2%). Ensuring the validity of per-instance criteria would require human verification for every new diagram, which does not scale.
  2. Cross-Instance Score Comparability: Item-level criteria produce non-uniform evaluation scales that hinder standardized comparison across different instances.
  3. Recurring Per-Instance Overhead: Item-level criteria incur repeated generation costs for every newly added diagram. In contrast, our type-specific criteria are calibrated once and generalize directly to unseen instances of the same type with no additional construction cost.

W2: This paper degrades the granularity from "diagram-specific Items" to "11 broad category templates," which fails to address the inherent uniqueness of individual diagrams or the issue of sub-categories

We respectfully disagree that our approach is intended to "degrade the granularity from 'diagram-specific Items' to '11 broad category templates'," since introducing diagram-specific items into the evaluation process was never our goal. As stated in our Abstract and Introduction, the core motivation of this work is that, compared with general images, scientific diagrams are particularly challenging to evaluate, as they span diverse types designed for different communicative purposes. We therefore evaluate diagrams from the perspective of their practical usage scenarios and communicative functions. Categorizing diagrams by function and usage scenario not only reflects real-world needs but also accounts for the distinct evaluation requirements of different diagram types. Moreover, it makes the evaluation criteria more specific and the evaluation process more stable. As for whether sub-categories exist within each type, we agree this is a promising direction worth exploring in this field. However, we do not consider diagram-specific items or diagram-specific metrics to be a sound approach to evaluation, as explained in our response to W1.

Regarding your concern that “category-level criteria generation may lead to inapplicable criteria for specific diagrams,” the human alignment results for Vanilla reported in Table 4 (85.7% directional agreement, Spearman’s ρ=0.733\rho = 0.733) and those on the additional PaperBanana human annotations collected during the rebuttal (85.6% directional agreement, ρ=0.752\rho = 0.752) demonstrate that our evaluation is closely aligned with human preferences and is therefore reliable. Indeed, evaluation criteria are inherently designed as scoring standards for a class of samples rather than for a single instance; grounding these shared criteria in each instance's source content is what ensures both applicability and cross-instance comparability.

W3: The reliance on reference images limits the practical applicability of the evaluator.

The calibrated criteria and relative comparison mechanism can support feedback in agent workflows in two ways:

  1. Criterion-level guidance for critique: Our calibrated criteria describe type-specific quality requirements, with human preferences distilled into explicit guidance (§3.3) that captures shared standards for each diagram type. Even without a reference image, a critic agent can directly apply this guidance to identify flaws—such as dependencies inconsistent with the source text or illegible typography—and request targeted revisions.
  2. Self-referenced relative scoring across iterations: In multi-turn agent workflows, the initial draft (I0I_0) or preceding iteration (It−1I_{t-1}) naturally serves as a comparative reference (anchor) for subsequent revisions (ItI_t). While criteria-based critiques drive the edits, our relative scoring mechanism can directly evaluate ItI_t against It−1I_{t-1} (grounded in the source text and criteria) to verify whether each revision yields positive quality gains without requiring an external ground-truth diagram.

W4: Assume reference images are the gold standard.

We do not treat reference diagrams as an infallible gold standard. Reference-based evaluation is a widely adopted practice in the LLM-as-a-judge paradigm for reducing evaluation error and improving stability [1, 2, 3]. A reference answer, even if it is not the only valid solution, still gives the judge a stable anchor for comparison and thereby reduces judgment variance. We agree that defective references pose an important risk; we have accounted for this risk in our design and mitigate it in two ways:

  1. Data-level: To ensure high reference quality, we used rigorous manual screening and careful annotation to curate the 165-instance dataset. In addition, the curation methodology established during this process is designed to filter out low-quality reference diagrams during automated dataset expansion.
  2. Method-level: Our evaluation framework uses the reference as a comparison baseline rather than as the optimal standard, and does not assume that a reference is flawless or always preferable. In §3.3, the judge receives the source paper and caption and scores both diagrams under the same criteria, with anonymized identities and randomized presentation order. The logical and content accuracy criteria require consistency with the source paper; the reference serves as a comparison anchor rather than a fixed quality ceiling.

[2] Du et al. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. 2026 [3] Zhu et al. AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations. 2026

W5: Lack of cross-analysis across scientific domains.

Our current experiments report results by diagram type because scientific diagrams differ in their visual structures, content organization, and communicative purposes, which directly determine the criteria needed to evaluate their quality.

Different diagram types have distinct disciplinary associations, arising naturally from how various fields use diagrams. As a complementary direction, we plan to construct a dataset with scientific domain as the primary sampling dimension, enabling systematic analysis of domain-specific performance and differences across fields.

W6: The core prompts are not reported.

We will open-source the evaluation framework, including the code, full prompt templates, type-specific criteria, and preference statements in the release documentation. The supporting definitions and evaluation guidance are provided in Appendices B (diagram-type definitions), C (evaluation dimensions and criteria), and D (type-specific evaluation weights) of the paper. The scoring prompt and the type-specific criteria for Timeline is provided below.

Scoring prompt. (Note: Engineering constraints for JSON parsing are omitted via [...] for brevity)

In this template, {criteria_list} contains the criteria for the target diagram type, {visual_intent} is the target caption, and {content} is the source paper text. The two diagrams are supplied as images in an anonymized, randomized order.

<system_role>You are a strict, meticulous, and objective scientific diagram evaluation expert. You excel at using specific assessment criteria to deeply compare two scientific diagrams generated for the same visual intent, providing precise scores and clear justifications based on the provided research paper content.</system_role>

<user_prompt>
**Task Background**
There is a scientific diagram generation task, and you need to evaluate two diagrams created for this task. We will assess the diagrams across three dimensions: Structure, Content, and Aesthetics.

**Diagrams to Evaluate**
The two diagrams you need to evaluate are provided as attached images in this message sequence. 
- <diagram_1>: The **first** image provided.
- <diagram_2>: The **second** image provided.

**Evaluation Criteria**
Now, you need to evaluate and compare these two diagrams based on the following **evaluation criteria list**, providing comparative analysis and scoring each on a scale of 0-10. Each criterion includes an explanation and a `human_preference_injection` field. **You must deeply understand the specific human aesthetic or functional preference conveyed in each criterion.**

<criteria_list>
"{criteria_list}"
</criteria_list>

<Instruction>
**Your Task**
Please strictly evaluate and compare `<diagram_1>` (the first image) and `<diagram_2>` (the second image) based on **each criterion** in the `<criteria_list>`. Treat this as a strictly blind evaluation; do not make assumptions about the source of either diagram (e.g., which is the original and which is generated). You need to:
1.  **Analyze Each Criterion & Human Preference**: Consider how each diagram fulfills the requirements of each criterion given the <visual_intent> and <paper_content>. Please use the human_preference_injection as a primary reference for high-quality performance across the criteria.
2.  **Comparative Evaluation**: Analyze how the two diagrams perform on each criterion, referencing their visual elements, the criterion explanation, and the injected human preference.
3.  **Score Separately**: Based on your comparative analysis, score each diagram on each criterion (0-10 points).

**Scoring Rules**
For each criterion, score both diagrams on a scale of 0-10 (continuous values):
* 0-2 points: Very poor performance. Almost completely fails to meet the criterion requirements.
* 2-4 points: Poor performance. Minimally meets the criterion requirements with significant deficiencies.
* 4-6 points: Average performance. Basically meets the criterion requirements, neither good nor bad.
* 6-8 points: Good performance. Largely meets the criterion requirements with notable strengths.
* 8-10 points: Excellent/outstanding performance. Fully meets or exceeds the criterion requirements and perfectly embodies the `human_preference_injection`.

[... Strict JSON parsing, escaping, and non-refusal formatting rules omitted ...]
</Instruction>

<output_format>
```json
{
    "structure": {
        "Visual Organizational Layout": [
            {
                "criterion": "[criterion text]",
                "analysis": "[comparative analysis]",
                "diagram_1_score": [0-10],
                "diagram_2_score": [0-10]
            },
            ...
        ],
        "Logical Accuracy": [
            {
                "criterion": "[criterion text]",
                "analysis": "[comparative analysis]",
                "diagram_1_score": [0-10],
                "diagram_2_score": [0-10]
            },
            ...
        ]
    },
    "content": {
        "Visual Element Representation": [ ... ],
        "Content Accuracy": [ ... ]
    },  
    "aesthetics": [ ... ]
}
```
</output_format>

The visual intent (caption) defining the task and the reference research paper content are provided below:

<visual_intent>
"{visual_intent}"
</visual_intent>

<paper_content>
"{content}"
</paper_content>

Now, evaluate the two diagram images. Output ONLY the json ...  code block — no other text whatsoever.
</user_prompt>

Timeline criteria. (Note: To stay within the character limit, we show the complete taxonomy, weights, and three representative full criteria below, with some descriptions omitted via [...])

Timeline

**Structure**

**Visual Organizational Layout**

1. Axis Continuity and Directional Flow
[...]
Weight: 0.15

2. Node-to-Axis Referential Clarity
[...]
Weight: 0.2

3. Spatial Distribution and Density Management
[...]
Weight: 0.15

**Logical Accuracy**

4. Strict Chronological Sequencing
Evaluates the absolute adherence to temporal order. All nodes and chronological markers must be placed in the correct historical or developmental sequence. Any structural hallucination that results in temporal inversions, backward jumps (unless explicitly representing a logical feedback loop), or out-of-order placement fundamentally violates the timeline's logical accuracy.
Human preference: Events and explicit temporal markers must follow the chronological order established by the source. The displayed markers should make that order verifiable at the source-supported temporal resolution. Simultaneous events and uncertain or interval-based dates should be represented consistently without inventing a more precise sequence.
Weight: 0.25

5. Temporal Scale and Structural Integrity
[...]
Weight: 0.15

6. Categorical and Hierarchical Mapping
[...]
Weight: 0.1

**Content**

**Visual Element Representation**

7. Iconographic and Typographic Fidelity
[...]
Weight: 0.15

8. Semantic Clarity and Legend Provision
[...]
Weight: 0.15

9. Temporal Marker Legibility
[...]
Weight: 0.1

**Content Accuracy**

10. Chronological and Factual Precision
The most critical accuracy metric for a timeline. Evaluates whether the specific events, milestones, or developmental stages are factually correct and accurately paired with their corresponding temporal markers. It strictly penalizes chronological hallucinations, where an event is erroneously attributed to the wrong year or date, or where the factual assertions contradict the established parameters of the source paper.
Human preference: Each event must be factually correct and paired with the date, interval, or period supported by the source. Preserve stated uncertainty and temporal precision rather than replacing them with unsupported exact dates. Evaluate correctness at the resolution appropriate to the intended timeline, without automatically favoring finer-grained chronology.
Weight: 0.25

11. Milestone Completeness and Relevance
[...]
Weight: 0.2

12. Terminological Accuracy and Consistency
[...]
Weight: 0.15

**Aesthetics**

13. Typographic Styling and Hierarchy
[...]
Weight: 0.35

14. Color Harmony and Functional Encoding
[...]
Weight: 0.3

15. Stylistic Consistency of Graphical Elements
[...]
Weight: 0.2

16. Visual Polish and Noise Reduction
Evaluates the overall refinement of the diagram's presentation layer. A strong timeline avoids superficial, non-functional decorative elements such as heavy drop shadows, exaggerated gradients, or complex background textures that distract from the chronological narrative. Line weights for the axis and connectors should be crisp and proportionate, resulting in a clean, academically rigorous visual polish.
Human preference: Refinement is demonstrated by clean rendering, proportionate line work, and the absence of visual distractions that interfere with the chronological narrative. Remove unnecessary embellishment while retaining information and visual cues needed to interpret events. Neither sparse content nor a particular rendering style is inherently evidence of greater polish.
Weight: 0.15
Reviewer_hY2jOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the fact that scientific diagrams have different structures and communication objectives depending on their type, and proposes SciDiagBench, a dataset consisting of 165 diagrams spanning 11 diagram types and 10 scientific fields. The authors construct a type-specific evaluation framework that generates different evaluation criteria and weights for each diagram type, based on Structure, Content, and Aesthetics. In this evaluation method, scoring instructions are added to the evaluation criteria based on human preferences observed in the development set, and generated diagrams are scored through relative comparison with human-created reference diagrams. In the experiments, the authors compare multiple scientific diagram generation agents and multiple VLM judges to analyze performance differences across diagram types and evaluation dimensions, differences in evaluation tendencies among judge models, and the effectiveness of the proposed evaluation elements.

Critical strengths
  1. SciDiagBench classifies scientific diagrams into 11 categories and generates different evaluation criteria based on the structure and communication objectives of each type. This enables a more nuanced evaluation of the diversity of scientific diagrams compared to traditional frameworks that evaluate all scientific diagrams using a single, generic standard.

  2. SciDiagBench incorporates human preferences into its evaluation criteria. Based on the results of human comparisons between reference diagrams and generated diagrams on the development set, preference statements are created and provided to the VLM judge as scoring instructions. This approach enables evaluation criteria that more closely align with human perception.

Critical weaknesses
  1. The dataset is relatively small. Although SciDiagBench covers 11 figure types and 10 scientific fields, it consists of only 165 samples in total, which is insufficient to fully evaluate the diversity of each type or its cross-domain generalization performance.

  2. There are still insufficient experiments to demonstrate that the proposed evaluation method is sufficiently human-aligned. Only the output generated by the Vanilla agent has been used to verify alignment with human evaluations; it has not been confirmed whether the proposed method maintains the same level of alignment with humans when applied to other generative methods, such as PaperBanana, AutoFigure, and Paper2Figure, that exhibit different failure patterns. Furthermore, since the proposed method has not been directly compared with existing evaluation methods, such as PaperBananaBench, FigureBench, and SciFig, using the same human preference data, it has not been demonstrated that the proposed method is more human-aligned than existing methods.

Questions for authors

Please see the weaknesses.

1 comment

W1: Insufficient Human Alignment Experiments across Generation Agents.

To address your concern regarding human alignment across generation agents, we added human preference annotations for PaperBanana on all 110 test instances during the rebuttal period. Three annotators blindly and independently compared each generated diagram with its human-created reference. After applying the same aggregation rule as in §4.1, 104 PaperBanana pairs were retained. Their agreement with our automatic evaluation is reported below:

AgentRetained pairsDirectional agreementKendall’s τ_bSpearman’s ρ
Vanilla10585.71%0.59430.7329
PaperBanana10485.58%0.63820.7521

These results confirm that our evaluation framework maintains a comparable level of human alignment on PaperBanana.

W2: No direct comparison with existing evaluation methods.

We compare our method (TSC+HPA) with the reference-based evaluation protocol of PaperBananaBench [1] on our test set. For a fair comparison, both protocols use Gemini 3.1 Pro as the judge VLM to assess the relative quality of generated diagrams against reference diagrams, and their results are evaluated against our human preference annotations.

GeneratorEvaluation ProtocolDARKendall’s τb\tau_bSpearman’s ρ\rho
VanillaPaperBananaBench [1]0.54290.35640.3954
VanillaOurs (TSC+HPA)0.85710.59430.7329
PaperBananaPaperBananaBench [1]0.57690.37370.4192
PaperBananaOurs (TSC+HPA)0.85580.63820.7521

As shown in the table, our evaluation framework consistently outperforms the PaperBananaBench protocol across both generators, improving DAR by 31.42 and 27.89 percentage points on Vanilla and PaperBanana, respectively, while achieving substantially higher rank correlations (τb\tau_b and ρ\rho) with human judgments.

[1] Zhu et al. PaperBanana: Automating Academic Illustration for AI Scientists. 2026

Reviewer_oTj8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses how to automatically evaluate AI-generated scientific diagrams, such as figures produced from a paper's text. It makes two contributions.

  1. A dataset. It holds 165 human-made diagrams from published papers, spread over 11 diagram types and 10 scientific domains. Each instance is a (paper, caption, aspect ratio, reference diagram) quadruple.
  2. An evaluation method.
Critical strengths
  1. Timely problem and clear framing. Evaluating diagram-generation agents is a real gap. Making the judge criteria depend on diagram type is intuitive and fits how reviewers judge diagrams.

  2. Careful splitting of visual quality from accuracy. Separating Visual Organizational Layout from Logical Accuracy, and Visual Element Representation from Content Accuracy, addresses a known judge failure: polished-looking diagrams that are wrong.

  3. Sound human study design

Critical weaknesses

The benchmark size is too small, authors acknowledge this. Filtering for contamination is manual. Domain coverage is skewed. AI/ML is 35.1%, and Agricultural & Food Sciences is 1.8% (about 3 diagrams), so the "10 broad domains" claim is not justified. The type taxonomy is built by the authors from the 291 diagrams. There is no reported agreement on assigning a diagram to a type, and some types overlap in the definitions (for example Conceptual Framework vs Algorithmic Framework vs Graphical Abstract; Mechanism Diagram vs Phased Evolution).

Questions for authors
  1. Will the dataset, the type-specific criteria, the preference statements and the code be released, and under what license, given the source figures?
  2. Can you collect human preference labels on PaperBanana, AutoFigure and Paper2Figure outputs and show that agent rankings agree with the automatic ranking?
  3. How were diagram types assigned, and what is the inter-annotator agreement on type labels?
2 comments

W1: Filtering for contamination is manual.

Yes. Since the dataset is relatively small, we manually screened the initial dataset during its construction for contamination to ensure data quality. During the rebuttal period, we designed an automated filtering pipeline based on the criteria described in Appendices A.1 (category exclusion) and A.2 (potential contamination) of the paper to support dataset expansion.

W2: The "10 broad domains" claim is not justified.

We acknowledge that domain coverage is imbalanced, with some domains represented by relatively few samples. Notably, SciDiagBench is designed around diagram types, with equal numbers of diagrams per type. Under this type-balanced design, the imbalanced domain distribution reflects real-world diagram usage, as certain diagram types appear far less frequently in some domains than in others.

Specifically, we used unbiased random sampling to collect papers and their figures from journals and conferences spanning multiple disciplines, such as The Innovation, Reviews of Modern Physics, and the International Conference on Information Systems (ICIS), without imposing a predefined diagram taxonomy. We collected 598 figures from 364 papers and retained 291 candidate diagrams after filtering. Then we developed the 11-type taxonomy inductively by identifying recurring visual structures and communicative functions, and subsequently selected equal numbers of diagrams for each type. This process grounds the taxonomy in actual scientific practice and captures most scenarios in which researchers use diagrams to communicate their work.

W3: No reported agreement on assigning a diagram to a type and some types overlap.

Agreement on assignments. To quantify how reliably diagrams are assigned to types, we asked Gemini 3.1 Pro to classify all 165 benchmark diagrams using only the type definitions given in appendix B. Its labels agree with the author labels on 151/165 diagrams (91.5%; Cohen’s κ = 0.907). This shows the definitions separate the types well in practice. For context, the 11 types were derived inductively from the 291 candidate diagrams, with no predefined categories. 7 authors iteratively grouped these diagrams and refined the definitions through discussion until reaching consensus. The 165 diagrams retained in the final benchmark carry these consensus labels.

Type overlap. To clarify the specific overlaps you identified, we distinguish the mentioned types as follows:

Conceptual Framework vs. Algorithmic Framework vs. Graphical Abstract: These three types differ in granularity and communicative scope. A Conceptual Framework maps a domain-level theoretical model, classification system, or research landscape using hierarchical, parallel, or progressive blocks. An Algorithmic Framework details a method-level modular blueprint of a specific algorithm, specifying its core components, data flow, inter-module interactions, and multi-stage execution logic. In contrast, a Graphical Abstract summarizes an entire paper’s narrative—spanning background, mechanisms, principal findings, and conclusions.

Mechanism Diagram vs. Phased Evolution: Mechanism Diagram explains how a single event occurs via microscopic entity interactions, whereas Phased Evolution illustrates how a system transforms across discrete stages using partitioned spatial blocks (where specific mechanisms only appear locally within individual stage blocks).

Q1: Will SciDiagBench be released, and under what license?

Yes, we will open-source the evaluation framework, including the code, type-specific criteria, and preference statements, with their specific licenses stated in the release documentation. For SciDiagBench, we will directly release papers and figures whose licenses permit redistribution, preserving their original licenses and required attribution. For the remaining materials, we will seek authorization from the copyright holders. If redistribution permission cannot be obtained, we plan to provide a web-based evaluation service covering the complete test set without redistributing the restricted source materials.

Q2: Do automatic agent rankings agree with human preference?

We added human preference annotations for PaperBanana on all 110 test instances, covering all 11 diagram types. Three annotators blindly and independently compared each generated diagram with its human-created reference. Applying the same aggregation rule retains 104 PaperBanana pairs for comparison with the main experimental results:

AgentRetained pairsDirectional agreementKendall’s τ_bSpearman’s ρ
Vanilla10585.71%0.59430.7329
PaperBanana10485.58%0.63820.7521

To examine agent ranking consistency, we conducted an exploratory paired analysis on the 99 instances retained for both agents. Humans preferred the generated diagram over its reference in 46/99 cases (46.5%) for PaperBanana, compared with 34/99 (34.3%) for Vanilla. This is consistent with the main results.

Q3: How were diagram types assigned? The 11 types were developed inductively from the 291 retained candidate diagrams, without imposing predefined types during initial collection. The authors iteratively grouped the diagrams and refined the definitions through repeated discussions until a consensus taxonomy was established.

As a supplementary check of model–author label agreement, we prompted Gemini 3.1 Pro to classify all 165 benchmark diagrams using the type definitions. Its assignments agree with the original author labels on 151/165 diagrams (91.52%; Cohen’s κ = 0.907).

Comments on the paper

1 comment

We thank all reviewers for their insightful comments, and we apologize for our delayed response, as expanding the dataset took additional time. Below, we first address the common concern about dataset size and then respond to each reviewer individually.

General Response to the Concern about Limited Benchmark Size

Although the current dataset is modest in size, it is fully human-annotated to ensure data quality. Such human annotation is essential for validating the reliability of our evaluation framework, i.e., measuring how well the LLM judge's assessments agree with human judgments (85.7% directional agreement for our framework). Once the evaluation model and metric design have been validated on this high-quality, human-annotated seed set, expanding this benchmark becomes an automated and scalable process.

Given the limited time of the rebuttal period, we added 330 instances through automated collection as a demonstration, expanding the test set fourfold from 110 to 440 instances (40 per type), and evaluated Vanilla (baseline) and PaperBanana (SOTA) on the added instances. The original and expanded results are compared below. These results support the framework's reliability and applicability at a larger scale. We promise to further expand this automatically constructed dataset to more than 1,000 instances and evaluate all models on it.

Diagram typeVanilla (110 / 440)PaperBanana (110 / 440)
Timeline0.4406 / 0.44290.5272 / 0.5272
Lineage Dendrogram0.4168 / 0.41620.4917 / 0.4659
Phased Evolution0.5626 / 0.56540.5975 / 0.5995
Algorithmic Framework0.4670 / 0.45940.5665 / 0.5786
Computer System Architecture0.4947 / 0.50140.4926 / 0.4997
Conceptual Framework0.5820 / 0.58310.5703 / 0.5699
Technical Roadmap0.5094 / 0.51240.5361 / 0.5535
Graphical Abstract0.6228 / 0.63560.6195 / 0.6147
Experimental Setup Diagram0.5258 / 0.53100.4610 / 0.4867
Illustrative Case0.3740 / 0.35530.4824 / 0.5014
Mechanism Diagram0.4566 / 0.46220.4985 / 0.5098
Overall (macro-average)0.4957 / 0.49680.5312 / 0.5370

The results remain broadly consistent before and after dataset expansion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.