- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces Crafter, an agentic system that generates scientific figures with plan, critique and correction. CraftEditor converts raster figures into editable SVGs using asset extraction and composition. CraftBench adds 279 examples covering three figure styles (academic from arXiv, poster from award-tier conferences, infographics from research blogs) and four input conditions (text-image, mask completion, sketch-image, key-element composition). Experiments show strong improvements over direct image generators (e.g., GPT-Image, Nano Banana) and agentic system baselines such as AutoFigure.
Critical strengths
- Combines scientific figure generation (Crafter) with conversion into editable SVGs (CraftEditor). The ablations show clear performance drops when removing each Crafter component.
- Benchmark covers four relevant input conditions (text, mask, element, sketch) across academic figures, posters, and infographics.
Critical weaknesses
- Evaluating scientific figure generation and editing is difficult, especially for complex methodological figures from long paper sections. Agreement between automatic and human judgments is moderate (κ = 0.58), and human–human agreement is not reported.
Questions for authors
- What is the human–human agreement?
- Which aspects or figure types produce most disagreement?
- Can you report raw scores alongside the Model/Tie/Human verdict results?
4 comments
Response to Reviewer Xuxw(2/2)
Q3: Raw scores alongside the Model/Tie/Human verdicts.
Can you report raw scores alongside the Model/Tie/Human verdict results?
R3:
Yes. We further report the raw 0–10 judge scores (Table A11, A12), and they tell the same story as the verdicts. Each image (candidate and human-drawn target) is scored independently on the weighted aspects of Appendix F; the table reports the mean total of the candidate on CRAFTBENCH (n=279) and its mean margin over the target of the same run:
| Method | T2I (n=179) | Mask (n=30) | Sketch (n=40) | KeyEl. (n=30) | Overall | Margin vs. target |
|---|---|---|---|---|---|---|
| Nano Banana Pro | 7.80 | 6.93 | 7.44 | 7.32 | 7.61 | −1.12 |
| PaperBanana (w/ Nano Banana Pro) | 7.77 | 7.12 | 7.75 | 7.93 | 7.72 | −1.05 |
| CRAFTER (w/ Nano Banana Pro) | 8.90 | 7.92 | 8.86 | 8.21 | 8.71 | −0.05 |
| Nano Banana 2 | 7.02 | 8.06 | 8.35 | 8.62 | 7.49 | −1.29 |
| CRAFTER (w/ Nano Banana 2) | 8.78 | 8.30 | 8.84 | 8.94 | 8.75 | +0.05 |
| Human-drawn target (CRAFTER w/ Nano Banana 2 run) | 8.78 | 9.01 | 7.79 | 9.21 | 8.71 | – |
As shown in this table, CRAFTER raises the mean raw score by about 1.0 point over PaperBanana under the identical backbone (8.71 vs. 7.72) and reaches parity with the human-drawn targets, while PaperBanana and the standalone generators stay more than 1 point below them.
The gain holds on every aspect. Candidate / target means with the Nano Banana Pro backbone:
| Method | Content faithfulness | Readability | Format (T2I, n=179) | Input fidelity (ref.-cond., n=100) |
|---|---|---|---|---|
| Nano Banana Pro | 7.49 / 9.01 | 8.02 / 8.57 | 8.79 / 8.45 | 6.63 / 8.34 |
| PaperBanana | 7.78 / 9.08 | 8.31 / 8.59 | 7.79 / 8.45 | 7.01 / 8.46 |
| CRAFTER | 8.70 / 9.08 | 8.86 / 8.58 | 9.22 / 8.38 | 8.12 / 8.55 |
The gain over PaperBanana is larger on content faithfulness (+0.92) and input fidelity (+1.11) than on readability (+0.55).
Meanwhile, we further computed confidence intervals on the verdicts. We report percentile-bootstrap 95% confidence intervals over samples (10,000 resamples) and exact sign tests:
| Benchmark (backbone) | CRAFTER | PaperBanana | Gap [95% CI] | Sign test |
|---|---|---|---|---|
| PaperBanana-Bench (NB2), n=292 | 50.34 | 33.73 | +16.61 [+10.3, +22.8] | p < 10⁻⁶ |
| PaperBanana-Bench (NB-Pro), n=292 | 50.00 | 35.96 | +14.04 [+7.7, +20.2] | p < 10⁻⁴ |
| CRAFTBENCH (NB-Pro), n=279 | 52.33 | 29.03 | +23.30 [+18.8, +27.6] | p < 10⁻¹⁸ |
The conclusions are also insensitive to the tie band: sweeping it from 0 to 0.5 keeps the gap on CRAFTBENCH (Nano Banana Pro) between 22.2 and 25.8 points, with the ranking of the systems unchanged.
We will include all of the above results in the revised pdf and mark them in red.
We hope these clarifications address your concerns. If our responses resolve your concerns, we would be grateful if you could consider raising your score. Please feel free to follow up with any further questions—we are happy to respond right away.
Best regards, Authors
Yes now its good. Increased my score
Dear Reviewer Xuxw,
Thank you very much for taking the time to read our response and for raising your score. We are glad that our clarifications and the additional results resolved your concerns.
We will include all of the above results in the final version. Please feel free to let us know if you have any further questions—we are happy to respond right away.
Best regards, Authors
Response to Reviewer Xuxw(1/2)
Response to Reviewer Xuxw
We thank Reviewer
Xuxwfor the constructive review and for recognizing the combination of figure generation with editable SVG conversion, the clear ablation trends, and the coverage of CRAFTBENCH across four input conditions and three figure types. We address the concerns below.Q1: Human–human agreement is not reported.
R1:
The annotators agree pairwise on 78.9% of cases (Fleiss' κ = 0.65), and the judge picks the same winner as their majority in 94.9% of the decided cases. Our paper already reports the agreement between the judge and the annotators (Table A8). To fully address your question, we further computed the human–human agreement among the three annotators and added it to Table A8:
As shown in this table, two annotators agree on 76.7–81.7% of cases, and the judge agrees with their majority on 71.7%. The moderate κ = 0.58 comes from the Tie option: the verdict is a three-way one (Model / Tie / Human), a near-tie is hard to separate from a narrow win, and 15 of the judge's 17 disagreements are between Tie and a preference. Restricted to the choice between the model-generated and the human-drawn figure, the agreement is high: when both sides name a winner, the judge picks the same figure as the majority in 37 of 39 cases (94.9%, κ = 0.89), vs. 85.4–87.8% between two annotators.
Meanwhile, as already reported in our paper, the lead of CRAFTER over PaperBanana is preserved under judges from three model families (19.9–22.2 points, Table A7), and it is far outside sampling uncertainty (see Q3). Regarding the evaluation of editing, the editable outputs of CRAFTEDITOR are scored by an ensemble of judges from three model families (Appendix E) and validated by a human editing study (Appendix I, Table A9).
Q2: Which aspects or figure types produce most disagreement?
R2:
Disagreement is highest on key-element composition and mask completion, the two tasks that build on given partial content; sketch-conditioned and text-to-image cases are the most consistent. To answer this question, we further conducted a breakdown analysis of the human study by task and figure type (Table A10):
Across tasks, the judge is least consistent exactly where the annotators are, which follows from the tasks themselves: in mask completion the verdict rests on one region, and in key-element composition the same elements can be arranged in several acceptable ways. Across figure types the differences are smaller (on infographics, n=9, the gap between the two columns is three verdicts), and on academic method figures, the type you single out as hardest to evaluate, the judge agrees with the majority most often (74.3%). The annotators give one overall verdict per case, so the breakdown is by task and figure type. Since some subgroups are small (6 to 13 cases), the differences between them should be read as indicative, and we will enlarge the study.