acceptodds
Under review as a conference paper at ICLR 2027

Crafter: A Multi-Agent Harness for Editable Scientific Figures Generation from Diverse Inputs

12 views

Abstract

Scientific figures are vital for communicating complex research, yet producing publication-quality illustrations remains labor-intensive. Existing automated systems each target a single figure type from text-only input, missing the diversity of types and conditions researchers actually need, and their raster outputs resist local revision. Since scientific figures are structured compositions of discrete semantic components, the localized error generators produce on such layouts demand not a stronger backbone but a harness. We instantiate this harness in two complementary systems:Crafter, a multi-agent harness for figure generation that generalizes across figure types and conditions without architectural changes, and CraftEditor, which converts raster outputs into editable SVGs. We introduce CraftBench, a benchmark spanning three figure types and four input conditions with human quality annotation. Experiments show that Crafter substantially outperforms standalone generators and agentic baselines on PaperBananaBench and CrafterBench, ablations confirm each component's contribution, and CraftEditor yields editable SVGs surpassing all baselines.

Primary area:
generative models
Keywords:
Agent, Harness, Scientific Figures, Multimodal Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XuxwOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces Crafter, an agentic system that generates scientific figures with plan, critique and correction. CraftEditor converts raster figures into editable SVGs using asset extraction and composition. CraftBench adds 279 examples covering three figure styles (academic from arXiv, poster from award-tier conferences, infographics from research blogs) and four input conditions (text-image, mask completion, sketch-image, key-element composition). Experiments show strong improvements over direct image generators (e.g., GPT-Image, Nano Banana) and agentic system baselines such as AutoFigure.

Critical strengths
  • Combines scientific figure generation (Crafter) with conversion into editable SVGs (CraftEditor). The ablations show clear performance drops when removing each Crafter component.
  • Benchmark covers four relevant input conditions (text, mask, element, sketch) across academic figures, posters, and infographics.
Critical weaknesses
  • Evaluating scientific figure generation and editing is difficult, especially for complex methodological figures from long paper sections. Agreement between automatic and human judgments is moderate (κ = 0.58), and human–human agreement is not reported.
Questions for authors
  • What is the human–human agreement?
  • Which aspects or figure types produce most disagreement?
  • Can you report raw scores alongside the Model/Tie/Human verdict results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Xuxw(1/2)

Response to Reviewer Xuxw

We thank Reviewer Xuxw for the constructive review and for recognizing the combination of figure generation with editable SVG conversion, the clear ablation trends, and the coverage of CRAFTBENCH across four input conditions and three figure types. We address the concerns below.

Q1: Human–human agreement is not reported.

Evaluating scientific figure generation and editing is difficult ... Agreement between automatic and human judgments is moderate (κ = 0.58), and human–human agreement is not reported.

What is the human–human agreement?

R1:

The annotators agree pairwise on 78.9% of cases (Fleiss' κ = 0.65), and the judge picks the same winner as their majority in 94.9% of the decided cases. Our paper already reports the agreement between the judge and the annotators (Table A8). To fully address your question, we further computed the human–human agreement among the three annotators and added it to Table A8:

PairAgreement (%)Cohen's κ
Annotator 1 vs. Annotator 281.70.70
Annotator 1 vs. Annotator 376.70.61
Annotator 2 vs. Annotator 378.30.64
Mean pairwise / Fleiss' κ (three annotators)78.90.65
Judge vs. majority verdict71.70.58
Judge vs. majority verdict, unanimous cases (n=41)90.20.83
Winner only: two annotators (cases where both name a winner)85.4–87.80.70–0.75
Winner only: judge vs. majority verdict (n=39)94.90.89

As shown in this table, two annotators agree on 76.7–81.7% of cases, and the judge agrees with their majority on 71.7%. The moderate κ = 0.58 comes from the Tie option: the verdict is a three-way one (Model / Tie / Human), a near-tie is hard to separate from a narrow win, and 15 of the judge's 17 disagreements are between Tie and a preference. Restricted to the choice between the model-generated and the human-drawn figure, the agreement is high: when both sides name a winner, the judge picks the same figure as the majority in 37 of 39 cases (94.9%, κ = 0.89), vs. 85.4–87.8% between two annotators.

Meanwhile, as already reported in our paper, the lead of CRAFTER over PaperBanana is preserved under judges from three model families (19.9–22.2 points, Table A7), and it is far outside sampling uncertainty (see Q3). Regarding the evaluation of editing, the editable outputs of CRAFTEDITOR are scored by an ensemble of judges from three model families (Appendix E) and validated by a human editing study (Appendix I, Table A9).

Q2: Which aspects or figure types produce most disagreement?

Which aspects or figure types produce most disagreement?

R2:

Disagreement is highest on key-element composition and mask completion, the two tasks that build on given partial content; sketch-conditioned and text-to-image cases are the most consistent. To answer this question, we further conducted a breakdown analysis of the human study by task and figure type (Table A10):

BreakdownnHuman–human agreement (%)Judge vs. majority (%)
Sketch-conditioned1288.983.3
Text-to-image2979.372.4
Mask completion1374.469.2
Key-element666.750.0
Academic3577.174.3
Poster1675.068.8
Infographic992.666.7

Across tasks, the judge is least consistent exactly where the annotators are, which follows from the tasks themselves: in mask completion the verdict rests on one region, and in key-element composition the same elements can be arranged in several acceptable ways. Across figure types the differences are smaller (on infographics, n=9, the gap between the two columns is three verdicts), and on academic method figures, the type you single out as hardest to evaluate, the judge agrees with the majority most often (74.3%). The annotators give one overall verdict per case, so the breakdown is by task and figure type. Since some subgroups are small (6 to 13 cases), the differences between them should be read as indicative, and we will enlarge the study.

AuthorsOpenReview ↗

Response to Reviewer Xuxw(2/2)

Q3: Raw scores alongside the Model/Tie/Human verdicts.

Can you report raw scores alongside the Model/Tie/Human verdict results?

R3:

Yes. We further report the raw 0–10 judge scores (Table A11, A12), and they tell the same story as the verdicts. Each image (candidate and human-drawn target) is scored independently on the weighted aspects of Appendix F; the table reports the mean total of the candidate on CRAFTBENCH (n=279) and its mean margin over the target of the same run:

MethodT2I (n=179)Mask (n=30)Sketch (n=40)KeyEl. (n=30)OverallMargin vs. target
Nano Banana Pro7.806.937.447.327.61−1.12
PaperBanana (w/ Nano Banana Pro)7.777.127.757.937.72−1.05
CRAFTER (w/ Nano Banana Pro)8.907.928.868.218.71−0.05
Nano Banana 27.028.068.358.627.49−1.29
CRAFTER (w/ Nano Banana 2)8.788.308.848.948.75+0.05
Human-drawn target (CRAFTER w/ Nano Banana 2 run)8.789.017.799.218.71–

As shown in this table, CRAFTER raises the mean raw score by about 1.0 point over PaperBanana under the identical backbone (8.71 vs. 7.72) and reaches parity with the human-drawn targets, while PaperBanana and the standalone generators stay more than 1 point below them.

The gain holds on every aspect. Candidate / target means with the Nano Banana Pro backbone:

MethodContent faithfulnessReadabilityFormat (T2I, n=179)Input fidelity (ref.-cond., n=100)
Nano Banana Pro7.49 / 9.018.02 / 8.578.79 / 8.456.63 / 8.34
PaperBanana7.78 / 9.088.31 / 8.597.79 / 8.457.01 / 8.46
CRAFTER8.70 / 9.088.86 / 8.589.22 / 8.388.12 / 8.55

The gain over PaperBanana is larger on content faithfulness (+0.92) and input fidelity (+1.11) than on readability (+0.55).

Meanwhile, we further computed confidence intervals on the verdicts. We report percentile-bootstrap 95% confidence intervals over samples (10,000 resamples) and exact sign tests:

Benchmark (backbone)CRAFTERPaperBananaGap [95% CI]Sign test
PaperBanana-Bench (NB2), n=29250.3433.73+16.61 [+10.3, +22.8]p < 10⁻⁶
PaperBanana-Bench (NB-Pro), n=29250.0035.96+14.04 [+7.7, +20.2]p < 10⁻⁴
CRAFTBENCH (NB-Pro), n=27952.3329.03+23.30 [+18.8, +27.6]p < 10⁻¹⁸

The conclusions are also insensitive to the tie band: sweeping it from 0 to 0.5 keeps the gap on CRAFTBENCH (Nano Banana Pro) between 22.2 and 25.8 points, with the ranking of the systems unchanged.

We will include all of the above results in the revised pdf and mark them in red.

We hope these clarifications address your concerns. If our responses resolve your concerns, we would be grateful if you could consider raising your score. Please feel free to follow up with any further questions—we are happy to respond right away.

Best regards, Authors

Yes now its good. Increased my score

Dear Reviewer Xuxw,

Thank you very much for taking the time to read our response and for raising your score. We are glad that our clarifications and the additional results resolved your concerns.

We will include all of the above results in the final version. Please feel free to let us know if you have any further questions—we are happy to respond right away.

Best regards, Authors

Reviewer_3FmYOpenReview ↗
Rating
1
Confidence
4(+2)
Informativeness
1
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper introduces Crafter, a multi-agent framework for scientific figure generation, and CraftEditor, a complementary system for converting raster figures into editable SVGs. Crafter coordinates planning, generation, verification, and refinement through an evolving structured specification. The authors also introduce CraftBench, a benchmark covering three figure types and four input conditions. Experiments on PaperBanana-Bench and CraftBench demonstrate substantial improvements over the evaluated image generators and agentic frameworks. CraftEditor also achieves promising results in raster-to-SVG conversion.

Critical strengths
  • The paper addresses important limitations of automated scientific figure generation, particularly support for diverse input conditions and the need for editable outputs.
  • The integration of figure generation and SVG reconstruction provides a useful end-to-end workflow. CraftBench extends existing evaluations beyond text-to-image generation and includes human-curated reference-conditioned tasks.
  • Crafter achieves substantial improvements over the evaluated baselines. The ablation studies, cross-family judge comparisons, and human evaluations provide useful supporting evidence.
Critical weaknesses
  • The proposed harness primarily combines existing techniques, including multi-agent orchestration, best-of-N candidate selection, iterative self-refinement, and structured specifications. While the typed corrective layer is a reasonable engineering design, its distinction from existing structured prompting and programmatic editing approaches is insufficiently established. Stronger comparisons with simpler structured-refinement baselines would help clarify the contribution.
  • Crafter employs multiple candidate generations and refinement rounds, introducing substantially greater inference cost than competing methods. Although the image-generation backbone and vision-dependent models are controlled, Crafter additionally uses Claude Opus 4.6 for its language-only agents. The performance also decreases considerably when the planner is replaced with Gemini 3.1 Pro. Comparisons under matched generation budgets, refinement rounds, and planner capabilities are needed to isolate the contribution of the harness architecture.
  • The main results are primarily based on VLM-as-a-Judge evaluations. Although the authors provide cross-family judge comparisons and a human study, the automatic judge agrees with majority human preferences on only 71.7% of the sampled cases. Additional human evaluation, particularly of scientific content correctness and structural relationships, would strengthen the reliability of the reported improvements.
Questions for authors
  1. How does Crafter compare with a simpler best-of-N generation and iterative refinement baseline under the same total inference budget?

  2. Can the authors isolate the contribution of the structured specification from the effects of stronger planning models and additional generation attempts?

  3. How accurately do the evaluation models detect scientific content errors, particularly incorrect labels, missing components, and invalid relationships?

  4. Does CraftEditor preserve independent editability of semantic objects, rather than merely producing visually faithful SVG renderings?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3FmY(1/3)

Response to Reviewer 3FmY

We thank Reviewer 3FmY for the careful review and for recognizing that the paper addresses important limitations of automated scientific figure generation, that the integration of generation and SVG reconstruction provides a useful end-to-end workflow, and that the ablation studies, cross-family judge comparisons and human evaluations provide useful supporting evidence. We address the concerns below.

Q1: The harness combines existing techniques; distinction of the typed corrective layer; simpler structured-refinement baselines.

The proposed harness primarily combines existing techniques ... While the typed corrective layer is a reasonable engineering design, its distinction from existing structured prompting and programmatic editing approaches is insufficiently established. Stronger comparisons with simpler structured-refinement baselines would help clarify the contribution.

R1:

We agree that the four-role loop builds on established orchestration and self-correction principles, which we cite explicitly (Madaan et al., 2023; Young, 2025; Pan et al., 2026). The agentic baselines we compare against are themselves planning-and-refinement pipelines, so the pattern alone cannot be what separates the systems. Our claimed contribution is what the harness is designed to achieve:

A single harness that covers three figure types and four input conditions. All task-specific behavior resides in agent prompts and the executor is pluggable (Section 3.1): the same architecture, with no structural change, handles text-to-image, mask completion, key-element composition and sketch-conditioned generation, and the same pattern converts raster outputs into editable SVGs (CRAFTEDITOR).

This generality does not come for free with the pattern. As already shown in our paper, under the same backbone and VLM, PaperBanana's gain over its backbone shrinks from +22.60 on PaperBanana-Bench to +8.06 on CRAFTBENCH once the inputs go beyond text-to-image academic diagrams, while CRAFTER's gain stays at +39.21 and +30.29 (Table 2, Section 5.1).

The simpler structured-refinement baseline is already in the paper: Table 3, "w/o corrective layer". This row keeps the planner, the structured specification, the critic and the budget, and only replaces typed edits with free-text revision instructions, which is how existing self-refinement pipelines revise. It costs 8.90 points (41.44 vs. 50.34), the largest drop among the four mechanisms. The distinction from structured prompting and programmatic editing is where the correction is applied. Existing approaches rewrite or append to the prompt in every round, so corrections from different rounds can contradict each other. In CRAFTER, the critic's per-dimension diagnostics are compiled into typed edits (set, resize, demote, add-note, add-avoid, add-region-fix), each of which modifies one field of a persistent structured specification. The next render is generated from the updated specification, so an earlier fix stays in place and a later round does not silently undo it. As Section 5.2.1 notes, the 8.90-point drop "supports the concern raised in Section 1 that unstructured corrections accumulate contradictions across rounds."

AuthorsOpenReview ↗

Response to Reviewer 3FmY(2/3)

Q2: Inference cost, planner model, matched budgets and rounds.

Crafter employs multiple candidate generations and refinement rounds, introducing substantially greater inference cost ... Crafter additionally uses Claude Opus 4.6 for its language-only agents ... Comparisons under matched generation budgets, refinement rounds, and planner capabilities are needed ...

  1. How does Crafter compare with a simpler best-of-N generation and iterative refinement baseline under the same total inference budget?
  2. Can the authors isolate the contribution of the structured specification from the effects of stronger planning models and additional generation attempts?

R2:

Each of the three controls is already in the paper, and CRAFTER leads under every one. We further add a matched-planner experiment.

As already reported in our paper, CRAFTER leads under a matched budget and matched rounds (Table A4). PaperBanana is itself an iterative-refinement baseline: it renders one plan and revises it for up to 3 critic rounds. In CRAFTER, K is the number of candidate plans rendered in parallel and T the number of refinement rounds. Restricted to PaperBanana's budget, i.e. a single plan with no best-of-K selection and the same 3 rounds (K=1), CRAFTER scores 41.78 vs. 33.73; with a single round instead (T=1), it scores 44.86. The advantage is therefore not purchased by extra generations. For reference, the full system costs 0.25 USD per figure vs. 0.11 USD for PaperBanana (Table A5), i.e. under 90 USD for the whole of CRAFTBENCH.

As already reported in our paper, CRAFTER also leads under a matched planner (Table A6). With every agent on Gemini 3.1 Pro, the single model PaperBanana uses for all of its agents, CRAFTER scores 38.4 on CRAFTBENCH vs. 27.96 for PaperBanana under the same backbone. Regarding the decrease you mention (50.2 to 38.4): the planner is a component of the harness and a stronger one helps it, which Table A6 reports; CRAFTER leads without it as well.

To fully address your concern, we further ran a new experiment that gives PaperBanana the same language model as CRAFTER (claude-opus-4-6). Moving all of PaperBanana's agents to Claude Opus 4.6 raises it only from 33.73 to 36.64 on PaperBanana-Bench, vs. 50.34 for CRAFTER with the same language model and backbone. A stronger language model does not close the gap: CRAFTER stays 13.70 points ahead (95% CI +7.2 to +20.2), so the lead comes from the harness and not from the planner model.

As already reported in our paper, Table 3 isolates the contribution of typed edits. Every row of Table 3 uses the same planner and backbone, and the "w/o corrective layer" row also keeps K and T unchanged. Its 8.90-point drop is therefore attributable to typed edits on the specification, and to neither a stronger planner nor additional attempts.

AuthorsOpenReview ↗

Response to Reviewer 3FmY(3/3)

Q3: Reliance on VLM judges; 71.7% agreement; detection of scientific content errors.

... the automatic judge agrees with majority human preferences on only 71.7% of the sampled cases. Additional human evaluation, particularly of scientific content correctness and structural relationships, would strengthen the reliability ...

  1. How accurately do the evaluation models detect scientific content errors, particularly incorrect labels, missing components, and invalid relationships?

R3:

The results are stable across judges from three model families (Table A7). As already reported in our paper, rescoring CRAFTBENCH with GPT-5.4 and Claude Sonnet 4.6 in place of the default Gemini 3.5 Flash leaves CRAFTER at 50.2 / 50.2 / 49.1 and its lead over PaperBanana between 19.9 and 22.2 points. The improvement is not an artifact of one evaluator.

The rubric targets exactly the errors you list. The content-faithfulness aspect (Fig. A3) instructs the judge to check that "the specific components, relationships and flow named in the caption are present and correct", to verify that labels are "REAL, specific and correct", and to check whether "any components [are] fabricated or missing"; and it carries the highest weight (3.0 vs. 1.0–1.5 for readability and format). The scoring anchors (Fig. A2) state: "Reward correctness and communication, not surface polish."

The judge separates the systems on content. We further report the raw judge scores by aspect. On content faithfulness the judge gives human-drawn targets 9.08 on average, CRAFTER 8.70, PaperBanana 7.78 and the standalone backbone 7.49 (CRAFTBENCH, Nano Banana Pro; Table A12), and CRAFTER's margin over PaperBanana is larger on content (+0.92) than on readability (+0.55).

The human study already evaluates content correctness. The annotators are instructed to choose the figure that is "faithful to the described content, readable, well-composed" (Fig. A5), so their verdicts cover scientific content and structure, and the judge is validated against these verdicts. We will also enlarge the human study in the final version.

We further computed the human–human agreement among the three annotators (Table A8). The decision is a three-way one (Model / Tie / Human), and the annotators themselves agree pairwise on 78.9% of cases (Fleiss' κ = 0.65), vs. 71.7% between the judge and their majority. Where they are unanimous (41 cases), the judge matches them in 90.2% and never returns the opposite verdict. The remaining gap comes from the Tie option: when the judge and the majority both name a winner, they pick the same figure in 94.9% of cases (37 of 39), vs. 85.4–87.8% between two annotators.

Q4: Does CRAFTEDITOR preserve independent editability of semantic objects?

  1. Does CraftEditor preserve independent editability of semantic objects, rather than merely producing visually faithful SVG renderings?

R4:

Yes, and Appendix I already measures exactly this with human editors. CRAFTEDITOR emits SVGs in which semantic elements are individually addressable objects and labels are genuine editable text nodes. In the editing study of Appendix I, annotators edit each converted output in a standard editor until it matches the original figure, and rate from 1 to 5 "whether elements behave like ordinary slide objects, that is, whether they can be selected, moved, and retyped individually." CRAFTEDITOR reaches a match in 80% of cases with a median time of 2 minutes and receives the highest editability rating (4.20 vs. 3.50 for AutoFigure-Edit and 1.70 for Edit-Banana; Table A9), while 70% of Edit-Banana outputs offer no editable structure.

We will include all of the above results in the revised pdf and mark them in red.

We hope these clarifications address your concerns. If our responses resolve your concerns, we would be grateful if you could consider raising your score. Please feel free to follow up with any further questions—we are happy to respond right away.

Best regards, Authors

Reviewer_f8jbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a practical problem, namely generating scientific figures such as method diagrams, posters and infographics from text and reference inputs, and converting them into editable vector graphics. CRAFTER wraps an off-the-shelf image generator, Nano Banana 2 or Pro, in a four-role "harness" organized around a shared structured specification. An intent reasoner writes the first version of the specification. A plan generator proposes one to three framings rendered in parallel. A VLM critic returns per-dimension diagnostics, which a refiner turns into typed edits on the specification rather than free-text prompt additions, for up to three rounds. CRAFTEDITOR reuses the pattern for raster-to-SVG conversion. The authors also introduce CRAFTBENCH, which has 279 samples spanning three figure styles and four input conditions and is scored by a per-image referenced VLM judge. CRAFTER reports 50.34 versus 33.73 for PaperBanana on PaperBanana-Bench and 50.18 versus 27.96 on CRAFTBENCH. CRAFTEDITOR scores 8.04 compared with 6.91 for AutoFigure-Edit on a three-VLM fidelity ensemble. I thank the authors for a carefully engineered systems paper. The problem setting reflects researchers' real needs, the new benchmark is a valuable extension, and the reporting is quite transparent. My main concern is the attribution of the gains. The controlled comparison described in the main text does not seem to control for the planning model, the evaluation judge is the same model as the system's in-loop critic, and the system is also more expensive. Combined with the fact that each method is run only once, it is currently hard to tell how much of the lead comes from the system design itself.

Critical strengths

The problem setting is practically relevant, and the benchmark is a useful extension. Supporting sketches, masked figures and element collages as inputs, and producing editable SVGs, is much closer to how researchers make figures than text-only generation. CRAFTBENCH's reference-conditioned tasks genuinely extend existing benchmarks, and Appendix B documents their construction with care. The reporting is transparent and internally consistent. Judge prompts are given in full in Figures A2 to A4, missing generations count as losses according to Appendix A.1, and the authors include a judge-family rescoring in Table A7 and a backbone swap in Table A6. I recomputed the sample-weighted overall scores in Tables 2, A2 and A3 from the per-task and per-style columns, and they agree with the reported values.

Critical weaknesses

Section 5 on page 7 shares the generator and VLM across agentic methods "to isolate the effect of orchestration design" and reports a lead over PaperBanana of 16.61 and 22.22 points on the two benchmarks. Appendix C on page 17 shows that CRAFTER's language-only agents use claude-opus-4-6, which the baselines do not. In Table A6 on page 19, CRAFTER with all agents on Gemini 3.1 Pro scores 38.4, about 10.4 points above PaperBanana's 27.96, so roughly half of the reported margin may come from the planner model. No matched-planner number is given for PaperBanana-Bench. In addition, the PaperBanana-Bench judge, which page 13 identifies as Gemini 3.1 Pro, is also CRAFTER's in-loop critic, which picks the best among up to three parallel plans and keeps the best result over up to three refinement rounds according to Appendix J, and CRAFTER costs about 2.3 times more per figure according to Table A5. Matched-planner results, a non-Gemini rescoring and a budget-matched baseline would resolve this. The statistical reliability and independence of the evaluation need to be strengthened. Each method is run once, as stated on page 13. The conditioned tasks have 30 to 40 samples each, and each sample is scored 100, 50 or 0, so a single sample can move a score by 1.25 to 3.33 points depending on the subset size and on whether it gains half or full credit. For example, in the key-element column of Table 2, which has 30 samples, CRAFTER's 40.00 and PaperBanana's 31.67 differ by the equivalent of two and a half samples. Ablations and thresholds appear to be set on the evaluation data. The method prompts are not released while the judge rubric is, and Figure A3 requires, for instance, that posters have a title banner, an author and affiliation line and at least 3 sections. It is therefore hard to tell whether the large poster gap in Table A3, where all baselines score at most 9.63 and CRAFTER reaches 33.94 and 36.70, reflects the harness or prompt alignment with the rubric. The human study in Table A8 covers 60 cases, with 71.7% agreement between the judge and the majority human verdict and a Cohen's kappa of 0.58, but does not test whether the judge recovers system rankings. Multi-seed results with confidence intervals, released prompts, and a ranking-level human check would address this.

Questions for authors
  1. Could the authors report CRAFTER with all agents on Gemini 3.1 Pro on both PaperBanana-Bench and CRAFTBENCH in the main table, or alternatively PaperBanana with the same claude-opus-4-6 planner? Which model do the baselines use for their text-only agents?
  2. It would strengthen the paper to rescore PaperBanana-Bench with non-Gemini judges, as done for CRAFTBENCH in Table A7, and to compare against PaperBanana with best-of-N selection by the same critic at a matched cost.
  3. Would the authors be willing to release all CRAFTER agent prompts, the typed-edit vocabulary and the schema of the specification? In particular, do the critic or planner prompts include the CRAFTBENCH format criteria, for example the poster requirements?
  4. Could the authors report results over several seeds with confidence intervals, state on which data the tie band, weights and early-exit threshold were chosen, and report whether the judge reproduces the human ordering of CRAFTER and PaperBanana, together with inter-annotator agreement?
  5. In Table 3, with each of 292 samples scored 100, 50 or 0, an overall of 45.30 for the "without directive critic" row does not seem attainable, since 264 gives 45.21 and 265 gives 45.38. Could the authors double-check this value and report the number of valid samples per row?
3 comments
AuthorsOpenReview ↗

Response to Reviewer f8jb(1/3)

Response to Reviewer f8jb

We thank Reviewer f8jb for the detailed and careful review, including recomputing our tables, and for recognizing that the problem setting reflects researchers' real needs, that CRAFTBENCH's reference-conditioned tasks genuinely extend existing benchmarks, and that the reporting is transparent and internally consistent. We address the concerns below.

Q1: Planner model; matched-planner results; which model the baselines use.

... CRAFTER's language-only agents use claude-opus-4-6, which the baselines do not ... roughly half of the reported margin may come from the planner model. No matched-planner number is given for PaperBanana-Bench ...

Could the authors report CRAFTER with all agents on Gemini 3.1 Pro ... or alternatively PaperBanana with the same claude-opus-4-6 planner? Which model do the baselines use for their text-only agents?

R1:

CRAFTER keeps a clear lead when the planner is matched, in both directions.

Which model the baselines use. PaperBanana's released configuration has a single model setting shared by all of its agents (retriever, planner, stylist, critic). We run it as released with gemini-3.1-pro-preview, which is also the model of every vision-dependent agent of CRAFTER (Appendix A.1). CRAFTER assigns only its language-only agents to claude-opus-4-6 (Appendix C).

As already reported in our paper, CRAFTER leads with every agent on Gemini 3.1 Pro (Table A6). No model difference is left between the two systems, and CRAFTER still scores 38.4 vs. 27.96 on CRAFTBENCH.

Following your suggestion, we further ran a new experiment that gives PaperBanana the same claude-opus-4-6. We set PaperBanana's model setting to claude-opus-4-6, which moves all of its agents to this model, and reran it on PaperBanana-Bench under the official judge:

System (w/ Nano Banana 2)Language modelPaperBanana-Bench
PaperBanana (reported in our paper)Gemini 3.1 Pro33.73
PaperBananaClaude Opus 4.636.64
CRAFTERClaude Opus 4.650.34

As shown in this table, the stronger language model moves PaperBanana by only 2.91 points, and CRAFTER stays 13.70 points ahead under the identical language model and backbone (95% CI +7.2 to +20.2). The lead comes from the harness, not from the planner model.

Q2: The PaperBanana-Bench judge is also the in-loop critic; cost; budget-matched baseline.

... the PaperBanana-Bench judge ... is also CRAFTER's in-loop critic, which picks the best among up to three parallel plans and keeps the best result over up to three refinement rounds ... and CRAFTER costs about 2.3 times more per figure ...

It would strengthen the paper to rescore PaperBanana-Bench with non-Gemini judges ... and to compare against PaperBanana with best-of-N selection by the same critic at a matched cost.

R2:

The lead does not come from sharing a model between the critic and the judge, nor from the extra generations. Both points are already covered in the paper:

(1) On CRAFTBENCH the judge is not the critic, and the lead is larger there. The lead is 22.2 points under the default judge (Gemini 3.5 Flash) and 19.9 / 20.4 under GPT-5.4 / Claude Sonnet 4.6, none of which appears in CRAFTER's loop (Table A7). If sharing a model inflated the score, the lead would be largest on PaperBanana-Bench; it is smallest there (16.61).

(2) The critic is shared by both systems. PaperBanana's own critic is the same Gemini 3.1 Pro and also steers up to 3 refinement rounds.

(3) Our ablations already serve as budget-matched baselines. You point to two selection steps: the choice among up to three parallel plans and the best result over up to three rounds. Removing the first (K=1, a single plan with the same 3 rounds PaperBanana uses), CRAFTER scores 41.78 vs. 33.73; removing the second (T=1, a single round), it scores 44.86 (Table 3, Table A4). K=1 gives CRAFTER the same generation budget as PaperBanana while keeping PaperBanana in its official configuration. Regarding the cost, the full system costs 0.25 USD per figure vs. 0.11 USD (Table A5), i.e. under 90 USD for the whole of CRAFTBENCH. The advantage is therefore not purchased by inference budget.

We keep PaperBanana-Bench under its official judge so that the numbers remain comparable with the published ones, and report the cross-family rescoring on CRAFTBENCH (Table A7).

AuthorsOpenReview ↗

Response to Reviewer f8jb(2/3)

Q3: Single run, subset sizes, confidence intervals.

Each method is run once ... CRAFTER's 40.00 and PaperBanana's 31.67 differ by the equivalent of two and a half samples ...

Could the authors report results over several seeds with confidence intervals ...

R3:

The lead of CRAFTER is far larger than the variation across runs and across samples.

To fully address your suggestion, we further ran a new experiment with repeated runs. We ran both systems three times on PaperBanana-Bench under the setting of Table 2:

Method (w/ Nano Banana 2)Run 1 (reported in our paper)Run 2Run 3Mean ± std
PaperBanana33.7334.0833.5633.79 ± 0.27
CRAFTER50.3450.6852.2351.08 ± 1.01

As shown in this table, each method varies by less than 2 points across runs, while CRAFTER leads by 17.29 points on average and by more than 16 points for any pairing of runs.

Meanwhile, we further computed confidence intervals for the lead. We report percentile-bootstrap 95% confidence intervals over samples (10,000 resamples) and exact sign tests:

CRAFTER vs. PaperBanananGap [95% CI]Better / worseSign test
PaperBanana-Bench (NB2)292+16.61 [+10.3, +22.8]93 / 37p < 10⁻⁶
PaperBanana-Bench (NB-Pro)292+14.04 [+7.7, +20.2]90 / 41p < 10⁻⁴
CRAFTBENCH (NB-Pro), all279+23.30 [+18.8, +27.6]133 / 24p < 10⁻¹⁸
CRAFTBENCH (NB-Pro), reference-conditioned100+19.00 [+11.5, +26.5]44 / 12p < 10⁻⁴

Every interval excludes zero by a wide margin. Regarding the subset sizes, our claim for the reference-conditioned setting rests on the 100 samples together (44 wins vs. 12 losses); the per-task columns of Table 2 are a breakdown of this result, and CRAFTER is ahead in all six of them (three tasks, two backbones).

Q4: Tie band, weights and early-exit threshold; human ordering; inter-annotator agreement.

Ablations and thresholds appear to be set on the evaluation data ...

... state on which data the tie band, weights and early-exit threshold were chosen, and report whether the judge reproduces the human ordering of CRAFTER and PaperBanana, together with inter-annotator agreement?

R4:

The judge's parameters were set to match human judgment, and none of our conclusions depends on their values.

  • Tie band: calibrated on the human-annotated set of Appendix H, on the plateau where the agreement with the annotators is stable (κ = 0.58 for every band between 0.29 and 0.42). It was thus chosen to make the judge agree with people, and the same band is applied to every system. The agreement is the same 71.7% with no tie band at all, so the reported agreement is not a product of the calibration.
  • Aspect weights: they encode the priority stated in the rubric, correctness before polish (Fig. A2, A3).
  • Early-exit threshold (6.5/10): selected on a held-out validation set and used unchanged on both benchmarks.
  • Ablations: Table 3 measures the contribution of each mechanism; no component was selected through it.

We further ran a new sensitivity analysis. We recomputed all verdicts from the stored scores under different tie bands and weights: the gap on CRAFTBENCH (Nano Banana Pro) stays between 22.2 and 25.8 points for every tie band from 0 to 0.5 (23.3 at the default 0.30), and it is 26.2 with equal aspect weights. The ranking of all systems is unchanged.

We further computed the inter-annotator agreement (Table A8). The three annotators agree pairwise on 78.9% of the 60 cases (Fleiss' κ = 0.65). The judge agrees with their majority on 71.7% (κ = 0.58), and on 90.2% of the 41 cases where they are unanimous. The system ordering depends on which figure wins: when both name a winner, the judge picks the same figure as the majority in 94.9% of cases (37 of 39), vs. 85.4–87.8% between two annotators. We will also enlarge the human study in the final version.

AuthorsOpenReview ↗

Response to Reviewer f8jb(3/3)

Q5: Prompt release; do the critic or planner prompts include the CRAFTBENCH format criteria?

The method prompts are not released while the judge rubric is ... It is therefore hard to tell whether the large poster gap ... reflects the harness or prompt alignment with the rubric.

Would the authors be willing to release all CRAFTER agent prompts, the typed-edit vocabulary and the schema of the specification? In particular, do the critic or planner prompts include the CRAFTBENCH format criteria, for example the poster requirements?

R5:

Yes, we will release all of them: all agent prompts, the specification schema and the typed-edit vocabulary (six operations: set, resize, demote, add-note, add-avoid, add-region-fix), together with the code and the benchmark.

The poster requirements are part of the task, and PaperBanana is given them explicitly. They are stated in the task instruction itself, e.g. "Multi-section layout with a clear poster structure: large title banner at the top with the paper title + author/affiliation strip, then 3-6 distinct content panels arranged in columns or a grid." PaperBanana receives this instruction on these samples, and the format aspect of the judge (Fig. A3) checks what the instruction asks for.

CRAFTER's prompts contain no judge rubric. The harness holds role-level layout guidance for each figure type, selects the one that matches the role of the requested figure, and has the planner expand it into a per-paper plan. This guidance was written before the judge rubric existed. It is the role-aware planning of Section 3 and the mechanism under test: a pipeline specialized for academic diagrams does not change its behavior when the instruction asks for a poster, while a harness that reasons about the communicative role of the figure does.

The lead does not depend on the non-academic types. On PaperBanana-Bench, which contains only academic diagrams and has no format aspect, CRAFTER leads by 16.61 points; on the academic subset of CRAFTBENCH it leads by 18.57 points, as already reported in Table A3.

Q6: The 45.30 in Table 3; valid samples per row.

In Table 3, with each of 292 samples scored 100, 50 or 0, an overall of 45.30 for the "without directive critic" row does not seem attainable ... Could the authors double-check this value and report the number of valid samples per row?

R6:

The value is correct: each ablation row is the mean of two runs, and every row uses all 292 samples. For this row the two runs score 45.38 and 45.21, exactly the two values you computed. Each run is rounded to two decimals before averaging, and the mean is rounded again (half up): (45.38 + 45.21) / 2 = 45.295 → 45.30. We now state this in Appendix A.1.

Thanks for the careful check, and we apologize for an oversight it revealed: in the per-dimension cells of the ablation rows, the few samples whose judge call failed were left out (N = 286–292) instead of being re-judged. We have re-scored these cells on all 292 samples and updated Table 3 and Table A4:

Configuration (mean of two runs, N = 292)Faith.Conc.Read.Aesth.Overall
w/o plan exploration28.3451.2038.2760.1941.78
w/o corrective layer27.4049.6636.9058.3141.44
w/o refinement loop30.0551.8841.3661.8244.86
w/o directive critic31.4252.9142.4763.4545.30

We will include all of the above results in the revised pdf and mark them in red.

We hope these clarifications address your concerns. If our responses resolve your concerns, we would be grateful if you could consider raising your score. Please feel free to follow up with any further questions—we are happy to respond right away.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading and constructive feedback. We are encouraged that the reviewers recognize that the paper addresses a real and practically important problem (f8jb, 3FmY), integrates figure generation and editable SVG conversion into a useful end-to-end workflow (Xuxw, 3FmY), contributes CRAFTBENCH, whose reference-conditioned tasks "genuinely extend existing benchmarks" (f8jb) across four input conditions and three figure types (Xuxw, 3FmY), achieves substantial improvements over the baselines with ablations that confirm each component (Xuxw, 3FmY), and reports results that are "transparent and internally consistent" (f8jb). We are pleased to note that most reviewers hold positive opinions towards this work. Below we briefly summarize our responses. Most concerns are addressed by evidence already in the submission:

  • Planner model (f8jb-W1, 3FmY-W2): with every agent on Gemini 3.1 Pro, the single model PaperBanana uses for all of its agents, CRAFTER still scores 38.4 vs. 27.96 on CRAFTBENCH (Table A6), so the lead does not come from the planner model.
  • VLM-judge validity (Xuxw-W1, f8jb-W1, 3FmY-W3): the CRAFTBENCH judge is not the in-loop critic, and the lead is preserved under judges from three model families (22.2 / 19.9 / 20.4 points under Gemini / GPT-5.4 / Claude, Table A7), so judge preference cannot explain the controlled gap; the judge is also human-validated (Appendix H).
  • Inference budget (f8jb-Q2, 3FmY-Q1): our ablations already serve as budget-matched baselines: restricted to a single plan and the same rounds as PaperBanana, CRAFTER scores 41.78 vs. 33.73 (Table A4), so the advantage is not purchased by inference budget.
  • Novelty and mechanism design (3FmY-W1): the contribution is a single harness that generalizes across figure types and input conditions without architectural change (Section 3.1); the equally agentic PaperBanana fails to generalize under the identical backbone (its gain shrinks from +22.60 to +8.06, Table 2), and replacing typed edits with free-text revisions costs 8.90 points (Table 3).
  • Poster requirements and prompts (f8jb-Q3): the poster requirements are stated in the task instruction that PaperBanana also receives, CRAFTER's prompts contain no judge rubric, and the lead holds on academic figures alone (16.61 / 18.57 points on PaperBanana-Bench / the academic subset of CRAFTBENCH, Table A3); all prompts will be released.
  • Editability (3FmY-Q4): independent editability is already measured with human editors (Appendix I): CRAFTEDITOR receives a 4.20/5 editability rating vs. 3.50 for AutoFigure-Edit and 1.70 for Edit-Banana (Table A9).

In addition, we conducted the following new experiments and analyses during the rebuttal period:

  • PaperBanana with Claude Opus 4.6 (f8jb-Q1, 3FmY-W2): moving all of PaperBanana's agents to CRAFTER's language model raises it only from 33.73 to 36.64 on PaperBanana-Bench, vs. 50.34 for CRAFTER (a lead of 13.70 points, 95% CI +7.2 to +20.2).
  • Repeated runs (f8jb-Q4): over three runs on PaperBanana-Bench, CRAFTER scores 51.08 ± 1.01 vs. 33.79 ± 0.27 for PaperBanana.
  • Confidence intervals (f8jb-W2): bootstrap 95% intervals of the lead exclude zero by a wide margin (+16.61 [+10.3, +22.8] on PaperBanana-Bench; +23.30 [+18.8, +27.6] on CRAFTBENCH with Nano Banana Pro).
  • Human–human agreement (Xuxw-Q1, f8jb-Q4, 3FmY-W3): the three annotators agree pairwise on 78.9% of cases (Fleiss' κ = 0.65) vs. 71.7% for the judge against their majority; when both name a winner, the judge picks the same figure in 94.9% of cases, vs. 85.4–87.8% between two annotators.
  • Raw judge scores (Xuxw-Q3): the raw 0–10 scores preserve the ordering of the verdicts (8.71 for CRAFTER vs. 7.72 for PaperBanana, on par with the human-drawn targets).
  • Tie-band sensitivity (f8jb-Q4): the gap on CRAFTBENCH (Nano Banana Pro) stays between 22.2 and 25.8 points for every tie band from 0 to 0.5.

We will include these results in the revised pdf (highlighted in red).


We hope these clarifications address the reviewers' concerns and we thank all reviewers for their thoughtful evaluation and recognition toward our paper.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.