- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formulates scientific methodology figure generation as recursive SVG program construction. They propose a multi-agent system called FigTree, composed of a Grounder, a tree of Generators, and a Critic. The Grounder extracts a content plan and a global style specification from the paper. The Generator then expands a layout tree top-down: each node is either drawn as a leaf SVG fragment or decomposed into 2~4 children. Each child returns its fragment, bounding box, and named ports, and the parent places the children and draws all cross-boundary connections. The Critic inspects both the rendered image and the SVG code and returns element-anchored repair instructions, guided by a verification memory distilled offline on AutoFigure-Edit Bench. Experiments are run on PaperBananaBench (292 cases), with Claude Sonnet 4.6 as the generator and Gemini 3.1 Pro used as both the critic and the judge. FigTree achieves the best reference-based score and the best fine-grained diagnostic score among tested methods.
Critical strengths
The paper shifts scientific figure generation from raster images to editable SVG programs. The recursive decomposition, with child-reported ports and parent-owned connections, is a clean way to handle long-horizon coordinate generation. The comparison with Direct-SVG, which uses the same Claude Sonnet 4.6 backbone (3.60 vs. 59.59 overall in Table 1), shows that the gain comes from the system design rather than the backbone. Also, FigTree shows clear improvement in faithfulness and hallucination control, which is arguably the most important property for scientific figures.
Critical weaknesses
The evaluation is closely coupled to the method itself. Gemini 3.1 Pro is used both as the in-loop critic and as the only automatic judge, and the verification memory is distilled by the same critic. Moreover, the generator and critic prompts (flat fills, no gradients, thin strokes, compact canvas, Times New Roman) mirror the authors’ own diagnostic rubric, where Visual Polish penalizes gradients, Compactness penalizes blank margins, and Text/Glyph targets raster artifacts. The method is therefore largely optimized against its own evaluator. The sample size is also unclear. Table 2 is described as evaluated on PaperBananaBench, but its means are consistent with integer scores averaged over only about 20–22 cases, with different n across methods. For example, every entry for Crafter, PaperBanana, Direct-SVG and AutoFigure-Edit is a multiple of 1/22, such as Crafter’s 7.41 = 163/22. If so, the 8.39 vs. 8.28 margin over GPT-Image-2 is within noise. The human study does not solve this either. A Spearman ρ of 0.891 over 10 methods is dominated by the gap between strong and weak systems, and tells little about the ordering among the top methods.
The ablation results are hard to explain, and the editability claim is weakly supported. Removing verification memory drops the overall score to 4.43. This is far below removing the entire render-critic loop (7.86), and even below Direct-SVG (6.40), which has no grounding, recursion, or critic. Content grounding also drops by 5.23, although the memory is described as visual and structural repair rules. Yet the paper gives almost no information about this memory, such as the number of rules, examples, or whether AutoFigure-Edit Bench overlaps with PaperBananaBench. Moreover, GPT-Image-2 is already close to FigTree in Table 2 and even better on topology and flow, so editability becomes the main advantage of FigTree. However, editability is only evaluated on 5 figures against a single raster model, without a vector baseline such as AutoFigure-Edit and without any human editing study.
Questions for authors
-
Please re-score Table 1 and Table 2 with a VLM judge from a model family not used in FigTree, and report whether the ranking and margins hold. For the human study, were the evaluators blinded, and are any of them authors? Please report inter-annotator agreement and the confidence intervals of HPS in Table 4.
-
Why does removing verification memory (4.43) perform worse than removing the whole render-critic loop (7.86) and Direct-SVG? How many rules does the memory contain? Was AutoFigure-Edit Bench disjoint from PaperBananaBench at the paper level?
-
Section 4.1 says each method receives only the method description and caption, but the grounder prompt in Appendix D.1 takes the full paper text. Which input did FigTree actually receive?
-
In the iterative-editing evaluation, which model edits each branch, and how were the defects injected? Why are AutoFigure-Edit and Crafter not included, given that Crafter’s title also claims editable figure generation?
4 comments
Table 2
| Method | Sem. Cov. | Halluc. Ctrl. | Topology | Flow | Text/Glyph | Polish | Acad. Style | Compactness | Overall |
|---|---|---|---|---|---|---|---|---|---|
| Wan2.6-Image | 3.70 | 3.65 | 3.84 | 5.55 | 3.41 | 4.24 | 3.11 | 6.73 | 4.28 |
| Nano Banana Pro | 6.06 | 3.16 | 5.03 | 5.66 | 5.30 | 3.89 | 2.69 | 6.07 | 4.73 |
| FLUX.2-Pro | 3.95 | 3.52 | 4.02 | 5.30 | 3.21 | 4.79 | 3.52 | 6.59 | 4.36 |
| Qwen-Image-2.0-Pro | 3.77 | 4.79 | 3.89 | 6.49 | 7.54 | 6.27 | 4.60 | 7.41 | 5.60 |
| GPT-Image-2 | 7.99 | 7.23 | 8.11 | 7.95 | 7.94 | 7.92 | 8.33 | 7.75 | 7.90 |
| Crafter | 6.52 | 5.68 | 6.40 | 7.73 | 8.08 | 7.47 | 6.95 | 6.56 | 6.91 |
| PaperBanana | 6.15 | 4.34 | 5.66 | 6.18 | 6.22 | 5.60 | 4.71 | 6.57 | 5.68 |
| Direct-SVG | 6.18 | 6.34 | 5.73 | 6.35 | 5.94 | 5.96 | 5.92 | 5.77 | 6.02 |
| AutoFigure-Edit | 1.74 | 1.37 | 1.45 | 1.45 | 2.15 | 1.73 | 1.05 | 1.55 | 1.56 |
| FIGTREE | 8.19 | 8.35 | 7.84 | 7.80 | 8.24 | 8.39 | 8.60 | 7.74 | 8.11 |
For the human study, we sampled 50 cases from PaperBananaBench. All three PhD-level evaluators were blinded to method identity, and none of them are authors of this paper. Each evaluator independently ranked all outputs, with ties allowed. Inter-annotator agreement is Fleiss’ on pairwise preferences, indicating fair agreement, and Kendall’s on full rankings, indicating substantial agreement. We report 95% bootstrap confidence intervals for HPS below and will add them to our manuscript.
Table 3
| Method | HPS | 95% CI |
|---|---|---|
| FIGTREE | 96.36 | [93.4, 99.0] |
| GPT-Image-2 | 92.45 | [88.7, 96.1] |
| Crafter | 74.17 | [66.9, 81.1] |
| PaperBanana | 67.22 | [59.2, 74.8] |
| Direct-SVG | 57.97 | [49.6, 66.0] |
| FLUX.2-Pro | 30.17 | [22.6, 38.2] |
| Wan2.6-Image | 30.14 | [22.5, 38.1] |
| Qwen-Image-2.0-Pro | 27.02 | [19.7, 34.9] |
| Nano Banana Pro | 24.49 | [17.5, 32.2] |
| AutoFigure-Edit | 0.00 | [0.0, 1.2] |
Q2. Explain the verification-memory ablation and provide details of the memory.
Thank you for identifying an ambiguity in our explanation. We agree that the paper should describe the verification memory and this ablation in more detail.
Why removing the memory is worse than removing the loop. Removing the entire render–critic loop preserves the initial output of paper-grounded recursive construction. This output can retain the correct components and relationships despite unresolved visual defects. In contrast, retaining refinement without verification memory exposes that output to additional edits without the same rule-guided diagnosis and repair constraints. Such edits can improve a local appearance while damaging an otherwise correct fragment—for example, by merging distinct components to reduce crowding, changing a label’s meaning, or moving a block without consistently updating its connections. Repeated edits across the tree can compound these errors. Thus, the two ablations remove different safeguards: one avoids refinement altogether, whereas the other retains a refinement process that can introduce additional errors.
This distinction also addresses the 5.23-point drop in Content. The consequences of an incorrect repair are not limited to typography or spacing. Deleting, conflating, or relabeling a paper-supported component, or changing a semantic connection, directly reduces content fidelity. Verification memory therefore supports both defect correction and preservation of already-correct scientific content; it is more than an aesthetic checklist.
Memory size and content. The memory contains 65 reusable condition–repair rules, distilled offline and frozen during evaluation. Representative principles are:
- Paper Grounding: Preserve paper-supported modules, notation, dimensions, and connection directions. Repair visual defects without deleting scientific content or introducing unsupported elements.
- Composition Before Coordinates: Resolve the scientific relationships and visual structure before adjusting positions. Layout repairs must preserve those relationships rather than replace distinct concepts with uniform text boxes.
- Scientific Visual Primitives: Use tokens, patches, matrices, tensors, and distributions where they clarify the method. Preserve their ordering and dimensional correspondence, and distinguish schematic illustrations from experimental results.
- Validation and Visual Review: Correct overflow, overlaps, and arrows crossing unrelated elements; then check scientific accuracy, proportions, typography, and readability. Anchor each repair to specific elements and preserve unrelated content.
Dataset separation. AutoFigure-Edit Bench, used to construct the memory, is fully disjoint from PaperBananaBench at the paper level, not merely at the image level. The frozen memory transfers reusable defect–repair rules rather than paper-specific examples from the evaluation set.
Q3. Clarify the input provided to FIGTREE.
Thank you for pointing out this inconsistency. FIGTREE, like all baselines, receives only the method description and caption, as stated in Section 4.1. PaperBananaBench also provides the method description, instead of full paper. The references to the “full paper” in Appendix D.1 are wording problem. We will revise them to refer to the method-section text, and update the notation in Section 3.1 accordingly. This does not affect the experimental setup or the fairness of the comparisons.
Q4. Clarify the iterative-editing setup and the exclusion of AutoFigure-Edit and Crafter.
We thank the reviewer. The vector branch uses Claude Sonnet 4.6 to edit the SVG code, while the raster branch uses GPT-Image-2 to edit the image. Both branches start from corresponding versions of the same manually defect-injected figure and use the same defect-free gold-standard reference. Injected defects cover misaligned arrows, overlapping elements, text overflow, incorrect connections, and poor color harmony.
At each round, Gemini 3.1 Pro receives the gold-standard figure and the current rendering of the respective branch, then generates an edit request using the same prompt template. The resulting requests are branch-specific: once the outputs diverge, their remaining defects also differ. Thus, we control the initial defects, reference target, feedback model, and feedback-generation template, rather than requiring identical edit requests across subsequent rounds. We will revise the manuscript to make this distinction explicit.
Moreover, we would like to clarify that this experiment is designed to compare the editability of SVG output with that of its raster counterpart. The generation quality of FIGTREE relative to other baselines has already been evaluated in Tables 1 and 2 of our manuscript.
Follow-up discussion
Dear reviewer H4Hp,
We greatly appreciate your efforts to review our submission and your invaluable comments. We have tried our best to carefully consider each of your suggestions and conduct all required experiments.
As the discussion period will conclude today, Nov 18, we respectfully invite you to review our responses. We hope that our additional experiments and clarifications have adequately addressed your concerns. Nevertheless, we remain fully prepared to provide further clarification or discussion on any outstanding points. If you find that our responses have resolved your concerns, we would be deeply grateful if you would consider raising your score in support of our submission.
Thank you again for your thoughtful and valuable review.
Best regards,
Paper14643 Authors
W1. Clarify the independence between the evaluation and the method.
Thank you for raising this important concern. We address them below.
Shared judge. We re-scored Tables 1 and 2 with an independent VLM judge,
doubao-seed-2-1-pro-260628, from a model family not used anywhere in FIGTREE (full results in our response to Q1). The method ranking is fully preserved: FIGTREE remains the best under both the reference-based protocol (56.65 vs. 48.74 for Crafter) and the fine-grained diagnostic protocol (8.11 vs. 7.90 for GPT-Image-2).Verification memory distilled by the critic. The memory is distilled offline on AutoFigure-Edit Bench, which is disjoint from PaperBananaBench at the paper level, and is frozen before evaluation. If the memory merely encoded Gemini-specific preferences, FIGTREE’s advantage would diminish under a different judge. Instead, it persists under the independent Doubao judge, suggesting that the rules capture general defect-repair patterns.
Prompts aligned with the rubric. To verify that the effectiveness of our method does not depend on our own rubric, we further evaluate it with the official PaperBananaBench reference-based protocol, whose criteria were not designed by us. Our method achieves the best overall score as well. (59.59 vs. 50.00 for the strongest baseline, Crafter; Table 1).
W2. Clarify the sample size of Table 2 and the ranking agreement among top methods.
We thank the reviewer for these sharp observations.
On sample size: You are exactly right about the 1/22 multiples. We acknowledge that these entries may have been contaminated by an early 22-case debugging subset. We therefore re-ran the four affected methods, Crafter, PaperBanana, Direct-SVG, and AutoFigure-Edit, and re-checked the remaining methods; the remaining methods are confirmed to be correct. The updated scores for the four affected methods are:
This does not affect the conclusion. We assess the margin with the full-set Win/Tie/Lose comparison described in our response to PzaL W2: against GPT-Image-2, FIGTREE wins 148, ties 96, and loses 42 across 286 cases, with a weighted score of 68.53% and a two-sided exact binomial p<0.001 on 190 non-tied cases.
On the Spearman ρ: We agree that the overall ρ=0.891 is heavily driven by the gap between strong and weak systems. Following your suggestion, we split the methods into Top-5 and Bottom-5 groups to evaluate intra-group ranking. Within the Top-5 group (FIGTREE, GPT-Image-2, Crafter, Direct-SVG, PaperBanana), the Spearman ρ is 0.90. Specifically, both the VLM and human judges agree perfectly on the top-3 (FIGTREE > GPT-Image-2 > Crafter), with only a minor swap between Direct-SVG and PaperBanana at ranks 4–5. This demonstrates that the VLM judge effectively distinguishes fine-grained ordering among strong methods. We will include this split-group analysis in the revision.
W3. Clarify the ablation results and strengthen the evidence for editability.
Please see Q2 and Q4.
Q1. Re-score with an independent VLM judge and report details of the human study.
Thank you for this helpful request. We re-scored Tables 1 and 2 with an independent VLM judge,
doubao-seed-2-1-pro-260628. The method ranking is preserved under this judge. FIGTREE remains the best under the reference-based protocol (56.65 vs. 48.74 for Crafter) and the fine-grained diagnostic protocol (8.11 vs. 7.90 for GPT-Image-2).Table 1