- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies scientific diagram editing from rendered images without access to the original source code. It introduces DiagramEditBench, comprising approximately 41K training examples and 718 human-curated evaluation examples, and proposes DiagramEditor, built on Qwen3.5-9B. The approach combines a Hierarchical Structuring Strategy (HSS) for organizing training programs with GRPO training using image-, code-, and vector-level rewards. Experiments demonstrate substantial improvements in compilation reliability and competitive automatic evaluation results, together with transfer to diagram reconstruction.
Critical strengths
1.A practically relevant problem with a clear task formulation. Editing diagrams when only rendered images are available addresses a common limitation in scientific workflows. Producing executable TikZ programs also supports subsequent modification and high-quality vector rendering.
2.A substantial dataset construction effort. The pipeline includes compilation checks, visual filtering, deduplication, code restructuring, instruction synthesis, and manual benchmark verification. The taxonomy of editing targets and compositional difficulty provides a useful foundation for studying diagram editing.
Critical weaknesses
. The discrepancy between automatic metrics and human evaluation weakens the central quality claims.
Although DiagramEditor leads on most automatic metrics, its human evaluation score is 3.42, below Gemini-3.5-Flash at 4.35 and GPT-5.5 at 4.01. This does not negate the contribution, but it shows that the automatic improvements do not straightforwardly establish superior editing quality.
Furthermore, non-compiling outputs receive minimum image and DVR scores. Consequently, these metrics combine compilation reliability with the quality of successfully rendered outputs. The current results do not sufficiently distinguish improvements from fewer compilation failures from improvements in instruction fulfillment and preservation of unrelated content.
The evaluation should report both overall performance and quality conditional on successful compilation, with the evaluated subsets clearly specified. The human evaluation protocol also needs clarification regarding sample size, sampling procedure, and treatment of compilation failures. Separate measurements of edit correctness and preservation would more directly support the structure-preserving editing claim than overall similarity to a reference image.
- The independent contributions of the proposed components are not yet sufficiently established.
The SFT and reward ablations are valuable. However, Table 2 shows that the largest improvement comes from image-reward RL: DreamSim increases from 67.68 to 81.18. Once image and code rewards are included, adding DVR increases DreamSim from 81.93 to 82.45, with gains of 1.34, 0.85, and 1.28 in entity, connection, and attribute scores, respectively.
These results suggest that DVR is useful, but the additional gains are modest and are primarily demonstrated through metrics directly optimized during training. Independent human comparisons or edit-completion measurements are needed to establish whether these gains translate into meaningful improvements in editing quality.
Similarly, the HSS ablation primarily reports compilation rates. It does not directly validate the claimed benefits for localized editing and preservation of unrelated content. Targeted comparisons against the corresponding ablated models would strengthen the methodological contribution without requiring an extensive expansion of the experiments.
- The relationship between DVR scores and semantic structural correctness requires further validation.
The connection reward is based on geometric matching within the connection pool, independently of entity matching. This can assess whether lines appear in similar locations, but the description does not clearly establish whether they connect the correct entities or encode the correct directional relationships.
A related concern applies to text rewards. After text-based matching, unmatched text boxes may receive geometric fallback matches, while the final reward primarily aggregates spatial overlap. It is therefore unclear whether an incorrect label at a similar location could still receive a high score.
These concerns do not imply that DVR is ineffective, but they limit its interpretation as a measure of semantic or topological correctness. A small set of controlled perturbations—such as replacing labels, changing connection endpoints, or reversing arrows—would help establish whether the reward reliably detects important errors. The conversion from PDF primitives to entities and connections also requires a clearer description.
- Dataset coverage and cross-task generalization need stronger documentation.
Training and evaluation examples are produced through similar instruction-synthesis and target-code-generation pipelines. Temporal splitting and visual deduplication help reduce overlap, but do not independently establish generalization across instruction styles, diagram categories, and generation patterns. The main paper reports aggregate results without sufficient breakdowns by editing operation, difficulty, or diagram type.
In addition, both the training corpus and DaTikZ v3 involve scientific diagrams and TikZ sources, while the main paper does not clearly document deduplication between all training sources and the reconstruction test set. The reconstruction results provide promising evidence of transfer, but their strength depends on the independence of the evaluation data.
The paper should provide dataset distributions, subgroup results, and a clear account of overlap checks against DaTikZ v3. This is a request for stronger documentation, rather than an allegation of data leakage.
Questions for authors
How many examples were included in the human evaluation, and how were compilation failures handled? What explains the ranking discrepancy between human and automatic evaluation?
Does the full model outperform the image-plus-code-reward variant under independent human judgments or edit-completion measurements?
How does DVR respond to incorrect labels, wrong connection endpoints, and reversed arrows? Does connection matching explicitly depend on entity correspondences?
Were all training sources deduplicated against the DaTikZ v3 test set? Can the complete HSS and DVR implementations be provided?
Possible Review-Submission Mismatch
Dear Reviewer fJAq,
Thank you for your time. It seems that this review may have been intended for another submission, as it discusses DiagramEditBench, DiagramEditor, HSS and DVR scores for TikZ diagram editing, whereas our submission focuses on adapting layered graphic designs to different canvas sizes. We would appreciate it if you could check this and update the review to reflect your assessment of our submission. We would be happy to clarify any aspects of our work.