acceptodds
Under review as a conference paper at ICLR 2027

UniDiagCode: A Multi-Format Benchmark for Image-to-Code Diagram Generation

Abstract

Image-to-code diagram generation converts rasterized diagrams into editable code representations. Prior work largely targets a single format, despite substantial differences in expressivity, code complexity, and downstream compatibility. We introduce UniDiagCode, a unified six-format benchmark comprising 341K real-world rendered-image/code pairs and 6.3K hand-drawn sketches. We evaluate large vision language models across formats and reveal substantial differences in visual fidelity, renderability, and generation cost. We further formulate target-format selection as an input-dependent decision, allowing the model to choose an appropriate format for each diagram. This model-driven selection achieves a better quality–cost trade-off than fixed-format generation, while fine-tuning on our data substantially improves performance under both format specification and selection. By treating the output format as a model decision rather than a predefined constraint, our work offers a flexible and practical framework for code-based diagram generation.

Primary area:
datasets and benchmarks
Keywords:
diagram generation, large vision-language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rNS1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UniDiagCode, a benchmark for image-to-code diagram generation covering six output formats, with approximately 341K rendered image–code pairs and 6.3K hand-drawn sketches. It compares generation in a specified format with model-driven format selection, evaluating reconstruction fidelity, visual quality, rendering failures, and code length. Experiments across commercial and open-weight models show that sufficiently capable models can reduce code length through format selection while maintaining comparable quality.

Critical strengths
  1. The paper takes an interesting perspective by treating the output format as a model decision rather than a fixed constraint in image-to-code diagram generation.
  2. The paper provides extensive technical details, making the experimental setup easier to understand and reproduce.
Critical weaknesses
  1. The paper does not yet make a strong enough case for adaptive format selection over a single general-purpose format. SVG and TikZ can already represent the diagrams considered in this benchmark. The experiments show that allowing the model to choose a format can reduce code length, but it remains unclear whether these savings are substantial enough in practice to justify treating format selection as a separate research problem. How much time or money do these reductions actually save per diagram? Without a clearer account of their practical impact, it is difficult to judge whether the benefits warrant the added complexity and further research effort.

  2. The cost evaluation does not fully establish the claimed efficiency gains. Code length is measured only for successful outputs, leaving out the tokens spent on failed attempts and any retries or repairs needed to obtain a usable result. Some models also produce shorter code at the expense of reconstruction quality or rendering reliability. Shorter successful outputs therefore do not necessarily mean lower overall cost.

Questions for authors
  1. Could the authors explain why the code-length savings justify devoting substantial research and engineering resources to choosing among diagram formats? Given that general-purpose formats can already represent these diagrams, what makes the benefits of format selection substantial enough to warrant this additional effort?

  2. Are code-length comparisons computed on the same subset of diagrams that succeed under both settings, or on each setting’s successful outputs separately? Based on the available results, does the efficiency advantage remain when failed-generation tokens are included and reconstruction quality is comparable?

5 comments

Thank you for your constructive review. We are glad that you found our perspective of treating the output format as a model decision rather than a fixed constraint interesting, and that our technical details make the setup easier to reproduce. Below, we address each of your concerns. In our responses, we refer to the parts added to the revised version, which are highlighted in blue.

Format selection vs. a single general-purpose format

We agree that SVG and TikZ can, in principle, represent the diagrams in our benchmark. However, representational coverage alone does not determine whether a format is equally suitable for model-based generation. The level of detail the LLM must specify for diagram generation varies across formats. For example, SVG requires the model to specify the geometry and placement of individual elements, often resulting in longer, lower-level code. By contrast, formats such as Graphviz, Mermaid, and PlantUML allow the model to specify diagram elements and their relationships while the rendering tool handles much of the layout, resulting in more concise, higher-level code.

This distinction matters because the formats place different demands on the model, which can affect both generation cost and output quality. In our experiments, our 8B model fine-tuned on our benchmark performs substantially better in these higher-level formats. As shown in Figure 3, generating Graphviz, Mermaid, and PlantUML diagrams in their own formats yields an IA 0.3–1.2 points higher than generating them in SVG. As shown in Table 4, even in the format-selection setting, its DQ is 3.7, higher than the 3.2 obtained when fixing the format to SVG. Enabling small models to generate high-quality diagrams at low cost has great practical value.

Format selection also reduces cost. For our fine-tuned 8B model, the number of output tokens per diagram, including failed generations, decreased from 2,890 when fixing the format to SVG to 1,893 with format selection, about one third fewer, as shown in Table 19 in Appendix H. Because output tokens are generated one by one, this reduction shortens generation time on the same hardware. For proprietary models, format selection reduced the API cost per 1,000 diagrams from \12.91to12.91 to \\10.08 for GPT-5.4 and from \27.13to27.13 to \\17.06 for Claude Opus 4.7, that is, by 22% and 37%, as shown in Table 20 in Appendix H. This corresponds to savings of about \2,800and2,800 and \\10,000 per million diagrams. For large-scale conversion, such as document digitization or training data creation, these savings are not negligible.

Our results suggest that format selection is not merely a way to reduce code length: for smaller models, choosing a format suited to their capabilities can also improve diagram quality. We therefore study whether LLMs can adapt their output representation to the target diagram and their own capabilities, and evaluate how this choice affects generation quality, rendering failures, and cost. This question is related to prior work on selecting representation formats or programming languages according to the task [1,2,3]. Our contribution is to examine this issue for diagram generation and to provide a benchmark for measuring these effects.

  • [1] Chen et al. Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication. EMNLP 2024 Findings.
  • [2] Twist et al. A Study of LLMs’ Preferences for Libraries and Programming Languages. ACL 2026 Findings.
  • [3] Xiao et al. Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding. CVPR 2025.

Cost analysis including failed generations

The code length in the paper is not restricted to the same diagrams that succeeded in both settings; it is computed separately over the outputs that rendered successfully in each setting. Note that even when we recomputed it only on the same diagrams that succeeded in both settings, the ratio to the original format hardly changed, as also described in Appendix G of the revised version.

On the other hand, as the reviewer points out, this computation does not include the tokens spent on failed generations. We therefore recounted the output tokens of all requests, including failed generations, and compared them as ratios to the setting that specifies the original format; the full results are shown in Table 18 in Appendix G of the revised version. As a result, for models with sufficient code-generation capability, the efficiency advantage of format selection was maintained even when failures are included. We also found that models without sufficient capability can acquire it through training.

For models with sufficient code-generation capability, format selection reduced the number of output tokens even when failures are included, and the ratios were almost the same as the code-length ratios reported in the paper. For example, the ratio is 0.59 for Gemini 3.1 Pro and 0.66 for Claude Opus 4.7. The newly evaluated open model Qwen3.8-27B also reduced the number of output tokens, with a ratio of 0.37, mainly because its failed generations in the original format were much longer than those in the format-selection setting. These models also largely maintain their quality. GPT-5.4 slightly exceeded the original format, with a ratio of 1.11, but remained below the 1.60 obtained when fixing the format to SVG.

In contrast, for zero-shot Qwen3-VL-4B and 8B, format selection increased the number of output tokens by 1.7–1.9 times. This is because they frequently fail by repeating the same content until they exhaust the output limit. After training the same 8B model on our benchmark data, the ratio dropped to 0.83, which is also lower than the 1.27 obtained when fixing the format to SVG.

From these results, we believe that format selection is an efficient option even in terms of the actual cost including failed generations, and that, combined with training, even small models can obtain its benefits.

Thank you again for your review. In our response, we explained why format selection is beneficial compared with a single general-purpose format, including its effects on quality and cost, and provided a cost analysis that includes failed generations. We’d appreciate any feedback on whether our response addresses your concerns. We’re happy to clarify any remaining points.

Reviewer_jMQyOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UniDiagCode, a six-format benchmark for image-to-code diagram generation comprising 341K real-world rendered-image/code pairs and 6.3K hand-drawn sketches. The authors fine-tune Qwen3 on the dataset, substantially improving its performance in both specified-format generation and model-driven format selection. The analysis in the paper identifies SVG as the most consistently robust general-purpose format across domains, while showing that input-dependent format selection can achieve a better trade-off between generation quality and cost.

Critical strengths
  • The dataset covers six diagram formats and includes both rendered diagrams and hand-drawn sketches, providing a comprehensive benchmark for evaluating LLMs’ diagram generation capabilities across formats.

  • Fine-tuning Qwen3 on the dataset substantially improves its diagram generation performance in both format-specification and format-selection settings, demonstrating the dataset’s utility for training multi-format models.

Critical weaknesses
  • The motivation for collecting a multi-format dataset should be better established. Strong proprietary models already perform well across formats and generally outperform the fine-tuned Qwen3-8B model. The paper would benefit from evidence of the practical advantages of multi-format training and format selection over a strong fixed-format baseline. Furthermore, the interpretation of code length as generation cost is insufficiently justified. Although output code length measures representation compactness, it does not capture input image tokens, reasoning tokens, or differences in inference speed. These factors may dominate overall latency and cost, making the practical savings from shorter code unclear.

  • The introduction claims that the jointly fine-tuned multi-format model matches models trained separately for individual formats, but I could not find a direct experimental comparison supporting this claim. A comparison with format-specific models should be included.

  • All evaluations rely on a LLM judge. Although the authors validate its agreement with human ratings, its relibility and potential biases remain a concern. Complementary quantitative measures, such as LPIPS or layout alignment, would help strengthen the evaluation, particularly for reconstructing rendered diagrams.

  • The main paper would benefit from representative qualitative comparisons across models and output formats. Such examples would help readers assess differences in visual fidelity, layout preservation, and characteristic failure modes.

  • The evaluated models, including Claude Opus 4.7 and GPT-5.4, are relatively outdated. Including more recent models would strengthen the experimental results.

Questions for authors

See Weaknesses.

6 comments

Thank you for your constructive review. We are glad that you found our benchmark comprehensive, covering six formats with both rendered images and hand-drawn sketches, and that fine-tuning on our data substantially improves performance. Below, we address each of your concerns. In our responses, we refer to the parts added to the revised version, which are highlighted in blue.

Motivation for a multi-format benchmark

We address this concern in two parts: the motivation for a multi-format benchmark, and the advantages of using multiple formats over a single general-purpose format.

Motivation for a multi-format benchmark. Multiple diagram formats are used in practice for different purposes. For example, Mermaid is often used to embed diagrams in text documents, TikZ in LaTeX papers, and draw.io when diagrams need to be edited in a graphical interface. People choose among these formats according to the type of diagram they want to create and how they intend to use it, and the formats differ in their trade-offs between expressive power and conciseness.

The ability to generate diagrams in the formats people actually use is therefore an important aspect of general-purpose diagram generation, regardless of whether the model chooses the format itself. Our Single setting evaluates this ability for each format, and the Multi setting additionally evaluates format selection.

Although several recent studies consider multiple output formats [1–3], they cover only a few diagram formats, focus primarily on charts, or rely on synthetic data. To our knowledge, our dataset is the largest fully real-world multi-format diagram corpus and the first to include hand-drawn sketches. Its scale and diversity support the systematic evaluation and training of models for general-purpose diagram generation. Training on our data substantially improves open models in the Single setting; for example, the IA of Qwen3-VL-8B with the original format specified rose from 1.9 to 3.4. This shows that our data helps open models acquire the ability to generate diagrams in multiple formats.

Advantages over a single general-purpose format. Our results also show that small open models struggle with general-purpose formats. As shown in Table 4, they frequently fail to render draw.io and TikZ code, unlike the proprietary models, and although they can render SVG reliably, its quality remains far below that of the proprietary models while requiring much longer code than the original formats. In contrast, using a more concise format suited to the target diagram helps these models; as shown in Figure 3, generating Graphviz and Mermaid diagrams in their own formats instead of SVG improves the diagram quality of Qwen3-VL-8B, and after training on our data, this advantage becomes larger for Graphviz and also appears for PlantUML.

Thus, while proprietary models can master general-purpose formats, small open models can achieve higher performance by making use of multiple formats. We believe that format selection, i.e., appropriately using the formats available in the real world, is especially important for models with limited capability.

  • [1] Zou et al. VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation. EMNLP 2024.
  • [2] Ni et al. VisCoder2: Building Multi-Language Visualization Coding Agents. ICLR 2026.
  • [3] Yang et al. OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward. ACL 2026 Findings.

Comparison with format-specific models

To support the claim in the introduction that a model trained jointly on multiple formats is comparable to models trained separately for each format, we have added a direct comparison to Appendix J of the revised version, as shown in Table 23. We compared a single model trained jointly on all six formats with six models trained separately for each format, all based on Qwen3-VL-8B, in the setting that specifies the original format.

The jointly trained model outperformed the per-format models in IA and DQ on the five formats other than Mermaid, with equal or lower RFR. On Mermaid, it was slightly below the per-format model, but the difference in IA was only 0.2. This shows that handling multiple formats in a single model does not impair its ability to generate each format. Therefore, the premise of format selection, namely that a single model chooses the format depending on the input, holds, and it also brings the practical advantage that there is no need to prepare and switch between separate models for each format.

Justification of code length as a proxy for generation cost

As the reviewer points out, code length does not include input image tokens, reasoning tokens, or inference speed. Reasoning tokens arise only when thinking is enabled, and our main experiments are conducted without thinking. We adopted this setting because it is standard in prior work on code-based diagram generation [4,5], and because enabling thinking does not consistently improve performance, as Table 13 in Appendix E shows, while it substantially increases the cost. Under this setting, we examined the shares of input and output in the cost and confirmed that output accounts for most of it, so code length is a valid proxy for cost.

For open models, output accounts for 66–95% of the tokens per request in the Original and SVG settings, including generations that failed to render, as shown in Table 19 in Appendix H of the revised version. In the Multi setting, the prompt becomes longer because it describes the formats, but the number of output tokens is still smaller than when fixing the format to SVG, a general-purpose format that can represent all diagrams, for all models except zero-shot Qwen3-VL-4B and 8B. Since input tokens are processed in parallel whereas output tokens are generated one by one, generation time depends mainly on the number of output tokens.

For proprietary models without thinking, output accounts for 75–92% of the API cost in the Original and SVG settings, as shown in Table 20 in Appendix H. Although the share of input increases in the Multi setting, the total cost is reduced by 22% for GPT-5.4 and 37% for Claude Opus 4.7 compared with SVG.

Therefore, format selection reduces the number of output tokens and thus the cost, both in generation time and in API pricing. In contrast, for Gemini 3.1 Pro, for which thinking cannot be disabled, reasoning tokens account for 65–90% of the cost, so the effect of differences in code length is small when thinking is enabled.

  • [4] Belouadi et al. TikZero: Zero-Shot Text-Guided Graphics Program Synthesis. ICCV 2025.
  • [5] Greisinger et al. TikZilla: Scaling Text-to-TikZ with High-Quality Data and Reinforcement Learning. ICLR 2026.

Complementary evaluation without VLM-as-a-judge

To complement the evaluation with VLM-as-a-judge, we added an evaluation with DreamSim [6], which measures the perceptual similarity between images; the results are shown in Appendix K of the revised version. As a result, the model ranking under DreamSim almost matched that under VLM-as-a-judge, indicating that the conclusions drawn from VLM-as-a-judge are not specific to the judge. Moreover, VLM-as-a-judge correlated more strongly with human ratings than DreamSim did, which further supports the validity of the evaluation with VLM-as-a-judge.

Comparing the rankings of the eight models by IA and by DreamSim in Table 24, Kendall's τb\tau_b per column was high, ranging from 0.86 to 1.00, so DreamSim yielded nearly the same ranking as VLM-as-a-judge. The improvement from training was also confirmed: for Qwen3-VL-8B in the Original setting, the distance decreased from 0.219 to 0.115.

Next, on the 300 human-rated samples, we compared the correlation with human ratings for VLM-as-a-judge and for DreamSim, as shown in Table 25.

VLM-as-a-judge correlated more strongly with human ratings than DreamSim, with Kendall's τb\tau_b of 0.765 versus 0.683 for IA and 0.724 versus 0.580 for DQ. The gap is larger for DQ, because DreamSim measures visual closeness to the reference and does not measure readability as a diagram. This indicates that VLM-as-a-judge is particularly effective for aspects that visual similarity cannot capture, such as readability.

Furthermore, we also evaluated the correctness of text labels and of elements and connections without using a judge, as shown in Appendix I of the revised version, and the model ranking there also largely agreed with that of the judge, with Kendall's τb\tau_b of 0.79–1.00 for text labels and 0.79–0.93 for elements and connections, except for 0.43 for element precision on Mermaid.

  • [6] Fu et al. DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data. NeurIPS 2023.

Qualitative comparisons across models and formats

Following your suggestion, we conducted qualitative comparisons across models and formats, and to systematically capture failure tendencies, we classified the errors in low-scoring outputs into error types. The definitions of the error types, the counts, and representative examples are shown in Appendix N.2 of the revised version, in Table 27 and Figure 26. We will move representative examples to the main text in the final version.

We analyzed 3,724 outputs from the seven models before training that were successfully rendered but received an IA of 3 or lower in the setting that specifies the original format. The classification was performed by an LLM, Claude Sonnet 5.5, given the reference image and the generated image, and we manually checked several dozen cases for each error type. A single output may fall into multiple error types.

The most frequent errors were local errors, in which an element is present but part of it is wrong, such as the direction of an arrow, the target of a connection, or a text string, and differences in the layout of elements. They appeared as major errors in 68.1% and 61.1% of all analyzed outputs, respectively, and in more than half of the outputs of both open and proprietary models. In contrast, missing and extra elements appeared in 48.9% and 23.4% of the analyzed outputs of open models, respectively, compared with 22.4% and 9.1% for proprietary models. In addition, failures in which the entire diagram collapses occurred almost exclusively in open models, accounting for 213 of the 216 cases. This indicates that, whereas the errors of proprietary models are mostly limited to details and layout, open models often get the composition of the diagram itself wrong. This tendency is also consistent with the result shown in Appendix I of the revised version that open models tend to omit necessary labels.

Evaluation of more recent models

We additionally evaluated two of the latest models, GPT 6 Sol as a proprietary model and Qwen3.8-27B as an open model, and added the results to Appendix L of the revised version, as shown in Table 26. For GPT 6 Sol, we evaluated only the setting that specifies the original format and the format selection setting. Both models performed better than previous-generation models of comparable scale, but the claims of our paper remained unchanged. That is, for sufficiently capable models, format selection shortens the code while maintaining quality, and the gap between open and proprietary models remains large.

GPT 6 Sol improved over GPT-5.4 when the original format is specified, raising IA from 3.7 to 4.3, and format selection improved it further to 4.5 with shorter code. Thus, the benefit of format selection persists as models become more capable. In contrast, Qwen3.8-27B reached an IA of only 2.8, far below the proprietary models and even below the 3.4 of the 8B model trained on our data, showing that our benchmark remains a useful foundation for improving open models.

Reviewer_DohTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniDiagCode introduces a six-format image-to-code diagram benchmark with rendered images and human-drawn sketches. It compares specified output formats with model-driven format selection and fine-tunes a multi-format model. The study examines reconstruction quality, renderability, and code length across formats and model families.

Critical strengths
  1. The real-world corpus and paired rendered/sketch evaluation provide a useful resource for comparing formats that are usually studied separately.

  2. Broad model comparisons reveal practical quality–length trade-offs. Blinded human evaluation supports the visual judge, and fine-tuning demonstrates the utility of the collected data.

Critical weaknesses
  1. Format selection needs stronger validation. Source-format supervision does not establish the best target format for each input. Gains vary across models, and comparisons with simple routing and per-input alternatives are needed to support the quality–cost claims.

  2. Structural correctness and editability need direct evaluation. Human validation supports the visual scores, but connectivity, label accuracy, and practical editing success remain indirectly assessed.

Questions for authors
  1. Could you compare selection with simple routing and the best available format per input, including failed generations in the cost analysis?

  2. Could you evaluate connectivity, label accuracy, and basic editing success on a small subset?

4 comments

Thank you for your constructive review. We are glad that you found our real-world corpus and the paired evaluation with rendered images and hand-drawn sketches a useful resource for comparing formats. Below, we address each of your concerns. In our responses, we refer to the parts added to the revised version, which are highlighted in blue.

Source-format supervision vs. the best format per input

As the reviewer points out, training only with the original format as supervision may not fully reflect the optimal format for each input. Following the reviewer's suggestion, we therefore considered how to define the best format for each input and conducted an additional analysis based on it. Specifically, we generated once in every format for each sample and regarded the format with the highest average of IA and DQ as the best format. Since this selection is made after observing the evaluation results, it can be viewed as a kind of oracle setting. As a result, as shown in Table 17 in Appendix F of the revised version, selecting the best format improved performance over the Multi setting, in which the model chooses the format by itself. This indicates that there is further room for improvement if the model can make better format choices.

Incorporating an oracle-like setting into training could improve performance, but it would require, at the stage of constructing the training data, generating, rendering, and evaluating each of the more than 300,000 samples in our training set in all six formats. This amounts to about 1.9 million additional generations and evaluations. Moreover, this is the minimum case of a single generation per format; increasing the number of generations per format to obtain better supervision would further increase the cost in proportion to that number. In contrast, our approach of using the original format as the target requires no inference at all to construct the supervision, so it can be carried out at very low cost while still improving performance.

We believe our approach is meaningful as a practical method that realizes effective format selection simply and at very low cost. As a way to further improve performance, we consider it a promising direction for future work to actually run inference on many samples, as in the oracle setting, and reflect the results in training.

Cost analysis including failed generations

Following your suggestion, we added a cost analysis that also includes generations that failed to render. Specifically, we recounted the output tokens of all requests, including failed generations, and compared them as ratios to the setting that specifies the original format; the full results are shown in Table 18 in Appendix G of the revised version.

For models with sufficient code-generation capability, format selection reduced the cost even when failures are included, and the ratios were almost the same as the code-length ratios reported in the paper, which exclude failures. For example, the ratio is 0.59 for Gemini 3.1 Pro and 0.66 for Claude Opus 4.7, and these models also largely maintain their quality. The newly evaluated open model Qwen3.8-27B also reduced the cost, with a ratio of 0.37. GPT-5.4 slightly exceeded the original format, but its cost was still lower than when fixing the format to SVG.

For zero-shot Qwen3-VL-4B and 8B, format selection increased the cost by 1.7–1.9 times. This is because they frequently fail by repeating the same content until they exhaust the output limit. However, after training the same 8B model on our benchmark data, the ratio dropped to 0.83, which is also lower than when fixing the format to SVG.

This shows that the cost benefit of format selection presupposes that the model can generate each format reasonably well, and that this prerequisite can be met through training. In other words, we believe that, combined with training, format selection is an effective means of reducing cost even for small models.

Direct evaluation of label accuracy and connectivity

Following your suggestion, we added evaluations of text label accuracy and of structural accuracy, namely elements and connections, that directly compare the generated outputs with the references. The results show that proprietary models reproduce text labels more accurately overall, and that in open models, missing required labels is more common than writing incorrect or extra labels. We also found that, for every model, correctly reproducing the connections between elements is more difficult than reproducing the elements themselves. Note that, since it is widely accepted that code-based representations are easier to edit than raster images [1,2], our work builds on this premise, and we leave the success of editing itself outside the scope of our evaluation.

Text label accuracy. We read the text in the reference and generated images with the OCR model PaddleOCR-VL-1.5 [3], normalized differences in notation such as letter case and symbols, and matched texts one-to-one only when they matched exactly. We then computed precision as the proportion of texts in the generated image that matched the reference, and recall as the proportion of texts in the reference image that were reproduced in the generated image. The full results over the samples for which both rendering and OCR succeeded are shown in Table 21 in Appendix I of the revised version.

Overall, both precision and recall were higher for proprietary models than for open models. For example, when the original format is specified, the recall of proprietary models is 0.86–0.90, whereas that of open models is 0.57–0.81. In addition, for the open models and the trained model, precision was significantly higher than recall in 20 of the 22 settings. High precision means that most of the generated texts match the reference, while lower recall means that some of the reference texts are not generated. In other words, in these models, missing required labels is more common than writing incorrect or extra labels.

Element and connection accuracy. For Graphviz and Mermaid, elements and connections can be extracted directly by parsing the code. We therefore extracted elements and connections from the reference and generated code, matched elements one-to-one only when their labels matched exactly, and evaluated whether each connection links the matched elements in the correct direction. The full results over the samples for which both parsing and rendering succeeded are shown in Table 22 in Appendix I of the revised version.

For all models and formats, the recall of connections was significantly lower than that of elements. While the recall of elements is high, at 0.84–0.99, that of connections is 0.68–0.97; the models can mostly generate the elements required for the diagram, but in many cases fail to connect them correctly.

These results indicate that, to improve open models, it is important to reduce missing labels and to correctly reproduce the connections between elements.

  • [1] Belouadi et al. AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ. ICLR 2024.
  • [2] Belouadi et al. DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ. NeurIPS 2024.
  • [3] Cui et al. PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing. arXiv:2601.21957.

Thank you again for your review. In our response, we provided an analysis of how well models select formats compared with the oracle selection, a cost analysis that includes failed generations, and a more detailed evaluation of label accuracy and diagram structure. We’d appreciate any feedback on whether our response addresses your concerns. We’re happy to clarify any remaining points.

We would be grateful to know whether our responses have addressed your concerns, and we are happy to address any remaining ones.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback.

Strengths noted by the reviewers.

  • A comprehensive benchmark built on a large-scale real-world multi-format dataset, evaluated with both rendered images and hand-drawn sketches.
  • The substantial performance gains from fine-tuning on our data.
  • The perspective of treating the output format as a model decision rather than a fixed constraint.

Weaknesses raised by the reviewers and our responses.

  • Necessity of format selection over a general-purpose format: we explained its benefits in terms of both quality and cost.
  • Whether models select the optimal format: we compared with selecting the best format for each input, showing that better selection could further improve quality, Appendix F.
  • Whether code length is a valid proxy for cost: we reported the tokens including failed generations and the cost breakdown, confirming that output dominates the cost and that format selection reduces cost even when failures are included, Appendices G and H.
  • Reliance on VLM-as-a-judge: we added an evaluation with an image-similarity metric other than VLMs and confirmed that our main conclusions remain unchanged, Appendix K.

We also added Appendices I, J, L, and N.2. All additions are highlighted in blue.

Clarification. Based on this discussion, we would like to further clarify our contributions.

  1. Format selection also improves quality. Strong proprietary models perform well in any format, but open and smaller models vary greatly in performance across formats. For our fine-tuned 8B model, the format-selection setting achieved higher quality than fixing the format to any general-purpose format, and our oracle analysis indicates substantial further room for improvement.
  2. The dataset itself is a contribution. Our large-scale real-world multi-format data substantially improves code generation not only for format selection but also in each individual format. Even in the setting that specifies the original format, fine-tuning raised the IA of our 8B model from 1.9 to 3.4, outperforming larger open models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.