- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces UniDiagCode, a benchmark for image-to-code diagram generation covering six output formats, with approximately 341K rendered image–code pairs and 6.3K hand-drawn sketches. It compares generation in a specified format with model-driven format selection, evaluating reconstruction fidelity, visual quality, rendering failures, and code length. Experiments across commercial and open-weight models show that sufficiently capable models can reduce code length through format selection while maintaining comparable quality.
Critical strengths
- The paper takes an interesting perspective by treating the output format as a model decision rather than a fixed constraint in image-to-code diagram generation.
- The paper provides extensive technical details, making the experimental setup easier to understand and reproduce.
Critical weaknesses
-
The paper does not yet make a strong enough case for adaptive format selection over a single general-purpose format. SVG and TikZ can already represent the diagrams considered in this benchmark. The experiments show that allowing the model to choose a format can reduce code length, but it remains unclear whether these savings are substantial enough in practice to justify treating format selection as a separate research problem. How much time or money do these reductions actually save per diagram? Without a clearer account of their practical impact, it is difficult to judge whether the benefits warrant the added complexity and further research effort.
-
The cost evaluation does not fully establish the claimed efficiency gains. Code length is measured only for successful outputs, leaving out the tokens spent on failed attempts and any retries or repairs needed to obtain a usable result. Some models also produce shorter code at the expense of reconstruction quality or rendering reliability. Shorter successful outputs therefore do not necessarily mean lower overall cost.
Questions for authors
-
Could the authors explain why the code-length savings justify devoting substantial research and engineering resources to choosing among diagram formats? Given that general-purpose formats can already represent these diagrams, what makes the benefits of format selection substantial enough to warrant this additional effort?
-
Are code-length comparisons computed on the same subset of diagrams that succeed under both settings, or on each setting’s successful outputs separately? Based on the available results, does the efficiency advantage remain when failed-generation tokens are included and reconstruction quality is comparable?
5 comments
Cost analysis including failed generations
The code length in the paper is not restricted to the same diagrams that succeeded in both settings; it is computed separately over the outputs that rendered successfully in each setting. Note that even when we recomputed it only on the same diagrams that succeeded in both settings, the ratio to the original format hardly changed, as also described in Appendix G of the revised version.
On the other hand, as the reviewer points out, this computation does not include the tokens spent on failed generations. We therefore recounted the output tokens of all requests, including failed generations, and compared them as ratios to the setting that specifies the original format; the full results are shown in Table 18 in Appendix G of the revised version. As a result, for models with sufficient code-generation capability, the efficiency advantage of format selection was maintained even when failures are included. We also found that models without sufficient capability can acquire it through training.
For models with sufficient code-generation capability, format selection reduced the number of output tokens even when failures are included, and the ratios were almost the same as the code-length ratios reported in the paper. For example, the ratio is 0.59 for Gemini 3.1 Pro and 0.66 for Claude Opus 4.7. The newly evaluated open model Qwen3.8-27B also reduced the number of output tokens, with a ratio of 0.37, mainly because its failed generations in the original format were much longer than those in the format-selection setting. These models also largely maintain their quality. GPT-5.4 slightly exceeded the original format, with a ratio of 1.11, but remained below the 1.60 obtained when fixing the format to SVG.
In contrast, for zero-shot Qwen3-VL-4B and 8B, format selection increased the number of output tokens by 1.7–1.9 times. This is because they frequently fail by repeating the same content until they exhaust the output limit. After training the same 8B model on our benchmark data, the ratio dropped to 0.83, which is also lower than the 1.27 obtained when fixing the format to SVG.
From these results, we believe that format selection is an efficient option even in terms of the actual cost including failed generations, and that, combined with training, even small models can obtain its benefits.
Thank you again for your review. In our response, we explained why format selection is beneficial compared with a single general-purpose format, including its effects on quality and cost, and provided a cost analysis that includes failed generations. We’d appreciate any feedback on whether our response addresses your concerns. We’re happy to clarify any remaining points.
Thank you for your constructive review. We are glad that you found our perspective of treating the output format as a model decision rather than a fixed constraint interesting, and that our technical details make the setup easier to reproduce. Below, we address each of your concerns. In our responses, we refer to the parts added to the revised version, which are highlighted in blue.
Format selection vs. a single general-purpose format
We agree that SVG and TikZ can, in principle, represent the diagrams in our benchmark. However, representational coverage alone does not determine whether a format is equally suitable for model-based generation. The level of detail the LLM must specify for diagram generation varies across formats. For example, SVG requires the model to specify the geometry and placement of individual elements, often resulting in longer, lower-level code. By contrast, formats such as Graphviz, Mermaid, and PlantUML allow the model to specify diagram elements and their relationships while the rendering tool handles much of the layout, resulting in more concise, higher-level code.
This distinction matters because the formats place different demands on the model, which can affect both generation cost and output quality. In our experiments, our 8B model fine-tuned on our benchmark performs substantially better in these higher-level formats. As shown in Figure 3, generating Graphviz, Mermaid, and PlantUML diagrams in their own formats yields an IA 0.3–1.2 points higher than generating them in SVG. As shown in Table 4, even in the format-selection setting, its DQ is 3.7, higher than the 3.2 obtained when fixing the format to SVG. Enabling small models to generate high-quality diagrams at low cost has great practical value.
Format selection also reduces cost. For our fine-tuned 8B model, the number of output tokens per diagram, including failed generations, decreased from 2,890 when fixing the format to SVG to 1,893 with format selection, about one third fewer, as shown in Table 19 in Appendix H. Because output tokens are generated one by one, this reduction shortens generation time on the same hardware. For proprietary models, format selection reduced the API cost per 1,000 diagrams from \12.91to10.08 for GPT-5.4 and from \27.13to17.06 for Claude Opus 4.7, that is, by 22% and 37%, as shown in Table 20 in Appendix H. This corresponds to savings of about \2,800and10,000 per million diagrams. For large-scale conversion, such as document digitization or training data creation, these savings are not negligible.
Our results suggest that format selection is not merely a way to reduce code length: for smaller models, choosing a format suited to their capabilities can also improve diagram quality. We therefore study whether LLMs can adapt their output representation to the target diagram and their own capabilities, and evaluate how this choice affects generation quality, rendering failures, and cost. This question is related to prior work on selecting representation formats or programming languages according to the task [1,2,3]. Our contribution is to examine this issue for diagram generation and to provide a benchmark for measuring these effects.