acceptodds
Withdrawn from ICLR 2027

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image-to-code generation tests whether a vision-language model (VLM) can write executable code to reproduce an image. Existing benchmarks suffer in several ways, including narrow scope, dependence on source code, and generic evaluation criteria across very different domains. To address this, we introduce Vision2Code, a multi-domain and reference-code-free benchmark with 539 test examples from 15 datasets spanning six domains: charts and plots, geometry, graphs, scientific imagery, documents, and 3D spatial scenes. We also create a dataset-specific rubric that judges image reconstruction quality with emphasis on criteria relevant to that dataset. Taken together, our evaluation framework aligns better with human judgments from four annotators than generic VLM rubrics and pixel, embedding, and perceptual baselines. We study the robustness of our evaluation framework by computing agreement between two judges, evaluator repeatability, and the sensitivity to rubric weights and domain-specific errors. Results show that both VLM judges have low variance across repeated runs and preserve the same ranking of all nine evaluated models. Introduced perturbations are also correctly detected by both judges using our rubric. We show that benchmark performance varies substantially across domains. Models do well on charts and graphs but struggle with spatial scenes, chemistry, documents, and scientific diagrams. To improve image-to-code generation capability, we introduce evaluator-guided self-training, where evaluator scores are used to select model-generated programs for fine-tuning. This improves Qwen3.5-9B from 1.60 to 1.86 and Phi-4-MM from 0.58 to 0.74 on test, with gains also holding under additional evaluators. Vision2Code provides a multi-domain testbed for measuring, diagnosing, and improving executable visual reconstruction.

Primary area:
datasets and benchmarks
Keywords:
image-to-code generation, multimodal benchmarks, vision-language models, fine-grained evaluation, reference-code-free evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PCQQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies generating executable code to reconstruct an input image. It addresses limited benchmark coverage, dependence on reference code in some benchmarks, and evaluation criteria that are insufficiently tailored to different domains. Vision2Code contains 539 test examples from 15 datasets across six domains, requires no original rendering code, and evaluates reconstructions with dataset-specific rubrics. Experiments with nine model configurations show stronger performance on charts and graphs than on tasks such as spatial reconstruction. The authors also validate agreement with human judgments and improve two smaller models through evaluator-guided self-training.

Critical strengths
  1. The unified procedure makes cross-domain evaluation convenient. Different image types are evaluated through code execution and image reconstruction, without requiring the original rendering code.

  2. The evaluator is supported by substantial validation. The authors compare against four human annotators and rescore every output with two VLM judges. The dataset-specific rubrics align better with human judgments than generic rubrics, and both judges produce the same overall ranking of the nine configurations.

Critical weaknesses
  1. The benchmark's added research value is limited. Table 1 emphasizes multi-domain coverage, but existing single-domain benchmarks were designed for specific tasks; not covering other domains is not itself a flaw in their evaluation. The table also shows that benchmarks such as ChartMimic already use domain-specific evaluation, while Image2Struct supports multiple domains without reference code. Recasting images from different sources as reconstruction tasks and refining their scoring rules mainly broadens coverage and makes evaluation more convenient. The paper does not sufficiently explain what important new insights this integration provides beyond using existing domain benchmarks together. I therefore see the contribution primarily as an extension of existing evaluation, with a limited research advance.

  2. The need for second-stage filtering is not sufficiently established. R1R_1 compares the source with the first reconstruction, whereas R2R_2 compares the first and second reconstructions: the two stages reconstruct different inputs. Thus, R2<R1R_2<R_1 does not itself establish that these examples are more useful for training than other high-scoring first reconstructions. With training-set size held fixed in Table 5, adding this condition raises Qwen3.5-9B from 1.8567 to 1.8645, offering almost no additional gain. Phi-4-MM improves more, from 0.6299 to 0.7414, but only one training seed is reported. The evidence supports filtered self-training, but does not yet establish a reliable advantage from the additional round of generation and scoring.

Questions for authors
  1. Phi-4-MM improves from 0.5756 to 0.7414 after six epochs on just 110 training examples with an effective batch size of 4. How do the authors explain this small-data gain? Does it mainly reflect fewer execution failures or better reconstruction quality among successful renders?

  2. Only 12 development examples are used to select the Phi-4-MM checkpoint. Have the authors checked whether using a different development sample changes the selected model or its test performance substantially?

  3. In Table 12, filtering only by a high first-stage score yields 3,220 Qwen examples and 663 Phi-4 examples, but these baselines are also reduced to the size of the second-stage subset. Have the authors evaluated training directly on these larger pools of high-scoring first reconstructions? This matters for whether second-stage filtering offers a practical advantage over simpler data construction.

Reviewer_J7FYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating VLMs' ability to generate code that reproduces an input reference image. The authors collected 539 examples across six domains and designed rubrics for evaluating these outputs automatically. They show that the evaluation scores align well with human judgment. Additionally, this paper presents a self-training approach by training on the model's own output, with the poorly scored samples rejected, and shows that it improves a Qwen VLM model on the benchmark.

Critical strengths

This paper benchmarks the image-to-code task, which is interesting and relates to the widely used "text-to-code-to-image" capability that many use for generating figures and plots. Although many works address this task, as noted by the authors, this paper introduces multiple domains in a unified benchmark and evaluates results at the image level without using reference code, which makes it more scalable.

Critical weaknesses

The benchmark proposed in this paper prompts the models to use Matplotlib and NumPy to reproduce the input image. However, there are many potential harnesses and tools that can drastically change the performance of the models in these tasks. For example, commercial models like GPT 5.4 with a coding harness can attempt to compile their initial attempts and iteratively debug to suppress runtime errors. There are also better tools like WebGL for generating some of the 3d scenes ("spatial domain") or documents present in this benchmark. Therefore, I believe the current evaluation settings cannot comprehensively assess the image-to-code ability of these VLMs.

Questions for authors

Table 3 (a) -- what is the difference between Generic (Qwen/Kiwi) and Ours (Qwen/Kiwi)?

Have the authors measured the cost difference of running the models on the benchmark? For example, in terms of dollars or tokens.

Reviewer_KgmEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Vision2Code introduces a reference-code-free image-to-code benchmark containing 539 test examples from 15 existing datasets across six visual domains. Models generate Python code using Matplotlib and NumPy, and the rendered outputs are evaluated using dataset-specific VLM rubrics.

The paper compares the evaluator with standard image metrics and generic VLM rubrics, validates agreement with four human annotators on 180 examples, and studies judge consistency and sensitivity to controlled errors. It also proposes a self-training filter that selects generated image–code pairs when the first reconstruction scores highly but a subsequent reconstruction receives a lower score.

The benchmark and evaluation protocol could be useful. However, the experiments do not sufficiently establish the significance of the contribution beyond dataset aggregation and rubric design, or a reliable advantage of the proposed self-training mechanism over simpler alternatives

Critical strengths
  • A useful evaluation setting with meaningful validation effort. Evaluating reconstruction without requiring reference source code allows broader coverage than code-paired datasets alone. The paper also recognizes that successful execution is not equivalent to faithful reconstruction. Human ratings, generic-rubric baselines, standard image metrics, repeated judgments, and controlled perturbations provide useful evidence about the evaluator. These experiments make the work more informative than a leaderboard alone.

  • Reasonably controlled comparisons within the proposed self-training framework. The filtering variants use matched training-set sizes, and improvements over the base models are checked with additional evaluators. These controls reduce straightforward explanations based on training-set size or preferences of a single judge. The appendix provides sufficient detail to understand the training procedure.

Critical weaknesses
  • The benchmark’s broader scientific significance and evaluator advantage are not sufficiently established.

The paper presents a reasonable evaluation question, but the evidence supports a narrower conclusion than a broadly superior measure of image-to-code capability. The human study contains 180 examples, with only 12 per dataset. Four ratings per example improve reliability, but do not increase the number of independent examples. The reported aggregate correlations do not establish uncertainty in the improvement over generic rubrics or reliability within individual domains. For example, Qwen’s Spearman correlation increases from 0.817 to 0.826, while Kimi’s increases from 0.820 to 0.861; the significance and consistency of these differences need to be assessed directly.

The comparisons with SSIM, PSNR, CLIP, DreamSim, embeddings, and generic VLM rubrics are useful, but do not establish superiority to applicable domain-specific evaluation methods or objective structural checks. The controlled-error experiment shows that the evaluator detects many introduced errors. Applying the competing evaluators to the same perturbations would establish whether the proposed rubric adds sensitivity beyond those alternatives.

Furthermore, requiring Matplotlib/NumPy across all domains confounds the interpretation of the cross-domain results. Charts are naturally supported by this environment, whereas documents and spatial scenes impose different representational demands. The experiments do not adequately separate visual-understanding limitations from difficulty expressing an image in the prescribed environment. The observed domain ranking therefore needs a narrower interpretation.

  • The specific self-training contribution and its underlying mechanism are weakly established.

The paper demonstrates that fine-tuning on selected model-generated pairs can improve performance. However, its comparisons primarily concern variants of the authors’ filtering procedure, rather than establishing an advantage over strong existing self-training or rejection-sampling fine-tuning approaches under matched resource budgets.

Even within the reported comparisons, the specific selection rule is not convincingly justified. For Qwen, the proposed filter scores 1.8645, compared with 1.8567 for the low-first-rating filter, a difference of 0.0078 on the 0–5 scale. The appendix reports one training seed. This result does not establish a reliable advantage for the proposed high-quality/self-reconstruction-drop criterion.

The interpretation of that criterion also needs validation. R1 compares the original image with the first reconstruction, while R2 compares the first reconstruction with the second. Because these are different image pairs, R2 < R1 does not isolate reconstruction instability. It may reflect generation randomness, evaluator noise, or differences in the difficulty of the two reconstruction tasks.

Section 5.3 provides useful descriptions of execution and reconstruction failures, but does not establish their root causes or explain which failures self-training corrects. In particular, the paper does not trace which semantic errors are preserved, introduced, or reinforced through the reconstruction stages and subsequent training. This matters because the training target is the model-generated image–code pair: a high aggregate first-stage score may still admit structural mistakes.

Taken together, these results establish a potentially useful heuristic, but not a sufficiently clear methodological advance or explanation of when and why it works.

Questions for authors

Clarifications that could materially affect my assessment:

  • Were the rubric and guardrails finalized before inspecting the human-validation examples and ratings? What evidence establishes that their advantage over generic rubrics generalizes beyond the examples used for validation?
  • Can you report paired uncertainty estimates for the correlation improvements over generic rubrics, together with domain-level results? Were the baseline evaluators tested on the same controlled perturbations?
  • What evidence separates domain-specific visual limitations from the constraints of Matplotlib/NumPy? Do the alternative-language examples support any quantitative conclusion about this confound?
  • Is the proposed self-training filter consistently better across training seeds and domains? In particular, how should the near-equality between the proposed and low-first-rating filters for Qwen be interpreted?
  • What evidence supports interpreting R2 < R1 as a useful learning signal rather than generation variability, evaluator noise, or a difference in reconstruction difficulty? Are improvements associated with correction of identifiable semantic errors, better execution, or changes in visual appearance?

Additional experiments, rather than clarification alone:

A matched-budget comparison with an established self-training or rejection-sampling fine-tuning baseline would directly address the claimed methodological advantage. A stage-by-stage analysis of structural errors would help explain the selection mechanism and its failure modes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.