- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies generating executable code to reconstruct an input image. It addresses limited benchmark coverage, dependence on reference code in some benchmarks, and evaluation criteria that are insufficiently tailored to different domains. Vision2Code contains 539 test examples from 15 datasets across six domains, requires no original rendering code, and evaluates reconstructions with dataset-specific rubrics. Experiments with nine model configurations show stronger performance on charts and graphs than on tasks such as spatial reconstruction. The authors also validate agreement with human judgments and improve two smaller models through evaluator-guided self-training.
Critical strengths
-
The unified procedure makes cross-domain evaluation convenient. Different image types are evaluated through code execution and image reconstruction, without requiring the original rendering code.
-
The evaluator is supported by substantial validation. The authors compare against four human annotators and rescore every output with two VLM judges. The dataset-specific rubrics align better with human judgments than generic rubrics, and both judges produce the same overall ranking of the nine configurations.
Critical weaknesses
-
The benchmark's added research value is limited. Table 1 emphasizes multi-domain coverage, but existing single-domain benchmarks were designed for specific tasks; not covering other domains is not itself a flaw in their evaluation. The table also shows that benchmarks such as ChartMimic already use domain-specific evaluation, while Image2Struct supports multiple domains without reference code. Recasting images from different sources as reconstruction tasks and refining their scoring rules mainly broadens coverage and makes evaluation more convenient. The paper does not sufficiently explain what important new insights this integration provides beyond using existing domain benchmarks together. I therefore see the contribution primarily as an extension of existing evaluation, with a limited research advance.
-
The need for second-stage filtering is not sufficiently established. compares the source with the first reconstruction, whereas compares the first and second reconstructions: the two stages reconstruct different inputs. Thus, does not itself establish that these examples are more useful for training than other high-scoring first reconstructions. With training-set size held fixed in Table 5, adding this condition raises Qwen3.5-9B from 1.8567 to 1.8645, offering almost no additional gain. Phi-4-MM improves more, from 0.6299 to 0.7414, but only one training seed is reported. The evidence supports filtered self-training, but does not yet establish a reliable advantage from the additional round of generation and scoring.
Questions for authors
-
Phi-4-MM improves from 0.5756 to 0.7414 after six epochs on just 110 training examples with an effective batch size of 4. How do the authors explain this small-data gain? Does it mainly reflect fewer execution failures or better reconstruction quality among successful renders?
-
Only 12 development examples are used to select the Phi-4-MM checkpoint. Have the authors checked whether using a different development sample changes the selected model or its test performance substantially?
-
In Table 12, filtering only by a high first-stage score yields 3,220 Qwen examples and 663 Phi-4 examples, but these baselines are also reduced to the size of the second-stage subset. Have the authors evaluated training directly on these larger pools of high-scoring first reconstructions? This matters for whether second-stage filtering offers a practical advantage over simpler data construction.