acceptodds
Withdrawn from ICLR 2027

MetroBench: Execution-Grounded Measurement for Visual Code Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual code generation couples interpretation of a reference artifact with program synthesis, yet artifact fidelity and calibrated capability are distinct measurement problems. We introduce MetroBench, an execution-grounded framework and diagnostic study toward their joint evaluation. The framework specifies Construct, Reflect, and Repair protocols, and separates a four-component chart scorer from an independently audited psychometric layer. On 200 procedural chart items and 1,000 program variants, mean fidelity declines from 1.000 to 0.658 across five perturbation levels; 197 item profiles decrease strictly, while three exhibit local reversals. Removing plotted-data agreement reduces the strictly ordered profiles to 145, exposing sensitivity to the scoring rubric. In a separate synthetic-response experiment with 20 repetitions, 40 Fisher-selected items yield held-out ability RMSE 0.104 versus 0.110 for random selection and 0.090 for the 200-item full test; the paired 40-item gain is uncertain. A matched-bank study shows that anchor linkage can recover scale despite nearly unchanged rank correlation. However, nominal 95% plug-in ability intervals cover only 45.8% of held-out abilities, compared with 95.8% when item parameters are known. These findings demonstrate why execution grounding, adaptive efficiency, and uncertainty calibration require separate evidence. The present studies evaluate procedural scoring and synthetic measurement, not model capabilities; a validated common scale across multimodal perception and code synthesis remains an open empirical requirement.

Primary area:
datasets and benchmarks
Keywords:
multimodal evaluation, code synthesis, execution-based evaluation, visual grounding, item response theory, adaptive testing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5P6HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MetroBench, a framework for systematically evaluating the execution quality and model capabilities in visual code generation, utilizing both execution-based scoring and psychometric experiments. The results indicate that composite scores are highly dependent on the underlying rules, that adaptive item selection offers only limited advantages, and that capability rankings do not necessarily guarantee calibrated measurement scales.

Critical strengths
  1. By distinguishing between quality scoring, ability ranking, and scale calibration—all from the perspective of measurement validity—this work reveals that these three approaches can lead to divergent conclusions. It encourages a greater focus on the credibility of scores in the evaluation of visual code generation.
  2. The experiments cover ablation studies, score reversals, budget variations, and issues related to linkage and interval coverage. The evidence robustly supports the conclusions, providing a detailed look at the risks associated with ranking and calibration in measurement.
Critical weaknesses
  1. The core framework has not yet been validated end-to-end in real-world visual code generation tasks; it lacks actual model responses, and execution-based scores have not been incorporated into ability modeling. Concepts such as "Construct," "Reflect," and "Repair" remain at the level of protocol design, and it has yet to be demonstrated whether this overall approach actually improves practical evaluation.
  2. While existing research already addresses quality comparison, adaptive testing, and anchor-based scale linkage, this paper primarily serves to synthesize these concepts. It does not propose new statistical principles, nor does it empirically verify the combined benefits of these approaches. Some findings align with established expectations in measurement theory; thus, the contribution leans more toward diagnostic insight than the introduction of a novel methodology.
  3. Certain experiments lack sufficient numerical reliability and mechanistic explanation. The calibration scores exhibit significant marginal density errors, yet there is no verification of whether the conclusions remain stable after correction. Furthermore, the "linkage" process simultaneously adjusts anchor parameters and calibration priors, making it difficult to isolate the independent contribution of each factor.
Questions for authors
  1. Could the authors provide a preliminary validation based on real-world model outputs—for instance, by collecting outputs from existing chart-generation models and comparing the composite scores against independent human quality judgments? Additionally, if these scores were used to fit an ability measurement model, would the calibration issues identified in the paper’s synthetic experiments still manifest?
Reviewer_fkaXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

MetroBench proposes an execution grounded framework for visual code generation plus a diagnostic study of its measurement properties. A four component fidelity score over pixels, mask IoU, plotted data and text recall is run on 200 procedurally generated charts with five cumulative perturbation levels, where mean fidelity falls from 1.000 to 0.658 and 197 of 200 profiles decrease strictly. A separate and deliberately disconnected synthetic study audits IRT calibration, adaptive item selection, version linkage and interval coverage, finding that nominal 95 percent plug in intervals cover only 45.8 percent.

Critical strengths
  1. The design discipline in the measurement study is nice. Generating one response matrix per repetition before administration, having Fisher and random read from that same matrix, and checking that every 200 item sequence reproduces the full test estimate to 1e-10 is the control adaptive evaluation work usually skips. The 40 item gain is then reported honestly as not significant.

  2. Two of the negative findings are worth knowing. Free and linked calibration both hit Spearman 0.9989 against true ability while their scale slopes are 1.706 and 1.004, which makes vivid the point that rank correlation cannot detect scale distortion. The coverage result, 45.8 percent against 95.8 percent with known parameters, is a sharp reminder that accurate point estimates say nothing about interval validity.

Critical weaknesses
  1. There is no benchmark here and no model is evaluated. The chart candidates are the authors' own mutations of their own templates, and Appendix A.4 admits some operators delete precisely the content the heuristic channels count, so the monotonicity result is close to circular. The synthetic respondents come from the same model that is then fitted, making the IRT study an implementation check. The two halves are explicitly independent, so the paper's own thesis, that execution grounding and measurement need to be connected and audited together, is argued but never instantiated.

  2. Once the scope is accounted for, what is left is a careful redemonstration of known statistics, and the paper says as much. It claims no new IRT family and no new selection principle, the scorer is a weighted average of standard similarities with weights never checked against human judgment, and plug in undercoverage follows from ignoring estimation error. The response model is acknowledged to be invalid on bounded support, calibration is factorized rather than joint, and the optimizer does not persist convergence status.

Questions for authors
  1. What actually prevents running three or four real models on the 200 Construct items and fitting the measurement layer to those scores? That one step would connect the halves and would change my score.

  2. Any human judgments of perceptual fidelity on a stratified sample? Without them the composite cannot be said to track quality at all.

  3. Do authors try a beta or logit normal response model, and how much undercoverage survives once the support is right and item uncertainty is propagated?

Reviewer_wF4KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents MetroBench, a framework for evaluating human sensitivity to errors in chart reconstruction. It constructs 1,000 controlled perturbations across five chart types and uses Item Response Theory (IRT) to model item difficulty and evaluator ability. The framework further employs Fisher information-based adaptive item selection to reduce evaluation costs. Experiments with simulated respondents examine parameter recovery, estimation accuracy, and cross-version calibration.

Critical strengths
  1. The paper addresses a relevant problem in chart reconstruction evaluation, particularly the need for systematic assessment of error sensitivity.
  2. The controlled perturbations across multiple chart types provide a structured foundation for evaluating different types and levels of reconstruction errors.
  3. The experiments examine several aspects of measurement reliability, including parameter recovery, adaptive selection, and cross-version calibration.
Critical weaknesses
  1. Limited methodological novelty. The main framework relies on established psychometric techniques, including IRT, Fisher information-based adaptive testing, and scale linking. Closely related approaches have already been explored in visualization literacy assessment, particularly Adaptive Assessment of Visualization Literacy (TVCG 2023). The methodological advances beyond these prior approaches are not sufficiently clear.

  2. Insufficient validation with real evaluators. The core experiments rely primarily on simulated responses generated under the assumed IRT model. While these experiments establish feasibility under controlled conditions, they provide limited evidence that the framework can reliably measure human sensitivity to chart reconstruction errors in practice.

  3. Limited evidence of practical benefits. The improvements of adaptive selection over random selection are relatively modest at several evaluation budgets. The paper does not sufficiently establish whether these gains translate into meaningful reductions in evaluation cost under realistic conditions.

Questions for authors

Given prior work on IRT-based adaptive visualization assessment, what is the main methodological contribution of MetroBench beyond applying established techniques to chart reconstruction errors, and how can the validity of the resulting measurements be established with real human evaluators?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.