- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces functional equivalence as a criterion for evaluating generated and edited charts, grounding it in a taxonomy of 36 chart transformations organized into five severity tiers. It constructs CHARTFUN, in which six generators apply two to five requested transformations to 400 PlotQA-derived charts, yielding 2,400 candidate–target pairs, each judged by three annotators. Only 60.5% of candidates are judged equivalent to their rule-based targets, and existing metrics show limited agreement with these labels. Among three proposed evaluators, a theory-guided VLM judge achieves the strongest agreement, with a Pearson correlation of 0.60 and an AUROC of 0.84, while the structured comparison and the distilled image evaluator perform less well.
Critical strengths
-
The framework makes a meaningful distinction between visual resemblance and preservation of chart information and interpretation.
-
CHARTFUN covers multiple chart families and generators, with three distinct annotators per pair, randomized image order, and reported agreement statistics. The comparison covers several evaluator designs and transparently reports the weaker performance of the structured and distilled alternatives.
Critical weaknesses
-
The theoretical framework is not connected to the human labels. Section 3 defines functional equivalence across encoding, decoding, and interpretation and organizes 36 transformations into five severity tiers, but the human study does not measure these distinctions. Section 4.2 operationalizes the target through a binary question about the same data and meaning, and the paper reports no human results by transformation or severity tier. It therefore remains untested whether the tiers correspond to human judgments of equivalence, and it is unclear how annotators should classify information-preserving changes that increase decoding effort or alter the natural comparison. The reported κ and α of 0.51 make this boundary consequential. The tier penalties in the structured comparison (0 to 0.8) and the score ranges in the VLM judge rubric are likewise study-specific choices that the human labels do not validate.
-
The benefit of theoretical guidance is not established. The VLM Judge and GPTScore-GPT-5 both prompt GPT-5, so the comparison mainly tests the rubric. Table 3 shows an improvement from 0.54 to 0.60 in Pearson correlation but only from 0.83 to 0.84 in AUROC, without uncertainty estimates. The pooled results do not establish whether theoretical guidance improves detection of subtle semantic discrepancies or mainly separates generators with different overall quality. The other two proposed evaluators do not strengthen this evidence: the structured comparison underperforms GPTScore, and the image evaluator is trained to predict the VLM Judge's own scores.
Questions for authors
- Could you provide the annotation instructions and qualification examples, particularly for information-preserving changes that increase decoding effort or alter the natural comparison? How were these cases labeled, and what do representative annotator disagreements reveal about the boundary of functional equivalence? Given Section 4.3, which aspects of interpretation do the current labels actually establish?
- Could you provide the GPTScore baseline prompt and confirm whether its model and inference settings were matched to those of the theory-guided judge?
- What constitutes generation failure, what retry policy was used, and how many source replacements occurred for each model? Were replacement sources regenerated for all six models or only for the failing model?
- Could you provide code or complete pseudocode for the structured scorer, including series matching, numerical tolerances, null handling, discrepancy deduplication, and tier assignment, together with the parsing model and inference settings?
7 comments
Response to Reviewer AdXf (2/4)
W2: Benefit of theoretical guidance
Uncertainty estimates. Using a paired bootstrap over the 400 source charts, the VLM Judge's improvements over GPTScore are significant for all correlation measures and for the gap between pairs humans judged equivalent and non-equivalent (Sec. 5.3). AUROC is comparable, meaning both rank pairs similarly, while the VLM Judge separates the two classes more clearly.
| Measure | VLM Judge | GPTScore | Difference [95% CI] |
|---|---|---|---|
| Pearson | 0.60 | 0.54 | +0.06 [0.03, 0.09] |
| Spearman | 0.61 | 0.56 | +0.05 [0.02, 0.07] |
| Kendall | 0.54 | 0.46 | +0.08 [0.06, 0.10] |
| AUROC | 0.84 | 0.83 | +0.01 [−0.01, 0.03] |
| Gap | 0.41 | 0.18 | +0.24 [0.21, 0.26] |
Subtle discrepancies vs. generator quality. To rule out that the gain only reflects separating stronger from weaker generators, we computed correlations within each generator. Since all generators edit the same 400 source charts, this holds both generator and source fixed. The VLM Judge's Pearson correlation exceeds GPTScore's for every generator:
| Model | VLM Judge | GPTScore | Difference [95% CI] |
|---|---|---|---|
| Gemini 3.6 Flash | 0.35 | 0.27 | +0.07 [−0.02, 0.17] |
| GPT-5.5 | 0.34 | 0.27 | +0.07 [−0.04, 0.17] |
| Qwen3.8-27B | 0.52 | 0.43 | +0.09 [0.01, 0.17] |
| GPT Image 2 | 0.62 | 0.59 | +0.03 [−0.04, 0.09] |
| Gemini 3 Pro Image | 0.61 | 0.58 | +0.03 [−0.03, 0.09] |
| InternVL3.5-38B | 0.54 | 0.36 | +0.18 [0.11, 0.24] |
Isolating the rubric. We agree that the comparison with GPTScore tests the evaluation criteria. To separate the effect of the functional-equivalence definition from that of the rubric, we also ran a simple GPT-5 prompt asking only whether the charts "show the same data and convey the same meaning". The definition alone accounts for the correlation gain over GPTScore (Pearson 0.54 → 0.61). The rubric does not raise correlation further, but it significantly increases the gap (0.36 → 0.41) and makes scores sensitive to severity: without it, cosmetic, presentation and magnitude-distorting changes score almost identically (1.00, 0.91, 0.88 for T0–T2), whereas the rubric separates them (1.00, 0.67, 0.52).
The other evaluators. We agree they do not add evidence for the rubric, and we do not present them as such. They explore alternative mechanisms for the same target. The structured comparison is fully interpretable and outperforms the existing structured approach, ChartEval (Goswami et al., 2025) (Pearson 0.30 vs. 0.14–0.22). The image evaluator needs no API call at inference time, yet achieves a larger gap than all general visual similarity metrics (0.24 vs. at most 0.15).
Response to Reviewer AdXf (3/4)
Q1: Annotation instructions, disagreements and scope of the labels
Annotation instructions. The instructions do not change with the severity of the requested transformation. Each pair consists of the rule-based target, provided as ground truth, and the model-generated chart, so annotators judge whether the model reproduced the target, not whether the transformation itself preserves equivalence. Keeping the instructions fixed avoids introducing bias, and a model may well apply a severe transformation correctly. The full instructions shown to annotators were:
Compare two charts. Do Chart A and Chart B show the same data and convey the same meaning?
How to decide. Choose Yes only when a careful reader could get the same answers and conclusions from both charts. Look for:
- Values and patterns: compare the numbers or proportions, high and low values, rankings, and trends. Each value must still belong to the same label, category, or group.
- Meaning and structure: both charts must use the same kind of chart and the same way of showing the data. The subject, titles, units, axes, scales, categories, and legend labels must keep the same meaning and point to the correct data.
- Correct and complete: no important value, label, category, or group should be wrong, added, missing, or too hard to read.
Choose No if an important difference changes what the chart says or the answer a reader would get. Ignore only appearance changes that leave the information unchanged: fonts, background, spacing, gridlines, legend position, or a different colour palette when labels still point to the correct data. Changing the order of bars or pie slices is okay if every label still has the same value.
General rules. Compare all the information in both charts. Do not base your answer on which chart looks better. Do not guess. If an important part cannot be read well enough to compare, choose No.
Annotators completed 10 practice comparisons with feedback, followed by a 10-item qualification test without feedback (at least 7 correct to proceed). Chart order was randomised, and for a quarter of study pairs annotators also wrote a short explanation ("Briefly state the main similarities or differences and explain why they support your answer", max. 500 characters). We will include the instructions and qualification examples in the appendix.
Q2: GPTScore prompt and inference settings
The GPTScore baseline uses the score prompt from ChartMimic (Yang et al., 2025, Appendix E.1, Fig. 33) without modification, except that the backbone is replaced with GPT-5. The full prompt is:
You are an excellent judge at evaluating visualization chart plots. The first image (reference image) is created using ground-truth Matplotlib code, and the second image (AI-generated image) is created using Matplotlib code generated by an AI assistant. Your task is to score how well the AI-generated plot matches the ground-truth plot.
Scoring Methodology. The AI-generated image is evaluated according to the following six criteria, totaling 100 points:
- Chart Types (20 points): Does the AI-generated image include all chart types present in the reference image (e.g., line charts, bar charts, etc.)?
- Layout (10 points): Does the arrangement of subplots in the AI-generated image match the reference image (e.g., number of rows and columns)?
- Text Content (20 points): Does the AI-generated image include all text from the reference image (e.g., titles, annotations, axis labels), excluding axis tick labels?
- Data (20 points): How accurately do the data trends in the AI-generated image resemble those in the reference image? Is the number of data groups consistent with the reference image?
- Style (20 points): Does the AI-generated image match the reference in terms of colors (line colors, fill colors, etc.), marker types (point shapes, line styles, etc.), legends, grids, and other stylistic details?
- Clarity (10 points): Is the AI-generated image clear and free of overlapping elements?
Evaluation Instructions. Compare the two images head-to-head and provide a detailed assessment using the following format:
Comments:
- Chart Types: [Comment and subscore]
- Layout: [Comment and subscore]
- Text Content: [Comment and subscore]
- Data: [Comment and subscore]
- Style: [Comment and subscore]
- Clarity: [Comment and subscore]
Score: [Final score out of 100]
Please follow this format to ensure that the evaluation is clear, consistent, and comprehensive.
The model and inference settings are identical for both evaluators: both use GPT-5 with the same hyperparameters, receive the same 2 images in the same order (reference first, candidate second). Scores are rescaled to [0, 1] before computing correlations (GPTScore from 0–100, the VLM Judge from 0–10).
Response to Reviewer AdXf (1/4)
We thank the reviewer for the detailed feedback. We address each point below.
W1: Connection between the framework and the human labels
We clarify the role of the framework in the human study.
What annotators judge. Annotators compare a model-generated chart with its rule-based target, in which all requested transformations have already been applied. They do not judge whether a transformation itself preserves equivalence; they judge whether the model reproduced the target. Annotators therefore never need to classify information-preserving changes that increase decoding effort or alter the natural comparison: such changes are part of both charts, and a pair is non-equivalent only if the model applied the requested edits incorrectly. The boundary annotators applied was defined through 10 practice comparisons with feedback and a 10-item qualification test covering both equivalent and non-equivalent pairs (Sec. 4.2).
Role of the tiers. The framework and its tiers organise which transformations are requested and how our evaluators weigh discrepancies; the levels of encoding, decoding and interpretation are drawn from prior empirical work (Sec. 3), and we do not claim that our human labels validate them separately. However, we report human results by tier:
Percentage of model-generated charts judged functionally equivalent to their rule-based target (at least 2 of 3 annotators), by the most severe transformation requested.
Acceptance declines from T0–T1 to T2–T3 across generators, showing that higher-tier edits are harder for models to reproduce faithfully.
Agreement. Fleiss' κ and Krippendorff's α of 0.51 indicate moderate agreement, and final labels use majority vote over three qualified annotators.
Penalties and score ranges. We agree these are study-specific design choices (Sec. 5.1). The penalties are evenly spaced across tiers, and the most severe discrepancy dominates the score (Eq. 1). We evaluate the resulting scores against the human labels (Table 3) and check that they decrease with severity: we score each source chart against its rule-based target and group the pairs by their most severe transformation.
Both evaluators' scores decrease at every tier.