acceptodds
Under review as a conference paper at ICLR 2027

Theory-Driven Conceptualization and Implementation of Evaluation Metrics for Chart Editing

Abstract

Charts can be visually similar yet communicate different information, while visually different charts may support the same interpretation. This makes chart comparison fundamentally different from conventional image similarity and poses considerable evaluation challenges in tasks such as chart editing and chart generation. We define this as a matter of functional equivalence: whether a candidate chart preserves the information and inferences supported by a reference chart. To support the study of functional equivalence, we introduce ChartFun, a benchmark of human judgments on the functional equivalence of 2,400 model-generated charts against rule-based targets. ChartFun spans five chart types and six image- and code-based chart-generation models. Our analysis reveals substantial variation across models: only 60.5% of generated charts are judged functionally equivalent to their rule-based targets overall, despite many outputs being visually similar. More importantly, correlation studies reveal that existing evaluation metrics used in chart-generation studies show limited agreement with human judgments of functional equivalence. We explore three new evaluation approaches grounded in prior theory on data visualization: a vision-language model (VLM) judge, a structured theory-driven comparison pipeline, and a lightweight learned evaluator. Our best-performing method, the VLM Judge, reaches a Pearson correlation of 0.60. Our work opens new avenues for evaluating chart generation and editing through the theory-grounded framework of functional equivalence.

Primary area:
datasets and benchmarks
Keywords:
chart evaluation, functional equivalence, chart generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AdXfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces functional equivalence as a criterion for evaluating generated and edited charts, grounding it in a taxonomy of 36 chart transformations organized into five severity tiers. It constructs CHARTFUN, in which six generators apply two to five requested transformations to 400 PlotQA-derived charts, yielding 2,400 candidate–target pairs, each judged by three annotators. Only 60.5% of candidates are judged equivalent to their rule-based targets, and existing metrics show limited agreement with these labels. Among three proposed evaluators, a theory-guided VLM judge achieves the strongest agreement, with a Pearson correlation of 0.60 and an AUROC of 0.84, while the structured comparison and the distilled image evaluator perform less well.

Critical strengths
  1. The framework makes a meaningful distinction between visual resemblance and preservation of chart information and interpretation.

  2. CHARTFUN covers multiple chart families and generators, with three distinct annotators per pair, randomized image order, and reported agreement statistics. The comparison covers several evaluator designs and transparently reports the weaker performance of the structured and distilled alternatives.

Critical weaknesses
  1. The theoretical framework is not connected to the human labels. Section 3 defines functional equivalence across encoding, decoding, and interpretation and organizes 36 transformations into five severity tiers, but the human study does not measure these distinctions. Section 4.2 operationalizes the target through a binary question about the same data and meaning, and the paper reports no human results by transformation or severity tier. It therefore remains untested whether the tiers correspond to human judgments of equivalence, and it is unclear how annotators should classify information-preserving changes that increase decoding effort or alter the natural comparison. The reported κ and α of 0.51 make this boundary consequential. The tier penalties in the structured comparison (0 to 0.8) and the score ranges in the VLM judge rubric are likewise study-specific choices that the human labels do not validate.

  2. The benefit of theoretical guidance is not established. The VLM Judge and GPTScore-GPT-5 both prompt GPT-5, so the comparison mainly tests the rubric. Table 3 shows an improvement from 0.54 to 0.60 in Pearson correlation but only from 0.83 to 0.84 in AUROC, without uncertainty estimates. The pooled results do not establish whether theoretical guidance improves detection of subtle semantic discrepancies or mainly separates generators with different overall quality. The other two proposed evaluators do not strengthen this evidence: the structured comparison underperforms GPTScore, and the image evaluator is trained to predict the VLM Judge's own scores.

Questions for authors
  1. Could you provide the annotation instructions and qualification examples, particularly for information-preserving changes that increase decoding effort or alter the natural comparison? How were these cases labeled, and what do representative annotator disagreements reveal about the boundary of functional equivalence? Given Section 4.3, which aspects of interpretation do the current labels actually establish?
  2. Could you provide the GPTScore baseline prompt and confirm whether its model and inference settings were matched to those of the theory-guided judge?
  3. What constitutes generation failure, what retry policy was used, and how many source replacements occurred for each model? Were replacement sources regenerated for all six models or only for the failing model?
  4. Could you provide code or complete pseudocode for the structured scorer, including series matching, numerical tolerances, null handling, discrepancy deduplication, and tier assignment, together with the parsing model and inference settings?
7 comments
AuthorsOpenReview ↗

Response to Reviewer AdXf (1/4)

We thank the reviewer for the detailed feedback. We address each point below.

W1: Connection between the framework and the human labels

We clarify the role of the framework in the human study.

What annotators judge. Annotators compare a model-generated chart with its rule-based target, in which all requested transformations have already been applied. They do not judge whether a transformation itself preserves equivalence; they judge whether the model reproduced the target. Annotators therefore never need to classify information-preserving changes that increase decoding effort or alter the natural comparison: such changes are part of both charts, and a pair is non-equivalent only if the model applied the requested edits incorrectly. The boundary annotators applied was defined through 10 practice comparisons with feedback and a 10-item qualification test covering both equivalent and non-equivalent pairs (Sec. 4.2).

Role of the tiers. The framework and its tiers organise which transformations are requested and how our evaluators weigh discrepancies; the levels of encoding, decoding and interpretation are drawn from prior empirical work (Sec. 3), and we do not claim that our human labels validate them separately. However, we report human results by tier:

Max tierGemini 3.6 FlashGPT-5.5Qwen3.8-27BGPT Image 2Gemini 3 Pro ImageInternVL3.5-38BAccepted
T0100.087.537.5100.075.062.577.1
T188.195.275.483.379.433.375.8
T280.072.068.048.020.016.050.7
T372.750.054.522.79.113.637.1
T484.581.768.539.332.021.054.5
All85.083.869.254.045.825.060.5

Percentage of model-generated charts judged functionally equivalent to their rule-based target (at least 2 of 3 annotators), by the most severe transformation requested.

Acceptance declines from T0–T1 to T2–T3 across generators, showing that higher-tier edits are harder for models to reproduce faithfully.

Agreement. Fleiss' κ and Krippendorff's α of 0.51 indicate moderate agreement, and final labels use majority vote over three qualified annotators.

Penalties and score ranges. We agree these are study-specific design choices (Sec. 5.1). The penalties are evenly spaced across tiers, and the most severe discrepancy dominates the score (Eq. 1). We evaluate the resulting scores against the human labels (Table 3) and check that they decrease with severity: we score each source chart against its rule-based target and group the pairs by their most severe transformation.

Most severe transformationVLM JudgeStructured Comparison
T01.000.91
T10.670.48
T20.520.45
T30.380.24
T40.240.10

Both evaluators' scores decrease at every tier.

AuthorsOpenReview ↗

Response to Reviewer AdXf (2/4)

W2: Benefit of theoretical guidance

Uncertainty estimates. Using a paired bootstrap over the 400 source charts, the VLM Judge's improvements over GPTScore are significant for all correlation measures and for the gap between pairs humans judged equivalent and non-equivalent (Sec. 5.3). AUROC is comparable, meaning both rank pairs similarly, while the VLM Judge separates the two classes more clearly.

MeasureVLM JudgeGPTScoreDifference [95% CI]
Pearson0.600.54+0.06 [0.03, 0.09]
Spearman0.610.56+0.05 [0.02, 0.07]
Kendall0.540.46+0.08 [0.06, 0.10]
AUROC0.840.83+0.01 [−0.01, 0.03]
Gap0.410.18+0.24 [0.21, 0.26]

Subtle discrepancies vs. generator quality. To rule out that the gain only reflects separating stronger from weaker generators, we computed correlations within each generator. Since all generators edit the same 400 source charts, this holds both generator and source fixed. The VLM Judge's Pearson correlation exceeds GPTScore's for every generator:

ModelVLM JudgeGPTScoreDifference [95% CI]
Gemini 3.6 Flash0.350.27+0.07 [−0.02, 0.17]
GPT-5.50.340.27+0.07 [−0.04, 0.17]
Qwen3.8-27B0.520.43+0.09 [0.01, 0.17]
GPT Image 20.620.59+0.03 [−0.04, 0.09]
Gemini 3 Pro Image0.610.58+0.03 [−0.03, 0.09]
InternVL3.5-38B0.540.36+0.18 [0.11, 0.24]

Isolating the rubric. We agree that the comparison with GPTScore tests the evaluation criteria. To separate the effect of the functional-equivalence definition from that of the rubric, we also ran a simple GPT-5 prompt asking only whether the charts "show the same data and convey the same meaning". The definition alone accounts for the correlation gain over GPTScore (Pearson 0.54 → 0.61). The rubric does not raise correlation further, but it significantly increases the gap (0.36 → 0.41) and makes scores sensitive to severity: without it, cosmetic, presentation and magnitude-distorting changes score almost identically (1.00, 0.91, 0.88 for T0–T2), whereas the rubric separates them (1.00, 0.67, 0.52).

The other evaluators. We agree they do not add evidence for the rubric, and we do not present them as such. They explore alternative mechanisms for the same target. The structured comparison is fully interpretable and outperforms the existing structured approach, ChartEval (Goswami et al., 2025) (Pearson 0.30 vs. 0.14–0.22). The image evaluator needs no API call at inference time, yet achieves a larger gap than all general visual similarity metrics (0.24 vs. at most 0.15).

AuthorsOpenReview ↗

Response to Reviewer AdXf (3/4)

Q1: Annotation instructions, disagreements and scope of the labels

Annotation instructions. The instructions do not change with the severity of the requested transformation. Each pair consists of the rule-based target, provided as ground truth, and the model-generated chart, so annotators judge whether the model reproduced the target, not whether the transformation itself preserves equivalence. Keeping the instructions fixed avoids introducing bias, and a model may well apply a severe transformation correctly. The full instructions shown to annotators were:

Compare two charts. Do Chart A and Chart B show the same data and convey the same meaning?

How to decide. Choose Yes only when a careful reader could get the same answers and conclusions from both charts. Look for:

  1. Values and patterns: compare the numbers or proportions, high and low values, rankings, and trends. Each value must still belong to the same label, category, or group.
  2. Meaning and structure: both charts must use the same kind of chart and the same way of showing the data. The subject, titles, units, axes, scales, categories, and legend labels must keep the same meaning and point to the correct data.
  3. Correct and complete: no important value, label, category, or group should be wrong, added, missing, or too hard to read.

Choose No if an important difference changes what the chart says or the answer a reader would get. Ignore only appearance changes that leave the information unchanged: fonts, background, spacing, gridlines, legend position, or a different colour palette when labels still point to the correct data. Changing the order of bars or pie slices is okay if every label still has the same value.

General rules. Compare all the information in both charts. Do not base your answer on which chart looks better. Do not guess. If an important part cannot be read well enough to compare, choose No.

Annotators completed 10 practice comparisons with feedback, followed by a 10-item qualification test without feedback (at least 7 correct to proceed). Chart order was randomised, and for a quarter of study pairs annotators also wrote a short explanation ("Briefly state the main similarities or differences and explain why they support your answer", max. 500 characters). We will include the instructions and qualification examples in the appendix.

Q2: GPTScore prompt and inference settings

The GPTScore baseline uses the score prompt from ChartMimic (Yang et al., 2025, Appendix E.1, Fig. 33) without modification, except that the backbone is replaced with GPT-5. The full prompt is:

You are an excellent judge at evaluating visualization chart plots. The first image (reference image) is created using ground-truth Matplotlib code, and the second image (AI-generated image) is created using Matplotlib code generated by an AI assistant. Your task is to score how well the AI-generated plot matches the ground-truth plot.

Scoring Methodology. The AI-generated image is evaluated according to the following six criteria, totaling 100 points:

  1. Chart Types (20 points): Does the AI-generated image include all chart types present in the reference image (e.g., line charts, bar charts, etc.)?
  2. Layout (10 points): Does the arrangement of subplots in the AI-generated image match the reference image (e.g., number of rows and columns)?
  3. Text Content (20 points): Does the AI-generated image include all text from the reference image (e.g., titles, annotations, axis labels), excluding axis tick labels?
  4. Data (20 points): How accurately do the data trends in the AI-generated image resemble those in the reference image? Is the number of data groups consistent with the reference image?
  5. Style (20 points): Does the AI-generated image match the reference in terms of colors (line colors, fill colors, etc.), marker types (point shapes, line styles, etc.), legends, grids, and other stylistic details?
  6. Clarity (10 points): Is the AI-generated image clear and free of overlapping elements?

Evaluation Instructions. Compare the two images head-to-head and provide a detailed assessment using the following format:

Comments:

  • Chart Types: [Comment and subscore]
  • Layout: [Comment and subscore]
  • Text Content: [Comment and subscore]
  • Data: [Comment and subscore]
  • Style: [Comment and subscore]
  • Clarity: [Comment and subscore]

Score: [Final score out of 100]

Please follow this format to ensure that the evaluation is clear, consistent, and comprehensive.

The model and inference settings are identical for both evaluators: both use GPT-5 with the same hyperparameters, receive the same 2 images in the same order (reference first, candidate second). Scores are rescaled to [0, 1] before computing correlations (GPTScore from 0–100, the VLM Judge from 0–10).

Reviewer_eF6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of this paper introduces functional equivalence as a criterion for evaluating chart editing and constructs ChartFun from 2,400 model-generated pairs across five chart types, collecting 7,200 human judgments involving six generators. The paper benchmarks existing metrics from the literature and other proposed evaluators to conclude that a theory-guided (LLM) judge achieves the strongest alignment with majority-vote labels.

Critical strengths
  1. This paper tries to address an important shortcomings of existing chart evaluators by connecting functional equivalence to several other factors to establish a useful conceptual framework for chart editing.

  2. The ChartFun benchmark includes 2,400 chart Paris spanning five chart types and includes both image-based and code-based generators.

Critical weaknesses
  1. The framework is not independently validated as the human annotators provide only binary judgements about whether charts show the same data and convey the same meaning. Do the authors have empirical evidence that the dimensions of encoding, decoding, and interpretation are distinguishable in human judgements?

  2. ChartFun is derived from only 400 source charts drawn from PlotQA, using five comparatively simple chart types and applied transformations on those, this makes the individual failure sources difficult to isolate. The evaluation doesn’t report separately and the controlled synthetic edits leaves the question open whether these findings generalize to realistic settings of chart editing, complex visualizations etc.

Questions for authors

See the "Critical Weaknesses" as well.

  1. Do the gains remain consistent when correlations are computed separately for each model and chart type from the shared source charts?

  2. What is the impact of the theory-grounded rubric? It’s not clearly disentangled.

  3. What are the distributions of 36 transformations in ChartFun and are the results consistent?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eF6g (1/3)

We thank the reviewer for the thoughtful feedback. We address each point below.

W1: Validation of the theoretical framework

Our study validates the overall functional-equivalence decisions, and we do not claim that our human judgments separate encoding, decoding and interpretation. These levels are not dimensions we propose to measure separately; they come from prior theoretical work on graph perception and comprehension (Sec. 3). We use them to organise the taxonomy, while the human study targets the decision an evaluation metric must make: whether a candidate chart is functionally equivalent to its reference. We will discuss level-specific human judgments as future work in the revision.

W2: Dataset scope, failure sources and generalisation

Chart types. The five families in ChartFun (horizontal bar, vertical bar, line, dot-line and pie) are among the most widely used for conveying information in visualisations (Battle et al., 2018; Zeng & Battle, 2023), and the source charts come from real-world data (World Bank, Open Government Data, Global Terrorism Database). As our results show, the problem is not solved even for these types. We should therefore first solve the quantification of functional equivalence for these widely used types, and then extend the approach to dense charts, multi-panel figures and other complex visualisations. We leave this to future work and will clarify it in the revision.

Isolating failure sources. The controlled edits are what make failure sources isolable: because every rule-based target is generated programmatically, the exact discrepancies requested for each chart are known. Table 2 of the paper already reports equivalence rates separately for each generator and chart type. Below we further break the results down by the most severe discrepancy requested:

Max tierGemini 3.6 FlashGPT-5.5Qwen3.8-27BGPT Image 2Gemini 3 Pro ImageInternVL3.5-38BAccepted
T0100.087.537.5100.075.062.577.1
T188.195.275.483.379.433.375.8
T280.072.068.048.020.016.050.7
T372.750.054.522.79.113.637.1
T484.581.768.539.332.021.054.5
All85.083.869.254.045.825.060.5

Percentage of model-generated charts judged functionally equivalent to their rule-based target (at least 2 of 3 annotators).

Acceptance generally declines as the tier increases, and the decline is much steeper for image-generation models. For example, GPT Image 2 drops from 100% at T0 to 22.7% at T3. We will add this breakdown to the revision.

Generalisation. We further applied our VLM Judge, GPTScore and a simple prompt ("Do these two charts show the same data and convey the same meaning? Give a score from 0 to 10.", run on GPT-5) to the test_small split of ChartCraft (Yan et al., 2024), a chart-editing dataset with its own charts and edit categories (105 style, 50 layout, 100 format and 300 data-centric edits):

EvaluatorStyleLayoutFormatData-centric
VLM Judge (ours)0.990.990.480.37
Simple prompt0.990.990.760.52
GPTScore0.960.960.500.66

The key category is data-centric edits, where the underlying data has been changed, so the edited chart conveys different information and should receive a low score. GPTScore still gives these charts 0.66 and the simple prompt 0.52, while our VLM Judge gives them 0.37. For style and layout edits, which leave the data unchanged, all three evaluators give high scores.

AuthorsOpenReview ↗

Response to Reviewer eF6g (2/3)

Q1: Consistency across models and chart types

Yes. We computed agreement with human judgments separately for each generation model (400 pairs each, all sharing the same 400 source charts) and each chart type (480 pairs each). Overall, the VLM Judge reaches 0.60 Pearson versus 0.54 for GPTScore (+0.06, 95% CI [0.03, 0.09]), and its correlation is higher in all 11 subgroups.

By generation model

ModelVLM Judge (ours)GPTScore (GPT-5)Difference [95% CI]
Gemini 3.6 Flash0.350.27+0.07 [−0.02, 0.17]
GPT-5.50.340.27+0.07 [−0.04, 0.17]
Qwen3.8-27B0.520.43+0.09 [0.01, 0.17]
GPT Image 20.620.59+0.03 [−0.04, 0.09]
Gemini 3 Pro Image0.610.58+0.03 [−0.03, 0.09]
InternVL3.5-38B0.540.36+0.18 [0.11, 0.24]

By chart type

Chart typeVLM Judge (ours)GPTScore (GPT-5)Difference [95% CI]
H. bars0.620.58+0.04 [−0.01, 0.09]
V. bars0.580.57+0.01 [−0.03, 0.06]
Lines0.610.57+0.05 [−0.02, 0.11]
Marked0.680.60+0.07 [0.01, 0.14]
Pies0.500.42+0.08 [−0.02, 0.20]

Since each subgroup contains only a fifth or sixth of the data, most individual intervals include zero, but the direction of the gain is consistent across all models and chart types. The gain is largest for the generators with the most non-equivalent outputs, InternVL3.5-38B (+0.18) and Qwen3.8-27B (+0.09), which is where the rubric's penalties for meaning-changing discrepancies matter most. Correlations are lower for Gemini 3.6 Flash and GPT-5.5 for both evaluators, since most of their outputs are equivalent (85% and 84%), leaving few non-equivalent pairs to separate.

Q2: Impact of the theory-grounded rubric

We disentangle the rubric's impact by comparing three evaluators that use the same backbone (GPT-5) and differ only in their prompt: GPTScore (visual similarity), a simple prompt asking only for functional equivalence ("Do these two charts show the same data and convey the same meaning? Give a score from 0 to 10."), and our VLM Judge (functional equivalence with the theory-grounded rubric). All are evaluated against human judgments on the 2,400 ChartFun pairs.

EvaluatorPearsonSpearmanKendallAUROCGap
GPTScore0.540.560.460.830.18
Simple prompt0.610.630.570.850.36
VLM Judge (ours)0.600.610.540.840.41

Moving from similarity to functional equivalence significantly improves all three correlations (e.g. Pearson 0.54 → 0.61, 95% CI of the difference [0.04, 0.09]). Adding the rubric does not improve correlation further, but it significantly improves the separation between pairs humans judged equivalent and non-equivalent (Gap 0.36 → 0.41, 95% CI of the difference [0.03, 0.08]).

AuthorsOpenReview ↗

Response to Reviewer eF6g (Part 3/3)

Q3: Distribution of transformations

We would like to clarify how ChartFun is constructed. Each source chart receives two to five transformations sampled from Table 1, which are requested together as a single editing instruction. Human annotators then compare the model's output with the rule-based target, in which all requested transformations have already been applied correctly. A pair is therefore judged non-equivalent not because of a transformation itself, but because the model applied the requested edits incorrectly. Per-transformation results would not be separable, since each pair combines several transformations (2–5). For this reason, we report results by generator and chart type (Table 2) and by the most severe transformation requested (see W2).

The number of the 400 source charts in which each transformation appears (each source chart is edited by all six models):

TransformationSourcesTransformationSources
X/Y-axis title removal99Metadata removal46
Chart title removal78Unrelated title replacement45
Legend relocation74Value reassignment39
Background colour change73Data value modification39
Font family change71Legend colour reassignment38
Consistent colour palette change64Legend removal38
Data addition/removal64Legend label replacement37
Reduced colour discriminability60Category reordering25
Layout adjustment57Axis inversion24
Contradictory value labels50Linear-to-log scale change22
Axis tick-mark removal49Secondary-axis assignment18
X/Y-axis tick-label removal49Unequal bar thickness12
Additional minor axis ticks48Grouped-to-stacked bars11
Irregular tick intervals48Bars-to-lines conversion11
Expanded/Truncated axis range47Bars-to-points conversion10
Gridline toggle46Categorical line connection8

References

  • Battle, L., et al. (2018). Beagle: Automated extraction and interpretation of visualizations from the web. CHI 2018.
  • Yan, P., et al. (2024). ChartReformer: Natural language-driven chart image editing. ICDAR 2024.
  • Zeng, Z., & Battle, L. (2023). A review and collation of graphical perception knowledge for visualization recommendation. CHI 2023.

I thank the authors for their rebuttal and additional analyses, clarifications are helpful.

But I remain unconvinced about the theoretical framework and validation through human judgments, also the rubric offers limited improvement over a simple functional-equivalence prompt. Moreover, the heavy reliance on relatively simple chart types also raises questions about generalizability.

After considering the rebuttal and other reviewers' discussions, I will maintain my original rating.

Reviewer_utP3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies functional equivalence for chart evaluation: preserving the same information and principal viewer inferences, while allowing changes in appearance. It releases CHARTFUN, which contains 2,400 generated charts from 400 source charts, five chart families, and six models. four code generation and two image generation models. Three independent human judges evaluate each pair. It Uses 36 transformations covering cosmetic changes, interpretability and changes to encoded information. Overall, 60.5% of candidates are judged equivalent to their targets. Three evaluators: a rubric-based GPT-5 judge, structured chart extraction followed by deterministic comparison, and a learned evaluator trained on the judge’s scores. The VLM judge performs best, with Pearson correlation 0.60 against binary human majority labels and AUROC 0.84.

Critical strengths
  • It is a standing problem. Preserving data values alone does not guarantee the same interpretation. Considering readability and visual impression is a reasonable direction.
  • Substantial human evaluation effort. 7,200 judgments, three distinct annotators per editing pair.
  • Inspectable judge rubric. Concrete examples of discrepancies that matter for chart meaning.
  • Reasonable model coverage, including both code and direct image generation.
Critical weaknesses
  • The limitations of SSIM, PSNR and LPIPS for chart correctness are already well established. Showing these limitations again adds little novelty. VLM judging and structured chart comparison are also established approaches, including in ChartEval and FigEdit.
  • Limited improvement over the strongest baseline. Both use GPT-5, with Pearson increasing from 0.54 to 0.60 and AUROC from 0.83 to 0.84.
  • Scoring is largely heuristic. Theory motivates the discrepancy categories, but the numerical penalties, score ranges and aggregation rule are not validated. Table 1 defines the transformations, but their assignment to severity tiers is unclear, particularly when severity depends on context.
  • Limited chart diversity. Five source families, several closely related. No clear evaluation on dense charts, multipanel figures or more varied visual encodings.
Questions for authors
  • How much does the proposed rubric improve over a simple GPT-5 prompt asking whether the charts preserve the same data and meaning? Which rubric components account for the gain?
  • How are discrepancies assigned to severity tiers? How sensitive are the results to the chosen penalties and score ranges?
  • How would the methods generalize to more complex charts and unseen chart families?
3 comments
AuthorsOpenReview ↗

Response to Reviewer utP3 (1/3)

We thank the reviewer for the careful reading and constructive feedback. We address each point below.

W1: Novelty relative to established metrics and evaluators

We agree that visual similarity metrics have severe limitations for charts, and we do not claim this as a contribution. We evaluate them because they remain standard in chart and figure generation (Belouadi et al., 2024a; 2024b; Rodriguez et al., 2025; Li et al., 2025), so they are needed to establish the baseline and the current state of the art.

Similarly, we do not claim VLM judging or structured comparison as new evaluator types. Our contribution is to show that these established directions are also severely limited for judging whether an edited chart preserves its information. ChartEval reports Pearson correlations of 0.45–0.89 with human ratings (Goswami et al., 2025, Table 2), but we were unable to reproduce this: on ChartFun, ChartEval reaches only 0.14–0.22, and GPTScore 0.54. Reproducibility is also limited because ChartEval did not release its dataset or the official code for the metric described in the paper, only a demo version.

To address these shortcomings, we formally define functional equivalence, grounded in visualisation theory, and provide ChartFun as a benchmark of human judgments against which current and future chart evaluators can be compared.

W2: Improvement over GPTScore

Our VLM Judge and GPTScore both use GPT-5 with identical inputs, so the difference lies only in the rubric prompt. The VLM Judge's improvement in Pearson correlation over GPTScore, from 0.54 to 0.60, is statistically significant (+0.06, 95% CI [0.03, 0.09]). Their AUROC is comparable (0.84 vs. 0.83), but AUROC only measures whether equivalent pairs are ranked above non-equivalent ones, not how far apart their scores are. This distinction matters in practice, since chart-editing benchmarks report average scores rather than rankings. Here the VLM Judge separates the two classes much more clearly: the gap between the average scores of pairs humans accepted and rejected is 0.41 for the VLM Judge and 0.18 for GPTScore. GPTScore originally relied on GPT-4o, which reaches 0.45 Pearson on ChartFun; we run it on GPT-5 to match our judge.

We attribute the gain to our definition of functional equivalence. GPTScore (Yang et al., 2025) scores similarity over six criteria, with data worth only 20 of 100 points, the same as style. As a result, a chart with falsified data can keep most of its score.

W3: Severity tiers, penalties and aggregation

We agree that the theory motivates the categories, while the numerical values are design choices (Sec. 5.1). Each transformation is assigned to a tier by what it changes: T0 cosmetic, T1 harder to read, T2 distorted magnitudes, T3 altered comparisons, T4 falsified information (Sec. 3). We will add a tier column to Table 1 so that every assignment is explicit.

Context is handled by the comparison rules rather than by the transformation type. Differences that preserve the same information are not penalised, such as 10 vs. 10.0, reordered unordered categories, or horizontal vs. vertical bars (Sec. 5.2). The five tiers map to evenly spaced penalties (0, 0.2, 0.4, 0.6, 0.8), so that no tier is weighted arbitrarily relative to its neighbours. The most severe discrepancy contributes its full penalty and the others contribute through a bounded term (Eq. 1), so many minor differences cannot outweigh one falsification.

We score each source chart against its rule-based target and group the pairs by their most severe transformation. Structured Comparison scores decrease at every tier, from T0 to T4:

Most severe edit appliedVLM JudgeGPTScoreStructured Comparison
T0: cosmetic1.000.900.91
T1: presentation0.670.740.48
T2: magnitude distortion0.520.750.45
T3: structural0.380.470.24
T4: falsification0.240.550.10
AuthorsOpenReview ↗

Response to Reviewer utP3 (2/3)

W4: Chart diversity

These five chart types are among the most widely used for conveying information in visualisations (Battle et al., 2018; Zeng & Battle, 2023). As we demonstrate in our work, the problem is not solved even for these five types, so we should first solve the quantification of functional equivalence for such widely used visualisation types before generalising and expanding this approach to other types, such as dense charts and multi-panel figures. We leave this to future work and will clarify this in the revision of our paper.

Q1: Rubric vs. a simple prompt

We compared our VLM Judge with a simple GPT-5 prompt: "Do these two charts show the same data and convey the same meaning? Give a score from 0 to 10."

Agreement with human judgments. Both were scored on the 2,400 ChartFun pairs (rule-based target vs. model-generated chart) against the human majority labels. Gap is the difference in mean normalised score between pairs humans judged equivalent and non-equivalent (Sec. 5.3, Table 4).

EvaluatorPearsonSpearmanKendallAUROCGap
VLM Judge (ours)0.600.610.540.840.41
Simple prompt0.610.630.570.850.36
GPTScore0.540.560.460.830.18

The rubric and the simple prompt agree with humans equally well (no significant difference), and both outperform GPTScore. The rubric separates equivalent from non-equivalent pairs significantly better (0.41 vs. 0.36; 95% CI of the difference [0.03, 0.08]).

Sensitivity to severity. We scored each source chart against its rule-based target, grouped by most severe discrepancy:

Most severe discrepancyVLM Judge (rubric)Simple prompt
T0: cosmetic1.001.00
T1: presentation0.670.91
T2: magnitude distortion0.520.88
T3: structural0.380.56
T4: falsification0.240.38

The simple prompt scores T0–T2 almost identically, treating truncated or log-scaled axes as nearly cosmetic. The gain comes mainly from the rubric's caps on axis range and scale changes (T2: 0.52 vs. 0.88) and its checks on series identity, values and labels (T4: 0.24 vs. 0.38).

Transfer to ChartCraft. We applied the evaluators unchanged to the test_small split of ChartCraft (Yan et al., 2024), which groups edits into four categories:

EvaluatorStyleLayoutFormatData-centric
VLM Judge (ours)0.990.990.480.37
Simple prompt0.990.990.760.52
GPTScore0.960.960.500.66

The clearest case is data-centric edits, which alter the underlying data so that the edited chart conveys different information. Here the rubric gives the lowest score (0.37), compared with 0.52 for the simple prompt and 0.66 for GPTScore. For style and layout edits, which leave the data unchanged, all three evaluators give high scores. In sum, the rubric matches the simple prompt on agreement with humans, while being more sensitive to meaning-changing discrepancies.

AuthorsOpenReview ↗

Response to Reviewer utP3 (3/3)

Q2: Assignment of discrepancies to severity tiers

Discrepancies are assigned to severity tiers by the purpose and the function we get from applying each of them. If there are multiple discrepancies with different tiers, the dominant discrepancy, which incurs the highest penalty, is the one in the highest tier. Each discrepancy can be assigned to a tier unambiguously because each tier is isolated and serves a different purpose. Furthermore, the rule-based edits are applied systematically: we take the original underlying PlotQA annotations and modify them ourselves, so that they serve as the ground truth for each discrepancy.

Each tier from T0 to T4 loses more information and function from the original chart. Discrepancies that are negligible and do not affect the viewer's understanding of the chart, such as a colour or font change, are assigned to T0. T1 is also considered functionally equivalent, but the chart is harder to read; it includes discrepancies such as removing the chart title. From T2 onwards, functional equivalence no longer holds. In T2, the data is the same but its magnitudes can be misperceived, as with a truncated axis or a log scale. In T3, the data can still be recovered but the visual encoding is substantially altered, for example by changing a bar plot to a line plot, which can produce visualisation mirages. In T4, the changes give the viewer a completely different view of the chart, for example removing a series or changing the value of a data point.

Max tierGemini 3.6 FlashGPT-5.5Qwen3.8-27BGPT Image 2Gemini 3 Pro ImageInternVL3.5-38BAccepted
T0100.087.537.5100.075.062.577.1
T188.195.275.483.379.433.375.8
T280.072.068.048.020.016.050.7
T372.750.054.522.79.113.637.1
T484.581.768.539.332.021.054.5
All85.083.869.254.045.825.060.5

Percentage of model-generated charts judged functionally equivalent to their rule-based target (at least 2 of 3 annotators), by the most severe discrepancy requested.

We can see that the proportion of accepted charts declines on average as the tier increases. This means functionally equivalent discrepancies are easier for models to apply. T4 is the exception, with more than half of its instances accepted, which can have two reasons: a) however function-deteriorating a discrepancy is, it can still be easy for models to apply; for example, changing one value or adding a random title is not hard. b) These discrepancies can be subtle: at a glance, the two charts look the same, except for a value that has been added or removed.

Q3: Generalisation to unseen charts

To test generalisation beyond ChartFun, we applied our VLM Judge, the simple prompt (see Q1) and GPTScore unchanged to the test_small split of ChartCraft (Yan et al., 2024), a separately constructed chart-editing dataset with its own charts and edit categories (105 style, 50 layout, 100 format and 300 data-centric edits).

EvaluatorStyleLayoutFormatData-centric
VLM Judge (ours)0.990.990.480.37
Simple prompt0.990.990.760.52
GPTScore0.960.960.500.66

The key category is data-centric edits, where the underlying data has been changed, so the edited chart conveys different information and should receive a low score. GPTScore still gives these charts 0.66 and the simple prompt 0.52, while our VLM Judge gives them 0.37. For style and layout edits, which leave the data unchanged, all three evaluators give high scores.

References

  • Battle, L., et al. (2018). Beagle: Automated extraction and interpretation of visualizations from the web. CHI 2018.
  • Belouadi, J., Lauscher, A., & Eger, S. (2024a). AutomaTikZ: Text-guided synthesis of scientific vector graphics with TikZ. ICLR 2024.
  • Belouadi, J., Ponzetto, S. P., & Eger, S. (2024b). DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ. NeurIPS 2024.
  • Goswami, K., et al. (2025). ChartEval: LLM-driven chart generation evaluation using scene graph parsing. IJCNLP-AACL 2025 (System Demonstrations).
  • Li, S., et al. (2025). Charts are not images: On the challenges of scientific chart editing. arXiv:2512.00752.
  • Rodriguez, J. A., et al. (2025). StarVector: Generating scalable vector graphics code from images and text. CVPR 2025.
  • Yan, P., et al. (2024). ChartReformer: Natural language-driven chart image editing. ICDAR 2024.
  • Yang, C., et al. (2025). ChartMimic: Evaluating LMM's cross-modal reasoning capability via chart-to-code generation. ICLR 2025.
  • Zeng, Z., & Battle, L. (2023). A review and collation of graphical perception knowledge for visualization recommendation. CHI 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.