acceptodds
Under review as a conference paper at ICLR 2027

ChartVL: Benchmarking SVG Chart Generation through Visual Layer Reconstruction and Verifiable Judgment

Abstract

Evaluating SVG charts generated by large language models is challenging because appropriate layout, layering, and styling depend on the roles of chart elements and the task requirements. Existing approaches often use vision-language models (VLMs) to evaluate the visual quality of charts, but they are prone to missing or misjudging visual issues, and their judgments, expressed as scores or textual explanations, are difficult to verify. We introduce CHARTVL, a benchmark for SVG chart generation built on visual layer reconstruction and verifiable judgment. We reconstruct visual layers to inform VLM judgments and require executable conditions alongside each determinate visual-design judgment. Executing these conditions confirms or revises the initial verdicts using evidence from the rendered SVG. CHARTVL contains 360 tasks spanning four commonly used chart categories and three task levels with increasingly complex design requirements. Across four evaluated models, 83.1%–99.2% of outputs use the requested chart type and correctly present the required data and content, but only 9.7%–58.6% also pass the verified visual evaluation of layout, layering, and styling. Our findings show that chart generation requires more than correct content. Coordinating visual layers into a readable whole is a distinct challenge that calls for structured, verifiable evaluation.

Primary area:
datasets and benchmarks
Keywords:
chart generation, SVG, benchmark, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_3bfEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The framework makes individual design judgments inspectable through element references, rationales, executable conditions, and retained execution evidence.
  • The reporting distinguishes structural evaluator errors and unresolved judgments from confirmed chart defects.
Critical strengths

This paper introduces CHARTVL, a benchmark of 360 SVG chart-generation tasks spanning four chart categories and three design-requirement levels. Its evaluation framework reconstructs visual layers, anchors them to rendered SVG elements, and executes VLM-written conditions for three visual-design dimensions. Experiments on four models show substantially higher task fulfillment than overall success under the proposed protocol. Overall, I lean toward weak reject because the evaluator's accuracy and the benefit of its central components remain empirically unvalidated, and the benchmark artifacts are not available for review.

Critical weaknesses
  1. Evaluator accuracy is not independently established. My main concern is the validity of the judgments underlying the benchmark results in Secs. 4–5. The paper motivates CHARTVL by the unreliability of VLM judges, yet reconstruction, task fulfillment, and all design judgments rely on a single VLM, gemini-3.8-flash, and no human agreement study is reported. Structural anchoring establishes element correspondence, but does not certify semantic roles. Task fulfillment remains a model judgment, and the VLM selects the scope, condition, and numerical allowances that execution subsequently checks. A correctly executed assertion can therefore miss a relevant defect or test an inappropriate criterion. The results contain patterns that make this concern concrete. Claude Opus 5 fulfills 96.4% of tasks but has confirmed out-of-bound failures on 59.8% of eligible charts, compared with 3.1% for GPT 5.6 Sol (Table 8). Since responses are rendered exactly as returned, including any Markdown fences (Appendix A.1), it is unclear whether such differences reflect chart defects or artifacts of the rendering protocol.

  2. The benefit of reconstruction and execution is not measured. In Sec. 5, the experiments compare generation models but do not establish the incremental benefit of the two central evaluation components. Fig. 1 illustrates a useful occlusion case, yet no comparison shows whether reconstruction improves defect detection beyond the task, code, image, and advisory hints. Similarly, although Sec. 4.3 records corrections when execution changes an initial verdict, neither their frequency nor their accuracy is reported. This leaves the practical value of reconstruction and executable conditions unresolved, which is central because both are claimed as primary contributions.

Questions for authors
  1. Was any human agreement study conducted for task fulfillment, structural verification, or the three design dimensions? If so, what were the agreement rates, false positives, and missed defects?
  2. How often does assertion execution overturn the initial verdict, in which direction, and how often are these corrections themselves correct?
  3. How many outputs from each model contain Markdown fences or surrounding prose, and how do the out-of-bound and success rates change if these are removed before rendering? This seems especially relevant to the 59.8% out-of-bound rate for Claude Opus 5.
  4. How sensitive are the model rankings and design pass rates to the choice of judge?
Reviewer_V7LUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ChartVL is a benchmark for LLM-generated SVG charts. It contains 360 tasks (bar, line, pie and glyph charts; 30 task groups per category, each with three levels of design requirements) built from 24 professional reports and public data, with instructions drafted and reviewed by LLMs and validated by humans. Models must return a complete HTML document whose primary graphic is an inline SVG. The evaluation first asks a VLM to write a layer tree that is uniquely anchored to the rendered SVG DOM, and then asks separate VLM judges to evaluate spatial planning, layer logic and inter-layer design coherence; each determinate judgment comes with a JavaScript assertion that is executed on the rendering snapshot to confirm or overturn the initial verdict. With gemini-3.8-flash as the judge, four models reach 83.1–99.2% task fulfillment but only 9.7–58.6% overall success, with spatial planning the weakest dimension.

Critical strengths

1, The evaluation harness is carefully engineered and transparent: strict unique anchoring with bidirectional coverage, explicit pass/fail/unresolved/invalid states, a sandboxed assertion interface with documented measurement limits, and retained evidence for every judgment.

2, The reporting is statistically careful: counts alongside rates, cluster-bootstrap confidence intervals over task groups, paired comparisons, and full outcome tables.

Critical weaknesses

1, The central claim — more reliable evaluation — is not validated. The paper motivates ChartVL by arguing that VLM judges miss or misjudge visual issues, yet it provides no evidence that its own framework judges more accurately. There is no comparison with human judgments, no comparison with a plain VLM judge, no ablation without layer reconstruction or without assertion execution, and no statistics on how often execution overturns initial verdicts or whether those corrections are right; a single judge model is used throughout. Since every reported number depends on this evaluator, its validity is currently unknown.

2, "Verifiable" is narrower than presented. The judge chooses which elements to check, the scope of each judgment and the numerical allowances in the assertion, and there is no universal threshold, execution then only tests the judge's own stated condition. Defects that the judge never examines cannot be caught, and task fulfillment, which gates every design metric, is a pure model judgment without any assertion.

3,The significance of the task is weakly motivated. The choice of SVG as the generation target is justified by a single sentence citing two works, whereas in practice charts are mostly produced through declarative libraries that handle layout automatically. Libraries such as D3 are allowed, but the paper does not report how many outputs rely on them, so it is unclear whether failures reflect layout ability or the construction route. The heavily specified, LLM-drafted design requirements make the task closer to fine-grained instruction following than to visualization design, and because levels replace rather than add requirements, the level analysis does not isolate design complexity.

4, Limited scope and insight. Only four API models are evaluated, each with a single sample at temperature 1.0, and the analysis is largely descriptive. For example, Claude Opus 5's 59.8% out-of-bound failure rate, versus 3.1% for GPT 5.6 Sol, is reported but not explained

Questions for authors

1,How well do the final verdicts agree with human annotators, how often does assertion execution overturn initial verdicts, and are those corrections correct? How do the results change without layer reconstruction or without assertion execution?

2,Why is SVG the right generation target, and what fraction of each model's outputs are hand-written SVG versus library-generated, with success rates for each?

Reviewer_SwhGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ChartVL, a benchmark and evaluation framework for SVG chart generation. It contains 360 tasks organized into 120 groups with three levels of design requirements. To evaluate generated charts, the framework reconstructs visual layers, maps them to rendered SVG elements, and asks a vision-language model to produce rubric-based judgments together with executable pass conditions. These conditions are executed against a rendering snapshot. The authors evaluate four generation models and report task fulfillment, overall success, and visual-design outcomes covering spatial planning, layer logic, and inter-layer coherence.

Critical strengths
  1. The paper addresses visual quality beyond content correctness. A chart can contain the requested values while remaining difficult to read because of clipping, overlap, or problematic layering. Separating these aspects makes the benchmark relevant to practical chart generation.

  2. The evaluation produces inspectable evidence. Reconstructed layers, selected elements, rationales, executable conditions, and execution results provide a useful record of what was evaluated. The paper also distinguishes structural verification failures and unresolved outcomes from confirmed design defects.

Critical weaknesses
  1. The reliability of the evaluator itself is not sufficiently established. Executing a condition can verify that the chosen condition holds on the captured rendering, but it does not establish that the model selected the right elements, expressed the intended visual criterion correctly, or identified all relevant defects. Task fulfillment is also judged by a model without an executable assertion. Since evaluator reliability is central to the paper’s contribution, comparisons against independent human judgments and existing evaluation methods are needed to interpret the reported results.

  2. The experimental setting limits the generality of the conclusions. The 360 tasks come from 120 task groups and cover four chart categories under a fixed static SVG setting. Each model produces only one response per task without feedback-based repair. This is a useful controlled setting, but it does not establish how the results transfer to diverse real-world charts or iterative workflows in which an agent renders, inspects, and revises its output.

Questions for authors
  1. Human validation of the evaluator: Have independent human judges assessed a stratified sample of generated charts and compared their judgments with the full ChartVL pipeline? Please report agreement and false-positive/false-negative rates for task fulfillment, individual visual defects, and overall success, together with annotator agreement.

  2. Contribution over simpler evaluators: How does the full pipeline compare with a direct VLM judge, existing chart-evaluation methods, and variants without layer reconstruction or executable assertions? Which component improves agreement with human judgments, and by how much?

  3. Errors in executable conditions: How often does an assertion execute correctly but test the wrong elements or an insufficient condition? How often does the evaluator omit a defect that human reviewers identify?

  4. Generality of the generation setting: Do the model rankings and the content-versus-visual-quality gap remain similar when models can inspect a rendered chart and perform a limited number of revisions? Can the authors also test tasks drawn from additional real-world sources or chart-generation workflows?

Reviewer_wDPbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CHARTVL, a benchmark comprising 360 SVG chart generation tasks, organized into 120 groups across three levels of design difficulty. It also proposes an evaluation framework in which a VLM first reconstructs the layer tree and anchors it to the rendered SVG DOM. Subsequently, the VLM evaluates the generated charts along three design dimensions and produces an executable Boolean assertion for each deterministic judgment to verify or refute its initial assessment. Experimental results show that although four models achieve high task completion rates, their overall pass rates remain considerably low, with spatial planning identified as the weakest dimension.

Critical strengths
  1. The research problem is meaningful, and the task decomposition is well-motivated. The paper distinguishes between content completion and visual design quality in its evaluation. The within-group paired comparisons further demonstrate that these two aspects can diverge: the success rate decreases by 20–28 percentage points from L1 to L3, while the task completion rate remains largely unchanged. This provides a clear and convincing empirical justification for the proposed distinction.
  2. The design of having evaluators formulate conditions before executing verification is novel. Unlike VisEval and ChartAnno, which provide reference measurements to evaluators during assessment, this work turns the evaluator's own judgments into auditable programs, thereby establishing a complete and traceable evidence chain.
  3. The evaluation pipeline demonstrates methodological rigor and conservative design choices. The authors deliberately avoid using position and color as disambiguation cues during anchoring. Cases with insufficient evidence are marked as unresolved rather than being counted as passes; empty responses are not considered successful; and different outcome statuses are reported separately (Tables 9 and 10).
  4. The statistical reporting is rigorous and appropriately conducted. The authors employ cluster bootstrap resampling at the task-group level, use shared resamples across models, and report confidence intervals for paired differences. They also explicitly acknowledge that the three difficulty levels do not constitute a nested hierarchy.
Critical weaknesses
  1. The validity of the central claim is insufficiently substantiated. The paper is motivated by the concern that VLM-based evaluators may overlook or misjudge defects. However, it provides neither an inter-annotator agreement study comparing VLM judgments with human annotations nor precision and recall evaluations on a curated set of known defects. The executable assertions only demonstrate that the conditions formulated by the VLM hold in the rendered output; they do not establish that the selected conditions are themselves appropriate. Consequently, subjectivity has not been eliminated but merely shifted from natural-language judgments to the thresholds and condition-selection decisions embedded in the code. Furthermore, the experimental evaluation is limited in scope, with too few models tested and insufficient coverage of diverse model families.
  2. The most critical evaluation statistics are missing. The paper states that when the execution results disagree with the initial judgments, the outcomes are recorded as "corrections." However, the paper does not report the correction rate, the direction of these corrections, or whether the corrections are actually accurate. These statistics constitute the most direct empirical evidence for the paper's central contribution of verifiable judgments, and their omission significantly weakens the supporting evidence. Furthermore, the paper lacks ablation studies to investigate the individual contributions of its key components. For example, how would the results change if layer-tree reconstruction, executable assertions, or the corresponding prompts were removed? How does the proposed framework compare with a baseline that relies solely on VLM-based scoring? In particular, Contribution 1 claims that layer-tree reconstruction provides structural and semantic evidence for VLM-based evaluation, yet no experimental results demonstrate that this component actually improves evaluation quality.
  3. The out-of-bounds failure rate may be confounded by protocol-related factors. Claude Opus 5 has 59.8% of otherwise acceptable charts classified as out of bounds, whereas GPT-5.6 Sol has a rate of only 3.1%. This substantial discrepancy is unexpectedly large and raises concerns about potential confounding factors in the evaluation protocol. I have two specific concerns: Appendix A.1 states that model responses are loaded verbatim, meaning that Markdown code fences and extraneous text are also rendered as page content. However, Appendix B indicates that the outer code fences are removed from the input provided to the reconstruction model. If a model tends to generate HTML enclosed in Markdown code fences, these additional elements could shift the rendered chart downward and affect the out-of-bounds assessment. The viewport is fixed at 960 × 540. If the generated HTML does not reset the browser's default body margin, an SVG with the same dimensions could overflow the viewport by approximately 8 pixels. I encourage the authors to report a breakdown of the underlying causes of out-of-bounds failures and conduct sensitivity analyses under alternative settings, such as using a larger viewport or removing Markdown code fences before rendering. In addition, including more visual experimental results and error analyses would make the findings more intuitive and improve the paper's readability. Currently, the paper provides only one visualization in Figure 1, which is insufficient to effectively illustrate the proposed framework and its experimental findings. 4.The model evaluation is limited in scope and lacks sufficient reproducibility details. The paper evaluates only four models and does not specify their exact model versions or the dates on which they were accessed. Moreover, the introduction refers to Claude Opus 5.5 and GPT-6 Astra, whereas the actual experiments appear to involve different model versions. The authors should clarify this discrepancy and ensure consistency between the model versions discussed in the introduction and those used in the experiments. Providing precise model identifiers and access dates is also necessary to improve the reproducibility of the evaluation. The experimental analysis lacks sufficient granularity. The paper does not provide a breakdown of results by chart type, such as bar charts, line charts, pie charts, and glyph charts. Furthermore, it lacks qualitative analyses of representative failure cases, making it difficult to understand the specific limitations of the evaluated models across different chart types.
Questions for authors
  1. What proportion of initial judgments are overturned after executing the corresponding assertions? Which direction of correction is more frequent, from pass to fail or from fail to pass? Furthermore, among a manually reviewed sample, what proportion of these corrections are verified to be correct?
  2. On a manually annotated subset, how well do CHARTVL's final judgments agree with human annotations? How does this agreement compare with that achieved by a baseline relying solely on direct VLM scoring?
  3. For the out-of-bounds failures observed in Claude Opus 5 and Qwen models, what proportion can be attributed to page-level factors, such as Markdown code fences and default body margins, rather than genuine chart-generation defects?

If the authors address the following three issues in their rebuttal, I would consider raising my overall assessment to slightly above the acceptance threshold: (1) providing a human annotation agreement study; (2) reporting statistics on the rate and direction of judgment corrections; and (3) conducting at least one ablation study comparing the framework with and without executable assertions or layer-tree reconstruction. In addition, a sensitivity analysis of out-of-bounds failures would further strengthen the rebuttal and positively influence my assessment.

Reviewer_RPR9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

CHARTVL is a benchmark and evaluation framework for SVG chart generation. Given an instruction and a data table, a model generates an HTML document containing an SVG chart. The benchmark evaluates both task fulfillment, including chart type, data values, and required content, and visual design, including spatial planning, layer logic, and inter-layer design coherence. The framework reconstructs visual layers from the generated code and image, uses VLMs to produce rubric-based judgments, and verifies these judgments with executable assertions. Four models are evaluated, with task fulfillment ranging from 83.1% to 99.2% and full evaluation success ranging from 9.7% to 58.6%.

Critical strengths
  1. The paper studies visual design quality in SVG chart generation in addition to content correctness. This is cool.

  2. The layer reconstruction and executable assertion framework provides a structured way to inspect individual evaluation decisions.

Critical weaknesses
  1. The benchmark lacks sufficient diversity and scale, with only 120 task groups, limited source charts, and repeated data tables across difficulty levels.

  2. The evaluation includes too few models, especially chart-specific systems and open-source VLMs, making the conclusions about current chart-generation capabilities incomplete.

Questions for authors
  1. How accurately do the reconstructed layers and executable assertions identify the correct chart elements and design requirements when checked by human experts?

  2. How does CHARTVL compare with existing structural metrics, scene-graph methods, or VLM judges without layer reconstruction?

  3. Can the authors evaluate the framework on an external set of SVG charts or include more generation models and chart styles?

  4. The distinction among task fulfillment, structural verification, design-dimension pass rates, and overall success could be made clearer in the main results table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.