acceptodds
Under review as a conference paper at ICLR 2027

ChartDesign: Towards LLM Designer of Data Visualization

Abstract

Charts require many design choices, and the same data table can support multiple reasonable visualizations depending on the intended task and audience. We introduce ChartDesign, a framework that post-trains compact language models to predict a renderer-neutral chart-design specification from tabular data. We construct 2,118 human-authored chart examples from PewResearch and CharXiV, reserve 200 manually verified and corrected charts for evaluation, and audit 300 training examples to characterize annotation quality. Post-training substantially improves reference-match accuracy over matched zero-shot, rule-based, and direct-copy retrieval baselines, with Phi-3 full fine-tuning reaching 82.0% on the combined human-verified evaluation set. Human evaluation further finds high visual plausibility and frequent preference for generated charts, while showing that reference agreement and design quality are distinct. These results support learning reusable human-authored chart-design preferences without tying the output representation to a single rendering library.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data visualization, chart design, large language models, vision-language models, structured generation, JSON representations, parameter-efficient fine-tuning, LoRA, human-centered AI

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_urZ6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ChartDesign, a framework that maps tabular data to a structured, renderer-neutral chart-design specification. The authors construct a corpus of 2118 human-authored charts from PewResearch and CharXiV, using a vision-language model to extract data tables and design annotations. They manually verify 200 evaluation examples and audit 300 training examples. Compact language models are adapted through LoRA or full fine-tuning to predict design attributes, including chart type, orientation, axes, legends, and mark layout. The best reported configuration achieves 82% macro-averaged attribute-level reference-match accuracy. A human study additionally evaluates source-design fidelity, visualization quality, and pairwise preference. The results demonstrate that task-specific fine-tuning improves agreement with source-chart designs relative to the evaluated baselines.

Critical strengths
  1. The renderer-neutral design specification separates design decisions from plotting syntax, making the output inspectable and potentially reusable across libraries. Matched-backbone experiments show clear improvements in reference agreement after task-specific fine-tuning.

  2. The manually verified evaluation set and training-annotation audit provide useful checks on automatically generated supervision. The paper also appropriately distinguishes reference agreement from visualization quality and complements automatic metrics with human evaluation.

Critical weaknesses
  1. The paper defines design attributes but does not sufficiently establish what constitutes good design or how reference imitation improves design quality. Although the schema includes family-specific fields, its adequacy across chart types is not systematically evaluated, and the human study does not establish a clear quality advantage over the original charts.

  2. The corpus covers only two sources and lacks chart-type counts, rare-design frequencies, and template-overlap analysis, leaving its representativeness and the model’s generalization unclear. Comparisons also omit strong general-purpose models, including proprietary models, while the retrieval baseline copies specifications without adapting data-dependent fields, limiting the evidence for the method’s competitive advantage.

Questions for authors

As mentioned in the Weaknesses session.

Reviewer_N1U2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChartDesign, which post-trains compact language models to predict renderer-neutral chart-design JSON from CSV tables. It constructs 2,118 chart examples, manually verifies 200 evaluation examples, and audits 300 training examples. Matched-backbone experiments show substantial reference-match gains, reaching 82.0% for Phi-3 full fine-tuning, while a blinded human study evaluates rendered-chart fidelity, quality, and preference. Overall, I lean towards rejection because the novelty over existing specification-generation work is limited and the claimed design preferences are learned and evaluated against references of limited design quality.

Critical strengths
  • The paper clearly distinguishes agreement with a source design from visualization quality, appropriately recognizing that multiple designs can be reasonable.
  • Manually corrected evaluation references and a separate training-annotation audit provide useful checks on automatically generated supervision.
Critical weaknesses
  1. Limited novelty. My main concern is that the contribution over prior work remains limited. Data2Vis (Dibia & Demiralp, 2019) already generates declarative specifications from tabular data, and Vega-Lite is already a high-level declarative specification. Sec. 2 distinguishes ChartDesign mainly by its output representation, but the paper neither compares its schema with Vega-Lite nor includes any prior specification-generation baseline. The training method is standard fine-tuning with inverse-frequency sampling, and renderer neutrality is shown only qualitatively (Fig. 2, Appendix A.7). The contribution therefore reduces largely to fine-tuning compact models on a newly annotated corpus.

  2. Low-quality design examples. The paper claims to learn "human-authored chart-design preferences", but the design JSONs in the training corpus are inferred from chart images by Phi-3.5-Vision Instruct, and the source charts were not selected for design quality. Several references shown in the paper, such as the CharXiV examples in Fig. 7 (Examples 4–5) and the sample chart in Fig. 2, resemble default-styled scientific plots with dense series and overlapping marks rather than deliberately designed visualizations. Since both reference matching and the human study compare outputs against such references without an independent quality anchor, high reference-match accuracy and human ratings close to the originals (Table 4) do not show that the model produces good designs. Moreover, the annotators are not described as visualization designers: five reported experience with visualization, data science, or machine learning, and five had general technical backgrounds (Sec. 4.4), and ratings are not reported separately by expertise. This makes me concerned that the paper's claim of learning design preferences rests on references of limited design quality.

Questions for authors
  1. How is the headline score in Table 1 computed from attribute-level accuracies? Fig. 6 suggests averages of roughly 0.48 for the Phi-3 base model and 0.75 for the fully fine-tuned model, compared with 32.0% and 82.0% in Table 1. Could the authors list all scored attributes and explain this difference?

  2. Tables 1–2 imply zero-shot CharXiV accuracies of about 10–11%. How often do the zero-shot models produce unparsable or schema-violating JSON, and do the zero-shot prompts include the schema?

  3. Table 13 reports latency for three-shot prompting, but no three-shot results appear in the paper. Were these experiments run, and how do they compare with the adapted models?

  4. Which design decisions does the proposed schema express that cannot be expressed in Vega-Lite?

  5. Phi-3-mini-4k-instruct is usually reported as 3.8B parameters with a 4k context window, whereas the paper states 4.2B and a 14k maximum sequence length (Sec. 3.4, Appendix A.2). Which model variant was used?

  6. In the human study, were the original charts the published images or re-rendered from the reference specifications, and which checkpoint generated the ChartDesign outputs?

  7. How does the model predict the "Title text" attribute in Fig. 6 when its input is only a CSV table?

Reviewer_uZgbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a framework, ChartDesign, that post-trains compact langue models to create a chart-design specification for tabular data. They use two datasets, namely PewResearch and CharXiV, for their experiments.

In their post-training pipeline, they use standard autoregressive cross-entropy to map an instruction and CSV table to a JSON design. In order to make the underrepresented attributes to have more exposure, a sampling approach that is similar to TFIDF is applied, while the adaptation uses either LoRA or full fine-tuning.

They claim this training method has improved the reference-match accuracy over 3 other methods; matched zero-shot, rule-based, and direct-copy retrieval baselines, reaching 82% under fine-tuning with Phi-3. Human evaluation is included. They report high visual plausibility and preference for generated charts from ChartDesign. It also shows reference agreement and design quality.

Critical strengths

The authors have come up with a method, ChartDesign, that generates multiple visualisation according to the downstream tasks. The authors use two different datasets, PewResearch and CharXiV, to demonstrate the effectiveness of their proposed methods. The authors have also employed human evaluation to show the subjects' preference for the generated charts.

Critical weaknesses

[W1] The authors kept mentioning "combined human-verified evaluation set”, but I do not think they have clearly defined what was combined with the human-verified evaluation set. [W2] It is always good to demonstrate the method is not specific to a particular dataset, and the authors have stated the usage of PewResearch and CharXiV. Although the authors briefly describe Pew is presentation-oriented for public, while CharXiV contains scientific figures. Are they any differences between the characteristics of charts from these two datasets? If not, the experimental results make no difference, but from different domains? If yes, what are the specifics that pertain in one domain and not in another. A figure of comparing the two charts from these two datasets will help understand your results. [W3] I am aware that Phi-3.5-Vision Instruct was used without justification, as there other available models. I think Phi-3.5-Vision Instruct is not the best to extract numbers, when numbers are pervasive in a chart. [W4] There are not descriptive statistics of the two datasets, and the complexity of the charts in order to understand the strength and weakness of the proposed method. [W5] The accuracy of ChartXiV is consistently lower than Pew, what is your suggested interpretation of this result? [W6] Figure 7 has shown five charts and authors generally show the best charts. I would like to know what charts fail because that is where we can improve the method. For these 5 charts, do they come from Pew or CharXiV? [W7] In Table 4, it is showed that human subjects prefer ChartDesign in the pairwise preference, did you have a feedback from the human subjects of their preference on their selection, as this also helps improve the method? Is it always a particular subject? Or is it a particular type of charts? Or is it mostly from a particular dataset?

Questions for authors

Please refer to the above weakness section as well as the following

  • Table 1 and Table 2 are very similar in content. Why did you separate into two tables?
Reviewer_hxduOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChartDesign, a framework for predicting chart design directly from json data. Instead of generating library-specific code, the model outputs a hierarchical JSON specification with chart family, text, axes, legends, and other layout. The authors used a vision-language model to construct a dataset including 2,118 charts and design specifications derived from PewResearch and CharXiV. They then post-train small language models and report substantial gains over zero-shot, rule-based, and nearest-neighbor baselines. A human study further compares reference agreement with visualization quality.

Critical strengths

The paper has several strengths. It explicitly acknowledges that one single csv table can be matched to multiple visualizations. The human verified test set and the separate training annotation audit are also good decisions. Besides aggregated scores, the authors also reported attribute-level analysis, a scaling experiment, rendered examples, and a human evaluation. The paper is clear and stated its limitations transparently.

Critical weaknesses

My main concern is the main contribution/ framing of the paper, that formulating chart design as tabular data. While the motivation is good, I think the paper is solving a different problem. I like the motivation that same table can support multiple good designs depending on the analyst's objective, audience, and context. But ChartDesign receives a single table. None of the design principles or intentions are given to the model or varied. I would find the claim much stronger if the input charts are intent conditioned or design principle conditioned, the dataset contains multiple human-authored acceptable designs for the same table, or the test set in evaluation doesn’t come from Pewreserach or Charxiv. The human evaluation has the same issue. Annotators are not given the original designer’s communication intent or the target audience. They rather annotate on perceived quality and appropriateness.

I am also not fully convinced that the main empirical gain comes from learning general chart-design principles, beyond recurring source or template regularities. The learning method itself is comparatively standard autoregressive post-training with LoRA or full fine-tuning, so I would like to see more convincing task formulation, dataset, and empirical evidence.

While training and test examples are held out, they still both come from PewResearch and CharXiV. Repeated publisher/community conventions, common figure templates, and typography/layout patterns could therefore help predict the reference design. The authors themselves acknowledge that the scaling experiment does not rule out “source-specific regularities or template similarity.”

The baselines also seem relatively weak. The nearest-neighbor system copies design JSON unchanged. The rule based baseline didn’t use some of the more recent baselines.

Overall, I think this is a thoughtful and transparent paper, and I like the distinction between reference matching and actual visualization quality. While I do see convincing evidence that post-training can reproduce chart-design patterns from these two corpora better than the controls, I’m not sure the system is a general chart designer from tabular data.

Questions for authors

Minor question: There is an tension that I would like the author to clarify: the original charts have slightly higher mean scores on every reported absolute quality criterion, while ChartDesign wins the direct pairwise preference 65% versus 25%. If the direct comparison is not measuring the same thing as the scores do, what might the evaluators comparing instead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.