ChartDesign: Towards LLM Designer of Data Visualization

Charts require many design choices, and the same data table can support multiple reasonable visualizations depending on the intended task and audience. We introduce ChartDesign, a framework that post-trains compact language models to predict a renderer-neutral chart-design specification from tabular data. We construct 2,118 human-authored chart examples from PewResearch and CharXiV, reserve 200 manually verified and corrected charts for evaluation, and audit 300 training examples to characterize annotation quality. Post-training substantially improves reference-match accuracy over matched zero-shot, rule-based, and direct-copy retrieval baselines, with Phi-3 full fine-tuning reaching 82.0% on the combined human-verified evaluation set. Human evaluation further finds high visual plausibility and frequent preference for generated charts, while showing that reference agreement and design quality are distinct. These results support learning reusable human-authored chart-design preferences without tying the output representation to a single rendering library.