- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ChartCraft, a synthetic dataset of approximately 1.92 million samples covering six chart-image generation and editing tasks, together with adapted ChartCrafter models and the CraftEval benchmark. Evaluation shows substantial gains over the original Qwen backbones, supplemented by human preferences, an external judge, extraction audits, and split-integrity checks. The task-organized supervision is useful, but the evidence establishes reference-conditioned adaptation more clearly than general design quality or reliable preservation of chart semantics. Overall, I lean towards rejection because the contribution reduces largely to distilling a proprietary teacher under teacher-aligned evaluation, and the data fidelity of the generated charts remains low.
Critical strengths
- The dataset clearly organizes six generation and editing tasks, with explicit source–target pairings and detailed prompt-construction procedures.
- ChartCrafter achieves substantial improvements over its original backbones, and the paper appropriately distinguishes these adaptation gains from a controlled data-only comparison.
Critical weaknesses
-
Teacher distillation under teacher-aligned evaluation. My main concern is that the approach essentially distills a proprietary teacher, while the evaluation shares the teacher's model family. Training targets are produced by Nano Banana Pro, instructions and consistency filtering by Gemini-3-Pro, and CraftEval is judged by Gemini-3-Pro, which also supplies the RL reward. Sec. 2.3 acknowledges that the results demonstrate task adaptation rather than an advantage over the teacher, and Nano Banana Pro remains the strongest system. Moreover, AQA explicitly frames evaluation as design replication, while CSA rewards the reference's exact thematic appearance, so an output that follows the instruction through a different, equally valid design may receive a lower score. The improvements therefore primarily establish reference-conditioned alignment with the teacher's outputs, which limits the contribution beyond distillation into an open backbone.
-
Low data fidelity. The absolute data fidelity of the generated charts remains low. ChartCrafter reaches DSI scores of only 54.9 on T2I and 51.2 on Edit, with Text Preservation at 37.8 and 30.1. For charts, incorrect values or labels compromise the visualization regardless of its aesthetic quality. This is difficult to reconcile with Table 12, which reports numerical or label corruption in only 15 of 200 T2I samples without identifying the evaluated model. The training targets compound this concern: automated filtering checks consistency among generated artifacts, and the independent audit (Appendix A.6) checks extracted code against source images, but neither quantifies the fidelity of the final training targets. Some conversions are also semantically inappropriate. For example, Fig. 16 requests stacking Min, Q1, Median, Q3, and Max, which are not additive components. Prior pictorial-chart generation work, such as ChArtist (Xiao et al., CVPR 2026) and Semantic-Structural Alignment for Generative Pictorial Charts (2026), explicitly conditions generation on chart structure to preserve data encoding and evaluates data accuracy directly, whereas ChartCraft relies on unconstrained pixel-space generation and neither discusses nor compares with these approaches.
Questions for authors
- Which model's outputs are summarized in Table 12, and how should its 15 of 200 T2I corruption cases be reconciled with the Text Preservation score of 37.8 in Table 3?
- The SFT stage uses about 50,000 examples from a resource of 1.92 million. How was this subset selected, and is there any evidence that training on larger portions of ChartCraft improves performance?
- How many human evaluators took part in Appendices A.3–A.5, what was their expertise, and how many judgments were collected per example? How were tied scores handled, and how were the sub-item ordered-pair counts in Table 6 constructed?
- Did the authors measure numerical accuracy directly, for example by extracting values from generated charts and comparing them with the source data, independently of the reference-based judge?
We thank the reviewer for recognizing the value of ChartCraft’s paired supervision and the improvements over the corresponding backbones. We provide additional human evaluation results and clarify the scope of our evidence below.
Human evaluation provides additional evidence about the quality of the curated training targets. Conventional charts received an average aesthetic rating of 2.4/5, compared with 3.9/5 for ChartCraft’s synthetic training charts. The ChartCraft assessment involved 10 human evaluators and over 500 training samples.
These results support the aesthetic quality of our curated supervision. We agree, however, that this comparison does not replace a matched chart-domain training control. Our comparisons between the original backbones and the fine-tuned models establish the combined benefit of ChartCraft supervision, but do not separately quantify the contributions of domain adaptation and aesthetic/contextual curation. We will clarify this distinction and avoid attributing the entire improvement to individual construction stages.
We conducted a user study involving more than 20 participants, measuring information-extraction accuracy and efficiency. Compared with conventional charts, participants exhibited an 11% reduction in information-extraction error rate and a 13% reduction in task completion time.
These behavioral measurements provide evidence beyond aesthetic preference: they support improved information-extraction accuracy and efficiency in the evaluated tasks. We will include the study protocol, comparison baseline, and results, and limit broader claims about comprehension and usability to the tasks and conditions actually evaluated.
We checked the audit criteria and confirmed a difference in evaluation scope. The numerical/label-corruption category in Table 12 counts incorrect numerical values or labels, but does not include missing text in this category. In contrast, Text Preservation evaluates both the completeness and correctness of titles, axis labels, and numerical values. It uses per-output scores of 0/1/2, subsequently aggregated and rescaled to 0–100.
An output can therefore receive a low Text Preservation score because required textual elements are missing, while not being counted as numerical/label corruption. Accordingly, the counts of 15/200 and 31/400 describe this specific failure category; they do not establish complete value-and-label correctness for the remaining outputs. Likewise, the Text Preservation scores of 37.8 and 30.1 are graded scores, rather than percentages of fully correct charts.
We agree that omissions are also fidelity failures. We will make the distinction between incorrect and missing content explicit, identify the checkpoints and output sets associated with each analysis, and clarify that Table 12’s corruption counts should not be interpreted as an overall fidelity success rate. The low Text Preservation scores remain an important limitation, even when explicit numerical/label corruption is less frequent.
We conducted a direct human comparison between SFT and RL on 150 editing examples, with 10 evaluators per example. Model identities were hidden, and left/right placement was randomized. After aggregating judgments within each example, 46% favored RL, 23% favored SFT, and 31% were rated “both good.” This directly supports an incremental preference benefit from RL over SFT.
Separately, the independent external judge in Table 7 corroborates this ordering: Qwen3.5-27B scores increase from 55.4 for SFT to 62.1 for RL. This judge was not used for reward optimization.
We also agree that preference and aesthetic improvements must be considered alongside fidelity. Table 4 reports increases from 47.5 to 51.2 in DSI and from 25.4 to 30.1 in Text Preservation after RL. Thus, independent evidence of increased preference is accompanied by improvements in CraftEval’s fidelity dimensions. These scores do not establish complete numerical correctness. We will present the preference and fidelity findings together and explicitly acknowledge the remaining text-preservation limitations.