- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ChartRevise, a dataset and evaluation protocol for program-grounded chart editing. It evaluates whether models complete explicit instructions, make necessary coupled updates, and avoid gratuitous changes, covering both under-editing and over-editing. The dataset contains 92,438 examples across 20 chart types, three plotting libraries, and 344 edit types. Its reference-free evaluation combines LLM-based code judging with execution checks. Experiments across five models and four external benchmarks show stronger requirement completion and exact-edit performance, while improvements in preventing unrelated edits are less consistent.
Critical strengths
1.The dataset is constructed based on chart grammar, which is an interesting idea. It is diverse in chart types, edit types, and plotting libraries, with automatic audits to improve data quality.
2.The evaluation considers both under-editing and over-editing, including missed coupled updates and unnecessary changes. This is very interesting and diagnostically useful.
Critical weaknesses
1.The LLM judge only checks code without seeing the rendered charts. This can miss visual problems such as overlapping elements, even when the code looks correct.
2.Fine-tuning reduces unnecessary edits in only half of the comparisons. The results are more convincing for reducing under-editing than over-editing.
3.Human validation finds only 69.5% of compound examples fully correct, which raises concerns about using them to teach exact editing.
Questions for authors
1.How much would the exact-edit results change if the judge also checked the rendered charts?
2.Why does fine-tuning increase unnecessary edits in some settings? Are there common types of over-editing?
3.What are the main errors in the compound training examples, and could these errors explain some of the remaining failures after fine-tuning?
4 comments
Response (Part 2)
W3 & Q3. Compound-Data Noise and Its Impact on Fine-Tuning
R: The compound records still provide high-quality editing supervision, with 95.6% of their individual requirements correctly satisfied. The lower task-level correctness (69.5%) reflects our strict all-or-nothing criterion over 2–12 requirements. At 96% per-requirement correctness, 79.3% of records are expected to satisfy every requirement under an independent-error assumption (measured: 81.5%), even before preservation is considered.
The main remaining errors are gratuitous changes, typically code left over from earlier attempts or reasoning comments (79% of defective compound records). The human audit also applies stricter preservation judgments, counting unrequested code changes even when they do not affect the rendered chart.
Impact on fine-tuning. These errors may contribute to some remaining failures, but are unlikely to be their primary cause, for three reasons:
- Limited training noise. Clear editing defects in compound records account for only 0.75% of the training data.
- Persistent completion failures. Among tasks still incomplete after fine-tuning, 61–84% on ChartEdit and 87–99.8% on Chart2Code-L2 were also incomplete for the corresponding base models. Thus, most remaining completion failures reflect limitations already present in the base models rather than failures newly introduced by fine-tuning.
- Model capability limitations. Preservation and executability remain challenging for weaker models on longer requests. As discussed in W2 & Q2, this is consistent with a completion–preservation trade-off.
Despite the remaining noise, fine-tuning improves exact-edit rates for all five models on ChartEdit (Table 3), demonstrating that the editing supervision provides useful learning signals.
Follow-up
Dear reviewer,
We sincerely thank you again for the thoughtful and constructive feedback. We have carefully addressed the concerns through additional experiments, analyses, and clarifications. We believe your suggestions have helped us strengthen the paper and better communicate its contributions.
We truly appreciate the time and effort you have dedicated to reviewing our work. And, we would be happy to address any further concerns.
Best Regards
Thank you for the clarifications and the additional experiments. I’m keeping my original rating. Wish you good luck!
Response (Part 1)
We sincerely thank the reviewer for the thoughtful and constructive feedback and for recognizing the contributions of our work. We appreciate the insightful suggestions and have conducted additional analyses to address the concerns, as detailed below.
W1 & Q1. Limitations of Code-Only Evaluation
R: We thank the reviewer for highlighting this limitation. Our code-only protocol supports fine-grained verification of explicit requirements and coupled updates, including in non-executable programs and for changes that are difficult to assess visually. We agree that rendered charts can reveal visual defects that code inspection may miss.
Robustness of exact-edit results. To directly assess the impact of visual evidence, we re-evaluated outputs from all five model families using both code and rendered charts. On ChartEdit, the fine-tuned exact-edit rates change by only 1–3 percentage points for four of five models. Importantly, fine-tuning continues to improve exact-edit rates for all five models on both ChartEdit and Chart2Code-L2 under code-and-image evaluation. Thus, the direction of the main fine-tuning gains is preserved when visual evidence is included.
Comparison of judge strategies. We evaluated all three strategies on reference programs from four benchmarks:
Visual-only judging identifies fewer reference edits, particularly on ChartM3, where subtle or non-visible changes can be difficult to verify. Incorporating code evidence substantially increases reference AppliedRate, supporting the importance of program-level inspection.
Scope of visual issues. Our protocol evaluates visual constraints when they are part of the editing requirements. For example, 57 of 1,405 ChartEdit tasks explicitly require avoiding overlap. Nevertheless, implicit layout defects may remain undetected by code-only judging. We will clarify this limitation and distinguish exact-edit verification from general visual-quality assessment, which can be evaluated using complementary metrics such as ChartScore from ChartEdit.
W2 & Q2. Inconsistent Improvements in Over-Editing
R: We agree that fine-tuning improves requirement completion more consistently than it reduces gratuitous changes. However, these regressions do not necessarily indicate unreliable supervision. Our analysis suggests a completion–preservation trade-off: fine-tuning encourages models to implement more requested operations, which can increase the opportunities for unintended changes, particularly when models have limited capacity to manage complex requests.
This trade-off is particularly evident in two settings:
Common over-editing errors. Our additional analysis identifies modifications to text or labels (31%) and changes to colors or styles (28%) as the most frequent categories of gratuitous edits.
These findings help explain why improving requirement completion does not necessarily improve preservation. They also identify complex multi-operation requests as an important remaining challenge for exact chart editing.