- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to make multimodal large models generate executable chart editing code from chart images and natural language instructions, and proposes a two-stage training framework. In the first stage, structured chart descriptions are used to assist in generating reasoning traces and code, which, after filtering, are used for supervised fine-tuning (SFT); in the second stage, it adopts online privileged self-distillation (OPSD): a frozen teacher model can access privileged information during training, while the student receives only the image and the editing instruction. The authors evaluate the method on ChartEdit and ChartMIMIC, and compare the effects of different forms of privileged information, the degree of alignment with the target code, and SFT initialization. The paper reports that combining SFT with OPSD improves multiple task metrics; compared with GRPO, the optimization phase of OPSD is faster and does not require a separate large reward evaluation model.
Critical strengths
- The training pipeline design is clear. The paper chains data construction, SFT, and online self-distillation into a complete pipeline, attempting to reduce dependence on expensive teacher models and online reward evaluators, which has some practical value.
- It attends to the important risk of privileged information leakage. The student sees only the image and the instruction during training and inference, while the teacher can use additional descriptions or code. The paper compares different privileged information and reports that OPSD without SFT leads to performance degradation and noticeable leakage, whereas SFT initialization significantly reduces the leakage rate.
- It analyzes the content of the privileged information and its relationship with the target. The paper compares reference chart descriptions, reference code, target code, and their combinations, and finds that target code — privileged information more aligned with the expected output — is generally more effective; additionally incorporating reference context does not necessarily bring gains. These analyses provide clues for understanding when self-distillation is effective.
- The experiments cover multiple chart tasks and baselines. The paper compares against general-purpose multimodal models and chart-specific models on ChartEdit and ChartMIMIC's reproduction and customized editing tasks; it also reports ablations on data filtering, distillation targets, and inference efficiency.
Critical weaknesses
- Target code as privileged information may directly provide the answer. The teacher can see the target program, while the student sees only the chart and the instruction. Even if the authors report a reduced leakage rate, this setting may still mainly be transferring the target code's behavior to the student, without necessarily improving its ability to independently infer edits from visual evidence.
- The large-scale SFT data depends on automatic generation and automatic filtering. The paper uses a large-scale synthetic corpus constructed from charts and programs, and filters samples through model-generated reasoning and quality checks. If the generator or the evaluator has systematic biases, erroneous demonstrations may still enter the training set; the main text provides limited description of independent human review.
- The evaluation scope is not yet sufficient to demonstrate robustness in real-world scenarios. The experiments are concentrated on ChartEdit and ChartMIMIC, and the training data comes from the charts and programs in the paper. It remains unclear how the method performs on real charts with complex styles, unclear annotations, data points that are difficult to read from the image, or ambiguous editing instructions.
- The cost advantage is mainly reflected in the OPSD stage, and the cost of the complete pipeline is not yet clear enough. The paper reports that SFT requires about 266k samples and about 32 hours of training on four A800 GPUs; the time comparison between OPSD and GRPO is limited to a finite number of update steps. If data generation, quality checks, and two-stage training are included, the overall cost advantage needs further quantification.
Questions for authors
- Regarding target code as privileged information: How do the authors distinguish between "the student learns to perform edits based on the chart and the instruction" and "the student indirectly memorizes answers from the target code through distillation"? What is the specific definition of the leakage rate, and has it been tested on information that is present in the target code but cannot be reliably recovered from the image?
- Regarding synthetic data quality: Which models or checkers are used for the automatic filtering of reasoning traces and code? Is there independent human sampling and review, and can the authors report error types, false-filter rates, and the impact of these errors on final performance?
- Regarding real-world generalization: Can the method be evaluated on charts from different sources with more diverse styles, or on real user editing tasks? In particular, does it maintain editing quality and avoid altering unrelated attributes for data that is visually hard to read accurately, unseen chart types, and ambiguous instructions?
- Regarding the complete cost: Can the authors report the end-to-end resources and cost from synthetic data generation, quality filtering, and SFT to OPSD, and compare with GRPO or other methods under the same data volume, training budget, and hardware conditions? This would allow a more accurate judgment of whether the overall training cost is actually reduced.
6 comments
Response to Reviewer jBWp (Part 2/4)
W2 & Q2: Automatic filtering and independent human review
Thank you for asking about filtering reliability. We clarify the automatic checks and add an independent human review. Automatic and human decisions agree on 91.0% of code assessments and 88.0% of trace assessments.
-
Automatic checks. The SFT construction stage uses Qwen3.7-Flash for description generation, reasoning/code synthesis, and quality assessment. We first execute and render the generated code. Code and image checks assess edit fulfillment, unintended changes, and readability; trace checks assess visual grounding, technical correctness, and consistency with the generated code.
-
Human review. We randomly sample and manually reassess 300 code-level decisions and 100 trace-level decisions, covering both automatically accepted and rejected samples. The tables below report the decisions and summarize agreement and errors in both directions.
Table 4. Human review of automatic filtering decisions.
Assessment level Automatic decision Human pass Human fail Total Code level Pass 174 8 182 Code level Fail 19 99 118 Trace level Pass 55 5 60 Trace level Fail 7 33 40 -
Results. Among automatically accepted samples, the human-identified failure rates are 4.40% for code and 8.33% for traces. Valid samples also occur among automatic rejections (16.10% and 17.50%, respectively).
Table 5. Agreement and error proportions between automatic filtering and human review.
Human-audit measure Code level Trace level Automatic–human agreement 273/300 (91.0%) 88/100 (88.0%) Human-failed samples among automatic passes 8/182 (4.40%) 5/60 (8.33%) Human-passed samples among automatic failures 19/118 (16.10%) 7/40 (17.50%)
Response to Reviewer jBWp (Part 3/4)
W3 & Q3: Evaluating Cross-Source Generalization in Chart Editing
Thank you for suggesting evaluation beyond the original benchmarks. We conducted an exploratory evaluation of 336 editing tasks using images from five external sources. OPSD achieves the highest rendering success and average editing score among the four evaluated models.
-
Test dataset composition
-
Size and sources. The test subset contains 336 manually designed editing tasks on 230 images from ChartMuseum [1], ChartQAPro [2], VisTalk [3], VCG-Bench [4], and Geometry3K [5].
-
Visual coverage. Images include statistical charts, infographics, structural diagrams, and geometric figures. Challenge groups cover dense or small details, different diagram structures, and bounded stylistic underspecification; these groups can overlap.
-
Task scope. For stylistically underspecified requests, the desired effect and content to preserve are stated, while the exact styling is left open. This is an exploratory, post hoc selected subset.
-
-
Evaluation method
-
Model input. Each model receives only the reference image and editing instruction.
-
Judge input. GPT-6 Luna receives the original image, instruction, and rendered result, without the generating model's identity.
-
Scoring. Edit fulfillment, data/structure preservation, non-target appearance preservation, and readability are each scored out of 25, totaling 100. All models are evaluated on the same 336 tasks; failed renders receive zero in every dimension.
-
-
Results and analysis
-
Overall performance and content preservation.
Table 6. Editing performance on the exploratory cross-source subset.
Model Rendering success Total /100 Qwen3.5-9B base 69.64% 40.50 Qwen3.5-9B + SFT 82.44% 55.78 Qwen3.5-9B + SFT + OPSD 88.69% 58.47 Qwen3.5-27B 73.51% 54.18 -
Relative to SFT, OPSD improves rendering success by 6.25 percentage points and the all-task score by 2.68 points, calculated before rounding. Its all-task data/structure preservation score increases from 12.68 to 13.24, and non-target appearance preservation from 10.92 to 11.44, each out of 25.
-
Performance on challenging inputs.
Table 7. Performance across challenge groups in the exploratory subset.
Challenge group Tasks SFT /100 SFT + OPSD /100 Dense or small visual details 68 38.03 45.19 Structural diagrams 114 56.79 57.12 Geometric diagrams 54 63.33 72.24 Bounded stylistic underspecification 19 31.42 33.00 The largest subgroup gains are on geometric diagrams (+8.91 points) and dense or small visual details (+7.16 points). Structural diagrams and stylistically underspecified requests show smaller gains.
-
Reference
[1] Tang, L., Kim, G., Zhao, X., et al. (2025). ChartMuseum: Testing visual reasoning capabilities of large vision-language models. Advances in Neural Information Processing Systems (NeurIPS 2025), Datasets and Benchmarks Track.
[2] Masry, A., Islam, M. S., Ahmed, M., et al. (2025). ChartQAPro: A more diverse and challenging benchmark for chart question answering. Findings of the Association for Computational Linguistics: ACL 2025, 19123-19151.
[3] Wang, Y., Hou, Z., Shen, L., et al. (2023). Towards natural language-based visualization authoring. IEEE Transactions on Visualization and Computer Graphics, 29(1), 1222-1232.
[4] Su, X., Dong, P., Tang, Z., et al. (2026). VCG-Bench: Towards a unified visual-centric benchmark for structured generation and editing. arXiv preprint arXiv:2605.15677.
[5] Lu, P., Gong, R., Jiang, S., et al. (2021). Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 6774-6786.
Response to Reviewer jBWp (Part 1/4)
We sincerely thank the reviewer for the careful and constructive review.
W1 & Q1: Definition and Evaluation of Privileged-Information Leakage
Thank you for asking us to clarify PI leakage and examine hidden implementation details. SFT initialization substantially reduces measured leakage in both evaluations below.
Definition. The frozen teacher is conditioned on target code; the student receives only the image and editing instruction. We count output claims of access to unavailable descriptions or programs, or unsupported assertions about their contents, as PI leakage. Agreement with target code, visually grounded estimates, and implementation choices alone do not count. For example, choosing a random seed differs from claiming that the original program used it.
Leakage evaluation. On the held-out test set, we use characteristic expressions from manual inspection to screen candidates, manually review every flagged output, and inspect sampled unflagged outputs for missed cases. We also obtain GPT-6 Luna annotations. For target-code PI at step 100, SFT initialization reduces human-reported leakage from 52.8% to 3.4% and GPT-6 Luna’s measured leakage from 46.0% to 4.4%, as shown below.
Table 1. Human and GPT-6 Luna leakage annotations at step 100.
Hidden-detail evaluation. We randomly select the same 100 tasks from the test set and evaluate target-code OPSD under both initializations. GPT-6 Luna receives the image, instruction, generated response, and stored reference and target programs to identify claims about implementation details that cannot reliably be inferred from the image. Explicit references to unavailable context decrease from 39/100 to 3/100, while hidden-detail assertion candidates decrease from 7/100 to 2/100. The results are reported separately below.
Table 2. Source-aware annotation of unavailable-context references and hidden-detail assertion candidates.
Representative cases illustrate the distinction between choosing an implementation and claiming to have read an unavailable program.
Table 3. Representative output claims and target-code evidence.