acceptodds
Under review as a conference paper at ICLR 2027

Learning Programmatic Chart Editing through Privileged On-Policy Self-Distillation

Abstract

Chart editing requires a multimodal large language model (MLLM) to translate a reference chart and a natural-language instruction into executable plotting code. Existing approaches often rely on powerful proprietary models to synthesize reasoning data for supervised fine-tuning (SFT) and evaluate rollouts during group relative policy optimization (GRPO), making supervision costly to scale. To reduce this reliance, we introduce a two-stage fine-tuning framework that combines SFT on verified reasoning and code pairs with on-policy self-distillation (OPSD), where a frozen copy of the fine-tuned model receives privileged information (PI) to supervise a student model given only the original input. First, for SFT data construction, we design a reasoning trajectory generation pipeline guided by privileged information (PI). Specifically, we annotate the reference chart to obtain a structured textual description and provide it, together with the reference chart and editing instruction, to a low-cost MLLM of comparable capability to Qwen3.5-9B (the base model) to obtain the reasoning trajectory. This yields a training set of 266k verified trajectories, which, after SFT of the base model, brings improvements of over 10% on both ChartEdit and ChartMIMIC. Next, after SFT, OPSD further improves the model through dense token-level feedback, without requiring a separate larger teacher or repeated calls to an external reward evaluator during optimization. We conduct experiments to systematically study how the choice of PI and SFT initialization affect OPSD. On ChartEdit and ChartMIMIC, our model achieves relative improvements of 17% and 32%, respectively, over the base model and delivers the best overall performance among the open-source models of comparable size. Our comparisons favor precise PI aligned with the desired output, with target code outperforming both reference information and combinations containing additional context. Direct OPSD degrades performance and exhibits substantial PI leakage, whereas SFT initialization reduces the leakage rate from over 40% to approximately 2% and enables further performance gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Chart Editing, Multimodal Large Language Models, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jBWpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to make multimodal large models generate executable chart editing code from chart images and natural language instructions, and proposes a two-stage training framework. In the first stage, structured chart descriptions are used to assist in generating reasoning traces and code, which, after filtering, are used for supervised fine-tuning (SFT); in the second stage, it adopts online privileged self-distillation (OPSD): a frozen teacher model can access privileged information during training, while the student receives only the image and the editing instruction. The authors evaluate the method on ChartEdit and ChartMIMIC, and compare the effects of different forms of privileged information, the degree of alignment with the target code, and SFT initialization. The paper reports that combining SFT with OPSD improves multiple task metrics; compared with GRPO, the optimization phase of OPSD is faster and does not require a separate large reward evaluation model.

Critical strengths
  • The training pipeline design is clear. The paper chains data construction, SFT, and online self-distillation into a complete pipeline, attempting to reduce dependence on expensive teacher models and online reward evaluators, which has some practical value.
  • It attends to the important risk of privileged information leakage. The student sees only the image and the instruction during training and inference, while the teacher can use additional descriptions or code. The paper compares different privileged information and reports that OPSD without SFT leads to performance degradation and noticeable leakage, whereas SFT initialization significantly reduces the leakage rate.
  • It analyzes the content of the privileged information and its relationship with the target. The paper compares reference chart descriptions, reference code, target code, and their combinations, and finds that target code — privileged information more aligned with the expected output — is generally more effective; additionally incorporating reference context does not necessarily bring gains. These analyses provide clues for understanding when self-distillation is effective.
  • The experiments cover multiple chart tasks and baselines. The paper compares against general-purpose multimodal models and chart-specific models on ChartEdit and ChartMIMIC's reproduction and customized editing tasks; it also reports ablations on data filtering, distillation targets, and inference efficiency.
Critical weaknesses
  • Target code as privileged information may directly provide the answer. The teacher can see the target program, while the student sees only the chart and the instruction. Even if the authors report a reduced leakage rate, this setting may still mainly be transferring the target code's behavior to the student, without necessarily improving its ability to independently infer edits from visual evidence.
  • The large-scale SFT data depends on automatic generation and automatic filtering. The paper uses a large-scale synthetic corpus constructed from charts and programs, and filters samples through model-generated reasoning and quality checks. If the generator or the evaluator has systematic biases, erroneous demonstrations may still enter the training set; the main text provides limited description of independent human review.
  • The evaluation scope is not yet sufficient to demonstrate robustness in real-world scenarios. The experiments are concentrated on ChartEdit and ChartMIMIC, and the training data comes from the charts and programs in the paper. It remains unclear how the method performs on real charts with complex styles, unclear annotations, data points that are difficult to read from the image, or ambiguous editing instructions.
  • The cost advantage is mainly reflected in the OPSD stage, and the cost of the complete pipeline is not yet clear enough. The paper reports that SFT requires about 266k samples and about 32 hours of training on four A800 GPUs; the time comparison between OPSD and GRPO is limited to a finite number of update steps. If data generation, quality checks, and two-stage training are included, the overall cost advantage needs further quantification.
Questions for authors
  • Regarding target code as privileged information: How do the authors distinguish between "the student learns to perform edits based on the chart and the instruction" and "the student indirectly memorizes answers from the target code through distillation"? What is the specific definition of the leakage rate, and has it been tested on information that is present in the target code but cannot be reliably recovered from the image?
  • Regarding synthetic data quality: Which models or checkers are used for the automatic filtering of reasoning traces and code? Is there independent human sampling and review, and can the authors report error types, false-filter rates, and the impact of these errors on final performance?
  • Regarding real-world generalization: Can the method be evaluated on charts from different sources with more diverse styles, or on real user editing tasks? In particular, does it maintain editing quality and avoid altering unrelated attributes for data that is visually hard to read accurately, unseen chart types, and ambiguous instructions?
  • Regarding the complete cost: Can the authors report the end-to-end resources and cost from synthetic data generation, quality filtering, and SFT to OPSD, and compare with GRPO or other methods under the same data volume, training budget, and hardware conditions? This would allow a more accurate judgment of whether the overall training cost is actually reduced.
6 comments
AuthorsOpenReview ↗

Response to Reviewer jBWp (Part 1/4)

We sincerely thank the reviewer for the careful and constructive review.

W1 & Q1: Definition and Evaluation of Privileged-Information Leakage

Thank you for asking us to clarify PI leakage and examine hidden implementation details. SFT initialization substantially reduces measured leakage in both evaluations below.

  • Definition. The frozen teacher is conditioned on target code; the student receives only the image and editing instruction. We count output claims of access to unavailable descriptions or programs, or unsupported assertions about their contents, as PI leakage. Agreement with target code, visually grounded estimates, and implementation choices alone do not count. For example, choosing a random seed differs from claiming that the original program used it.

  • Leakage evaluation. On the held-out test set, we use characteristic expressions from manual inspection to screen candidates, manually review every flagged output, and inspect sampled unflagged outputs for missed cases. We also obtain GPT-6 Luna annotations. For target-code PI at step 100, SFT initialization reduces human-reported leakage from 52.8% to 3.4% and GPT-6 Luna’s measured leakage from 46.0% to 4.4%, as shown below.

    Table 1. Human and GPT-6 Luna leakage annotations at step 100.

    InitializationHuman-reported leakage (%)GPT-6 Luna leakage (%)
    w/o SFT52.8046.00
    w/ SFT3.404.40
  • Hidden-detail evaluation. We randomly select the same 100 tasks from the test set and evaluate target-code OPSD under both initializations. GPT-6 Luna receives the image, instruction, generated response, and stored reference and target programs to identify claims about implementation details that cannot reliably be inferred from the image. Explicit references to unavailable context decrease from 39/100 to 3/100, while hidden-detail assertion candidates decrease from 7/100 to 2/100. The results are reported separately below.

    Table 2. Source-aware annotation of unavailable-context references and hidden-detail assertion candidates.

    Metric (GPT-6 Luna annotation)w/o SFTw/ SFT
    Explicit references to unavailable context39/100 (39%)3/100 (3%)
    Code-grounded hidden-detail assertion candidates7/100 (7%)2/100 (2%)

Representative cases illustrate the distinction between choosing an implementation and claiming to have read an unavailable program.

Table 3. Representative output claims and target-code evidence.

CaseOutput claimTarget-code evidence
Random seedThe supplied code uses seed 42.np.random.seed(42)
Data generationThe supplied target code uses np.logspace.np.logspace(0, 12, 1000)
AuthorsOpenReview ↗

Response to Reviewer jBWp (Part 2/4)

W2 & Q2: Automatic filtering and independent human review

Thank you for asking about filtering reliability. We clarify the automatic checks and add an independent human review. Automatic and human decisions agree on 91.0% of code assessments and 88.0% of trace assessments.

  • Automatic checks. The SFT construction stage uses Qwen3.7-Flash for description generation, reasoning/code synthesis, and quality assessment. We first execute and render the generated code. Code and image checks assess edit fulfillment, unintended changes, and readability; trace checks assess visual grounding, technical correctness, and consistency with the generated code.

  • Human review. We randomly sample and manually reassess 300 code-level decisions and 100 trace-level decisions, covering both automatically accepted and rejected samples. The tables below report the decisions and summarize agreement and errors in both directions.

    Table 4. Human review of automatic filtering decisions.

    Assessment levelAutomatic decisionHuman passHuman failTotal
    Code levelPass1748182
    Code levelFail1999118
    Trace levelPass55560
    Trace levelFail73340
  • Results. Among automatically accepted samples, the human-identified failure rates are 4.40% for code and 8.33% for traces. Valid samples also occur among automatic rejections (16.10% and 17.50%, respectively).

    Table 5. Agreement and error proportions between automatic filtering and human review.

    Human-audit measureCode levelTrace level
    Automatic–human agreement273/300 (91.0%)88/100 (88.0%)
    Human-failed samples among automatic passes8/182 (4.40%)5/60 (8.33%)
    Human-passed samples among automatic failures19/118 (16.10%)7/40 (17.50%)
AuthorsOpenReview ↗

Response to Reviewer jBWp (Part 3/4)

W3 & Q3: Evaluating Cross-Source Generalization in Chart Editing

Thank you for suggesting evaluation beyond the original benchmarks. We conducted an exploratory evaluation of 336 editing tasks using images from five external sources. OPSD achieves the highest rendering success and average editing score among the four evaluated models.

  • Test dataset composition

    • Size and sources. The test subset contains 336 manually designed editing tasks on 230 images from ChartMuseum [1], ChartQAPro [2], VisTalk [3], VCG-Bench [4], and Geometry3K [5].

    • Visual coverage. Images include statistical charts, infographics, structural diagrams, and geometric figures. Challenge groups cover dense or small details, different diagram structures, and bounded stylistic underspecification; these groups can overlap.

    • Task scope. For stylistically underspecified requests, the desired effect and content to preserve are stated, while the exact styling is left open. This is an exploratory, post hoc selected subset.

  • Evaluation method

    • Model input. Each model receives only the reference image and editing instruction.

    • Judge input. GPT-6 Luna receives the original image, instruction, and rendered result, without the generating model's identity.

    • Scoring. Edit fulfillment, data/structure preservation, non-target appearance preservation, and readability are each scored out of 25, totaling 100. All models are evaluated on the same 336 tasks; failed renders receive zero in every dimension.

  • Results and analysis

    • Overall performance and content preservation.

      Table 6. Editing performance on the exploratory cross-source subset.

      ModelRendering successTotal /100
      Qwen3.5-9B base69.64%40.50
      Qwen3.5-9B + SFT82.44%55.78
      Qwen3.5-9B + SFT + OPSD88.69%58.47
      Qwen3.5-27B73.51%54.18
    • Relative to SFT, OPSD improves rendering success by 6.25 percentage points and the all-task score by 2.68 points, calculated before rounding. Its all-task data/structure preservation score increases from 12.68 to 13.24, and non-target appearance preservation from 10.92 to 11.44, each out of 25.

    • Performance on challenging inputs.

      Table 7. Performance across challenge groups in the exploratory subset.

      Challenge groupTasksSFT /100SFT + OPSD /100
      Dense or small visual details6838.0345.19
      Structural diagrams11456.7957.12
      Geometric diagrams5463.3372.24
      Bounded stylistic underspecification1931.4233.00

      The largest subgroup gains are on geometric diagrams (+8.91 points) and dense or small visual details (+7.16 points). Structural diagrams and stylistically underspecified requests show smaller gains.

Reference

[1] Tang, L., Kim, G., Zhao, X., et al. (2025). ChartMuseum: Testing visual reasoning capabilities of large vision-language models. Advances in Neural Information Processing Systems (NeurIPS 2025), Datasets and Benchmarks Track.

[2] Masry, A., Islam, M. S., Ahmed, M., et al. (2025). ChartQAPro: A more diverse and challenging benchmark for chart question answering. Findings of the Association for Computational Linguistics: ACL 2025, 19123-19151.

[3] Wang, Y., Hou, Z., Shen, L., et al. (2023). Towards natural language-based visualization authoring. IEEE Transactions on Visualization and Computer Graphics, 29(1), 1222-1232.

[4] Su, X., Dong, P., Tang, Z., et al. (2026). VCG-Bench: Towards a unified visual-centric benchmark for structured generation and editing. arXiv preprint arXiv:2605.15677.

[5] Lu, P., Gong, R., Jiang, S., et al. (2021). Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 6774-6786.

Reviewer_JeNEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a two-stage framework for programmatic chart editing and reproduction. First, it fine-tunes Qwen3.5-9B on 266k reasoning–code examples generated with structured chart descriptions and filtered through automated verification. Second, it applies privileged on-policy self-distillation (OPSD) on 26k additional examples, using a frozen SFT teacher conditioned on privileged information while restricting the student to the chart image and instruction. Evaluation on ChartEdit and ChartMIMIC shows improvements over the base model. Ablations examine reasoning supervision, privileged-information choices, initialization, and distillation objectives. The results favor target-code supervision after SFT and show that lower distillation divergence alone does not guarantee better task performance.

Critical strengths

s1. The comparisons across privileged-information types and initialization conditions reveal meaningful limitations of self-distillation. In particular, direct OPSD reduces divergence while degrading task performance, and combining reference information with target code weakens the gains (Figure 5; Table 6). These findings provide useful guidance beyond the benchmark improvements.

s2. The final 9B model performs strongly against the evaluated general-purpose and chart-specialized baselines (Table 1). OPSD improves ChartMIMIC Customized Overall from 69.81 to 76.40 while avoiding online rendering and external reward evaluation. This substantial improvement beyond SFT supports the practical value of the approach.

Critical weaknesses

w1. The advantage of OPSD over matched alternatives is not obvious. OPSD uses 26k additional examples, but there is no continued-SFT baseline using the same target programs. Moreover, the GRPO comparison reports runtime and API cost without downstream task scores (Table 3), so the reported speedup does not establish a better performance–cost trade-off. This matters especially because OPSD’s incremental gains on ChartEdit and ChartMIMIC Direct are only 0.63 and 1.84 points, respectively, with no uncertainty estimates.

Questions for authors

q1. How does OPSD compare with continued SFT on the same 26k examples under comparable training budgets? Are the smaller gains on ChartEdit and ChartMIMIC Direct robust across runs?

q2. How should the statement about uniformly lower base-teacher scores under code-based PI be reconciled with Table 7? Does this affect the explanation for direct OPSD’s degradation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer JeNE (Part 1/2)

We sincerely thank the reviewer for the careful and constructive review.

W1 & Q1: Comparison with the performance of continued-SFT and GRPO models.

Thank you for suggesting additional training comparisons and for asking us to assess run-to-run robustness. We added continued-SFT and offline-distillation controls and evaluated GRPO to assess OPSD’s additional value.

  • Setting. All methods start from the same SFT checkpoint. Continued SFT follows the original full-parameter training procedure. Offline distillation uses the same PI-conditioned teacher and LoRA student-update configuration as OPSD. Both controls are evaluated with 25.6k and 51.2k additional examples; GRPO uses four rollouts per prompt.

  • Training efficiency. With 256 examples/update, continued SFT takes 78.3 s with USD 0.24 in data-synthesis costs. Offline distillation takes 35.5 s/update for optimization, or approximately 93.5 s/update including response preparation. OPSD and GRPO take 128.3 and 740.4 s/update, respectively.

  • Results. Overall scores are summarized below.

    Table 1. Overall performance of OPSD and post-training comparison methods.

    MethodChartEditDirectCustomized
    SFT initialization83.680.369.8
    OPSD84.382.176.4
    Continued SFT (+25.6k data)8481.269.4
    Continued SFT (+51.2k data)83.782.169.3
    Offline distillation (25.6k data)81.280.570.2
    Offline distillation (51.2k data)81.180.465.3
    GRPO83.382.672.4

We repeated OPSD training with two additional random seeds, 43 and 44, giving three training runs in total.

  • Results. The table reports Overall scores and absolute gains over the fixed SFT baseline.

    Table 2. OPSD performance across three training runs.

    Training runChartEdit OverallΔ vs. SFTDirect OverallΔ vs. SFT
    SFT baseline83.6—80.3—
    OPSD, original run84.30.6482.11.83
    OPSD, seed 4384.10.4581.81.49
    OPSD, seed 4484.20.5583.43.16
    OPSD, mean ± sample SD84.20 ± 0.10+0.55 ± 0.1082.45 ± 0.88+2.16 ± 0.88
    • Gains remain positive across all three runs, ranging from 0.45–0.64 points on ChartEdit and 1.49–3.16 points on Direct.
AuthorsOpenReview ↗

Response to Reviewer JeNE (Part 2/2)

Q2: Clarifying Table 7 of the Submitted Manuscript and the Role of Initialization

Thank you for identifying this inconsistency. We clarify the correction, its implications, and the additional initialization experiments below.

  • Correction to the statement. Code-based PI does not uniformly lower the base teacher’s scores. In Table 7 of the submitted manuscript, reference-code PI improves ChartEdit from 64.21 to 64.50 and Customized from 56.55 to 58.12. Target-code PI also improves Customized to 57.90. Its effects are therefore task-dependent, and we will correct the corresponding statement in Section 4.5.3.

  • Implications for direct OPSD. This correction removes a uniform decline in PI-conditioned teacher performance as an explanation for direct OPSD’s degradation. The observed task-performance degradation and reduction in teacher–student divergence remain unchanged, supporting the conclusion that closer distribution matching does not necessarily improve task performance.

  • Crossed initialization experiments. We added two crossed teacher–student conditions to distinguish the contributions of their initialization. All entries below are Overall scores.

Table 3. Overall performance under crossed teacher and student initialization.

StudentTeacherChartEditDirectCustomized
BaseBase68.358.357.5
SFTSFT84.382.176.4
SFTBase68.666.858.5
BaseSFT75.567.165.8

Thank you for your detailed reply. I have no further questions at this point, I have raised my soundness to 3 and my overall assessment remains unchanged.

Thank you for your thoughtful consideration of our responses and for updating your soundness assessment!

Reviewer_AvaUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies programmatic chart editing, where a multimodal model receives a reference chart image and a natural-language instruction and generates executable plotting code without access to the original source code. It proposes a two-stage training pipeline. First, structured chart descriptions guide the synthesis of reasoning and code, with separate checks for reasoning quality and program correctness. Approximately 266,000 retained examples are used to fine-tune Qwen3.5-9B. Second, a frozen copy of the SFT model receives privileged information (PI) and supplies next-token distributions on student-generated prefixes. The student receives only the image and instruction and is trained with Jensen–Shannon divergence on another 26,000 examples. Experiments on ChartEdit and the Direct and Customized tasks of ChartMIMIC show strong improvements from SFT and additional gains from on-policy self-distillation (OPSD), particularly on Customized. The best configuration supplies target code to the teacher. The paper also compares PI types, distillation objectives, and initialization, and reports lower per-update cost than a GRPO implementation requiring external reward evaluation.

Critical strengths
  1. The data-construction pipeline addresses a concrete difficulty in visual code supervision. Reference programs can contain details unavailable from the image, making synthesized reasoning unsuitable for an image-only student. The structured-description step explicitly distinguishes observations, inferences, and uncertain properties, while separate reasoning and program checks address errors that execution alone would miss. Constructing 266,454 retained examples and showing substantial SFT improvements across the evaluated tasks constitutes a meaningful empirical contribution.

  2. The final model performs well, with a meaningful second-stage gain on customized editing. OPSD improves ChartMIMIC Customized Overall from 69.81 to 76.40, while also improving ChartEdit and Direct. The PI comparisons and the uniform use of an image-and-instruction student input make the training-time supervision design explicit. The observation that direct OPSD can reduce teacher–student divergence while degrading task performance is also useful: it demonstrates that distribution matching alone is an inadequate measure of successful transfer.

Critical weaknesses
  1. The comparisons do not sufficiently establish the additional value or cost-effectiveness of the proposed second stage. The 26,000 OPSD examples already include target code, making continued SFT from the same SFT checkpoint a particularly relevant baseline. Offline distillation from the same PI-conditioned teacher would further test the value of supervision on student-generated prefixes. Neither comparison is reported. The first-stage code-only SFT ablation in Table 2 does not substitute for these controls because it uses a different initialization and training stage. Although the Customized gain is substantial, the additional gains on ChartEdit and Direct are only 0.63 and 1.84 points; comparing against the unchanged SFT checkpoint cannot separate the proposed mechanism from the benefit of additional data and optimization. Furthermore, Table 3 reports a 5.77-fold wall-time difference over 100 updates but no corresponding GRPO task scores. This establishes lower per-update cost in the tested setup, not better performance at equal cost or lower cost at comparable performance. These gaps are consequential because efficient privileged OPSD is central to the paper's contribution.

  2. The analysis of initialization and PI leakage is insufficiently resolved. The direct-OPSD and SFT-initialized conditions change both teacher and student initialization. They establish that SFT benefits the complete process, but do not distinguish improved teacher guidance from improved student readiness. The reported reduction in leakage from over 40% to approximately 2% is potentially informative, yet its measurement is not specified sufficiently to reproduce: the paper gives a conceptual definition and examples, but does not clearly report the labeling procedure, evaluated sample counts, or validation of the detector or judge. This matters because legitimate transfer from target-code supervision must be distinguished from outputs that falsely assume access to missing context; matching the target program is explicitly insufficient to establish leakage. The headline leakage numbers therefore provide less explanatory evidence than their prominence suggests.

Questions for authors
  1. Starting from the same SFT checkpoint and using the same 26,000 examples, how does OPSD compare with continued SFT on target code and offline distillation from the same PI-conditioned teacher? Matching the student parameterization and reporting training and teacher-generation costs would help establish the additional value of on-policy supervision.

  2. What task scores are achieved by the GRPO checkpoint used for Table 3? Can you compare performance at matched wall time or report the cost of reaching comparable task performance? The existing per-update measurements alone do not resolve the practical efficiency comparison.

  3. How exactly are the leakage rates in Figures 4 and 5 computed? Please specify the sample sizes, labeling criteria, detector or judge and its prompt if applicable, and any manual validation. How do you distinguish unsupported assumptions about available context from valid predictions learned through privileged supervision?

  4. Can you hold the SFT teacher fixed while comparing base and SFT students, or hold the SFT student fixed while changing the teacher? Such a control would help determine whether SFT primarily improves the teacher's use of PI, the student's ability to learn from it, or both.

6 comments
AuthorsOpenReview ↗

Response to Reviewer AvaU (Part 1/4)

We sincerely thank the reviewer for the careful and constructive review.

W1 & Q1: Comparison with Continued SFT and Offline Distillation

Thank you for suggesting continued-SFT and offline-distillation comparisons. We added continued SFT and offline distillation experiments. OPSD outperforms both baselines across all three tested tasks, with the largest gains on ChartMIMIC Customized. We summarize the setup, training efficiency, and benchmark results below.

  • Setting: OPSD, continued SFT, and offline distillation all start from the same SFT checkpoint. For continued SFT, we use the same setting as the original SFT stage. For offline distillation, we use the same PI-conditioned teacher as OPSD to generate responses offline, then fine-tune the student using the same student parameter-update configuration as OPSD. We evaluate two data scales (25.6k or 51.2k generated by the original synthesis pipeline) to assess whether additional data enables continued SFT and offline distillation to match OPSD’s gains.

  • Training efficiency: OPSD takes 128.3 s/update, requiring no separate response preparation. Continued SFT takes 78.3 s/update but incurs an additional USD 0.24/update for data synthesis. Offline distillation takes ~93.5 s/update, including preparation (58.0 s generation + 35.5 s optimization).

  • Results: The table below compares Overall scores on ChartEdit and the Direct and Customized tasks of ChartMIMIC.

    Table 1. Overall performance of OPSD, continued SFT, and offline distillation.

    MethodChartEditDirectCustomized
    SFT initialization83.680.369.8
    OPSD84.382.176.4
    continued SFT (25.6k data)8481.269.4
    continued SFT (51.2k data)83.782.169.3
    offline distillation (25.6k data)81.280.570.2
    offline distillation (51.2k data)81.180.465.3
    • OPSD achieves the strongest performance on Customized, as it exceeds continued SFT by 7.0–7.1 points and offline distillation by 6.2–11.1 points. On ChartEdit and Direct, OPSD exceeds or matches continued SFT, and consistently outperforms offline distillation.

    • Continued SFT brings marginal improvements to the ChartEdit and Direct tasks but also slightly degrades on the Customized task (dropping from 69.8 to 69.4 and 69.3).

    • Offline distillation yields no consistent improvement over the SFT initialization. Specifically, with 51.2k examples, its Customized score drops by 4.5 points.

These results support the benefit of OPSD over the tested continued-SFT and offline-distillation alternatives.

AuthorsOpenReview ↗

Response to Reviewer AvaU (Part 2/4)

W1 & Q2: Task Performance of the GRPO Checkpoint and Cost-Effectiveness.

Thank you for requesting the GRPO checkpoint’s task scores and a performance–cost comparison. We supplemented the task scores of the GRPO checkpoint used in Table 3 of the submitted manuscript, which was trained on the same 25.6k examples as OPSD.

Table 2. Task performance of OPSD and GRPO.

MethodChartEditDirectCustomized
SFT initialization83.680.369.8
OPSD84.382.176.4
GRPO83.382.572.4

For 100 updates on four H100 GPUs, GRPO takes 5.77 times as long as OPSD (Table 3 of the submitted manuscript) but achieves lower scores on ChartEdit and Customized. Specifically, while GRPO attains a comparable score on the Direct task (82.5 vs. 82.1), it lags behind OPSD on both ChartEdit (83.3 vs. 84.3) and the Customized task (72.4 vs. 76.4).

AuthorsOpenReview ↗

Response to Reviewer AvaU (Part 3/4)

W2 & Q3: Supplementary Description and Experiments on Data (Privileged Information) Leakage

Thank you for requesting clarification of the leakage evaluation. Both human and model annotations show lower PI leakage with description-guided synthesis and SFT-initialized OPSD. We summarize the definition, annotation procedure, and results below.

  • Definition: We define PI leakage as expressed reliance on descriptions or programs unavailable to the student, including unsupported claims about their contents. Visually grounded estimates, implementation choices, and agreement with target code alone do not count as leakage.

  • Annotation procedure:

    • Samples. For SFT data synthesis, GPT-6 Luna evaluates the same 500 tasks across three pipelines guided by reference descriptions, reference code, or target code. For OPSD, we evaluate each PI type and checkpoint on a fixed set of 500 examples held out before fine-tuning, with and without SFT initialization. We evaluate checkpoints at steps 20, 40, 60, 80, and 100.
    • Human annotation. We first manually inspect sampled outputs to identify phrases that may indicate PI leakage (e.g., "Based on the provided target code, …"). We use these phrases to flag candidate outputs and manually review each in context. We also inspect a sample of unflagged outputs for missed leakage.
    • Model annotation. We provide GPT-6 Luna with the original instruction, reference image, and generated text to identify PI leakage. We report human and model leakage rates separately.
  • Results:

    • SFT data synthesis: Description-guided synthesis produces substantially less PI leakage. Human and model annotations both show lower leakage with reference descriptions than with either form of code guidance.

      Table 3. Human and GPT-6 Luna leakage annotations for synthesized SFT reasoning.

      Extra informationOriginal human-reported (%)GPT-6 Luna (%)
      Reference Chart Description1.00.0
      Reference Chart Code12.816.2
      Target Code12.416.0
    • OPSD

      • Without SFT initialization, code-based PI leakage increases substantially during training. Both human and model annotations show this trend for target-code and reference-code PI, while description-based PI maintains low leakage throughout training.

        Table 4. PI leakage across OPSD checkpoints without SFT initialization (human / GPT-6 Luna, %).

        PIs20s40s60s80s100
        gtcode0.40 / 5.604.80 / 5.0023.80 / 11.0045.20 / 28.4052.80 / 46.00
        refcode4.60 / 11.2038.60 / 30.4069.20 / 70.4086.80 / 89.2084.60 / 87.80
        refdesc2.60 / 3.601.00 / 1.402.40 / 1.002.80 / 2.204.00 / 1.20
        • With SFT initialization, the large increase in code-based PI leakage is substantially mitigated. Both annotation methods show much lower leakage at later checkpoints than in the corresponding runs without SFT initialization.

        Table 5. PI leakage across OPSD checkpoints with SFT initialization (human / GPT-6 Luna, %).

        PIs20s40s60s80s100
        gtcode0.40 / 5.002.20 / 6.002.40 / 5.003.00 / 5.403.40 / 4.40
        refcode1.00 / 5.601.60 / 5.602.20 / 5.601.00 / 6.803.00 / 10.80
        refdesc0.40 / 4.000.00 / 3.200.80 / 3.000.00 / 5.000.60 / 5.00

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear AC and Reviewers,

We sincerely thank you for your time and constructive feedback on our paper, Learning Programmatic Chart Editing through Privileged On-Policy Self-Distillation.

We are grateful that reviewers recognized the strengths of our work, including: 11 the structured data-construction and verification pipeline (Reviewers AvaU and jBWp), 22 the strong empirical performance and additional gains from OPSD (Reviewers AvaU and JeNE), and 33 the informative analysis of privileged information and SFT initialization (Reviewers AvaU, JeNE, and jBWp).

During the rebuttal period, we conducted additional experiments and analyses addressing training comparisons, initialization, PI leakage, data quality, generalization, and computational costs. We also clarified the evaluation procedures and corrected the interpretation of Table 7 in the revised manuscript. The reviewer feedback and key updates are summarized below:

Reviewer Feedback

  • Reviewer AvaU confirmed that “My concerns have been addressed” and raised the overall score to 3.

  • Reviewer JeNE reported “no further questions,” raised the soundness assessment to 3, and maintained the positive overall score of 3.

  • Reviewer jBWp raised concerns about target-code PI, filtering reliability, generalization, and computational costs. We provided expanded leakage analyses, an independent human audit, cross-source evaluation, and a detailed resource breakdown. The reviewer has not yet provided follow-up feedback.

Additional Experiments and Analysis

  • Comparison with Continued SFT and Offline Distillation:

  • GRPO Task Performance and Training Cost: We compared OPSD with continued SFT and offline distillation from the same SFT checkpoint at two additional data scales. (Reviewers AvaU and JeNE)

  • Robustness across Training Runs. We reported the GRPO checkpoint’s task scores alongside its training time and API cost for comparison with OPSD. (Reviewers AvaU and JeNE)

  • PI-Leakage and Hidden-Detail Analysis. We added model-based leakage annotations alongside human annotations and separately examined claims about implementation details unavailable from the image. (Reviewers AvaU and jBWp)

  • Independent Human Audit of Automatic Filtering. We manually reassessed code-level and trace-level filtering decisions and reported agreement and errors among automatically accepted and rejected samples. (Reviewer jBWp)

  • Exploratory Cross-Source Evaluation. We conducted an exploratory evaluation on 336 editing tasks using images from five external sources. (Reviewer jBWp)

  • Resource and Cost Breakdown. We reported data-construction API cost estimates and training resources for the shared SFT stage, OPSD, and GRPO. (Reviewer jBWp)

Clarifications and Correction

  • Definition and Annotation of PI Leakage. We specified the leakage criteria, evaluated samples, and human and model annotation procedures. (Reviewers AvaU and jBWp)

  • Automatic Code and Reasoning Checks. We specified the models and checks used to assess generated code, rendered charts, and reasoning traces. (Reviewer jBWp)

  • Interpretation of Table 7. We corrected the statement about code-based PI uniformly lowering base-teacher scores and clarified its implications for explaining direct OPSD’s degradation. ReviewerJeNE,∗∗Section45˙3˙∗∗Reviewer JeNE, **Section 4\.5\.3**

Thank you again for helping us strengthen the evidence and clarify the scope of our conclusions. We appreciate Reviewer AvaU’s confirmation that the concerns have been addressed and Reviewer JeNE’s updated soundness assessment. We hope the additional results and clarifications assist the AC and reviewers in assessing the contribution and remaining limitations of our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.