- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- EdiTikZ mines plausible figure-edit pairs from scientific sources and infers editing instructions from images and code. Joint reconstruction and editing training, followed by multi-reward RL, produces compact models with strong results on a human-refined benchmark.
Critical strengths
- Mining existing human-authored figures provides a substantial and useful source of editing supervision.
- Group-level data separation, human-corrected benchmark instructions, and human evaluation strengthen the empirical evidence.
Critical weaknesses
- Half of the audited training instructions contain some error. The consequences of these mismatches for training and the two reward signals remain insufficiently quantified.
- The data-mixture ablations do not clearly establish whether joint training benefits from reconstruction supervision specifically or from additional training exposure.
Questions for authors
- Clarification: Were editing-only, sequential, and joint-training ablations matched for total training tokens and optimization steps?
- Evidence request: Is there evidence separating performance on clean and noisy instructions, or a controlled filtering ablation? This would clarify how reliably the method learns from the mined supervision.
2 comments
Solved my concerns, so I will keep my rating.
Thank you for pointing this out. This is acknowledged as a limitation in our conclusion (p9, l478-479). Despite still containing errors, 82.9% of our instructions were rated either very good or good by human expert annotators with high agreement (weighted κ=0.79) as seen in p4, l213-215. Moreover, these errors also contain hardly observable changes such as minimally increasing the size of an arrowhead which may even improve model robustness and generalization to non-meaning changing perturbations. Lastly, with growing capabilities of VLMs in the future, we expect the performance of our approach to increase as error rates go down.
Editing-only and reconstruction-only models were trained for 4 epochs on 754K samples, whereas joint and sequential training used two epochs over 1.51M combined examples (~3M combined samples for all four ablation settings). We also used the same effective batch size as indicated in Table 14, matching the optimization steps. Input images and instructions are masked from the loss and a maximum sequence length of 2K is used throughout training. Due to DaEdiTikZ being a subset of the extended DaTikZ-V4, all plausible figure editing pairs can be used for both editing and reconstruction. Standalone figures not included in DaEdiTikZ but in our extended DaTikZ-V4 can still be used for reconstruction supervision alone. Therefore, our joint approach does not rely on additional data.
To assess sensitivity to noisy instructions, we introduced controlled perturbations of the DaEdiTikZ-Bench edit instructions. For each sample, we replaced one text span with either:
A paraphrase that preserves meaning (positive)
A visual property (e.g., annotation, data, relationship, …) that changes meaning (negative)
Afterwards, we judged if the outputs fulfill the changed requirement and report success under meaning-preserving/meaning-changing instructions conditioned on success under the original instruction. Our results are as follows:
Meaning-preserving perturbations
Meaning-changing perturbations
With the following takeaways:
EdiTikZ-9B-RL retains correctness under meaning-preserving changes in 94.7% of initially successful cases which is highest amongst all models.
RL does further improve the adaptation of visual requirements that change over SFT (81.4% vs. 70.3%) which is almost as good as GPT-5.6-Sol (84.1%). This demonstrates the scalability of our additional RL step post SFT, making it more robust to perturbations in the edit instructions that alter the figure.
Meaning-changing perturbations reduce overall success more for our models than for the non-meaning changing perturbations (e.g., 11.1% vs. 2.3% for EdiTikZ-9B-RL vs. GPT-5.6-Sol). However, this gap is mostly due to the very high performance on instructions that do not alter the figure and less so because it does not change properties when requested.
These findings indicate that our two-stage training approach learns very reliably from the mined supervision where it is robust to changes on the input edit instructions. We added our results to the revised paper (p8, Table 6) and thanks again for pointing this out.