- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submitted paper builds a dataset for generating and editing academic diagrams, such as geometry figures, molecules, sheet music, and state machines. It has about 1.2M samples across ten subjects. Most of the data is synthetic. The authors render it from code (SVG, TikZ, RDKit, LilyPond, etc.), mask text labels to create editing pairs, and filter open-source educational images. For the model, they use a frozen Qwen3-VL-8B to rewrite prompts. They then LoRA-finetune Qwen-Image for generation and Qwen-Image-Edit for editing. The finetuned models outperform other open-source models on GenExam (51.4 vs. 35.8 base) and GRADE (58.7 vs. 30.0 base). They also show improvements on WISE and RISE. An ablation study adds each data pipeline step by step to show its contribution.
Critical strengths
-
Most edit pairs are made by changing the source code and re-rendering, so the two images only differ where the edit is. Clean pairs like this are hard to collect, and the data would be useful if released.
-
Over the same base models, GenExam goes up by 15.6 and GRADE by 28.7.
-
The data covers subjects like music and sports that prior datasets rarely include. The appendix is also detailed enough to reproduce the pipeline.
Critical weaknesses
- It is unclear how much of the gain comes from the data itself. Prompt enhancement alone gives a large part of the improvement, and the ablation does not control training budget.
- The dataset is not well documented or verified. The claimed 1.2M does not match the training size in Appendix C, and there is no quantified human audit.
- The benchmark gains are clear, and the WISE and RISE results provide some evidence of transfer. However, the paper does not report controlled evaluations on unseen templates or edit types, and test-set overlap checks are not fully described for the synthetic pipelines.
Questions for authors
Q1: Effect of prompt enhancement. Table 6 shows prompt enhancement alone brings large gains (GenExam 35.8→41.0, GRADE 30.0→41.2, WISE 0.50→0.59, RISE 19.4→23.5). There is no result for the finetuned model without it, and the baselines in Tables 2–5 do not use the same rewriter. Can you report the full 2×2 setting (finetuned or not × prompt enhancement or not), and run the main baselines with the same rewriter?
Q2: Ablation design. (a)In Table 6, each data type is added in a fixed order, and every step also adds more samples and training steps. The gains could come from having more data rather than from the data type itself, and a different order might give a different picture. (b)I am also confused by two rows in Table 6: OCR editing data is added to the T2I model, and T2I filtering data is added to the editing model. Section 4.1 says each model only uses its own task data, so how were these used? (c) It would help to see leave-one-out results with the total training budget fixed. (d)A comparison with an existing dataset such as StructVisuals or S1-Omni-Image, using the same base model and budget, would also show whether this data is better than what is already available.
Q3: The training setup in Appendix C (43,993 and 23,384 steps, effective batch size 8, one pass, repeat factor 1) implies about 352K T2I and 187K editing samples, about 539K in total. How does this relate to the reported 1.2M? Could you give the sample counts by pipeline before filtering, after filtering, and used in training?
Q4: Appendix B.3 mentions manual checks, but only for the SVG branch built from textbook knowledge (economics and biology), and no numbers are given. The other pipelines seem to rely only on compilers, rules, or VLM judges. Could you report a human audit on a sample from each pipeline and subject, showing how often there are knowledge errors, mismatches between the instruction and the target, or changes outside the edit region?
Q5: Some training tasks are similar to those evaluated in the benchmarks, such as the Go and music cases in Figure 5. This is reasonable for coverage, but generalization to unseen task types remains unclear. Could you evaluate held-out templates or edit types? Appendix B.5 describes pHash-based benchmark overlap checks for retrieved images. Were synthetic samples also checked at the source level, such as shared game positions, score fragments, or graph instances?
Minor:
- Table 4 incorrectly lists SenseNova-U1-8B-MoT as closed-source, although its weights are publicly available. Its reported score is also higher than the proposed model’s (0.68 vs. 0.66). Please correct the classification and any affected ranking statements.
- Section 4.3 claims gains over the strongest closed-source baseline on RISE, but Table 5 reports 26.4 for the proposed model versus 49.4 for GPT-Image-2. Please clarify or correct this statement.