- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies failures of image-generation models on visually simple tasks requiring precise execution. It proposes an L1–L5 “AI Obedience” taxonomy and introduces Violin, a benchmark of 7,745 examples covering pure-color generation, geometric shape generation, and image masking. The reported comparisons generally favor proprietary models. Additional experiments examine explicit color codes, fine-tuning, and generalization. The authors attribute the observed failures to semantic and aesthetic priors overriding exact constraints.
Critical strengths
- The paper addresses a useful evaluation gap: visually convincing images can still violate numerical or pixel-level requirements. This motivates complementary testing beyond semantic alignment.
- The tasks support programmatically constructed reference outputs and cover complementary aspects of precision (Section 3.1; Figure 4). The explicit-code extension and generalization tests in Appendix E strengthen the benchmark’s diagnostic potential.
Critical weaknesses
-
The evaluation does not reliably measure the exact instruction following claimed by the paper. The paper defines L4 obedience as exact numerical execution (Section 2.2, p. 4), but the metrics can overlook violations of the instructions. The masking metrics do not directly measure preservation of the source content (Section 3.2; Appendix D.2, Eqs. 7–10), while geometric purity measures color variation rather than correctness of the required fill color (Appendix D.2, Eq. 14). In synthetic checks of the released evaluators, changing every retained pixel from grayscale 100 to 200 still produces zero error on all five masking metrics. Adding an extra disconnected white square similarly produces zero error on all five shape metrics. These counterexamples undermine the interpretation of the reported scores as exact obedience and warrant corrected evaluation. The main color task also uses natural-language color names but scores against fixed RGB representatives (Section 3.1; Table 5, p. 18). Appendix E, p. 21, explicitly acknowledges this ambiguity, yet the explicit-code experiments in Table 7 exclude the proprietary models used in the main comparison. Consequently, the principal color results do not cleanly separate numerical execution from differences in color-name interpretation.
-
The proposed explanation and conclusions about fine-tuning exceed the supporting evidence. The examples in Section 2.3 and Figure 3 illustrate failures, but do not establish that aesthetic priors or emergent abilities cause them. Alternative explanations, including training-data coverage, numerical encoding, and image reconstruction, remain unresolved. The architectural and scaling claims in Appendix F, p. 25, therefore require controlled evidence or more cautious wording. The fine-tuning results also deserve a more balanced interpretation (Appendix E.3). Janus-Pro-1.5’s color error decreases from 0.191 to 0.045 (Tables 7 and 9), approximately a 76% relative reduction. Although this remains short of exact execution, it represents substantial improvement. Moreover, Qwen-Image’s post-tuning Var-1 score differs between Table 8 (0.048) and Table 9 (0.079) without a clear explanation. Figure 7 demonstrates degradation under the tested setup, rather than a general limitation of learning these tasks through additional data.
Questions for authors
- After correcting the metric blind spots and using explicit numerical color prompts, do the main model rankings remain stable? (Section 3.2; Appendices D.2 and E)
- What controlled evidence distinguishes the aesthetic/emergence explanation from data or reconstruction limitations? (Section 2.3; Appendix F)
- What explains the discrepancy between Tables 8 and 9, and which model and configuration produced Figure 7?