- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is a benchmark work. It evaluate image generator in a setting where multiple references composed under multiple and heterogeneous visual-instruction images. It consists 1,241 tasks with 777 reference images. It proposes three findings: (1) models face an adherence-artifact trade-off, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute, (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text.
Critical strengths
The explicit design of conflict tasks is interesting while the comparison conditions are not entirely rigorous.
Critical weaknesses
The evaluation score based on GPT-6 and Gemini and manual validation is limited. The evaluation results may be unfair and instable especially when GPT-Image is included in the evaluated generators.
Questions for authors
- Comparing to existing benchmarks, the authors should further explain their advancements, and these advancements must be substantial enough for the paper to be accepted again at a top-tier conference. For instance, Multi-Ref [R1].
[R1] Chen, R., Chen, D., Wu, S., Wang, S., Lang, S., Sushko, P., ... & Krishna, R. (2025, October). Multiref: Controllable image generation with multiple visual references. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 13325-13331).
-
The reference pool contains only 777 images, and 359 synthetic references are generated by only Nano Banana and GPT-Image families, limiting the diversity of source distributions and visual styles of the benchmark.
-
The construction pipeline relies on GPT-5 to propose labels. The potential construction bias is not sufficiently quantified and the generalization to natural controls is unclear.
-
Why is the comparison between visual and textual instructions only conducted on a 200-task subset, as it produce a main conclusion of this benchmark.
-
Are all 1,241 tasks include a layout instruction? If so, does this setting introduce a strong distributional bias?