- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces the "MIB Sliver and Gold Set" benchmark, along with an evaluator (MIE) designed to assess multi-subject image generation.
Critical strengths
- The paper proposes a benchmark for evaluating customized multi-subject image generation.
Critical weaknesses
-
The overall academic readability is poor; references are incomplete, and the flowcharts are ambiguous. This work bears the hallmarks of being generated by AI.
-
The data source is too limited, covering only two models. It is recommended to incorporate more data for both evaluation and the training of the evaluator; otherwise, the system risks learning to favor specific models rather than the quality of the generated images themselves.
Questions for authors
-
The paper appears heavily AI-generated, and the motivation is weak—it essentially serves only to provide an evaluation framework.
-
There is a complete lack of supplementary material or visualizations of images produced by the generative models, making it impossible to assess the effectiveness of the proposed evaluator.
-
The data sources are too sparse, originating only from "Nano Banana" and "MOSAIC"—furthermore, there is no citation provided for "MOSAIC."
Response to Reviewer f2po
Thank you for the review. A revised PDF has been uploaded. We respond to each question below.
Q1: Writing and motivation
We revised the full manuscript for readability. The Introduction has been rewritten to state the motivation and significance more directly. In short, current metrics cannot reliably judge multi-subject generation: on MIB-Gold, PickScore and HPS v2.1 agree with human preference only 48.6% and 52.0% of the time, close to chance (Sec. 4.1). In contrast, MIE, trained on MIB-Silver, agrees with human preference on 92.2% of pairs, including 88.4% on generators unseen in training (Sec. 4.2). Using MIE, we find that even the strongest current generator (GPT-Image-1.5) satisfies all three dimensions in only 52.4% of images (Table 3). We also redrew the overview diagram (Fig. 2, p. 3). It now shows the candidate pair explicitly and omits non-essential numbers, so the main pipeline is easier to follow.
Our use of AI tools, including editing for readability, is disclosed in the AI Use Statement (p. 11) in accordance with ICLR policy. The research, experiments, and claims are the authors' own.
Q2: Visualizations
The submission includes visualizations of generated images. Because of the page limit, and since our contribution is a benchmark and an evaluator rather than a new generation method, they appear in the appendix:
In the revision, the Introduction and Sec. 4 cite these figures where each failure mode is discussed (Figs. 5–7), so they are easier to find.
Q3: Data sources and MOSAIC citation
MOSAIC is now cited wherever it is used, and we rechecked the rest of the bibliography.
Only training uses two generators; evaluation uses six.
If MIE had only learned to favor its training generators, it would have nothing to rely on in these unseen pairs. The results show that it still generalizes:
We agree that more training generators would make MIE more robust, and these results do not fully rule out generator-specific shortcuts. The revised Limitations paragraph (Sec. 4) now states that MIB-Silver relies on two training generators and dual-VLM supervision, which may retain residual generator priors.