- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a benchmark for evaluating interactive multi-turn image generation. The authors collected a dataset with both real-world and synthesized data, and annotated the data with a taxonomy of capabilities and scenarios. They have also introduced a multi-agent system to perform multi-turn image generation and an assessment system that delivers scores and failure attributions. This benchmark is run on several state-of-the-art commercial image generation methods.
Critical strengths
The presented benchmark is carefully designed and curated. It covers a diverse set of domains and tasks, which makes it somewhat generalizable. The evaluation pipeline is fully automated and produces assessments in multiple dimensions, which could be useful for the practical purpose of improving multi-turn image generation models. Compared to prior works, this paper presents an interactive benchmark rather than using a static data source. The ablations on using different evaluation models and human alignment are welcome. Overall, I think this is a useful benchmark.
Critical weaknesses
None
Questions for authors
Did the authors measure the soundness of the requested edits made by the supervisors in the loop in step 1?
Why is the line plot in Figure 4, right, the appropriate graph for scores per category? To my understanding, the category within each group (turn/session) is interchangeable, which means the trend is meaningless.
What is the cost of running this benchmark? Presumably higher than prior works due to the use of multiple agents in a loop during evaluation.
Suggestions:
- “PGC” is mentioned in the abstract without being defined (I initially thought it meant “procedurally generated content”. It is clarified later, but if the definition were given at first mention, it would be clearer.
3 comments
I thank the authors for the response and for providing these new results. I see that they have been included in the revised manuscript, and I have no further questions.
Response to Reviewer cjd5
We are glad that the new results have been incorporated into the revised manuscript and that the reviewer has no further questions. We sincerely thank the reviewer for taking the time to review our work and provide constructive suggestions, which have substantially improved the clarity and rigor of our paper.
Response to Reviewer cjd5
Rebuttal for Submission 2498: IUX-GenBench
To reviewer cjd5: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.
Q1. Soundness of supervisor-requested edits
Following the reviewer's feedback, we have conducted a subjective experiment to manually audit the
judgeoutputs in the generation stage. We reused the sampling results from the experiment reported in Appendix D.1 (page 28, line 1482) and checked two intermediate iterations and one final iteration for 908 session-level samples. This amounts to 1,816 edit-instruction judgments and 908 stop-decision judgments. During the experiment, annotators observed only the generated image results and the judge's feedback for the corresponding turn; they could not access the full interaction history or model information. Annotators provided binary judgments for each edit instruction/stop decision, indicating whether they agreed. The approval rates for these two types of instructions were 94.27% and 91.52%, respectively.Q2. Line plot in Figure 4
We clarify that Figure 4 (right) (page 9, line 443) is intended to compare several candidate models, rather than to show trends in model scores across evaluation dimensions. A grouped bar chart can be used in place of the line chart, and we have implemented this fix in the manuscript.
Q3. Running cost
Some cost statistics have been submitted to the anonymous code repository (page 2, line 81); here we provide a direct answer: the average consumption per model at each stage is shown in the table below.
Because the evaluation loop uses multiple models, the running cost of this benchmark is higher than that of single-turn static evaluation. Specifically, in the generation stage, the supervisor/judge participates in each round of review and edit-instruction generation, making the generation stage the most expensive. Because API prices and model performance are not constant, the above cost estimates are for reference only; in practice, users can substitute lower-cost models to trade off cost against evaluation quality.
Suggestion. PGC undefined at first mention
We thank the reviewer for this suggestion. We have elaborated on "User-Generated Content (UGC)" and "Professional-Generated Content (PGC)" at their first mention in the abstract.