acceptodds
Under review as a conference paper at ICLR 2027

IUX-GenBench: Benchmarking Interactive User Experience for Multi-Turn Image Generation

Abstract

Existing benchmarks for image generation and editing models have made considerable progress in evaluating static output quality across diverse tasks. Nevertheless, such evaluations tend to concentrate on isolated capabilities, thereby neglecting the dynamic interaction requirements and multifaceted scenarios that users encounter in practice. We therefore introduce **IUX-GenBench**, a **bench**mark for **i**nteractive **u**ser e**x**perience in multi-turn image **gen**eration and editing tasks. Our main contributions include: (i) constructing a hybrid dataset of UGC, PGC, and LLM-synthesized augmented data with a fine-grained taxonomy of capabilities and scenarios; (ii) introducing a multi-model collaborative simulator to replace static evaluations with dynamic assessment across simulated interaction loops; (iii) designing a dual-layer assessment system that jointly measures turn-level quality and session-level coherence; and (iv) proposing automated attribution analysis and structured diagnostic reporting as a novel evaluation practice that transcends scalar scoring to deliver interpretable, actionable failure diagnostics. Extensive experiments show that most models perform poorly on IUX-GenBench, with significant session-level degradation. Attribution analysis further reveals that unintended background alterations and ignored local constraints account for most session-level failures, highlighting persistent deficiencies in sustaining user intent and contextual coherence. By exposing these limitations, IUX-GenBench provides a demanding testbed for generative AI development.

Primary area:
datasets and benchmarks
Keywords:
Image Generation and Editing Benchmark, User Centric, Multi-turn Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cjd5OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark for evaluating interactive multi-turn image generation. The authors collected a dataset with both real-world and synthesized data, and annotated the data with a taxonomy of capabilities and scenarios. They have also introduced a multi-agent system to perform multi-turn image generation and an assessment system that delivers scores and failure attributions. This benchmark is run on several state-of-the-art commercial image generation methods.

Critical strengths

The presented benchmark is carefully designed and curated. It covers a diverse set of domains and tasks, which makes it somewhat generalizable. The evaluation pipeline is fully automated and produces assessments in multiple dimensions, which could be useful for the practical purpose of improving multi-turn image generation models. Compared to prior works, this paper presents an interactive benchmark rather than using a static data source. The ablations on using different evaluation models and human alignment are welcome. Overall, I think this is a useful benchmark.

Critical weaknesses

None

Questions for authors

Did the authors measure the soundness of the requested edits made by the supervisors in the loop in step 1?

Why is the line plot in Figure 4, right, the appropriate graph for scores per category? To my understanding, the category within each group (turn/session) is interchangeable, which means the trend is meaningless.

What is the cost of running this benchmark? Presumably higher than prior works due to the use of multiple agents in a loop during evaluation.

Suggestions:

  • “PGC” is mentioned in the abstract without being defined (I initially thought it meant “procedurally generated content”. It is clarified later, but if the definition were given at first mention, it would be clearer.
3 comments
AuthorsOpenReview ↗

Response to Reviewer cjd5

Rebuttal for Submission 2498: IUX-GenBench

To reviewer cjd5: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.

Q1. Soundness of supervisor-requested edits

Following the reviewer's feedback, we have conducted a subjective experiment to manually audit the judge outputs in the generation stage. We reused the sampling results from the experiment reported in Appendix D.1 (page 28, line 1482) and checked two intermediate iterations and one final iteration for 908 session-level samples. This amounts to 1,816 edit-instruction judgments and 908 stop-decision judgments. During the experiment, annotators observed only the generated image results and the judge's feedback for the corresponding turn; they could not access the full interaction history or model information. Annotators provided binary judgments for each edit instruction/stop decision, indicating whether they agreed. The approval rates for these two types of instructions were 94.27% and 91.52%, respectively.

Q2. Line plot in Figure 4

We clarify that Figure 4 (right) (page 9, line 443) is intended to compare several candidate models, rather than to show trends in model scores across evaluation dimensions. A grouped bar chart can be used in place of the line chart, and we have implemented this fix in the manuscript.

Q3. Running cost

Some cost statistics have been submitted to the anonymous code repository (page 2, line 81); here we provide a direct answer: the average consumption per model at each stage is shown in the table below.

StageAvg. API callsAvg. tokens (i/o)Avg. time
Generation1486017.8M / 19.6M69.77h
Generation (generator only)3031980.7k / 12.6M31.2h
Evaluation22097.6M / 901.4k12.8h
Attribution383561.9k / 80.77k1.12h
Report574986 / 5218111.3 sec

Because the evaluation loop uses multiple models, the running cost of this benchmark is higher than that of single-turn static evaluation. Specifically, in the generation stage, the supervisor/judge participates in each round of review and edit-instruction generation, making the generation stage the most expensive. Because API prices and model performance are not constant, the above cost estimates are for reference only; in practice, users can substitute lower-cost models to trade off cost against evaluation quality.

Suggestion. PGC undefined at first mention

We thank the reviewer for this suggestion. We have elaborated on "User-Generated Content (UGC)" and "Professional-Generated Content (PGC)" at their first mention in the abstract.

I thank the authors for the response and for providing these new results. I see that they have been included in the revised manuscript, and I have no further questions.

AuthorsOpenReview ↗

Response to Reviewer cjd5

We are glad that the new results have been incorporated into the revised manuscript and that the reviewer has no further questions. We sincerely thank the reviewer for taking the time to review our work and provide constructive suggestions, which have substantially improved the clarity and rigor of our paper.

Reviewer_WFFmOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents IUX-GenBench for evaluating interactive experience in multi-turn image generation and editing. The benchmark combines manually curated prompts from public web content, professional-scenario prompts, and human-reviewed LLM-augmented data, totaling 1,767 samples organized into ten application scenarios and seven capabilities. Evaluation uses multi-model supervision and adjudication-driven iterative generation, then separately assesses single-turn output quality and, within sessions, edit execution, non-edited region fidelity, and context consistency, with error attribution and diagnostic reports for low-scoring samples. Results on eight generative models reveal clear difficulties in multi-turn editing; a human preference study, evaluator substitution, and repeated-rating analysis support the validity of automatic scoring.

Critical strengths

Incorporates the multi-turn editing process into evaluation, filling a blind spot of single-output assessment. The benchmark jointly examines immediate output quality, preservation of unedited content, and consistency with historical requirements, with breakdowns by scenario and capability. It can reveal repeated repair and content drift that are hard to observe from single-generation results alone, offering a useful evaluation resource for interactive generation systems.

Provides multiple validations of automatic scoring rather than relying on a single evaluator model. The paper includes a 1,379-pair output preference study with ten annotators, reports correlation with human judgment, and checks stability via evaluator substitution, augmented data-source validation, and repeated-rating analysis. The appendix gives scoring rules, role prompts, and output structure, making the evaluation pipeline easier to understand and inspect. Critical Weaknesses

Critical weaknesses

The correspondence between simulated feedback and real user interaction is not sufficiently validated. Current interaction mainly centers on model correction of the initial request; real users' preference shifts, feedback expression, and tolerance are limitedly covered. The existing human preference study supports correlation between output scores and human judgment, but does not directly verify whether feedback instructions and stopping decisions resemble real users. The authors acknowledge this in limitations; since interactive experience is the paper's core positioning, this limitation still affects the scope of the conclusions.

Session sample selection may affect cross-model comparison and the interpretation of "multi-turn degradation." Session evaluation covers only samples with multi-turn records, and whether different models enter multi-turn depends on earlier generation and approval results; the aggregate metric also treats samples lacking session scores as session passes. Thus, each model's session score may correspond to a different difficulty set of samples, and single-turn and session metrics measure different dimensions. Comparing the two pass rates alone is not enough to show that the same samples generally degrade through iteration.

Reliability validation of error attribution and diagnostic reports remains insufficient. Existing human alignment experiments mainly test quality scoring, not directly the accuracy of automatic error classification, main failure-reason proportions, or recommendations. The attribution module also reads upstream scoring rationales and feedback, so it may inherit bias from prior judgments. Structured reports have practical potential, but current evidence supports "root causes" and "actionable recommendations" more weakly than the scores themselves.

Questions for authors

Can a limited number of existing sessions be used to manually check the simulator's correction instructions and stopping decisions, or to conduct a small-scale comparison with real user feedback? Can such results support calling it a proxy for interactive experience rather than just a model-supervised correction process? If evidence is limited, please narrow the conclusions accordingly.

What are the actual numbers of samples, turns, and reasons for missing data for each model entering session evaluation? Can multi-turn performance be compared on common samples, and can the impact of treating missing session scores as passes on aggregate ranking be explained? To argue iterative degradation, please distinguish task-selection effects from content-quality changes within the same session.

Can an independent manual review of error classification and main failure-reason proportions be conducted on a small number of existing low-scoring sessions, reporting typical attribution errors? Please distinguish observed failure symptoms from inferred causes; no full new model-optimization experiment is needed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WFFm (Part I)

Rebuttal for Submission 2498: IUX-GenBench

To reviewer WFFm: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.

W1 / Q1. Correspondence between simulated feedback and real user interaction

We thank the reviewer for this concern. Our current work focuses on image-generation scenarios in which user goals are clear. Although user's objective remain constant, the model's misinterpretation of the user's intent and the complexity of the generation requirements may necessitate multi-turn generation. And the setting of a clearly defined user goal facilitates the establishment of distinct evaluation criteria at both the turn-level and the session-level. We have not yet considered scenarios in which user goals change freely or exploratory creative workflows, because this will lead unclear and ununified evaluation standard. We consider this a promising direction for future work and have added a discussion in Appendix E (page 32, line 1680).

In addition, we have conducted a subjective experiment to manually audit the judge outputs in the generation stage. We reused the sampling results from the experiment reported in Appendix D.1 (page 28, line 1482) and checked two intermediate iterations and one final iteration for 908 session-level samples. This amounts to 1,816 edit-instruction judgments and 908 stop-decision judgments. During the experiment, annotators observed only the generated image results and the judge's feedback for the corresponding turn; they could not access the full interaction history or model information. Annotators provided binary judgments for each edit instruction/stop decision, indicating whether they agreed. The approval rates for these two types of instructions were 94.27% and 91.52%, respectively. Combined with the user feedback collected in our initial survey and dataset construction, we believe this result is sufficient to support calling it a proxy for interactive experience in weakly personalized usage scenarios (with exceptions noted above).

AuthorsOpenReview ↗

Response to Reviewer WFFm (Part II)

W2 / Q2. Session sample selection

Thanks for the comments. It should be noted that the session-level statistics reported in Table 1 (page 5, line 217) are used to describe each model's performance on the corresponding evaluation dimension. These statistics are not used for cross-model comparison. The comprehensive pass rate (Comp pr) metric comprehensively measures model capability by assessing all valid dimensions (regardless of whether they are turn-level or session-level) and is used for horizontal comparison.

For session-level pass-rate statistics, we currently compute results on a separate subset consisting of samples that actually entered the session stage for each model. This approach effectively reflects the model's performance on the corresponding dimension.

  • Why not use a single unified subset for session-level statistics?

We considered three ways to construct a unified statistics:

  1. Intersection subset: samples that entered the session stage for all models. However, this subset is very small (44 samples) and is not statistically meaningful.
  2. Union subset: samples that entered the session stage for at least one model. This subset has 1,256 samples and is essentially similar to the Comp pr statistics (the denominator subtracts single-turn samples on all models and a few API-call failures). Moreover, if new models are introduced, the subset must be recomputed, which is not practical.
  3. Full-set statistics: this is consistent with Comp pr and therefore redundant.

The table below provides results for session statistics on different subsets.

ModelSession-entry samplesSession pass rate (original/separate subset)Session pass rate (intersection subset)Session pass rate (union subset)Comp pr (full-set)
GPT-Image-219859.60%13.64%64.79%69.43%
GPT-Image-1.531728.08%0.00%43.32%52.02%
Gemini-3-Pro-Image-Preview47241.31%2.27%39.27%47.70%
Gemini-3.1-Flash-Image-Preview45247.12%0.00%39.51%46.67%
Doubao-Seedream-5-037824.07%2.27%33.86%41.45%
GPT-Image-159521.51%0.00%30.13%41.34%
Qwen-Image-2.062920.83%2.27%21.78%31.62%
Gemini-2.5-Flash-Image74616.35%0.00%16.22%26.63%
  • How does Comp pr reflect overall model capability? Why are samples lacking session evaluation treated as passing by default?

The current comprehensive pass rate is defined as the proportion of samples that pass all valid dimensions. We believe this metric is reasonable as an overall capability indicator: if a model can return a satisfactory result from the first-turn input, this itself reflects strong model capability; models that enter subsequent rounds can also resolve problems through strong long-horizon repair ability, thereby compensating for an insufficient first-turn result. The comprehensive pass rate therefore rewards both capability paths: direct first-turn success and subsequent repair success.

  • How should "multi-turn degradation" be interpreted?

It should be noted that because samples passing the first turn do not enter subsequent rounds, we cannot use the before/after performance of the same samples to directly demonstrate that the model exhibits "iterative degradation". We also clarify that the core argument of this paper is not that model performance degrades through iteration on given samples, but that turn-level and session-level evaluations have different emphases: turn-level evaluation measures single-turn generation quality, while session-level evaluation measures the model's capability and consistency in multi-turn dialogue. The comparison between the two reflects the model's relative insufficiency in multi-turn scenarios. To eliminate ambiguity, we have revised the "degradation" wording in the manuscript to "session-level performance gap" (Section 1, page 2, line 68/73).

Iteration counts. The table below reports partial iteration-distribution statistics and relevant session-level average scores for each model.

ModelMean iterationsIteration quartiles Q1 / M / Q3Session-level average score for different iterations(2/5/10)
GPT-Image-24.002 / 3 / 44.823 / 4.400 / 3.287
GPT-Image-1.54.792 / 3 / 84.445 / 4.476 / 2.542
Gemini-3-Pro-Image-Preview4.422 / 3 / 64.524 / 3.965 / 2.395
Gemini-3.1-Flash-Image-Preview4.442 / 3 / 64.672 / 3.729 / 2.803
Doubao-Seedream-5-05.172 / 3 / 104.292 / 3.511 / 1.991
GPT-Image-16.072 / 5 / 104.523 / 4.250 / 2.332
Qwen-Image-2.06.252 / 8 / 104.486 / 1.741 / 1.028
Gemini-2.5-Flash-Image6.233 / 6 / 104.133 / 3.286 / 2.458
AuthorsOpenReview ↗

Response to Reviewer WFFm (Part III)

Reasons for missing data. Missing or non-applicable data at each stage arises as follows:

StageMissing / non-applicable reason
GenerationAPI call failure
Turn evaluationAPI call failure; Generation failure
Session evaluationAPI call failure; Generation failure, or only a single turn (first-turn pass; no session entered)
AttributionOnly valid low-scoring samples are processed

W3 / Q3. Reliability of attribution and diagnostic reports

To validate the reliability of the model outputs in the attribution stage, we have conducted an additional subjective experiment. Annotators were provided with sampled low-scoring sessions, the evaluator's results and comments, and the predefined error taxonomy, and were asked to select the primary error category. We then compared their choices with the conclusions of the attribution model. The definition of low-scoring samples follows Section 4.3 (line 370). For each model, we sampled 10 low-scoring turn-level cases and 10 low-scoring session-level cases, totaling 160 items. The table below shows the accuracy of the attribution model and two reference baselines evaluated against human attribution conclusions. It can be observed that the attribution model shows substantial improvement over the majority-class baseline, indicating meaningful agreement with human judgment. At the same time, we have also softened some overly strong statements in the manuscript. For example, "root cause" has been replaced with "possible failure factor" (Section 4.3, page 8, line 390).

MethodTurnSession
Random baseline0.25 (4 classes)0.33 (3 classes)
Majority-class baseline0.58750.6875
Attribution model0.73750.7625

The table below further shows the distribution of primary error categories among attributed samples for each model in the complete experimental records. Referring to Appendix B.3 (Table 5, page 23, line 1204), the primary error categories are logically inferred causes, while the secondary error categories can be described as observed failure symptoms. The two are explicitly separated.

ModelTurn-attributed samplesText-Image ConsistencySemantic Understanding & ReasoningGeneration Quality & AestheticsInstruction Compliance & ConstraintsSession-attributed samplesIterative & Editing CapabilityDialogue & Intent AlignmentWorkflow & Task Completion
GPT-Image-23810121603524110
GPT-Image-1.582442315011777391
Gemini-3-Pro-Image-Preview915222143165131331
Gemini-3.1-Flash-Image-Preview935027160129101271
Doubao-Seedream-5-01426351262208163414
GPT-Image-1160107331732951811113
Qwen-Image-2.0242159304494192301845
Gemini-2.5-Flash-Image354293431534931523338

Finally, regarding diagnostic suggestions, their validation requires domain experts to judge and verify them in actual production practice; we currently lack the conditions for validation. Therefore, we have explicitly adjusted the relevant wording in the manuscript and updated Appendix E (page 32, line 1718): these are diagnostic hypotheses, rather than validated post-training prescriptions.

AuthorsOpenReview ↗

We are wondering if the reviewer have any other questions

Dear Reviewer WFFm,

Thank you again for your constructive comments. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.

Following your suggestions, we have clarified the rationale for fixing the initial prompt and user goal, distinguished session-level statistics from the comprehensive pass rate (Comp pr), explained the construction of session-level subsets, reported iteration-distribution statistics and revised the "degradation" wording to "session-level performance gap", added subjective experiments auditing the judge outputs and the reliability of attribution, softened the diagnostic-report claims to "possible failure factors" and "diagnostic hypotheses", expanded the Limitations and Future Work section (Appendix E and others).

We would be happy to clarify any remaining questions and concerns.

Best Regards,

The Authors

Reviewer_PB4wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces IUX-GenBench, a benchmark for evaluating text-to-image (T2I) generation and multi-turn image editing models in an interactive loop. The dataset consists of 1,767 bilingual (Chinese and English) prompts spanning 10 application scenarios and 7 technical capability dimensions. It combines 713 manually rewritten prompts inspired by user-generated and professionally generated web content with 1,220 LLM-synthesized prompts, filtered through manual quality inspection. Rather than scoring single-turn outputs in isolation, the authors propose a four-stage evaluation pipeline. In Stage 1, a target model generates an initial image and iteratively edits it for up to 10 turns based on modification instructions from three MLLM supervisors (Gemini 3.5 Flash, Qwen3.6 Flash, and GPT-5.4 mini) aggregated via majority vote by a judge model (Gemini 2.5 Pro). In Stage 2, an MLLM evaluator (Gemini 3.5 Flash) scores the interaction history on a 1 to 5 scale across four turn-level metrics (Instruction Adherence, Visual Quality, Logic and Physics Reasoning, Text Rendering) and three session-level metrics (Edit Execution, Non-Edit Fidelity, Context Consistency), alongside a dedicated safety rubric for 20 ethics and safety samples. In Stages 3 and 4, low-scoring trajectories are categorized into a predefined error taxonomy and summarized via a five-step MLLM prompting protocol into a diagnostic report. Evaluating eight models, the authors show that while recent models achieve decent turn-level pass rates, session-level pass rates drop substantially, largely driven by local edit execution failures and unintended alterations to background regions. The authors validate their Stage 2 evaluator against human pairwise preferences on 1,379 image pairs and verify ranking consistency across three alternative MLLM evaluators.

Critical strengths

1.Practical closed-loop evaluation uniting initial generation and iterative editing. Existing benchmarks largely treat text-to-image synthesis and instruction-guided editing as separate single-turn tasks. Evaluating models in a closed loop up to 10 turns captures a common real-world workflow where a user tries to repair an imperfect initial generation through follow-up edits. The empirical contrast in Table 1 and Section 5.3 showing that nearly all evaluated models suffer a steep drop from turn-level quality to session-level coherence (with models frequently altering unedited background areas or forgetting earlier constraints during repair cycles) highlights a concrete limitation in current multimodal generators. 2.Thorough validation of the MLLM evaluation rubric against human judgments and alternative judges. Because the benchmark relies on Gemini 3.5 Flash for Stage 2 scoring, verifying that the resulting leaderboard is not an artifact of evaluator bias or API stochasticity is crucial. The authors substantiate their scoring protocol with a human preference study over 1,379 pairs across ten annotators (reaching 0.895 turn-level and 0.933 session-level Spearman rank correlation), cross-check rankings against GPT-5.4 mini, Kimi-K2.5, and Qwen3.6 Flash, and quantify multi-run API variance in Appendix D. This validation provides solid empirical support for the reliability of the Stage 2 scores.

Critical weaknesses

1.Discrepancy between the claimed interactive user intent simulation and the actual static-prompt correction loop, which induces selection bias in session-level metrics. In Section 1 and Section 2.3, the paper motivates IUX-GenBench by arguing that real users communicate goals incrementally and indirectly rather than providing fully specified prompts upfront. However, inspecting the dataset construction (Appendix A.3) and the Supervisor and Judge system prompts (Appendix B.1, lines 1007 to 1062) reveals that the entire interaction loop is governed by a single, fully specified initial prompt provided at Turn 1. At each step, the supervisors simply check whether the generated image matches all details of that static initial prompt and issue repair instructions if anything is missing. Consequently, the simulator does not test evolving user goals or exploratory creative workflows; it strictly tests multi-turn self-correction toward a fixed initial specification. More importantly, this setup introduces a confounding selection bias into the session-level metrics in Table 1. As stated in Section 5.1 (lines 410 to 413) and the caption of Figure 11, samples that pass on Turn 1 do not undergo multi-turn editing; they are excluded from the session-level denominator and automatically assumed to pass session-level assessment when computing the Comprehensive Pass Rate. As a result, each model's session-level scores (EE, NF, CC, and session pass rate) are evaluated on a different subset of prompts—namely, only the prompts that the model failed to generate correctly on Turn 1. Stronger models that solve easier prompts on Turn 1 are evaluated at the session level only on their hardest remaining failures, making raw session-level comparisons across models difficult to interpret without controlling for Turn 1 approval rates. 2.AI pre-population bias and very small per-skill sample sizes in the full evaluation suite (Sections 5 and 6). First, Appendix C (lines 965 to 968) notes that to accelerate annotation in Section 5, all BQA answers were initialized by a VLM and displayed to human annotators as AI suggestions to confirm or correct. Because 11 of the 18 skills rely on BQA, showing VLM predictions upfront introduces strong anchoring and confirmation bias, which confounds the human score convergence analysis in Figure 5 for those skills. Second, the full suite in Section 5 comprises only 43 prompts total (Appendix F, Table 6), resulting in several skills having only 3 to 6 prompts and 3 to 6 validation questions in total (for example, View has 3 questions, Camera has 4, Mood/Feeling has 4, and Language Complexity has 6). Benchmarking four models and computing per-skill VLM-human Spearman correlations (Figure 6, where multiple bars had to be omitted due to constant predictions) on 3 to 4 questions per skill is statistically fragile.

Questions for authors

1.Impact of artifact score normalization (Section 4.1 and Table 2): If you compute Krippendorff's alpha and EDR for the brush protocol after normalizing the artifact pixel ratios by the empirical range observed across the 90 images (e.g., min-max or percentile scaling, or normalizing by foreground/union mask area rather than total image pixels), what are the resulting alpha and EDR values? Additionally, if you report spatial overlap (such as pairwise IoU or Dice coefficient across annotators' brush masks) directly, how well do annotators agree on exact localization? 2.Distribution of iteration counts across models: In Stage 1, the simulator runs up to a cap of 10 iterations. What is the distribution of total turns taken per sample across the evaluated models? Since Stage 2 evaluates the full interaction history, does a model that fails early and accumulates visual degradation over 10 repeated retry attempts get penalized more heavily on Non-Edit Fidelity and Context Consistency than a model that stops after 2 turns? 3.Scoring of Text Rendering (TR) on prompts without text: In Section 4.3 and Section 5.1, a sample only counts as passing if every dimension receives a perfect score of 5. In Table 1, all models achieve high turn-level TR pass rates between 75.3% and 95.9% across the full dataset, whereas in Table 3, pass rates on the Text Rendering and Layout Control (TRC) capability subset are much lower (20.2% to 75.4%). Does the evaluator assign a default score of 5 for TR when a prompt does not call for embedded text, and how many prompts in the benchmark actually require text generation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer PB4w (Part I)

Rebuttal for Submission 2498: IUX-GenBench

To reviewer PB4w: Thanks for the constructive comments. Responses to specific comments are given below, which have also been incorporated into the revised paper.

W1. Fixed initial prompt & Session-level selection

Fixed initial prompt. Thanks for the concern. Our current work focuses on image-generation scenarios in which users' goals are clear. Although user's objective remain constant, the model's misinterpretation of the user's intent and the complexity of the generation requirements may necessitate multi-turn generation. And the setting of a clearly defined user goal facilitates the establishment of distinct evaluation criteria at both the turn-level and the session-level. We have not yet considered scenarios in which user goals change freely or exploratory creative workflows, because this will lead unclear and ununified evaluation standard. We consider this a promising direction for future work and have added a discussion in Appendix E (page 32, line 1680).

Session-level selection. Thanks for the comments. It should be noted that the session-level statistics reported in Table 1 (page 5, line 217) are used to describe each model's performance on the corresponding evaluation dimension. These statistics are not used for cross-model comparison. The comprehensive pass rate (Comp pr) metric comprehensively measures model capability by assessing all valid dimensions (regardless of whether they are turn-level or session-level) and is used for horizontal comparison.

For session-level pass-rate statistics, we currently compute results on a separate subset consisting of samples that actually entered the session stage for each model. This approach effectively reflects the model's performance on the corresponding dimension.

  • Why not use a single unified subset for session-level statistics?

We considered three ways to construct a unified statistics:

  1. Intersection subset: samples that entered the session stage for all models. However, this subset is very small (44 samples) and is not statistically meaningful.
  2. Union subset: samples that entered the session stage for at least one model. This subset has 1,256 samples and is essentially similar to the Comp pr statistics (the denominator subtracts single-turn samples on all models and a few API-call failures). Moreover, if new models are introduced, the subset must be recomputed, which is not practical.
  3. Full-set statistics: this is consistent with Comp pr and therefore redundant.

The table below provides results for session statistics on different subsets.

ModelSession-entry samplesSession pass rate (original/separate subset)Session pass rate (intersection subset)Session pass rate (union subset)Comp pr (full-set)
GPT-Image-219859.60%13.64%64.79%69.43%
GPT-Image-1.531728.08%0.00%43.32%52.02%
Gemini-3-Pro-Image-Preview47241.31%2.27%39.27%47.70%
Gemini-3.1-Flash-Image-Preview45247.12%0.00%39.51%46.67%
Doubao-Seedream-5-037824.07%2.27%33.86%41.45%
GPT-Image-159521.51%0.00%30.13%41.34%
Qwen-Image-2.062920.83%2.27%21.78%31.62%
Gemini-2.5-Flash-Image74616.35%0.00%16.22%26.63%
  • How does Comp pr reflect overall model capability? Why are samples lacking session evaluation treated as passing by default?

The current comprehensive pass rate is defined as the proportion of samples that pass all valid dimensions. We believe this metric is reasonable as an overall capability indicator: if a model can return a satisfactory result from the first-turn input, this itself reflects strong model capability; models that enter subsequent rounds can also resolve problems through strong long-horizon repair ability, thereby compensating for an insufficient first-turn result. The comprehensive pass rate therefore rewards both capability paths: direct first-turn success and subsequent repair success.

AuthorsOpenReview ↗

Response to Reviewer PB4w (Part II)

W2. Sample size

It should be noted that the number of prompts involved in our study is not 43 but 1,767 (including 20 ESP samples). Regarding the reviewer's concern about sample size, dataset construction is based on pre-collected UGC/PGC portions and LLM-augmented samples; for a benchmark, this total is sufficient. We also clarify that we did not use BQA, and all LLM annotations of dataset samples underwent complete manual review, as detailed in Section 3.3 (page 5, line 267), rather than in Appendix C (lines 965--968). It should also be noted that Appendix C describes the preliminary analysis of experimental results, whereas lines 965--968 describe part of the rules for manual verification of data quality, not the acceleration of annotation. Finally, this study does not yet involve the construction of agent skills. In the actual benchmarking, each sample for each model must undergo turn-level evaluation and, where applicable, session-level evaluation; this process does not correspond to 3--6 validation questions, but is implemented through the complete evaluation system prompt (Appendix B.2, page 21, line 1091).

Q1. Annotator consistency

We clarify that we did not use empirical range to process images. In response to the reviewer's concern about annotator consistency in the subjective experiments, we have added the calculation of Krippendorff's α\alpha. As our subjective study uses pairwise preference annotations, which do not directly support computing α\alpha on raw scores. We therefore apply a principled transformation: preferences are first converted to scores, then z-standardized, and Krippendorff's α\alpha is computed with an interval metric. Across turn-level and session-level, the average α\alpha is 0.601.

To eliminate potential bias introduced by the preference-to-score transformation, we additionally computed ICC(2,k), obtaining 0.934 (missing values were imputed with column means, with an imputation rate below 5%). This indicates high consistency among annotators in ranking samples by relative quality; the gap between alpha and ICC mainly reflects differences in annotation scale rather than fundamental disagreement about quality direction.

Q2. Iteration counts

The table below reports partial iteration-distribution statistics and related session-level average scores. Notably, although 10-iteration samples have lower average session scores, this is not essentially because more rounds lead to a heavier penalty. As shown in the evaluator's system prompt (Appendix B.2, page 22, line 1135), we did not explicitly write an instruction assigning lower scores to larger numbers of turns. One root cause of the low scores is that the generation model has insufficient long-horizon dialogue capability, which prevents the model from correctly handling cumulative visual degradation; consequently, the evaluator gives corresponding scores after analyzing the complete interaction history.

ModelMean iterationsIteration quartiles Q1 / M / Q3Session-level average score for different iterations(2/5/10)
GPT-Image-24.002 / 3 / 44.823 / 4.400 / 3.287
GPT-Image-1.54.792 / 3 / 84.445 / 4.476 / 2.542
Gemini-3-Pro-Image-Preview4.422 / 3 / 64.524 / 3.965 / 2.395
Gemini-3.1-Flash-Image-Preview4.442 / 3 / 64.672 / 3.729 / 2.803
Doubao-Seedream-5-05.172 / 3 / 104.292 / 3.511 / 1.991
GPT-Image-16.072 / 5 / 104.523 / 4.250 / 2.332
Qwen-Image-2.06.252 / 8 / 104.486 / 1.741 / 1.028
Gemini-2.5-Flash-Image6.233 / 6 / 104.133 / 3.286 / 2.458
AuthorsOpenReview ↗

Response to Reviewer PB4w (Part III)

Q3. Evaluation dimensions vs. Capability dimensions

We clarify that evaluation dimensions and capability dimensions are two distinct concepts in this study.

  • For evaluation dimensions, a sample is considered passing as long as it satisfies the relevant evaluation criterion; if the prompt does not specify such a requirement, it passes by default. Currently, for prompts without text-editing requirements, the default score for Text Rendering (TR) is 5. In Table 1 (page 5, line 217), TR statistics use the number of samples with a single-dimension score of 5 as the numerator and all valid turn-level samples (~1,747) as the denominator. This is consistent with the relatively high overall pass rate for this dimension and indicates that the models can reasonably complete user-specified text-rendering requirements in single-turn generation; it does not represent the models' actual TR capability ranking.

  • For capability dimensions, capability labels describe the nature of the sample (input prompt) and are not individually and specifically evaluated. In our evaluation set, there are 669 samples labeled with Text Rendering and Layout Control (TRC). In Table 3 (page 7, line 325), TRC statistics use only samples that pass all valid dimensions as the numerator and samples whose capability label contains TRC as the denominator. Because each sample does not necessarily have only one capability-dimension label, the varied performance across models reflects gaps in comprehensive capability.

  • Given that the two are easily confused in naming, we have explicitly added relevant clarification in the manuscript (Section 3.2, page 5, line 238). In addition, finer-grained evaluation data construction and evaluation-dimension design based on capability-dimension labels can be future work; the relevant discussion will be updated in Appendix E (page 32, line 1706).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses & Paper Revisions

We sincerely thank all the reviewers for their valuable and constructive reviews. We appreciate that the reviewers acknowledged the significance and strengths of this work, including its novel closed-loop benchmark design for multi-turn image generation, the multiple and thorough validations of automatic scoring, and the extensive evaluation of state-of-the-art models. These acknowledgments have encouraged us to further clarify the benchmark design, strengthen the empirical validation, and soften overly strong claims in the manuscript.

The major revisions are summarized as follows.

  1. Fixed initial prompt. We clarified the rationale for fixing the initial prompt and user goal: this design enables reproducible and controllable turn-level and session-level evaluation criteria, while multi-turn interaction may still arise from model misinterpretation and complex generation requirements. We also discussed personalized tolerance modeling and exploratory creative workflows as future work.

  2. Session-level statistics vs. the comprehensive pass rate. We distinguished session-level statistics from the comprehensive pass rate (Comp pr). Session-level statistics are computed on model-specific session-entry subsets and are not used for cross-model comparison, whereas Comp pr evaluates all valid dimensions and is used for horizontal comparison. We also added comparisons across intersection, union, and full-set constructions.

  3. Additional manual audits and statistics. We added a subjective audit of the judge outputs in the generation stage (1,816 edit-instruction judgments and 908 stop-decision judgments, with approval rates of 94.27% and 91.52%), an attribution-reliability experiment against human judgments with random and majority-class baselines, annotator-consistency analyses (Krippendorff’s α and ICC), and iteration-count distribution statistics.

  4. Evaluation dimensions vs. technical capability labels. We explicitly distinguished evaluation dimensions from technical capability labels, using TR and TRC as an illustrative example, and discussed finer-grained evaluation design based on capability labels as future work.

  5. Softened wording. We revised "(session-level) degradation" to "session-level performance gap", replaced "root cause" with "possible failure factor", and described diagnostic suggestions as diagnostic hypotheses rather than validated post-training prescriptions.

  6. Complete manuscript updates. We incorporated the above discussions throughout the manuscript, expanded the Limitations and Future Work section (Appendix E), and made additional revisions, including replacing the line chart in Figure 4 with a grouped bar chart and expanding "User-Generated Content (UGC)" and "Professional-Generated Content (PGC)" at their first mention in the abstract.

The major revised contents in the manuscript are highlighted in blue. Point-for-point responses to specific comments are given in the following reviewer-specific responses. We welcome any further discussions and will address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.