- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
IMUG-Bench evaluates unified multimodal models through multi-turn dialogues that interleave image understanding, generation, and editing. It contains 3,113 samples and 12,034 turns across 19 domains and 97 tasks. A distinctive component is Dynamic-MCQ: the correct answer is determined from the model's own generated image, allowing understanding to be assessed even when generation deviates from the intended target. Eight models are evaluated using a VLM judge and structured evaluation points. The paper reports substantial generation degradation over dialogue turns and explores chain-of-thought, self-verification, and best-of-N sampling on BAGEL.
Critical strengths
- The evaluation setting addresses a meaningful gap. Combining generation, editing, and questions about self-generated content tests dependencies that isolated single-turn evaluations miss. Dynamic-MCQ is particularly useful because it separates understanding of the actual image from successful execution of an earlier generation instruction. The breadth of manually verified task templates makes this a potentially reusable resource.
- The evaluation is more carefully specified than a purely qualitative benchmark. The paper supplies structured scoring points and judge prompts, evaluates multiple model families, and includes human validation and formatting sensitivity analyses in the appendix. These details support reproducibility and make the observed capability gaps informative.
Critical weaknesses
-
Section 4.3 and Table 3 show generation scores falling over dialogue turns, but task composition and difficulty also change. These aggregates cannot determine how much degradation comes from errors in earlier outputs. The exposure-bias claim needs a paired comparison of the same target turn under model-generated and corrected histories.
-
Table 1 compares benchmark features, but the paper does not relate its generation rankings to those on existing generation benchmarks. It remains unclear whether IMUG-Bench reveals differences in multi-turn robustness beyond the models' single-turn generation strength.
Questions for authors
-
On a human-verified subset, could you compare the same target turn using the model-generated image history and a minimally corrected reference history? Match target instructions, sampling settings, and scoring. Keep text history fixed where consistent, and check that referents and Dynamic-MCQ labels remain valid. Report paired score changes and uncertainty by turn and task type. Does correcting earlier outputs recover later-step performance?
-
For overlapping model versions, compare generation-only rankings with GenEval or DPG-Bench under comparable inference budgets, separating text-to-image generation from editing. Report rank correlation and notable inversions, accounting for uncertain ties. Within IMUG-Bench, do single-turn, free-running multi-turn, and reference-history results distinguish models with similar conventional generation scores?
5 comments
Response to Reviewer VRZq (Part 2/2)
R2. Generation Benchmark Rankings and Evaluation Settings (Weakness 2 & Question 2)
We separately report IMUG-Bench Text-to-Image (T2I) and Image Editing scores and compare them with GenEval, DPG-Bench, WISE, ImgEdit-Bench, and GEdit-Bench to identify capability differences not fully captured by existing benchmarks.
IMUG T2I aggregates text-to-image tasks without editing historical images; IMUG Editing aggregates tasks that modify historical images. Both use the corresponding task subsets from the original evaluation. Full Free-Running results, covering interleaved understanding and generation, appear in Table 2 of the original manuscript. For fairness, the rankings below include only the same six open-source models.
Table 2. Comparison with existing text-to-image generation benchmarks.
| Model | GenEval | DPG-Bench | WISE | IMUG T2I |
|---|---|---|---|---|
| BAGEL | 0.82 | 85.07 | 0.52 | 69.4 |
| BLIP3-o 4B | 0.81 | 79.36 | 0.50 | 81.1 |
| BLIP3-o 8B | 0.83 | 80.73 | 0.52 | 82.9 |
| OmniGen2 | 0.80 | 83.57 | 0.47 | 53.8 |
| Ovis-U1 | 0.89 | 83.72 | 0.42 | 60.1 |
| UniWorld-V1 | 0.80 | 81.38 | 0.55 | 81.0 |
Table 3. Comparison with existing image editing benchmarks.
| Model | ImgEdit-Bench | GEdit-Bench-EN | IMUG Editing |
|---|---|---|---|
| BAGEL | 3.20 | 6.52 | 62.8 |
| BLIP3-o 4B | 2.44 | 2.34 | 54.8 |
| BLIP3-o 8B | 2.40 | 2.73 | 60.6 |
| OmniGen2 | 3.44 | 6.41 | 54.4 |
| Ovis-U1 | 4.00 | 6.42 | 75.3 |
| UniWorld-V1 | 3.26 | 4.85 | 60.7 |
(1) Benchmark ranking correlations. As shown in Tables 2–3, the Spearman rank correlations between IMUG T2I and GenEval, DPG-Bench, and WISE are 0.174, −0.714, and 0.580, respectively. For IMUG Editing, the correlations with ImgEdit-Bench and GEdit-Bench are 0.314 and 0.600. Notably, existing generation benchmarks also show substantial ranking differences: the correlations of GenEval with DPG-Bench and WISE are 0.232 and −0.353, respectively, while that between DPG-Bench and WISE is −0.203. These results indicate that benchmarks emphasize different aspects of model performance, and ranking correlation alone does not determine benchmark validity.
(2) Ranking reversals. Model rankings differ substantially across evaluation settings. For text-to-image generation, Ovis-U1 ranks first on GenEval but only fifth on IMUG T2I. For image editing, OmniGen2 ranks second on ImgEdit-Bench but drops to sixth on IMUG Editing. In the full multi-turn Free-Running evaluation, BAGEL ranks first on DPG-Bench but only fourth in IMUG Free-Running generation. These ranking reversals show that IMUG-Bench captures model capability differences related to multi-turn interaction and historical dependencies that are not reflected in conventional generation benchmarks.
(3) Distinguishing models with similar scores. OmniGen2 and UniWorld-V1 obtain identical GenEval scores (0.80), but rank sixth and third, respectively, in the full Free-Running generation evaluation. For image editing, OmniGen2 and Ovis-U1 achieve nearly identical GEdit-Bench scores (6.41/6.42), yet obtain 54.4 and 75.3 on IMUG Editing, a difference of 20.9 points. Moreover, there are no tied model rankings within IMUG-Bench itself. These results show that IMUG-Bench can further distinguish models with similar scores on existing generation benchmarks but different multi-turn interaction capabilities.
(4) Clarification of evaluation settings within IMUG-Bench. In the new Tables 2–3, we further divide the generation tasks in IMUG-Bench into T2I and Editing, evaluating direct image generation and history-dependent image editing, respectively. Since multi-turn evaluation in IMUG-Bench relies on the model's actual generated history for subsequent interactions and scoring, Free-Running and Reference-History Evaluation using the model's own generated history as the reference are the same setting under our original evaluation protocol. As shown in (2)–(3), the T2I, Editing, and full multi-turn Free-Running evaluations in IMUG-Bench can all distinguish models that achieve similar scores on conventional image generation benchmarks.
Overall, IMUG-Bench reveals ranking differences from existing generation benchmarks and further distinguishes models' ability to handle historical dependencies and maintain generation consistency in multi-turn interleaved interactions, providing an additional evaluation dimension for image generation models.
We thank the reviewer again for the constructive feedback. We hope these responses address your concerns and welcome further discussion.
Thanks for the replies and the new experiments. They address my previous questions. I agree that the benchmark can capture some aspects of multi-turn generation, and I will raise my score to weak accept.
I still think the paper could discuss whether the full interleaved history is necessary for generation. In the application settings I work with, we often collect the relevant images and summarize the dialogue and current request into one instruction. For example, text → img0 → text → img1 → text → img2 can be reformulated as img0 + img1 + summarized instruction → img2, making the current step an image editing or composition task.
Section 4.1 already adapts some models to image-list inputs, but the full dialogue is still retained. My question is about replacing that dialogue with a summary. Are there tasks where this reformulation loses important information or leads to worse results? How much of the observed difficulty would remain with this input format? A small comparison using the same generated images and past information would be helpful.
This is the main reason I am not giving a higher score. I would like to see more discussion of when the interleaved formulation is needed in practice.
Response to Reviewer VRZq (Part 1/2)
We thank the reviewer for the constructive suggestions. We address the evidence for exposure bias and comparisons with existing generation benchmarks below.
R1. Difficulty Controls and Paired History Correction (Weakness 1 & Question 1)
To distinguish and control for the influence of task difficulty on score changes, we conducted two additional analyses: independent task-difficulty assessment and paired history-correction experiments.
(1) Independent task-difficulty assessment. We independently evaluated each image-generation turn using DeepSeek V4.1 along three dimensions: understanding, reasoning, and execution difficulty. Each dimension was assigned an integer score from 0 to 5, and scores were averaged by turn. Execution difficulty measures the overall generation complexity for image tasks.
Image-task difficulty rises from 2.13 (T1) to 3.22 (T4), then falls to 2.90 (T6). Across six equally weighted turns, the 90% confidence interval for Kendall's τb (turn index vs. difficulty) is [−0.088, +0.230]. The interval includes zero, indicating no clear monotonic increase in image-task difficulty. Thus, increasing difficulty alone cannot explain the sustained decline in image generation performance.
(2) Paired history-correction experiment. Since open-ended image generation does not have a unique ground-truth image, we adopted Human-Verified Reference History instead of a strict Oracle History. Specifically, we sampled 482 multi-turn dialogues from the original 3,113 samples according to the sample distribution across the 97 subtasks, and conducted controlled evaluations on BAGEL and OmniGen2.
For image outputs, we constructed reference histories using high-quality images generated by GPT-5 Image Mini and screened through human verification. For text outputs, we replaced historical responses with standard answers. The experiments kept the target tasks and interaction turns unchanged, modifying only the history visible to the models. Dynamic-MCQ answers were also determined based on the reference history. On the evaluated subset, human verification found that the Dynamic-MCQ reference answers returned by the judge model had an accuracy of 95.5%, and we corrected the remaining erroneous answers.
Table 1. Per-turn understanding and generation scores before and after history correction.
The results in Table 1 show that both models achieve overall improvements in text understanding after history correction, indicating that high-quality historical information helps models perform subsequent understanding tasks.
For image generation, both models also achieve higher scores after history correction. Specifically, the T6 scores of BAGEL and OmniGen2 increase from 22.13/27.25 to 70.20/45.65, corresponding to gains of 48.06/18.40 points. Meanwhile, their score declines from T2 to T6 are reduced from 28.46/15.42 to 4.90/6.17 points, respectively, demonstrating that correcting historical outputs effectively mitigates multi-turn generation degradation.
Together, these experiments show that (1) task difficulty does not consistently increase across turns, and (2) correcting historical outputs improves subsequent understanding and generation performance. These findings provide further evidence that accumulated historical generation errors are an important contributor to multi-turn performance degradation, rather than the degradation being attributable to increasing task difficulty alone.
(3) Statistical uncertainty. We sampled dialogues according to the original distribution across the 97 subtasks and conducted paired comparisons on identical target tasks and interaction turns. We further used dialogue-level bootstrap resampling to evaluate the paired score gains from history correction. Both BAGEL and OmniGen2 show positive average gains in text understanding and image generation, with all four 95% confidence intervals excluding zero. These results further support the improvements in subsequent understanding and generation performance brought by history correction.