acceptodds
Under review as a conference paper at ICLR 2027

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

Abstract

In recent years, unified multimodal models (UMMs) have emerged to support both understanding and generation within a single framework. Mastering dynamic, multi-turn interleaved image-text dialogues is a crucial task for UMMs in real-world applications. However, existing benchmarks fail to evaluate this important task, as they are often limited to single-turn or static settings, and typically overlook exposure bias in multi-turn interactions. To bridge this gap, we propose IMUG-Bench, a comprehensive benchmark for multi-turn interleaved image-text dialogue of UMMs that jointly evaluates their understanding and generation capabilities. Our IMUG-Bench comprises three classes: Static Spatial, Temporal Causal, and Hybrid—covering 3,113 samples and 12,034 interaction turns. It also includes dynamic understanding questions, thereby supporting evaluation that better reflects real-world multi-turn interaction scenarios. Large-scale experiments on IMUG-Bench systematically evaluate mainstream open-source and closed-source UMMs, revealing their capability boundaries and failure modes, and uncovering pronounced exposure bias on the generation side in multi-turn interactions. We further explore several test-time scaling strategies, including Chain-of-Thought, Self-Verification, and Best-of-N Sampling, which effectively improve generation accuracy and mitigate exposure bias in generation tasks. These findings provide insights for improving UMM robustness and multi-turn interaction.

Primary area:
datasets and benchmarks
Keywords:
Unified Multimodal Models, Multimodal Benchmark, Multimodal Understanding and Generation, Exposure Bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VRZqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IMUG-Bench evaluates unified multimodal models through multi-turn dialogues that interleave image understanding, generation, and editing. It contains 3,113 samples and 12,034 turns across 19 domains and 97 tasks. A distinctive component is Dynamic-MCQ: the correct answer is determined from the model's own generated image, allowing understanding to be assessed even when generation deviates from the intended target. Eight models are evaluated using a VLM judge and structured evaluation points. The paper reports substantial generation degradation over dialogue turns and explores chain-of-thought, self-verification, and best-of-N sampling on BAGEL.

Critical strengths
  1. The evaluation setting addresses a meaningful gap. Combining generation, editing, and questions about self-generated content tests dependencies that isolated single-turn evaluations miss. Dynamic-MCQ is particularly useful because it separates understanding of the actual image from successful execution of an earlier generation instruction. The breadth of manually verified task templates makes this a potentially reusable resource.
  2. The evaluation is more carefully specified than a purely qualitative benchmark. The paper supplies structured scoring points and judge prompts, evaluates multiple model families, and includes human validation and formatting sensitivity analyses in the appendix. These details support reproducibility and make the observed capability gaps informative.
Critical weaknesses
  1. Section 4.3 and Table 3 show generation scores falling over dialogue turns, but task composition and difficulty also change. These aggregates cannot determine how much degradation comes from errors in earlier outputs. The exposure-bias claim needs a paired comparison of the same target turn under model-generated and corrected histories.

  2. Table 1 compares benchmark features, but the paper does not relate its generation rankings to those on existing generation benchmarks. It remains unclear whether IMUG-Bench reveals differences in multi-turn robustness beyond the models' single-turn generation strength.

Questions for authors
  1. On a human-verified subset, could you compare the same target turn using the model-generated image history and a minimally corrected reference history? Match target instructions, sampling settings, and scoring. Keep text history fixed where consistent, and check that referents and Dynamic-MCQ labels remain valid. Report paired score changes and uncertainty by turn and task type. Does correcting earlier outputs recover later-step performance?

  2. For overlapping model versions, compare generation-only rankings with GenEval or DPG-Bench under comparable inference budgets, separating text-to-image generation from editing. Report rank correlation and notable inversions, accounting for uncertain ties. Within IMUG-Bench, do single-turn, free-running multi-turn, and reference-history results distinguish models with similar conventional generation scores?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VRZq (Part 1/2)

We thank the reviewer for the constructive suggestions. We address the evidence for exposure bias and comparisons with existing generation benchmarks below.

R1. Difficulty Controls and Paired History Correction (Weakness 1 & Question 1)

To distinguish and control for the influence of task difficulty on score changes, we conducted two additional analyses: independent task-difficulty assessment and paired history-correction experiments.

(1) Independent task-difficulty assessment. We independently evaluated each image-generation turn using DeepSeek V4.1 along three dimensions: understanding, reasoning, and execution difficulty. Each dimension was assigned an integer score from 0 to 5, and scores were averaged by turn. Execution difficulty measures the overall generation complexity for image tasks.

Image-task difficulty rises from 2.13 (T1) to 3.22 (T4), then falls to 2.90 (T6). Across six equally weighted turns, the 90% confidence interval for Kendall's τb\tau_b (turn index vs. difficulty) is [−0.088, +0.230]. The interval includes zero, indicating no clear monotonic increase in image-task difficulty. Thus, increasing difficulty alone cannot explain the sustained decline in image generation performance.

(2) Paired history-correction experiment. Since open-ended image generation does not have a unique ground-truth image, we adopted Human-Verified Reference History instead of a strict Oracle History. Specifically, we sampled 482 multi-turn dialogues from the original 3,113 samples according to the sample distribution across the 97 subtasks, and conducted controlled evaluations on BAGEL and OmniGen2.

For image outputs, we constructed reference histories using high-quality images generated by GPT-5 Image Mini and screened through human verification. For text outputs, we replaced historical responses with standard answers. The experiments kept the target tasks and interaction turns unchanged, modifying only the history visible to the models. Dynamic-MCQ answers were also determined based on the reference history. On the evaluated subset, human verification found that the Dynamic-MCQ reference answers returned by the judge model had an accuracy of 95.5%, and we corrected the remaining erroneous answers.

Table 1. Per-turn understanding and generation scores before and after history correction.

ModelModalityHistoryT1T2T3T4T5T6
BAGELImageOriginal64.0350.6046.8144.6843.0422.13
BAGELImageCorrected75.4675.0964.8662.9066.9070.20
BAGELTextOriginal46.1544.2946.5649.5544.5337.20
BAGELTextCorrected48.4063.2060.5061.5377.0878.05
OmniGen2ImageOriginal54.7642.6735.7432.4929.8627.25
OmniGen2ImageCorrected58.2851.8252.5749.6040.7445.65
OmniGen2TextOriginal70.3069.6766.0361.9667.4571.95
OmniGen2TextCorrected64.0873.3577.8667.3271.8885.37

The results in Table 1 show that both models achieve overall improvements in text understanding after history correction, indicating that high-quality historical information helps models perform subsequent understanding tasks.

For image generation, both models also achieve higher scores after history correction. Specifically, the T6 scores of BAGEL and OmniGen2 increase from 22.13/27.25 to 70.20/45.65, corresponding to gains of 48.06/18.40 points. Meanwhile, their score declines from T2 to T6 are reduced from 28.46/15.42 to 4.90/6.17 points, respectively, demonstrating that correcting historical outputs effectively mitigates multi-turn generation degradation.

Together, these experiments show that (1) task difficulty does not consistently increase across turns, and (2) correcting historical outputs improves subsequent understanding and generation performance. These findings provide further evidence that accumulated historical generation errors are an important contributor to multi-turn performance degradation, rather than the degradation being attributable to increasing task difficulty alone.

(3) Statistical uncertainty. We sampled dialogues according to the original distribution across the 97 subtasks and conducted paired comparisons on identical target tasks and interaction turns. We further used dialogue-level bootstrap resampling to evaluate the paired score gains from history correction. Both BAGEL and OmniGen2 show positive average gains in text understanding and image generation, with all four 95% confidence intervals excluding zero. These results further support the improvements in subsequent understanding and generation performance brought by history correction.

AuthorsOpenReview ↗

Response to Reviewer VRZq (Part 2/2)

R2. Generation Benchmark Rankings and Evaluation Settings (Weakness 2 & Question 2)

We separately report IMUG-Bench Text-to-Image (T2I) and Image Editing scores and compare them with GenEval, DPG-Bench, WISE, ImgEdit-Bench, and GEdit-Bench to identify capability differences not fully captured by existing benchmarks.

IMUG T2I aggregates text-to-image tasks without editing historical images; IMUG Editing aggregates tasks that modify historical images. Both use the corresponding task subsets from the original evaluation. Full Free-Running results, covering interleaved understanding and generation, appear in Table 2 of the original manuscript. For fairness, the rankings below include only the same six open-source models.

Table 2. Comparison with existing text-to-image generation benchmarks.

ModelGenEvalDPG-BenchWISEIMUG T2I
BAGEL0.8285.070.5269.4
BLIP3-o 4B0.8179.360.5081.1
BLIP3-o 8B0.8380.730.5282.9
OmniGen20.8083.570.4753.8
Ovis-U10.8983.720.4260.1
UniWorld-V10.8081.380.5581.0

Table 3. Comparison with existing image editing benchmarks.

ModelImgEdit-BenchGEdit-Bench-ENIMUG Editing
BAGEL3.206.5262.8
BLIP3-o 4B2.442.3454.8
BLIP3-o 8B2.402.7360.6
OmniGen23.446.4154.4
Ovis-U14.006.4275.3
UniWorld-V13.264.8560.7

(1) Benchmark ranking correlations. As shown in Tables 2–3, the Spearman rank correlations between IMUG T2I and GenEval, DPG-Bench, and WISE are 0.174, −0.714, and 0.580, respectively. For IMUG Editing, the correlations with ImgEdit-Bench and GEdit-Bench are 0.314 and 0.600. Notably, existing generation benchmarks also show substantial ranking differences: the correlations of GenEval with DPG-Bench and WISE are 0.232 and −0.353, respectively, while that between DPG-Bench and WISE is −0.203. These results indicate that benchmarks emphasize different aspects of model performance, and ranking correlation alone does not determine benchmark validity.

(2) Ranking reversals. Model rankings differ substantially across evaluation settings. For text-to-image generation, Ovis-U1 ranks first on GenEval but only fifth on IMUG T2I. For image editing, OmniGen2 ranks second on ImgEdit-Bench but drops to sixth on IMUG Editing. In the full multi-turn Free-Running evaluation, BAGEL ranks first on DPG-Bench but only fourth in IMUG Free-Running generation. These ranking reversals show that IMUG-Bench captures model capability differences related to multi-turn interaction and historical dependencies that are not reflected in conventional generation benchmarks.

(3) Distinguishing models with similar scores. OmniGen2 and UniWorld-V1 obtain identical GenEval scores (0.80), but rank sixth and third, respectively, in the full Free-Running generation evaluation. For image editing, OmniGen2 and Ovis-U1 achieve nearly identical GEdit-Bench scores (6.41/6.42), yet obtain 54.4 and 75.3 on IMUG Editing, a difference of 20.9 points. Moreover, there are no tied model rankings within IMUG-Bench itself. These results show that IMUG-Bench can further distinguish models with similar scores on existing generation benchmarks but different multi-turn interaction capabilities.

(4) Clarification of evaluation settings within IMUG-Bench. In the new Tables 2–3, we further divide the generation tasks in IMUG-Bench into T2I and Editing, evaluating direct image generation and history-dependent image editing, respectively. Since multi-turn evaluation in IMUG-Bench relies on the model's actual generated history for subsequent interactions and scoring, Free-Running and Reference-History Evaluation using the model's own generated history as the reference are the same setting under our original evaluation protocol. As shown in (2)–(3), the T2I, Editing, and full multi-turn Free-Running evaluations in IMUG-Bench can all distinguish models that achieve similar scores on conventional image generation benchmarks.

Overall, IMUG-Bench reveals ranking differences from existing generation benchmarks and further distinguishes models' ability to handle historical dependencies and maintain generation consistency in multi-turn interleaved interactions, providing an additional evaluation dimension for image generation models.

We thank the reviewer again for the constructive feedback. We hope these responses address your concerns and welcome further discussion.

Thanks for the replies and the new experiments. They address my previous questions. I agree that the benchmark can capture some aspects of multi-turn generation, and I will raise my score to weak accept.

I still think the paper could discuss whether the full interleaved history is necessary for generation. In the application settings I work with, we often collect the relevant images and summarize the dialogue and current request into one instruction. For example, text → img0 → text → img1 → text → img2 can be reformulated as img0 + img1 + summarized instruction → img2, making the current step an image editing or composition task.

Section 4.1 already adapts some models to image-list inputs, but the full dialogue is still retained. My question is about replacing that dialogue with a summary. Are there tasks where this reformulation loses important information or leads to worse results? How much of the observed difficulty would remain with this input format? A small comparison using the same generated images and past information would be helpful.

This is the main reason I am not giving a higher score. I would like to see more discussion of when the interleaved formulation is needed in practice.

Reviewer_Zpz6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces IMUG-Bench, a benchmark designed to evaluate Unified Multimodal Models (UMMs) on multi-turn interleaved image-text understanding and generation tasks. The benchmark comprises 3,113 samples and 12,034 interaction turns, revealing that models suffer from severe performance degradation in multi-turn generation due to "exposure bias". Additionally, the authors test standard test-time scaling strategies, such as Chain-of-Thought and Best-of-N sampling, in an attempt to mitigate this error accumulation phenomenon.

Critical strengths
  1. The benchmark features "Dynamic-MCQ" questions whose correct options depend entirely on the model's previously generated images rather than a static ground truth. This prevents models from utilizing static prior knowledge to cheat, enhancing the rigor of testing long-horizon context awareness.
  2. The paper uses empirical data to quantitatively highlight the "exposure bias" phenomenon that UMMs are prone to in multi-turn image generation, effectively identifying current models' deficiencies in instruction following and cross-turn consistency.
  3. The benchmark covers 19 domains and 97 tasks under three main categories (Static Spatial, Temporal Causal, and Hybrid), providing a rich and diverse set of real-world interaction scenarios.
Critical weaknesses
  1. Limited Conceptual and Theoretical Novelty: The multi-turn "exposure bias" and error accumulation highlighted in the paper are classic, well-known problems in traditional sequence generation and dialogue systems. Introducing this concept into the evaluation of Unified Multimodal Models (UMMs) is merely an incremental empirical observation, lacking breakthrough theoretical or conceptual innovation.
  2. Mitigation Strategies Lack Depth and Fail to Address the Root Cause: The test-time scaling solutions explored by the authors (such as Chain-of-Thought, Self-Verification, and Best-of-N sampling) are existing standard baseline practices rather than algorithmic contributions. Furthermore, the authors admit that these methods only partially mitigate the issue, and significant performance degradation still persists in long-horizon interactions, failing to resolve the fundamental architectural limitations of the models.
Questions for authors

same as weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer Zpz6 (Part 1/1)

We sincerely thank the reviewer for the careful evaluation and constructive feedback on IMUG-Bench. We address your concerns regarding novelty and the limitations of our mitigation strategies below.

R1. Novelty of Multimodal Exposure Bias Analysis (Weakness 1)

We agree that exposure bias and error accumulation are not new theoretical concepts. Our contribution lies in systematically revealing, evaluating, and exploring ways to mitigate these issues in the context of multi-turn interleaved understanding and generation with Unified Multimodal Models (UMMs).

Our benchmark dataset and experiments reveal exposure bias in multi-turn multimodal interactions, along with a pronounced asymmetry across modalities: image generation performance declines substantially as interaction turns increase, while text understanding remains relatively stable. This finding highlights the modality-dependent nature of error accumulation and provides a new perspective on the limitations of UMMs in long-horizon interactions.

We further explore mitigation through test-time scaling, showing that inference-time strategies can improve generation performance and reduce degradation. Thus, our contribution is not to introduce exposure bias as a new phenomenon, but to reveal its asymmetric manifestation across modalities in unified multimodal interactions and explore potential mitigation strategies.

R2. Scope and Limitations of Test-Time Mitigation (Weakness 2)

We acknowledge that CoT, Self-Verification, and Best-of-N are established test-time scaling methods. Our contribution is to examine their applicability to multi-turn interleaved generation and systematically evaluate their effectiveness in improving generation performance and mitigating exposure bias.

Unlike single-turn generation, multi-turn interleaved tasks require models to satisfy the current instruction while maintaining visual consistency across turns, with each generated output further influencing subsequent interactions. We therefore investigate whether existing inference-time strategies can improve this continuous, history-dependent generation process.

Our experiments show that all three strategies improve BAGEL's generation performance. In particular, CoT increases its average image generation score from 54.1 to 65.9 (+11.8 points; Table 4 in the original manuscript), demonstrating the effectiveness of the evaluated test-time scaling methods in this setting.

These results establish practical mitigation baselines for multi-turn interleaved generation while also highlighting the remaining limitations and room for improvement of existing strategies. They provide evidence to motivate future methods specifically designed to optimize history-dependent, multi-turn generation.

We thank the reviewer again for the constructive feedback. We hope these responses address your concerns and welcome further discussion.

AuthorsOpenReview ↗

Response to Reviewer Zpz6: Final Appreciation

We sincerely thank the reviewer once again for your positive assessment and recognition of IMUG-Bench, as well as your valuable feedback throughout the review process. We greatly appreciate your support and are truly grateful for the time and effort you have devoted to reviewing our paper.

Reviewer_4joSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IMUG-Bench, a multi-turn benchmark to evaluate the understanding and generation capabilities of Unified Multimodal Models (UMMs), comprising 3,113 samples and 12,034 interaction turns across three task classes. Constructed by LLM/VLM-assisted template filling and human verification, it evaluates three question types: Static-MCQ, Dynamic-MCQ, and Image-Generation tasks scored by a VLM judge. The authors evaluate eight UMMs and reveal a large performance gap between open-source and closed-source models. Furthermore, they observe a turn-wise decline in image generation performance, which they attribute to exposure bias, and find that three test-time scaling methods, CoT, self-verification, and Best-of-N sampling, can alleviate this issue.

Critical strengths
  1. The paper empirically demonstrates that three test-time scaling strategies (CoT, Self-Verification, and Best-of-N) can improve multi-turn image generation and mitigate turn-wise performance degradation, providing practical insights into improving UMMs robustness.

  2. The dynamic evaluation setup gets Dynamic-MCQ ground truths directly from the model's previously generated images rather than fixed answers, which is a much fairer way to test multi-turn interleaved understanding and generation.

Critical weaknesses
  1. IMUG-Bench samples are constructed through LLM/VLM-based template filling under the explicit constraint "Do NOT rewrite the template text outside brackets", with predefined turn counts and modality sequences. Despite covering 97 tasks, the paper provides insufficient evidence of diversity beyond template-level variations. Consequently, the benchmark may primarily evaluate model performance on predefined interaction patterns.

  2. The paper attributes the declining T1–T6 image-question scores (Table 3) to exposure bias. However, since each turn evaluates a different subset of tasks, this drop could simply reflect increasing task difficulty rather than error accumulation. Without an oracle-history baseline (feeding ground-truth history at each turn), it is hard to tell whether the performance drop comes from accumulated generation errors or the tasks themselves. Thus, the exposure bias claim is not fully convincing.

Questions for authors
  1. Since the benchmark relies on template filling with strict text-preservation rules across the 97 tasks, how diverse are the templates in practice (e.g., how many distinct templates are used per task, and how much do they vary in phrasing and structure)?

  2. To rule out task-subset differences across turns in Table 3, could the authors report the T1–T6 trend purely on the subset of full six-turn dialogues?

  3. Could the authors run an ablation comparing self-generated history against oracle (ground-truth) history on the same tasks, to confirm how much of the multi-turn performance drop actually comes from accumulated generation errors?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4joS (Part 1/2)

We sincerely thank Reviewer 4joS for the careful evaluation and constructive feedback on IMUG-Bench. To address your concerns regarding template diversity and the evidence for exposure bias, we conducted additional statistical analyses and controlled experiments. Our responses are provided below.

R1. Template Diversity (Weakness 1 & Question 1)

The fixed wording in IMUG-Bench templates primarily serves to standardize question structures, rather than restricting the content of instantiated samples. Specifically, IMUG-Bench covers 19 domains and 97 tasks, comprising 346 distinct template combinations instantiated with diverse keywords and visual content. Template-based construction is also common in existing generation benchmarks; for example, GenEval organizes its evaluation around 6 task categories.

We further quantified the similarity between samples within the same subtask. The median TF-IDF cosine similarity for text is only 0.215, while the median image-feature cosine similarity is 0.653. Given that samples within the same subtask naturally share similar task objectives, these results indicate that fixed interaction structures do not lead to repetitive sample content. IMUG-Bench achieves diversity through its task coverage, template combinations, and instantiated content.

R2. Full Six-Turn Performance (Weakness 2 & Question 2)

To address the potential impact of task-subset differences across turns, we followed your suggestion and recomputed the per-turn scores of all eight models using only dialogues with exactly six interaction turns (Table 1).

Table 1. Per-turn performance on the subset of full six-turn dialogues (Text/Image).

ModelT1T2T3T4T5T6
GPT-5 Image Mini97.1/96.583.0/84.587.5/86.892.6/73.470.4/80.377.2/54.8
Nano Banana88.2/94.566.0/76.190.6/74.085.3/74.959.3/81.072.8/58.7
BAGEL51.7/77.050.9/59.645.7/47.660.8/57.557.5/41.444.8/24.7
BLIP3-o 4B55.9/86.347.5/67.979.6/60.658.7/61.346.2/62.167.0/48.9
BLIP3-o 8B68.6/88.363.3/71.675.5/49.880.3/65.475.0/61.174.4/51.7
OmniGen283.6/55.660.4/34.882.1/39.881.7/41.170.0/34.165.1/31.4
Ovis-U159.8/59.154.0/40.278.1/34.056.2/46.653.8/36.658.5/27.3
UniWorld-V123.0/84.621.2/68.637.8/59.336.7/58.335.0/48.327.6/42.9

The results show that text understanding scores still exhibit no consistent downward trend, whereas image generation scores generally decline as interaction turns increase. Although image scores fluctuate at certain turns, this may be related to the uneven distribution of modality output positions and samples within six-turn dialogues.

To quantify the overall trends in both modalities, we fitted least-squares linear trends to each model's average scores across T1–T6. All eight models exhibit negative slopes in image generation, ranging from −8.75 to −3.48, whereas text understanding slopes range from −3.77 to +1.96, showing no consistent downward trend. These findings agree with our original observations and further confirm the difference in performance degradation between modalities within the full six-turn subset.

To further reduce the influence of sample-composition differences across turns, we additionally analyzed paired score changes between successive outputs of the same modality within each dialogue.

Table 2. Image score changes between successive image-generation steps.

Model1→22→33→41→4
BAGEL-8.4-20.5-3.6-26.4
BLIP3-o 4B-9.3-1.6-0.9-19.1
BLIP3-o 8B-7.6-5.4+3.5-15.0
OmniGen2-3.9-10.7-2.2-17.4
Ovis-U1-15.6-3.2-5.1-23.3
UniWorld-V1-2.9-12.9-17.7-40.9

As shown in Table 2, all six models exhibit substantial declines from their first to fourth image-generation outputs, while paired text-score changes alternate between positive and negative values without a consistent downward trend. Thus, image generation degradation persists even when restricting the analysis to full six-turn dialogues and comparing outputs within the same dialogue, indicating that the original observation cannot be explained solely by mixing dialogues of different lengths.

AuthorsOpenReview ↗

Response to Reviewer 4joS (Part 2/2)

R3. Task Difficulty and Paired History Correction (Weakness 2 & Question 3)

To distinguish and control for the influence of task difficulty on score changes, we conducted two additional analyses: independent task-difficulty assessment and paired history-correction experiments.

(1) Independent task-difficulty assessment. We independently evaluated each image-generation turn using DeepSeek V4.1 along three dimensions: understanding, reasoning, and execution difficulty. Each dimension was assigned an integer score from 0 to 5, and scores were averaged by turn. Execution difficulty measures the overall generation complexity for image tasks.

Image-task difficulty rises from 2.13 (T1) to 3.22 (T4), then falls to 2.90 (T6). Across six equally weighted turns, the 90% confidence interval for Kendall's τb\tau_b (turn index vs. difficulty) is [−0.088, +0.230]. The interval includes zero, indicating no clear monotonic increase in image-task difficulty. Thus, increasing difficulty alone cannot explain the sustained decline in image generation performance.

(2) Paired history-correction experiment. Since open-ended image generation does not have a unique ground-truth image, we adopted Human-Verified Reference History instead of a strict Oracle History. Specifically, we sampled 482 multi-turn dialogues from the original 3,113 samples according to the sample distribution across the 97 subtasks, and conducted controlled evaluations on BAGEL and OmniGen2.

For image outputs, we constructed reference histories using high-quality images generated by GPT-5 Image Mini and screened through human verification. For text outputs, we replaced historical responses with standard answers. The experiments kept the target tasks and interaction turns unchanged, modifying only the history visible to the models.

Table 3. Per-turn understanding and generation scores before and after history correction.

ModelModalityHistoryT1T2T3T4T5T6
BAGELImageOriginal64.0350.6046.8144.6843.0422.13
BAGELImageCorrected75.4675.0964.8662.9066.9070.20
BAGELTextOriginal46.1544.2946.5649.5544.5337.20
BAGELTextCorrected48.4063.2060.5061.5377.0878.05
OmniGen2ImageOriginal54.7642.6735.7432.4929.8627.25
OmniGen2ImageCorrected58.2851.8252.5749.6040.7445.65
OmniGen2TextOriginal70.3069.6766.0361.9667.4571.95
OmniGen2TextCorrected64.0873.3577.8667.3271.8885.37

The results in Table 3 show that both models achieve overall improvements in text understanding after history correction, indicating that high-quality historical information helps models perform subsequent understanding tasks.

For image generation, both models also achieve higher scores after history correction. Specifically, the T6 scores of BAGEL and OmniGen2 increase from 22.13/27.25 to 70.20/45.65, corresponding to gains of 48.06/18.40 points. Meanwhile, their score declines from T2 to T6 are reduced from 28.46/15.42 to 4.90/6.17 points, respectively, demonstrating that correcting historical outputs effectively mitigates multi-turn generation degradation.

Together, these experiments show that (1) task difficulty does not consistently increase across turns, and (2) correcting historical outputs improves subsequent understanding and generation performance. These findings provide further evidence that accumulated historical generation errors are an important contributor to multi-turn performance degradation, rather than the degradation being attributable to increasing task difficulty alone.

We thank the reviewer again for the constructive feedback. We hope these responses address your concerns and welcome further discussion.

Reviewer_4joSOpenReview ↗

Response to Authors

Thank you for your response and the additional analyses. However, I remain concerned about the diversity of the benchmark. TF-IDF similarity mainly reflects differences in the filled-in keywords, which is expected by construction for template-based samples. It does not capture whether samples share the same question skeleton and turn structure, which is the core of my concern. If many samples follow the same skeleton, models may exploit these structural patterns, which could bias the evaluation results. So I will maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer 4joS: Additional Experiment & Final Appreciation

We sincerely thank the reviewer for carefully reading our previous responses and supplementary analyses, and for providing further insightful and constructive comments on the template diversity of IMUG-Bench. Your continued attention has encouraged us to further examine the semantic differences among samples and the impact of fixed templates on evaluation results. To address your concerns regarding potential content repetition and structural bias introduced by fixed templates, we conducted two additional experiments, as detailed below.

(1) Experiment 1: Semantic Similarity Analysis

We first analyzed the semantic similarity of textual and visual inputs. For image inputs, as previously reported, the cosine similarity in the image feature space extracted by DINOv3 is 0.653. For text questions without image inputs, we concatenated the question texts in their original turn order and encoded the complete requests using all-mpnet-base-v2 to calculate semantic similarity. The results show that the median semantic cosine similarity is 0.759 for sample pairs within the same subdomain and 0.513 for sample pairs from different subdomains within the same domain. These results indicate that questions constructed using the same template within a subdomain already exhibit substantial semantic differences, while questions across different subdomains demonstrate even greater diversity. Furthermore, considering the overall task composition, IMUG-Bench covers 19 domains, 97 subdomains, and 346 template combinations, with an average of approximately 5.1 subdomains per domain and 9 samples per template combination. Samples within the same subdomain focus on similar capability dimensions, while different subdomains and domains cover a broader range of task objectives and capability dimensions. The extensive subdomain divisions and template combinations further expand the semantic coverage of the entire benchmark, effectively enhancing its overall task diversity.

(2) Experiment 2: Semantic-Preserving Question Rewriting

To further examine the impact of fixed template phrasing on model evaluation results, we conducted an additional semantic-preserving question rewriting experiment. We randomly sampled 500 turns according to the text-to-image turn ratio of the full benchmark. We used DeepSeek V4.1 to rewrite the current-turn instructions, focusing on modifying syntactic structures and information organization while preserving all original task-specific semantic details. This ensured that the rewritten questions differed in grammar and sentence structure while maintaining their original task semantics. During evaluation, the original and rewritten questions used the same original conversation history, with only the current-turn user instruction replaced. The results are shown below:

ModalityOriginal RequestRewritten RequestAverage Change95% CI
Overall51.8251.94+0.11[-1.78, +1.98]
Text Understanding39.4840.30+0.81[-1.73, +3.47]
Image Generation60.3560.55+0.20[-1.66, +2.09]

The overall average score changes by only +0.11 percentage points. The 95% confidence intervals for both text and image modalities include zero, and the changes in average scores are minimal. These results demonstrate that questions expressed using fixed templates and those written naturally with different sentence structures have comparable effects on model responses.

Together, these two experiments show that samples sharing the same template in IMUG-Bench contain diverse instance-specific content despite their shared task structures. The non-template text constitutes the core semantic content of each question, playing a central role in determining and ensuring content diversity, while the templated text primarily facilitates question construction and has no major impact on model understanding or question diversity.

We sincerely thank the reviewer once again for the careful evaluation and continued discussion. Your valuable feedback has helped us strengthen our experimental analyses and further examine the design and validity of our benchmark. We hope our additional results address your concerns, and we would be truly grateful if you could reconsider your rating in light of this new evidence. We greatly appreciate your support and are truly grateful for the time and effort you have devoted to reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Response to All Reviewers and Area Chair

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for the valuable time and effort devoted to the review and discussion process, as well as for the insightful and constructive feedback on IMUG-Bench. We also greatly appreciate your positive assessment and recognition of our work.

During the rebuttal period, we carefully addressed the reviewers' comments and conducted additional experiments, yielding the following findings:

  1. Template Diversity and Evaluation Robustness: We provided additional diversity statistics across templates, subdomains, and domains, and analyzed sample similarities at both the textual and semantic levels, further demonstrating the diversity of IMUG-Bench in task structures and instance-specific content. Additionally, our semantic-preserving question rewriting experiment confirmed the robustness of evaluation results to changes in template wording.

  2. Multi-Turn Performance Degradation and Historical Error Accumulation: We further investigated the effects of task difficulty and historical information quality on model performance. Through analyses of complete six-turn dialogues, task difficulty, and paired history-correction experiments, we provided further evidence that task difficulty changes are not the primary contributor to multi-turn performance degradation, while confirming the important role of accumulated historical errors.

  3. Comparison with Existing Generation Benchmarks: We separately compared model rankings on text-to-image generation and image editing tasks between IMUG-Bench and existing benchmarks. The substantial ranking differences further highlight the complementary evaluation value of IMUG-Bench in assessing multi-turn interactions and historical dependencies.

  4. Test-Time Scaling and History Management: We further clarified the research scope of test-time scaling methods and conducted an additional self-summarized history experiment. The results show that self-summarized history substantially improves multi-turn image generation performance, while also revealing its limitations in text understanding.

We sincerely hope that these additional experiments and analyses more fully address the reviewers' concerns and further clarify the evaluation design, empirical findings, and research value of IMUG-Bench. We also hope that IMUG-Bench can provide a useful reference for future research on multi-turn understanding and generation in unified multimodal models.

We greatly value the questions and suggestions raised by the reviewers and sincerely appreciate the insightful and valuable discussions throughout this process. Finally, we extend our heartfelt thanks to all reviewers and the Area Chair for their careful evaluation and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.