- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
UniG2U-Bench studies whether unified image generation and understanding models improve multimodal understanding, and whether generating an intermediate image at inference time helps answer visual questions. The benchmark contains 3,000 instances spanning seven categories and 30 subtasks, with evaluations of more than 30 models. It distinguishes two comparisons: unified models versus their selected base or proxy understanding models under Direct answering, and Generate-then-Answer versus Direct answering within unified models.
The paper reports that 12 of 16 paired unified models underperform their selected comparators under Direct answering, while six of eight applicable models underperform under Generate-then-Answer. It also identifies task-specific benefits, particularly in some spatial and iterative reasoning settings. The analysis includes a conceptual gradient-alignment framework, a likelihood-based account of commitment to generated intermediates, correlation analyses, failure categorization, and reference-intermediate interventions.
Critical strengths
-
The benchmark separates two important questions that are often conflated. Improvements associated with unified training and improvements from explicitly generating visual intermediates at inference time are different phenomena. Evaluating them separately across a broad task taxonomy makes the benchmark useful for diagnosing where generation capabilities help understanding and where they introduce additional failure modes. The task-level results are more informative than a single aggregate comparison of unified models.
-
The paper includes meaningful diagnostic controls and examines limitations of its own analysis. Appendix D controls for the shared-baseline effect in the correlation analysis by residualizing against base-model performance. Appendix N supplies manually checked reference intermediates, which helps distinguish failures of intermediate-image generation from failures to use an intermediate effectively. The separation of unsupported editing protocols into additional analyses is also appropriate. These controls make the empirical investigation more substantive than a broad leaderboard comparison alone.
Critical weaknesses
- The identities and matching rationale of the comparison models are not consistently specified, which affects the central performance deltas and grouping analyses. For example, Table 2 compares OneCAT-3B against Qwen2.5-VL-7B, while the original OneCAT paper identifies Qwen2.5-3B-Instruct as its language-model initialization. This requires a clear distinction between initialization and a chosen multimodal comparator. Using the already evaluated Qwen2.5-VL-3B as an alternative proxy changes the reported Direct deficit from −3.30 to −1.24, although its sign remains negative.
Similar ambiguity appears between Tables 2 and 13: the latter describes the backbone column as the paired base VLM, but lists Qwen-2.5 for BAGEL where Table 2 uses LLaVA-OneVision, and Qwen3-1.7B for Ovis-U1 where Table 2 uses Qwen2.5-VL-3B. Proxy comparisons can be informative, and the paper appropriately acknowledges that these are not causal estimates of unified training. Nevertheless, the exact initialization, visual architecture, selected comparator, and rationale need to be distinguished consistently. Otherwise, the magnitude of the reported deltas and the interpretation of “same-backbone” groupings are difficult to reproduce.
- The documented Direct/GtA protocols do not yet establish a controlled comparison of the contribution of generated images. Appendix K is labeled Non-CoT, but K.4 requests multiple generated images and substantially duplicates the visual-CoT prompts in L.10. The Direct object-motion prompt in K.1.2 uses camera-motion framing, whereas L.1.4 uses object-motion framing. The Direct and GtA MSR prompts also differ in whether they explicitly ask for multi-step spatial reasoning. These may be documentation errors; the PDF does not establish that they affected execution. If they were actually used, however, the comparison changes task framing and reasoning instructions in addition to image generation. The claimed budget matching is also not accompanied by sufficiently concrete per-model text-token, image-generation, and iteration limits. Because inference-time G2U is a principal research question, uncertainty about the executed prompts and resource controls is consequential.
Questions for authors
-
[Clarification] Please provide an exact checkpoint and pairing table that separately identifies each model’s language initialization, visual component, selected understanding comparator, and matching rationale. Please also specify how the checkpoint labeled LLaDA-Instruct processes visual inputs and how backbone families are defined in the correlation analysis. How do the principal conclusions change on the subset with the most defensible matching?
-
[Clarification] Which prompts were actually executed for Direct and GtA? Please resolve the generated-image requests in Appendix K.4 and the object-motion and MSR discrepancies between Appendices K and L. Are these documentation-only issues, or do any reported experiments require correction?
-
[Assessment-relevant validation] Please report the actual text, image, and interaction budgets for both protocols. For the tasks where GtA changes reasoning scaffolding as well as adding an image, do the principal conclusions survive comparison with a text-only reasoning condition using equivalent task instructions and a clearly stated budget? Paired uncertainty for these comparisons would help establish how stable the reported effects are.
4 comments
Response to Reviewer 3yav (Part 2/3): Prompts Actually Executed
W2 / Q2: "Which prompts were actually executed for Direct and GtA? Please resolve the generated-image requests in Appendix K.4 and the object-motion and MSR discrepancies between Appendices K and L. Are these documentation-only issues, or do any reported experiments require correction?"
We thank the reviewer for checking the appendices so carefully. Both observations are correct. Appendix K.4 reproduces the visual-CoT prompt of L.10, and K.1.2 shows the camera-motion prompt instead of the object-motion one. These are documentation errors:
- The Direct prompts actually executed for the Uni-MMMU tasks are text-only and do not ask for image generation. The jigsaw prompt, for example, asks the model to "mentally visualize placing each candidate into the bottom-right cell".
- The executed object-motion prompt uses the object-motion wording.
Appendix U now gives every executed prompt, and K.4 and K.1.2 are corrected.
For MSR, the Direct prompt asks for step-by-step reasoning, while GtA gives this instruction in its generation stage. We reran MSR with a single shared instruction, so the conditions differ only in the intermediate.
Table B. MSR with one instruction shared by all conditions (accuracy, %, same input images).
| Model | Direct | Text-CoT | GtA |
|---|---|---|---|
| Ovis-U1 | 32 | 23 | 28 |
| BAGEL | 30 | 28 | 31 |
| Janus-Pro | 23 | 24 | 18 |
| OmniGen2 | 23 | 32 | 27 |
Response to Reviewer 3yav (Part 3/3): Budget, Text-Only Control and Paired Uncertainty
Q3: "Please report the actual text, image, and interaction budgets for both protocols. For the tasks where GtA changes reasoning scaffolding as well as adding an image, do the principal conclusions survive comparison with a text-only reasoning condition using equivalent task instructions and a clearly stated budget? Paired uncertainty for these comparisons would help establish how stable the reported effects are."
Budgets. Appendix O lists the token caps, which are the same for Direct and the GtA answer stage within each task, the image-generation settings of each model, and the tokens and images actually used per item (see Tables D and E in our response to Reviewer AbYh).
Text-only control. To separate the extra step from the image, we added the Text-CoT and blank-image conditions (Appendix P). Both keep the GtA instructions, step structure and answer-stage cap. With the budget matched, GtA is below Text-CoT for all four models (significantly for Ovis-U1, Janus-Pro and OmniGen2 on 2,354 items), and for Ovis-U1 the blank-image pipeline is 2.8 points below Direct while the generated image accounts for a further 5.7. On all 3,000 items, including the four LLM-judged tasks, the principal conclusions hold (Table C).
Table C. Paired differences on all 3,000 items (points, bootstrap 95% CI).
| Model | GtA − Direct | GtA − Text-CoT |
|---|---|---|
| Ovis-U1 | −8.3 [−9.9, −6.7] | −5.4 [−7.0, −3.8] |
| Janus-Pro | −7.8 [−9.4, −6.2] | −4.5 [−6.1, −2.9] |
| OmniGen2 | −0.8 [−2.0, +0.4] | −1.1 [−2.5, +0.2] |
| BAGEL | +0.3 [−0.9, +1.4] | −0.9 [−2.3, +0.4] |
Paired uncertainty. Every comparison between Direct, Text-CoT and GtA is now paired at the item level, with bootstrap 95% confidence intervals and exact McNemar tests, Holm-corrected across subtasks.
Table D. Examples of paired comparisons (Appendices P and T).
| Comparison | Model | Difference (points) | Significance |
|---|---|---|---|
| GtA − Direct (original runs) | TokenFlow | −3.9 | p = 3×10⁻⁴ |
| GtA − Direct (original runs) | GPT-4o | −1.7 | not significant |
| GtA − Text-CoT, IllusionBench | Ovis-U1, Janus-Pro, BAGEL | −11.8, −12.5, −4.5 | significant after Holm |
Appendices P and T give the full tables.
We hope these responses address the reviewer's concerns. We would be sincerely grateful if the reviewer would take them into account in the final assessment, and we would be glad to provide any further clarification during the discussion period.
Gentle Follow-up on Our Rebuttal
Dear Reviewer 3yav,
Thank you again for the time and effort you have devoted to reviewing our paper. We have revised our responses to make them more precise, with each of your questions quoted and answered in turn. The responses cover the full pairing table with matching tiers, the prompts actually executed, the rerun of MSR with a shared instruction, and the budget-matched text-only control with paired uncertainty.
We would be grateful if you could take a look at the updated responses at your convenience and let us know whether they address your concerns. If any question remains, we would be very happy to discuss it further during the discussion period. If our responses have resolved your concerns, we would sincerely appreciate it if you would consider updating your assessment.
Thank you again for your effort and your constructive feedback.
Best regards,
The Authors
Response to Reviewer 3yav (Part 1/3): Pairing of Unified Models and Comparators
We sincerely thank the reviewer for the very careful and detailed review, and for recognizing that the benchmark separates two questions that are often conflated, and that its diagnostic controls make the study more substantive than a leaderboard comparison. The reviewer's points on pairing and prompts were precise and correct, and addressing them has made the paper more reproducible. We answer each point below, and all changes are in the revised PDF (changes in blue).
Table 13 lists each unified model's language-model initialization, while Table 2 lists the model it is compared with; the two columns were labelled as if they were the same. The revision adds a pairing table (Appendix S) that gives, for each unified model, its language initialization, vision encoder, comparator, matching tier and the reason for the choice. Comparators follow the vision component and the training data, in three tiers:
MMaDA is compared with LLaDA-Instruct, the model it was initialized from, and is listed as Tier 2.
As the reviewer suggested, OmniGen2 is built on Qwen2.5-VL-3B, and OneCAT-3B is initialized from Qwen2.5-3B-Instruct, the language model of Qwen2.5-VL-3B; both are now compared with Qwen2.5-VL-3B (Tier 1 and Tier 2, respectively). On the subset with the most defensible matching, the principal conclusion holds: every Tier-1 model with Direct results scores below its comparator, and so does OneCAT-3B.
Table A. Direct gap to the comparator (points), Tier-1 models and OneCAT-3B (Appendix S).