acceptodds
Under review as a conference paper at ICLR 2027

Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

Abstract

As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs). In this work, we propose Self-Generative-Understanding (SGU), a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. Without requiring new annotations, SGU leverages the dual understanding-and-generation abilities of UMMs by asking them to first perceive an image and produce a textual description, subsequently reconstruct a visual context based on that description, and finally perform reasoning over the self-generated output. This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems. Extensive experiments show that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone. Our work provides a complementary holistic evaluation framework and offers a foundation for benchmarking the development of next-generation unified multimodal models.

Primary area:
datasets and benchmarks
Keywords:
Unified Multimodal Models, Multimodal Evaluation, Closed-Loop Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HTEBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Self-Generative-Understanding (SGU), an evaluation protocol that asks a unified multimodal model to describe an image, reconstruct it from its own description, and answer the original visual question using the reconstructed image. By reusing existing VQA questions and reference answers, SGU measures task performance through a combined understanding-and-generation pipeline. Experiments on six models and four benchmarks reveal performance differences that separate capability evaluations do not directly capture.

Critical strengths
  1. Simple and meaningful joint evaluation. SGU brings understanding and generation into a shared task and evaluates whether task-relevant information remains usable across modalities. This provides a useful complement to evaluating the two capabilities separately.

  2. Practical and reusable protocol. The method requires no new task annotations and can reuse existing VQA benchmarks. Its explicit intermediate descriptions and images, together with component-replacement experiments, also support diagnosis of pipeline failures.

Critical weaknesses
  1. Limited coverage of recent models. The main evaluation covers only six models and omits recent representative unified models such as SenseNova-U1 [Diao et al., 2026] and JoyAI-Image [Song et al., 2026]. Broader coverage would strengthen the evidence that the observed performance gaps and diagnostic findings generalize to current models and architectures.

  2. Limited conceptual novelty relative to related evaluations. The core pipeline is straightforward and closely related to prior joint evaluation methods. In particular, Unified-Bench [Yan et al., 2025] already evaluates an image-to-description-to-reconstruction loop. The paper should discuss this work and more clearly establish what the original-VQA endpoint adds beyond reconstruction-based evaluation, ideally through a direct comparison.

  3. Limited assessment of fine-grained visual quality. A textual description cannot fully capture an image's subtle textures, local appearance, and complex spatial details. Moreover, the final VQA score tests only information relevant to the selected question, so visual details may be lost during reconstruction without affecting the score. SGU therefore emphasizes task-relevant semantic preservation and provides limited coverage of fine-grained visual fidelity and overall image quality.

Questions for authors
  1. Could you extend the evaluation to recent unified models, such as SenseNova-U1 and JoyAI-Image, or clarify any practical constraints on including them?
  2. How does SGU compare with Unified-Bench on the same models and source images? What additional failures or distinctions does the VQA endpoint reliably reveal beyond reconstruction similarity?
  3. How would you extend the protocol to assess fine-grained visual details that are omitted from the intermediate description or not tested by the original question?
Reviewer_mQHhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Self-Generative-Understanding (SGU), which asks a unified multimodal model to describe an image, reconstruct it from the description, and answer the original visual question using the reconstruction. It reuses existing answers without additional annotations for scoring. Experiments on six models and four benchmarks, with component replacements and input controls, reveal substantial losses along this pipeline.

Critical strengths
  1. SGU tests whether information remains usable for answering concrete questions after reconstruction. Reusing existing VQA answers makes the protocol scalable and provides a useful complement to image similarity metrics.
  2. Caption-only QA, component replacements, image controls, cross-model readers, and independent human judgments help interpret the results. For example, UniWorld scores 63.30% from captions but only 28.67% after reconstruction on OCR-VQA, exposing substantial downstream loss.
Critical weaknesses
  1. Intermediate data quality and error attribution remain insufficiently validated. A reliable original VQA sample does not guarantee a correct or sufficiently detailed intermediate description. If a description omits or misstates the evidence needed for a question, a generator may follow it faithfully while the loop still fails against the original answer. An omission may also reflect generic captioning rather than failed visual understanding. Caption-only QA and component replacements provide useful diagnostic signals, but neither independently establishes description correctness, evidence sufficiency, or generation fidelity to that description. The human audit in Appendix G.6 assesses reconstructed evidence without jointly validating the captions. SGU therefore measures pipeline success, but the attribution of losses to understanding versus generation remains uncertain. This is my main concern because the benchmark should distinguish information lost upstream from information that a generator fails to render.
  2. The added validity of SGU is unevenly established. On MathVista, SGU correlates only slightly more strongly with human evidence judgments than Caption-QA (0.314 vs 0.301). Adding SGU produces mixed predictive changes. Moreover, 25.51% of correct loop answers lack majority-judged supporting evidence. These results limit interpreting SGU as evidence preservation, although it remains a valid task-success measure. Unreported benchmark splits, sample counts, human audit size, and confidence intervals make the strength of the claimed added information difficult to assess.
Questions for authors
  1. What independent criteria distinguish an incorrect or insufficient caption from a reconstruction that fails to follow a correct caption? How do the current analyses separate these cases?
  2. Can you report a jointly audited subset covering caption correctness, task-relevant sufficiency, and reconstruction fidelity? Among correct and sufficient captions, how often does generation lose the required evidence?
  3. What exact splits, question and source-image counts, and human audit size were used? Do confidence intervals accounting for shared source images support the reported ranking differences and added validity over Caption-QA?
Reviewer_Q847OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Self-Generative-Understanding (SGU), a closed-loop protocol for evaluating unified multimodal models. Given an image and a question, the model first describes the image, generates a new image from that description, and then answers the original question using the generated image. The authors evaluate six models across four benchmarks and observe consistent accuracy drops relative to direct VQA and caption-based QA. Through component replacement experiments and intermediate-output analyses, the paper investigates how visual understanding and generation contribute to these performance losses.

Critical strengths

The closed-loop evaluation is interesting and intuitive. In particular, the consistent drop from Caption-QA accuracy to SGU accuracy across models and benchmarks reveals a meaningful system-level limitation of current UMMs: information that supports correct answers in textual form does not reliably support correct answers after image generation and subsequent visual understanding. This provides a useful perspective on how well these capabilities work together.

Critical weaknesses
  1. The description stage complicates the interpretation of SGU failures.

    The default question-blind prompt may omit details needed to answer the question, limiting downstream performance without necessarily indicating a lack of visual understanding. This is particularly concerning for the benchmarks used, such as MathVista, where answering questions may require details that generic descriptions overlook. Appendix G.4 partially addresses this through question guidance and longer captions, but does not quantify how often captions lack the required information.

  2. The specific value for evaluating unified models remains unclear.

    The evaluated behavior is decomposed into independent stages, and specific bottlenecks are identified by isolating or replacing individual components. This raises the question of whether evaluating understanding and generation separately would provide a more direct and informative assessment of these capabilities. The paper needs to clarify what additional insight the closed loop offers into unified modeling that would justify it over such separate evaluations.

Questions for authors
  1. What fraction of Stage 1 captions omit information required to answer the question, broken down by benchmark? Could this be assessed on an annotated subset under both question-blind and question-guided prompting?

  2. Could the authors extend the evidence assessment to include captions and report a quantitative breakdown of failures across the description, generation, and final answering stages?

3 comments

Dear Reviewer,

Thank you again for your time and thoughtful feedback on our paper. As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal and additional analyses. We hope our responses have helped clarify the points you raised, and we would greatly appreciate any further feedback on whether your concerns have been adequately addressed. If any questions or concerns remain, we would be more than happy to provide additional clarification and continue the discussion.

Thank you very much for your time and consideration.

Thank you for the detailed audit and failure breakdown, which address my concerns about the description stage. However, I am not fully convinced of the specific value for unified models, so I would prefer to keep my current score.

Thank you for taking the time to consider our response. We appreciate your acknowledgment that the additional audit and failure breakdown addressed your concerns about the description stage. We would also like to note that SGU offers a complementary system-level evaluation perspective by examining how a unified multimodal model's understanding and generation capabilities work together toward a shared task outcome, rather than assessing them only in isolation. This provides a distinct perspective alongside existing capability-specific evaluations. We would be happy to answer any further technical questions.

Thank you again for your thoughtful feedback and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.