- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Self-Generative-Understanding (SGU), an evaluation protocol that asks a unified multimodal model to describe an image, reconstruct it from its own description, and answer the original visual question using the reconstructed image. By reusing existing VQA questions and reference answers, SGU measures task performance through a combined understanding-and-generation pipeline. Experiments on six models and four benchmarks reveal performance differences that separate capability evaluations do not directly capture.
Critical strengths
-
Simple and meaningful joint evaluation. SGU brings understanding and generation into a shared task and evaluates whether task-relevant information remains usable across modalities. This provides a useful complement to evaluating the two capabilities separately.
-
Practical and reusable protocol. The method requires no new task annotations and can reuse existing VQA benchmarks. Its explicit intermediate descriptions and images, together with component-replacement experiments, also support diagnosis of pipeline failures.
Critical weaknesses
-
Limited coverage of recent models. The main evaluation covers only six models and omits recent representative unified models such as SenseNova-U1 [Diao et al., 2026] and JoyAI-Image [Song et al., 2026]. Broader coverage would strengthen the evidence that the observed performance gaps and diagnostic findings generalize to current models and architectures.
-
Limited conceptual novelty relative to related evaluations. The core pipeline is straightforward and closely related to prior joint evaluation methods. In particular, Unified-Bench [Yan et al., 2025] already evaluates an image-to-description-to-reconstruction loop. The paper should discuss this work and more clearly establish what the original-VQA endpoint adds beyond reconstruction-based evaluation, ideally through a direct comparison.
-
Limited assessment of fine-grained visual quality. A textual description cannot fully capture an image's subtle textures, local appearance, and complex spatial details. Moreover, the final VQA score tests only information relevant to the selected question, so visual details may be lost during reconstruction without affecting the score. SGU therefore emphasizes task-relevant semantic preservation and provides limited coverage of fine-grained visual fidelity and overall image quality.
Questions for authors
- Could you extend the evaluation to recent unified models, such as SenseNova-U1 and JoyAI-Image, or clarify any practical constraints on including them?
- How does SGU compare with Unified-Bench on the same models and source images? What additional failures or distinctions does the VQA endpoint reliably reveal beyond reconstruction similarity?
- How would you extend the protocol to assess fine-grained visual details that are omitted from the intermediate description or not tested by the original question?
Dear Reviewer,
Thank you again for your time and thoughtful feedback on our paper. As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal and additional analyses. We hope our responses have helped clarify the points you raised, and we would greatly appreciate any further feedback on whether your concerns have been adequately addressed. If any questions or concerns remain, we would be more than happy to provide additional clarification and continue the discussion.
Thank you very much for your time and consideration.