acceptodds
Under review as a conference paper at ICLR 2027

Generation as Evidence: Improving Visual Understanding in Unified Multimodal Models

Abstract

Unified multimodal models support visual understanding and image generation, but whether generation provides useful evidence beyond strong understandingonly methods remains unclear. We study this question through caption discrimination: selecting the correct description of an image from two candidates. We score the observed image conditioned on each caption using the generation pathway, then combine generation and understanding scores with a linear combiner calibrated on a small labeled set. The pretrained model remains frozen, and no new images are generated. We compare against four progressively stronger understanding-only baselines, with the strongest using the same understanding scores, calibration data, and combiner. Across six unified models and a crossmodel control on five compositional benchmarks, generation significantly improves 10 of 27 evaluated pairs by 1.2–8.1 percentage points over this baseline. Eight gains replicate in an independent implementation; uninformative replacements of generation scores yield no significant gains. The median gain falls from 4.8 to 1.1 points as understanding baselines are strengthened, showing why generation’s contribution must be measured against strong controls. Localization, occlusion, and image-shuffling analyses show that generation scores depend on image content and concentrate on regions that distinguish the captions. These findings show that generation pathways can provide complementary visual evidence for caption discrimination within frozen unified models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, Visual Understanding, Generative Scoring, Compositional Reasoning, Inference-Time Capability Transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3V1aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether generation can help understanding in unified models.

The authors investigate this question through a task called caption discrimination. They use the understanding and generation branches to compute scores, such as likelihood, and then train a linear model to combine these scores and select the correct caption between two similar captions. This design does not require generating an actual image.

They find that generation can enhance understanding, although the improvement depends on the capability of the baseline understanding model.

Critical strengths
  1. The caption discrimination setting does not require generating an actual image and can be adapted to different unified model architectures.
Critical weaknesses
  1. The evaluation is limited to a single task, caption discrimination, which is a relatively simple binary classification problem for unified models. More complex and practical tasks, such as visual question answering and multimodal reasoning, are not investigated.

  2. The paper proposes 12 Understanding Readouts and 4 Generation Readouts for score computation, but does not provide experiments to justify the choice of these numbers.

Questions for authors

Please see weaknesses.

Reviewer_GG2kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether generation-side scores from frozen unified models supplement understanding scores. A regularized linear combiner fuses twelve understanding margins and four generation margins using labeled target-task calibration. On two-caption discrimination, 10 of 27 model–benchmark pairs meet the authors' improvement criterion against learned U-only with matched labels and learner.

Critical strengths
  1. Progressively stronger understanding baselines, matched calibration, and null-feature replacements help distinguish generation evidence from weak-baseline or added-capacity effects.
  2. Image-grouped splits, paired statistics, length controls and visual interventions provide substantial checks. Negative results and limitations are transparently reported.
  3. Some gains are substantial, including +8.1 points on Emu3.5/SugarCrepe and +7.3 on Janus-Pro/SPEC, supporting conditional incremental predictive value.
Critical weaknesses

1. Limited methodological novelty and insufficiently established significance of the residual gains. Bidirectional likelihood fusion has precedents such as BLiM (ICCV 2025, §3.1); SUDER (2025 preprint, §3.2) also evaluates text through a unified model's generation likelihood, using it for training. The paper cites these works; its distinction is primarily controlled inference-time analysis with frozen backbones. This can be valuable, but the median gain over the strongest internal U baseline is only 1.1 points, and 17/27 pairs do not meet the improvement criterion (Table 1). Target-task calibration and four G readouts per caption add cost, while final-pipeline latency remains unmeasured (Appendix D). Large improvements in individual settings therefore do not establish broad practical value; the contribution needs clearer insight or evidence of a useful benefit–cost trade-off.

2. Multi-feature fitting leaves the source of improvement insufficiently resolved. Twelve U features combine manually chosen templates, mean/sum aggregation and PMI, then are fitted jointly with four G features. Regularized linear fusion is reasonable, but can compensate for template, length and prior biases; final accuracy does not fully separate such readout corrections from distinctive visual evidence in G. Simple fusion, null-feature and length controls exist, but do not directly isolate feature-group necessity and stability under the same learner. Table 7's G-alone selects one readout, without an explicit four-feature learned G-only comparison.

3. Independent generalization evidence is limited. The main panel is retrospective with overlapping splits; SugarCrepe++ shares images and most negatives with SugarCrepe, and non-COCO tasks underwent screening (Appendix B.6). These histories are disclosed, but 10/27 is not a general success rate on new tasks. Eight passing pairs recur in the second implementation, while the separately frozen prospective CREPE test finds no gain: both systems score 96.65% (Appendix F). The final combiner is recalibrated per target task, so cross-task stability is not established. Image-dependence diagnostics also do not directly identify the visual source of the final fusion gains. Although acknowledged, these boundaries limit the breadth of the empirical contribution.

Questions for authors
  1. What is the principal generalizable insight beyond existing bidirectional scoring? Can final U-only/U+G inference costs on one representative backbone establish the practical significance of the gains?
  2. Using cached scores, can you provide a small, prespecified feature-group ablation under the same learner and splits, plus learned G-only? Paired uncertainty would help distinguish robust G increments from dependence on the current templates and calibration recipe.
Reviewer_Jn6TOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is an interesting work. The authors formulate a two-candidate caption discrimination task to investigate whether generation can provide useful gains for visual understanding. With the model frozen and without generating new images, they compute log⁡p(c∣x)\log p(c\mid x) through the understanding pathway and log⁡p(x∣c)\log p(x\mid c) for the observed image through the generation pathway, then fuse these scores using a linear combiner fitted on a labeled calibration set. The authors conduct extensive controlled experiments to validate this evaluation approach. Further analyses suggest that gains arise when the understanding pathway still makes errors and the generation scores can distinguish correct from incorrect captions. Overall, the method offers the community a useful approach to measuring the incremental value of generation capabilities in unified multimodal models (UMMs), while the extensive and carefully controlled experiments provide valuable guidance for future UMM development.

Critical strengths
  1. The research question is clearly defined, and the baseline design carefully controls relevant variables. The baseline ladder clearly demonstrates how apparent gains shrink as the understanding baseline becomes stronger, offering methodological guidance for evaluating claims that generation improves understanding.

  2. The experimental controls are extensive, including replication with an independent implementation, zero-information replacements, caption-length controls, occlusion and localization analyses, and a prospective test with a protocol frozen in advance. The authors also transparently report negative results, which strengthens the credibility and reproducibility of the work.

  3. The paper identifies conditions for gains that can be measured on labeled data, providing practical guidance on when generation-based evidence may be useful.

Critical weaknesses
  1. As shown in Table 1, the observed gains primarily occur in models with weaker understanding performance. The largest gain comes from Emu3.5-Image, a generation-specialized model with relatively weak understanding performance, whereas no significant gains are observed in settings with high understanding accuracy, in the prospective test, or with the strong understanding model Qwen3.8. This raises the question of whether generation contributes additional visual evidence or primarily compensates for weaknesses in the understanding pathway.

  2. The strongest understanding-only baseline does not include prompted readouts, such as yes/no judgments, even though these are among the most natural ways to use a UMM’s understanding capabilities. The appendix audits these readouts only on SugarCrepe. Would the gains persist after incorporating prompted readouts on datasets where gains are observed, such as ColorSwap and SPEC?

  3. Theoretically, for an exact joint probabilistic model, G is equivalent to PMI-corrected U, so the gains arise from inconsistencies between the two conditional estimates. Do these inconsistencies reflect complementary visual evidence, or could they instead arise from text priors in the generation head or estimation noise? Could the authors provide further explanation or analysis to distinguish these possibilities?

Questions for authors

Please see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.