acceptodds
Under review as a conference paper at ICLR 2027

Let Vision-Language Models Generate Images through Their Representations

Abstract

Recent unified multimodal models (UMMs) aim to combine visual understanding and generation within a single model, yet effectively integrating the two capabilities remains challenging. Leveraging pretrained vision-language models (VLMs) offers a way to build on already learned multimodal knowledge, but making this knowledge effectively accessible to visual generation remains nontrivial. We find that a central challenge lies in the heterogeneous visual representations used for understanding and generation. To address this challenge, we introduce Integral, which converts a pretrained VLM into a UMM by constructing a shared visual representation space across the two capabilities. Integral implements this design through two complementary components. Abstraction Fusion constructs shared visual tokens by aggregating hierarchical representations from the frozen vision encoder, while Block-wise MoT introduces a lightweight generative stream that interacts with the VLM only at block boundaries. Our approach preserves nearly all of the pretrained VLM's understanding capability while extending it to strong visual generation. Controlled comparisons further show that sharing the VLM representation space improves generative learning beyond what can be explained by reconstruction quality alone. Our results indicate that pretrained VLM representations already contain knowledge useful for visual generation, and that preserving their representation hierarchy enables this knowledge to be used without sacrificing visual understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Modeling, Vision-Language Models, Multimodal Language Models, Visual Generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_AsDLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Integral, a unified multimodal framework that extends a pretrained vision-language model (VLM) with image generation capabilities while keeping the original VLM backbone frozen. The authors introduce the Propagation-Bottleneck Hypothesis to explain the trade-off between semantic understanding and image reconstruction. To address this, they propose Abstraction Fusion (AF), which aggregates hierarchical features from the frozen vision encoder and learns a shared visual representation through semantic and reconstruction objectives. Additionally, Block-wise MoT reduces generation-specific parameters by interacting with the frozen VLM only at selected block boundaries. Experiments demonstrate competitive performance across image generation and editing tasks while largely preserving multimodal understanding capabilities.

Critical strengths
  1. Insightful motivation and promising representation design. The Propagation-Bottleneck Hypothesis provides an interesting perspective on the trade-off between semantic understanding and image reconstruction. Building on this motivation, AF effectively leverages hierarchical features from the pretrained VLM's native vision encoder, achieving strong reconstruction quality while largely preserving its understanding capabilities.

  2. Parameter-efficient generation with competitive performance. Block-wise MoT reduces generation-specific parameters while achieving validation Flow Matching losses comparable to Full MoT. The complete Integral system also demonstrates strong performance across multiple image generation and editing benchmarks, highlighting the potential of reusing pretrained VLM capabilities.

Critical weaknesses
  1. Limited methodological novelty

The main components of Integral build upon established techniques. Aggregating intermediate features from pretrained vision encoders has been widely explored in visual representation learning / visual tokenization. For instance, DualToken [1] also discusses the distinct feature requirements of semantic understanding and visual reconstruction, and proposes leveraging shallow intermediate features for reconstruction and deeper features for semantic representation, while RAEv2 [2] and DecQ [3] similarly fuse multi-layer features to enhance both visual understanding and reconstruction quality. The authors should discuss these closely related works and clarify how their approach differs. Furthermore, semantic reconstruction, pixel-level reconstruction, and variational bottlenecks are all well-established techniques for learning reconstruction-capable visual representations. Similarly, Block-wise MoT primarily adjusts the interaction frequency between modalities within an existing MoT architecture, rather than introducing a fundamentally new generative modeling mechanism.

The paper's more interesting contribution is its hypothesis that preserving and reusing the pretrained VLM's representation hierarchy facilitates knowledge transfer from understanding to generation. However, the current experiments do not sufficiently isolate this mechanism. See below.

  1. Insufficient ablations to isolate the contributions of key components.

The final generation results are obtained using AF and Block-wise MoT together, making their individual contributions difficult to assess. Figure 3 compares Block-wise MoT with Full MoT only in terms of validation velocity MSE, without reporting downstream generation performance. Moreover, the contributions of multi-layer feature fusion and semantic reconstruction supervision within AF are not sufficiently examined. The relative importance of features from different encoder depths also remains unclear.

  1. A gap between the main results and the controlled ablation evidence.

While the main experiments demonstrate the competitiveness of Integral as a complete system, the ablation studies are relatively limited in explaining the sources of these improvements. In particular, Figure 4 compares AF with FLUX.1-VAE under the same generation setup, but only during early text-to-image training, without corresponding comparisons on the main downstream benchmarks. Meanwhile, the comparisons with BAGEL and other unified models involve multiple architectural and training differences. As a result, the current evidence does not sufficiently isolate the contribution of AF to the final generation performance, which is important for supporting the paper's central claim about shared visual representations.

[1] DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies, ICLR 2026

[2] Improved Baselines with Representation Autoencoders, Arxiv 2026

[3] DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders, Arxiv 2026

Questions for authors
  1. Component contributions and AF design: Could the authors provide downstream generation results for AF + Full MoT, along with ablations on multi-layer feature fusion and semantic reconstruction supervision? Could they also analyze the learned cross-layer attention distributions and the relative contributions of features from different encoder depths?

  2. Main results and controlled comparisons: Could the authors provide additional downstream generation comparisons that isolate the contribution of AF under otherwise consistent model architectures and training settings? In particular, extending the controlled comparison in Figure 4 to representative main benchmarks would help establish whether the advantages of AF also contribute to the final system performance.

  3. Novelty and related work: What are the key differences between AF and existing multi-layer feature fusion approaches for visual tokenization? Could the authors further clarify the necessity of using the VLM's native visual representations and explicit semantic alignment, and strengthen the discussion of related pretrained-VLM-based generation frameworks, such as BAGEL, LaTtE-Flow, and Bifrost-1?

Reviewer_Jt4mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework that extends a pretrained vision-language model with image generation while largely preserving its visual understanding capabilities. It introduces two components: (1) Abstraction Fusion (AF), which aggregates intermediate representations from a frozen vision encoder to construct visual tokens supporting both semantic understanding and image reconstruction, and (2) Block-wise Mixture-of-Transformers (MoT), which adds a generation-specific pathway conditioned on frozen VLM representations at selected block boundaries. The authors evaluate it on various tasks and the model reportedly preserves 99.8% of the pretrained VLM's understanding performance while achieving competitive generation results with 0.89B generation-specific parameters.

Critical strengths
  1. Integral achieves competitive results on compositional image generation, knowledge-intensive generation, and image editing while preserving 99.8% of the pretrained VLM's understanding performance across several benchmarks. The breadth of evaluation provides meaningful evidence of the framework's practical effectiveness.
  2. The approach retains a frozen pretrained VLM and introduces only 0.89B generation-specific transformer parameters. This provides a practical alternative to retraining large unified multimodal models and reduces the risk of degrading existing understanding capabilities.
  3. Abstraction Fusion exploits intermediate vision encoder features to retain fine-grained visual information while preserving high-level semantic representations.
  4. Beyond conventional text-to-image benchmarks, the paper evaluates generation requiring cultural, temporal, spatial, and scientific knowledge, as well as instruction-based image editing. The strong results are particularly relevant to the proposed goal of leveraging pretrained multimodal knowledge.
Critical weaknesses
  1. Related works missing comparisons on reconstructive multimodal learning and hierarchical representation methods. The central idea of combining semantic and reconstruction-oriented visual information has substantial precedent. X-Former (ECCV 2024) [1] and Reconstructive Visual Instruction Tuning (ICLR 2025) [2] explore image reconstruction objectives for improving fine-grained VLM understanding. More directly, RAEv2 (2026) [3] demonstrates that aggregating intermediate representations from a frozen vision encoder improves reconstruction and generation, while UniAR (ICML 2026) [4] employs multi-level feature fusion for unified understanding and generation. Although Integral differs architecturally, its underlying motivation and several design principles are well established. The paper should more clearly distinguish its contribution from these works.
  2. The paper does not sufficiently isolate the contributions of layer selection, learned attention-based fusion, the variational bottleneck, semantic reconstruction supervision, and pixel reconstruction supervision. In particular, the absence of simple aggregation baselines, such as intermediate-layer summation as explored in RAEv2 [3], makes it difficult to justify the additional complexity of AF.
  3. Unclear distinction between unified representation learning and attaching a generative pathway to a frozen VLM. Integral uses a frozen pretrained VLM and a separate generative stream with one-way conditioning. Although AF is trained to reconstruct the VLM's semantic representation, this does not necessarily establish that the learned visual latents occupy the same representation space. Furthermore, understanding performance is primarily preserved rather than improved through interaction with generation. The paper should clarify the operational meaning of a shared visual representation and distinguish its form of unification from conventional architectures that condition a separate generator on pretrained multimodal features. This distinction is particularly relevant given prior unified-tokenizer approaches such as UniAR [4].
  4. Insufficient targeted evaluation of fine-grained visual understanding preservation. The paper reports 99.8% average retention across 20 understanding benchmarks, but this aggregate result may obscure changes in specific visual capabilities. Since the method explicitly aims to retain fine-grained visual information through intermediate representations, targeted evaluations of object counting, fine-grained object and attribute identification, spatial relations, and compositional understanding would be particularly relevant. Prior work such as X-Former [1] and ROSS [2] evaluates the importance of reconstruction-oriented learning for detailed visual comprehension. Similar analyses would help establish whether AF genuinely preserves fine-grained information rather than simply maintaining the pretrained VLM's overall benchmark performance.
  5. Limited end-to-end generalization across pretrained vision-language models. Although the appendix investigates alternative tokenizer configurations using Qwen3.5-2B and SigLIP2, the complete generation framework is evaluated primarily with Qwen3.5-9B. Consequently, it remains unclear whether the proposed representation-sharing and Block-wise MoT designs offer comparable benefits across different pretrained VLM architectures and scales. This limits the evidence supporting Integral as a general method for extending pretrained VLMs with generation.

References: [1] Sirnam Swetha et al. X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs. ECCV 2024 [2] Haochen Wang et al. Reconstructive Visual Instruction Tuning. ICLR 2025. [3] Jaskirat Singh et al. Improved Baselines with Representation Autoencoders. arXiv:2605.18324 [4] Wujian Peng et al. Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification. ICML 2026

Questions for authors
  1. How does Abstraction Fusion compare with simpler hierarchical feature aggregation methods, such as RAEv2 [3], under matched training conditions?
  2. How does Block-wise MoT compare with full MoT in final generation quality, inference latency, and computational cost, beyond validation velocity MSE?
  3. Does the observed propagation bottleneck correlate with measurable loss of fine-grained semantic information across encoder layers, beyond changes in CKA similarity?
  4. Does AF preserve fine-grained visual understanding capabilities, such as object counting, multi-class identification, attribute recognition, and spatial relationships? Can the authors provide targeted evaluations, such as the OC/MCI tasks used in X-Former [1], to substantiate semantic preservation beyond aggregate benchmark retention?
  5. Shared representation versus conditional generation: How do the authors establish that Integral benefits from a genuinely shared visual representation rather than simply conditioning a separate generative pathway on frozen VLM features?
  6. Do the benefits of AF and Block-wise MoT extend to other pretrained VLM architectures or model scales?
Reviewer_8EXMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about making an existing pretrained vision-language model able to both understand and generate images. It says the main problem is that the features used for understanding and for generation do not match. Its method builds one shared visual feature space in two parts: one part takes features from different depths of a frozen visual encoder and combines them; the other adds a small generation module only at the joins between groups of backbone layers, so it uses few extra parameters but still uses features from many levels. This shared space keeps the meaning of the frozen encoder and is still good enough to rebuild images from. The paper reports results on understanding, image generation, prompt-following, image rebuilding, and some examples, to show generation can be added to a frozen model while understanding is mostly kept.

Critical strengths

One strong point is how the problem is framed and solved. Instead of retraining the encoder, the paper keeps it frozen and combines its features, which avoids the usual fight between "good for understanding" and "good for generation", and does so without adding new parameters. This is exactly what you want when reusing an existing model. A second strong point is the light generation module. Adding generation only at the joins between layer groups keeps the cost low without losing much quality, which matters for real adoption.

Critical weaknesses

The main weakness is that the comparison is not complete. A paper like this is a claim about a new model's quality, so it should be compared against the strongest all-in-one models that exist now. Instead, most comparisons are against partial models, and the true comparable ones are few, older, and missing several recent strong open systems. So the headline numbers are hard to read as proof that the shared-representation idea, rather than a strong base model, gives the quality.

A second weakness is that the cost of sharing one visual space is never isolated. There is no fair, budget-matched test against the closest methods, so it is unclear how much generation quality is paid for sharing, and whether that trade is worth it.

Questions for authors

First, how did you choose which all-in-one models to compare with, and why are several recent strong ones missing? This is the point that matters most to me.

Second, can you give a fair head-to-head against the closest methods at a similar model size and training budget, so the shared-representation benefit can be separated from the base model's strength?

Third, is there any controlled test, single shared space versus two separate spaces under the same budget, that shows what sharing actually costs? If sharing only avoids harm rather than helps, the framing should say so.

Reviewer_6Zy6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a representation that aggregates features from all layers of the VLM's vision encoder and preserves both understanding and generation abilities. Based on layer-wise CKA analysis, the authors argue that when the encoder is fine-tuned, understanding and reconstruction compete for the same high-level features, since reconstruction can only read from the final layer, and propose a cross-layer self-attention module integrated with dual decoders. This provides a direct readout of low-level visual features for reconstruction, while keeping the semantic features used for understanding intact. Combined with a lightweight generation stream on top of the frozen VLM, the method achieves strong results in both understanding and generation.

Critical strengths
  • AF latents train much faster than FLUX.1-VAE latents of comparable reconstruction quality (GenEval 0.41 vs. 0.18 after 50k steps under the same generator, Fig. 4). The resulting system reaches GenEval 0.88 with only 0.89B generation-specific parameters.
  • Higher-level VLM features are kept intact without sacrificing reconstruction ability, avoiding the semantics–reconstruction trade-off of encoder fine-tuning.
  • A flexible, learnable fusion across encoder layers; the same design transfers to Qwen3.5-2B and SigLIP2 (Table 9), suggesting its potential for unified multimodal modeling.
Critical weaknesses
  • Fig. 4 (AF vs. FLUX.1-VAE) controls for reconstruction quality, but cannot separate the claimed benefit of sharing from the known benefit of semantic latents, which speed up generative training even without a VLM (e.g., REPA, ReDi). Training the same generator on a semantic latent that does not come from the conditioning VLM (e.g., the SigLIP2-AF in Table 9) would isolate this; otherwise the claim should be narrowed.
  • The extra cost of AF is not reported. Model size excludes the visual tokenizer, and since the reported understanding results are those of "Qwen3.5-9B w/ AF", every input image also passes through AF and the semantic decoder. Please report the parameters, latency and memory of AF and the semantic decoder.
  • In Table 3, the BAGEL row (47.71 / 52.17 / 40.23 / 47.76) matches the original KRIS-Bench paper, but since June 2025 the official KRIS-Bench leaderboard has listed the BAGEL team's updated results (BAGEL 56.40; BAGEL-Think 60.41). With the updated number, Integral (52.72) no longer outperforms BAGEL as stated in Sec. 3.2. Please update the row and revise the claim accordingly.
Questions for authors
  1. Could you train the same generator (Qwen3.5-9B conditioning, same schedule as Fig. 4) on the SigLIP2-AF latents from Table 9, and report GenEval and WISE^V? This would show whether the gain comes from sharing the VLM's own representation space or from semantic latents in general.
  2. Which judge model and evaluation code version were used for Integral on KRIS-Bench?
  3. Eq. 6 states t ~ U(0,1), while Table 7 lists logit-normal sampling with a time shift of 4.0. Which one was used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.