acceptodds
Withdrawn from ICLR 2027

OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper presents a family of advanced vision encoders, named OpenVision 3, that learns a **single**, **unified** visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that *synergize* and *generalize* well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder ***frozen***. For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.87 vs. 2.54 on ImageNet). For multimodal understanding, we plug the encoder into the LLaVA-1.5 and LLaVA-NeXT framework: it achieves performance on par with or superior to a standard CLIP vision encoder (e.g., 59.2 vs. 58.1 on GQA and 35.0 vs. 32.0 on ChartQA), and exhibits remarkable scalability. We provide empirical evidence that generation and understanding are mutually beneficial in our architecture, while further underscoring the critical role of the VAE latent space. We hope this work can spur future research on unified modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Pretraining, Unified Vision Encoder, Vision Tokenizer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TygmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • OpenVision 3 combines a frozen VAE with a trainable ViT to learn a shared visual representation for understanding and generation.
  • Training jointly optimizes reconstruction, contrastive, and image-captioning objectives.
Critical strengths
  • The paper demonstrates that jointly optimizing reconstruction, contrastive, and captioning objectives can learn a unified visual representation supporting both understanding and generation.
  • Key evaluations and ablations use controlled settings, helping assess representation quality and the contributions of individual objectives.
Critical weaknesses
  • The architecture is highly similar to TUNA, with limited conceptual novelty. The additional standalone tokenizer-training stage introduces computational overhead whose necessity is not demonstrated.
  • The paper claims to learn a single representation for understanding and generation, yet the understanding, generation, and reconstruction evaluations appear to use different features.
  • The generation experiments are too limited to substantiate the central claim. Lack of comparisons with representative works such as VTP, UniFlow, and UAE.
  • The ablations in Tab.4 and Tab.5 are insufficient, leaving the proposed design choices inadequately validated.
Questions for authors
  • What distinguishes the method from TUNA, and what evidence justifies the additional standalone tokenizer-training stage?
  • Do understanding, generation, and reconstruction use the same features? If not, how does this support the claim of a single unified representation?
  • Can stronger generation experiments, including comparisons with VTP, UniFlow, and UAE, substantiate the claimed benefits of multi-objective training?
  • Can more comprehensive ablations in Tab.4 and Tab.5 establish the effectiveness of the proposed design choices?
Reviewer_ms5bOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OpenVision 3, which learns a unified representation that serves both image understanding and generation. The architecture is simple and effective: a newly trained ViT encoder and decoder are inserted between the VAE encoder and decoder, jointly optimizing the reconstruction capability and semantic expressiveness of image embeddings in the VAE latent space. Within the separate experimental settings for generation and understanding, OpenVision 3 achieves better results than comparable unified encoders. In addition, the paper explores the mutual benefits between understanding and generation. Overall, the work is fairly solid in terms of both novelty and experimental design.

Critical strengths
  1. This paper proposes a simple and effective unified encoding scheme, and further training a ViT in the latent space is a relatively novel attempt.

  2. OpenVision 3 achieves better results in the separate experiments on understanding and generation. For example, it achieves better reconstruction results on ImageNet, and its understanding performance in frameworks such as LLaVA-1.5 is also better than that of OpenAI-CLIP.

Critical weaknesses
  1. Although OpenVision 3 performs well on the separate understanding and generation tasks, it still appears to be inferior to specialized tokenization models for each task. Moreover, the paper does not train a unified understanding-generation model, which prevents OpenVision 3 from truly achieving unified understanding and generation and substantially undermines the competitiveness of such models.
Questions for authors
  1. Why are there no comparisons with other unified tokenizers in the LLaVA-1.5 and LLaVA-Next experiments? Does this suggest that your approach may be inferior to comparable methods in understanding performance?

  2. I have some doubts about the claim that the semantic loss promotes reconstruction. If semantics truly facilitate reconstruction, should OpenVision 3's reconstruction performance in Table 1 exceed that of FLUX-VAE?

  3. How capable is OpenVision 3 in practical applications, for example at resolutions of 512 × 512 or even 1024 × 1024? The highest resolution used in the experiments is 256 × 256; does this suggest that it is not capable of real-world deployment?

Reviewer_6YVROpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes OpenVision 3, a unified visual encoder that stacks a from-scratch ViT on a frozen FLUX VAE. It trains the encoder jointly with a reconstruction branch (decoding back through the VAE) and an understanding branch (contrastive and captioning losses), so that one representation can serve both understanding and generation. The authors evaluate the frozen encoder on reconstruction (rFID 0.187 on ImageNet), class-conditional generation under RAE (gFID 1.87 vs. 2.54 for CLIP), and understanding under LLaVA-1.5 and LLaVA-NeXT (comparable to or better than CLIP). Ablations suggest that the two objectives help each other and that the VAE input is beneficial.

Critical strengths
  1. The question is important. How to build one visual representation that serves both understanding and generation is exactly what the field needs right now. RAE showed that a strong understanding representation can support generation, but its reconstruction quality is limited. This paper’s own CLIP-based RAE baseline in Table 1 reaches only 17.44 PSNR. So a natural next step is to add reconstruction while training the understanding encoder itself, and get a latent that works for both. This paper takes that step directly and trains the encoder from scratch with an open recipe, which I think is a valuable direction.
  2. The evaluation covers all three sides. The paper evaluates the same frozen encoder on reconstruction, generation (under RAE) and understanding (under LLaVA-1.5 and LLaVA-NeXT), so it is easy to see how one representation behaves across tasks. Some findings are useful for anyone thinking about how to design such representations. For example, in Table 5, adding semantic supervision slightly hurts rFID but substantially improves downstream gFID. The results offer some helpful ideas on how to construct a unified visual latent.
Critical weaknesses
  1. The contrastive loss is reintroduced without justification or ablation.

This series is specifically about how to train vision encoders with captioning-based objectives, and the direct predecessor, OpenVision 2, removed the text encoder and contrastive loss. It showed that captioning alone matched OpenVision on multimodal benchmarks at a much lower training cost. This paper adds the contrastive branch back (Eq. 5, α = 1.0), but doesn’t explain why and doesn’t test what it contributes. Since this design choice is exactly the question the series has been studying, reversing the previous conclusion without any supporting experiment is quite confusing to me. It makes it hard to judge whether this choice is actually correct for the unified setting. It also makes it unclear whether the reported benefit of semantic supervision for generation comes from the contrastive term or the captioning term.

  1. The frozen VAE is not well justified.

Putting a frozen FLUX VAE in front of the ViT is a central design choice, but it has clear costs. It caps reconstruction and generation at FLUX’s level, which the authors acknowledge in Sec. 5.3, and it adds complexity to the framework. So I’d expect strong evidence that it is needed, but the ablation in Sec. 5.2 is not convincing to me.

In Table 6, the w/o-VAE variant is actually better on PSNR (32.82 vs. 30.33) and SSIM (0.935 vs. 0.885), with essentially the same LPIPS (0.060 vs. 0.061); only rFID is clearly worse (0.980 vs. 0.216). This looks more like a result of the supervision than of the VAE itself. The reconstruction loss is only L1 + LPIPS, while the VAE variant decodes through FLUX’s decoder, which was trained with much stronger perceptual supervision. In my own experience, a pixel-space encoder trained with captioning and reconstruction losses on DataComp, using MSE, LPIPS and Gram-matrix losses, reaches similar PSNR/SSIM with a much better rFID. So I don’t think a gap in rFID alone is enough to justify adding a frozen VAE. The paper also doesn’t explain why the VAE improves rFID. It may simply be that FLUX’s decoder already produces outputs that suit this kind of distribution-level metric, rather than the VAE latent being a better space for unified learning.

The generation ablation in Table 7 has a similar problem. The gFID of 8.45 vs. 9.68 comes from generators trained for only 100 epochs, far from convergence. At this stage, the difference mostly reflects a small gap in convergence speed and says little about final generation quality. Comparing gFIDs at this level is not very meaningful when they are so far from the final results (e.g., the 1.87 reported in Table 11).

  1. Understanding ability drops compared with the OpenVision and OpenVision2, and this trade-off is not analyzed.

The paper compares understanding performance only with OpenAI CLIP, DFN and LAION. The more relevant baselines are OpenVision and OpenVision 2: they are the direct predecessors, share the same data and captioning-based training, and the understanding branch here “generally follows OpenVision”. Using the numbers reported in those papers under LLaVA-1.5, OpenVision 3 is lower on every listed benchmark:

\hline \text{Encoder} & \text{Source} & \text{MME-P} & \text{SEED} & \text{SQA} & \text{GQA} & \text{POPE} & \text{ChartQA} & \text{OCRBench} \\\\ \hline \text{OpenAI CLIP-L/14 (224)} & \text{OpenVision, Tab. 1} & 1443 & 66.0 & 73.4 & 60.8 & 85.0 & 13.2 & 177 \\\\ \text{OpenVision-L/14 (224)} & \text{OpenVision, Tab. 1} & 1487 & 69.5 & 73.6 & 62.9 & 86.4 & 13.9 & 315 \\\\ \text{OpenVision 2-L/14 (224)} & \text{OpenVision 2, Tab. 1} & 1460 & 69.3 & 76.5 & 62.6 & 87.1 & 13.7 & 327 \\\\ \hline \text{OpenAI CLIP-L/14 (224)} & \text{this paper, Tab. 2} & 1468 & 65.4 & 73.9 & 60.6 & 84.7 & 12.6 & 179 \\\\ \text{OpenVision 3 (VAE+L/2)} & \text{this paper, Tab. 2} & 1405 & 65.8 & 72.4 & 61.0 & 85.2 & 11.9 & 244 \\\\ \hline \end{array}$$ The LLM backbone isn’t specified here, so the protocols may differ slightly. Still, the CLIP-L/14 baseline re-run in this paper closely matches the one in OpenVision. So the unified design (VAE input plus reconstruction) costs a noticeable part of the understanding ability. That trade-off should be measured and discussed openly rather than hidden by comparing only against CLIP. I’d like to see OpenVision 1/2 evaluated in the same pipeline at matched size and token count, including TextVQA, the main metric of the previous two papers.
Questions for authors
  1. Why add the contrastive loss back? OpenVision 2 showed that captioning alone was enough and cheaper. Does that conclusion not hold in the unified setting, or does the contrastive loss give some extra benefit here, for example to generation? An ablation comparing contrastive-only, caption-only and both would answer this directly.
  2. What does the frozen VAE actually bring? In Table 6, the w/o-VAE variant has better PSNR and SSIM and similar LPIPS. Only rFID is worse, and rFID alone doesn’t fully reflect visual quality. Is the rFID gain really from the VAE latent space, or from the supervision (L1 + LPIPS only) and the strength of FLUX’s decoder? Qualitative comparisons, and a w/o-VAE baseline with matched decoder capacity and supervision, would make this clear.
  3. Why is understanding weaker than OpenVision 1/2? Under LLaVA-1.5, OpenVision 3 falls behind both predecessors on all listed benchmarks. Is this caused by the frozen VAE losing semantic or fine-grained information, by the reconstruction objective, or by the smaller training budget? Please report OpenVision 1/2 in the same pipeline at matched size and token count.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.