OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation

This paper presents a family of advanced vision encoders, named OpenVision 3, that learns a **single**, **unified** visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that *synergize* and *generalize* well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder ***frozen***. For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.87 vs. 2.54 on ImageNet). For multimodal understanding, we plug the encoder into the LLaVA-1.5 and LLaVA-NeXT framework: it achieves performance on par with or superior to a standard CLIP vision encoder (e.g., 59.2 vs. 58.1 on GQA and 35.0 vs. 32.0 on ChartQA), and exhibits remarkable scalability. We provide empirical evidence that generation and understanding are mutually beneficial in our architecture, while further underscoring the critical role of the VAE latent space. We hope this work can spur future research on unified modeling.