- The contrastive loss is reintroduced without justification or ablation.
This series is specifically about how to train vision encoders with captioning-based objectives, and the direct predecessor, OpenVision 2, removed the text encoder and contrastive loss. It showed that captioning alone matched OpenVision on multimodal benchmarks at a much lower training cost. This paper adds the contrastive branch back (Eq. 5, α = 1.0), but doesn’t explain why and doesn’t test what it contributes. Since this design choice is exactly the question the series has been studying, reversing the previous conclusion without any supporting experiment is quite confusing to me. It makes it hard to judge whether this choice is actually correct for the unified setting. It also makes it unclear whether the reported benefit of semantic supervision for generation comes from the contrastive term or the captioning term.
- The frozen VAE is not well justified.
Putting a frozen FLUX VAE in front of the ViT is a central design choice, but it has clear costs. It caps reconstruction and generation at FLUX’s level, which the authors acknowledge in Sec. 5.3, and it adds complexity to the framework. So I’d expect strong evidence that it is needed, but the ablation in Sec. 5.2 is not convincing to me.
In Table 6, the w/o-VAE variant is actually better on PSNR (32.82 vs. 30.33) and SSIM (0.935 vs. 0.885), with essentially the same LPIPS (0.060 vs. 0.061); only rFID is clearly worse (0.980 vs. 0.216). This looks more like a result of the supervision than of the VAE itself. The reconstruction loss is only L1 + LPIPS, while the VAE variant decodes through FLUX’s decoder, which was trained with much stronger perceptual supervision. In my own experience, a pixel-space encoder trained with captioning and reconstruction losses on DataComp, using MSE, LPIPS and Gram-matrix losses, reaches similar PSNR/SSIM with a much better rFID. So I don’t think a gap in rFID alone is enough to justify adding a frozen VAE. The paper also doesn’t explain why the VAE improves rFID. It may simply be that FLUX’s decoder already produces outputs that suit this kind of distribution-level metric, rather than the VAE latent being a better space for unified learning.
The generation ablation in Table 7 has a similar problem. The gFID of 8.45 vs. 9.68 comes from generators trained for only 100 epochs, far from convergence. At this stage, the difference mostly reflects a small gap in convergence speed and says little about final generation quality. Comparing gFIDs at this level is not very meaningful when they are so far from the final results (e.g., the 1.87 reported in Table 11).
- Understanding ability drops compared with the OpenVision and OpenVision2, and this trade-off is not analyzed.
The paper compares understanding performance only with OpenAI CLIP, DFN and LAION. The more relevant baselines are OpenVision and OpenVision 2: they are the direct predecessors, share the same data and captioning-based training, and the understanding branch here “generally follows OpenVision”. Using the numbers reported in those papers under LLaVA-1.5, OpenVision 3 is lower on every listed benchmark:
\hline
\text{Encoder} & \text{Source} & \text{MME-P} & \text{SEED} & \text{SQA} & \text{GQA} & \text{POPE} & \text{ChartQA} & \text{OCRBench} \\\\
\hline
\text{OpenAI CLIP-L/14 (224)} & \text{OpenVision, Tab. 1} & 1443 & 66.0 & 73.4 & 60.8 & 85.0 & 13.2 & 177 \\\\
\text{OpenVision-L/14 (224)} & \text{OpenVision, Tab. 1} & 1487 & 69.5 & 73.6 & 62.9 & 86.4 & 13.9 & 315 \\\\
\text{OpenVision 2-L/14 (224)} & \text{OpenVision 2, Tab. 1} & 1460 & 69.3 & 76.5 & 62.6 & 87.1 & 13.7 & 327 \\\\
\hline
\text{OpenAI CLIP-L/14 (224)} & \text{this paper, Tab. 2} & 1468 & 65.4 & 73.9 & 60.6 & 84.7 & 12.6 & 179 \\\\
\text{OpenVision 3 (VAE+L/2)} & \text{this paper, Tab. 2} & 1405 & 65.8 & 72.4 & 61.0 & 85.2 & 11.9 & 244 \\\\
\hline
\end{array}$$
The LLM backbone isn’t specified here, so the protocols may differ slightly. Still, the CLIP-L/14 baseline re-run in this paper closely matches the one in OpenVision. So the unified design (VAE input plus reconstruction) costs a noticeable part of the understanding ability. That trade-off should be measured and discussed openly rather than hidden by comparing only against CLIP. I’d like to see OpenVision 1/2 evaluated in the same pipeline at matched size and token count, including TextVQA, the main metric of the previous two papers.