- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the geometric structure of the modality gap in contrastively pretrained vision-language models and proposes a training paradigm for leveraging unpaired text data in multimodal large language model (MLLM) pretraining. The authors first conduct an empirical analysis across six vision-language encoders and report three observations: (1) the modality gap cannot be primarily explained by a global centroid shift, (2) the residual cross-modal discrepancy exhibits significant anisotropy, and (3) image and text representations share similar normalized eigenspectra and substantially overlapping dominant subspaces. Motivated by these findings, the authors propose ReAlign, a training-free alignment method based on centroid translation, global covariance-trace calibration, spherical normalization, and a final centroid correction. Building on ReAlign, they introduce ReVision, which transforms unpaired text embeddings into pseudo-visual representations for projector pretraining, followed by conventional visual instruction tuning.
Critical strengths
S1. The paper presents a systematic geometric analysis across six pretrained vision-language encoders. The observation that image and text representations can exhibit substantial cross-modal discrepancies while retaining similar dominant geometric structures is interesting.
S2. The proposed ReVision framework provides an interesting way to exploit abundant unpaired text data for projector pretraining. This is a practically relevant direction, given the cost of collecting large-scale, high-quality image-text pairs.
Critical weaknesses
W1. The motivation of this paper is rather confusing. The paper clearly conflates two conceptually distinct research problems: understanding the modality gap in contrastive vision-language pretraining(e.g. CLIP) and reducing reliance on paired image-text data during Multimodal Large Language Models (MLLM) (e.g. LLaVA) pretraining.
The modality gap is primarily studied in the context of contrastively pretrained vision-language models, where people investigate why such a gap emerges during contrastive pretraining and how it affects cross-modal representation alignment, zero-shot image-text retrieval, and related downstream tasks.
However, the modality gap is not inherently a challenge that conventional MLLM pretraining needs to address. In projector-based architectures such as LLaVA, image features are directly mapped into the LLM's embedding space through a trainable projector. The CLIP text encoder is not involved in this process, and explicitly correcting the image-text modality gap in the original CLIP representation space is not a prerequisite for learning the visual projector.
More importantly, modern MLLM pretraining paradigms, such as Qwen-VL and DeepSeek-VL, incorporate visual features as token representations into the language model, and train the vision-language interface directly through next token prediction. Although they may also employ visual encoders and projection modules, their training objectives do not require explicitly correcting the modality gap between independently extracted CLIP image and text embeddings. Consequently, the geometric discrepancy investigated in this paper is not a fundamental obstacle to their pretraining procedures.
In fact, the modality gap becomes relevant to the proposed framework primarily because the authors introduce a specific text-only pretraining strategy in which text embeddings are substituted for real image embeddings. Under this formulation, the distributional mismatch between the two becomes a challenge for transferring a projector pretrained on pseudo-visual features to real visual representations.
Therefore, the actual problem addressed by this work is how to leverage unpaired text data for projector pretraining in a particular class of MLLMs, rather than how to resolve a general modality-gap problem in MLLM pretraining. ReAlign should be understood as a statistical calibration technique for enabling this feature substitution, rather than as a broadly applicable solution to a fundamental MLLM pretraining challenge.
This distinction also raises concerns about the scope and generality of the proposed framework. The experiments focus on a frozen-encoder, projector-based training setup, while the paper does not establish whether the proposed strategy can be extended to more integrated multimodal pretraining pipelines, such as those adopted by Qwen-VL and DeepSeek-VL.
I believe the paper should substantially clarify its problem formulation and scope. The central research question should be framed around data-efficient projector pretraining using unpaired text, with modality-gap correction positioned as a technical mechanism rather than the overarching research objective. The authors should also discuss how their approach relates to modern MLLM architectures and explain the extent to which their conclusions generalize beyond the specific projector-based training paradigm considered in the experiments.
W2. The geometric analysis does not sufficiently justify the proposed alignment method. The paper's main technical motivation is that the modality gap contains substantial anisotropic residuals beyond a global centroid shift. However, ReAlign does not explicitly model or correct these residuals.
In Sec 3.2, the residual covariance is defined through paired cross-modal discrepancies: with covariance , where the cross-covariance terms depend on paired image-text correspondences. In contrast, the core transformation of ReAlign in Section 4 is which only depends on marginal statistics and preserves the eigenvectors and relative eigenvalue ratios of the original text covariance before spherical normalization.
Importantly, anisotropy in the paired residual covariance does not necessarily imply anisotropic mismatch between the marginal image and text distributions. Even when the two modalities have identical marginal covariance matrices, their paired residual covariance can still exhibit substantial anisotropy. Therefore, the empirical findings in Sec. 3.1–3.2 do not directly establish the need for, or effectiveness of, the proposed statistical calibration. Although Sec. 3.3 reports similar normalized eigenspectra and substantially overlapping dominant subspaces, these observations do not imply that simple centroid and global-scale calibration is sufficient to align the two modalities.
Overall, there is a substantial disconnect between the geometric properties investigated in Sec. 3 and the transformation proposed in Sec. 4. The authors should clarify precisely which components of the modality gap ReAlign corrects and how the geometric analysis motivates its specific operations.
W3. The geometric interpretation of Trace Alignment is questionable. Consider that $ \Sigma =\mathbb{E}\left[e e^{\top}-e \mu^{\top}-\mu e^{\top}+\mu \mu^{\top}\right] =\mathbb{E}\left[e e^{\top}\right]-\mu \mu^{\top}=1-|\mu|_2^2 $, the scaling factor used in Trace Alignmen, the second step of ReAlign, can be simplified as: $ s= \sqrt{ \frac{1-|\mu_x|_2^2} {1-|\mu_y|_2^2} }. $
Therefore, the covariance traces used by ReAlign are entirely determined by the centroid norms and do not provide independent information about the second-order covariance geometry. This do not invalidate the empirical usefulness of ReAlign, but they raise questions about the proposed geometric interpretation and the extent of its methodological novelty.
W4. The central assumption behind ReVision is that a projector trained on pseudo-visual embeddings generated from text will generalize to real image embeddings during subsequent visual instruction tuning. However, this assumption is not sufficiently examined.
In Sec. 5, ReVision trains the projector to reconstruct text from pseudo-visual embeddings derived from that same text. This objective may primarily teach the projector to decode text-encoder representations into a form usable by the frozen LLM, rather than establish compatibility with real visual representations.
Furthermore, aligning global marginal statistics does not guarantee sample-level semantic correspondence. Two representation distributions can share similar centroids, covariance spectra, and dominant subspaces while exhibiting substantially different semantic correspondences. The geometric evaluations in Sec. 6 primarily measure centroid distances, spectral properties, and pairwise angular distributions. In particular, the near-zero centroid gap in Figure 7 is largely expected because the final step of ReAlign explicitly enforces centroid matching.
W5. The paper claims that statistically aligned unpaired text can effectively substitute for expensive paired image-text data in MLLM pretraining. However, the evidence supporting this claim is limited.
In Table 2, ReVision-1M achieves an average score of 48.05, below the paired image-text baseline at 48.91. ReVision surpasses this baseline only when the number of text-only pretraining samples is doubled to 2M, achieving 49.75. Therefore, the experiments do not establish that unpaired text is competitive with or superior to paired image-text data under matched sample budgets.
More importantly, the proposed framework still relies on substantial paired visual instruction data during Stage 2. Hence, the current experiments demonstrate that unpaired text may partially replace paired supervision during the initial projector-pretraining stage, rather than establishing a broadly applicable replacement for paired image-text supervision in MLLM training.
Questions for authors
Please see weaknesses
Thank you for your careful review. We address your two questions below.
We clarify that the 1M paired pretraining baseline and ReVision-1M trained by us in the original Table 2 use the same data and semantic supervision. Both settings use the same Bunny samples and caption supervision targets. The main difference lies in the Stage-1 input representations: paired pretraining uses global features from real images, whereas ReVision uses pseudo-visual features obtained by passing the corresponding text through the text encoder and ReAlign. Both settings use the same model configuration, projector, and training protocol, together with the same 417K Stage-2 dataset.
Paired image–text pretraining directly uses real images to provide visual supervision. Therefore, under the same semantic supervision, we use the performance of paired image–text pretraining as an empirical upper bound for pretraining with text substitution. ReVision aims to approach this level while reducing the requirement for per-sample image–text pairing, and to expand its supervision sources through more flexible text data acquisition. In the original Table 2, ReVision-1M and 1M paired pretraining achieve 48.05 and 48.91, respectively, with a gap of 0.86 points.
ReVision’s pretraining stage uses unpaired text to construct pseudo-visual supervision, so adding new text samples does not require collecting a corresponding image for each sample. Higher-quality and more diverse text data can be obtained by integrating existing descriptive corpora, incorporating texts from different domains, and using large language models to generate or expand descriptions of visual content. The original 1M–2M experiments were an initial scaling comparison conducted under the same Bunny data distribution. To further examine whether these approaches to expanding text data translate into performance gains beyond 2M samples, we introduce high-quality text from Unicorn as a supplementary source and extend the experimental scale to 3M. All supplementary experiments use the LLM2CLIP-Openai-L-14-336 encoder, the Llama-3-8B-Instruct language model, and a two-layer MLP projector, while keeping the 417K Stage-2 dataset and Stage-2 training and evaluation protocols consistent.
At a fixed scale of 2M text samples, incorporating Unicorn texts improves the average score from 49.75 with the original Bunny corpus to 52.12, a gain of 2.37 points. We further increase the number of distinct text samples while maintaining an approximately 2:1 ratio of Bunny to Unicorn. The results are as follows:
At the same scale of 2M samples, mixed-text pretraining and paired pretraining achieve average scores of 52.12 and 52.35, respectively, with a numerical gap of 0.23 points. Further increasing the text scale to 3M yields an average score of 53.86, improving over mixed text 2M by 1.74 points. These results demonstrate the value of expanding text supervision: introducing new text sources and increasing the corpus scale can further improve pretraining performance.
To assess the extent to which the 0.84-point difference in the original Table 2 is affected by evaluation randomness, we use three different evaluation random seeds (0, 1, and 2) to repeat the evaluations under the same evaluation protocol, keeping each model’s checkpoint fixed, and report the mean and standard deviation of the average scores for each setting.
Each evaluation run follows the original procedure to calculate the average score across 11 metrics. We then compute the mean and standard deviation of these average scores across the three evaluation runs. The results are as follows:
Across repeated evaluations, ReVision-2M and 1M paired pretraining achieve mean scores of 49.87 and 49.06, respectively, with a gap of 0.81 points. The ordering of the mean scores is consistent with the original Table 2.