acceptodds
Under review as a conference paper at ICLR 2027

Modality Gap–Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

Abstract

Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. We reveal the structured geometric characteristics of the Modality Gap: it not only contains a global centroid shift, but also exhibits significant anisotropic residuals, while the two modalities still share similar dominant geometric structures. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable pre-training paradigm for Multimodal Large Language Models (MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Modality Gap, Vision-Language Alignment, Representation Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mgo5OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the geometric structure of the modality gap in contrastively pretrained vision-language models and proposes a training paradigm for leveraging unpaired text data in multimodal large language model (MLLM) pretraining. The authors first conduct an empirical analysis across six vision-language encoders and report three observations: (1) the modality gap cannot be primarily explained by a global centroid shift, (2) the residual cross-modal discrepancy exhibits significant anisotropy, and (3) image and text representations share similar normalized eigenspectra and substantially overlapping dominant subspaces. Motivated by these findings, the authors propose ReAlign, a training-free alignment method based on centroid translation, global covariance-trace calibration, spherical normalization, and a final centroid correction. Building on ReAlign, they introduce ReVision, which transforms unpaired text embeddings into pseudo-visual representations for projector pretraining, followed by conventional visual instruction tuning.

Critical strengths

S1. The paper presents a systematic geometric analysis across six pretrained vision-language encoders. The observation that image and text representations can exhibit substantial cross-modal discrepancies while retaining similar dominant geometric structures is interesting.

S2. The proposed ReVision framework provides an interesting way to exploit abundant unpaired text data for projector pretraining. This is a practically relevant direction, given the cost of collecting large-scale, high-quality image-text pairs.

Critical weaknesses

W1. The motivation of this paper is rather confusing. The paper clearly conflates two conceptually distinct research problems: understanding the modality gap in contrastive vision-language pretraining(e.g. CLIP) and reducing reliance on paired image-text data during Multimodal Large Language Models (MLLM) (e.g. LLaVA) pretraining.

The modality gap is primarily studied in the context of contrastively pretrained vision-language models, where people investigate why such a gap emerges during contrastive pretraining and how it affects cross-modal representation alignment, zero-shot image-text retrieval, and related downstream tasks.

However, the modality gap is not inherently a challenge that conventional MLLM pretraining needs to address. In projector-based architectures such as LLaVA, image features are directly mapped into the LLM's embedding space through a trainable projector. The CLIP text encoder is not involved in this process, and explicitly correcting the image-text modality gap in the original CLIP representation space is not a prerequisite for learning the visual projector.

More importantly, modern MLLM pretraining paradigms, such as Qwen-VL and DeepSeek-VL, incorporate visual features as token representations into the language model, and train the vision-language interface directly through next token prediction. Although they may also employ visual encoders and projection modules, their training objectives do not require explicitly correcting the modality gap between independently extracted CLIP image and text embeddings. Consequently, the geometric discrepancy investigated in this paper is not a fundamental obstacle to their pretraining procedures.

In fact, the modality gap becomes relevant to the proposed framework primarily because the authors introduce a specific text-only pretraining strategy in which text embeddings are substituted for real image embeddings. Under this formulation, the distributional mismatch between the two becomes a challenge for transferring a projector pretrained on pseudo-visual features to real visual representations.

Therefore, the actual problem addressed by this work is how to leverage unpaired text data for projector pretraining in a particular class of MLLMs, rather than how to resolve a general modality-gap problem in MLLM pretraining. ReAlign should be understood as a statistical calibration technique for enabling this feature substitution, rather than as a broadly applicable solution to a fundamental MLLM pretraining challenge.

This distinction also raises concerns about the scope and generality of the proposed framework. The experiments focus on a frozen-encoder, projector-based training setup, while the paper does not establish whether the proposed strategy can be extended to more integrated multimodal pretraining pipelines, such as those adopted by Qwen-VL and DeepSeek-VL.

I believe the paper should substantially clarify its problem formulation and scope. The central research question should be framed around data-efficient projector pretraining using unpaired text, with modality-gap correction positioned as a technical mechanism rather than the overarching research objective. The authors should also discuss how their approach relates to modern MLLM architectures and explain the extent to which their conclusions generalize beyond the specific projector-based training paradigm considered in the experiments.

W2. The geometric analysis does not sufficiently justify the proposed alignment method. The paper's main technical motivation is that the modality gap contains substantial anisotropic residuals beyond a global centroid shift. However, ReAlign does not explicitly model or correct these residuals.

In Sec 3.2, the residual covariance is defined through paired cross-modal discrepancies: r=(ex−μx)−(ey−μy)r=(e_x-\mu_x)-(e_y-\mu_y) with covariance Σr=Σx+Σy−Σxy−Σyx\Sigma_r=\Sigma_x+\Sigma_y-\Sigma_{xy}-\Sigma_{yx}, where the cross-covariance terms depend on paired image-text correspondences. In contrast, the core transformation of ReAlign in Section 4 is e~y=μx+s(ey−μy),\tilde e_y=\mu_x+s(e_y-\mu_y), which only depends on marginal statistics and preserves the eigenvectors and relative eigenvalue ratios of the original text covariance before spherical normalization.

Importantly, anisotropy in the paired residual covariance does not necessarily imply anisotropic mismatch between the marginal image and text distributions. Even when the two modalities have identical marginal covariance matrices, their paired residual covariance can still exhibit substantial anisotropy. Therefore, the empirical findings in Sec. 3.1–3.2 do not directly establish the need for, or effectiveness of, the proposed statistical calibration. Although Sec. 3.3 reports similar normalized eigenspectra and substantially overlapping dominant subspaces, these observations do not imply that simple centroid and global-scale calibration is sufficient to align the two modalities.

Overall, there is a substantial disconnect between the geometric properties investigated in Sec. 3 and the transformation proposed in Sec. 4. The authors should clarify precisely which components of the modality gap ReAlign corrects and how the geometric analysis motivates its specific operations.

W3. The geometric interpretation of Trace Alignment is questionable. Consider that $ \Sigma =\mathbb{E}\left[e e^{\top}-e \mu^{\top}-\mu e^{\top}+\mu \mu^{\top}\right] =\mathbb{E}\left[e e^{\top}\right]-\mu \mu^{\top}=1-|\mu|_2^2 $, the scaling factor used in Trace Alignmen, the second step of ReAlign, can be simplified as: $ s= \sqrt{ \frac{1-|\mu_x|_2^2} {1-|\mu_y|_2^2} }. $

Therefore, the covariance traces used by ReAlign are entirely determined by the centroid norms and do not provide independent information about the second-order covariance geometry. This do not invalidate the empirical usefulness of ReAlign, but they raise questions about the proposed geometric interpretation and the extent of its methodological novelty.

W4. The central assumption behind ReVision is that a projector trained on pseudo-visual embeddings generated from text will generalize to real image embeddings during subsequent visual instruction tuning. However, this assumption is not sufficiently examined.

In Sec. 5, ReVision trains the projector to reconstruct text from pseudo-visual embeddings derived from that same text. This objective may primarily teach the projector to decode text-encoder representations into a form usable by the frozen LLM, rather than establish compatibility with real visual representations.

Furthermore, aligning global marginal statistics does not guarantee sample-level semantic correspondence. Two representation distributions can share similar centroids, covariance spectra, and dominant subspaces while exhibiting substantially different semantic correspondences. The geometric evaluations in Sec. 6 primarily measure centroid distances, spectral properties, and pairwise angular distributions. In particular, the near-zero centroid gap in Figure 7 is largely expected because the final step of ReAlign explicitly enforces centroid matching.

W5. The paper claims that statistically aligned unpaired text can effectively substitute for expensive paired image-text data in MLLM pretraining. However, the evidence supporting this claim is limited.

In Table 2, ReVision-1M achieves an average score of 48.05, below the paired image-text baseline at 48.91. ReVision surpasses this baseline only when the number of text-only pretraining samples is doubled to 2M, achieving 49.75. Therefore, the experiments do not establish that unpaired text is competitive with or superior to paired image-text data under matched sample budgets.

More importantly, the proposed framework still relies on substantial paired visual instruction data during Stage 2. Hence, the current experiments demonstrate that unpaired text may partially replace paired supervision during the initial projector-pretraining stage, rather than establishing a broadly applicable replacement for paired image-text supervision in MLLM training.

Questions for authors

Please see weaknesses

Reviewer_AcQwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes the geometry of the image-text modality gap. ReAlign uses unpaired-data statistics for centroid translation, scalar trace alignment, and a final centroid correction. ReVision uses the resulting pseudo-visual features for text-only Stage-1 pretraining before visual instruction tuning. Experiments include three backbone combinations and paired-versus-unpaired pretraining comparisons.

Critical strengths
  1. The alignment is lightweight and interpretable, with component ablations and tests across three backbone combinations.

  2. Table 1 reports higher average scores than C3 Align for all three backbones; LLM2CLIP-Qwen3 improves from 52.11 to 54.39.

Critical weaknesses
  1. Equal-scale substitution is not established: Table 2 gives 48.05 for ReVision-1M against 48.91 for 1M paired data. The 49.75 result uses 2M text samples, so budget-matched efficiency remains unclear.

  2. The 2M-versus-1M paired advantage is 0.84 average points without reported run-to-run uncertainty. Variance estimates are needed to establish whether this small lead is reliable.

Questions for authors
  1. What are the results at matched Stage-1 token and compute budgets?

  2. What is the multi-seed variability of the Table 2 comparison?

2 comments

Thank you for your careful review. We address your two questions below.

W1 & Q1

We clarify that the 1M paired pretraining baseline and ReVision-1M trained by us in the original Table 2 use the same data and semantic supervision. Both settings use the same Bunny samples and caption supervision targets. The main difference lies in the Stage-1 input representations: paired pretraining uses global features from real images, whereas ReVision uses pseudo-visual features obtained by passing the corresponding text through the text encoder and ReAlign. Both settings use the same model configuration, projector, and training protocol, together with the same 417K Stage-2 dataset.

Paired image–text pretraining directly uses real images to provide visual supervision. Therefore, under the same semantic supervision, we use the performance of paired image–text pretraining as an empirical upper bound for pretraining with text substitution. ReVision aims to approach this level while reducing the requirement for per-sample image–text pairing, and to expand its supervision sources through more flexible text data acquisition. In the original Table 2, ReVision-1M and 1M paired pretraining achieve 48.05 and 48.91, respectively, with a gap of 0.86 points.

ReVision’s pretraining stage uses unpaired text to construct pseudo-visual supervision, so adding new text samples does not require collecting a corresponding image for each sample. Higher-quality and more diverse text data can be obtained by integrating existing descriptive corpora, incorporating texts from different domains, and using large language models to generate or expand descriptions of visual content. The original 1M–2M experiments were an initial scaling comparison conducted under the same Bunny data distribution. To further examine whether these approaches to expanding text data translate into performance gains beyond 2M samples, we introduce high-quality text from Unicorn as a supplementary source and extend the experimental scale to 3M. All supplementary experiments use the LLM2CLIP-Openai-L-14-336 encoder, the Llama-3-8B-Instruct language model, and a two-layer MLP projector, while keeping the 417K Stage-2 dataset and Stage-2 training and evaluation protocols consistent.

At a fixed scale of 2M text samples, incorporating Unicorn texts improves the average score from 49.75 with the original Bunny corpus to 52.12, a gain of 2.37 points. We further increase the number of distinct text samples while maintaining an approximately 2:1 ratio of Bunny to Unicorn. The results are as follows:

SettingStage-1 dataSamplesAverage score
Mixed text 1MBunny 0.67M + Unicorn 0.33M1M49.07
Mixed text 2MBunny 1.33M + Unicorn 0.67M2M52.12
Mixed text 3MBunny 2M + Unicorn 1M3M53.86
Paired image–text pretrainingBunny 2M image–text pairs2M52.35

At the same scale of 2M samples, mixed-text pretraining and paired pretraining achieve average scores of 52.12 and 52.35, respectively, with a numerical gap of 0.23 points. Further increasing the text scale to 3M yields an average score of 53.86, improving over mixed text 2M by 1.74 points. These results demonstrate the value of expanding text supervision: introducing new text sources and increasing the corpus scale can further improve pretraining performance.

W2 & Q2

To assess the extent to which the 0.84-point difference in the original Table 2 is affected by evaluation randomness, we use three different evaluation random seeds (0, 1, and 2) to repeat the evaluations under the same evaluation protocol, keeping each model’s checkpoint fixed, and report the mean and standard deviation of the average scores for each setting.

Each evaluation run follows the original procedure to calculate the average score across 11 metrics. We then compute the mean and standard deviation of these average scores across the three evaluation runs. The results are as follows:

SettingNumber of runsaverage score: mean ± standard
1M paired image–text pretraining349.06 ± 0.32
ReVision-1M348.27 ± 0.19
ReVision-2M349.87 ± 0.11

Across repeated evaluations, ReVision-2M and 1M paired pretraining achieve mean scores of 49.87 and 49.06, respectively, with a gap of 0.81 points. The ordering of the mean scores is consistent with the original Table 2.

Thank you for the clarifications and additional experiments. I have also read the other reviews and the authors’ responses. The larger-scale results strengthen the evidence that text-based pretraining is useful. However, the near-parity comparison changes the text corpus, and matched-compute comparisons remain unavailable. Moreover, repeating evaluation with a fixed checkpoint does not address the training variability raised in my review. The evidence therefore remains insufficient to establish the claimed efficiency of replacing paired pretraining. I am keeping my rating at 2.

Reviewer_5eiHOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the geometry of the modality gap in pretrained vision–language encoders and proposes ReAlign, a training-free transformation that calibrates text features through centroid translation, global variance scaling, and post-normalization centroid correction. ReVision uses these features as pseudo-visual inputs to pretrain an MLLM projector, followed by real-image instruction tuning. Experiments show consistent improvements over unaligned text features and C³ across three backbone combinations.

Critical strengths

The method is simple and inexpensive. It requires only unpaired feature statistics and introduces no additional alignment network. The comparisons against No Align and C³ use matched training settings, providing useful evidence that the calibration improves downstream performance.

The geometric analysis goes beyond centroid distance by examining residual anisotropy, eigenspectra, subspace overlap, and neighborhood preservation. The comparison with stronger covariance matching is informative, as it shows that closer statistical matching need not produce better downstream results. Component ablations and domain-sensitivity experiments further support the practical design.

Critical weaknesses

The connection between the geometric analysis and the implemented training interface needs clarification. The analysis concerns contrastive embeddings, but the paper does not sufficiently specify whether MLLM training uses global image embeddings or patch features, sentence-level or token-level text representations, and how their sequence structures are reconciled. The statistics should be estimated on the same features actually supplied to the projector.

The experiments support the usefulness of ReAlign, but do not fully establish why it works. The method adjusts means and overall scale rather than correcting the directional residuals highlighted in the analysis. Comparisons with ordinary normalization, a tunable scalar, or a lightweight affine calibration would help determine whether the gains require the proposed geometric treatment or mainly reflect better-conditioned projector inputs.

The scaling evidence is limited. The paper presents ReVision as a scalable alternative to paired pretraining, but Stage-1 text scaling is tested only from 1M to 2M samples. The 2M result exceeds a fixed 1M paired-data baseline by 0.84 points. Larger text scales and corresponding paired-data or compute-matched comparisons are needed to establish whether the advantage persists, grows, or saturates.

Questions for authors

Which encoder layers and feature granularities are used for geometric analysis, pseudo-visual pretraining, and real-image inference? How are text and image token sequences matched at the projector interface?

Does ReAlign outperform simpler normalization or affine-calibration baselines under the same data and optimization budget?

Does the improvement persist beyond 2M text samples? Can the authors provide larger-scale learning curves alongside paired-data and compute-matched baselines?

Was Unicorn retrained using exactly the same Stage-2 data and protocol? If so, why is its original SFT distribution used to explain the reported results?

4 comments

Thank you for your careful review. We address your four questions below.

W1 & Q1

Encoder outputs. The geometric analysis and statistical calibration use the global representations produced by the image encoder and text encoder in the shared contrastive space. The feature extraction procedures for the two encoders used in the MLLM experiments are as follows:

EncoderImage encoderText encoderOutput dimension
LLM2CLIPThe [CLS] representation from the final layer of the vision encoder is passed through LayerNorm and the visual projection layer to obtain the global image representation.LLM2Vec performs mean pooling over valid text tokens from the final layer, followed by a text adapter that maps the pooled representation into the contrastive space to obtain the global text representation.1280
SigLIP2The final-layer representations of the vision encoder are processed by LayerNorm and multi-head attention pooling to obtain the global image representation.The final-layer representations of the text encoder are processed by LayerNorm, and the representation at the last sequence position is passed through a linear projection head to obtain the global text representation.1152

These descriptions correspond to the encoder architectures of LLM2CLIP-Openai-L-14-336 and SigLIP2-So400M-patch14-384, respectively. The model configurations are provided in Section 6, RQ2, and Appendix E.1 of the original manuscript.

Statistics estimation. After L2 normalization, the text representations produced by the encoder are used to estimate the text centroid and covariance trace; image statistics are estimated from the corresponding global image representations. The calibration procedure further estimates the centroid of the text features after scaling and normalization for the final centroid correction. Image and text statistics are computed independently and do not require paired samples. The representations used for statistics estimation follow the same feature extraction and preprocessing procedures as the corresponding training branches. ReAlign operates after global feature extraction and before sequence expansion and projector mapping. Stage-1 uses calibrated text features, while Stage-2 and inference use the corresponding global image features.

Training pipeline. In the verified LLM2CLIP training implementation, both text and image global representations are repeated along the sequence dimension to produce sequences of the same length before entering the projector, and are then mapped into the LLM hidden space through the same projector. The projector is a two-layer MLP: a linear layer first maps the features to the LLM hidden dimension, followed by a GELU activation and another linear layer that preserves this hidden dimension. The resulting feature sequence is inserted at the image placeholder positions in the LLM input. Stage-1 freezes the LLM and trains only the projector; Stage-2 jointly updates the projector and LLM using real-image instruction data. The sequence lengths and input configurations for each encoder will be specified in Appendix E.1.

W2 & Q2

First, it is necessary to distinguish between the geometry of paired residuals and the geometry of the marginal distributions of the two modalities. Section 3.2 and Figures 3–4 of the original manuscript analyze the paired residuals after removing the centroid difference:

r=(ex−μx)−(ey−μy).r=(e_x-\mu_x)-(e_y-\mu_y).

ReAlign instead estimates the centroids and total variances of the two modalities from unpaired samples to calibrate their marginal distributions, as described in Sections 4.1–4.2. The anisotropy of paired residuals indicates that image–text differences contain directional structure, but does not directly imply that the marginal covariances of the two modalities must be matched along individual directions.

A stronger statistical matching baseline. To examine whether finer directional statistical matching provides additional benefits, we compare against Blockwise Covariance Alignment (BC Align) in Appendices D.1–D.2. This method directly matches first-order and blockwise second-order statistics: beyond matching the means, it adjusts directional variances and correlations within each feature block. It therefore applies a stronger covariance-matching transformation than the global scalar scaling used by ReAlign. However, Appendix D.2, Table 3 reports an average score of 48.74 for BC Align, compared with 50.16 for ReAlign. Appendix D.2, Figure 18(b) further shows that they preserve 10.1% and 87.3% of the original local neighborhoods, respectively. These results indicate that finer covariance matching does not necessarily improve downstream performance and support the design choice of preserving the local structure of the original representations.

Meanwhile, Section 3.3, Figures 5(a)–(b) show that the two modalities already exhibit similar spectral decay patterns and a degree of overlap between their dominant subspaces. ReAlign is therefore designed to preserve this existing structure while calibrating the centroids and overall variance scale, followed by correction of normalization-induced centroid drift. Global scalar calibration preserves the covariance eigendirections and normalized spectral structure before normalization, as analyzed in Sections 4.1–4.2 and Appendix A.1. Normalization-induced centroid drift and its correction are discussed in Section 4.3 and Appendices A.2–A.3. This design does not aim to eliminate all directional residuals for every individual image–text pair.

Component ablation. Appendix F.1, Table 4 further evaluates the contribution of each calibration step:

Calibration methodAverage score
Centroid translation only46.68
Centroid translation + total variance calibration48.67
Full ReAlign50.16

These results show that both total variance calibration and the final centroid correction provide additional gains in the evaluated setting.

Simple input calibration baselines. To directly address the reviewer’s question, we add comparisons with ordinary normalization, fixed scalar scaling, and per-dimension affine calibration. These experiments follow the LLM2CLIP–Llama3 configuration in Section 6, RQ2, Table 1, and the training protocol in Appendix E.1, using the same data, projector, optimization settings, and training budget. The methods differ only in the processing of Stage-1 text inputs; Stage-2 uses the same image inputs and training protocol.

MethodProcessingAverage score
L2-only (No Align)Apply only L2 normalization to the text representations.47.50
L2 + fixed scalar (0.5)Multiply the normalized text representations by a shared scalar of 0.5.48.25
Per-dimension affine calibrationApply per-dimension scaling and translation based on the means and standard deviations of image and text features in each dimension.42.35
ReAlignApply the complete procedure of centroid translation, total variance calibration, and post-normalization centroid correction.50.16

The fixed scalar baseline multiplies L2-normalized text representations by 0.5. Per-dimension affine calibration uses the same unpaired samples as ReAlign and applies per-dimension scaling and translation to the text representations based on the means and standard deviations of image and text features; the calibration parameters of both methods remain fixed during training. ReAlign achieves the highest average score among these comparisons, while per-dimension affine calibration performs worse than No Align. Applying different scaling factors to different dimensions may perturb the original relative variance structure and local neighborhoods; in contrast, ReAlign’s uniform scaling preserves covariance eigendirections and normalized spectral structure before normalization. This observation is consistent with the motivation to preserve the existing geometric structure.

W3 & Q3

ReVision’s pretraining stage uses text-only data, so adding new text samples does not require collecting a corresponding image for each sample. The text corpus can be expanded with higher-quality and more diverse data by integrating existing descriptive corpora, incorporating texts from different domains, and using large language models to generate or expand captions of visual content. The original 1M–2M experiments were an initial scaling comparison conducted under the same Bunny data distribution. To further examine whether these approaches to expanding text data translate into performance gains beyond 2M samples, we introduce high-quality text from Unicorn as a supplementary source.

To further examine the role of corpus quality and semantic coverage, we select a fixed set of 1M high-quality texts from Unicorn-1.2M and combine it with the original Bunny texts for the supplementary experiments. We apply fixed sampling and cross-source deduplication rules and construct nested 1M, 2M, and 3M text subsets with an approximately 2:1 ratio of Bunny to Unicorn, keeping the source proportions consistent as the data scale increases.

3.1 Introducing high-quality text at a fixed sample size

With the total number of Stage-1 text samples fixed at 2M, we compare the original Bunny corpus with a mixed corpus incorporating high-quality Unicorn texts.

SettingStage-1 data compositionDistinct text samplesAverage score
Original corpusBunny 2M2M49.75
Incorporating high-quality textBunny 1.33M + Unicorn 0.67M2M52.12

At the same text scale and with the same model training settings, incorporating Unicorn texts improves the average score from 49.75 to 52.12, a gain of 2.37 points. This indicates that, in addition to sample count, corpus composition is an important factor affecting the effectiveness of text pretraining.

3.2 Scaling with a fixed mixture ratio and comparison with paired pretraining

Keeping the Bunny-to-Unicorn mixture ratio consistent, we increase the number of distinct text samples from 1M to 2M and 3M. The 2M setting is the same mixed-corpus experiment reported in Section 3.1. We also add a paired pretraining baseline using 2M Bunny image–text pairs.

SettingStage-1 data compositionDistinct samplesTraining epochsAverage score
Mixed text 1MBunny 0.67M + Unicorn 0.33M1M149.07
Mixed text 2MBunny 1.33M + Unicorn 0.67M2M152.12
Mixed text 3MBunny 2M + Unicorn 1M3M153.86
Paired image–text baselineBunny 2M image–text pairs2M152.35

Within the evaluated range, the average score of mixed-text pretraining increases from 49.07 to 53.86. At the same scale of 2M samples, mixed-text pretraining and paired image–text pretraining achieve 52.12 and 52.35, respectively, with a gap of 0.23 points. Increasing the text scale to 3M distinct samples yields an average score of 53.86, exceeding the 2M paired image–text baseline by 1.51 points. These results extend the experiments beyond 2M text samples and show that adding more distinct texts continues to provide gains under the current data and training settings.

Q4

We clarify that the statement “using the same Stage-2 data and training protocol” refers to the paired pretraining baseline and ReVision comparisons trained by us in Table 2. The Unicorn scores were quoted directly from its original paper; Unicorn was not retrained using our Stage-2 data and training protocol. The discussion of Unicorn’s SFT data distribution refers to its original training setup and explains the differences in data conditions between the quoted results and our experiments.

To further compare performance under the same data setting, the additional experiment trains ReVision using the same pretraining and instruction-tuning data as Unicorn. In this supplementary experiment, both the pretraining and instruction-tuning stages of ReVision use only text data, and the training data are identical to those used by Unicorn. The additional ReVision comparison in this section uses the LLM2CLIP-Openai-L-14-336 encoder, the Llama-3-8B-Instruct language model, and a two-layer MLP projector. The Unicorn comparison corresponds to the Unicorn-8B model quoted from its original paper.

The table below retains the Unicorn scores quoted in Table 2 of the original manuscript and reports the ReVision results under the same data setting.

BenchmarkUnicorn-8BReVision
MME60.2463.12
MMStar35.1336.79
ScienceQA68.8170.03
RealWorldQA42.3544.14
MMMU36.8737.74
MMMU-Pro34.0535.19
VisuLogic26.8027.50
LogicVista29.5330.15
CRPE42.3247.96
POPE64.2166.78
HallusionBench43.0145.33
Average score43.9445.88

Thank you for the clarifications and additional experiments. The confirmed use of repeated global embeddings limits the demonstrated scope to this particular interface. My main concern about the scaling comparison also remains: the reported Bunny-only 2M text setting scores 49.75 versus 52.35 for 2M paired data, while the near-parity result uses a different text mixture. The 3M experiment shows further gains, but still lacks a compute-matched paired comparison. These results support the usefulness of text pretraining, but do not sufficiently establish the claimed efficiency of replacing paired pretraining. I am therefore lowering my rating from 3 to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.