- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposed a finite-sample statistical analysis of multimodal contrastive learning by interpreting InfoNCE as an estimator of pointwise mutual information (PMI). Via localized complexity analysis, it leads to fast convergence rates and establishes minimax-optimal PMI estimation rates for both joint-input networks and CLIP-style dual encoders. The results are further extended to representation sufficiency and zero-shot classification.
Critical strengths
-
The paper provides a clear theoretical analysis of InfoNCE, establishing fast convergence rates while accounting for the dependence among negative pairs within a batch. The localized analysis offers a stronger statistical guarantee than conventional global generalization bounds.
-
The matching minimax upper and lower bounds establish the optimality of PMI estimation under Holder smoothness assumptions. Extending these results to CLIP-style dual encoders, along with theoretical guarantees for downstream representation learning, further strengthens the contribution.
Critical weaknesses
-
The theoretical setting differs from practical CLIP architectures and training procedures. The minimax-optimal rate for dual encoders allows the embedding dimension and network capacity to increase with the sample size, while the analysis assumes unnormalized similarity scores and empirical risk minimization over fixed batches. These assumptions limit the direct applicability of the theoretical results to practical CLIP models.
-
The connection between PMI estimation and downstream performance remains somewhat limited. Proposition 5.2 depends on the excess risk of the ideal PMI-based classifier and an additional prompt coverage condition, which may lead to loose guarantees in realistic settings. Moreover, the real-data experiments show relatively low zero-shot classification accuracy without directly evaluating PMI estimation error or representation sufficiency. Thus, the practical significance of the downstream guarantees remains unclear.
Questions for authors
-
Theorem 4.5 establishes minimax-optimal rates for CLIP-style dual encoders, but the embedding dimension and network capacity are allowed to increase with the sample size. How should these results be interpreted for practical CLIP models with fixed embedding dimensions and normalized similarity scores? Clarification of which guarantees remain applicable under these settings would help assess the scope of the theoretical contribution.
-
The theoretical analysis assumes empirical risk minimization over fixed disjoint batches, whereas practical contrastive learning typically uses stochastic optimization with reshuffled batches. How essential are these assumptions to the fast convergence rates? A clarification of their role in the analysis would help establish the applicability of the results to practical training procedures.
-
Proposition 5.2 includes the excess risk of the ideal PMI-based classifier and an additional prompt coverage constant. Under what conditions does this bound provide a meaningful guarantee for downstream classification? Clarifying the effect of these terms would help assess the significance of the theoretical results.
5 comments
Regarding stochastic optimization, our paper focuses on the finite-sample statistical analysis, as is typical in statistical learning theory. Whether SGD converges to the empirical risk minimizer is a question of optimization. The InfoNCE loss is highly nonconvex in the network parameters, so a convergence theory for SGD on the contrastive loss should be treated as a separate work, and it is beyond the scope of this paper. As we pointed out in the original paper, we view this as an interesting direction for future work. We also note that our results still apply to the output of any training procedure, including SGD, in two ways. First, we have now added Theorem E.1 to the revised paper, which holds for every approximate minimizer, and the optimization error appears in addition to the approximation and estimation errors, so any guarantee for the convergence of the optimization algorithm can be combined with it. Second, the lower bound of Theorem 4.6 holds for every estimator based on the pairs, including the output of SGD with reshuffled batches, so no training procedure can improve on the rate in terms of when the PMI is Hölder smooth.
Response to W2 and Q3
(i) W2 and Q3 (downstream guarantee). We thank the reviewer for this question. We discussed the conditions under which Proposition 5.2 gives a meaningful guarantee in Appendix D.2 of the original paper. Since most of this discussion appeared only in the appendix, we have also added a short summary of it to the main text of the revised paper after Proposition 5.2. We summarize it here.
The first term, , is the excess risk of the classifier that uses the true PMI. It does not depend on the learned score or on the sample size . It is small when the logits of the true PMI approximately recover the class probabilities. For example, let be the set of texts that describe class and take the prompt distribution with the class bias . If the image affects the text only through its class, then the logits of the true PMI are exactly the log class probabilities , and the first term is zero (see Appendix D.2 of the original paper for further details). This corresponds to the conditions discussed in Uesaka et al. '25, who also give a bound when the conditions hold only approximately. In the revised paper we also state this bound in Appendix D.2.
The second term, , reflects how accurately the learned score estimates the PMI . For the prompt distributions considered above we showed in Appendix D.2 of the original paper. In this case the second term equals , so it is of the same order as the PMI error . When the first term is small, our results for the PMI error then directly translate into guarantees for zero-shot classification.
(ii) W2 (real data experiments). We would like to first clarify that the PMI error and the representation sufficiency both require the true joint distribution of images and texts, so they cannot be evaluated on real data. In this experiment we instead report zero-shot accuracy, which is a standard evaluation metric for contrastive pretraining (e.g. Radford et al. '21) and can be seen as a proxy for the quality of the learned score.
Regarding the low accuracy, we did not tune the training for accuracy, and both encoders are trained from scratch on at most image-text pairs (far fewer than the 400M pairs used for CLIP in Radford et al. '21). Even so, the top-1 accuracy on ImageNet-1K, which has classes, is about at , which is still far above the random chance (). On Caltech-101, which has classes, the top-1 and top-5 accuracies are also up to and , respectively. More importantly, the goal of this experiment is not to obtain high accuracy but to examine whether the predictions of our theory hold on real data. The results show that they do, i.e., the performance is mainly governed by the total number of pairs , and the batch size is less important.
To further address the reviewer's comment about evaluating the PMI error, we note that our theory also bounds the excess InfoNCE risk (see Theorem 4.2 of the original paper), and this risk can be estimated on real data. We therefore additionally evaluated the held-out InfoNCE loss on the WIT dataset, which is a good proxy for the excess InfoNCE risk. It is computed on pairs of the WIT dataset that are not used for training, for the models of the original paper (fixed batches) and for models trained with batches reshuffled at every epoch. As the table below shows, the loss decreases clearly with , and the effect of the batch size is smaller than that of .
Held-out InfoNCE loss on the WIT dataset
| batches | ||||
|---|---|---|---|---|
| fixed | 6.859 | 6.769 | 7.046 | |
| reshuffled | 6.519 | 6.830 | 7.051 | |
| fixed | 5.784 | 5.792 | 6.024 | |
| reshuffled | 5.391 | 5.876 | 6.107 | |
| fixed | 4.722 | 4.909 | 5.156 | |
| reshuffled | 4.233 | 4.801 | 5.197 | |
| fixed | 3.627 | 3.826 | 4.136 | |
| reshuffled | 3.057 | 3.577 | 4.086 |
We thank the reviewer for the helpful comments and thoughtful feedback. We respond to each point below. All major changes in the revised paper have been highlighted in blue, and the list of references can be found in the global response.
Response to W1 and Q1 and Q2
(i) W1 and Q1 (embedding dimension). Our Theorem 4.5 is interpreted as follows: for a given sample size n, we can find network sizes, including the embedding dimension p, such that the InfoNCE minimizer attains the minimax rate. Choosing the network size as a function of the sample size is very common in statistical learning theory for neural networks (e.g. Schmidt-Hieber '20, Nakada & Imaizumi '20, Han et al. '25, Ma et al. '26). This is also known as the sieve method in nonparametric statistics (Shen & Wong '94), where the class has to grow with n so that its approximation error vanishes as n grows. It is a standard technique for deriving convergence rates in terms of the sample size n.
For a fixed embedding dimension p, a minor modification of Theorem 4.5 also gives the result in terms of the approximation and estimation errors. Specifically, we obtain
ErrQ(s)≲p−2α/mindX,dY+np(dX+dY)/mindX,dY,
up to logarithmic factors, where the first term is the approximation error and the second is the estimation error. We now state this as Corollary E.3 in the revised paper. For a fixed p the approximation error does not go to zero as n grows, so it remains in the bound. If we instead choose p≍nmindX,dY/(2α+dX+dY) in the above, the two terms balance and we can recover the minimax rate n−2α/(2α+dX+dY) for the Hölder class.
(ii) W1 and Q1 (normalized scores). We have now added results for normalized scores (see Lemma E.4 of the revised paper). We consider the class of scores ⟨f/∣f∣2,g/∣g∣2⟩/τ, where f and g are ReLU networks whose norms are bounded away from zero. These scores are bounded by 1/τ, so Proposition 4.1 and Theorem 4.2 of the original paper apply without change. The only change occurs when we compute the covering number of this class, where we use that the normalization u↦u/∣u∣2 is Lipschitz away from zero. This gives the same bound as for unnormalized scores, i.e.,
ErrQ(s)≲ϵnor2+nminS,W2L
up to logarithmic factors, where ϵnor is the approximation error of the class and minS,W2L is the size of the networks as in Corollary 4.4.
(iii) W1 and Q2 (fixed batches). We first note that many existing finite-sample theories of contrastive learning analyze the empirical loss or its minimizer with fixed batches, or under an even more restrictive assumption on the negative pairs (e.g. Saunshi et al. '19, HaoChen et al. '21, Lei et al. '23, Chen et al. '24, Oko et al. '25, Lin & Mei '25, Hieu et al. '25, Li et al. '26). Following this literature, we also analyzed the empirical risk minimizer with fixed batches. Having said this, the fixed-batch assumption is not essential for our analysis, and the same results extend to reshuffled batches. Specifically, for each epoch e=1,…,E, let Be,1,…,Be,N be the reshuffled batches and consider the objective that averages the batch losses over all epochs, i.e.,
LK,E(s)=E1∑e=1EN1∑b=1NℓK(s;Be,b),
where ℓK(s;B) is the InfoNCE loss of a batch B as in the paper. Similar objectives that average a contrastive loss over a collection of mini-batches have been also considered in Cho et al. '24. Then the same concentration bound of Proposition B.7 also holds for the loss averaged over the mini-batches, which allows us to obtain the same convergence rates as in the fixed-batch setting (see Theorem E.1 in Appendix E.1 of the revised paper, and also our response to Reviewer TU7F for further details).