- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper is analyzing the affect of the projection head on the mutual information loss in terms of distribution. They give some intuition on the Gaussian case, and show experiments on how it differs when selecting different types of projection heads.
Critical strengths
-
Fluent writing, every logical step is well explained and even more appreciated - it is highly motivated. The notations are introduced well such that it helps in let the reader keep track the information flow of the paper.
-
The analysis based on the projection head seems new and novel to me, and a very interesting direction.
Critical weaknesses
-
I would have push several propositions and proofs (i.e. proposition 5.2, which is for gaining more intuition) to the supplementary and devote more of the paper space to justify the claims with more experiments such as: larger datasets, more architectures, maybe even try on different losses or variation of the loss (i.e. temperature). For example the table of performance should be in the main paper on the expanse of the proofs.
-
Several other things can push samples to a specific distribution, like loss of course [1], and architecture (as you are suggesting here) - such as normalizations, layer-normalization, or simply huge amount of multiplications and additions (which can prone to the law of large numbers), the scale of the trained dataset. I think that a discussion on the interplay of the architecture of the critic with respect to other parameters would be appreciated.
-
The experimental section is relatively shallow - the datasets are super small - CiFAR10 and MNIST are toy datasets, to verify the claims it should be examined in larger scale.
-
Maybe I missed it in the paper, but it suggest that the projection head deforming the distribution, but I fail to understand how, for which distribution, and what is preferable. Obviously you have the Gaussian example, but it would be much more interesting to investigate it on real cases, or to gain intuition of how the projection head affect in these cases.
refs:
[1] infoNce induces Gaussian distribution. Betser et al. ICLR 26. (which you cited)
Questions for authors
-
Why have you used only seven classes of imageNet?
-
Where are the implementation details - for someone who would want to reproduce your results? LR? exact architecture? number of epochs? scheduler etc? If they appear in the supp, then I think it should at least be mentioned and referred in the main paper.
-
The visualizations text is too small and hard to read. Moreover you didnt mention in Figs 4 and 5 how do you made the dimensionality reduction.
-
I think this ref is highly relevant to the paper: [1]
This paper is generally interesting, and taking a new approach to tackle the distribution of models, learn it from different perspective. I didnt like that the experimental section is shallow, and I didnt get what is the distribution itself that the critic can push for. Nevertheless I think this paper is shed light on different direction of the infoNCE loss, hence I like it. A better validated and justified claims with a more convincing experiments could lift this paper.
refs:
[1] Aligning Multimodal Representations through an Information Bottleneck. Almudevar et al. ICML 25.
We thank the reviewer for their valuable comments and constructive suggestions.
W1 — Main-text results and additional controls. Section 6.1 reports the score controls (Table 1) and discusses the matched critics (Table 8). We will move the proof of Proposition 5.2 to the supplement and keep the score and intuition in Section 5. On seven ImageNet classes, the depth-2 separable critic achieved 91.15% frozen-feature linear-probe accuracy versus 87.93% for the matched joint MLP, with InfoNCE estimates of 1.579 and 4.289, respectively (Table 8, three seeds). The probe advantage persisted on all three seeds with bounded MLP scores, including with normalized inputs. In a two-seed sweep over temperatures of 0.25, 0.5, and 1.0, the InfoNCE estimate peaked at 0.25 in both seeds, while the best probe temperature differed across seeds. In the seven-class fixed-critic experiment, symmetric InfoNCE raised the 2D probe from 60.8–62.5% to 63.9–64.5% across three paired seeds. ResNet-34 gave mixed results relative to ResNet-18.
W2 — Normalization and training-set size. On seeds 42 and 43, removing final BatchNorm reduced probe accuracy from 61.79/62.50% to 57.81/60.94%. Training on 100 images per class gave 51.56/51.70% and also reduced the number of optimizer steps at fixed epochs. With the optimizer update count and learning-rate schedule matched to the full-data runs, the same 100-image-per-class condition gave 61.08%, 59.52%, and 60.94%, versus 61.79%, 62.50%, and 60.80% with all training images on seeds 42–44. The mean paired gap was 1.18 percentage points (Table 10). LayerNorm remains untested. We will discuss these dependencies alongside Betser et al., who characterize when low-dimensional projections become approximately Gaussian under InfoNCE. Betser et al. (ICLR 2026)
W3 — Experimental scale. The fixed-critic evaluation also includes selected CIFAR-100 coarse superclasses, with permutation-matched k-means accuracy of 69.2–92.1% across three configurations (Table 6). The additional ImageNet experiments cover up to seven selected classes. Full ImageNet-1K validation remains open.
W4 — Which distribution is shaped? The fixed score combines pairwise alignment with the negative log density of an equally weighted Gaussian mixture. This term changes the relative weights of candidate embeddings in InfoNCE. The reference is a marginal over retained encoder embeddings after training-time noise. Proposition 5.2 derives the critic conditional on this marginal. Section 5 now treats recovery of the prescribed marginal by encoder training as an empirical question. Across three ImageNet configurations, the fixed critic achieved 87.02% mean permutation-matched k-means accuracy, versus 81.21% for the separable critic and 84.87% for the joint MLP (Table 7). In 12 of 18 audited 2D runs, squared maximum mean discrepancy between noisy embeddings and the target exceeded the empirical 95th percentile of simulated target-to-target values. To test the role of dimension, we padded the seven heptagon vertices with zeros. Mean validation probe accuracy increased from 61.7% for the 2D heptagon to 70.0% for the same heptagon in 9D, then to 75.5% for the simplex in 9D. Both increases occurred in every paired seed. For both 9D layouts, class silhouette remained near zero and MMD² exceeded the target-to-target reference percentile in all three seeds (Appendix E.2). The 9D layouts thus improved linear prediction despite weak class clustering.
Q1 — Seven ImageNet classes. We used all seven categories in our experimental subset for a controlled comparison with the same data split and training protocol across critics. We selected categories informally for visual distinguishability. The evidence applies to this selected subset.
Q2 — Implementation details. Appendix B gives the fixed-critic architecture in Table 2, the CIFAR augmentations in Table 3, and the training settings for MNIST and CIFAR in Table 4. Appendix D describes the 128D ImageNet comparison, which uses ResNet-18, AdamW (learning rate 0.001, weight decay 0.0001), cosine decay to 0.00001, batch size 256, and 200 epochs. The separate Figure 3 notebook specifies a CIFAR-adapted ResNet-18 with 128D outputs, batch size 1024, 800 epochs, and Adam at 0.01 with no scheduler. Section 6.1 now describes the Figure 3 training and evaluation protocol.
Q3 — Figures 4 and 5. The captions now state that Figures 4 and 5 plot the two learned encoder coordinates directly, without dimensionality reduction. We will enlarge the labels and legends for readability.
Q4 — Almudévar et al. We will cite their information-bottleneck analysis, which motivates regularization for cross-modal alignment. Our experiments study how the pairwise critic shapes retained embeddings from augmented views of one modality. Almudévar et al. (ICML 2025)