acceptodds
Under review as a conference paper at ICLR 2027

Your Projection Head Secretly Shapes Representations

Abstract

Projection heads are discarded after contrastive pretraining, yet they strongly affect the representation that remains. We study this effect through the critic induced by the projection head and similarity function, showing that critic form shapes not only how representations are compared, but also which encoder-output distributions are easy to realize. This yields a tradeoff between critic expressiveness, self-supervised loss value, and downstream usability, where restricted critics impose distributional biases that highly expressive non-separable critics can weaken. Experiments support this mechanism: a separable projection-head critic improves linear-probe accuracy over a non-separable MLP critic despite a much lower InfoNCE estimate, and fixed critics steer embeddings toward Gaussian-mixture structures without explicit prior matching.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
information theory, self-supervised representation learning, distribution matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y8HhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is analyzing the affect of the projection head on the mutual information loss in terms of distribution. They give some intuition on the Gaussian case, and show experiments on how it differs when selecting different types of projection heads.

Critical strengths
  • Fluent writing, every logical step is well explained and even more appreciated - it is highly motivated. The notations are introduced well such that it helps in let the reader keep track the information flow of the paper.

  • The analysis based on the projection head seems new and novel to me, and a very interesting direction.

Critical weaknesses
  • I would have push several propositions and proofs (i.e. proposition 5.2, which is for gaining more intuition) to the supplementary and devote more of the paper space to justify the claims with more experiments such as: larger datasets, more architectures, maybe even try on different losses or variation of the loss (i.e. temperature). For example the table of performance should be in the main paper on the expanse of the proofs.

  • Several other things can push samples to a specific distribution, like loss of course [1], and architecture (as you are suggesting here) - such as normalizations, layer-normalization, or simply huge amount of multiplications and additions (which can prone to the law of large numbers), the scale of the trained dataset. I think that a discussion on the interplay of the architecture of the critic with respect to other parameters would be appreciated.

  • The experimental section is relatively shallow - the datasets are super small - CiFAR10 and MNIST are toy datasets, to verify the claims it should be examined in larger scale.

  • Maybe I missed it in the paper, but it suggest that the projection head deforming the distribution, but I fail to understand how, for which distribution, and what is preferable. Obviously you have the Gaussian example, but it would be much more interesting to investigate it on real cases, or to gain intuition of how the projection head affect in these cases.

refs:

[1] infoNce induces Gaussian distribution. Betser et al. ICLR 26. (which you cited)

Questions for authors
  • Why have you used only seven classes of imageNet?

  • Where are the implementation details - for someone who would want to reproduce your results? LR? exact architecture? number of epochs? scheduler etc? If they appear in the supp, then I think it should at least be mentioned and referred in the main paper.

  • The visualizations text is too small and hard to read. Moreover you didnt mention in Figs 4 and 5 how do you made the dimensionality reduction.

  • I think this ref is highly relevant to the paper: [1]

This paper is generally interesting, and taking a new approach to tackle the distribution of models, learn it from different perspective. I didnt like that the experimental section is shallow, and I didnt get what is the distribution itself that the critic can push for. Nevertheless I think this paper is shed light on different direction of the infoNCE loss, hence I like it. A better validated and justified claims with a more convincing experiments could lift this paper.

refs:

[1] Aligning Multimodal Representations through an Information Bottleneck. Almudevar et al. ICML 25.

1 comment

We thank the reviewer for their valuable comments and constructive suggestions.

W1 — Main-text results and additional controls. Section 6.1 reports the score controls (Table 1) and discusses the matched critics (Table 8). We will move the proof of Proposition 5.2 to the supplement and keep the score and intuition in Section 5. On seven ImageNet classes, the depth-2 separable critic achieved 91.15% frozen-feature linear-probe accuracy versus 87.93% for the matched joint MLP, with InfoNCE estimates of 1.579 and 4.289, respectively (Table 8, three seeds). The probe advantage persisted on all three seeds with bounded MLP scores, including with normalized inputs. In a two-seed sweep over temperatures of 0.25, 0.5, and 1.0, the InfoNCE estimate peaked at 0.25 in both seeds, while the best probe temperature differed across seeds. In the seven-class fixed-critic experiment, symmetric InfoNCE raised the 2D probe from 60.8–62.5% to 63.9–64.5% across three paired seeds. ResNet-34 gave mixed results relative to ResNet-18.

W2 — Normalization and training-set size. On seeds 42 and 43, removing final BatchNorm reduced probe accuracy from 61.79/62.50% to 57.81/60.94%. Training on 100 images per class gave 51.56/51.70% and also reduced the number of optimizer steps at fixed epochs. With the optimizer update count and learning-rate schedule matched to the full-data runs, the same 100-image-per-class condition gave 61.08%, 59.52%, and 60.94%, versus 61.79%, 62.50%, and 60.80% with all training images on seeds 42–44. The mean paired gap was 1.18 percentage points (Table 10). LayerNorm remains untested. We will discuss these dependencies alongside Betser et al., who characterize when low-dimensional projections become approximately Gaussian under InfoNCE. Betser et al. (ICLR 2026)

W3 — Experimental scale. The fixed-critic evaluation also includes selected CIFAR-100 coarse superclasses, with permutation-matched k-means accuracy of 69.2–92.1% across three configurations (Table 6). The additional ImageNet experiments cover up to seven selected classes. Full ImageNet-1K validation remains open.

W4 — Which distribution is shaped? The fixed score combines pairwise alignment with the negative log density of an equally weighted Gaussian mixture. This term changes the relative weights of candidate embeddings in InfoNCE. The reference is a marginal over retained encoder embeddings after training-time noise. Proposition 5.2 derives the critic conditional on this marginal. Section 5 now treats recovery of the prescribed marginal by encoder training as an empirical question. Across three ImageNet configurations, the fixed critic achieved 87.02% mean permutation-matched k-means accuracy, versus 81.21% for the separable critic and 84.87% for the joint MLP (Table 7). In 12 of 18 audited 2D runs, squared maximum mean discrepancy between noisy embeddings and the target exceeded the empirical 95th percentile of simulated target-to-target values. To test the role of dimension, we padded the seven heptagon vertices with zeros. Mean validation probe accuracy increased from 61.7% for the 2D heptagon to 70.0% for the same heptagon in 9D, then to 75.5% for the simplex in 9D. Both increases occurred in every paired seed. For both 9D layouts, class silhouette remained near zero and MMD² exceeded the target-to-target reference percentile in all three seeds (Appendix E.2). The 9D layouts thus improved linear prediction despite weak class clustering.

Q1 — Seven ImageNet classes. We used all seven categories in our experimental subset for a controlled comparison with the same data split and training protocol across critics. We selected categories informally for visual distinguishability. The evidence applies to this selected subset.

Q2 — Implementation details. Appendix B gives the fixed-critic architecture in Table 2, the CIFAR augmentations in Table 3, and the training settings for MNIST and CIFAR in Table 4. Appendix D describes the 128D ImageNet comparison, which uses ResNet-18, AdamW (learning rate 0.001, weight decay 0.0001), cosine decay to 0.00001, batch size 256, and 200 epochs. The separate Figure 3 notebook specifies a CIFAR-adapted ResNet-18 with 128D outputs, batch size 1024, 800 epochs, and Adam at 0.01 with no scheduler. Section 6.1 now describes the Figure 3 training and evaluation protocol.

Q3 — Figures 4 and 5. The captions now state that Figures 4 and 5 plot the two learned encoder coordinates directly, without dimensionality reduction. We will enlarge the labels and legends for readability.

Q4 — Almudévar et al. We will cite their information-bottleneck analysis, which motivates regularization for cross-modal alignment. Our experiments study how the pairwise critic shapes retained embeddings from augmented views of one modality. Almudévar et al. (ICML 2025)

Reviewer_R8dzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at how the critic, including the projection head and similarity function, affects learned representations. It compares separable and joint MLP critics and builds fixed critics that encode a desired Gaussian-mixture structure.

Critical strengths
  1. I like the constructive idea: put information about the desired distribution into the critic and use it to guide representation learning. This is an interesting and useful direction.

  2. Figure 3 shows that a better InfoNCE estimate can come with lower linear-probe accuracy. The depth- and parameter-matched comparisons in Table 7 make this observation more convincing.

  3. Lemma 5.1 gives a concrete result in an idealized setting. The fixed-critic visualizations and clustering results provide useful support for the distribution-shaping idea.

Critical weaknesses
  1. My main concern is how the class subsets were chosen for the geometry plots. Different prototype layouts use different classes, so it is hard to judge how representative the examples are or isolate the effect of geometry. Explaining the selection process and keeping the data fixed in a comparison would help.

  2. Some experimental details need clarification, especially Figure 3’s CIFAR-10 baseline of 68.86% after the reported 800 epochs. Please give the feature layer, training settings, and linear-probe protocol. I could not find a code link in the manuscript.

  3. Section 6.2 says no learnable projection head is used, but Appendix B lists an MLP projection head. Please clarify this boundary and which features are plotted and evaluated.

Questions for authors
  1. What parts of Lemma 5.1 carry over when Q\mathbb{Q} changes with the encoder and P\mathbb{P} must come from actual positive pairs? For general augmentation pairs, could the difference between the supports of P\mathbb{P} and Q\mathbb{Q} limit the interpretation of the Gibbs optimum?

  2. In Proposition 5.2, does finding the optimal critic for an assumed target also establish that target as optimal when training the encoder? Could you pls clarify this step and how the Gaussian-noise model relates to image-augmentation pairs.

  3. What exactly makes the joint MLP critic more expressive (I mean: projection head with inner product ⊂\subset MLP class?), and why should that capacity lead it to take on more of the discrimination task during training?

  4. I would find the paper easier to follow if the mutual-information tools appeared closer to where they are used. Their connection to the later fixed-critic construction could be clearer. Specifically, I’m not sure the MI discussion in Section 4 is needed at that point. It distracted me from the main argument as I was reading.

  5. A brief comparison with Distribution Matching [1] in the "Latent priors and distribution matching" could help clarify the paper’s positioning and contribution. Both works report similar geometric clustering in the learned representation space, although they use different mechanisms to obtain it.

Overall

Overall, I think this is an interesting paper with a novel and useful idea. Some analyses use idealized or toy settings, but I like the exploration of shaping representations through critic design. The issues above do not outweigh the core contribution. I recommend 3.

[1] Y. Jiao, W. Ma, D. Sun, H. Wang, and Y. Wang, “Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching,” arXiv:2502.14424v4.

1 comment

We thank the reviewer for their valuable comments and constructive suggestions.

W1 — Class selection and geometry. We chose visually distinguishable categories informally and matched the number of classes to the number of prototypes in each illustration. We then compared elongated and equilateral triangles with the same data split, training settings, paired seeds, and RMS prototype radius (1.41). On airliner, golden retriever, and sports car, elongated minus equilateral linear-probe accuracy was +0.99, +0.99, +0.33, and +0.33 percentage points (seeds 42–45). On a disjoint subset specified before training (bullfrog, container ship, horse sorrel), the differences were −1.99 and −0.33 points (seeds 42–43). Section 6.2 reports the reversal and labels the original plots as illustrations. These selected subsets cannot establish how often either layout wins across ImageNet.

W2 — Figure 3 protocol. The 68.86 ± 1.38% CIFAR-10 result for the separable critic is final test linear-probe accuracy (mean ± SD, six seeds). The available notebook uses a CIFAR-adapted ResNet-18, two random augmented views per image, batch size 1024, 800 epochs, and Adam at 0.01 with no scheduler. Logistic regression (maximum 2,000 iterations) is fitted on frozen 128D encoder outputs before the critic and evaluated on the official test set. The shuffled training loader drops the last incomplete batch, so each fit uses 49,152 of 50,000 training images. Section 6.1 documents this protocol. The exact code revision for each historical run and a public code link are pending.

W3 — Encoder and critic boundary. The MLP in Appendix B is part of the trainable encoder. In the ImageNet implementation it maps 512 → 512 → 128 → dd. Non-affine BatchNorm follows the MLP and outputs the retained representation. The Gaussian-mixture critic uses fixed parameters. Plots and evaluations use encoder outputs before Gaussian noise. Figures 4–5 show the two learned coordinates directly. Section 6.2 now identifies the output of non-affine BatchNorm as the retained representation.

Q1 — Lemma 5.1. For fixed Q,TQ,T, the DV identity is LDV=KL(P∣Q)−KL(P∣G)\mathcal L_{\mathrm{DV}}=\mathrm{KL}(P\\|Q)-\mathrm{KL}(P\\|G), with dG/dQ=eTEQ[eT]dG/dQ=\frac{e^T}{\mathbb E_Q[e^T]}. It yields the Gibbs optimum over the stated KL-constrained class. Under revised Definition 3.1, normalizing TT to T−log⁡EQeTT-\log\mathbb E_Q e^T makes NWJ equal DV, as in Lemma 5.1. During encoder training, the positive-pair law PfP_f and product of marginals QfQ_f both vary. The identity holds pointwise in ff when the terms are finite, but the KL constraint and attainability of GfG_f by positive pairs do not follow. If PfP_f charges a QfQ_f-null set, KL is infinite and the density ratio is undefined. Identical views with continuous embeddings can give this case because their joint law lies on the diagonal.

Q2 — Proposition 5.2 and noise. Proposition 5.2 derives the optimal critic conditional on the assumed noisy marginal. We removed the preceding global-encoder-optimum claim. Encoder training changes both the pair law and the marginal. The Gaussian conditional describes adding noise to the clean embedding. The experiment scores the augmented embedding against that noisy clean embedding, so the conditional given the augmented embedding also depends on the encoder and augmentation pipeline.

Q3 — Joint MLP expressivity. The shared normalized projector and dot product give a symmetric positive-semidefinite kernel with rank bounded by the projector dimension. The joint MLP can express asymmetric scores and nonlinear interactions across views. Exact class inclusion for the particular finite-width MLP would require a separate argument. These interactions could let the critic exploit cues already present in the retained features but hard to use with a linear probe. On seven ImageNet classes, three-seed probe means were 90.39% for separable, 87.22% for matched joint MLP, 88.12% with the MLP scores bounded to ±2, and 87.45% with bounded scores and normalized inputs. This pattern is consistent with the explanation, while the division of work between encoder and critic remains an empirical question.

Q4 — Placement of the MI tools. We will shorten Section 4 to the encoder–critic decomposition and the data-processing observation about the retained layer. We will place the variational tools where Section 5 uses them and move proofs to the supplement. The transition will explain how restricting the score affects which representations make positive pairs distinguishable.

Q5 — Distribution Matching. Jiao et al. combine augmentation alignment with a Mallows–Wasserstein discrepancy between the encoder marginal and a spherical reference. Our construction embeds the Gaussian-mixture reference in the pairwise InfoNCE score. We will cite their work and explain how both objectives can yield clustered representations. Jiao et al. (arXiv v4)

Reviewer_WaScOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how projection heads influence retained representations in contrastive self-supervised learning. It interprets the projection head and similarity function as a critic, arguing that critic restrictions impose distributional biases on encoder outputs. Experiments compare separable and joint MLP critics and demonstrate prototype-oriented embeddings using fixed Gaussian-mixture critics. The paper supports its interpretation with information-theoretic arguments and a proposed distribution-matching duality.

Critical strengths
  • An interesting representation-learning question. Understanding how discarded training components affect retained features is useful for designing self-supervised objectives.

  • Evidence that contrastive objective values and downstream performance can disagree. The CIFAR-10 comparison uses six seeds. Appendix D additionally controls critic depth and parameter count on a seven-class ImageNet subset, where the separable critic achieves higher linear-probe accuracy despite a lower InfoNCE estimate.

  • An interesting constructive experiment. Encoding prescribed geometry into a fixed critic offers a simple way to influence embedding organization. The visualizations and clustering results motivate further investigation.

Critical weaknesses
  1. The NWJ part of Lemma 5.1 is false as stated.

    Taking T = 0 gives G = Q, so the constraint permits only P = Q, and KL(G || Q) = 0. However, Definition 3.1 gives L_NWJ = -1/e, contradicting the claimed equality. The proof incorrectly assumes equality between NWJ and DV at P = G. This requires an additional normalization of the fixed critic; the DV statement does not have this problem.

  2. Proposition 5.2 does not establish encoder-level distribution matching.

    The derivation identifies an optimal critic assuming that the noisy marginal already equals the target distribution. This does not establish the surrounding claim that the target distribution globally maximizes the encoder objective.

    A simple Gaussian example illustrates the issue. Let the noise follow N(0, 1), the target marginal follow N(0, 2), and the encoder output follow Z ~ N(0, v). The proposed fixed critic becomes T(z, y) = -(z-y)^2/2 + y^2/4. The noisy marginal matches the target at v = 1, but the infinite-negative InfoNCE objective has derivative 1/4 > 0 with respect to v at that point. Thus, moving away from the target improves the objective unless additional constraints are imposed.

  3. The proposed encoder–critic tradeoff is not established by Proposition 4.1.

    Data processing and reassignment of layers between encoder and critic establish information ordering and equivalent function decompositions. They do not establish reduced encoder gradients, improved linear accessibility, or a particular output distribution when separately trained critics become more expressive. Critic-induced inductive bias and the mismatch between MI objectives and downstream utility were already emphasized by Tschannen et al. (2020). The additional mechanism claimed here needs stronger support.

  4. The critic comparison leaves score calibration and normalization confounded with architecture.

    The separable critic uses normalized vectors and a fixed-temperature dot product, whereas the joint MLP produces unrestricted scores. Even with matched parameter counts, these choices produce different score ranges and attainable objective values. Temperature sweeps or learned scaling, together with comparable normalization controls, would help determine how much of the observed difference comes from separability.

  5. The distribution-shaping evidence is limited, and implementation details need clarification.

    Most demonstrations use two-dimensional embeddings and selected three- or four-class subsets. Prototype-oriented plots and clustering accuracy do not establish matching of mixture weights, covariances, or density. Quantitative distributional discrepancies and sensitivity to incorrect prototype counts would strengthen the claim. Additionally, Table 3 reports sigma_class = 0 for MNIST, although the displayed critic divides by sigma_class^2. The implemented objective needs clarification.

Questions for authors

See weakness.

1 comment

Thank you for the counterexamples and suggested controls.

W1 — NWJ normalization in Lemma 5.1. Your T=0T=0 example gives −1/e-1/e under the submitted NWJ definition, although G=QG=Q and the lemma predicts zero. Revised Definition 3.1 uses LNWJ=1+EP[T]−EQ[eT]L_{\mathrm{NWJ}}=1+\mathbb E_P[T]-\mathbb E_Q[e^T]. Writing Z=EQ[eT]Z=\mathbb E_Q[e^T], the gap LDV−LNWJ=Z−1−log⁡ZL_{\mathrm{DV}}-L_{\mathrm{NWJ}}=Z-1-\log Z vanishes exactly when Z=1Z=1. Revised Lemma 5.1 assumes this normalization. Then dG/dQ=eTdG/dQ=e^T and both objectives equal EP[T]\mathbb E_P[T] for every admissible PP. Under the lemma’s constraint KL(P∥Q)≤KL(G∥Q)\mathrm{KL}(P\Vert Q)\leq\mathrm{KL}(G\Vert Q), the DV argument gives the common maximum KL(G∥Q)\mathrm{KL}(G\Vert Q) at P=GP=G. For T=0T=0, the revised NWJ value is zero.

W2 — Scope of Proposition 5.2. In your Gaussian example, the noisy marginal matches the target at v=1v=1, where the infinite-negative InfoNCE derivative is 1/41/4. A small increase in vv thus improves the objective. We withdrew the claim that encoder training globally maximizes the objective at the target marginal. Proposition 5.2 now derives an optimal pairwise score given a Gaussian corruption law and a specified noisy marginal. Section 6.2 tests which geometry emerges during encoder training.

W3 — What Proposition 4.1 establishes. Proposition 4.1 leaves open how critic choice affects encoder gradients, the learned marginal, and linear-probe accuracy. The paragraph following it reads:

This proposition establishes a natural information ordering among encoder layers: subsequent internal representations cannot contain more information than the preceding ones. On the other hand, later layers can yield representations that are more accessible to simpler critics, since they allow the same lower bound on MI to be attained with shallower critic architectures.

The ordering concerns MI between the two views’ layer outputs. In a fixed network, moving one layer from critic to encoder lets a shallower critic compute the same score for every pair and attain the same MI bound. Tschannen et al. (2020) already highlighted the role of critic choice and the gap between MI objectives and downstream utility. Our contribution here is the explicit layer-wise construction linking that information ordering to the depth of the extended critic. The revised Discussion treats the proposed division of work during training as a hypothesis. We corrected the figure caption and will qualify the corresponding claims in Section 4, the abstract, and the contribution statements.

W4 — Score range, normalization, and temperature. Table 1 compares four critics on the same seven ImageNet classes and three seeds. Mean linear-probe accuracy was 90.39% for the separable critic, 87.22% for the matched unrestricted joint MLP, 88.12% after bounding the MLP scores to [−2,2][-2,2], and 87.45% after also normalizing its inputs. The corresponding mean InfoNCE estimates were 1.581, 4.206, 2.761, and 2.630. The separable critic led each MLP in probe accuracy on every paired seed. In the separate two-seed sweep (Appendix E.1), raising the separable critic’s temperature from 0.25 to 1.0 lowered InfoNCE from about 2.90 to 0.83 on both seeds. Probe accuracy peaked at 1.0 on seed 42 and 0.5 on seed 43. Even with matched bounds, learned score distributions can differ, so the contribution of separability alone remains open.

W5 — Distributional fit, prototype count, and MNIST variance. In 12 of 18 audited 2D runs, squared RBF MMD between the noisy embeddings and the target mixture exceeded the empirical 95th percentile from ten independent target-to-target sample pairs per run. We use this percentile as a descriptive reference. Across all 18 runs, nearest-prototype assignment frequencies differed from uniform weights by 0.003–0.057 in total variation. The Frobenius distance between the sample covariances of the noisy embeddings and target draws ranged from 0.068 to 0.760. These measure prototype occupancy and overall spread. Recovery of individual component covariances remains open. Both seven-class 9D layouts exceeded their MMD reference on all three seeds (Appendix E.2). The experiments show prototype-oriented geometry, with measurable departures from the target mixture.

With the same three ImageNet classes, two, three, and four prototypes gave probe accuracies of 86.42%, 88.41%, and 89.74% on seed 42 and 87.42%, 89.07%, and 88.08% on seed 43. Matching the class count gave the best probe on one seed.

For MNIST, Table 3 reports the pre-noise component scale σclass=0\sigma_{\mathrm{class}}=0. The implemented mixture uses the post-noise component variance vtarget=σclass2+σnoise2=02+0.62=0.36v_{\mathrm{target}}=\sigma_{\mathrm{class}}^2+\sigma_{\mathrm{noise}}^2=0^2+0.6^2=0.36, so the denominator in its exponent is 2vtarget=0.722v_{\mathrm{target}}=0.72. Proposition 5.2 and Appendix B now specify the post-noise variance explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.