Your Projection Head Secretly Shapes Representations

Projection heads are discarded after contrastive pretraining, yet they strongly affect the representation that remains. We study this effect through the critic induced by the projection head and similarity function, showing that critic form shapes not only how representations are compared, but also which encoder-output distributions are easy to realize. This yields a tradeoff between critic expressiveness, self-supervised loss value, and downstream usability, where restricted critics impose distributional biases that highly expressive non-separable critics can weaken. Experiments support this mechanism: a separable projection-head critic improves linear-probe accuracy over a non-separable MLP critic despite a much lower InfoNCE estimate, and fixed critics steer embeddings toward Gaussian-mixture structures without explicit prior matching.