acceptodds
Under review as a conference paper at ICLR 2027

On the Limitations of Identifiability Theory for Contrastive Learning

Abstract

Recent identifiability results for contrastive learning connect successful representation learning to recovery of latent factors underlying an invertible data-generating process. We study a limitation of using such recovery as an explanation of downstream performance: latent recoverability, task sufficiency, and stability are distinct properties. Focusing on classification, we first introduce class-conditional structure into standard latent-variable models and show how concentration simultaneously reduces class ambiguity and shrinks the scale of within-class variation. Thus, even exact access to the latent state does not by itself guarantee discriminability, while resolving individual instances at fixed effective precision becomes harder as within-class variation contracts. We then derive pairwise lower bounds linking recovery error, classification error, representation geometry, and probe Lipschitz constants. These bounds expose a recovery–prediction–stability trilemma: when a learned representation compresses latent distinctions that a downstream probe must preserve, accuracy can be maintained only at the cost of increased sensitivity. We further characterize a directional mechanism by which variations in distinct latent factors can interfere through a shared representation. Controlled simulations and experiments on four disentanglement benchmarks exhibit the predicted accuracy–recovery tensions and coupled degradation under perturbations. Our results suggest that identifiability guarantees should be complemented by explicit assumptions on task structure, representation geometry, and stability when they are used to explain downstream performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive learning, Self-supervised learning, Representation learning, Identifiability, Downstream generalization, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tumkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether latent-factor identifiability results for contrastive learning can explain downstream classification performance and stability. It distinguishes latent recoverability, task sufficiency, and robustness. In a class-conditional latent model, it analyzes how class concentration can reduce ambiguity while shrinking within-class variation. It then derives pairwise bounds relating recovery and prediction errors to representation distances and probe Lipschitz constants, and gives a conditional result for interference between latent factors that act along a shared representation direction. Simulations and experiments on four disentanglement benchmarks compare latent recovery, category performance, and behavior under random or adversarial perturbations.

Critical strengths

The paper makes a useful conceptual distinction: recovering the complete latent state does not determine whether class labels are discernible from that state, and clean prediction does not establish stability. The class-conditional discussion helps specify the extra assumptions needed when identifiability results are used to explain downstream performance.

The authors test the proposed tensions in a controlled setting with known latent factors. Reporting category and other-latent recovery across dSprites, shapes3D, smallNORB, and cars3D, including perturbed observations, provides more insight than clean classification accuracy alone. The paper also acknowledges that its theoretical conditions do not imply every contrastive learner exhibits the same trade-off.

Critical weaknesses

The central “trilemma” does not yet establish a mechanism specific to contrastive learning. Theorem 1 follows from pairwise triangle inequalities and applies to arbitrary representations and Lipschitz probes; its consequence requires the learned representation to compress relevant pairs. Corollary 2 further assumes exact alignment between a label-preserving change and a class-changing representation direction. The benchmark experiments show associated changes in recovery and prediction, but do not measure the representation separations, local probe sensitivities, or directional alignment required to connect those observations to the bounds.

Some empirical evidence is difficult to interpret as loss of latent recoverability. In the vMF experiment, Table 1 compares classification using the true latent state with reconstruction (R^2) from a learned representation as concentration changes. Because within-class variance also shrinks, a decline in (R^2) alone does not show that absolute reconstruction error increases. The proposed effective-resolution threshold is not measured for the trained system. The image experiments rely mainly on one instance-discrimination framework and three split seeds; the appendix gives limited details for reproducing the category-targeted PGD attacks. These choices limit how broadly the observed patterns support the paper’s contrastive-learning claim.

Questions for authors

Can the authors directly test Theorem 1 on the trained representations by reporting latent-to-representation distance ratios, recovery and classification errors, and estimates of local probe sensitivity? For the cars3D cross-factor result, how close are the relevant representation changes to the shared-direction assumption in Corollary 2?

In the vMF experiment, how do absolute latent reconstruction error and within-class-normalized error vary with (\kappa)? Can the effective resolution (\epsilon_{\mathrm{eff}}) be measured or estimated independently, rather than chosen to explain the observed concentration range?

1 comment

Thank you for asking when the bounds are informative. Theorem 1 applies to any encoder. DIET assigns augmented views the same instance target. We tested whether its trained encoders compress pose differences needed downstream.

1. When is Theorem 1 informative? We compared clean Cars3D views at adjacent elevations with car identity and azimuth fixed. We fitted affine probes on separate reference views, standardized elevation, and RMS-normalized the centered features. Appendix A.2 reports all five settings. For each pair, B=[∣Δz∣−ei−ej]+/∥Δh∥B=[|\Delta z|-e_i-e_j]_+/\|\Delta h\| is the required gain and G=∣Δz^∣/∥Δh∥G=|\Delta\hat z|/\|\Delta h\| the observed gain, with absolute endpoint errors ei,eje_i,e_j.

Clean Cars3D diagnosticNo augmentationSimCLR-style augmentation
Median elevation distance / representation distance0.86 ± 0.012.34 ± 0.08
95th-percentile required gain BB0.60 ± 0.021.88 ± 0.12
95th-percentile observed gain GG1.13 ± 0.062.94 ± 0.14
Elevation RMSE (standardized levels)0.708 ± 0.0070.609 ± 0.005
Car-identity classification error (%)69.4 ± 2.944.3 ± 0.5

Entries are means ± sample SD across three runs. The same elevation step occupied less representation distance, and the probe recovered it more accurately with a larger directional gain. Under SimCLR-style augmentation, the bound was positive for 63% of elevation pairs, 7% of category pairs, and less than 1% of azimuth pairs.

In a separate vMF evaluation at matched κ=1000\kappa=1000, the larger MLP's recovery bound was positive for 99.6% of different-class pairs and only 2.8% of same-class pairs. For most same-class pairs the summed endpoint errors exceeded the separation. On those features, a logistic probe had 4.69% classification error, mean ℓ2\ell_2 error of 0.340 against one-hot targets, and median local probability Jacobian norm of 3.72 in RMS-calibrated units.

2. How close is Cars3D to exact alignment? At the strongest augmentation, PGD increased median absolute cosine by 0.053 for category–elevation, 0.086 for category–azimuth, and 0.060 for elevation–azimuth (Fig. 6B). For category–elevation in a separate sample, the median relative orthogonal residual was 0.990 clean and 0.979 under PGD. Exact alignment requires zero.

Revised Corollary 2 (Section 5, proof in Appendix C.2) gives a sufficient condition for an affine probe. Write an elevation displacement as u=tv+eu=tv+e, where vv is a category displacement and e⊥ve\perp v. The category scores must change if ∣t∣∥Av∥>∥Ae∥|t|\|Av\|>\|Ae\|. In one selected clean example the terms were 0.111 and 0.091. Changing elevation moved the score vector by 0.123 and preserved the predicted identity.

Fig. 8A–B shows that pose errors can grow as alignment moves in either direction. Under PGD, car 154’s azimuth prediction went from the correct 120° to 0° as alignment fell (0.855 to 0.311). Car 119’s elevation prediction went from the correct 33% to 100% as alignment rose (0.319 to 0.958).

3. Does vMF reconstruction deteriorate in absolute units? For the linear readout, mean squared Euclidean error rose from 0.096 ± 0.074 to 0.631 ± 0.056 between κ=5\kappa=5 and 10001000 (6.6-fold, ten runs). Within-class variance fell from 0.578 to 0.00399, so normalized error rose from 0.166 to 157.987 (Table 1, Fig. 9A, Appendix A.3). For the larger MLP, absolute error was 0.0256 and 0.0288 at the same endpoints, while normalized error rose from 0.044 to 7.204 (appendix table on nonlinear reconstruction). Thus high global R2R^2 (0.971 at κ=1000\kappa=1000) coexists with error exceeding the within-class variance. Classification in Table 1 uses the true latents. At κ=100\kappa=100 and 250250, the accuracies (95% and 99%) are close to the nearest-prototype reference (95.9% and 99.3%, Fig. 9B).

4. Can effective resolution be estimated independently? We rewrote Appendix L to distinguish the RMS angular spread (2/κ2/\sqrt{\kappa} for d=5d=5 at large κ\kappa) from the mean cosine gap (2/κ2/\kappa to leading order). At κ=50\kappa=50 and 100100, the mean gaps are 0.0396 and 0.0199. These values describe the latent distribution. We did not independently calibrate the trained system’s resolution as this is more of a problem in limited by the precision of the machine (see Colbrook et al. in the references, the paper on Smale's 18th problem). The text was adapted accordingly.

5. Reproducibility and scope. Appendix E.3 now specifies the PGD objective, preprocessing, random start, projection, 40 steps of 2/2552/255, and budgets of 4/2554/255 for dSprites/SmallNORB and 8/2558/255 for Shapes3D/Cars3D. PGD maximizes cross-entropy for the true category index on the frozen DIET output, with no target class or fitted probe in the loss. We will provide code and configurations in the anonymous supplement and release them publicly at publication. The image results cover four datasets, one instance-discrimination framework, and three split seeds. Generalization across objectives remains open.

Reviewer_6T9wOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the limitations of the identifiability theory of contrastive learning for downstream tasks, specifically classification. It derives pairwise lower bounds that jointly relate latent-recovery error, downstream prediction error, representation distances, and probe Lipschitz constants. Such bounds reveal a so-called recovery-prediction-stability trilemma. Namely, it claims when a learned representation compresses latent directions that a downstream probe must preserve, accuracy can be maintained only at the cost of increased sensitivity.

Critical strengths

The problem studied in the paper is definitely very interesting and valuable for the field. It puts quite an effort in deriving the low bounds of pair-wise distances in representation space in connection to their latent space. The motivation is pretty clear as illustrated in Figure 2. Experiments are well-structured and illustrative.

Critical weaknesses

The main critics from the reviewer is its mathematical presentation of the major body of the paper, i.e., Section 3, 4, and 5. There are so many undefined symbols and abuse of symbols, which make these parts almost unreadable. And the writing quality also doesn't meet a typical academic standard, e.g., repeated identical references. Without proper professional writing, the paper in its present form can't be accepted at the ICLR.

Questions for authors

A general comment here: Although the problem studied in the paper is valid and interesting, the loss used for identifiability analysis of contrast learning was or has always been used solely for disentanglement. The error bound results presented in this paper seem to make sense, but without constructive insight into the issue. Can you imagine combining the CL loss and downstream loss, such as classification loss, and deriving the bounds under the conditions of reaching some global minima?

2 comments

Thank you for suggesting joint instance-discrimination and classification training. We derived a bound for this objective and tested its downstream effects. The link to classification is explicit in Reizinger et al. (ICLR 2025), who show that supervised cross-entropy can recover latent factors up to a linear transformation under their generative assumptions. We ask what such recovery implies for prediction and stability.

Global minima. Let JJ index instances, C=c(J)C=c(J) denote class, and LJ,LCL_J,L_C be the population cross-entropies of two heads on a shared encoder. We optimize

Fλ(θ)=LC(θ)+λLJ(θ).F_\lambda(\theta)=L_C(\theta)+\lambda L_J(\theta).

If the encoder and heads can realize both true posteriors, every global minimum at λ>0\lambda > 0 attains LC=H(C∣X)L_C=H(C\mid X) and LJ=H(J∣X)L_J=H(J\mid X), where XX is the input. Each excess loss is a nonnegative expected KL divergence, and joint realizability makes their weighted sum zero. The optimal class probability therefore equals the sum of the optimal instance probabilities within that class.

A bound without joint realizability. Fix the model family and distribution. For 0≤a<b0 \le a < b, let θa,θb\theta_a,\theta_b be global minimizers at the two weights, with finite losses. Define the reduction in instance loss and increase in class loss by

G=LJ(θa)−LJ(θb),D=LC(θb)−LC(θa).G=L_J(\theta_a)-L_J(\theta_b),\qquad D=L_C(\theta_b)-L_C(\theta_a).

Comparing the two solutions at weight aa gives the lower bound and comparing them at bb gives the upper bound.

aG≤D≤bG.aG \le D \le bG.

Hence G,D≥0G,D \ge 0. When G>0G > 0, D/GD/G is the class cross-entropy paid per unit of instance cross-entropy gained. For objective gaps at most δa,δb\delta_a,\delta_b, the same argument gives aG−δa≤D≤bG+δbaG-\delta_a \le D \le bG+\delta_b.

Connection to Theorem 1. Two same-class instances have identical class targets but distinct instance targets. If a probe with Lipschitz constant LL reconstructs both latents to error at most ε\varepsilon, Theorem 1 gives

L,∥hi−hj∥≥[∥zi−zj∥−2ε]+.L\\,\lVert h_i-h_j\rVert \ge [\lVert z_i-z_j\rVert-2\varepsilon]_+.

The classification lower bound for this pair is zero. Thus Bayes-optimal classification can coexist with a high sensitivity requirement for recovery when well-separated latents have nearby representations.

Experiment. We fine-tuned the same DIET-pretrained checkpoints at five weights and fitted fresh downstream probes. Each mean weights the 21 pretraining–downstream concentration combinations equally, with ten encoder runs per setting.

Instance weightClean accuracyLinear recovery R2R^2PGD accuracy (ϵ=0.01\epsilon=0.01)
0 (classification only)78.56%0.63011.68%
0.0178.38%0.62911.63%
0.177.75%0.62311.74%
176.91%0.61312.08%
1076.25%0.60712.31%

Increasing the weight from 0 to 10 cost 2.32 percentage points of clean accuracy and 0.023 in recovery R2R^2, with a 0.63-point gain under PGD. These are downstream effects. Testing the bound directly requires both training-head losses on a common distribution and control of the optimization gaps. We will release the code.

Presentation. Section 3 defines the generator and class map, Section 4 defines the probes, and Theorem 1 states its pairwise error assumptions. We reviewed the citations for redundancy and deliberately retained a citation where a later argument uses the source, so readers returning to that section can trace the claim locally. Could you identify one or two symbols you found undefined or inconsistently used, together with the affected equation or proof step? This would let us address the precise concern.

Thanks for the efforts from the authors to address my concerns. An analysis of the combination of the CL loss and downstream loss could be further investigated to enhance the quality of the work. So I keep my score.

Reviewer_1TPAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper looks at the self-supervised representation learning methods with known identifiability guarantees and studies the interplay between three elements: the ability of an encoder to recover the true latents, to perform a downstream task of interest (predicting the class for example) and the stability of the downstream prediction and latent recovery process to noise. Authors formalize how the latent recovery implied by the identifiability results does not imply the representation can be used to decode target labels with a linear probe. They also formalize how a good predictability or latent recovery does not imply a better robustness to noise. These contributions are Theoretical first but authors provide some numerical and synthetic experiment to show/confirm this interplay in practice.

Critical strengths
  • Form: the presentation meets the requirements to be published at ICLR, substantial efforts was put into presenting and explaining the contributions in a clear way, writing is smooth, reading this paper is an enjoyable experience. Empirical setup meets the standard in the identifiability field and I don't have concerns when it comes to soundness. The DGP used by the authors is taken from existing published work.

  • Topic: the aim of this paper is to link identifiability results and practical use of SSL representations which is useful and interesting to the community. Authors help formalize the weaknesses of using identifiability results on their own to predict the quality of SSL representations. While the findings might seem relatively expected given prior identifiability research, I think formalizing the pitfalls is something the community will be able to build on top of and refer to.

Critical weaknesses

Few actionable takeaways: my takeaway from this paper is that 1) the data generating process (DGP) impacts significantly how well one can predict the class from representations inverting the DGP 2) class predictability from the learned representation does not imply robustness to noise for linear probes. I think formalizing these observations is valuable but the paper does not explore ways of mitigating these effects beyond "ablate across the augmentation strength to find the setup which maximizes downstream prediction". It formalizes weaknesses of the identifiability framework but does not explore ways to improve SSL in practice. This means the impact of the paper remains limited from a practical standpoint. Studying how the complexity of the probe can influence the robustness to noise/help improve stability could be an interesting addition to the paper (see question below).

Questions for authors
  • Can you clarify the reason why in figure 3, the accuracy is higher for low concentration values? and can you provide the results for the recovery of all generative factors?
  • How is the sensibility to noise mitigated by more complex probes? Did you study this relaxation and can it help improve stability of the downstream prediction?
1 comment

Thank you for suggesting that we test whether a more expressive probe can improve stability. We added nonlinear probes to the vMF and image experiments. We also withdraw the dSprites comparison in our earlier reply because its quoted values came from an illustrative table. Below we report a measured result from three SmallNORB splits.

Why is clean accuracy higher at low concentration in Figure 3? Each top panel fixes the downstream concentration and varies the pretraining concentration. At κdown=50\kappa_{\mathrm{down}}=50, for example, clean linear-probe accuracy is 79.8% for κpre=10\kappa_{\mathrm{pre}}=10 and 42.2% for κpre=1000\kappa_{\mathrm{pre}}=1000 (Appendix E.2, Table 7). Table 1 suggests one explanation. At low pretraining concentration, a linear readout recovers the latents more accurately. In our equal-prior vMF model, the optimal classifier selects the prototype with the largest dot product with the latent. Composing a linear reconstruction with that dot product gives a linear class score on the frozen representation. Better reconstruction could therefore explain the higher clean accuracy. Increasing downstream concentration has a separate effect: it improves class separation when classification uses the true latents (Figure 9B).

How well is each generative factor recovered? Appendix F.1, Table 10 reports linear recovery R2R^2 for each of the five vMF coordinates at every concentration. All five show lower recovery at κ=1000\kappa=1000 than at κ=5\kappa=5.

κ\kappaz1z_1z2z_2z3z_3z4z_4z5z_5
50.9270.9200.8970.8790.882
10000.5340.2830.3480.3340.325

Entries are means over ten independent encoder runs. For Shapes3D, SmallNORB, and Cars3D, Appendix F, Table 9 gives factor-wise results for linear and MLP probes. It reports every factor on clean images and category recovery under uniform noise and PGD.

Does a more complex probe improve stability? At κpre=κdown=1000\kappa_{\mathrm{pre}}=\kappa_{\mathrm{down}}=1000, mean coordinate-wise recovery R2R^2 rose from 0.385 with an affine readout to 0.641 with a one-hidden-layer MLP and 0.975 with a three-hidden-layer MLP (Figure 4, Table 3). For classification, we trained separate probes on the same frozen encoders. The larger MLP exceeded logistic regression by 1.13 percentage points on clean inputs and scored 3.41 points lower after adding 0.01 to every input coordinate. Both MLPs had lower mean classification accuracy than logistic regression under this shift at all seven matched concentrations.

SmallNORB illustrates why we distinguish the perturbed score from the drop relative to clean performance. At augmentation setting η=4\eta=4, the 256-unit MLP achieved higher category correlation under uniform noise than the linear probe (0.714 versus 0.660, averaged over three splits). Its clean correlation was also higher (0.959 versus 0.888), so its clean-to-noisy drop was slightly larger (0.245 versus 0.228).

We added two practical recommendations to the Discussion. First, compare probe capacities on the same frozen encoder and report both the perturbed score and its change from clean performance. Second, evaluate related tasks on the same perturbed examples. In Cars3D, category-conditioned perturbations also changed elevation and azimuth predictions (Figure 6).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of overall rebuttal and revised version

We thank Reviewers 1TPA, 6T9w, and tumk for their careful reading and for suggesting experiments that helped us connect the theoretical results to the behavior of trained encoders. Most changes are marked in blue in the revised manuscript.

Probe capacity and practical evaluation (Reviewer 1TPA). We now compare linear and nonlinear probes on the same frozen encoders in the vMF experiments and across all four disLib datasets, with results for individual factors. At κ=1000\kappa=1000, replacing the affine probe with the larger MLP raises mean latent-recovery R2R^2 from 0.382 to 0.971 (Table 8). For classification, the MLP’s clean accuracy gains did not consistently persist under perturbation (Fig. 4). The new Discussion (Sec. 8) recommends comparing probe capacities and evaluating related tasks on the same perturbed examples. In Cars3D, for instance, perturbations directed at car identity also changed elevation and azimuth estimates.

Joint training (Reviewer 6T9w). Appendix A.4 now analyzes a shared encoder trained with both class and instance cross-entropy. When the model can represent both true posteriors and both losses have positive weight, every population minimum recovers both posteriors. For a fixed model family, we also bound how much class loss can increase per unit decrease in instance loss when comparing global minima at different weights. In a separate experiment, we fine-tuned the same checkpoints at five weights. Raising the instance weight from 0 to 10 lowered mean clean accuracy by 2.32 percentage points and raised accuracy under PGD by 0.63 points (Table 6).

Testing the geometric claims (Reviewer tumk). We now measure the quantities in Theorem 1 on trained Cars3D representations. For views of the same car at adjacent elevations, the 95th-percentile probe gain required by the bound rose from 0.60 without augmentation to 1.88 with SimCLR-style augmentation. The observed gain at the same percentile rose from 1.13 to 2.94 (Table 5). We also generalized Corollary 2 from exact alignment to partial directional overlap. In one clean example satisfying the new condition, changing elevation moved the car-identity scores while preserving the predicted identity (Sec. 5, Appendix A.3).

For vMF, we now report reconstruction error both in absolute units and relative to within-class variance. As κ\kappa increases from 5 to 1000, the linear probe’s mean squared reconstruction error rises from 0.096 to 0.631 while within-class variance shrinks (Fig. 9). Appendix L clarifies that the trained system’s effective resolution remains uncalibrated. Appendix E.3 gives the full perturbation protocol, and we will release the code and configurations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.