- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines whether latent-factor identifiability results for contrastive learning can explain downstream classification performance and stability. It distinguishes latent recoverability, task sufficiency, and robustness. In a class-conditional latent model, it analyzes how class concentration can reduce ambiguity while shrinking within-class variation. It then derives pairwise bounds relating recovery and prediction errors to representation distances and probe Lipschitz constants, and gives a conditional result for interference between latent factors that act along a shared representation direction. Simulations and experiments on four disentanglement benchmarks compare latent recovery, category performance, and behavior under random or adversarial perturbations.
Critical strengths
The paper makes a useful conceptual distinction: recovering the complete latent state does not determine whether class labels are discernible from that state, and clean prediction does not establish stability. The class-conditional discussion helps specify the extra assumptions needed when identifiability results are used to explain downstream performance.
The authors test the proposed tensions in a controlled setting with known latent factors. Reporting category and other-latent recovery across dSprites, shapes3D, smallNORB, and cars3D, including perturbed observations, provides more insight than clean classification accuracy alone. The paper also acknowledges that its theoretical conditions do not imply every contrastive learner exhibits the same trade-off.
Critical weaknesses
The central “trilemma” does not yet establish a mechanism specific to contrastive learning. Theorem 1 follows from pairwise triangle inequalities and applies to arbitrary representations and Lipschitz probes; its consequence requires the learned representation to compress relevant pairs. Corollary 2 further assumes exact alignment between a label-preserving change and a class-changing representation direction. The benchmark experiments show associated changes in recovery and prediction, but do not measure the representation separations, local probe sensitivities, or directional alignment required to connect those observations to the bounds.
Some empirical evidence is difficult to interpret as loss of latent recoverability. In the vMF experiment, Table 1 compares classification using the true latent state with reconstruction (R^2) from a learned representation as concentration changes. Because within-class variance also shrinks, a decline in (R^2) alone does not show that absolute reconstruction error increases. The proposed effective-resolution threshold is not measured for the trained system. The image experiments rely mainly on one instance-discrimination framework and three split seeds; the appendix gives limited details for reproducing the category-targeted PGD attacks. These choices limit how broadly the observed patterns support the paper’s contrastive-learning claim.
Questions for authors
Can the authors directly test Theorem 1 on the trained representations by reporting latent-to-representation distance ratios, recovery and classification errors, and estimates of local probe sensitivity? For the cars3D cross-factor result, how close are the relevant representation changes to the shared-direction assumption in Corollary 2?
In the vMF experiment, how do absolute latent reconstruction error and within-class-normalized error vary with (\kappa)? Can the effective resolution (\epsilon_{\mathrm{eff}}) be measured or estimated independently, rather than chosen to explain the observed concentration range?
Thank you for asking when the bounds are informative. Theorem 1 applies to any encoder. DIET assigns augmented views the same instance target. We tested whether its trained encoders compress pose differences needed downstream.
1. When is Theorem 1 informative? We compared clean Cars3D views at adjacent elevations with car identity and azimuth fixed. We fitted affine probes on separate reference views, standardized elevation, and RMS-normalized the centered features. Appendix A.2 reports all five settings. For each pair, B=[∣Δz∣−ei−ej]+/∥Δh∥ is the required gain and G=∣Δz^∣/∥Δh∥ the observed gain, with absolute endpoint errors ei,ej.
Entries are means ± sample SD across three runs. The same elevation step occupied less representation distance, and the probe recovered it more accurately with a larger directional gain. Under SimCLR-style augmentation, the bound was positive for 63% of elevation pairs, 7% of category pairs, and less than 1% of azimuth pairs.
In a separate vMF evaluation at matched κ=1000, the larger MLP's recovery bound was positive for 99.6% of different-class pairs and only 2.8% of same-class pairs. For most same-class pairs the summed endpoint errors exceeded the separation. On those features, a logistic probe had 4.69% classification error, mean ℓ2 error of 0.340 against one-hot targets, and median local probability Jacobian norm of 3.72 in RMS-calibrated units.
2. How close is Cars3D to exact alignment? At the strongest augmentation, PGD increased median absolute cosine by 0.053 for category–elevation, 0.086 for category–azimuth, and 0.060 for elevation–azimuth (Fig. 6B). For category–elevation in a separate sample, the median relative orthogonal residual was 0.990 clean and 0.979 under PGD. Exact alignment requires zero.
Revised Corollary 2 (Section 5, proof in Appendix C.2) gives a sufficient condition for an affine probe. Write an elevation displacement as u=tv+e, where v is a category displacement and e⊥v. The category scores must change if ∣t∣∥Av∥>∥Ae∥. In one selected clean example the terms were 0.111 and 0.091. Changing elevation moved the score vector by 0.123 and preserved the predicted identity.
Fig. 8A–B shows that pose errors can grow as alignment moves in either direction. Under PGD, car 154’s azimuth prediction went from the correct 120° to 0° as alignment fell (0.855 to 0.311). Car 119’s elevation prediction went from the correct 33% to 100% as alignment rose (0.319 to 0.958).
3. Does vMF reconstruction deteriorate in absolute units? For the linear readout, mean squared Euclidean error rose from 0.096 ± 0.074 to 0.631 ± 0.056 between κ=5 and 1000 (6.6-fold, ten runs). Within-class variance fell from 0.578 to 0.00399, so normalized error rose from 0.166 to 157.987 (Table 1, Fig. 9A, Appendix A.3). For the larger MLP, absolute error was 0.0256 and 0.0288 at the same endpoints, while normalized error rose from 0.044 to 7.204 (appendix table on nonlinear reconstruction). Thus high global R2 (0.971 at κ=1000) coexists with error exceeding the within-class variance. Classification in Table 1 uses the true latents. At κ=100 and 250, the accuracies (95% and 99%) are close to the nearest-prototype reference (95.9% and 99.3%, Fig. 9B).
4. Can effective resolution be estimated independently? We rewrote Appendix L to distinguish the RMS angular spread (2/κ for d=5 at large κ) from the mean cosine gap (2/κ to leading order). At κ=50 and 100, the mean gaps are 0.0396 and 0.0199. These values describe the latent distribution. We did not independently calibrate the trained system’s resolution as this is more of a problem in limited by the precision of the machine (see Colbrook et al. in the references, the paper on Smale's 18th problem). The text was adapted accordingly.
5. Reproducibility and scope. Appendix E.3 now specifies the PGD objective, preprocessing, random start, projection, 40 steps of 2/255, and budgets of 4/255 for dSprites/SmallNORB and 8/255 for Shapes3D/Cars3D. PGD maximizes cross-entropy for the true category index on the frozen DIET output, with no target class or fitted probe in the loss. We will provide code and configurations in the anonymous supplement and release them publicly at publication. The image results cover four datasets, one instance-discrimination framework, and three split seeds. Generalization across objectives remains open.