- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that representation collapse is not inherently helpful or harmful; its effect depends on whether contraction is aligned with semantic or nuisance directions. It formalizes this using semantic Fisher information, proposes a counterfactual secant estimator, and separately studies fold-type information loss.
Critical strengths
- This paper shows clear conceptual distinction between collapse magnitude, collapse orientation, overlap degeneracy, and folding.
- The idea that conventional metrics (e.g. effective rank, participation ratio, RankMe, or the singular values of the full Jacobian) measuring the amount of contraction but not whether the contracted directions are semantic or nuisance directions, is very interesting.
- Useful controls separate representation failure from classifier-head failure.
Critical weaknesses
- The theory assumes local linearization, continuous semantic coordinates, Gaussian nuisance variation, and known semantic/nuisance directions. Two main experiments instead use categorical semantics and a secant approximation.
- In my understanding, the proposed estimator requires known factors and single-factor counterfactuals. Given this information, simpler contraction measures and LiDAR perform similarly, while labelled counterfactual augmentation performs much better.
- Evidence is limited to small networks and mostly synthetic (32\times32) datasets. The motivating setting of large-scale self-supervised representation learning is not tested.
Questions for authors
- How robust are conclusions to the selected nuisance factors, counterfactual step sizes, and representation reparameterizations?
- Can the method be evaluated on modern pretrained self-supervised encoders without access to a ground-truth generative-factor oracle?
2 comments
Thank you for your response and answering my questions. I understand this is a theory-oriented paper but standard dataset (ImageNet or its subset) is highly appreciated which could better examine the robustness of the theory in a more realistic setup. Nonetheless, I think my concerns about on the theoretical part have been partially addressed. I therefore maintain my positive scores.
What the estimator needs, a new robustness test over 36 choices of declared nuisances, step size and floor, the exact reparameterisation class, and a new self-supervised experiment
Thank you for the support and the precise summary.
Assumptions vs experiments. Correct. The theorem assumes continuous coordinates and Gaussian nuisance. For categorical factors the secant estimator is a counterfactual Mahalanobis discriminability motivated by Eq. (1): it predicts the sign of effects, not a Cramer-Rao bound (§3.1). §4.1 is the one place where part (ii)'s hypotheses can be met exactly (a fixed operator on a frozen encoder, two declared nuisances), and there the predicted identity holds to within 4⋅10−10; parts (i) and (iii) are not met there. Table 4 lists each formal result with its assumptions and test; the new Table 1 gives each claim's status.
Known factors; LiDAR and C^S similar; counterfactual augmentation better. We agree with all three, and report each as a finding.
Q1. Robustness.
Q2. Pretrained SSL without an oracle. Not without some declared interventions, and we do not claim otherwise. One partial answer: on Colored-CIFAR the semantic pairs are natural images of different classes, though the nuisance pairs still use the planted border colour (App. C). Separately, a small VICReg experiment (§4.3, fixed in advance, 3 seeds) carries the orientation question over to SSL. Its views come from the generator's factors, so it does not remove the oracle:
The spectral statistics rate the best encoder as the most collapsed. The deficit, and equally C^S, order all four arms (with object hue) correctly, though only "shape is worst" was fixed in advance. LiDAR on the same pool rates the shape-invariant encoder highest (1.93 vs 1.84 for the best one; post hoc). Frozen large pretrained encoders remain untested.
Scale. We agree: every experiment uses small networks on 32x32 images (§5).
Also new: the narrowed contrast (semantic arm worse than a neutral-nuisance arm under contraction, better under protection) held prospectively on five unused seeds (+30.5 and -30.3 points). In pairs matched on three spectral statistics, the semantic arm is on the predicted side in all 9 pairs within 5% and in 32 of 33 within 10%.