acceptodds
Desk rejected at ICLR 2027

Orientation Can Reverse the Harm of Representation Collapse

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation collapse is blamed for brittle features, yet neural collapse accompanies good generalisation, and no label-free statistic says which of the two a model is in. We show that collapse magnitude alone does not decide it: at a matched contraction, the orientation of that contraction relative to semantic and nuisance directions changes how much task-relevant information survives, in a construction where every spectral statistic of the Jacobian is identical. We formalise this with a representation's linear semantic Fisher information and test the formal claim where its hypotheses hold exactly, on a frozen encoder: the predicted change in semantic information is reproduced to better than , while the companion statement about nuisance contraction has no exact realisation on these encoders, which is consistent with contracting a nuisance response costing information rather than saving it. Holding one regulariser fixed and rotating only the factor it acts on, the representation-level out-of-distribution error is for a semantic contraction against for a neutral nuisance, and, at effective ranks within of each other, for a semantic protection against for a neutral one; six of our eight pre-registered predictions nonetheless failed and one of two kill criteria fired. Over unregularised networks swept through two training regimes our index keeps one sign in all thirty strata where each of eight collapse statistics reverses, but an information-matched control shows most of that advantage is the counterfactual pool rather than our summary of it. The one failure the index provably cannot see is a fold, which we exhibit and measure.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation collapse, neural collapse, dimensional collapse, representation quality metrics, effective rank, RankMe, Fisher information, nuisance invariance, out-of-distribution generalization, diagnostics for learned representations

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GCsqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that representation collapse is not inherently helpful or harmful; its effect depends on whether contraction is aligned with semantic or nuisance directions. It formalizes this using semantic Fisher information, proposes a counterfactual secant estimator, and separately studies fold-type information loss.

Critical strengths
  1. This paper shows clear conceptual distinction between collapse magnitude, collapse orientation, overlap degeneracy, and folding.
  2. The idea that conventional metrics (e.g. effective rank, participation ratio, RankMe, or the singular values of the full Jacobian) measuring the amount of contraction but not whether the contracted directions are semantic or nuisance directions, is very interesting.
  3. Useful controls separate representation failure from classifier-head failure.
Critical weaknesses
  1. The theory assumes local linearization, continuous semantic coordinates, Gaussian nuisance variation, and known semantic/nuisance directions. Two main experiments instead use categorical semantics and a secant approximation.
  2. In my understanding, the proposed estimator requires known factors and single-factor counterfactuals. Given this information, simpler contraction measures and LiDAR perform similarly, while labelled counterfactual augmentation performs much better.
  3. Evidence is limited to small networks and mostly synthetic (32\times32) datasets. The motivating setting of large-scale self-supervised representation learning is not tested.
Questions for authors
  1. How robust are conclusions to the selected nuisance factors, counterfactual step sizes, and representation reparameterizations?
  2. Can the method be evaluated on modern pretrained self-supervised encoders without access to a ground-truth generative-factor oracle?
2 comments
AuthorsOpenReview ↗

What the estimator needs, a new robustness test over 36 choices of declared nuisances, step size and floor, the exact reparameterisation class, and a new self-supervised experiment

Thank you for the support and the precise summary.

Assumptions vs experiments. Correct. The theorem assumes continuous coordinates and Gaussian nuisance. For categorical factors the secant estimator is a counterfactual Mahalanobis discriminability motivated by Eq. (1): it predicts the sign of effects, not a Cramer-Rao bound (§3.1). §4.1 is the one place where part (ii)'s hypotheses can be met exactly (a fixed operator on a frozen encoder, two declared nuisances), and there the predicted identity holds to within 4⋅10−104\cdot 10^{-10}; parts (i) and (iii) are not met there. Table 4 lists each formal result with its assumptions and test; the new Table 1 gives each claim's status.

Known factors; LiDAR and C^S\hat C_S similar; counterfactual augmentation better. We agree with all three, and report each as a finding.

  • The index needs a declared set of single-factor interventions, and its semantic ones change the class (on Colored-CIFAR they are picked by class label), so it is supervised; the abstract and §5 state the requirement.
  • Given our counterfactuals, LiDAR keeps its sign in 13 of 14 strata (groups of networks that share the value of one training knob), ours in 14. A new post hoc analysis asks where C^F\hat C_F and C^S\hat C_S disagree at all: rarely (Spearman 0.90 to 0.99 on three sweeps, 0.71 on the fourth). Among pairs of networks they order oppositely, failure follows C^F\hat C_F in 81 of 98 on the 3D Shapes spurious sweep (83%, 95% interval [64%, 94%]), 45 of 60 under label noise (interval includes chance) and 7 of 15 on the pre-registered factorial (App. E.4).
  • Counterfactual augmentation uses the same pairs with labels during training and reaches 0.1% error (Table 12, post hoc warm-up): where labelled pairs exist they are the better remedy, as App. D.7 says.

Q1. Robustness.

  • New (fixed in advance). On 15 networks that carry the narrowed claim (ERM; contract or protect, semantic or neutral) we recomputed the index under every combination of nuisance step size (1, 2 or 3 values), declared nuisance set (all five, object hue only, floor hue only, all but object hue) and floor (σ2=10−3,10−2,10−1\sigma^2 = 10^{-3}, 10^{-2}, 10^{-1}). All 36 variants keep a positive rank correlation with the probe OOD error, between +0.76 and +0.91 (default +0.87; C^S\hat C_S gives +0.91), and every variant agrees with the default ranking at Spearman 0.91 or higher (Table 24, App. E.5). Dropping object hue, whose pixel change overlaps the shape change most, does not weaken it, although we expected it to; a smaller floor and larger steps weaken it slightly. With 15 networks in five arm-clusters this shows a stable ordering of these arms, not calibration.
  • Floor across the four sweeps: σ2\sigma^2 from 0.01 to 0.1 keeps rankings at Spearman 0.91 to 0.99 (App. D.6).
  • Reparameterisation: invariant, to machine precision, to rotations of zz and to a joint rescaling of zz and σ\sigma. It is deliberately not invariant to an anisotropic rescaling at a fixed floor (0.43 nats for a moderate diagonal rescaling, unbounded in general), because it measures information at a stated readout resolution; so we compare it only across networks of one architecture and output scale (Prop. 3, App. D.16).
  • Factor choice: factors are not orthogonal pixel directions (object hue overlaps the shape change 0.49 vs 0.11 for floor hue; App. D.15).

Q2. Pretrained SSL without an oracle. Not without some declared interventions, and we do not claim otherwise. One partial answer: on Colored-CIFAR the semantic pairs are natural images of different classes, though the nuisance pairs still use the planted border colour (App. C). Separately, a small VICReg experiment (§4.3, fixed in advance, 3 seeds) carries the orientation question over to SSL. Its views come from the generator's factors, so it does not remove the oracle:

Views differ inProbe OOD errorEffective rankRankMeDeficit
shape (semantic)73.6%9.8511.621.91
floor hue52.9%10.1012.00-0.49
any nuisance0.1%7.2310.00-1.66

The spectral statistics rate the best encoder as the most collapsed. The deficit, and equally C^S\hat C_S, order all four arms (with object hue) correctly, though only "shape is worst" was fixed in advance. LiDAR on the same pool rates the shape-invariant encoder highest (1.93 vs 1.84 for the best one; post hoc). Frozen large pretrained encoders remain untested.

Scale. We agree: every experiment uses small networks on 32x32 images (§5).

Also new: the narrowed contrast (semantic arm worse than a neutral-nuisance arm under contraction, better under protection) held prospectively on five unused seeds (+30.5 and -30.3 points). In pairs matched on three spectral statistics, the semantic arm is on the predicted side in all 9 pairs within 5% and in 32 of 33 within 10%.

Thank you for your response and answering my questions. I understand this is a theory-oriented paper but standard dataset (ImageNet or its subset) is highly appreciated which could better examine the robustness of the theory in a more realistic setup. Nonetheless, I think my concerns about on the theoretical part have been partially addressed. I therefore maintain my positive scores.

Reviewer_vGXhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to reconcile the observed downsides of representation collapse with the fact that neural collapse accompanies good generalization. They study how this difference in result depends on whether semantic or nuisance directions are being collapsed and formalize this using linear semantic Fisher information.

Critical strengths

The paper uses good scientific practices and openly discusses its limitations and negative results.

Critical weaknesses
  1. Pre-registered predictions are framed as an important part of the paper yet are only properly introduced in the appendix (Table 6) making mentions of ‘these predictions’ in the main text feel out of place. While I appreciate the authors’ honesty in acknowledging that 6 of their 8 predictions failed and one of their kill criteria was triggered, this puts the paper in a weird spot since it does not treat the hypothesis as falsified.
  2. The novelty of the claims that are not falsified (that semantic contraction hurts more than nuisance contraction) seems limited as it is close to results the authors already note exist in the literature ‘That an invariance is harmful exactly when it discards a task-relevant factor is the premise of this line, not our claim’ (lines 121-122).
  3. The relevance of the results is limited by the simplicity of the frozen encoder setting and the requirement of single factor counterfactuals.
  4. The writing makes the paper hard to read, long winded sentences and excessive caveats hurt readability. Key concepts like ‘folds’ , ‘caustics’, the difference between neural collapse and representation collapse, and how collapse is measured in this paper should be introduced more clearly.
Questions for authors

To improve my rating, I would need to see a substantial improvement in writing and clarity. The authors would need to convey more clearly why their results are relevant despite the failed predictions and how the remaining results contribute to the existing literature.

2 comments
AuthorsOpenReview ↗

The pre-registered predictions are now in the main text and treated as falsified, the narrower contrast that survived then held on five new seeds with the test fixed in advance, and the abstract and introduction are rewritten

Thank you for the specific guidance. Each point below names where the change is in the revised PDF.

1. The pre-registration is in the main text, and treated as falsified. Table 2 (§4.2) lists all eight predictions and both kill criteria (outcomes under which we had committed to abandon the claim) with their outcomes. The paper now says plainly that, as a set, the predictions are falsified (six of eight failed; kill criterion 1 fired), and it withdraws the pre-registered claim that contracting a spurious nuisance helps. The only remaining evidence for helpful collapse is a post hoc warm-up control, labelled as such in §4.2 and §5. What failed: penalties on factors the task does not use were predicted to leave the error unchanged and moved it by up to 14 points; and contracting the spurious nuisance lost to contracting the semantic factor on the model's OOD error, because four of its five networks never fitted their training data.

2. What survived, then tested in advance. What did hold on seeds 0 to 4 is a contrast the plan did not name: under one operator, the semantic arm is worse than a neutral-nuisance arm when the operator contracts and better when it protects (representation-level OOD error: a probe retrained on frozen features). Found after seeing those seeds, it was only exploratory. During the discussion we wrote it down before running anything, with its measure and a kill rule (withdraw the claim if either half fails), and ran the identical protocol on five seeds never used before:

Probe OOD error contrastSeeds 0-4 (exploratory)Seeds 5-9 (fixed in advance)
contract: semantic minus neutral+23.3 [+17.0, +28.4]+30.5 [+25.9, +35.2]
protect: semantic minus neutral-32.0 [-35.3, -29.4]-30.3 [-32.4, -28.2]

Both halves held in every new seed, and on the model's own OOD error too. The original eight predictions fail again in the same way (two of eight hold).

3. Why the remaining results matter, and what is new. We agree with the sentence you quote and keep it: that discarding a task-relevant factor hurts is not our claim. Our question is whether the statistics used to diagnose collapse can tell which factor was compressed. In trained encoders they are often informative (Table 1), but they can miss it, in three settings:

  • two networks of one operator agree on effective rank (3.73 vs 3.74), participation ratio, RankMe and NC1, yet reach 7.5% vs 39.3% probe OOD error (new matched comparison, rule fixed in advance: all 9 pairs matched within 5%, and 32 of 33 within 10%, go the predicted way);
  • in self-supervised learning (new, VICReg, fixed in advance), views differing in shape give 73.6% error and views differing in any nuisance 0.1%, and effective rank and RankMe rate the 0.1% encoder as the most collapsed (its views come from the generator's factors, so it shares the counterfactual requirement);
  • by construction, every spectral statistic can stay equal while the semantic information falls (Theorem 1(iii)) or the Bayes accuracy moves from chance to perfect (Proposition 2).

Relative to the diagnostics in use (effective rank, RankMe, NC1, CDNV, LiDAR), the paper adds two things (§2). First, a design that holds the regulariser and its weight fixed and changes only the factor it acts on, so that the amount of collapse is measured rather than varied. Second, a measurement of how much of a direction-aware diagnostic's power comes from its counterfactual pairs rather than its formula: given our counterfactuals, LiDAR keeps its sign in 13 of 14 strata (groups of networks that share the value of one training knob), against our 14. The practical lesson, as §5 says: report a collapse statistic together with the factors it refers to.

4. Frozen encoders and single-factor counterfactuals. The frozen-encoder operator test appears only in §4.1, where part (ii) of the theorem can be met exactly. Every other experiment, including the new ones, trains networks end to end and freezes them only to fit the probe that measures what the representation retains. The counterfactual requirement is a real limit, now stated in the abstract and §5.

5. Writing. The abstract and introduction are rewritten in shorter sentences (abstract: about 30 words per sentence, down from over 40), with three numbers in the abstract. The introduction defines representation, neural and dimensional collapse, how collapse is usually measured, semantic and nuisance factors, orientation, overlap and fold. The abstract glosses folds; Figure 1's caption glosses "caustic". A new Table 1 lists each main claim with its status and evidence, and folds moved from the appendix into the main text as a scope boundary (§4.5). §4 now leads its main Q1 results with a sentence in words and keeps only the numbers each claim needs, though it remains the densest section.

Reviewer_vGXhOpenReview ↗

Maintaining my score

Thank you for your response.

While the writing in the introduction is somewhat improved, my concerns about novelty and significance of the theory and the empirical results that are not falsified remain. The paper does a good job being honest with its own limitations but not in conveying why their results are meaningful despite these limitations. The new Table 1 is emblematic of a broader problem throughout the paper, as it tries to do too many things and is hard to read due to focusing too much on details of experiments that have not yet been introduced. This extends to the rest of the paper which prioritizes detailed numerical values and caveats over the flow of the paper and clearly conveying the main points. I am also still not convinced by how the predictions are introduced and how their falsification is handled.

I will maintain my score.

Reviewer_qDrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper investigates why representation collapse can be beneficial in some settings while harmful in others. The authors argue that the magnitude of collapse alone is insufficient to determine its effect on downstream performance, and that the orientation of collapse relative to task-relevant semantic and nuisance directions plays a crucial role. In particular, suppressing nuisance variation can preserve or improve the usefulness of a representation, whereas suppressing semantic variation can destroy information necessary for the task. The work aims to formalise this distinction and investigate whether conventional collapse measures adequately capture it. The authors introduce a semantic information deficit (SID) metric based on Fisher information and theoretically demonstrate that contractions with identical spectral properties can retain different amounts of semantic information and demonstrate that collapse orientation can substantially affect generalisation, even when conventional collapse measures are similar. They also show that their diagnostic provides more consistent indications of representation failure, although much of this advantage comes from access to factor-specific counterfactual information. Finally, the paper examines representation folding as a distinct form of information loss that the proposed metric cannot reliably detect.

Critical strengths

The paper presents an interesting perspective on representation collapse by highlighting the importance of the directions in which information is lost, rather than considering collapse magnitude alone. The theoretical formulation provides a way to distinguish between semantic and nuisance collapse

Critical weaknesses

The theoretical and empirical results are not fully connected. Theorem 1 establishes how semantic information changes under controlled contractions of semantic or nuisance Jacobians, but the regularisation interventions used in the trained networks do not generally satisfy these assumptions. While the experiments demonstrate substantial differences in generalisation across collapse orientations, the extent to which these differences are explained by the proposed theoretical formulation remains unclear. Furthermore, the principal matched-collapse comparison only approximately matches effective rank, limiting how conclusively the results isolate orientation from other changes in representation geometry.

Questions for authors

1)Given the similar performance of SID and simpler factor-specific contraction measures in several experiments, are there settings where accounting for the interaction between semantic and nuisance directions provides a clear empirical advantage? Could the authors demonstrate cases where these measures make different predictions about representation quality? 2)Can the authors provide additional comparisons where the collapse magnitude is more closely matched across different orientations, preferably under multiple spectral measures? Would the observed differences in downstream performance persist under such controls?

2 comments
AuthorsOpenReview ↗

New results for both questions: a comparison matched on three spectral statistics at once (Q2), the network pairs on which the deficit and the plain contraction index disagree (Q1), and which part of Theorem 1 trained networks can test

We thank the reviewer for two precise questions. Both now have new results in the revised PDF (Appendix E, summarised in §4.2 and §4.4).

Q2. Matching collapse on several statistics at once. We trained 48 new networks: contract and protect, each along the semantic factor (shape) and the neutral nuisance (floor hue), at λ∈{0.03,0.1,0.3,3}\lambda \in \{0.03, 0.1, 0.3, 3\} with 3 seeds, pooled with the submitted λ=1\lambda=1 runs. Before reading any error we fixed the rule: a semantic and a neutral network of the same operator are a matched pair when effective rank, participation ratio and RankMe all agree within 5%. The strength may differ within a pair.

OperatorTolerancePairs (networks, sem./neu.)Probe OOD error, semantic minus neutralOn predicted side
contract5%8 (5/4)+15.7 [+0.5, +31.7]8 of 8
contract10%27 (8/16)+14.0 [+7.6, +21.4]26 of 27
protect5%1 (1/1)-31.81 of 1
protect10%6 (3/5)-29.6 [-32.6, -25.6]6 of 6

Pairs share networks, so the intervals resample networks rather than pairs (Table 21).

The direction survives matching. On these pairs the spectral statistics are equal by construction, and a lower effective rank picks the worse network in 14 of 33 pairs (10% tolerance), while C^F\hat C_F picks it in 29 and C^S\hat C_S in 28 (post hoc). Two qualifications. For contraction the gap shrinks from the same-strength +23.3 to about +15, so part of the original contrast was extra collapse. And protection rarely matches on all three statistics: its 6 pairs use 3 semantic networks, all at λ=3\lambda=3. Its one pair within 5% agrees on effective rank (3.73 vs 3.74), participation ratio (2.74 vs 2.73), RankMe (9.17 vs 9.46) and even NC1 (0.058 vs 0.053), yet the probe errors are 7.5% vs 39.3%.

Q1. When does the interaction term matter? Rarely on our trained networks, and §4.4 now says so.

  • By construction (new, Appendix E.4): with JS=e1J_S=e_1, JN=(cos⁡θ,sin⁡θ)⊤J_N=(\cos\theta, \sin\theta)^\top and σ2=0.01\sigma^2=0.01, both contraction indices stay at CS=CN=0C_S=C_N=0, but IS=100 (1−cos⁡2θ/1.01)I_S = 100\,(1-\cos^2\theta/1.01), so the deficit rises by 12log⁡101=2.31\frac{1}{2}\log 101 = 2.31 nats as the semantic response rotates into the nuisance direction. (The full Jacobian spectrum does change here.)
  • On trained networks the two indices order networks almost identically (Spearman 0.90 to 0.99 on three sweeps, 0.71 on 3D Shapes label noise). For every pair of networks they rank in opposite orders, we asked which order the failure follows. It follows C^F\hat C_F in 81 of 98 pairs on the 3D Shapes spurious sweep (83%, 95% interval [64%, 94%], resampling whole configurations), 45 of 60 under label noise (interval includes chance) and 7 of 15 on the pre-registered factorial; dSprites has too few such pairs. This is post hoc (Table 23, App. E.4).
  • In the alignment control (Table 12; post hoc warm-up), holding the semantic response inside the nuisance span at the same Fisher-length floor raised C^F−C^S\hat C_F-\hat C_S, the part of the deficit that C^S\hat C_S lacks, by 0.29 to 0.54 nats, and OOD error by 9 to 16 points, in every seed (Appendix E.4).

So the interaction is detectable on one sweep and in one post hoc control, but small: most of what the index knows is in the counterfactual pool, and we claim the design and that measurement, not a better formula.

Theorem vs trained networks. Agreed: the regularised networks do not satisfy Theorem 1. The scope table (Table 4, App. A) maps each part, and Table 1 summarises it.

  • (ii) is met exactly by a fixed operator on a frozen trained encoder with two declared nuisances, and there holds to within 4⋅10−104\cdot 10^{-10} (§4.1).
  • (i) is not met by the projector contraction I−(1−β)PI-(1-\beta)P we apply in representation space (the induced map has ∥Q∥2\|Q\|_2 up to 174, not ≤1\le 1). This gives a post hoc account (App. D.2) of the sign, not the size, of the cost of contracting a nuisance.
  • (iii) needs a Jacobian with orthonormal columns, which trained encoders lack: there the deficit rises 0.82 vs 0.09 nats (predicted 1.535 vs 0), and effective rank also separates the two cases.

The factorials test a prediction the theorem licenses, not the theorem. One consequence of its form does show in trained networks: the deficit is a log-determinant, so it registers a uniform loss of semantic volume that a rank summary of the same matrix cannot. Between ERM and the semantic contraction (39 points of probe OOD error), LiDAR on our pool moves by less than 0.1 while the deficit moves by 2.66 nats (§4.4, App. D.12).

Also new: the narrowed contrast (semantic arm worse than the neutral-nuisance arm under contraction, better under protection), fixed in advance, held on five unused seeds: +30.5 [+25.9, +35.2] and -30.3 [-32.4, -28.2] points (Table 2).

Reviewer_qDrVOpenReview ↗

My concerns are partially adressed

Thank you for your response.

The new matched collapse comparisons in Appendix E.2 address part of my concern regarding the influence of collapse magnitude. However, I still find the empirical evidence somewhat limited. The regularisation strength differs between matched pairs, and the protection comparison has only one pair within the 5% tolerance. I do not think the results conclusively establish that orientation is responsible for the observed differences independently of other changes in representation geometry. Regarding my first question, the analysis in Appendix E.4 provides some evidence that SID can outperform the simpler contraction index when the two disagree. However, the indices agree on most comparisons, and the advantage is clearly supported in only one of the reported sweeps. About Theorem 1, while I appreciate this clarification, it reinforces my original concern that the theoretical and empirical contributions are not sufficiently connected.

The theoretical formulation presented by the authors does not directly explain the empirical results, and the proposed approach has not been sufficiently stress tested to establish its broader applicability. Therefore, I maintain my original score. Reading the paper again made me reconsider the presentation score as well

Reviewer_8gCcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to find when representation collapse is harmful to a model. According to it, the direction of the collapse matters more than the size. Collapsing directions that carry label information is harmful, while collapsing nuisance directions is not. The authors formalize this with semantic Fisher information and share results on 3D shapes, dSprites, and a colored version of CIFAR-10. Alongside, they propose a diagnostic that uses counterfactual image pairs to measure retention of label information by the representation.

No AI was used is this review.

Critical strengths

Proposition 2 is a good example of the paper's thesis. It says that folding a 2D Gaussian at an angle α leaves every spectral collapse statistic unchanged, while the Bayes error drops from 1/2 to 0 as α grows. The derivation seems to be correct. The paper is honest about its negative results, even though most of it is shared in Tables 2 and 6 in the supplementary. Using the same penalty at the same weight seems to be a sound choice to separate effects of orientation from the cost of adding a penalty.

Critical weaknesses

The paper is very hard to read. I could not figure out the main claims from the abstract and introduction. Specifically, the abstract has more than 10 quantitative figures in one paragraph. It uses terms that are not defined anywhere else, like "kill criteria," "strata," and "counterfactual pool." It says 6 out of 8 pre-registered predictions failed, with no context on where those predictions were. The main text also reads in a similar fashion. Results are listed as long strings of in-line numbers, with heavy references to the appendix sections, making the main paper not so self-sufficient. One of the three research questions, Q3, only appears in the appendix. The theory feels a bit too simplistic and doesn't really apply to trained models. Theorem 1 follows from simple matrix algebra, I think. Inverting a matrix flips the Loewner order, and that determinants are multiplicative. I_S is the standard Fisher information from population coding. Table 2 says that Theorem 1 cannot be realized on these networks. The experiments are too small to support the general claims in the paper. All of the networks are quite small in terms of the number of parameters and are trained only on 32-by-32 images. There are no self-supervised learning experiments, even though the authors say that SSL collapse is their main motivation.

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

A rewritten abstract and introduction with the flagged terms defined, what Theorem 1 is (and is not) for, and a new self-supervised experiment

Thank you for the careful reading, and for singling out Proposition 2 and the same-weight design.

Readability. You are right, and we rewrote what you name.

  • The abstract now gives three numerical results instead of more than ten. It no longer uses "kill criteria", "strata" or "counterfactual pool", and it glosses "folds".
  • The pre-registered predictions and kill criteria (outcomes under which we had committed to abandon the claim) now sit in a main-text table with each outcome (§4.2, Table 2). On five new seeds the original eight fail again (two of eight hold). The narrower contrast, written down in advance this time, held: the semantic arm is worse than the neutral-nuisance arm under contraction (+30.5 points of probe OOD error) and better under protection (-30.3), both intervals excluding zero.
  • "Strata" (networks that share one value of one training knob), "counterfactual pool" (pairs of inputs that differ in exactly one factor) and "kill criterion" are defined in the caption of Table 1, where they first appear, and again in §4.2 and §4.4.
  • A new Table 1 lists each main claim with its status (proved, measured, failed, or held on a test fixed in advance) and where its evidence is.
  • Q3 no longer exists as a question: folds are a main-text scope boundary (§4.5) with their key numbers.
  • §4 now leads its main Q1 results with a sentence in words, keeps only the numbers each claim needs (the full grids are in Tables 2, 3 and 21) and splits its longest sentences. It still points to the appendix for details that do not fit in 10 pages, and §4.4 remains its densest part.

Theorem 1. Agreed: the algebra is elementary (inversion reverses the Loewner order; determinants multiply) and ISI_S is the linear Fisher information of population coding, which we cite. §1 and §3 now say so. Its role is a matched counterexample: contracting a semantic and a nuisance response by the same factor can leave every spectral statistic of the Jacobian identical while only the first lowers the semantic information.

On "Table 2 says that Theorem 1 cannot be realized" (that scope table is now Table 4): you are right for parts (i) and (iii); only part (ii) is met exactly in our test.

  • Part (ii) is met exactly by a fixed operator on a frozen trained encoder once the declared nuisance set is reduced to two factors, and there the predicted change holds to within 4⋅10−104\cdot 10^{-10} at every base point (§4.1).
  • Part (i) is not met by the projector contraction I−(1−β)PI-(1-\beta)P we apply in representation space: the induced map has ∥Q∥2\|Q\|_2 up to 174, not ≤1\le 1. This gives a post hoc account (App. D.2) of the sign, not the size, of the cost of contracting a nuisance, which the factorials showed against our prediction.
  • Part (iii) needs a Jacobian with orthonormal columns, which trained encoders lack. There the deficit rises 0.82 vs 0.09 nats (predicted 1.535 vs 0), and effective rank also separates the two cases (1.79 vs 2.35), so the blindness belongs to the construction, not to trained encoders.

The theorem bounds no error; the trained networks test its prediction. The new matched comparison complements this: in all 9 pairs of networks matched within 5% on effective rank, participation ratio and RankMe, the semantic arm is on the predicted side. The one protection pair (one operator, different strengths) agrees on all three statistics and on NC1 (effective rank 3.73 vs 3.74), yet reaches 7.5% vs 39.3% probe OOD error.

Self-supervised learning. We added a small self-supervised experiment (§4.3; predictions fixed in advance, 3 seeds). In SSL the augmentation decides which factor the objective contracts. We trained VICReg on 3D Shapes without labels, with arms differing only in the factor that separates the two views; the views are built from the generator's factors (§5).

Views differ inProbe OOD errorEffective rankRankMeDeficit
shape (semantic)73.6%9.8511.621.91
floor hue52.9%10.1012.00-0.49
object hue61.3%9.1410.541.58
any nuisance0.1%7.2310.00-1.66

The shape-invariant encoder is at chance, and the spectral statistics rate the best encoder as the most collapsed. The deficit, like the plain contraction index, orders all four arms correctly; only "shape is worst" was fixed in advance.

Scale. We agree: every experiment uses small networks on 32x32 images, and large pretrained SSL models remain untested. §5 states this limit and says we test the SSL reading only in a small setting, and the abstract names the data set behind each number it gives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.