- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how the relative scale of a linear prediction head's initial output and its regression targets affects representation collapse in self-supervised learning. The authors introduce a readout-to-target norm ratio and show that, under standard layer normalization and readout initialization, this ratio is approximately , independent of model width.
Using MelHuBERT on LibriSpeech, the authors find that representations undergo rapid rank collapse early in training, with larger values of (r) consistently associated with longer recovery times. To explain the observations, the authors analyze the initial encoder gradients and show that, at large (r), a target-independent gradient term dominates. While this provides a plausible explanation for the onset of collapse, the analysis does not establish the full mechanism governing collapse or recovery. The empirical validation is restricted to one model family and dataset, and the generality and downstream significance of the findings remain uncertain.
Critical strengths
S1. The paper identifies a simple initialization-dependent quantity, the readout-to-target ratio, that strongly predicts the duration of early representation collapse in the studied setting. The authors support this finding through controlled interventions on target scale, target spectral shape, and readout initialization. The consistent ordering of recovery times across configurations is compelling and suggests a potentially useful diagnostic for regression-based ssl.
S2. The paper shows that much of the observed rank reduction arises from a dominant shared mean vector, while a smaller but meaningful reduction remains after centering. This distinction is important because it demonstrates how uncentered spectral measurements can exaggerate the apparent loss of representation dimensionality. The accompanying analysis of CKA's insensitivity to the shared offset provides a useful methodological insight for studying representation collapse.
Critical weaknesses
W1. The mechanistic explanation does not establish why representations collapse or why recovery time depends on (r). Section 4.4 shows that, at large (r), the initial encoder-output gradient is dominated by a target-independent term proportional to (W^\top Wh). However, this observation does not establish why representations from different inputs converge toward a shared vector, nor does it explain the subsequent recovery dynamics.
W1.1. The paper's central contribution is not merely the observation of collapse but the proposed relationship between initialization scale and its duration which makes W1 important.
W1.2. The dominant shared-mean component substantially amplifies the reported reduction in uncentered rank, raising questions about how much of the observed phenomenon reflects a genuine loss of input-dependent representation dimensionality.
W2. The experimental validation is restricted to MelHuBERT on LibriSpeech, with only two seeds for the main experiments and one for several ablations. Importantly, the paper does not provide full-length experiments near the standard initialization ratio (r\approx 0.577), nor does it experimentally establish that the proposed failure mode occurs in other commonly used regression-based SSL frameworks such as MAE or data2vec.
Questions for authors
Q1. In continuation to W2, does the relationship between the initial readout-to-target ratio (r) and collapse duration hold in another regression-based SSL architecture, such as MAE or data2vec?
Q2. In continuation to W2, would two readouts with identical initial output norms but different singular-value spectra or alignments with the encoder representations exhibit similar recovery times?
Q3. In continuation to W1, how does the relative contribution of the target-dependent and target-independent gradient terms evolve throughout training, and does this evolution predict the recovery point? The current initialization-time analysis does not directly explain the duration of collapse.
Responses to Q1 and Q3 can substantially change my assessment.
Edit: increasing my score from 2 to 3 post-rebuttal.
4 comments
Response to Reviewer LHtH (2/2): W1, W1.2, W2 and Q2
W1: Why do inputs converge to one shared vector, and why does the duration depend on r? We agree that Section 4.4 showed neither. The new analysis separates the two: the shared vector forms the same way at every r, and r sets how long it lasts.
How it forms. For a linear layer z = Au + c, the weight gradient over a batch splits exactly into (Σᵢ gᵢ)ūᵀ, the summed output gradient times the mean input, plus the centered part Σᵢ (gᵢ − ḡ)(uᵢ − ū)ᵀ. The first part moves every frame's output in the same direction. Inputs and output gradients are not centered across frames (after the GELU, the mean carries 24% of the input's energy), so this part dominates at initialization. This is the classical effect of uncentered inputs (LeCun et al., 1998), sharpened by Adam, whose early updates are close to sign steps. Tests, one seed at each of r = 0.58, 4.52 and 37.3:
| Test | Result |
|---|---|
| Share of the first part in the initial gradient | 83–99% of the norm in the MLP layers and attention query, value and output projections, at all three r |
| First Adam update, r = 4.52 | Each layer moves the mean by 83–95% of the most a sign step allows; random signs give 2–4% |
| Remove the first part from every encoder weight gradient | Block 5's mean share at step 20 is 0.14 instead of 0.95 (r = 4.52 and 37.3), 0.33 instead of 0.89 (r = 0.58) |
| Keep only the first part | Reproduces the normal run over those 20 steps (r = 4.52) |
| Onset speed (3 seeds) | Mean share at step 10: 0.82–0.89 at r = 0.58, 0.86–0.92 at r = 37.3 |
In a 2,000-step run without the first part, the shared vector still forms by step ~150 as later updates realign with the mean input, so this tests the route rather than offering a fix. Since the onset is nearly as fast at r = 0.58 as at 37.3, the paper's initialization result explains the duration, not the onset; Section 4.4 is rewritten accordingly.
Why r sets the duration. The collapse lasts until the target pull catches up with the pull toward zero (A ≈ 1, see the table in our Q3 answer). The shrinking comes from the per-frame part of the target-free term: at r = 37.3, removing it from the encoder's gradient leaves the output at 6.6–7.0 times the target's size at step 1,000, against 1.4 normally, while removing a frame-shuffled copy of the same size changes nothing (3 seeds each). Adam also slows the catch-up: at r = 4.52 its updates are about 1% of the learning rate by step 50, mainly because its second-moment estimate still reflects the first gradients, 70–150 times larger. We measure the resulting curve (about 26-fold longer from r = 0.58 to 4.52, about 6-fold or more from 4.52 to 37.3) but do not derive it.
W1.2: Genuine loss or offset? Both. Block 5 rank (singular values above 1% of the largest), from about 480 at initialization:
| r | Raw rank, lowest | Mean-removed rank, lowest in the first 600 steps | Mean-removed rank at step 1,000 / 8,000 | Phone probe at 8,000 |
|---|---|---|---|---|
| 0.58 | 29–43 | 180–253 | 393–448 / 767 | 50.9% |
| 1.6 | 17–25 | 175–280 | 347–423 / 767 | 49.5% |
| 4.52 | 16–21 | 202–287 | 329–383 / 767 | 46.6% |
| 37.3 | 14–15 | 224–292 | 268–295 / 394–439 | 40.9% |
Most of the raw drop is the offset, but mean-removed rank still falls by roughly half during the onset at every r: a genuine loss of input-dependent dimensions, of similar depth across r. r sets how long it lasts: by step 8,000 mean-removed rank is back at its ceiling (767) for r ≤ 4.52 but not at r = 37.3, where recovery is also uneven (while A is below 1, setbacks briefly push it as low as 123). The probe, which ignores the offset, follows r, and on 960 h r = 4.52 still trails r = 0.58 by 3.3–4.5 points from step 8,000 to 20,000 (3 seeds to step 12,000, then 2). The revision reports mean-removed rank and standardized probes beside the raw measures throughout.
W2 and Q2. All main comparisons now have 3 seeds (5 for the small-readout pair), with full-length runs at r = 0.58 (8,500 steps on 100 h, 20,000 on 960 h). At r = 0.58 the collapse is brief but present (a median of 80 steps over 10 runs), and it sets in between r = 0.2 and 0.25. Q1 covers MAE and data2vec. Downstream, a smaller readout (r = 0.04) never collapses and beats r = 0.58 in all 5 paired seeds at step 20,000 on 960 h (+0.94 points, 90% CI 0.53–1.35). For Q2, six readouts at matched r = 4.52 (3 seeds each) finish on Saturday: flat or steep singular spectra, alignment with the encoder's strongest or weakest directions, and alignment or anti-alignment with the target. We will post the result here.
Thank you for the detailed rebuttal.
The MAE and data2vec-style experiments strengthen the generality of the empirical findings. The analysis of the target-dependent and target-independent update contributions is useful, although the proposed quantity (A) appears to be an empirical indicator of recovery rather than a complete explanation of its dynamics.
That being said, the rebuttal has surely improved my assessment of the paper, and I am inclined to raise my rating from 2 (Weak Reject) to 3 (Weak Accept).
I would still appreciate if the authors could submit a revised manuscript during the discussion phase.
Revision uploaded: Q1 completed and W1.2 corrected
Thank you for reading our replies so closely, and for raising your score. Your questions were precise, and answering them improved the paper.
The revised PDF is now uploaded, and its pdfdiff against the submission shows the changes. With it is a new supplementary zip: the code and every run's configuration and logged curves, with scripts that rebuild each table, figure and number.
On A. The revision reads A the way your comment does, as a marker of the end rather than its cause: balance is neither sufficient nor necessary for the collapse to end, and our forecast from A is rougher than we first reported (general comment). The length is explained instead by distance. The readout's output begins at r times the target's norm, and the collapse ends only when it has shrunk to about a quarter to a third of the target's size. So a larger r has farther to go along a path shared by every r, at a pace that depends on Adam's memory of the early gradients (general comment; reply to Reviewer fGBW). What fixes that end point we cannot yet explain, and the revision says so.
Q1, completed (Section 4.1, Table 2). MAE now has five ratios, and in the data2vec-style teacher we separated r from the target's normalization, which our first answer changed together.
MAE ViT-B/16 on ImageNet-100, the arms differing only in the target's scale (three seeds; kNN accuracy at epoch 25, the length of the three new arms):
| r | 0.28 | 0.89 (MAE's own) | 2.8 | 8.9 | 24.8 |
|---|---|---|---|---|---|
| Peak mean share, encoder output | 0.48–0.51 | 0.65–0.71 | 0.85–0.88 | 0.91–0.93 | 0.93–0.94 |
| Collapse ends at step | 50, 50, none | 250 | 300 | 500 | 1,000 |
| kNN accuracy (%) | 22.5 | 21.6 | 21.0 | 17.9 | 15.4 |
All 78 pairs of runs whose order is clear end in the order of r at the encoder's output (p ≤ 0.0005), and all 27 at its middle block (p = 0.009; Holm-corrected, Appendix F). The kNN figures in our first reply (24.9 and 15.3% at r = 0.89 and 24.8) were after all 50 epochs. The reruns at r = 24.8 with Adam's ε at 1e-12 now have three seeds, and the first collapse is unchanged in all three (Appendix B.4).
data2vec-style teacher on 960 h, each kind of target scaled to both ratios (three seeds; phone probe at step 8,000):
| Targets | r = 1.7 | r = 16.1 |
|---|---|---|
| data2vec's own normalization | ends at 150–200; 55.3% | ends at 8,000–9,600; 45.6% |
| Unit norm per frame | ends at 150–200; 52.8% | not ended by 12,000; 44.3% |
With either normalization, r = 16.1 instead of 1.7 makes the collapse at least 40 times longer and the probe 8.5 to 9.7 points lower; at the same r, swapping the normalization costs at most 2.5 points. So in data2vec, as in MAE, r sets the duration.
W1.2, with a correction. Our first reply said most of the raw rank drop is the offset and that mean-removed rank falls by about half. Over all the sweep's seeds, raw rank falls from about 480 to 9 to 43 at every r ≥ 0.58, and mean-removed rank to 117 to 292 at every r ≥ 0.3. The offset is part of the raw drop, not most of it: it inflates the largest singular value and so raises the 1% bar. The loss of input-dependent dimensions is real, at a depth that does not depend on r, and its length grows with r: by step 8,000 rank with the mean removed has fully recovered for every r ≤ 4.52 but not at 13 or 37.3 (Section 4.2, Table 1).
W2 and Q2. The main sweep now runs every ratio at full length with three seeds, the standard r = 0.58 included (Table 1), and Q1 above covers MAE and data2vec. The seven readouts at matched r for Q2 are in the general comment, which answers you and Reviewer fGBW together.
We are happy to answer anything else before the discussion closes.
Response to Reviewer LHtH (1/2): Q1 and Q3
Thank you for the careful review. Q1 and Q3 were the tests the paper most needed, and we ran both; they are answered here, and W1, W1.2, W2 and Q2 in part 2. As in the general response, the collapse ends when the shared mean's share of the representation's energy falls below 0.5 for three evaluations, probes are linear, and runs have three seeds unless stated.
Q1: Does the relationship hold in MAE or data2vec? Yes in both, at the two ratios tested so far in each.
Q3: How does the balance evolve, and does it predict recovery? Mostly yes: it tracks recovery closely, with two exceptions. We log A, the target-driven part of the encoder's update divided by the target-free part, after Adam's per-weight normalization, on fixed evaluation batches throughout every run. A by training step (means over 3 seeds, 100 h):
We would value your view on whether these results answer Q1 and Q3.