acceptodds
Under review as a conference paper at ICLR 2027

Target Scale Is Readout Scale: One Ratio Orders How Long Collapse Lasts

Abstract

Many self-supervised methods train an encoder through a linear readout that regresses a dense target. When the encoder ends in a layer normalization, the readout's output at initialization has a norm set by the widths alone, regardless of the target. Scaling the target is therefore equivalent to scaling the readout, and what matters is the ratio between the readout's initial output and the target's norm. The standard recipe, a unit-gain layer normalization with a fan-in initialized readout and a unit-variance target, fixes at every width. We test what happens when it does not, in a controlled case study: MelHuBERT trained on LibriSpeech with regression targets of different scales. Rank collapsed early in every run with a default readout, and ordered how long the collapse lasted: about a thousand steps at and more than twelve thousand at . Of 94 pairs of runs at different ratios, 90 are ordered as predicts, none against it, and 4 are undetermined because neither run had recovered. Most of the collapse is a shared offset, with 98% of the representation's energy in its mean over frames; with the mean removed rank falls by over a third, and further under a softmax readout, which builds no offset. At 8,500 steps orders both the offset and the rank that remains without it. In 500-step runs at , a sixty-fold change in the readout's learning rate moved rank by two, while lowering restored 100 to 419 dimensions. At large the gradient at the encoder's output is dominated by a term that does not contain the target, and an approximate closed form for its direction matches simulation to 0.3%. MAE (0.89) and data2vec (0.58 to 1.63) sit close to . Distance from initialization measured by CKA, a common diagnostic, can miss the collapse: CKA centers the representation, and two runs 0.12% apart on it differ sevenfold in rank, twofold with the mean removed. Checking costs one batch, before training starts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation collapse, dimensional collapse, target normalization, initialization, layer normalization, speech representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LHtHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how the relative scale of a linear prediction head's initial output and its regression targets affects representation collapse in self-supervised learning. The authors introduce a readout-to-target norm ratio rr and show that, under standard layer normalization and readout initialization, this ratio is approximately 1/31/\sqrt{3}, independent of model width.

Using MelHuBERT on LibriSpeech, the authors find that representations undergo rapid rank collapse early in training, with larger values of (r) consistently associated with longer recovery times. To explain the observations, the authors analyze the initial encoder gradients and show that, at large (r), a target-independent gradient term dominates. While this provides a plausible explanation for the onset of collapse, the analysis does not establish the full mechanism governing collapse or recovery. The empirical validation is restricted to one model family and dataset, and the generality and downstream significance of the findings remain uncertain.

Critical strengths

S1. The paper identifies a simple initialization-dependent quantity, the readout-to-target ratio, that strongly predicts the duration of early representation collapse in the studied setting. The authors support this finding through controlled interventions on target scale, target spectral shape, and readout initialization. The consistent ordering of recovery times across configurations is compelling and suggests a potentially useful diagnostic for regression-based ssl.

S2. The paper shows that much of the observed rank reduction arises from a dominant shared mean vector, while a smaller but meaningful reduction remains after centering. This distinction is important because it demonstrates how uncentered spectral measurements can exaggerate the apparent loss of representation dimensionality. The accompanying analysis of CKA's insensitivity to the shared offset provides a useful methodological insight for studying representation collapse.

Critical weaknesses

W1. The mechanistic explanation does not establish why representations collapse or why recovery time depends on (r). Section 4.4 shows that, at large (r), the initial encoder-output gradient is dominated by a target-independent term proportional to (W^\top Wh). However, this observation does not establish why representations from different inputs converge toward a shared vector, nor does it explain the subsequent recovery dynamics.

W1.1. The paper's central contribution is not merely the observation of collapse but the proposed relationship between initialization scale and its duration which makes W1 important.

W1.2. The dominant shared-mean component substantially amplifies the reported reduction in uncentered rank, raising questions about how much of the observed phenomenon reflects a genuine loss of input-dependent representation dimensionality.

W2. The experimental validation is restricted to MelHuBERT on LibriSpeech, with only two seeds for the main experiments and one for several ablations. Importantly, the paper does not provide full-length experiments near the standard initialization ratio (r\approx 0.577), nor does it experimentally establish that the proposed failure mode occurs in other commonly used regression-based SSL frameworks such as MAE or data2vec.

Questions for authors

Q1. In continuation to W2, does the relationship between the initial readout-to-target ratio (r) and collapse duration hold in another regression-based SSL architecture, such as MAE or data2vec?

Q2. In continuation to W2, would two readouts with identical initial output norms but different singular-value spectra or alignments with the encoder representations exhibit similar recovery times?

Q3. In continuation to W1, how does the relative contribution of the target-dependent and target-independent gradient terms evolve throughout training, and does this evolution predict the recovery point? The current initialization-time analysis does not directly explain the duration of collapse.

Responses to Q1 and Q3 can substantially change my assessment.

Edit: increasing my score from 2 to 3 post-rebuttal.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LHtH (1/2): Q1 and Q3

Thank you for the careful review. Q1 and Q3 were the tests the paper most needed, and we ran both; they are answered here, and W1, W1.2, W2 and Q2 in part 2. As in the general response, the collapse ends when the shared mean's share of the representation's energy falls below 0.5 for three evaluations, probes are linear, and runs have three seeds unless stated.

Q1: Does the relationship hold in MAE or data2vec? Yes in both, at the two ratios tested so far in each.

FrameworkrPeak mean shareCollapse ends at stepProbe accuracy
MAE ViT-B/16, ImageNet-100, official recipe at batch 512, 50 epochs0.89 (MAE's normalized-pixel targets) / 24.8 (same targets times one constant)0.65–0.71 / 0.93–0.94250 / 1,000Linear 44.0 / 30.0%, kNN 24.9 / 15.3%, after training
data2vec-style EMA teacher on MelHuBERT, 960 h (mean of the teacher's top 8 blocks, each instance-normalized; decay 0.999 → 0.9999)1.7 (data2vec's targets) / 16 (unit norm per frame)0.98 / 0.99150–200 / not by 12,000Phone 55.3 / 44.3% at step 8,000
  • In MAE only the target's scale differs between the arms, so both the longer collapse and the lower probe are effects of r. In the data2vec-style runs, unit norm also evens out each frame's size, so that comparison changes more than r.
  • At r = 24.8, MAE collapses a second time from step 3,000–3,750 and stays collapsed (mean share 0.59–0.73 at the end). This second collapse is largely Adam's ε at work: the small target brings the encoder's gradients down to about 1e-8 and below, where ε cuts weight updates more than bias updates, so per-patch structure stalls while the shared part holds. Rerun with ε = 1e-12 (seed 0), the first collapse is unchanged (same depth, over by about step 1,000), and through step 4,000 the second is much weaker (middle-block mean share 0.19 vs 0.52). So the first collapse, the one r orders, does not depend on ε.
  • MAE at r = 0.28, 2.8 and 8.9 finishes on Sunday, giving five ratios.

Q3: How does the balance evolve, and does it predict recovery? Mostly yes: it tracks recovery closely, with two exceptions. We log A, the target-driven part of the encoder's update divided by the target-free part, after Adam's per-weight normalization, on fixed evaluation batches throughout every run. A by training step (means over 3 seeds, 100 h):

rStep 202001,0002,0004,0008,000Collapse ends at step
0.580.190.791.131.041.161.1360–80
1.60.120.211.061.101.131.05300–400
4.520.050.110.500.831.121.051,800–2,100
37.30.010.050.060.080.110.46 (0.80 at 11,000)11,100 or later
  • A starts small, and smaller at larger r: the target-free part dominates at initialization, as Section 4.4 found. It rises as the output shrinks, because the target-free part scales with the output and the target-driven part does not. At r = 37.3 it also rises faster late than shrinking alone explains, one reason we have no closed form for the duration.
  • Every recovery happens near A = 1: A is 0.57–1.68 at the end in all 58 runs where the collapse ends (speech and MAE; median 0.92). No run recovers while A is small.
  • A near 1 does not guarantee recovery. With unit-norm EMA targets, A stays at 0.97–1.06 from step 2,000 to 12,000 while the collapse persists. With the paper's γ = 2 target at r = 37.3, A's moving average reaches 0.7 at steps 1,900–2,400, yet the runs are still collapsed at 12,000.
  • As a forecast (the end is 0.97 times the step where A's moving average first reaches 0.7, with 0.97 fitted on seed 0 of the main sweep), A places 17 of the 18 held-out runs that recover within a factor of 1.5 (median error factor 1.20), against 1.38 and 2.00 for power laws in the current or initial r. A reaches 0.7 before the end in 11 of the 18, so it marks recovery more than it warns of it.
  • Causal test: switching r from 37.3 to 4.52 at step 200 makes the run recover at steps 1,500–2,600, like runs started at 4.52 (1,800–2,100), instead of after 11,000.

We would value your view on whether these results answer Q1 and Q3.

AuthorsOpenReview ↗

Response to Reviewer LHtH (2/2): W1, W1.2, W2 and Q2

W1: Why do inputs converge to one shared vector, and why does the duration depend on r? We agree that Section 4.4 showed neither. The new analysis separates the two: the shared vector forms the same way at every r, and r sets how long it lasts.

How it forms. For a linear layer z = Au + c, the weight gradient over a batch splits exactly into (Σᵢ gᵢ)ūᵀ, the summed output gradient times the mean input, plus the centered part Σᵢ (gᵢ − ḡ)(uᵢ − ū)ᵀ. The first part moves every frame's output in the same direction. Inputs and output gradients are not centered across frames (after the GELU, the mean carries 24% of the input's energy), so this part dominates at initialization. This is the classical effect of uncentered inputs (LeCun et al., 1998), sharpened by Adam, whose early updates are close to sign steps. Tests, one seed at each of r = 0.58, 4.52 and 37.3:

TestResult
Share of the first part in the initial gradient83–99% of the norm in the MLP layers and attention query, value and output projections, at all three r
First Adam update, r = 4.52Each layer moves the mean by 83–95% of the most a sign step allows; random signs give 2–4%
Remove the first part from every encoder weight gradientBlock 5's mean share at step 20 is 0.14 instead of 0.95 (r = 4.52 and 37.3), 0.33 instead of 0.89 (r = 0.58)
Keep only the first partReproduces the normal run over those 20 steps (r = 4.52)
Onset speed (3 seeds)Mean share at step 10: 0.82–0.89 at r = 0.58, 0.86–0.92 at r = 37.3

In a 2,000-step run without the first part, the shared vector still forms by step ~150 as later updates realign with the mean input, so this tests the route rather than offering a fix. Since the onset is nearly as fast at r = 0.58 as at 37.3, the paper's initialization result explains the duration, not the onset; Section 4.4 is rewritten accordingly.

Why r sets the duration. The collapse lasts until the target pull catches up with the pull toward zero (A ≈ 1, see the table in our Q3 answer). The shrinking comes from the per-frame part of the target-free term: at r = 37.3, removing it from the encoder's gradient leaves the output at 6.6–7.0 times the target's size at step 1,000, against 1.4 normally, while removing a frame-shuffled copy of the same size changes nothing (3 seeds each). Adam also slows the catch-up: at r = 4.52 its updates are about 1% of the learning rate by step 50, mainly because its second-moment estimate still reflects the first gradients, 70–150 times larger. We measure the resulting curve (about 26-fold longer from r = 0.58 to 4.52, about 6-fold or more from 4.52 to 37.3) but do not derive it.

W1.2: Genuine loss or offset? Both. Block 5 rank (singular values above 1% of the largest), from about 480 at initialization:

rRaw rank, lowestMean-removed rank, lowest in the first 600 stepsMean-removed rank at step 1,000 / 8,000Phone probe at 8,000
0.5829–43180–253393–448 / 76750.9%
1.617–25175–280347–423 / 76749.5%
4.5216–21202–287329–383 / 76746.6%
37.314–15224–292268–295 / 394–43940.9%

Most of the raw drop is the offset, but mean-removed rank still falls by roughly half during the onset at every r: a genuine loss of input-dependent dimensions, of similar depth across r. r sets how long it lasts: by step 8,000 mean-removed rank is back at its ceiling (767) for r ≤ 4.52 but not at r = 37.3, where recovery is also uneven (while A is below 1, setbacks briefly push it as low as 123). The probe, which ignores the offset, follows r, and on 960 h r = 4.52 still trails r = 0.58 by 3.3–4.5 points from step 8,000 to 20,000 (3 seeds to step 12,000, then 2). The revision reports mean-removed rank and standardized probes beside the raw measures throughout.

W2 and Q2. All main comparisons now have 3 seeds (5 for the small-readout pair), with full-length runs at r = 0.58 (8,500 steps on 100 h, 20,000 on 960 h). At r = 0.58 the collapse is brief but present (a median of 80 steps over 10 runs), and it sets in between r = 0.2 and 0.25. Q1 covers MAE and data2vec. Downstream, a smaller readout (r = 0.04) never collapses and beats r = 0.58 in all 5 paired seeds at step 20,000 on 960 h (+0.94 points, 90% CI 0.53–1.35). For Q2, six readouts at matched r = 4.52 (3 seeds each) finish on Saturday: flat or steep singular spectra, alignment with the encoder's strongest or weakest directions, and alignment or anti-alignment with the target. We will post the result here.

Thank you for the detailed rebuttal.

The MAE and data2vec-style experiments strengthen the generality of the empirical findings. The analysis of the target-dependent and target-independent update contributions is useful, although the proposed quantity (A) appears to be an empirical indicator of recovery rather than a complete explanation of its dynamics.

That being said, the rebuttal has surely improved my assessment of the paper, and I am inclined to raise my rating from 2 (Weak Reject) to 3 (Weak Accept).

I would still appreciate if the authors could submit a revised manuscript during the discussion phase.

AuthorsOpenReview ↗

Revision uploaded: Q1 completed and W1.2 corrected

Thank you for reading our replies so closely, and for raising your score. Your questions were precise, and answering them improved the paper.

The revised PDF is now uploaded, and its pdfdiff against the submission shows the changes. With it is a new supplementary zip: the code and every run's configuration and logged curves, with scripts that rebuild each table, figure and number.

On A. The revision reads A the way your comment does, as a marker of the end rather than its cause: balance is neither sufficient nor necessary for the collapse to end, and our forecast from A is rougher than we first reported (general comment). The length is explained instead by distance. The readout's output begins at r times the target's norm, and the collapse ends only when it has shrunk to about a quarter to a third of the target's size. So a larger r has farther to go along a path shared by every r, at a pace that depends on Adam's memory of the early gradients (general comment; reply to Reviewer fGBW). What fixes that end point we cannot yet explain, and the revision says so.

Q1, completed (Section 4.1, Table 2). MAE now has five ratios, and in the data2vec-style teacher we separated r from the target's normalization, which our first answer changed together.

MAE ViT-B/16 on ImageNet-100, the arms differing only in the target's scale (three seeds; kNN accuracy at epoch 25, the length of the three new arms):

r0.280.89 (MAE's own)2.88.924.8
Peak mean share, encoder output0.48–0.510.65–0.710.85–0.880.91–0.930.93–0.94
Collapse ends at step50, 50, none2503005001,000
kNN accuracy (%)22.521.621.017.915.4

All 78 pairs of runs whose order is clear end in the order of r at the encoder's output (p ≤ 0.0005), and all 27 at its middle block (p = 0.009; Holm-corrected, Appendix F). The kNN figures in our first reply (24.9 and 15.3% at r = 0.89 and 24.8) were after all 50 epochs. The reruns at r = 24.8 with Adam's ε at 1e-12 now have three seeds, and the first collapse is unchanged in all three (Appendix B.4).

data2vec-style teacher on 960 h, each kind of target scaled to both ratios (three seeds; phone probe at step 8,000):

Targetsr = 1.7r = 16.1
data2vec's own normalizationends at 150–200; 55.3%ends at 8,000–9,600; 45.6%
Unit norm per frameends at 150–200; 52.8%not ended by 12,000; 44.3%

With either normalization, r = 16.1 instead of 1.7 makes the collapse at least 40 times longer and the probe 8.5 to 9.7 points lower; at the same r, swapping the normalization costs at most 2.5 points. So in data2vec, as in MAE, r sets the duration.

W1.2, with a correction. Our first reply said most of the raw rank drop is the offset and that mean-removed rank falls by about half. Over all the sweep's seeds, raw rank falls from about 480 to 9 to 43 at every r ≥ 0.58, and mean-removed rank to 117 to 292 at every r ≥ 0.3. The offset is part of the raw drop, not most of it: it inflates the largest singular value and so raises the 1% bar. The loss of input-dependent dimensions is real, at a depth that does not depend on r, and its length grows with r: by step 8,000 rank with the mean removed has fully recovered for every r ≤ 4.52 but not at 13 or 37.3 (Section 4.2, Table 1).

W2 and Q2. The main sweep now runs every ratio at full length with three seeds, the standard r = 0.58 included (Table 1), and Q1 above covers MAE and data2vec. The seven readouts at matched r for Q2 are in the general comment, which answers you and Reviewer fGBW together.

We are happy to answer anything else before the discussion closes.

Reviewer_wMmuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate and analyze how a scale mismatch between the linear readout and the regression target can cause supervised learning to collapse. By conducting comprehensive experiments that vary this scale ratio, r, and other parameters using LibriSpeech’s train-clean-100 subset and MelHuBERT, the authors find that larger r often means longer periods of collapse and more time needed for recovery. Further analysis shows that this collapse may be due to a dominant shared mean vector. The authors suggest checking the initialization of the readout layer instead of tuning the learning rate of the linear layer when the model experiences collapse.

Critical strengths
  • The suggestion to check r can be helpful in some cases and can provide insights when designing training recipes.
  • The paper provides a mathematical analysis of why this makes training less efficient.
  • The ablation experiments are quite comprehensive.
Critical weaknesses
  • The paper investigates what seems to be an extreme edge case, and I am not sure how likely such a large scale mismatch is to occur in practice. As the authors note, standard recipes with normalized targets already keep r within a fairly reasonable range. The paper sometimes reads to me as, “Let’s introduce a substantial scale mismatch and see how training goes wrong.” The experiments show that it can go wrong, but I am less convinced that this is a problem practitioners commonly encounter. It is a standard procedure that the training target and the model's parameter should be normalized before training.
  • The learning-rate comparison is limited to 500 steps, when the measured ranks are all close to their minimum. This shows that changing the learning rate does not substantially improve early rank in that setting, but it does not establish whether the learning rate affects recovery several thousand steps later.
  • The study is conducted using only one audio model and approximately 100 hours of training data. Many aspects of training can change as the scale increases, and it is unclear whether the same recovery-ordering result would hold with larger datasets or different architectures. There are also no actual training results for text or vision; the paper only estimates r for selected published systems. I am not fully convinced that the findings would generalize beyond the current setting.
Questions for authors

I am more interested in the small r, and want to see how this can help to further improve self supervised learning. Do the improvements observed at r = 0.04 and r = 0.37 persist throughout training? Are there any trade-offs between faster rank recovery and downstream performance when r becomes very small?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wMmu

Thank you for the review, and for steering us toward small r: it produced the most practical result of the new runs. The standard ratio r ≈ 0.58 already sits above the onset of the collapse, and a smaller readout avoids it and gives a small, consistent gain. We also repeated the learning-rate comparison over full training and extended the study to 960 h, vision and a data2vec-style teacher. As in the general response, the collapse ends when the shared mean's share of the representation's energy falls below 0.5 for three evaluations, probes are linear, and runs have three seeds unless stated.

Q: Do the small-r gains persist, and is there a trade-off? On 960 h through 20,000 steps, the gain at r = 0.04 persists, the smaller one at r = 0.37 fades, and we see no meaningful trade-off.

r (seeds)Peak mean shareCollapse ends at stepPhone probe at step 2,000 / 8,000 / 20,000Speaker probe at 20,000
0.04 (5)0.15–0.21no collapse48.2 / 52.4 / 57.1%98.9%
0.37 (3)0.70–0.8130–6047.1 / 51.9 / 56.6%98.9%
0.58, default (5)0.85–0.9980–40046.3 / 51.5 / 56.1%98.9%
  • r = 0.04 leads r = 0.58 in all 5 paired seeds at steps 1,000, 2,000, 8,000, 12,000 and 20,000 (+0.94 points at 20,000, 90% CI 0.53–1.35). In the 40,000-step extension, the pairs that have passed step 20,000 still favor it (+0.9 at 28,000, +1.1 at 24,000); all three pairs finish on Saturday.
  • r = 0.37 still collapses briefly. It leads r = 0.58 in every seed through step 12,000, but at steps 16,000 and 20,000 the difference is within seed noise.
  • Trade-offs: r = 0.04 is not slower early (it leads from step 1,000). Speaker identification, a second probe, is near ceiling for every r; r = 0.04 is 0.1–0.4 points lower at steps 1,000–2,000 and within 0.3 points from step 8,000 on. The extreme case, r = 0 (a zeroed readout), finishes on Sunday.
  • For practice: scaling the default readout initialization down (here by 0.04/0.58) costs nothing and gives this gain. Whether it holds at full pre-training length is what the 40,000-step runs begin to test.

W1: Is a large mismatch an edge case? Partly. The standard recipe keeps r ≈ 0.58, and the revision says so up front. But normalizing the target does not by itself keep r small; which normalization is used decides it. Under PyTorch's default readout initialization, a target standardized per coordinate gives r ≈ 0.58 at any width, while the same target at unit norm per frame gives r ≈ √(m/3), about 16 at width 768. In our data2vec-style runs, data2vec's own targets (an average of 8 instance-normalized layers) give r ≈ 1.7, already about three times the standard ratio. Measured costs:

  • data2vec-style EMA teacher at r ≈ 16 instead of 1.7: still collapsed at step 12,000 instead of ending by step 200, and the phone probe at step 8,000 is 11 points lower.
  • r = 4.52 on 960 h: 3.3–4.5 phone-probe points behind r = 0.58 at every checkpoint from step 8,000 to 20,000 (3 seeds to step 12,000, then 2), long after its collapse ends.
  • Near the standard recipe the effect is small but present: the collapse sets in between r = 0.2 and 0.25, so r = 0.58 is already above the onset, and going below it helps (table above).

Checking r costs one forward pass at initialization.

W2: The learning-rate comparison covered only 500 steps. Agreed; it now covers full training. At r = 4.52 over 8,500 steps, multiplying or dividing the readout's learning rate by 7.79 moves the end of the collapse at most 1.7-fold and leaves the final probe unchanged (47.1–47.2% vs 47.2%). The same 7.79-fold change in r moves the end about 26-fold and raises the final probe to 51.4%. Halving the global learning rate delays the end, and doubling it makes the collapse total and longer (table in the general response).

W3: One audio model, 100 h, no vision or text.

  • More data: on 960 h the durations match 100 h (r = 4.52: 1,600–1,900 vs 1,800–2,100 steps; r = 37.3: about 11,000 or later on both), and the probe keeps the order of r at five ratios (54.2 / 53.6 / 53.1 / 49.5 / 43.0% at step 12,000 for r = 0.04 / 0.37 / 0.58 / 4.52 / 37.3).
  • Vision: MAE ViT-B/16 on ImageNet-100 (official recipe at batch 512, 50 epochs), with MAE's targets (r = 0.89) and the same targets times a constant (r = 24.8). The collapse ends by step 250 vs 1,000 and the final linear probe is 44.0 vs 30.0%; r = 24.8 also collapses a second time later, linked to Adam's ε (reply to LHtH). MAE at r = 0.28, 2.8 and 8.9 finishes on Sunday.
  • A second regression target: the data2vec-style EMA teacher above.
  • We have not trained text models; the revision says so.

We are glad to run further checks if any point remains unclear.

AuthorsOpenReview ↗

Follow-up: the gain at r = 0.04 holds to step 40,000

Thank you again for pointing us to small r. The revision, now uploaded and rewritten for clarity, follows it to 40,000 steps (Section 4.1).

Do the gains at small r persist? At r = 0.04, which never collapses, yes, through step 40,000. At r = 0.37, which still collapses briefly, the lead holds only through step 12,000, as in our first reply. All three 40,000-step pairs of r = 0.04 against the standard 0.58 on 960 h are now complete, each pair sharing its seed, data order and target. Phone-probe gain of r = 0.04, mean over the three pairs:

Step2,0008,00012,00016,00020,00024,00028,00032,00036,00040,000
Gain (points)+2.03+1.30+1.40+0.76+0.84+0.83+0.88+1.02+0.92+0.74

The lead narrows early, then holds between 0.7 and 1.0 points from step 20,000 to 40,000 while both arms gain about 5 points, and it is positive in every pair at each of those checkpoints (18 of 18). At step 40,000 the arms reach 61.9 and 61.1%; by pair the gain is 0.3, 1.8 and 0.1 points, so its size varies with the seed but not its sign. The gain also appears in MAE: at r = 0.28, below MAE's own 0.89, kNN accuracy at epoch 25 is 22.5% against 21.6%, higher in all three seeds (Table 2).

Trade-offs at very small r. None that we see. Speaker identification is 98.9% in both arms at step 40,000. At the extreme, a zeroed readout is not better: on 100 h it reaches 52.4% at step 8,000 against 52.9% at r = 0.14, lower in all three seeds. The gain comes from a small readout, not from removing it.

W1: how often a large mismatch happens in practice. You are right that published recipes keep r near the standard ratio: they sit between 0.39 and 1.63 (Section 5, now with I-JEPA and V-JEPA). The main sweep now spans r = 0 to 37.3, so it shows where r helps as well as where it hurts, and two results make r worth checking in practice. First, it matters across the published range: on 100 h, phone accuracy at step 8,000 is 51.5, 50.9, 50.4 and 49.5% at r = 0.3, 0.58, 1 and 1.6 (Table 1). Second, normalizing the target does not settle r, and r, not the normalization, carries the cost. In the data2vec-style teacher we scaled each kind of target to the other's ratio (Table 2): going from r = 1.7 to 16.1 lengthens the collapse at least 40-fold and costs 8.5 to 9.7 points of phone accuracy under either normalization, while at a fixed r the normalization moves the probe by at most 2.5 points. Learning-rate warmup does not make a large r safe either (general comment).

W2 and W3. The readout's learning rate now also covers r = 37.3 over 12,000 steps, with the same conclusion: even where a faster readout shortens the collapse, it still lasts at least 3.5 times as long as at r = 4.52 (Section 4.3, Table 3). MAE now has five ratios, from 0.28 to 24.8. Its collapse ends at 50 to 1,000 steps in the order of r for all 78 pairs whose order is clear, and kNN accuracy falls from 22.5% at r = 0.28 to 17.9% at 8.9, and to 15.4% at 24.8 during the second collapse noted in our first reply (Table 2, Appendix F).

We hope this answers your question about practical relevance and earns the revision a second look in your assessment.

Thanks for the detailed rebuttal and the additional experiments results across different modalities and methods. I really appreciate the authors' effort in providing these empirical findings and insights.

However, I still consider the practical gain from this technique to be limited. Within the range of r used by published recipes, the improvement is small and close to seed noise, especially when training long enough. In addition, finding the optimal r appears to require a sweep, so whether tuning r provides a benefit beyond what is typically obtained by tuning other hyper-parameters, such as the learning rate, still remain a question to me.

Therefore, I prefer to maintain my original rating.

AuthorsOpenReview ↗

Where r matters in practice

Thank you for the kind words about the new results and for being clear about what still concerns you.

Near the standard ratio, where published recipes sit, the gain from a smaller r is small, as the paper says (Section 4.1). Early in training it is consistent, though: on 100 h at step 8,000, r = 0.58 leads r = 1.6, about data2vec's ratio, by 1.2 to 1.5 points of phone accuracy in every seed, while seeds at either ratio differ by under half a point. It may shrink with long training, as you note. In the paper it is a side result; the practical point is at large r.

There the cost is large, and it lasts. On 100 h at step 8,000, phone accuracy falls from 50.9% at the standard r = 0.58 to 46.6% at 4.52 and 40.9% at 37.3 (Table 1). On 960 h, r = 4.52 trails 0.58 by 3.3 to 4.5 points at each checkpoint from step 8,000 until its runs end, though its collapse is over by step 1,900 (Section 4.1). Under a data2vec-style teacher, r = 16.1 costs 8.5 to 9.7 points against 1.7, and in MAE, k-NN accuracy falls from 21.6% at MAE's own r = 0.89 to 17.9% at 8.9 (Table 2).

Your review noted that checking r can help when designing training recipes, and that is where a large r arises. Published recipes stay in range, but the changes that leave it are the kind a new recipe makes (Section 5): regressing an unnormalized target such as codec latents or a teacher's outputs, cutting the target to unit norm per input instead of per coordinate (r ≈ 16 at width 768), dropping the layer normalization before the readout or initializing the readout at a fixed standard deviation.

Avoiding that cost needs no sweep. Checking r is one forward pass at initialization (Section 7). If it is large, normalizing the target per coordinate or shrinking the readout to match restores the recipe (Section 5). In Tables 1 and 2, no smaller nonzero r scored below a larger one, so landing at or below the standard ratio is enough. Finding the best small r would take a sweep; avoiding a large one does not.

On the learning rate, Section 4.3 and Table 3 test your question at large r, and there it cannot stand in for lowering r. At r = 4.52, scaling the readout's learning rate up or down 7.79-fold moves the collapse's end at most 1.64-fold, while lowering r by that factor moves it 26 to 30-fold. No learning-rate or Adam setting we tried reached the accuracy of lowering r (37.1 to 46.3% against 48.4% at step 4,000). Warmup lowers later accuracy and lets the collapse at r = 13 come back. Section 4.4 gives the reason: before the collapse ends, the readout's output has to shrink by about 3r to 4.5r times, so r sets the distance, while the optimizer's settings change only the pace. Inside the published range we have not compared the two, and the stakes there are small either way.

Beyond the check, most of the paper explains the collapse: what starts it, why one ratio orders how long it lasts in three frameworks and why the optimizer's settings do not. We hope that part is useful even where r is already small.

Thank you again for the careful review and the discussion.

Reviewer_fGBWOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates how the relative scale between a linear readout's initial output and its regression target affects representation collapse in self-supervised learning. Experiments on MelHuBERT show that larger (r) leads to longer collapse duration, primarily driven by a shared representation offset. The authors further provide gradient analysis and ablation studies to explain this behavior.

Critical strengths
  1. Useful mechanistic analysis. The gradient decomposition and comparisons across initialization scales, learning rates, and loss functions provide useful insights into the observed collapse.

  2. A simple and interpretable explanation of an important training phenomenon: The proposed readout-to-target ratio provides an intuitive way to characterize scale mismatch between model outputs and regression targets.

Critical weaknesses
  1. Incomplete mechanistic explanation: The gradient analysis explains the behavior at initialization but does not establish why different ratios lead to different recovery times.

  2. Limited practical validation: Most pronounced effects occur at unusually large ratios, while experiments near standard initialization scales are limited to short runs.

Questions for authors
  1. Can the authors explain how the initialization gradient mechanism determines the subsequent recovery time?

  2. Would readout matrices with identical (r) but different singular spectra or target alignments exhibit similar collapse behavior?

  3. Does the conclusion that initialization scale matters more than learning rate still hold over full training trajectories?

4 comments
AuthorsOpenReview ↗

Response to Reviewer fGBW

Thank you for the careful review and three precise questions. We agree with W1: the submitted analysis stopped at initialization. The new runs follow the mechanism through training and repeat the learning-rate comparison over full trajectories. As in the general response, the collapse ends when the shared mean's share of the representation's energy falls below 0.5 for three evaluations, and runs have three seeds unless stated.

W1, Q1: How does the initialization mechanism set the recovery time? It sets how far behind the target pull starts. The gradient reaching the encoder is Wᵀ(Wh + b − y): a target-free part that pulls each output toward zero, and −Wᵀy, which pulls it toward the target. Adam divides out the overall gradient scale, so scaling the readout and the target together leaves the encoder's updates unchanged (apart from the readout's relative step size, which matters little; see Q3), and the two scales act mainly through r. At initialization the target-free part dominates by a margin that grows with r (Section 4.4). It weakens as the output shrinks while the target pull does not, so the collapse lasts until the output has shrunk enough for the two to be comparable. A larger r leaves a larger gap to close. We tested each step:

TestResult
r = 4.52 vs 37.3 from the same initialization (one seed)The output's size agrees within 1% for 150 steps, then the r = 4.52 run separates as its target pull grows
A, the target pull divided by the zero pull in Adam's updateRises as the output shrinks; at the end of the collapse it is 0.57–1.68 in all 58 runs that recover
Forecast: 0.97 × the step where A's moving average reaches 0.7 (factor fitted on seed 0 of the main sweep)Within a factor of 1.5 in 17 of 18 held-out runs that recover (median error factor 1.20, against 1.38 and 2.00 for power laws in the current or initial r)
Switch r from 37.3 to 4.52 at step 200From then on its output size tracks a run started at 4.52 at the same step, and it recovers at 1,500–2,600 (started at 4.52: 1,800–2,100; left at 37.3: 11,000 or later)
At r = 37.3, remove the per-frame target-free part from the encoder's gradientThe output is still 6.6–7.0 times the target's size at step 1,000, against 1.4 normally; removing a frame-shuffled copy of the same size changes nothing

We have no closed form for the resulting curve (about 26-fold longer from r = 0.58 to 4.52, about 6-fold or more from 4.52 to 37.3), and in two settings the pulls balance but the collapse persists (the paper's γ = 2 target and unit-norm EMA targets; reply to LHtH). The revision states both.

Q3: Does the conclusion hold over full training? Yes. At r = 4.52 over the full 8,500 steps on 100 h:

ChangeCollapse ends at stepFinal probe
None (default)1,800–2,10047.2%
Readout lr ×7.79 / ÷7.791,100–1,700 / 2,000–2,50047.2 / 47.1%
Global lr ×0.5 / ×22,500–2,900 / not by 3,70044.8% / 37.1% (step 4,000; default 43.2%)
r ÷7.79 instead (r = 0.58)60–8051.4%

The same 7.79-fold change moves the end at most 1.7-fold through the readout's learning rate and about 26-fold through r. Neither direction of the global learning rate shortens the collapse; doubling it makes the collapse total (mean share near 1 for at least 1,000 steps). Readout lr ×7.79 and ÷7.79 at r = 37.3 finish on Monday, and we will add them here.

W2: Practical validation near the standard ratio. We agree the submission had only short runs there. We now have full-length runs at r = 0.58: 8,500 steps on 100 h (3 seeds) and 20,000 on 960 h (5 seeds). The collapse is brief there but present: it sets in between r = 0.2 and 0.25 and lasts a median of 80 steps at r = 0.58 (10 runs). Going below the standard ratio helps: a smaller readout (r = 0.04) never collapses and beats r = 0.58 in all 5 paired seeds at step 20,000 (57.1 vs 56.1%; 90% CI of the gain 0.53–1.35 points).

Large ratios are also less unusual than they look. Under PyTorch's default readout initialization, normalizing the target to unit norm per frame instead of per coordinate gives r ≈ √(m/3), about 16 at width 768. At r ≈ 16 a data2vec-style teacher is still collapsed at step 12,000, and at step 8,000 its phone probe is 11 points below the same teacher at its own scale (r ≈ 1.7). Even r = 4.52 trails r = 0.58 by 3.3–4.5 probe points at every checkpoint from step 8,000 to 20,000 on 960 h (3 seeds to step 12,000, then 2), long after its collapse ends.

Q2: Same r, different spectra or alignments. Six readouts at matched r = 4.52, 3 seeds each: flat or steep singular spectra, alignment with the encoder's strongest or weakest directions, and alignment or anti-alignment with the target through the least-squares readout. They finish on Saturday, and we will post the result here.

We hope this answers your questions, and we are glad to run further checks.

AuthorsOpenReview ↗

Follow-up: all three questions answered in the revision

Thank you again for the three precise questions. The revised paper is uploaded, rewritten for clarity, and each question now has a direct answer in it. Q1 changed the paper the most: Section 4.4 is now "How the collapse starts, and what sets its length".

Q1 / W1: the initialization sets how far the output must shrink (general comment; Section 4.4, new Figure 2). At initialization the target-free term is about r times the target term, and it shrinks the readout's output along the same path at every r. With our main target, no collapse ends before the output is down to about a third of the target's size, and from r = 1.6 up most end near a quarter. Since the output starts at r times the target's size, it has to shrink roughly 3r- to 4.5r-fold before the collapse can end, so the initial ratio fixes the distance and the shared path fixes the time. The general comment gives the evidence, including an intervention that cuts the distance mid-run.

The pace depends on Adam's memory of the large early gradients. At step 200 of an r = 37.3 run, shrinking the readout 8.25-fold lowers r exactly as raising the target 8.25-fold does, but it also cuts the encoder's gradient 34 to 38-fold while Adam's second moment still holds the old values: from step 100 to 2,000 the encoder's weights move 1.6 in norm, against 10.6 to 11.1 after the raise, and the collapse lasts about three times longer (Figure 3). With a shorter memory, β2 = 0.99, long collapses end sooner and the ends still follow r: 100 to 400, 900 to 1,100 and 7,300 to 7,700 steps at r = 0.58, 4.52 and 37.3 (two seeds at 37.3; Table 3). The general comment also corrects two rows of our first reply's table: our measure A tracks the end of the collapse but does not set it, and its forecast is rougher than we reported.

Q2 is answered in the general comment, since Reviewer LHtH asked the same: alignment with the target makes no difference, a steep spectrum about doubles the duration and alignment with the encoder matters most.

Q3: yes, now at r = 37.3 as well (Section 4.3, Table 3). Over 12,000 steps, a 7.79-fold faster readout ends the collapse in two of three seeds, at 7,600 and 7,300 steps against 11,100 and more than 12,000 for the same seeds by default, still 3.5 to 4.2 times as long as the default runs at r = 4.52. With a 7.79-fold slower readout all three seeds are still collapsed at 12,000. Phone accuracy at step 4,000 stays at 38.9 to 39.1% under either change (39.0% without), against 43.2% for runs started at r = 4.52. So at both ratios, over full training, the readout's learning rate moves the end far less than r and leaves accuracy where it is; Section 4.3's title now says so. Learning-rate warmup also keeps the order by r (general comment).

W2: full-length runs near the standard ratio (Section 4.1, Table 1). The sweep now has r = 0.14, 0.3, 1 and 1.6 beside 0.58, three seeds each. Collapses there last at most 600 steps, yet phone accuracy at step 8,000 falls steadily with r, from 52.9% at r = 0.14 to 49.5% at 1.6. That is the range where published regression recipes sit, 0.39 to 1.63 (Section 5, which now includes I-JEPA and V-JEPA). On 960 h, r = 0.04 still leads 0.58 at step 40,000 in all three pairs.

We would be glad to hear whether the revision now answers W1 and W2, and whether it changes your assessment.

Thank you to the authors for their detailed response, my concerns have been addressed. I like the research taste of this work. I would like to raise my score to 3.

After further consideration, I realize that some aspects of the paper fall outside my primary area of expertise. I would therefore like to lower my confidence score to better reflect my familiarity with the topic.

Thank you for following up, for raising your score and for the kind words about the work. We are glad the revision answered your questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of new results

We thank the reviewers for careful, specific reviews. They found r simple and interpretable (fGBW, LHtH), its ordering compelling and the offset and CKA analysis useful (LHtH), and the ablations comprehensive (wMmu). The new runs show four things: the ordering holds in MAE, in a data2vec-style teacher and on 960 h; downstream accuracy follows r, and its gap outlasts the collapse; learning-rate changes do not substitute for changing r; and every recovery happens when two gradient pulls balance.

In the collapse, one vector shared by all frames dominates the representation. Its mean share is the fraction of energy in the mean over frames, at block 5 of 12 (MAE: encoder output). It starts near 0.12 (speech), nears 1 at the peak of a full collapse, and the collapse is over once it falls below 0.5 for three evaluations in a row. Probe accuracy shows what the model learned: a linear phone probe on LibriSpeech dev-clean over all 12 layers, with standardized features so the shared vector cannot help or hurt (MAE: ImageNet-100 linear probe). Three seeds unless stated; step ranges span seeds, accuracies are seed means. All speech runs are new and continuous; the submitted ones restarted every 500 steps with the same seed, which kept comparisons across r fair but changed absolute durations.

Setting (asked by)rCollapse ends at stepProbe accuracy
MelHuBERT, LibriSpeech 100 h (LHtH W2)0.58 / 1.6 / 4.52 / 37.360–80 / 300–400 / 1,800–2,100 / 11,100 or later50.9 / 49.5 / 46.6 / 40.9% at step 8,000
MelHuBERT, 960 h (wMmu)0.04 / 0.37 / 0.58 / 4.52 / 37.3 (5 / 3 / 5 / 3 / 3 seeds)no collapse / 30–60 / 80–400 / 1,600–1,900 / about 11,000 or later54.2 / 53.6 / 53.1 / 49.5 / 43.0% at step 12,000
MAE ViT-B/16, ImageNet-100, official recipe at batch 512, 50 epochs (LHtH Q1, wMmu)0.89 (MAE's targets) / 24.8 (same targets times a constant)250 / 1,000; at 24.8, collapsed again from step 3,000–3,750 (linked to Adam's ε)44.0 / 30.0% after training
data2vec-style EMA teacher, MelHuBERT, 960 h (LHtH Q1)1.7 (data2vec's targets) / 16 (unit norm per frame)150–200 / not by 12,00055.3 / 44.3% at step 8,000

At step 20,000 on 960 h, r = 0.04 leads r = 0.58 in all 5 paired seeds (57.1 vs 56.1%; 90% CI of the gain, 0.53–1.35 points), and r = 4.52 trails r = 0.58 by 3.3–4.5 points at every checkpoint from step 8,000 (3 seeds to step 12,000, then 2). Near the standard ratio the collapse is brief but present: it sets in between r = 0.2 and 0.25 and lasts a median of 80 steps at r = 0.58 (10 runs).

At r = 4.52, 100 h, 8,500 steps (fGBW Q3, wMmu)Collapse ends at stepFinal probe
Default1,800–2,10047.2%
Readout lr ×7.79 / ÷7.791,100–1,700 / 2,000–2,50047.2 / 47.1%
Global lr ×0.5 / ×22,500–2,900 / not by 3,70044.8% / 37.1% (step 4,000; default 43.2%)
r ÷7.79 instead (r = 0.58)60–8051.4%

Why r sets the duration (LHtH W1, Q3; fGBW Q1; details in the LHtH reply). The gradient reaching the encoder is Wᵀ(Wh + b − y). Its target-free part (dominant at large r, Section 4.4) pulls every output toward zero; −Wᵀy pulls it toward the target. Adam divides out the overall gradient scale, so the readout and target sizes act mainly through their ratio r. At large r the pull toward zero wins, and the collapse lasts until the output shrinks enough for the target pull to catch up. A, the target pull divided by the zero pull in Adam's update, is 0.57–1.68 at that moment in all 58 runs that recover. Switching r from 37.3 to 4.52 at step 200 makes a run recover at steps 1,500–2,600, like runs started at 4.52, instead of after 11,000.

How the shared vector forms. A layer's weight gradient splits exactly into the summed output gradient times the mean input, which pushes every frame in the same direction, plus a per-frame part. Inputs and output gradients are not centered across frames, so the first part dominates. In one seed each at r = 0.58, 4.52 and 37.3, it is 83–99% of the gradient at initialization in the MLP and attention layers (keys aside), Adam's sign-like first steps turn it into one shared shift, and removing it largely prevents the shared vector over the first 20 steps.

Still open: a closed form for the duration curve, and two settings where the pulls balance but the collapse persists (the paper's γ = 2 target, unit-norm EMA targets). We have not trained text models.

Still running, posted as they finish: same-r readouts with different spectra or alignments (LHtH Q2, fGBW Q2) and 40,000-step small-readout runs (Saturday); r = 0, 0.14, 0.3, 1 and 13 (Sunday morning); MAE at r = 0.28, 2.8 and 8.9 (Sunday); readout lr at r = 37.3 (Monday).

Revision (by Monday, November 16): all results above, Section 4.4 rewritten to separate onset from duration, mean-removed rank and standardized probes beside raw ones, and smaller reporting fixes.

AuthorsOpenReview ↗

Revision uploaded: remaining results and corrections

We have uploaded the revised paper. Every run we listed as still running is in it, and the supplementary material holds the code and scripts that rebuild every table, figure and number. The sweep on 100 h now has nine ratios, three seeds each, from a zeroed readout to r = 37.3. Nothing collapses at r ≤ 0.14, and of the 188 pairs of runs whose order is clear, 183 end in the order of r (Section 4.1); the five exceptions all pair r = 1 with 1.6, whose ends overlap.

Why does the collapse last longer at larger r? (LHtH W1 and Q3, fGBW W1 and Q1; Section 4.4, new Figure 2, Appendix B) The gradient at the encoder's output has two parts: a target-free term that pulls the readout's output toward zero and a target term that pulls it toward the target. At initialization the first is about r times the second. Because the target-free term never sees the target, runs that share a seed and differ only in the target's scale shrink their output along the same path until the target's pull matters (within 1% of each other at step 100 for r from 4.52 to 37.3).

The end comes at nearly the same size, too: with our main target, a random readout and r held fixed, the output is at 0.22 to 0.33 of the target's size at all 43 ends from r = 1.6 to 37.3, on both datasets, with changed optimizer settings and with warmup. Each run's output starts at r times the target's size, so it shrinks roughly 3r- to 4.5r-fold before the collapse ends: r sets the distance, and the shared path sets how long it takes. Cutting the distance mid-run ends the collapse soon after: raising the target 8.25-fold at step 3,000 of an r = 37.3 run puts the output below a quarter of the new target's size at once, and the collapse ends 300 to 500 steps later. Why the end comes near that size is still open.

Our first replies explained the end by the balance of the two pulls, measured by A, the target's pull divided by the target-free pull in Adam's update. With every run in, A tracks the end but does not set it. At the 113 ends outside the warmup runs we added, it spans 0.39 to 1.68 (we reported 0.57 to 1.68 over 58), with 105 between 0.6 and 1.6. Under warmup the collapses end at A = 0.25 to 0.34, and in three settings A sits near one for thousands of steps while the collapse persists (Section 4.4).

Our forecast from A is also weaker than we reported: its 17 of 18 held-out runs within a factor of 1.5 of the end were the runs finished then. Over all 38 it does so for 79% (median error factor 1.28, still below 1.87 and 1.43 for power laws in the initial and the current r), and A's moving average reaches 0.7 before the end in only 53% (Appendix F).

Do readouts with the same r collapse alike? Not always (LHtH Q2, fGBW Q2; Section 4.5, Appendix E). The two random draws agree, and so do two readouts that differ only in their alignment with the target. With a steep spectrum the collapse lasts about twice as long, and alignment with the encoder changes it most. Seven readouts at r = 4.52, three seeds each:

ReadoutWeight norm (default = 1)Collapse ends at step
Default random draw1.01,800–2,100
Random orthogonal (flat spectrum)1.01,400–2,100
Steep spectrum (singular values ∝ 1/k)1.23,300–4,300
Least squares toward the target2.3not by 6,000
The same, negated2.3not by 6,000
On the encoder's 80 strongest directions0.3800
On its 80 weakest directions3.1not by 6,000

For the same output norm, readouts on weaker encoder directions need larger weights, as do both least-squares readouts, and the durations largely follow the weight norm. The revision therefore states the claim for readouts drawn the same way: among them, r orders the duration.

Learning-rate warmup, which we also added, does not make a large r safe. At r = 4.52 and 13 it makes the collapse shallower and shorter and keeps the order. But at r = 13 the collapse comes back in every run, and phone accuracy at step 12,000 falls (45.8% without warmup, 44.6 and 40.5% with it; Section 4.3). The other new runs are in the threads that asked for them: MAE at five ratios (LHtH), the readout's learning rate at r = 37.3 (fGBW) and r = 0.04 to step 40,000 (wMmu).

What else changed. Section 4.2 now holds the CKA analysis, Section 5 adds I-JEPA and V-JEPA (r = 0.39) and Section 6 adds the closest prior work (Kumar et al., 2024; Kosson et al., 2024; Atanasov et al., 2025). Appendix G keeps the submission's 500-step and softmax runs. The five tests of Appendix F, registered before the new runs (recorded in the supplementary material), replace the submission's list of predictions. The AI use statement now covers every use declared in the submission form. The text is revised throughout; the title, section order and central claim are unchanged. Where our first replies differ from the revision, the revision is right.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.