acceptodds
Under review as a conference paper at ICLR 2027

When Feature Rank Misses Model Collapse: Class Geometry in Recursive Classifiers

Abstract

Global feature rank can rise even when within- and between-class covariance components both become spectrally more concentrated. We study this behavior in image classifiers recursively retrained on generated images and inherited pseudo-labels. In a three-seed CIFAR-10 study with the original ResNet stem, four replacements increase centered-covariance effective rank by 14.5% and within-to-between-class scatter 3.22-fold, while accuracy falls from 88.4% to 62.2%. Exact entropy accounting shows how variance reweighting conceals concentration of both components. Later generations lose another 9.28 accuracy points while rank rises 3.30%; fixing the initial teacher avoids this additional loss on identical image pools. A standard CIFAR stem raises baseline accuracy to 95.4%, yet retains the full-trajectory mismatch: accuracy reaches 56.2% while rank increases 2.6%. Its later rank changes vary by seed. Alternative spectral definitions also give different directions, and GAN and ImageNet-100 diffusion workflows lose rank as geometry deteriorates. Disjoint subspace and frozen-feature readout analyses corroborate degraded class structure. Low-label audits estimate geometry but do not establish better warning or intervention decisions. These findings identify what increasing covariance rank can conceal, while showing why its interpretation depends on class-conditioned spectra, training stage and predictive performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
model collapse, synthetic data, recursive training, representation geometry, effective rank, pseudo-labeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F74LOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies a setting where image classifiers are trained recursively on synthetic data (pseudo labels from classifiers of previous iteration): a fixed pretrained generator (DDPM, GAN or DiT) produces the synthetic images. The authors study the effective rank of the final layer embeddings as an analytical measure, together with accuracy and a within/between class “scatter ratio”. A key observation is that in the CIFAR-10 + DDPM setting, global feature effective rank increases while both within and between class effective ranks fall. They derive mathematical decompositions of the change in log effective rank including two component spectrum terms, and show that the reweighting between the two components can be associated to this behavior. The paper also includes extensive experimentation on training stage, teacher, input stem, rank definition, generators, small labeled audit sets.

Critical strengths
  • Observation that global feature effective rank can rise even when both within and between class effective ranks fall.
Critical weaknesses
  • The paper is quite hard to follow, frequently mentions (even in the abstract/introduction sections) findings or terminology that has either not been explained/cited, or only introduced later in the paper.
  • The setup of measuring feature rank over recursive training of classifiers seems to be newly proposed, with no existing citation, therefore significance is limited.
  • Keeping the generator fixed is not the typical setting to measure “model collapse” in the literature.
  • The authors only observe this surprising behavior of feature rank not agreeing with classification in the CIFAR-10/DDPM setting. This limits the significance of this work further.
  • Much of the paper is spent for experimental results that do not follow this novel observation.
  • The fixed-teacher control does not quite make sense to me, it’s training with the same generator and same C0 for labeling - so it’s just increasing the training data, not sure if this can be associated with “fixing the initial teacher” as mentioned in the abstract.
Questions for authors
  • Is there any prior work that uses classifier feature rank to monitor recursive or training with pseudo labels?
  • Is there any combination other than CIFAR-10 with DDPM in the proposed recursive setting where we observe covariance rank to rise while classification performance falls?
4 comments
AuthorsOpenReview ↗

Response to Reviewer F74L

Thank you for the careful reading.

Readability (W1, W5). The introduction (L35–115) now defines the training loop, WW and BB, RR and LiDAR before the main quantitative results, and terms in the abstract and introduction, such as the first switch and the Fisher criterion, are defined where they first appear. Internal shorthand ("replacement", "original stem") was replaced, and "fresh", "historical" and "post-outcome" runs are defined in Section 5.1 (L253–288). To keep the main text focused on the main observation, the evidence map and the stem tables moved to the appendix, the new Section 5.5 (L404–456) tests the observation against LiDAR and class-aware statistics, and the conclusion (L513–538) collects the limitations in one paragraph.

Fixed-teacher control (W6). The original description was unclear. In both chains, each generation trains on exactly 50,000 generated images and nothing accumulates, so the training set does not grow. The control takes the same generated images of generations 2–4 and labels them with the initial classifier C0C_0 instead of the previous classifier. Inputs, budget, initialization and recipe stay fixed (Section 5.3, L320–366). From generation 1 to 4, accuracy falls 9.28±1.17 points under inheritance and rises 0.67±0.10 points with C0C_0 labels (Table 2, L281).

Significance and setting (W2, W3). Effective-rank scores such as RankMe and LiDAR are used to select or monitor models without labels. Recursive pseudo-labeling is a setting where labels are missing and models degrade, and in this setting these scores can rise while accuracy falls (Section 5.5, Table 3, L379). In an exploratory comparison, selecting the generation with the highest LiDAR would lose 24.2 accuracy points on CIFAR-10 with DDPM images and 16.0 on real images (7.3 and 9.1 among generations 1–4), whereas the Fisher criterion computed with true labels selects generation 0, the best one, in all six chains (Appendix F.5, L2696–2779). We agree that a fixed generator differs from model collapse in the usual sense. The revision states this scope, reports that joint retraining lowers rank together with accuracy after an initial rise while the generator's FID rises from 132 to 323 (Section 5.9, L501–511; Appendix F.4, L2643–2693), and adds a setting without any generator (Q2 below).

Q1. Prior work. We are not aware of prior work that validates feature rank for this loop. Feature rank is used to assess representations in related settings: RankMe and LiDAR select self-supervised models without labels, and in continual and reinforcement learning a falling feature rank is read as lost plasticity or capacity (Kumar et al., ICLR 2021; Lyle et al., ICLR 2022; Dohare et al., Nature 2024). The revision cites these works in the introduction. When each generation fine-tunes its predecessor, as in continual training, covariance rank and LiDAR still rise while accuracy falls (Appendix F.3, L2581–2640).

Q2 (W4). Other combinations. Yes, within CIFAR-10. Covariance rank and LiDAR rise while accuracy falls, in all three seeds, in these settings:

  • Real unlabeled CINIC-10 images, with the same C0C_0 and test images: rank +26.9±2.6%, LiDAR +68.5±6.1%, accuracy −16.0 points. Most of this rise occurs at generation 1, the first trained on these images. Over the inherited generations 1–4, rank still rises 3.0±2.0% and LiDAR 15.6±0.7% while accuracy falls 9.1 points (Section 5.2, L290–318; Appendix F.2, L2526–2578).
  • Label noise without recursion: with 40% of the CIFAR-10 training labels flipped, rank +98.8±7.4%, LiDAR +90.8±16.2%, accuracy −25.1 points (Appendix F.3).
  • Real CIFAR-10 images labeled by the generation-1 classifier, without generated images: rank +23.3±3.7%, LiDAR +51.9±4.7%, accuracy −16.3 points (Appendix F.3).
  • An ImageNet-pretrained ViT-S/16 (98.6% at generation 0) on the same DDPM images: rank +35.9±4.9%, LiDAR +37.4±3.4%, accuracy −3.8 points by generation 4 (Appendix F.6, L2782–2822).

Among the fresh generated-image chains, covariance rank rises only with DDPM images, while LiDAR also rises with StyleGAN2 images. We do not claim that rank always rises; the claim is that, without labels, its direction does not indicate whether accuracy rises or falls.

Thank you for the changes to the paper.

Readability: I think the paper is much easier to read. I believe the readability can still be improved to convey the novelty and significance of the work more clearly, especially in the abstract and introduction. The abstract still feels like a list of experiments instead of a coherent finding. Introduction has too many references to sections now, I believe introduction should be more self-contained and convincing. They should make clear the key contribution as it stands, which I believe to be that the direction of rank does not indicate whether accuracy rises or falls.

Fixed-teacher control: I appreciate the clarification on fixed teacher control, that makes sense and my initial understanding was incorrect.

Other combinations: I appreciate the additional experiments on real images with CINIC-10, in general, I agree with your assessment, of "its direction does not indicate whether accuracy rises or falls", which leads me to the next point and final assessment.

Significance, prior work: I believe this is the main weakness of this paper now. The recursive pseudo-labeling setting being constructed seems fair on its own but it is not a standard practical setting. The key contribution therefore becomes a critique of the effective rank and LiDAR, however, these scores are typically used to select models in the standard self-supervised learning, which is different from the specific setting in this paper. In your Response to Reviewer ckdZ's Follow-up Comment , you justify as "It was an open question whether LiDAR would avoid the mismatch." and "None of the label-free statistics we tested follows accuracy in every chain, so none can replace a labeled check.". I think if this is the main takeaway, these claims deserve much more emphasis and a more careful study to make a strong paper, for example, testing these scores in the self-supervised setting they were actually designed for, going beyond CIFAR-10, and comparing them against other standard label-free accuracy estimators.

I will raise my score from 1 to 2, however, I think this paper still leans reject.

AuthorsOpenReview ↗

Response to Reviewer F74L's Follow-up Comment

Thank you for reading the revision and for the updated assessment. Your statement of the contribution, that the direction of rank does not indicate whether accuracy rises or falls, is the one we intend. We will restructure the abstract and introduction around it, with fewer pointers to sections.

On the setting. The mismatch is not tied to the recursive loop. In ordinary supervised training on CIFAR-10 with 40% of the labels flipped, covariance rank rises 98.8±7.4% and LiDAR 90.8±16.2% while accuracy falls 25.1 points (Appendix F.3, L2581–2640). The same control on CIFAR-100, run during this discussion with the plan fixed beforehand, gives the same result: accuracy falls 25.2±0.2 points while covariance rank rises 85.9±0.5% and LiDAR 12.5±0.4%, in all three seeds.

On where rank is used. Feature rank is also used outside self-supervised learning. In continual and reinforcement learning, a falling rank is read as lost capacity (Kumar et al., ICLR 2021; Lyle et al., ICLR 2022; Dohare et al., Nature 2024). Our results show that the converse reading, a higher rank as a better classifier, does not hold in the settings above.

On other label-free accuracy estimators. We have now run the comparison you suggest, on the stored outputs of eight chains of the revision, three seeds each. It is exploratory. The PDF is unchanged and does not contain this comparison or the CIFAR-100 control.

  • Mean confidence and ATC (Garg et al., ICLR 2022) rise over generations 1–4 on CIFAR-DDPM (+1.1±0.1 and +2.6±0.4 points, in all three seeds) while accuracy falls 9.3 points. They also rise over ImageNet-100 generations 1–5.
  • The agreement of each classifier with the initial classifier's predictions needs no labels. Wherever accuracy falls in all three seeds, at the first switch or over the later generations of a chain, agreement falls in all three as well.

This qualifies the sentence you quote from our reply to Reviewer ckdZ: agreement with the initial classifier, tested since, does follow accuracy in every chain. It uses class predictions rather than feature rank, and it adds a positive recommendation to the negative result.

On what remains. Testing the scores in self-supervised training, the setting they were designed for, is the extension we plan next. The present claims concern supervised classifiers.

Thanks. If not tied to recursive loop that needs some clear phrasing in the paper again. The paper should not be presented as "We study recursive pseudo-labeling with a fixed generator", but rather a claim about rank can rise with poor data quality, which is aligned with findings in previous work (Ma et al., 2018, that you cite). The CIFAR-100 result can be included in a revision.

re: where rank is used, IMO falling rank for lost capacity studies something different than higher rank vs accuracy.

interesting that the agreement with the initial classifier follows accuracy with labels.

These are all interesting ideas and can be made a stronger work with more emphasis and experimentation.

Reviewer_ckdZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies image classifiers that are trained recursively on synthetic images. The generator is fixed (DDPM or StyleGAN2-ADA on CIFAR-10, DiT on ImageNet-100), and at each generation the previous classifier labels the new images. The authors measure the effective rank of the penultimate feature covariance, the scatter ratio R=tr W/tr BR=\mathrm{tr}\,W/\mathrm{tr}\,B, and accuracy on real test data. The main finding is that on CIFAR-10 with DDPM, the rank goes up by 14.5% while accuracy drops from 88.4% to 62.2% and RR gets 3.2 times worse. Both the within-class and between-class ranks go down, and the authors use an entropy decomposition to show that the total rank goes up because more of the variance moves to the within-class part. They also run a fixed-teacher control, a different CIFAR stem, other rank definitions, real data mixing, and some monitoring experiments, which do not show a clear advantage over just using accuracy.

Critical strengths
  1. The experiments are done carefully and the paper is very transparent. The splits are disjoint, post-hoc analyses are clearly marked, and the results are independently verified.
  2. The fixed-teacher experiment and the check in App. C (geometry computed with the model's own predicted labels looks better than with true labels) are useful. But both confirm things we already know, so in my opinion they are not enough for acceptance.
Critical weaknesses
  1. Novelty. The main message of the paper is that covariance rank does not see labels, so it mixes useful (between-class) and useless (within-class) variance, and you need the W/B split to interpret it. This was already discussed in LiDAR (Thilak et al., ICLR 2024), which builds a class-aware rank from exactly this split, using surrogate classes instead of real labels. The paper does not cite LiDAR and does not compare with it or with any other class-aware metric. The math is standard (Fisher's W+B decomposition, the Jensen-Shannon mixing term, a product rule between two endpoints). So what is left is an empirical observation in one setting, and I don't think the paper shows clearly what we learn from it.

  2. The main finding is not surprising and I don't see how it can be used. For a supervised classifier nobody expects higher rank to mean a healthier model; this intuition comes from SSL (RankMe). In fact the rank increase is mostly the scatter increase written in a different way: Dp=+0.267D_p=+0.267 alone is bigger than the total change +0.136+0.136 (Table 8). The effect is also not stable. With the standard CIFAR stem it is only +2.6%, in later generations it goes down in two of three seeds, and other rank definitions give the opposite sign (Table 4). The mitigations tested (fixed teacher, keeping real data) are already known, and the geometry-based interventions do not give a consistent advantage over accuracy or over switching at a fixed generation.

Questions for authors
  1. How is this work different from LiDAR (Thilak et al., ICLR 2024)? What happens if you compute a class-aware metric with the real labels on your trajectories, for example the regularized LDA rank erank((W+δI)−1/2B(W+δI)−1/2)\mathrm{erank}\big((W+\delta I)^{-1/2}B(W+\delta I)^{-1/2}\big) or the Fisher criterion tr(W−1B)\mathrm{tr}(W^{-1}B)?
  2. Do you see the same effects if you run the same loop on real unlabeled images instead of generated ones?
  3. In the main experiments the generator is fixed, so this is closer to iterated pseudo-labeling than to model collapse in the usual sense. Wouldn't a different title fit better? And how does the rank drop in the joint retraining experiment (§5.8) fit your message?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ckdZ [1/2]

Thank you for pointing us to LiDAR and for proposing the class-aware comparison and the real-image test. LiDAR assesses self-supervised representations without labels; we ask whether such label-free scores track accuracy when classifiers learn from their predecessors' labels. LiDAR's surrogate classes are images, so differences between images of the same task class count as signal. On CIFAR-10 with DDPM images (CIFAR-DDPM), after the first switch from real to generated training images, LiDAR rises 11.3±1.4% over generations 1–4 while linear-probe accuracy, which LiDAR was designed to predict, falls by 7.08±1.04 points. With true labels, the Fisher criterion falls with accuracy.

W1/Q1. LiDAR and class-aware metrics. The revision cites LiDAR. On all trajectories of Table 3 (L379; three seeds; Section 5.5, L404–456), we compute the quantities you proposed with δ=10−3,tr(W)/d\delta = 10^{-3}\\,\mathrm{tr}(W)/d: the regularized LDA rank and the Fisher criterion tr((W+δI)−1B)\mathrm{tr}((W+\delta I)^{-1}B), each with true labels and with the classifier's predictions. We also compute LiDAR itself, with 2,000 evaluation images and 16 training-augmentation views each, identical across generations.

LiDAR's surrogate-class split does not prevent the mismatch:

  • LiDAR rises 56.5±5.7% on CIFAR-DDPM while accuracy falls from 88.4% to 62.2%. It also rises with StyleGAN2 images (+19.1±2.4%), with 50% real data (+24.3±2.2%), on real unlabeled images (+68.5±6.1%, see Q2) and with an ImageNet-pretrained ViT-S/16 (+37.4±3.4%, Appendix F.6, L2782–2822). On ImageNet-100 it falls with accuracy (−3.5±0.7%).
  • The linear probes above are ridge-regularized classifiers refitted on the frozen features of the inherited generations 1–4. With fixed C0C_0 labels over the same generations, LiDAR changes by 2.0±3.4%.
  • Without generated images or recursion, LiDAR also rises under label noise: with 40% of the CIFAR-10 training labels flipped, it rises 90.8±16.2% (13.1±10.3% with SimCLR-style views) while accuracy falls 25.1 points, in all three seeds (Appendix F.3, L2581–2640).

LiDAR's surrogate classes. LiDAR's between-class matrix is the covariance of per-image mean features. With the true labels of the same 2,000 images, this matrix splits exactly into between- and within-class parts. By generation 4 on CIFAR-DDPM, the within-class part grows from 27.5% to 57.3% of its trace (67% to 87% after LiDAR's whitening), and it grows in all five CIFAR-10 chains and all three seeds (exploratory, Appendix F.5, L2696–2779). On ImageNet-100 the share also rises (from 57% to 78%), yet LiDAR falls 3.5%: as with covariance rank, the shift toward within-class spread alone does not determine whether LiDAR rises or falls. With the task classes in place of the image surrogates, the same entropy rank, the LDA rank, falls on CIFAR-10 (−11.4±0.5% on CIFAR-DDPM). On ImageNet-100 it is nearly unchanged at this ridge (+0.1±0.7%) and falls 7.4±0.6% at δ=10−1,tr(W)/d\delta = 10^{-1}\\,\mathrm{tr}(W)/d.

With true labels, the Fisher criterion tracks accuracy over every interval of Table 3 (e.g., −73.2±1.1% on CIFAR-DDPM, −63.2±0.6% on ImageNet-100) and under label noise (−71.8±1.4% with 40% of the labels flipped, −58.9±0.8% with 20%). Under label noise, however, the true-label LDA rank rises in all three seeds (+2.8±2.1% and +6.2±0.9%). The scatter ratio RR, which also uses the true classes, improves while accuracy falls in two runs (Section 5.2, L290–318): a weaker historical ConvNeXt and a one-step ViT-S control trained on DiT's conditioning labels. We did not compute the Fisher criterion for these runs. In Table 3, every statistic that moves against accuracy in all three seeds is computed without the true classes. Prediction-labeled statistics are less reliable in later generations: their Fisher criterion rises 1.6±1.4% over ImageNet-100 generations 1–5 while accuracy falls 4.9 points.

Most of LiDAR's gain on CIFAR-DDPM (40.6±4.0%) comes at the first switch. With SimCLR-style views the rise shrinks to 7.7±5.0%, but it remains positive in all three seeds while accuracy falls 26 points. We use the standard covariance identity only to quantify which spectral changes accompany these trajectories. Our contribution is an empirical test of rank-based monitoring on recursive classifiers: LiDAR rises while refitted linear probes lose accuracy, including after the first switch, and a growing share of its between-image scatter lies within the true classes (exploratory).

AuthorsOpenReview ↗

Response to Reviewer ckdZ [2/2]

W2. Interpretation, stability and practical value.

  • Feature rank is also used outside SSL to assess representations: in continual and reinforcement learning, a falling rank is taken as a sign of lost plasticity or capacity (Kumar et al., ICLR 2021; Lyle et al., ICLR 2022; Dohare et al., Nature 2024). In recursive pseudo-labeling, where each generation trains on its predecessor's pseudo-labels without ground-truth labels, both covariance rank and LiDAR can rise while accuracy falls. The same holds with sequential fine-tuning: when each generation fine-tunes its predecessor, rank rises 14.0±1.4% and LiDAR 58.0±4.5% by generation 4 while accuracy falls 21.1 points, in all three seeds (Appendix F.3, L2581–2640). This control measures accuracy rather than plasticity; it shows that a rising rank does not rule out falling accuracy.
  • The reweighting term is large on CIFAR-DDPM (DpD_p = +0.267 vs a total of +0.136 nats), but scatter alone does not fix the sign. On ImageNet-DiT, DpD_p is nearly the same (+0.270), yet total log rank falls (−0.408) because the within-class spectrum concentrates (DWD_W = −0.470; Table 8, L1135).
  • We agree that the direction of covariance rank depends on the input stem and the spectral definition (Section 5.6, L458–475), so the claim is only that a rising rank cannot be read as improvement. LiDAR's direction is more consistent across the tested settings: from generation 0 to 4 it rises in all three seeds of the five CIFAR-10 chains and of the ViT-S/16 chain. It also rises under label noise (W1/Q1 above) and falls on ImageNet-100.
  • Use. Selecting the generation with the highest LiDAR loses 24.2 accuracy points on CIFAR-DDPM and 16.0 on real images, whereas the true-label Fisher criterion selects generation 0 in all six chains (exploratory, Appendix F.5, L2696–2779). Restricted to generations 1–4, LiDAR's choice still loses 7.3 and 9.1 points. The Fisher criterion computed with the classifier's own predictions, which needs no labels, also selects generation 0 in all six chains but misses the later ImageNet-100 decline. We therefore recommend validating any label-free monitor against labeled accuracy before relying on it.
  • As you note, the mitigations are known, and our intervention tests do not establish an advantage over accuracy or over switching at a fixed generation (Section 5.8, L489–499; Appendix F).

Q2. Real unlabeled images. Yes. We ran the same loop on real unlabeled images, with the plan frozen before training. Starting from the same C0C_0, at each generation the preceding classifier labels 50,000 fresh ImageNet-derived CINIC-10 images of the CIFAR-10 classes, with CIFAR-10-derived files excluded (Section 5.2, L290–318; Appendix F.2, L2526–2578). By generation 4, accuracy falls from 88.4% to 72.4% while covariance rank rises 26.9±2.6% and LiDAR 68.5±6.1%, and the true-label Fisher criterion falls 61.8±0.4%. Most of this rise occurs at the first switch, with either label source: with CINIC-10's provided labels on the same images, generation 1 also raises rank 19.7±1.7% over C0C_0, whereas the same retraining on CIFAR-10 images leaves rank nearly unchanged. Over the inherited generations 1–4, accuracy falls a further 9.1±0.6 points while rank rises 3.0±2.0% and LiDAR 15.6±0.7% (+3.9±1.3% with SimCLR-style views), in all three seeds.

Q3. Title and joint retraining. We agree. The revised abstract (L11–32) and introduction (L35–115) state the scope as recursive pseudo-labeling, and we propose "Effective Rank versus Class Separation in Recursive Pseudo-Labeling" as the title of the final version. In joint retraining, rank first rises at generation 1, which still uses the pretrained generator, and then falls together with accuracy (Section 5.9, L501–511), while the generator degrades: FID against the CIFAR-10 training images rises from 132 to 323 by the fifth generator round (Appendix F.4, L2643–2693). Since rank can also rise while accuracy falls, its direction alone does not show whether the classifier is degrading.

Reviewer_ckdZOpenReview ↗

Reply to authors

Thanks for the detailed response and the revised paper. I think the revision is much clearer now, the scope is recursive pseudo-labeling which makes more sense, LiDAR is discussed and evaluated, and the CINIC-10 and label noise experiements answer my Q2. Also good that the new experiments were pre-registered.

But my main concern is still the significance, and honestly the new results make it more clear for me. In CINIC-10 the rank goes up at the first switch even with the provided labels (+19.7%), so this rise comes from changing the image source and not from the psudo labels or the recursion. And with label noise the rank basically doubles, which you also say agrees with Ma et al. (2018). So the story is: label free spread statistics react to domain shift and label noise and not to accuracy, and the separation statistics follow the accuracy. this is also in line with Xie et al. (2023) where the distance between class centers predicts accuracy and how close samples are to their own class mean dosent. Given these I still find the main conclusion expected. The recommendation (track accuracy on a labeled set and use true label separation stats to interpret geometry) is reasonable, but i dont think its a strong enough contribution by itself.

I agree spectral diagnostics are being proposed for monitoring model collapse, so a careful negative result has some value, and the paper is more sound now. Given all these points, I prefer to keep my overall rating.

AuthorsOpenReview ↗

Response to Reviewer ckdZ's Follow-up Comment

Thank you for the follow-up and for acknowledging the changes. We agree with your summary of the controls; the revision says the same (Appendix F.2, L2526–2578).

We would add one remark on whether the conclusion was expected. The distinction between spread and separation does not tell us which side a label-free statistic falls on. LiDAR is built from a between-class/within-class split, and before the test it was an open question whether its surrogate classes would avoid the mismatch. They do not: LiDAR rises in all five CIFAR-10 chains while accuracy falls, and by generation 4 on CIFAR-DDPM, 57.3% of what it counts as between-class scatter lies within the true classes (exploratory, Appendix F.5, L2696–2779). The dispersion score of Xie et al. (2023) also takes its classes from the model's predictions. On ImageNet-100, the Fisher criterion computed with predicted labels rises 1.6±1.4% over generations 1–5 while accuracy falls 4.9 points (Section 5.5, L404–456).

None of the label-free statistics we tested (covariance rank, LiDAR, and the LDA rank and Fisher criterion with predicted labels) follows accuracy in every chain. Recursive pseudo-labeling is used when labels are scarce, so we think it is useful to know that none of them can yet replace a check on labeled data in this setting.

Reviewer_B5KfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of global feature effective rank as an indicator of model collapse in recursively trained image classifiers. The authors show that effective rank can increase even as classification accuracy and class separability deteriorate. Through a covariance decomposition into within-class and between-class components, they explain how variance reweighting can mask spectral concentration. Experiments across multiple datasets, generative models, and training configurations validate these observations and highlight the importance of class-conditioned geometry.

Critical strengths
  1. Interesting and counterintuitive findings. The paper reveals that global effective rank can increase despite substantial performance degradation, challenging its interpretation as a reliable indicator of representation collapse.

  2. Rigorous mathematical analysis. The covariance decomposition and exact entropy accounting provide a clear mathematical explanation for how global rank can increase even when within-class and between-class components become more concentrated.

  3. Extensive empirical analysis. The paper investigates multiple datasets, generative models, classifier architectures, and training configurations, including fixed-teacher controls and alternative rank definitions, providing useful evidence about when the observed phenomenon occurs.

Critical weaknesses
  1. Limited novelty. Effective rank is not inherently correlated with classification accuracy, and prior work has already explored the limitations of rank-based metrics and the importance of class geometry [1–3]. The paper's incremental contribution remains unclear.

  2. Questionable training setup. The main experiments repeatedly train classifiers from scratch on synthetic data with inherited pseudo-labels. The relevance of this setting to realistic continual training is insufficiently justified.

  3. Limited mechanistic understanding. The entropy decomposition is descriptive rather than causal. The effects of pseudo-label errors, synthetic data distribution shifts, and optimization are not adequately disentangled.

  4. Limited generalizability. The rank-increase phenomenon is primarily observed with ResNet-18 on CIFAR-10/DDPM. Results vary across architectures and settings, while the ViT baselines are relatively weak.

  5. Unclear practical value. Class-geometry analysis requires labeled data but does not consistently outperform accuracy-based monitoring in detecting degradation or guiding interventions.

[1] Self-Supervised Anomaly Detection in the Wild: Favor Joint Embeddings Methods. 2024.

[2] Feature Learning in Deep Classifiers through Intermediate Neural Collapse. ICML 2023.

[3] Do Coresets, Pruning, and Quantization Preserve Neural Network Representations? 2026.

Questions for authors
  1. Does the rank–accuracy mismatch also occur under conventional supervised training, label noise, or distribution shifts without recursive training?

  2. Does the phenomenon persist when classifiers are continually fine-tuned or historical training data are retained?

  3. Can the authors compare inherited pseudo-labels with generator-conditioning labels to better isolate the sources of degradation?

  4. Could the authors provide FID scores, representative generated images across generations, and diversity statistics, particularly for joint generator–classifier retraining?

  5. Can the authors validate the findings using stronger ViT baselines and additional random seeds?

4 comments
AuthorsOpenReview ↗

Response to Reviewer B5Kf [1/3]

Thank you for the concrete questions, which we address with new experiments or analyses. The post-review experiments were pre-registered and use three seeds. For example, an ImageNet-pretrained ViT-S/16 that starts at 98.6% accuracy loses 3.8±0.3 points over four generations while covariance rank rises 35.9±4.9% and LiDAR 37.4±3.4% (W4).

W1. Novelty and [1]–[3]. The revision cites all three and the dispersion score (Xie et al., 2023). [1] finds RankMe unreliable for anomaly detection, [2] shows within-class covariance shrinking relative to between-class covariance across layers, [3] combines centered-activation rank and class separation to evaluate compressed networks, and the dispersion score links inter-class dispersion to accuracy under shift. None studies recursive retraining without labels. In continual and reinforcement learning, falling feature rank is read as a sign of lost plasticity or capacity (Kumar et al., 2021; Lyle et al., 2022; Dohare et al., 2024). Our contributions in this setting:

  • Label-free rank monitors can rise while accuracy falls, with generated and real unlabeled images. This includes LiDAR, which already splits variance by surrogate classes. A growing share of its between-image scatter lies within the true classes (exploratory, Appendix F.5, L2696–2779).
  • With true labels, the Fisher criterion tracks accuracy over every interval of Table 3 (L379).
  • An exact entropy accounting of the covariance-rank changes shows how total rank rises while both component spectra concentrate (Section 4, L211–249).
  • Controls with a fixed teacher, label noise, real-image pseudo-labels and sequential fine-tuning (Section 5.3, L320–366; Appendix F.3, L2581–2640; see W3).

W2. Setting. With continual training, where each generation fine-tunes its predecessor, the mismatch remains. After the first switch from real to generated training images, generations 1–4 lose 11.1±0.5 accuracy points while covariance rank rises 7.0±1.5% and LiDAR 15.3±3.3%, in all three seeds (Appendix F.3; generations 0 to 4 in the table under W3). A fixed generator isolates the classifier side of recursion, where ground-truth labels are unavailable and label-free monitors would be used. Without a generator, the same loop on real unlabeled images (ImageNet-derived CINIC-10) shows the same mismatch (Section 5.2, L290–318). Real-data retention and joint generator–classifier retraining are in Section 5.9 (L501–511).

W3. Mechanism. We agree that the decomposition is descriptive, and we do not claim a causal mechanism. The new controls show that the mismatch occurs without generated images or recursion, and the fixed-teacher control shows that fixing the teacher avoids the later loss on the same DDPM images. Within the recursive chain, they do not identify each factor's contribution. Each row reports the change from C0C_0, trained on the real CIFAR-10 images with their true labels, to a model trained on the listed data, where C1C_1 is the generation-1 classifier (three seeds; Appendix F.3):

Training dataAccuracy (points)Covariance rankLiDARFisher, true labels
Real images, 40% of labels flipped−25.1±0.3+98.8±7.4%+90.8±16.2%−71.8±1.4%
Real images, 20% of labels flipped−11.0±0.5+86.8±3.5%+79.0±14.5%−58.9±0.8%
Real images, labels from C1C_1 (73% correct)−16.3±0.6+23.3±3.7%+51.9±4.7%−59.7±0.2%
CINIC-10 images, labels CINIC-10 provides−16.7±0.3+19.7±1.7%+56.5±4.2%−56.6±0.6%
DDPM images, C0C_0's labels (first switch)−16.9±0.3+10.9±2.4%+40.6±4.0%−62.4±0.4%
DDPM images, inherited labels, generation 4−26.2±1.0+14.5±2.3%+56.5±5.7%−73.2±1.1%
DDPM images, inherited labels, fine-tuning the predecessor, generation 4−21.1±0.5+14.0±1.4%+58.0±4.5%−66.7±1.2%
  • Label errors: without generated images, training on real images with 20% or 40% flipped labels or with C1C_1's labels reproduces the mismatch in all three seeds (rows with real images).
  • Image source: with the same labeler C1C_1 and training seed, training on the real images gives 5.2±0.5 points higher accuracy than training on the DDPM images of generation 2, and covariance rank is also higher (+10.6±8.3%). The real-image labels are also less imbalanced (largest class 13% versus 38%), so this comparison does not separate image source from label imbalance.
  • Label inheritance: after the first switch, accuracy falls a further 9.28±1.17 points under inherited labels and rises 0.67±0.10 with C0C_0 labels on the same images (Table 2, L281).
  • Class imbalance: in the CINIC-10 chain, pseudo-labels are much less imbalanced than the DDPM labels (largest class 12.6–13.5% versus 37–41%), yet rank still rises over generations 1–4, so DDPM-level imbalance is not needed; within the DDPM chains the evidence is mixed (Section 5.3).
AuthorsOpenReview ↗

Response to Reviewer B5Kf [2/3]

W4. Generalizability. We revised the claim: without the true classes, effective rank measures spectral spread, and its direction does not indicate whether accuracy is rising or falling.

  • Covariance rank rises on CIFAR-10 with DDPM images (CIFAR-DDPM), in all six seeds of the fresh and historical cohorts, and on real unlabeled CINIC-10 images, but falls on CIFAR-10 StyleGAN2 and ImageNet-DiT, while RR worsens in all four.
  • From generation 0 to 4, LiDAR rises in all three seeds of the five CIFAR-10 chains, including the StyleGAN2 chain, whose pools have FID 2.5 versus 131.6 for the DDPM pools (Q4 below). It falls on ImageNet-100.
  • The mismatch needs neither a generator nor recursion: under label noise, covariance rank and LiDAR rise while accuracy falls (table under W3; Appendix F.3, L2581–2640).
  • With a standard CIFAR stem, rank still rises 2.60±1.08% over the full trajectory, but over generations 1–4 it falls in two of three seeds.
  • The mismatch also occurs with an ImageNet-pretrained ViT-S/16, which reaches 98.6% at generation 0 (pre-registered, Appendix F.6, L2782–2822). On the same DDPM images, accuracy falls 3.8±0.3 points by generation 4 while covariance rank rises 35.9±4.9% and LiDAR 37.4±3.4%, in all three seeds, and the true-label Fisher criterion falls 53.3±1.4%. Most of the rank rise occurs at the first switch. Over generations 1–4, LiDAR rises 7.1±0.8% in all three seeds, while covariance rank is mixed (+1.0±3.3%). As in the ResNet chain, the pseudo-labels are imbalanced (largest class 39% at generation 1, 45% at generation 4).

W5. Practical value. We agree that geometry does not outperform accuracy when labels are available: accuracy should be measured on a labeled set large enough for the decision, and geometry-triggered interventions did not establish a consistent advantage (Section 5.8, L489–499). For label-free monitoring, selecting the generation with the highest LiDAR loses 24.2 accuracy points on CIFAR-DDPM and 16.0 on real images, and 7.3 and 9.1 points among generations 1–4 (exploratory, Appendix F.5, L2696–2779). The Fisher criterion computed with the classifier's own predictions also needs no labels. Like the true-label Fisher criterion, it selects generation 0 in all six chains, but it misses the later ImageNet-100 decline.

AuthorsOpenReview ↗

Response to Reviewer B5Kf [3/3]

Q1. Without recursion.

  • Label noise in conventional supervised training: yes. With 20% or 40% of the CIFAR-10 training labels flipped, accuracy falls 11.0 or 25.1 points while covariance rank and LiDAR rise by 79–99%, in all three seeds (table under W3; Appendix F.3, L2581–2640). This agrees with Ma et al. (ICML 2018), who found that the local intrinsic dimensionality of learned representations rises once networks start to fit noisy labels.
  • Pseudo-label errors on real images, one step: training on the real CIFAR-10 images labeled by the generation-1 classifier C1C_1 (73.4% of these labels correct) yields 72.1% accuracy, 16.3 points below C0C_0, while covariance rank rises 23.3% and LiDAR 51.9%, in all three seeds (table under W3).
  • Distribution shift: one generation on CINIC-10 images with their provided labels, and the first switch to DDPM images, before labels from later classifiers are used, both raise rank while accuracy falls (table under W3; Table 2, L281; Appendix F.2, L2526–2578). Both also change the label source, and the CINIC-10 labels are not verified, so neither isolates the shift in images. One-step retraining on the original labeled CIFAR-10 set reaches 88.48% and leaves rank nearly unchanged.

Q2. Fine-tuning or retained data.

  • Fine-tuning: each generation fine-tunes its predecessor for 50 epochs (peak learning rate 0.01) on the same DDPM images, labeled by the predecessor (Appendix F.3). The accuracy loss at the first switch is smaller than with a new classifier (10.1±0.4 versus 16.9±0.3 points). Generations 1–4 lose 11.1±0.5 points (9.3±1.2 with new classifiers), and generation-4 accuracy is 5.0±1.0 points higher. Rank still rises: from generation 0 to 4, covariance rank rises 14.0±1.4% and LiDAR 58.0±4.5%, and over generations 1–4 by 7.0±1.5% and 15.3±3.3%, in all three seeds, while the true-label Fisher criterion falls in both intervals.
  • Retained data: in the fresh runs that keep 50% real data (Mix-50), covariance rank rises 10.2±2.2% and LiDAR 24.3±2.2% while accuracy falls 4.9 points (Appendix Table 45, L2500). The historical Mix-25/50/75 and Accumulate runs also end with a higher RR and lower accuracy than generation 0 (Section 5.9, L501–511).

Q3. Conditioning labels. On CIFAR-10 StyleGAN2 images of generation 1, training on the generator-conditioning labels instead of C0C_0's pseudo-labels (same images and seed; the two agree on 88.5% of images) changes accuracy by −0.3±0.5 points (mixed signs) but lowers covariance rank 5.3±0.6%, LiDAR 8.3±1.5% and RR 8.7±0.6%, in all three seeds (pre-registered, Appendix F.3). On DiT images, a one-step ViT-S control trained on the conditioning labels has 2.43±0.24 points lower accuracy than the pseudo-label control, while its RR improves (8.47 vs 9.05) and its effective rank falls (108.6 vs 126.5; Section 5.2, L290–318). In both settings the conditioning labels lower rank without raising accuracy; they are not verified labels (agreement 88.5% and 67.4%).

Q4. FID, images and diversity. The revision adds FID against the 50,000 CIFAR-10 training images, Inception-space precision and recall as a diversity measure, and sample grids (Appendix F.4, L2643–2693; the CIFAR-10 test images give FID 3.15).

  • The fixed DDPM, sampled with 50 steps in all our DDPM runs, produces visibly noisy images: its pools have FID 131.6±0.4, precision 0.54 and recall 0.16, with no trend across generations. Its sampling distribution thus stays fixed, while image samples, labeling teachers and newly trained students change.
  • The StyleGAN2 pools have FID 2.52±0.02 and recall 0.61 (the 10,000 real test images give 3.15 and 0.69), and the ImageNet-derived CINIC-10 images FID 9.9. LiDAR still rises while accuracy falls in both chains (+19.1±2.4% and +68.5±6.1%), and covariance rank rises 26.9±2.6% on the CINIC-10 images, so neither rise requires the low-quality DDPM samples.
  • In joint retraining, FID rises from 132 for the pretrained generator to 323 after five rounds of generator fine-tuning, and recall falls from 0.15 to 0.00. The jointly retrained generator degrades, and the classifier's rank falls with it (Section 5.9).
  • The class shares of the pseudo-labels in the first and last generation are in Appendix Table 46 (L2514).

Q5. Stronger ViTs and more seeds. ViT: an ImageNet-pretrained ViT-S/16 (98.6% at generation 0) reproduces the mismatch on the CIFAR-DDPM chain in all three pre-registered intervals (generations 0–1, 1–4 and 0–4; W4 above; Appendix F.6, L2782–2822). In each interval, accuracy falls and LiDAR rises in all three seeds. Covariance rank also rises, except over generations 1–4, where it is mixed. Seeds: the CIFAR-DDPM rank rise holds in all six seeds of two independent cohorts, fresh seeds 101–103 and historical seeds 42–44 trained with an earlier recipe. In the historical cohort, rank goes from 12.3/12.1/12.4 to 15.3/14.7/13.3 by generation 6 as accuracy falls to 56–57%.

AuthorsOpenReview ↗

Additional Result on Label-Free Monitoring (W5)

We would like to add one result that bears on W5 (practical value), obtained after our response. It uses the stored outputs of the models in the revision; no model was retrained, and the PDF is unchanged.

We computed standard label-free accuracy estimators on eight chains of the revision, three seeds each: mean confidence, ATC with its threshold set at generation 0 (Garg et al., ICLR 2022), the nuclear norm of the prediction matrix (Deng et al., ICML 2023), the dispersion score (Xie et al., NeurIPS 2023), and the agreement of each classifier with the predictions of the initial classifier C0C_0 on the same test images.

  • Confidence. Mean confidence and ATC miss the later decline in two chains. Over generations 1–4 on CIFAR-DDPM, accuracy falls 9.3 points while they rise 1.1±0.1 and 2.6±0.4 points, in all three seeds. They also rise over ImageNet-100 generations 1–5 while accuracy falls 4.9 points.
  • Dispersion score. From generation 0 to 4, it rises with real unlabeled images and with sequential fine-tuning while accuracy falls.
  • Nuclear norm. It does not move against accuracy, but its direction differs between seeds in two cases.
  • Agreement with C0C_0. It needs no labels. Wherever accuracy falls in all three seeds, at the first switch or over the later generations of a chain, agreement falls in all three as well. Among the generations after generation 0, the one with the highest agreement is the most accurate in every chain and seed.

A label-free monitor that follows accuracy in our chains therefore exists. It rests on class predictions rather than on feature rank, and it offers a way to monitor recursive training when no labeled set is available. The analysis is exploratory. Agreement requires keeping C0C_0, and it would not detect a classifier that improves on C0C_0.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers. In recursive pseudo-labeling, label-free effective-rank monitors, including LiDAR, can rise while accuracy falls, whereas the Fisher criterion computed with true labels falls with accuracy from the first to the last generation. The revision tests this with six new pre-registered experiments and three new analyses. All other new results use the original submission's stored checkpoints, images and evaluation sets. Each new experiment uses three seeds, and unless stated otherwise the changes it reports have the same sign in all three. L denotes line numbers in the revised PDF.

RequestWhat the revision adds
ckdZ Q1, B5Kf W1: LiDAR and class-aware metricsLiDAR, the regularized LDA rank and the Fisher criterion on all trajectories (Section 5.5, L404–456). From generation 0 to 4, LiDAR rises in all five CIFAR-10 chains and all three seeds: DDPM images with inherited labels, with labels from the initial classifier C0C_0 or with 50% real data, StyleGAN2 images, and real images. With DDPM images and inherited labels (CIFAR-DDPM) it rises 56.5±5.7% while accuracy falls from 88.4% to 62.2%. With true labels, the within-class share of LiDAR's between-image scatter grows from 27.5% to 57.3% on CIFAR-DDPM (exploratory, Appendix F.5, L2696–2779), and the Fisher criterion tracks accuracy over every interval of Table 3 (L379).
ckdZ W2, B5Kf W5, F74L W2: practical valueIn an exploratory comparison, selecting the generation with the highest LiDAR would lose 24.2 accuracy points on CIFAR-DDPM (7.3 among generations 1–4), whereas the true-label Fisher criterion selects generation 0 in all six chains (Appendix F.5).
ckdZ Q2, F74L Q2: real unlabeled images, other settingsThe same loop on ImageNet-derived CINIC-10 images: accuracy falls 16.0 points while covariance rank rises 26.9±2.6% and LiDAR 68.5±6.1% (Section 5.2, L290–318; Appendix F.2, L2526–2578).
B5Kf Q1, W3: label errors without generated imagesWith 40% of the CIFAR-10 training labels flipped, accuracy falls 25.1±0.3 points while covariance rank rises 98.8±7.4% and LiDAR 90.8±16.2%. With the real images labeled by the generation-1 classifier, accuracy falls 16.3±0.6 points while they rise 23.3±3.7% and 51.9±4.7%. The true-label Fisher criterion falls in both (Appendix F.3, L2581–2640).
B5Kf W2/Q2: continual fine-tuningWhen each generation fine-tunes its predecessor instead of training a new classifier, accuracy falls 21.1±0.5 points by generation 4 while covariance rank rises 14.0±1.4% and LiDAR 58.0±4.5% (Appendix F.3).
B5Kf Q3: conditioning labelsOn CIFAR-10 StyleGAN2 images, training on the generator-conditioning labels instead of C0C_0's pseudo-labels (88.5% agreement) changes accuracy by −0.3±0.5 points (mixed signs) but lowers covariance rank 5.3%, LiDAR 8.3% and RR 8.7% (Appendix F.3).
B5Kf Q5, W4: stronger architectureAn ImageNet-pretrained ViT-S/16 (98.6% at generation 0) on the same DDPM images: accuracy falls 3.8±0.3 points by generation 4 while covariance rank rises 35.9±4.9% and LiDAR 37.4±3.4%, and the true-label Fisher criterion falls 53.3±1.4% (Appendix F.6, L2782–2822).
B5Kf Q4: FID, images, diversityFID against the CIFAR-10 training images, precision and recall, and sample grids (Appendix F.4, L2643–2693). Across its 12 pools the fixed DDPM, sampled with 50 steps, has FID 131.6±0.4 with no trend over generations. The StyleGAN2 pools have FID 2.5 and the CINIC-10 images 9.9, and LiDAR rises with both. In joint retraining, FID rises from 132 to 323 by the fifth generator round.
ckdZ Q3: title, joint retrainingWe rewrote the abstract (L11–32) and introduction (L35–115) to state the scope as recursive pseudo-labeling with a fixed generator, and we propose "Effective Rank versus Class Separation in Recursive Pseudo-Labeling" as the title of the final version. In joint retraining, rank falls with accuracy while the generator degrades (Section 5.9, L501–511).
F74L W6: fixed-teacher controlIn both chains, each generation trains on exactly 50,000 generated images and nothing accumulates; only the labeling teacher differs (Section 5.3, L320–366).
F74L W1, ckdZ: readabilityThe introduction defines the loop, WW, BB, RR and LiDAR before the results; internal shorthand was replaced; fresh, historical and post-outcome runs are defined in Section 5.1 (L253–288); limitations are collected in the conclusion (L513–538).

We also cite the two works suggested by B5Kf that the submission did not cite (Otero et al., 2024; Rangamani et al., ICML 2023).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.