- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies a setting where image classifiers are trained recursively on synthetic data (pseudo labels from classifiers of previous iteration): a fixed pretrained generator (DDPM, GAN or DiT) produces the synthetic images. The authors study the effective rank of the final layer embeddings as an analytical measure, together with accuracy and a within/between class “scatter ratio”. A key observation is that in the CIFAR-10 + DDPM setting, global feature effective rank increases while both within and between class effective ranks fall. They derive mathematical decompositions of the change in log effective rank including two component spectrum terms, and show that the reweighting between the two components can be associated to this behavior. The paper also includes extensive experimentation on training stage, teacher, input stem, rank definition, generators, small labeled audit sets.
Critical strengths
- Observation that global feature effective rank can rise even when both within and between class effective ranks fall.
Critical weaknesses
- The paper is quite hard to follow, frequently mentions (even in the abstract/introduction sections) findings or terminology that has either not been explained/cited, or only introduced later in the paper.
- The setup of measuring feature rank over recursive training of classifiers seems to be newly proposed, with no existing citation, therefore significance is limited.
- Keeping the generator fixed is not the typical setting to measure “model collapse” in the literature.
- The authors only observe this surprising behavior of feature rank not agreeing with classification in the CIFAR-10/DDPM setting. This limits the significance of this work further.
- Much of the paper is spent for experimental results that do not follow this novel observation.
- The fixed-teacher control does not quite make sense to me, it’s training with the same generator and same C0 for labeling - so it’s just increasing the training data, not sure if this can be associated with “fixing the initial teacher” as mentioned in the abstract.
Questions for authors
- Is there any prior work that uses classifier feature rank to monitor recursive or training with pseudo labels?
- Is there any combination other than CIFAR-10 with DDPM in the proposed recursive setting where we observe covariance rank to rise while classification performance falls?
4 comments
Thank you for the changes to the paper.
Readability: I think the paper is much easier to read. I believe the readability can still be improved to convey the novelty and significance of the work more clearly, especially in the abstract and introduction. The abstract still feels like a list of experiments instead of a coherent finding. Introduction has too many references to sections now, I believe introduction should be more self-contained and convincing. They should make clear the key contribution as it stands, which I believe to be that the direction of rank does not indicate whether accuracy rises or falls.
Fixed-teacher control: I appreciate the clarification on fixed teacher control, that makes sense and my initial understanding was incorrect.
Other combinations: I appreciate the additional experiments on real images with CINIC-10, in general, I agree with your assessment, of "its direction does not indicate whether accuracy rises or falls", which leads me to the next point and final assessment.
Significance, prior work: I believe this is the main weakness of this paper now. The recursive pseudo-labeling setting being constructed seems fair on its own but it is not a standard practical setting. The key contribution therefore becomes a critique of the effective rank and LiDAR, however, these scores are typically used to select models in the standard self-supervised learning, which is different from the specific setting in this paper. In your Response to Reviewer ckdZ's Follow-up Comment , you justify as "It was an open question whether LiDAR would avoid the mismatch." and "None of the label-free statistics we tested follows accuracy in every chain, so none can replace a labeled check.". I think if this is the main takeaway, these claims deserve much more emphasis and a more careful study to make a strong paper, for example, testing these scores in the self-supervised setting they were actually designed for, going beyond CIFAR-10, and comparing them against other standard label-free accuracy estimators.
I will raise my score from 1 to 2, however, I think this paper still leans reject.
Response to Reviewer F74L's Follow-up Comment
Thank you for reading the revision and for the updated assessment. Your statement of the contribution, that the direction of rank does not indicate whether accuracy rises or falls, is the one we intend. We will restructure the abstract and introduction around it, with fewer pointers to sections.
On the setting. The mismatch is not tied to the recursive loop. In ordinary supervised training on CIFAR-10 with 40% of the labels flipped, covariance rank rises 98.8±7.4% and LiDAR 90.8±16.2% while accuracy falls 25.1 points (Appendix F.3, L2581–2640). The same control on CIFAR-100, run during this discussion with the plan fixed beforehand, gives the same result: accuracy falls 25.2±0.2 points while covariance rank rises 85.9±0.5% and LiDAR 12.5±0.4%, in all three seeds.
On where rank is used. Feature rank is also used outside self-supervised learning. In continual and reinforcement learning, a falling rank is read as lost capacity (Kumar et al., ICLR 2021; Lyle et al., ICLR 2022; Dohare et al., Nature 2024). Our results show that the converse reading, a higher rank as a better classifier, does not hold in the settings above.
On other label-free accuracy estimators. We have now run the comparison you suggest, on the stored outputs of eight chains of the revision, three seeds each. It is exploratory. The PDF is unchanged and does not contain this comparison or the CIFAR-100 control.
- Mean confidence and ATC (Garg et al., ICLR 2022) rise over generations 1–4 on CIFAR-DDPM (+1.1±0.1 and +2.6±0.4 points, in all three seeds) while accuracy falls 9.3 points. They also rise over ImageNet-100 generations 1–5.
- The agreement of each classifier with the initial classifier's predictions needs no labels. Wherever accuracy falls in all three seeds, at the first switch or over the later generations of a chain, agreement falls in all three as well.
This qualifies the sentence you quote from our reply to Reviewer ckdZ: agreement with the initial classifier, tested since, does follow accuracy in every chain. It uses class predictions rather than feature rank, and it adds a positive recommendation to the negative result.
On what remains. Testing the scores in self-supervised training, the setting they were designed for, is the extension we plan next. The present claims concern supervised classifiers.
Thanks. If not tied to recursive loop that needs some clear phrasing in the paper again. The paper should not be presented as "We study recursive pseudo-labeling with a fixed generator", but rather a claim about rank can rise with poor data quality, which is aligned with findings in previous work (Ma et al., 2018, that you cite). The CIFAR-100 result can be included in a revision.
re: where rank is used, IMO falling rank for lost capacity studies something different than higher rank vs accuracy.
interesting that the agreement with the initial classifier follows accuracy with labels.
These are all interesting ideas and can be made a stronger work with more emphasis and experimentation.
Response to Reviewer F74L
Thank you for the careful reading.
Readability (W1, W5). The introduction (L35–115) now defines the training loop, W and B, R and LiDAR before the main quantitative results, and terms in the abstract and introduction, such as the first switch and the Fisher criterion, are defined where they first appear. Internal shorthand ("replacement", "original stem") was replaced, and "fresh", "historical" and "post-outcome" runs are defined in Section 5.1 (L253–288). To keep the main text focused on the main observation, the evidence map and the stem tables moved to the appendix, the new Section 5.5 (L404–456) tests the observation against LiDAR and class-aware statistics, and the conclusion (L513–538) collects the limitations in one paragraph.
Fixed-teacher control (W6). The original description was unclear. In both chains, each generation trains on exactly 50,000 generated images and nothing accumulates, so the training set does not grow. The control takes the same generated images of generations 2–4 and labels them with the initial classifier C0 instead of the previous classifier. Inputs, budget, initialization and recipe stay fixed (Section 5.3, L320–366). From generation 1 to 4, accuracy falls 9.28±1.17 points under inheritance and rises 0.67±0.10 points with C0 labels (Table 2, L281).
Significance and setting (W2, W3). Effective-rank scores such as RankMe and LiDAR are used to select or monitor models without labels. Recursive pseudo-labeling is a setting where labels are missing and models degrade, and in this setting these scores can rise while accuracy falls (Section 5.5, Table 3, L379). In an exploratory comparison, selecting the generation with the highest LiDAR would lose 24.2 accuracy points on CIFAR-10 with DDPM images and 16.0 on real images (7.3 and 9.1 among generations 1–4), whereas the Fisher criterion computed with true labels selects generation 0, the best one, in all six chains (Appendix F.5, L2696–2779). We agree that a fixed generator differs from model collapse in the usual sense. The revision states this scope, reports that joint retraining lowers rank together with accuracy after an initial rise while the generator's FID rises from 132 to 323 (Section 5.9, L501–511; Appendix F.4, L2643–2693), and adds a setting without any generator (Q2 below).
Q1. Prior work. We are not aware of prior work that validates feature rank for this loop. Feature rank is used to assess representations in related settings: RankMe and LiDAR select self-supervised models without labels, and in continual and reinforcement learning a falling feature rank is read as lost plasticity or capacity (Kumar et al., ICLR 2021; Lyle et al., ICLR 2022; Dohare et al., Nature 2024). The revision cites these works in the introduction. When each generation fine-tunes its predecessor, as in continual training, covariance rank and LiDAR still rise while accuracy falls (Appendix F.3, L2581–2640).
Q2 (W4). Other combinations. Yes, within CIFAR-10. Covariance rank and LiDAR rise while accuracy falls, in all three seeds, in these settings:
Among the fresh generated-image chains, covariance rank rises only with DDPM images, while LiDAR also rises with StyleGAN2 images. We do not claim that rank always rises; the claim is that, without labels, its direction does not indicate whether accuracy rises or falls.