acceptodds
Withdrawn from ICLR 2027

Understanding and Mitigating Catastrophic Inheritance via Discriminative Spectrum Drift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pre-training on large-scale data has become a standard paradigm for visual transfer learning, but noisy supervision in pre-training data can be inherited by downstream models and degrade their generalization. While prior studies have shown that noisy pre-training affects downstream transfer, its structural effect on downstream representations remains insufficiently understood. In this work, we recast catastrophic inheritance as discriminative spectrum drift, and study how noisy pre-training reshapes class-relevant discriminative information in spectral space. Across multiple commonly used cross-domain visual datasets and controlled noisy pretrained checkpoints, we show that the inherited effect of noisy pre-training is not merely reflected in performance variation, but manifests as a consistent drift of the downstream discriminative spectrum. To characterize this phenomenon, we introduce two continuous descriptors that capture the overall deviation from the clean reference and the average spectral position of discriminative information. Motivated by these findings, we further propose a lightweight spectral adaptation framework for the realistic setting where no clean reference is available and the pretrained backbone remains frozen. Experiments show that our method mitigates discriminative spectrum drift and improves downstream adaptation performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
catastrophic inheritance, noise machine learning, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DmwfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how noisy pretraining affects downstream transfer with frozen visual backbones. It characterizes feature changes using a normalized discriminative spectrum, its Jensen–Shannon divergence from a clean reference, and its spectral centroid. The proposed Self-Guided Spectral Adaptation (SGSA) adds a residual bottleneck adapter and regularizes its features using a spectral prior constructed from PCA-direction stability, augmentation consistency, and positional decay. Alignment, leakage, and feature-preservation losses are intended to mitigate inherited representation damage without requiring a clean checkpoint for adaptation. Experiments use ImageNet-1K- and YFCC15M-pretrained ResNet-50 backbones on Office31 and Office-Home, with adapter-only and component comparisons.

Critical strengths
  1. The paper addresses a meaningful downstream constraint. Users often inherit pretrained checkpoints without access to clean pretraining data or the resources to retrain the backbone. An adaptation procedure that operates with a frozen backbone and does not require a clean reference would be useful in this setting. Separating feature variance from task-relevant discrimination is a reasonable motivation for studying which directions to preserve.

  2. The experimental design includes initial controls for the adapter's contribution. The adapter-only baseline and component comparisons attempt to separate spectral regularization from the benefit of adding trainable parameters. These are relevant tests of the proposed approach, although the reporting problems below currently limit the conclusions that can be drawn from them.

Critical weaknesses
  1. The central discriminative spectrum is not operationally defined. Section 2.3, Eq. (2), normalizes direction-wise strengths as qi=di/∑jdjq_i=d_i/\sum_j d_j, but the supplied manuscript does not specify how did_i is computed. This is consequential for both the diagnostic and the algorithm: Eq. (15) uses the batch-dependent spectrum q(z)q(z) as a training objective. Different definitions of discriminability would produce different spectra, gradients, and interpretations of drift. Without the exact statistic, label dependence, handling of missing classes or zero denominators, and differentiation procedure, the central results cannot be reproduced or assessed for correctness.

  2. The baseline results and experimental protocol do not support a coherent assessment of effectiveness. Table 1 reports clean-pretraining baseline accuracies of 57.21 and 47.01 for Office31 and Office-Home; Table 2 reports 74.6 and 33.3. At 30% noise, the corresponding pairs are 50.54 versus 63.0 and 42.89 versus 28.9. Table 1 explicitly averages over both backbone families, whereas Table 2 does not clearly identify a changed protocol. Table 4 is also described as averaging over both families and noise levels, but its baseline means match Table 2. Different settings could explain these discrepancies, but the manuscript needs to identify them to connect the diagnostic findings with the mitigation results. The lack of per-backbone/task results, clear label-access and split definitions, and repeated-run uncertainty further prevents assessing the roughly 0.5–0.6 percentage-point gains over NML in several Office-Home settings.

Questions for authors
  1. Clarification essential to assessing the method: What is the exact formula and executable procedure for did_i and q(z)q(z)? Please specify the labels used, normalization and numerical stabilization, handling of batches with absent classes, and how gradients through the spectrum are computed.

  2. Clarification essential to assessing the results: What protocol or aggregation difference explains the baseline discrepancies between Tables 1, 2, and 4? Please provide the corresponding per-backbone and per-transfer-task results, with the label-access protocol, train/validation/test splits, and run counts clearly identified.

  3. Clarification of the reported comparative gains: What paired repeated-run uncertainty supports the small Office-Home improvements over NML, and were SGSA and the compared methods evaluated with matched data access and tuning budgets?

These questions concern the definition and validity of the reported work, rather than a request for a separate future benchmark study. Complete answers could materially change my recommendation.

Reviewer_XdzROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how noisy pre-training affects downstream transfer from the perspective of discriminative spectrum drift. It proposes Self-Guided Spectral Adaptation to mitigate this effect with a frozen backbone and without a clean reference model.

Critical strengths

The idea that directions with larger feature variance are not necessarily more useful for classification is reasonable. Examining the discriminative ability of individual spectral directions provides a useful perspective and may inspire further research.

Critical weaknesses
  1. The core discriminative strength d_i is not defined. The paper uses d_i to construct the discriminative spectrum but does not explain how it is calculated. Since both the analysis metrics and training objectives depend on this quantity, their correctness cannot be verified.

  2. The connection between spectral drift and damaged discriminative information is not sufficiently established. A change in the spectral location of discriminative information does not necessarily imply that the information has been damaged. The key assumptions behind the reliability prior also lack sufficient validation.

  3. The baseline results and aggregation procedures are inconsistent across Tables 1, 2, and 4. The authors should explain the differences and clearly state how the results in each table are obtained and averaged.

  4. The experimental details are incomplete. The transfer protocol, sources of the baseline methods, and implementation parameters are not sufficiently described. This makes it difficult to assess whether the comparisons are fair and whether the results can be reproduced.

Questions for authors

Although changes in the spectral location of discriminative information provide an interesting analysis perspective, the current paper appears incomplete.

The core discriminative spectrum is not fully defined, key assumptions are not validated, and the reported baseline results are inconsistent across tables. The available evidence is therefore insufficient to support the main conclusions.

Reviewer_3Hw6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the adaptation of models pre-trained on noisy data. Specifically, it observes that increasing noise levels during pre-training lead to degraded transfer performance on downstream tasks. To better understand this phenomenon, the paper analyzes the feature representations through spectral decomposition and examines the deviations between models pre-trained on noisy and clean data. Based on this analysis, the paper proposes SGSA, which incorporates an adapter with spectrally motivated regularization to improve the adaptation of noisy pre-trained models to downstream tasks. Experimental results demonstrate that SGSA achieves state-of-the-art performance compared with existing methods.

Critical strengths
  • The paper is well-written and well-structured, starting with an empirical observation and analysis that motivate the proposed SGSA method. The overall presentation is easy to follow, and the motivations behind the proposed approach are clearly explained.
Critical weaknesses
  • The paper primarily focuses on standard classification tasks with relatively modest-scale pre-training compared to modern large-scale models. It remains unclear whether similar effects of noisy pre-training can be observed in more realistic large-scale pre-training scenarios, such as those involving LLMs. Moreover, the evaluation is limited to classification tasks, while models such as CLIP are increasingly used as components in more complex systems, including vision-language models (VLMs) and diffusion models. It remains unclear whether noisy pre-training would similarly affect performance in these broader and more practical downstream applications.

  • The analysis of noisy pre-training assumes that the pre-trained backbone remains frozen during downstream adaptation, which may oversimplify the problem and limit the realism of the experimental setting. In practice, parameter-efficient fine-tuning (PEFT) approaches, such as LoRA, are widely adopted for adapting pre-trained models to downstream tasks. It remains unclear whether the observed effects of noisy pre-training can be generalized to these more practical fine-tuning settings.

Questions for authors

Besides the weakness shown in the above section, please also see the following questions:

Q1: How is did_i in Equation 2 defined and computed? Its definition does not appear to be provided anywhere in the paper.

Q2: What are the key differences between the proposed spectral adapter in Section 3.2 and existing PEFT approaches, such as LoRA? Could other PEFT methods also be applied to mitigate the negative effects of noisy pre-training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.