acceptodds
Under review as a conference paper at ICLR 2027

Feature Support Erosion in Long-Tail Fine-Tuning

Abstract

Pretrained models provide strong prior knowledge for long-tailed recognition, making fine-tuning a promising alternative to training from scratch. However, we find that tail classes with similar training frequencies can behave very differently during fine-tuning: some remain stable, while others suffer severe forgetting. This suggests that class frequency alone is insufficient to explain tail-class stability. Our analysis reveals a key factor behind this difference: Feature Support Erosion, where pretrained discriminative directions of different classes receive unequal support from the dominant training signals. To characterize this effect, we use class-wise Average Gradient Outer Product (AGOP) to identify pretrained discriminative directions and measure their support from head-class training signals. Based on this analysis, we propose Support-Aware Directional Preservation (SADP), which assigns stronger preservation to poorly supported classes and selectively constrains lightweight feature updates along their pretrained discriminative directions. Extensive experiments demonstrate that SADP consistently outperforms strong pretrained fine-tuning methods, effectively reducing tail-class forgetting while preserving the flexibility of downstream fine-tuning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-Tailed Recognition, Class Imbalance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LdsoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why tail classes with similar training frequencies exhibit different degrees of forgetting during adaptation of pretrained models. It introduces Feature Support Erosion and uses the Average Gradient Outer Product to characterize the overlap between pretrained discriminative subspaces and gradients from head classes. The proposed SADP method applies stronger directional preservation to classes with weaker feature support and achieves consistent improvements across four benchmarks for recognition under class imbalance.

Critical strengths
  1. The paper presents a useful perspective on tail class forgetting by examining feature support beyond class frequency.
  2. The analysis of feature support, feature retention, and recognition margins provides a coherent motivation for the proposed method.
  3. SADP selectively preserves vulnerable feature directions, with subspaces and support weights computed only once before training.
  4. Experiments across benchmarks show gains on tail classes while maintaining competitive performance on frequent classes.
Critical weaknesses
  1. The proposed explanation relies mainly on correlations, since directional coverage does not distinguish beneficial from harmful updates.
  2. The ablations lack directional preservation with uniform weights, making the contribution of support weighting difficult to isolate.
  3. The reliability of AGOP subspaces is not empirically verified, particularly for tail classes with fewer samples than the retained subspace dimension.
  4. The paper should clarify how the exact feature difference is obtained without an additional backbone evaluation when adapters are inserted throughout the Transformer.
Questions for authors

Please refer to weaknesses.

Reviewer_TSJTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why tail classes with similar sample counts suffer different amounts of forgetting during fine-tuning. SADP uses class-wise AGOP subspaces to identify important pretrained directions and weights their preservation by overlap with head-class gradients. Experiments on four long-tailed benchmarks show consistent gains over LIFT and other baselines.

Critical strengths
  1. Preserving selected feature directions is a sensible way to balance adaptation and retention. The method is simple once the subspaces have been computed.
  2. The reported gains are consistent across four datasets. On ImageNet-LT, SADP improves LIFT from 77.1% to 79.0% overall and from 72.8% to 75.9% on tail classes.
Critical weaknesses
  1. Appendix D.1 fixes k = 16, but a class AGOP has rank at most the number of gradient samples used to estimate it. For classes with fewer than 16 such samples, some selected directions may belong to an arbitrary null space. Appendix C.5 discusses reliability qualitatively without resolving this case. This matters because the same subspace determines both the support score and the penalty.
  2. Table 3 lacks uniform-weight directional preservation, the most direct control for the proposed support weighting. The current comparison changes both weighting and the constrained directions. Figure 2 shows an association with forgetting, but squared gradient overlap does not distinguish helpful from conflicting updates. The benefit of support weighting therefore remains unclear.
Questions for authors
  1. How are rank-deficient AGOPs handled, and how sensitive are the results to k?
  2. What happens when the same directional penalty uses equal weights across tail classes?
  3. Which feature layer is constrained, and how is its residual relative to the pretrained model computed when adapters are inserted into multiple blocks (Appendix D.1)?
Reviewer_ftF1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies tail-class forgetting in long-tailed fine-tuning. the author use class-wise Average Gradient Outer Product (AGOP) to identify pretrained discriminative directions and measure their support from head-class training signals. Based on this analysis, the authors propose SADP to provide stronger preservation for poorly supported classes and selectively constrain lightweight feature updates along their pretrained discriminative directions. Experiments demonstrate significant gains over previous methods.

Critical strengths
  1. The paper studies tail-class forgetting in long-tailed fine-tuning and provides a novel perspective by analyzing it through AGOP.

  2. The experimental evaluation is extensive, spanning multiple datasets and imbalance settings, and consistently demonstrates the effectiveness of the proposed approach across tail classes.

Critical weaknesses
  1. The authors consider only head and tail classes: support is estimated from head classes, and preservation is applied only to tail classes, leaving medium classes neither analyzed nor protected; the authors should justify excluding medium classes and ablate including them in both steps.

  2. The authors should analyze the key hyperparameters, including λ\lambda, kk, and τ\tau, to demonstrate the robustness of the proposed method.

  3. The authors should discuss the computational cost and running time. Since the proposed approach involves computing class-wise AGOPs and performing eigendecomposition to extract discriminative subspaces, does this introduce significant additional computational overhead?

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.