Feature Support Erosion in Long-Tail Fine-Tuning

Pretrained models provide strong prior knowledge for long-tailed recognition, making fine-tuning a promising alternative to training from scratch. However, we find that tail classes with similar training frequencies can behave very differently during fine-tuning: some remain stable, while others suffer severe forgetting. This suggests that class frequency alone is insufficient to explain tail-class stability. Our analysis reveals a key factor behind this difference: Feature Support Erosion, where pretrained discriminative directions of different classes receive unequal support from the dominant training signals. To characterize this effect, we use class-wise Average Gradient Outer Product (AGOP) to identify pretrained discriminative directions and measure their support from head-class training signals. Based on this analysis, we propose Support-Aware Directional Preservation (SADP), which assigns stronger preservation to poorly supported classes and selectively constrains lightweight feature updates along their pretrained discriminative directions. Extensive experiments demonstrate that SADP consistently outperforms strong pretrained fine-tuning methods, effectively reducing tail-class forgetting while preserving the flexibility of downstream fine-tuning.