Depth-Graded Parameter-Efficient Adaptation of Foundation Models for Face Recognition

Vision foundation models are an attractive starting point for training face recognition (FR) models, which often require large identity-labelled datasets. Current approaches to use foundation models for FR adapt them with low-rank adapters that use the same rank in every transformer block. We ask *where in the foundation model network needs capacity for FR adaptation*. After probing CLIP ViT-L/14 with high-rank adapters, we find that the rank of the learned adapter grows with model depth, while a uniform rank-16 adapter is saturated in deep blocks and under-used in shallow ones. Motivated by this, we propose *depth-graded rank allocation*, a search-free schedule that increases adapter rank linearly with depth. We pair it with rank-stabilised scaling and instantiate it with low-rank and weight-decomposed adaptation (called DG-LoRA and DG-DoRA, respectively). We study eight ViT-L foundation models in our experiments. We show that depth-graded ranks improve adapted ViT-L over uniform rank adapters with the same budget. Trained on WebFace4M our model with 14M trainable parameters reaches 95.1% TAR on IJB-C (FAR=$10^{-4}$), surpassing prior parameter-efficient adaptation of foundation models and a ViT-L trained from scratch on the same data.