- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is concerned with finetuning vision foundation models for the face recognition (FR) task. Following up from recent work on LoRA finetuning of these models they perform spectral analysis on LoRA learned weights and find that high rank LoRAs are effectively saturated in earlier layers while contributing more to the final layers. This leads to the proposition of increasing LoRA rank linearly by depth. Thei final model achieves SOTA performance on FR.
Critical strengths
- The proposed approach offers benefits and can be used as a general solution for finetuning vision foundation models for FR.
- It is sensible and applies existing knowledge, earlier layers work as generic pattern detectors while later layers extract task specific features, to design the method. As such early layers require little adaptation sicne they use the pretrained existing feature extractors while later layers require more extensive finetuning. It is also interesting this observation is verified in practice.
- Ablations cover a significant part of the design space.
- Literature review is comprehensive
Critical weaknesses
- Training settings are nor consistent among tables. T2 uses PArtial Fc, T4 uses FC head, T5 trains output projection with Partial FC. A single method would make results more easily comparable.
- How checkpoints are selected is not clear, a central claim is needed about which checkpoint is used which is consistent among datasets, for example the final checkpoint after N epochs. Proposed method is only tested on CLIP ViT-L/14 and ViT-B/16, despite other foundation models available.
- For FR TAR@FAR=1e-4 is very lenient allowing 1 mistake every 10k comparisons, would be interesting to see more values reported for 1e-5 and 1e-6 too.
- No tables contain any variance info. For saturated sets like these FR sets it is critical to see that reported metrics do not fall in the same region when accounting for variance.
- Paper claims training efficiency is important, however, inference time for FR is much greater cost than training and all models have the same inference cost, as such the efficiency claim is very limited.
- No domains where FR data are really scarse such as IR are tested.
- No code is provided.
Questions for authors
- What is the TAR of your model @FAR=1e-5 and 1e-6 against comparisons?
- How do you select which checkpoint to test?
2 comments
Authors Reply to Reviewer Z929 [2/2]
- Reply to Weakness #6: We agree that heterogeneous face recognition, such as NIR-VIS, is an important data-scarce setting. Our study explores where capacity is needed when a foundation model is adapted to FR, and we evaluate it at two data scales on CASIA-WebFace and WebFace4M. Extending depth-graded allocation to cross-spectral FR with only a few hundred training identities can be studied in future work. However, following the reviewer's suggestion, we evaluated our method on near-infrared (NIR) to visible (VIS) face recognition using CASIA NIR-VIS 2.0 with the standard view-2 protocol with 10 folds. In each fold, the training set has about 8.7k NIR and VIS images of 357 identities. The test set matches about 6.2k NIR probe images against a VIS gallery with one image for each of 358 other identities. We report rank-1 identification accuracy and the verification rate (VR) at FAR = 0.1% and 0.01%, as mean ± std over the 10 folds. We consider two models trained on WebFace4M and then evaluate without any NIR training (zero-shot):
| Training data | Model | Rank-1 (%) | VR @ FAR=0.1% | VR @ FAR=0.01% |
|---|---|---|---|---|
| WebFace4M | rsLoRA | 98.53 ± 0.33 | 96.85 ± 0.38 | 88.16 ± 1.88 |
| WebFace4M | DG-LoRA (ours) | 99.40 ± 0.21 | 98.53 ± 0.21 | 93.54 ± 1.04 |
As an additional experiment, we consider the two models trained on WebFace4M and then fine-tune them on the NIR-VIS training set of each fold:
| Training data | Model | Rank-1 (%) | VR @ FAR=0.1% | VR @ FAR=0.01% |
|---|---|---|---|---|
| WebFace4M → NIR-VIS train | rsLoRA | 99.95 ± 0.03 | 99.90 ± 0.05 | 99.20 ± 0.28 |
| WebFace4M → NIR-VIS train | DG-LoRA (ours) | 99.96 ± 0.02 | 99.96 ± 0.02 | 99.55 ± 0.13 |
As the results show, the model adapted with depth-graded allocation outperforms the uniform-rank adaptation on both zero-shot and fine-tuned NIR-VIS face recognition. Further experiments on cross-spectral FR by adapting the foundation models can be studied in future work.
- Reply to Weakness #7: As mentioned in the Reproducubulity Statement, we will release the code upon acceptance of the paper.
- Reply to Question #1: Following the reviewer's suggestion, we added a comparison with FRoundation at low FAR on IJB-B and IJB-C in the revised version of the paper. The following table shows the comparison of our adaptation of CLIP ViT-L/14 with DG-LoRA and DG-DoRA (r: 16 → 256) on WebFace4M against FRoundation:
| Method | Adaptation | IJB-B @ 1e-6 | IJB-B @ 1e-5 | IJB-B @ 1e-4 | IJB-B @ 1e-3 | IJB-C @ 1e-6 | IJB-C @ 1e-5 | IJB-C @ 1e-4 | IJB-C @ 1e-3 |
|---|---|---|---|---|---|---|---|---|---|
| FRoundation | LoRA, r = 16 | 40.90 | 80.77 | 90.05 | 94.95 | 78.32 | 87.35 | 92.78 | 96.34 |
| FMFace (ours) | DG-LoRA, 16 → 256 | 42.33 | 86.55 | 93.32 | 96.31 | 82.01 | 90.90 | 95.30 | 97.66 |
| FMFace (ours) | DG-DoRA, 16 → 256 | 40.34 | 85.50 | 93.00 | 96.34 | 80.89 | 90.74 | 95.09 | 97.48 |
As the results show, our method outperforms FRoundation at all low FAR values on IJB-B and IJB-C datasets.
- Reply to Question #2: We use the model at the last epoch for evaluation. We clarified this in the revised version of the paper.
We hope we could adequately address the reviewer's concerns. In case reviewer found our reply convincing, we appreciate it if the reviewer can increase their scores and rating. We are happy to continue the discussion if any part remains unclear.
Authors Reply to Reviewer Z929 [1/2]
We thank the reviewer for their time in reviewing our paper and for their insightful comments. Below, we tried our best to address the remaining concerns raised by the reviewer:
Reply to Weakness #1: We thank the reviewer for this suggestion. We reran the experiments and revised the paper by updating the results in Table 2 and 4. In the revised version, we ensure that tables 2, 4, and 5 are consistent (output projection trained, Partial FC head). After the revision, the conclusions remain valid.
Reply to Weakness #2 (checkpoint): We use the model at the last epoch for evaluation. We clarified this in the revised version of the paper.
Reply to Weakness #2 (other foundation models): In addition to the results reported in the paper for CLIP ViT-L/14 and ViT-B/16, in the revised version of the paper, we also provide results for adapting DINO and EVA-CLIP with depth-graded rank allocation. The following table shows the comparison of our adaptation of CLIP ViT-L/14, EVA02-CLIP ViT-L/14 and DINOv2 ViT-L/14 with DG-LoRA (r: 16 → 256) on CASIA-WebFace:
As can be seen, our method can be used with other foundation models as well. In particular, we observe that DINOv2 achieves the highest average verification accuracy (0.95 points above CLIP) and the highest TAR on IJB-C (83.31 against 82.22 for CLIP), while CLIP is the best on IJB-B (77.51 against 76.85 for DINOv2). EVA02-CLIP is 2.40/1.64 points below CLIP on IJB-B/C. We also report complete spectral probe analysis for these models in the appendix which confirms our findings.
As the results show, our method outperforms FRoundation at all low FAR values on IJB-B and IJB-C datasets.
Reply to Weakness #4: Following the reviewer's suggestion, we added variance information to all tables for LFW, CFP-FP, AgeDB, CALFW, and CPLFW benchmarks. For comparison with baselines, we also reported TAR for low values of FAR (1e-6, 1e-5, 1e-4) on IJB-B and IJB-C in the appendix B of the revised version of the paper.
Reply to Weakness #5: We agree that when a face recognition model is deployed, its inference cost outweighs its training cost. However, we should note that our method does not change the inference cost. After training, the low-rank update can be merged into the weights. Hence, every adapted model in our tables has the architecture and inference cost of the CLIP ViT-L/14 image encoder. Our main claim is about accuracy at a fixed training cost. Depth-graded allocation improves IJB-B/C over uniform-rank adaptation and over other methods at the same inference cost. Training also costs the same as uniform adapters with same training budget, where the number of trainable parameters is the same, and unlike adaptive-rank methods such as SoRA, DyLoRA and PRILoRA, it needs no search or pruning. We should highlight that parameter-efficient adaptation still has practical value. Training 14M instead of 304M parameters needs no gradients or optimiser states for the frozen parametrs (304M). When inference cost is the constraint, a smaller encoder can also be used. Table 6 reports this trade-off for ViT-B/16, with 3.5 times fewer encoder parameters, which reaches 88.98/92.17 TAR at FAR = 1e-4 on IJB-B/C4. In the revised paper, we state explicitly that all compared adapted models have the same inference cost, and we have toned down the efficiency wording.