- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a computation-efficient network training method (QL) that selectively updates layers upon CKA measures for each layer. The lower CKA, the more stable the layer subject to update, so that the method avoids updating such layers. Two methods to filter out layers of stability are proposed: QL_\tau and QL_\rho. QL_\tau determines layer update upon pre-defined threshold value such that if CKA exceeds the threshold, the layer is subject to weight update. QL_\rho chooses the layer subject to update depending on relative magnitude of layerwise CKA. QL applied to lightweight models (MCUNet and ViT-B/32) on various datasets. The results highlight the feasible computation-efficient fine-tuning of lightweight models.
Critical strengths
The proposed method (QL) verifies the feasibility of the proposed method on small models on several datasets.
Critical weaknesses
- Limited impact. As the authors indicate in the manuscript, the proposed method is for on-device fine-tuning lightweight models. This significantly undermines the impact of this work.
- Little improvement from previous method. The comparisons with previous methods like SU indicate marginal improvements as shown in Table 1. This method may save several FLOPs by skipping weight updates for stable layers while not reducing any space complexity.
- Lack of scalability. A large impact may come from a larger coverage of the proposed method rather than the two relatively small models. The manuscript is missing such scalability demonstration.
Questions for authors
- What about QL_\rho on ImageNet, which is not reported in the manuscript?
- Have the authors checked any scalability of this method?