acceptodds
Under review as a conference paper at ICLR 2027

Efficient Resource-constrained Backpropagation through Quasi-stable Layer Freezing

Abstract

Recent studies have shown that different layers in a neural network converge at different stages of training. As a result, updating all layers throughout the entire process is often unnecessary and leads to significant computational overhead. Motivated by the observation that each layer progressively stabilizes its representation transformation, we introduce Quasi-stable Layer Freezing (QL), which dynamically estimates layer-wise transformation stability and skips updates to quasi-stable layers. This strategy eliminates a large portion of redundant backpropagation and significantly reduces training cost. QL is designed in the context of on-device learning, where models are trained under severely constrained computational resources. Experiments on standard benchmarks demonstrate that QL reduces the computational cost of backward pass by up to approximately 96% in terms of FLOPs, and achieves speedups of 16.7 on a Raspberry Pi 5. We will release the code upon paper acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
edge, training, efficient

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sZdSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a computation-efficient network training method (QL) that selectively updates layers upon CKA measures for each layer. The lower CKA, the more stable the layer subject to update, so that the method avoids updating such layers. Two methods to filter out layers of stability are proposed: QL_\tau and QL_\rho. QL_\tau determines layer update upon pre-defined threshold value such that if CKA exceeds the threshold, the layer is subject to weight update. QL_\rho chooses the layer subject to update depending on relative magnitude of layerwise CKA. QL applied to lightweight models (MCUNet and ViT-B/32) on various datasets. The results highlight the feasible computation-efficient fine-tuning of lightweight models.

Critical strengths

The proposed method (QL) verifies the feasibility of the proposed method on small models on several datasets.

Critical weaknesses
  1. Limited impact. As the authors indicate in the manuscript, the proposed method is for on-device fine-tuning lightweight models. This significantly undermines the impact of this work.
  2. Little improvement from previous method. The comparisons with previous methods like SU indicate marginal improvements as shown in Table 1. This method may save several FLOPs by skipping weight updates for stable layers while not reducing any space complexity.
  3. Lack of scalability. A large impact may come from a larger coverage of the proposed method rather than the two relatively small models. The manuscript is missing such scalability demonstration.
Questions for authors
  1. What about QL_\rho on ImageNet, which is not reported in the manuscript?
  2. Have the authors checked any scalability of this method?
Reviewer_zNXxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to reduce the backpropagation cost of on-device fine-tuning by skipping weight updates for layers that have stabilized. The authors argue that existing freezing methods need a validation set, an expensive pre-deployment search (SU), or additional memory for gradient accumulation and reference models, while LoRA-style adapters add extra parameters and so extra memory. On the other hand, QL freezes quasi-stable layers in an online manner and reversibly, either with a threshold τ\tau or a budget ρ\rho of frozen parameters. The layer stability is computed every Δt\Delta t epochs using a single scalar per layer computed through the RBF-kernel CKA method between the input and output activations of the layer on one mini-batch, and layers with the smallest DiD_i are frozen.

Critical strengths
  1. The paper is clearly written, and the proposed method is straightforward to implement
  2. The stability score is computed from activations already produced by the training forward pass, so no reference model, validation split or evolutionary search is needed
  3. The Appendix is complete and detailed, enabling full reproducibility once code is released
Critical weaknesses
  1. The comparison with the SU method is done only under matching trainable parameters, but not under same computational gain. As shown in Table 1, SU uses much less FLOPs than the proposed method, while achieving similar final accuracy. Since the goal is to boost on-device training, the comparison with SOTA should be made under same computational budget instead of same percentage of trained parameters. Morover, FreezeOut results in Table 4 seems broken.

  2. The idea of freezing layers based on representation convergence was already proposed in Raghu et al. (2017), which the paper cites only as an analysis tool intead of a direct comparison. The paper should also discuss bias-only and last-layer on-device baselines such as TinyTL (Cai et al., 2020).

  3. It is not clear if the CKA method selects better layers than simpler rules, and the only validation (shown in Fig. 3) uses a single dataset with 10 mini-batches after 2 steps. There is no comparison against trivial selections, for example random layer selection, last-k fine-tuning or a rule based on weight-change at the same ρ\rho (e.g., ∥W(t)−W(t−1)∥\|W^{(t)}-W^{(t-1)}\|).

  4. The main reported results use single runs without standard deviations computed on different seeds, so the small gains over baselines could be within evaluation noise. The authors should report experiments with different runs to have more solid results.

Questions for authors
  1. What are the total per-step FLOPs and end-to-end wall-clock (forward, input gradients, weight gradients, CKA) of QL, SU and LoRA, and which layers does QL freeze?
  2. What accuracy do random layer selection and last-kk fine-tuning reach at the same ρ\rho? How does QL compare with SU under same FLOPs rather than same trainable parameters?
  3. Are the gains over SU, LoRA and the other SOTA methods significant over multiple seeds?
Reviewer_vzciOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Quasi-stable layer freezing (QL) to freeze part of the model for constrained fine tuning. They select the layers to freeze based on its stability determined by the subspace variation - difference between CKA similarity of a layer's input and output activations at times t and t-1. They show that the subspace variation becomes small as the layer stabilises. Based on this insight they developed QL_T, a threshold based freezing and QL_rho, where rho is the frozen parameter budget. This method achieves considerable speedup with respect to vanilla full model backpropagation.

Critical strengths
  1. The proposed QL method uses CKA-based representation similarity to identify relatively stable layers and freezes them. This is practical and the trainable-parameter budget helps on-device implementation feasible. The results show convincing speedup and flop reduction at comparable performance to vanilla backpropagation.
Critical weaknesses
  1. The methods relies on CKA for identification of stable layers. However CKA is know to be highly sensitive to outliers and many transformations, as shown theoretically and empirically in [1]. In the context of this work, CKA similarity of input and output of a layer may not necessarily mean that the layer had little progress or is stable. This means that the assumption in Eq.11 of the error bound may be invalid.

  2. Comparison against simple layer selection metrics like magnitude-based selection, gradient fisher information based selection, gradient norm based selection, last k layer selection, or random layer selection are required to understand the reliability of CKA for stability measurement over these simple selection mechanisms that are being widely used. While Figure 3 tries to validate CKA as a metric for stability, it evaluates only 10 mini-batches, which is not sufficient to establish long-term stability.

[1] Davari et.al. Reliability of CKA as a Similarity Measure in Deep Learning. ICLR 2023.

Questions for authors
  1. The QL selection is based on mini-batches from the training data, which the model is already exposed to. How can you distinguish changes induced by data sampling from changes induced by training? Will the selection work as expected if an unseen and fixed dataset is used for selection to ensure bias-free selection? How sensitive is performance to the distribution of this selection set?

  2. Can this method be extended to channels and heads for further granularity and control of resource-consumption? Also, for ViT, what constitutes a layer: is it an individual linear layer or an entire transformer block?

  3. Could the analysis in Figure 3 be extended to individual layers rather than groups of layers? Since the proposed method freezes layers, the group-wise analysis may not present the variations in the importance of individual layers and the group-level conclusions may not hold at the layer-level.

  4. How frequently do the selected layers change throughout training? Do layers get selected approximately uniformly or are there layers that get selected more frequently? If yes, is there any relationship between frequently selected layers and their position in the model?

  5. Could the authors compare against a static strategy that selects layers once at initialization and maintains the same trainable-parameter budget throughout training? This would clarify whether QL's benefits come from the stability criterion itself or its partially frozen adaptation which causes a regularisation effect.

Suggestions for Future Work

  1. The experimental evaluation would benefit from comparisons against more recent resource-efficient training methods like TinyTrain (ICML 2024)[1] and AdaBet (CVPR 2026)[2], both of which address the same problem but static as well as ElasticTrainer (MobiSys 2023)[3] which is a dynamic method but focusing on speedup. Additionally, simple baselines such as last-k-layer training and random layer selection would also help the proposed method.

[1] Young D. Kwon, Rui Li, Stylianos I. Venieris, Jagmohan Chauhan, Nicholas D. Lane, and Cecilia Mascolo. Tinytrain: Resource-aware task-adaptive sparse training of dnns at the data-scarce edge. ICML 2024

[2] Tenison et. al. AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks. CVPR 2026.

[3] Kai Huang, Boyuan Yang, and Wei Gao. Elastictrainer: Speeding up on-device training with runtime elastic tensor selection. MobiSys 2023

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.