Efficient Resource-constrained Backpropagation through Quasi-stable Layer Freezing

Recent studies have shown that different layers in a neural network converge at different stages of training. As a result, updating all layers throughout the entire process is often unnecessary and leads to significant computational overhead. Motivated by the observation that each layer progressively stabilizes its representation transformation, we introduce Quasi-stable Layer Freezing (QL), which dynamically estimates layer-wise transformation stability and skips updates to quasi-stable layers. This strategy eliminates a large portion of redundant backpropagation and significantly reduces training cost. QL is designed in the context of on-device learning, where models are trained under severely constrained computational resources. Experiments on standard benchmarks demonstrate that QL reduces the computational cost of backward pass by up to approximately 96% in terms of FLOPs, and achieves speedups of 16.7$\times$ on a Raspberry Pi 5. We will release the code upon paper acceptance.