acceptodds
Under review as a conference paper at ICLR 2027

LocalProp: Neuro-Localized Memory-Efficient Backpropagation

Abstract

The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor memory efficient, since learning inside the brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a pruning operation is performed, followed by a short final fine-tuning phase. Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several datasets, including large-scale benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-memory trade-off. We publicly release our code at https://see.supplementary.

Primary area:
optimization
Keywords:
neural network optimization, efficient learning, neural network training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_huLZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a four-stage training pipeline for image classification task. The whole pipeline uses local backpropagation, reducing 58% peak memory usage for CIFAR-10 fine-tuning. However, the accuracy degradation is substantial on various datasets.

Critical strengths

None

Critical weaknesses
  1. The research scope of this paper is narrow. The authors only show how their methods can be used for image classification task to reduce GPU memory usage. It is not clear how this method can be transferred to other tasks where I-JEPA training is unavailable.

  2. For classification tasks, the accuracy is bad in fact. Other methods like activation compression training (for example, Mesa https://github.com/ziplab/Mesa) can perform as well as full training and reduce up to 50% GPU memory usage.

Questions for authors

Is 58% lower peak memory usage for only finetune stage?

What is the peak memory usage and wall time for the whole pipeline?

Which AI tool do you use? There is no clear model information provided in AI USE STATEMENT.

Reviewer_A8kaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training pipeline that restricts backpropagation to individual blocks or small groups of blocks. It combines local I-JEPA self-supervised pre-training, supervised learning with block-specific classifiers, structured pruning based on the effect of component ablation on the final classification loss, and local recovery after pruning. Experiments use ViT-S and ResNet-50 on CIFAR-10, CIFAR-100, and ImageNet-1K. The paper evaluates classification accuracy, peak GPU memory, fine-tuning time, pruning strategies, and representation development. Its central finding is that limiting gradient propagation reduces memory consumption, while increasing the propagation span can recover accuracy lost under strictly local training.

Critical strengths

The paper has exhaustive and comparable results. Studying local pre-training, supervised adaptation, pruning, and recovery using both convolutional networks and transformers is useful. The phase comparisons show that local recovery restores much of the immediate pruning loss, and the representation and gradient analyses help explain the limitations of strictly local training.

Critical weaknesses
  1. The methodological contribution and the benefit of pruning are insufficiently isolated. The pipeline combines block-local objectives, auxiliary classifiers, component-ablation scoring, and recovery training. The authors need to explain more clearly what this combination contributes beyond applying familiar operations throughout the training pipeline.

  2. It is unclear if the four-block configuration retains its favorable accuracy–memory trade-off on ImageNet. It is better to compare it with relevant local-learning methods[1-3] and memory-saving full-backpropagation training, such as activation checkpointing, under consistent experimental settings.

  3. What explains the reduced pretrained models’ ImageNet results in Table 1? ViT-S and ResNet-50 achieve 6.09% and 1.49%, respectively, compared with 21.48% and 47.68% when trained from scratch. Are these optimization failures, different configurations, or reporting errors?

[1] Kappel, David, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam, Christian Mayr, and Anand Subramoney. "A variational framework for local learning with probabilistic latent representations." In 5th Workshop on practical ML for limited/low resource settings.

[2] Zhang, Aozhong, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, and Penghang Yin. "Comq: A backpropagation-free algorithm for post-training quantization." IEEE Access (2025).

[3] Cheng, Anzhe, Heng Ping, Zhenkun Wang, Xiongye Xiao, Chenzhong Yin, Shahin Nazarian, Mingxi Cheng, and Paul Bogdan. "Unlocking deep learning: A bp-free approach for parallel block-wise training of neural networks." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4235-4239. IEEE, 2024.

Questions for authors

Please see the weakness above.

Reviewer_RkDROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework called LocalProp that aims to lower the cost of back-propagation. The authors propose to use (1) local I-JEPA training, that only updates a selected neural block; (2) local supervised fine-tuning, that trains block-specific linear classifiers that provide supervision signals; and finally (3) a local structured pruning. The authors show that the proposed algorithm achieves better performance compared to baselines such as Local FT.

Critical strengths
  • The presentation of the paper is good and therefore the algorithm is easy enough for understanding.
  • A multi-stage pipeline may allow for other applications other than local back propagation.
Critical weaknesses
  • The authors' writing seems to suggest that the motivation is coming from forward-forward algorithms (or no back-propagation) but in fact the authors' algorithm does have back-propagation. Also, [1] has done experiments on ImageNet as well, and somehow it seems that authors did not put it in the introduction when discussing the drawbacks of previous algorithms.

  • I did not fully understand the necessity of having a pruning pipeline. It seems like this work is already self-contained even without the pruning module. The best performance in terms of accuracy is coming from no pruning, and also it seems the pruning here can be quite generic (i.e. any structure pruning can work?).

  • It seems that the performance of VIT-S on CIFAR-100 is from fine-tuning but not from training from scratch.

  • The algorithm does not seem to be consistent for ViT-S and ResNet. The definition of "local" is different for these two architecture from my reading. Could it be the reason that in Table 2 there is no result of peak allocated memory?

[1] Blockwise Self-Supervised Learning at Scale

Questions for authors

Please read above weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.