- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Local learning denotes the idea to replace the usual adjoint of a linear layer in the computation of the weight gradients via backpropagation in a neural network with a learnable matrix. This paper studies the idea to measure the effectiveness of local learning via the per-layer alignment of the resulting update vector with the true gradient. It demonstrates that the commonly used alignment metric can fail, i.e., that a better alignment can yield networks that have lower test accuracy. The experiments focusses on ResNet-18 and similar networks without skip connection and evaluates image classification on CIFAR-10 after 15 epochs as a testbed.
Critical strengths
For future works relying on local learning, it is very important to understand the limitations of the commonly used technique of predicting the quality of the local learning from the resulting alignment with the true gradient. Thus, this work could be of interesting for the local learning community.
Critical weaknesses
Admittedly, I am far from being an expert on the topic. Yet, due to the following reasons the work was very difficult to follow for me.
- It uses terms like "layer's credit", "weight transport", "feedback matrix", "differing in alignment at their own weights", or "transposed of forward weights" to denote certain mathematical quantities. In my opinion, it is crucial to make such terms precise with formal mathematical definitions, including the underlying assumptions on the network architectures. In particular, I think that all "transposed forward weights" are supposed to refer to the adjoint operator of a given linear layer. The current text is very descriptive and hardly contains any formula or any details on the dimensions of the involved variables.
- Variables were used without introduction, e.g., the B_{tuned}, B_{low} and W the teaser figure refers to.
- Formulas like do not seem to make sense.
- The paper follows a non-standard structure with four methodological section (apart from introduction, related work, limitations, and conclusions), where I had difficulties following the train of thought.
- There are important sentences I did not understand, e.g., "With the shortcuts restored it trains below both once one diverged run is replaced". First, "trains below both" should be explained. Does it refer to training loss or validation metric? Second, it seems like the removal of a training run which diverged is a quiet significant interference. Similarly, "Forking alone costs almost nothing" - in terms of which measure of costs?
Second, I am worried about the related work and interest of the topic to the community. Among the peer-reviewed papers (i.e. excluding arxiv preprints) in the references, only two are younger than 2022. One is Caillon et al. (2026) which Section 8 states lies outside of what the authors measured. Thus, Roy et al. (WACV 2026) is the only recent remaining peer-reviewed paper. I suggest to detail their approach much more, relate the findings at hand to their work, and make the overall motivation clearer.
Finally, the experiments (CIFAR-10 with Resnet-18, 15 epochs of training) seem very limited.
Questions for authors
To address the second and third points in the weaknesses above, can you position your work more clearly in the recent literature? To what extend are the findings important to the community? Why are the limited experiments on CIFAR-10 sufficient to draw general conclusions?