acceptodds
Under review as a conference paper at ICLR 2027

Better Aligned, Trains Worse: Intervening on Gradient Alignment in Local Learning

Abstract

Local learning rules replace backpropagation's transposed weights with a feedback matrix and are routinely checked by per-layer gradient alignment. A better-aligned matrix is read as one whose credit trains better, a premise tested so far mostly by correlation across runs. We show that alignment can prefer the feedback matrix that trains worse, and that following its preference lowers accuracy. We study the weight mirror, which learns its matrix from random probes. We run it at the published mirror learning rate with few probes and at a tuned setting with a larger rate and more probes. Each run also carries the other setting's matrix without using it, so both matrices are measured on the same forward weights. Measured this way, in a residual network with and without shortcuts, alignment prefers the matrix of the setting that trains worse. We then fork tuned runs. From one snapshot, one branch keeps its matrix and the other lets the preferred matrix compute the weight updates. That branch ends with lower accuracy in each seed, by 1.12 percentage points with shortcuts and 2.73 without. Alignment scores for credit assignment should be validated by intervention, not by correlation.

Primary area:
optimization
Keywords:
gradient alignment, weight mirror, credit assignment, local learning rules, feedback alignment, evaluation diagnostics, backpropagation alternatives

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_fzMZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Local learning denotes the idea to replace the usual adjoint of a linear layer in the computation of the weight gradients via backpropagation in a neural network with a learnable matrix. This paper studies the idea to measure the effectiveness of local learning via the per-layer alignment of the resulting update vector with the true gradient. It demonstrates that the commonly used alignment metric can fail, i.e., that a better alignment can yield networks that have lower test accuracy. The experiments focusses on ResNet-18 and similar networks without skip connection and evaluates image classification on CIFAR-10 after 15 epochs as a testbed.

Critical strengths

For future works relying on local learning, it is very important to understand the limitations of the commonly used technique of predicting the quality of the local learning from the resulting alignment with the true gradient. Thus, this work could be of interesting for the local learning community.

Critical weaknesses

Admittedly, I am far from being an expert on the topic. Yet, due to the following reasons the work was very difficult to follow for me.

  • It uses terms like "layer's credit", "weight transport", "feedback matrix", "differing in alignment at their own weights", or "transposed of forward weights" to denote certain mathematical quantities. In my opinion, it is crucial to make such terms precise with formal mathematical definitions, including the underlying assumptions on the network architectures. In particular, I think that all "transposed forward weights" are supposed to refer to the adjoint operator of a given linear layer. The current text is very descriptive and hardly contains any formula or any details on the dimensions of the involved variables.
  • Variables were used without introduction, e.g., the B_{tuned}, B_{low} and W the teaser figure refers to.
  • Formulas like θ=\arccos⁡(cos⁡)\theta = \arc\cos(\cos) do not seem to make sense.
  • The paper follows a non-standard structure with four methodological section (apart from introduction, related work, limitations, and conclusions), where I had difficulties following the train of thought.
  • There are important sentences I did not understand, e.g., "With the shortcuts restored it trains below both once one diverged run is replaced". First, "trains below both" should be explained. Does it refer to training loss or validation metric? Second, it seems like the removal of a training run which diverged is a quiet significant interference. Similarly, "Forking alone costs almost nothing" - in terms of which measure of costs?

Second, I am worried about the related work and interest of the topic to the community. Among the peer-reviewed papers (i.e. excluding arxiv preprints) in the references, only two are younger than 2022. One is Caillon et al. (2026) which Section 8 states lies outside of what the authors measured. Thus, Roy et al. (WACV 2026) is the only recent remaining peer-reviewed paper. I suggest to detail their approach much more, relate the findings at hand to their work, and make the overall motivation clearer.

Finally, the experiments (CIFAR-10 with Resnet-18, 15 epochs of training) seem very limited.

Questions for authors

To address the second and third points in the weaknesses above, can you position your work more clearly in the recent literature? To what extend are the findings important to the community? Why are the limited experiments on CIFAR-10 sufficient to draw general conclusions?

Reviewer_CQcQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The author investigates the relationship between gradient alignment and performance for the weight mirror, an algorithm that addresses weight transport by training feedback weights to align with feedforward weights. Using two ResNet18 configurations, they show that feedback weight with higher gradient alignment could lead to lower final performance.

Critical strengths

The author studies interesting questions in the field of local learning. Many local learning methods (such as weight mirror, predictive coding, equilibrium propagation) attempt to approximate, or improve the approximation of, the backpropagated gradient to achieve good performance. However, the relationship between gradient alignment and final performance is not necessarily positively correlated. Therefore, studies on the relationship between gradient alignment and performance are meaningful for local learning.

Critical weaknesses

Although the broad question is interesting, the paper has two major weaknesses.

  1. The scope of the study and the methods have limitations: the study is focused on weight mirrors and one pair of specific ResNet configurations. The weight mirror only addresses the weight-transport problem, but the weight update rule is not local. It is therefore inappropriate to equate weight mirror with local learning. On the other hand, if the paper discards the claim about local learning and focuses solely on weight mirror, then empirical measurements on one pair of configurations for a specific algorithm do not have enough impact. It would be more interesting to either have empirical studies across multiple algorithms or have theoretical contributions on relationships between gradient alignment and performance.
  2. The evidence from the experiments is also not sufficient to support the claim that better alignment leads to worse performance. Angles of alignment are computed as an average over layers and epochs. However, average values do not capture changes in alignment throughout training or across depth. Alignment across depth or at different phases of training may impact the performance differently. Also, 15 epochs is an arbitrary number for comparing performance. Has the training loss/top-1 accuracy converged?
Questions for authors

Questions:

  1. Forking in the middle changes more than just the alignment: feedback magnitude, rank, and dynamics could all be meaningful for good performance. If we just look at the alignment without forking, "tuned" configuration shows better performance than “low budget”. If we consider forking, handing “low budget" B to “tuned” configuration also improves the performance (Table 3b). This evidence suggests “better aligned, better performance”. How would you reconcile this with your conclusion?
  2. Could you elaborate on the meaning of “from the same incoming error” in line 131? Mathematically, what is the incoming error? And how is credit computed?
  3. Why vary both learning rate and probe numbers between the two settings? It would be more interesting to isolate the effect of each knob. Figure 2 seems to suggest that given a fixed learning rate, more probes lead to better alignment and better performance. Is this also true for other learning rates?
  4. There is an iterative trainer and a sequential trainer. What is the reason for having two different trainers? It is not very clear from the writing.
  5. In the end, almost all local learning literature validates their methods with empirical performance. If the thorough empirical studies indeed find “better aligned, worse performance”, how might such conclusions influence future local learning research?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.