acceptodds
Under review as a conference paper at ICLR 2027

On the Role of Balanced Initialization in Feedback Alignment

Abstract

Feedback alignment (FA) and its variants replace exact backward weights with fixed random feedback matrices, and have recently been argued to retain important properties of gradient-based learning. We identify a fundamental distinction between these dynamics through balancedness, a key invariant of gradient flow in deep linear networks. We show that Random Feedback Alignment (RFA) generically fails to preserve balancedness: even from exactly balanced initialization, the balancedness defect departs from zero. We prove, under mild nondegeneracy conditions, that exact balancedness can occur only at isolated times. Surprisingly, balanced structure nevertheless remains beneficial: in nonlinear networks, balanced initialization improves RFA training over Xavier initialization, and this advantage persists after matching the layerwise norms of the Xavier initialization. These results separate the benefits of balanced structure from its exact conservation, and expose a structural difference between feedback alignment and backpropagation.

Primary area:
learning theory
Keywords:
Feedback Alignment, Biologically Plausible Learning, Implicit Bias, Continuous-Time Analysis, Initialization, Deep Linear Networks, Ordinary Differential Equations, Dynamical Systems

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dW3MOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies random feedback alignment (RFA) dynamics. The authors show that the classical W2⊤W2−W1W1⊤W_2^\top W_2 - W_1 W_1^\top conserved quantities under GF dynamics for a 2 layer linear NN are not conserved under RFA dynamics. The authors give experimental evidences that balanced initializations are beneficial for RFA training for a non linear NN.

Critical strengths

S1. The proof of Theorem 1 is rigorous

S2. It is interesting that balanced init could be beneficial for RFA training

Critical weaknesses

W1. The introduction is not well written

W2. The motivation of the paper is not clear. Why do we care about Theorem 1? In particular, while it is well known that balancedness is an invariant of gradient flow (which the authors refer to as backpropagation), the RFA dynamics that the present paper studies has no reason to share this invariant.

W3.There is a confusion between gradient flow and gradient descent in this paper. Line 139-146: it is GF that preserves δ\delta-balancedness, not GD. For GD, the conserved quantities remain approximatively conserved, not exactly conserved.

W4. The link between conservation and balancedness initialization is not well motivated. The authors give experimental evidence that balanced init is beneficial for RFA training for non linear networks. However, for non linear networks the quantity ∥∣W2⊤W2−W1W1⊤∥∣F\||W_2^\top W_2 - W_1 W_1^\top\||_F is not preserved.

Questions for authors

please comment the weaknesses above

Reviewer_cEhpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines balancedness under random feedback alignment (RFA). In a two-layer linear network initialized at zero, it shows that ordinary adjacent-layer Gram balance is lost immediately and can recur only at isolated times under the stated nondegeneracy condition. It then applies an existing balanced initialization scheme to nonlinear RFA networks and reports improvements over standard initializations in some MNIST and CIFAR-10 settings, including a CIFAR-10 norm-matching control. The advantage depends on feedback scale and reverses in one reported regime.

Critical strengths

The main theorem appears correct within its stated two-layer linear, zero-initialization setting. The norm-matching control is useful, and the experiments report both favorable settings and a feedback-scale regime where Xavier performs better.

Critical weaknesses
  1. The theorem provides very limited new insight. The balancedness identity for linear-network gradient flow arises from a specific cancellation between adjacent layer updates. Replacing the backward transpose by an unrelated fixed matrix removes that cancellation. The paper’s zero-initialization calculation makes this especially transparent: the first layer starts moving at first order in time, whereas the second starts at second order. The isolated-times refinement then follows from standard analyticity and the identity theorem. I do not find this a surprising structural discovery or a substantial technical contribution. More importantly, the theorem provides no quantitative control of approximate balance beyond the local departure calculation, nor a consequence for convergence, implicit bias, or the benefit of the nonzero balanced initializations used in the experiments.
  2. The paper does not establish the relevance of this linear calculation to the training dynamics it seeks to explain. Deep linear networks are a useful mathematical model, but conclusions about their special invariant structure do not automatically explain nonlinear learning. Here the theory concerns a deterministic two-layer linear flow from zero, while the experiments concern nonzero initialization, nonlinear activations, and finite-step stochastic optimization. The paper does not show that the identified departure from balance controls, predicts, or explains the behavior in those experiments. In a mature literature, a further elementary property of this model needs either a substantive mathematical consequence or a convincing connection to learning behavior. I do not see either established here.
  3. The discussion of prior work does not adequately situate the contribution. The paper cites Arora et al. (2019), which includes Nadav Cohen, but its account of balancedness omits the earlier Arora, Cohen, and Hazan (2018), On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization, as well as subsequent geometric analyses such as Bah et al., Learning Deep Linear Neural Networks: Riemannian Gradient Flows and Convergence to Global Minimizers. Govind Menon’s The Geometry of the Deep Linear Network gives a systematic account of this invariant-manifold and balanced-geometry framework, with attribution to the original results. These references matter because the paper presents the failure of one classical cancellation as a fundamental distinction without explaining what understanding it adds to this established framework. I am not claiming that these works prove the identical RFA theorem; the issue is that the paper does not establish a meaningful intellectual advance relative to the existing theory.
  4. The experiments do not supply a sufficiently informative practical contribution to compensate for the limited theory. The reported gains show that an existing structured initialization can help in particular settings. They do not establish which aspect of that initialization is responsible. Layerwise norm matching leaves singular-value distributions, rank, conditioning, and relative singular bases entangled, so it does not isolate the proposed coupling explanation. Moreover, the initialization ranking reverses with feedback scale, and an advantage over a comparably tuned baseline is not established. The authors appropriately label the mechanism conjectural, but this leaves the paper with a conditional optimization observation and no demonstrated explanation connecting it to the theorem.
  5. The framing and exposition overstate the content, and the BP comparison contains a substantive imprecision. Terms such as “fundamental distinction” and “surprisingly” assign considerable weight to a result that follows from a short local calculation and standard analytic facts. Substantial space is devoted to textbook ODE preliminaries, while the new scientific consequence remains unclear. In addition, Section 2.2, lines 141–143, attributes exact balance conservation to finite-step gradient descent. Exact conservation is a gradient-flow property; simultaneous finite-step BP generally has a nonzero second-order balance drift. The exact-versus-approximate distinction is handled explicitly in the earlier gradient-descent literature and needs to be respected in a paper centered on conservation laws.

I find the contribution insufficient for acceptance. Correctness of the stated theorem does not by itself make the result significant, and the empirical observations do not establish a new mechanism or a reliable practical advantage. My recommendation rests primarily on this lack of substantive new knowledge. Correcting the presentation and supplying additional controls would improve the paper, but acceptance would require a considerably clearer and more consequential contribution.

Questions for authors

Please comment on the above.

Reviewer_RSgFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the role of balanced initialization in Random Feedback Alignment (RFA). While under backpropagation gradient flow, the Gram-matrix is balanced, the authors show this is not preserved by RFA in a two-layer linear network initialized at zero. They further prove that exact balancedness can only occur at isolated times under a nondegeneracy assumption. This finding is complemented with experiments on synthetic linear tasks, MNIST, and CIFAR-10.

Critical strengths

The paper raises a legitimate question about whether the benefits of balanced initialization depend on its conservation during training. The observation that balanced initialization can remain beneficial under RFA, even after balancedness is lost, is potentially interesting.

Overall the paper is easy to follow and the mathematical derivations (although standard) seem correct.

Critical weaknesses

1. Very limited novelty and scientific significance.

The main theoretical result follows rather directly from the RFA dynamics and classical arguments. The nonconservation of balancedness can already be understood by directly differentiating the Gram-matrix discrepancy. Moreover, previous work has extensively studied structural differences between feedback alignment and backpropagation, including conservation laws and implicit regularization (Robertson & Koyejo, 2023, 2024), while Arora et al. (2019) already investigated the optimization benefits of balanced initialization.

There is in my opinion not a sufficiently substantial new theoretical or conceptual contribution. The paper does not establish new convergence guarantees or explain why balanced initialization improves optimization under RFA.

2. Very weak empirical validation and insufficient mechanistic evidence.

The experimental evaluation is restricted to synthetic linear tasks and relatively simple fully connected networks on MNIST and CIFAR-10. This is insufficient to establish a significant practical contribution, especially given the extensive existing literature on feedback alignment and variants.

I also think the experiments do not convincingly isolate the proposed mechanism. Matching initialization norms does not control differences in rank, singular-value spectra, or other structural properties induced by SVD-based initialization. The reported benefits could therefore arise from factors other than inter-layer singular-vector coupling. The advantage also disappears or reverses at larger feedback scales.


On a less critical (but still important) note, I would avoid using the term "biologically plausible" without further justification, as this is a strong claim. Removing the weight-transport requirement of backpropagation does not, by itself, establish biological plausibility, particularly when the proposed initialization requires explicit coordination between layers.

Questions for authors
  1. What is the main theoretical insight beyond the direct observation that RFA does not preserve the Gram-matrix balancedness invariant? What meaningful consequence does the isolated-time result establish for optimization or learning?

  2. Can the authors demonstrate that the observed improvements specifically result from inter-layer structural coupling, rather than differences in initialization rank or singular-value spectra?

  3. Can the authors show their method work on larger scales and more modern setups (ResNets on ImageNet, Transformer pretraining, etc.) ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.