- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates an early-training failure mode of direct feedback alignment (DFA): hidden representations become strongly concentrated and learning stalls near the loss of a class-prior predictor. The proposed explanation is that a shared component of output error, acting through fixed random feedback, mean presynaptic activity, and trainable biases, shifts tanh units toward saturation before the readout corrects the shared error.
The paper derives an exact mean–covariance decomposition and develops approximate dynamics for mean drift and sensitivity concentration. Predictions are evaluated across 48 settings and additional width–depth configurations. Frozen-feature experiments distinguish retained class decodability from slow readout optimization, while interventions involving calibration, centering, feedback strength, and optimizer choice test the mechanism and its limits. Related effects are examined in deeper networks, convolutional networks, and alternative objectives.
Critical strengths
A useful mechanistic account with quantitative predictions. The paper connects shared output error to hidden mean drift, saturation, and concentrated sensitivity, and evaluates a reduced model initialized from the network without fitted trajectory parameters. Its agreement across multiple experimental settings makes the contribution more informative than a descriptive observation of training plateaus.
A convincing distinction between decodability and optimization speed. The frozen-feature decoding and readout continuation experiments show that collapsed representations can retain substantial class information while supporting slow SGD learning. The centering and scalar-rescaling controls provide particularly useful evidence for an amplitude bottleneck at a fixed optimizer scale.
Critical weaknesses
The paper studies a specialised fixed-feedback learning rule, with its strongest quantitative evidence concerning small tanh networks under fixed-rate SGD. Adaptive optimization substantially shortens the stall, while the interventions have mixed effects across other settings. The paper should more clearly explain which important scientific or practical problem benefits from this analysis, and what guidance or transferable insight it provides beyond the examined transient. This affects my assessment of the contribution’s significance, rather than its algebraic correctness.
Questions for authors
- Scope and causal contribution of the rank-one approximation.
Equation (11) decomposes the shared-output-error update into a rank-one mean term and a gate–activity covariance term. Error centering removes both. Figure 6f measures the relative norms of these two contributions, while Table 11 reports their related mean-drift diagnostics, including substantial upstream contributions at initialization. Could the authors provide corresponding absolute component magnitudes and signed drift contributions for the two-class and squared-error settings where the reduced model misses gate concentration (Figure 2d)? In particular, is the discrepancy driven by changes in the mean-error signal during training, gate–error covariance, upstream feature motion, or other approximations in the reduced model? An intervention that separately removes the rank-one and covariance contributions would also help clarify their respective causal roles.
- Separating feedback orientation from the delivered mean-error drive.
The pre-aligned feedback control matches the Frobenius norm of the random feedback matrix, but this does not ensure that the projected mean-error signals have equal magnitudes. The paper acknowledges that the control therefore does not isolate the restoring contribution in Equation (5). Could the authors quantify the magnitude of the gated broadcast mean-error signal and the signed restoring contribution separately? If feasible, a control that matches the initial delivered mean-error magnitude while varying feedback orientation would help determine whether the reduced collapse and faster learning are primarily associated with the restoring direction or changes in the strength of the shared drive.
- Concrete utility and benefits under adaptive optimization.
What is the principal scientific or practical use of this analysis—for example, a transferable principle for biological credit assignment, guidance for hardware-constrained local learning, or improved DFA training? Please connect that use to the demonstrated evidence.
2 comments
Response to rebuttal and updated rating
Thank you for the detailed rebuttal. The added causal controls and drift diagnostics thoroughly address my technical questions, and the two adaptive-optimizer examples covering sign feedback under Adam and the normalized Transformer grokking task under AdamW help demonstrate utility beyond small tanh networks with fixed-rate SGD. For the final version, I hope the authors can further expand the validation and discussion on modern architectures. Overall, the paper is significantly strengthened, and I am raising my score from 2 to 3.
Thank you for proposing specific causal tests. We added selective removal, signed drift measurements and shared-only routing; Figure 5 brings the orientation and adaptive-optimizer results into the main text.
Rank-one versus covariance. Removing the shared rank-one weight and bias terms preserves approximately the initial feature variance; removing shared gate–activity covariance alone leaves baseline compression nearly unchanged (Table 33). Removing shared error only in lower layers preserves 88% of top-layer variance and advances the 80% accuracy crossing from 1,200 to 910 updates (Table 36). All retain the true readout gradient and fluctuating-error term, including the gate–error residual.
Absolute and signed drift. Table 39 reports top-layer displacement norms, with signed projections divided by total displacement magnitude in parentheses. At initialization, means across three seeds are:
By update 50, squared error's rank/bias fraction rises to +0.75, while shared covariance remains near zero. Figure 22 retains the fluctuating-error and finite-step cross terms. These measurements distinguish upstream motion from local shared covariance.
Two checks locate the approximation errors. For squared error, supplying observed mean error reduces participation-minimum error from 0.390 to 0.035, implicating readout dynamics. For two classes, class-conditional Gaussian moments reduce participation reconstruction error from 0.510 to 0.009 at update 300 (Figure 21). These are measured-data diagnoses, distinct from initialization-only predictions.
Orientation at matched initial strength. Aligned, random and reversed feedback have initial mean-gated probe norms of 19.8 (rounded) in all three arms. Their signed restoring quantities
q = ebar^T Wout Gamma_bar^2 B ebarare +31.06, +0.06 and −31.06. Here ebar is mean output error, Wout the readout, B the top-layer feedback matrix, and Gamma_bar the diagonal matrix of mean activation derivatives. Positive q denotes a corrective hidden contribution under the mean-only closure. Their minimum feature variances over the first 300 updates are 63%, 2.8% and 0.05% of initial values (Figure 5a; Table 35). The training-sample norm match does not constrain later accumulated drive.The trajectory control uses
B etilde + Bprime ebar: the fluctuating pathway stays random while shared error uses fixed aligned or reversed feedback. Shared-only alignment preserves 65% of initial variance versus 3% for ordinary DFA and reaches 80% accuracy in 920 versus 1,200 updates. Whole-error alignment preserves 63% and reaches it in 290 (Table 36), identifying an additional benefit from aligning input-dependent feedback.Same-state virtual steps hold the fluctuating gradient exactly fixed. Alignment improves the measured top-layer-only loss decrease at every validation checkpoint in every seed at the nominal rate (Figure 19b). This establishes a local directional benefit alongside the training comparisons.
Utility under adaptive optimization. Centering persistent sign feedback advances the 80% crossing from 370 to 100 updates under Adam, with every seed improving; the benefit also holds at the original feedback scale. In a normalized Transformer it speeds fitting in all eight original pairs. At AdamW decay 0.03, it also brings forward sustained 95% held-out accuracy in all eight fresh pairs, with a 19% median paired reduction (Figure 5c; Table 47). The generalization benefit is regime-specific. The practical lesson is to control dominant shared drive separately from useful input-dependent feedback.
Thank you for proposing these controls. They let us distinguish the shared signal’s strength and direction and separate local from upstream contributions.