- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates conditioned direct feedback alignment (nDFA), a family of learning rules that improves credit assignment in fixed random feedback networks by applying inverse-second-moment conditioning to either the presynaptic activity or the local error signals. Through a linearized spectral analysis and extensive experiments across synthetic and real-world datasets like CIFAR-10, the authors demonstrate that activity conditioning mitigates the distorting effects of high-variance nuisance directions, thereby stabilizing training and improving optimization speed compared to tuned DFA baselines. While activity conditioning provides robust gains and a favorable early-only efficiency tradeoff, the study establishes that error conditioning offers more limited benefits under stable training and that these geometric interventions do not resolve the fundamental stationary mismatch inherent in random feedback learning.
Critical strengths
-
The paper provides a geometric characterization of DFA failure modes by demonstrating how activity anisotropy weights task-irrelevant nuisance directions, and it proves that nDFA effectively reduces the input-side condition number to reweight these updates toward task-relevant directions.
-
The authors establish a practical and statistically validated efficiency tradeoff through early only activity conditioning on CIFAR-10, showing that applying inverse-moment operators during initial training phases retains most accuracy gains while significantly reducing the total computational work compared to persistent conditioning.
Critical weaknesses
-
The authors find that error conditioning improves short budget models but its additional benefit fails to survive the global correction under stable training on CIFAR-10. This outcome leaves a gap between the theoretical family of two factor conditioned DFA and the empirical evidence, as the study does not isolate whether this failure stems from the low rank nature of gated DFA errors or from an inherent limitation of the error side inverse moment normalization when combined with activity conditioning.
-
The primary empirical hypothesis suggests that nDFA should help most when high variance directions are task irrelevant nuisance, yet this prediction is not confirmed on the image background benchmarks. The failure to transfer synthetic nuisance geometry to real image backgrounds indicates that the spectral learning rate analysis may rely on an idealized task orientation assumption that does not hold for the complex nuisance structure found in naturalistic datasets.
Questions for authors
For weakness 1: Provide further evidence or a more detailed ablation to determine if the fragile estimation of error moments is the primary bottleneck preventing error conditioning from yielding sustained gains in converged models?
For weakness 2: Given that the predicted benefits on image background tasks were not confirmed, how do the authors reconcile the gap between the synthetic spectral gain and the realized performance on real world datasets where nuisance directions may not be as clearly separated?
3 comments
additional tests
Following our earlier response, we would like to add new evidence on your two main questions. Studies undertaken in response to Reviewer Jpt2's follow-up now include the diagonal-error control and a direct geometry intervention. We appreciate your role in motivating this work, which clarifies both the value of error conditioning for fixed-feedback learning and the role of background geometry. The new studies are in Appendix C and Fig. 7; main Fig. 5A,E highlight the error controls and prospective orientation comparison.
Additional evidence on error-moment estimation. A new, separately frozen study on 48 fresh seeds adds the diagonal-error control to the earlier comparison against activity conditioning alone (A). With averaged moments, it meets the original five-part criterion under its own correction family. The full error factor beats A by +0.70 points [95% CI 0.52, 0.87] and a diagonal error factor, tuned over ten settings, by +0.82 [0.65, 0.99]; no diagonal-factor gain is statistically established. Without re-selecting the error settings, the gain over A is +1.05 [0.89, 1.22] at batch 128 and +0.73 [0.56, 0.91] at batch 512. This establishes an additional benefit from the full error factor under the tested averaged-moment recipe and across these batch sizes.
Additional evidence on background geometry. Your question motivated a direct intervention: we rotated every background relative to the class-mean subspace with its spectrum, foregrounds and labels unchanged. The intervention establishes that background orientation affects the benefit of conditioning: in the paper's mixtures conditioning gains 2.7 and 2.1 accuracy points when the leading background variance masks the class-mean directions, and loses 0.2 and 0.3 when it is orthogonal to them. The interaction has the same sign in all 20 seeds; BP shows the same dependence. We then recorded the sign of this dependence as a prediction before training two further mixtures, and it was confirmed in all eight declared tests (20 fresh seeds); BP again shows the same dependence there. This is a prospective replication across mixture strengths using the same images and rotations.
The intervention sharpens our earlier interpretation. The ratio predicts the opposite ordering, so we now retain it as a descriptive measurement rather than the predictive boundary proposed in our first reply. Input ablations show that the advantage of conditioning remains detectable with the class-mean component of the input removed, including in all four new cells. The stronger prediction of no advantage with that component alone held in two of four. This supports a more precise hypothesis about usable signal beyond class means, while leaving the full mechanism open. Your questions helped us move from a plausible account to controlled evidence about when conditioning aids local learning.
Final Decision for this submission:
Thank you for the rigorous follow-up experiments and the detailed response!
-
On W1, it shows that off-diagonal error structure is useful, and that the earlier null result was not an inherent limitation of error-side conditioning.
-
On W2, the background-rotation intervention shows that the benefit of conditioning depends on how background variance is oriented relative to the class-mean subspace. The revised abstract states clearly that the predicted image-background benefits are not confirmed. I find this scoping appropriate.
My concerns have been addressed, and I will raise my rating to 3. For confidence, I do not have direct research experience in this field so I will keep 2. Hope you may understand.
Thank you for identifying these two questions. We designed follow-ups for both. They establish an additional error-side benefit and provide a direct measurement of the geometric difference between the synthetic and image-background tasks.
1. Is error-moment estimation the bottleneck?
Better estimation helps, while the larger study also resolves a benefit with the original estimator. On the stable CIFAR-10 protocol, with 48 paired seeds, adding the error factor early to activity-only conditioning (A) gives:
The averaged recipe also beats the minibatch rule by +0.37 points. All six accuracy and cross-entropy comparisons pass Holm correction. The minibatch estimate is consistent with the original small positive estimate; both the larger sample and smaller correction family improve sensitivity. This follow-up tests the gain over A, not the original full criterion, which also required a diagonal-error control.
We separated estimation quality from the other recipe changes in two ways. First, at saved model states the minibatch-conditioned direction agrees with a pooled reference at cosine 0.72–0.81, versus 0.93–0.96 for an independent pooled estimate. Second, in a Transformer control that changes only error-moment averaging, averaged moments improve test perplexity from 232 to 189 in 12/12 seeds. This isolates an averaging benefit more directly than the MLP recipe, which also changes calibration and inverse refresh.
The Transformer results extend the error-side benefit beyond the MLP. In the 36M Transformer it lowers DFA test perplexity from 391 with A to 195; the full error factor also beats its diagonal control. At three times the training budget, the corresponding comparison is 264 to 145, with the same absolute early-error window. Concentrated local errors therefore coexist with useful conditioning. These are fixed-budget results, not a claim about converged models or a complete explanation of the mechanism (Fig. 5A–C; Appendix M.1–M.2).
2. Why did the synthetic prediction not transfer to image backgrounds?
We now measure the relevant geometry by redrawing backgrounds while holding each foreground fixed. This separates background variation from foreground-locked variation. Applying the inverse activity operator changes the class-mean-to-background energy ratio as follows:
The operator enriches this task proxy relative to nuisance in the synthetic nuisance regime, but not in the image mixtures. Their class means overlap high-variance directions, so visible background does not by itself establish the task orientation assumed by the spectral analysis. This gives a measurable boundary to the prediction (Fig. 5E; Appendix M.4).
These three-seed measurements use class means as a task proxy, so they support a geometric interpretation rather than a complete causal account. The denser-checkpoint audit agrees with the original benchmark outcome. The revision now provides both the confirmed error-factor gains and the geometry measurements requested in your review.