- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper seeks to develop neuro-plausible mechanisms for learning in neural networks, building on a recently introduced mechanism termed error broadcast and decorrelation (EBD). The key idea is as follows. Denoting the gradient of the loss function with respect to the activations at the last layer (prior to the softmax, if one is used) by , the authors write a first order optimality condition as , where denotes the input. This generalizes the idea behind the orthogonality principle for minimum mean squared error (MMSE) used in the EBD paper, where delta = Y – E[Y|X]. It follows that for any well-behaved function . In particular, if denotes (a function of) activations at an intermediate layer , we can use global feedback of to drive layer-wise loss functions seeking to drive to zero. This provides an alternative to backprop.
The authors also point out that the approach can be expanded by setting , where is a function that is independent of layer index . They experiment with using multiple choices of together in an “expanded” scheme.
Evaluations are provided for classification on small vision datasets: CiFAR-10 and Tiny ImageNet. The proposed schemes are not competitive with standard backprop, but the proposed approach applied with cross-entropy loss is shown to be superior to the original EBD scheme, and the expanded scheme can yield some improvements.
Critical strengths
While the observation on first order optimality conditions is a straightforward extension of the concept in the EBD paper, its application to cross-entropy does lead to improved classification performance in the evaluation. It is also applied to a toy example (Appendix B) in which the loss function is a Poisson negative log likelihood.
Critical weaknesses
My main issue with the paper is regarding its potential for intellectual impact. It is unclear what the consequences are for either machine learning (not better than backprop) or computational neuroscience (what next?).
The evaluations are for small vision datasets, and the results are still inferior to backprop.
The paper as written might give an impression of optimality in training objective to the casual reader. While standard loss functions are convex in last layer activations, they are highly non-convex in the network parameters being learnt, hence all the framework is doing is providing a shortcut from the output to intermediate layers to drive reasonable training objective. While this is obvious, it should be clearly stated.
This is an observation, not necessarily a weakness: the derivations in the appendix are rather obvious. Not sure they are all needed, but they give the impression of more sophistication than is warranted for the rather simple idea the authors are bringing to the table.
Questions for authors
I would like for you to make a stronger case for intellectual impact in machine learning and/or computational neuroscience.
4 comments
Response to reviewer comments (Part II)
small vision datasets, and the results are still inferior to backprop.
We agree that BP remains the strongest accuracy reference in the reported experiments and that larger-scale evaluation would strengthen the practical case. The contribution of SBD is a general, theoretically grounded framework for credit assignment under local learning constraints based on broadcasted credit information. It connects a broad family of task losses to appropriate broadcast signals and three-factor updates, without requiring a backward chain of derivatives or feedback weights matched to forward weights. SVE further provides a principled mechanism for enriching that feedback. The residual-network and online CorInfoMax experiments also demonstrate applicability under different architectural and learning conditions, including single-sample updates, lateral interactions, and unshared local connections.
The gap to BP is an important remaining challenge, which we explicitly report. It identifies the performance still to be recovered when replacing exact end-to-end gradient transport with the proposed local mechanism. We therefore position the current results as controlled empirical support for a flexible credit-assignment framework, with broader theoretical and architectural implications, while reserving claims of large-scale competitiveness for further evaluation.
On optimality and convergence
We thank the reviewer for highlighting this distinction. We have added an explicit clarification in Section 4:
"These results provide an optimality-based foundation for constructing local learning objectives. The convexity assumption in the sufficiency argument concerns the conditional risk as a function of the output , whereas optimization over the network weights remains generally nonconvex, as in standard deep-network training. Accordingly, the population-optimality characterization does not itself imply global convergence of the parameter updates; convergence of the specified local decorrelation dynamics is established separately below under explicit assumptions."
In the revised article, we have also provided a convergence analysis for the update dynamics. Under the stated boundedness and step-size conditions, it establishes bounded weights and vanishing current-minibatch and EMA score–activation moments, even as inputs and scores change during training. This guarantees decorrelation for the specified local rule despite omitting output-score derivatives (Section 4 and Appendix A.7).
On the appendix derivations
Several derivations become straightforward once the appropriate score-orthogonality formulation is identified. We include these steps to make the assumptions, parameterization choices, and local approximations transparent, helping readers verify and build on the results. The central contribution lies in establishing the general formulation and developing its consequences for loss-dependent broadcast signals, three-factor learning, and orthogonality-preserving score expansion. The appendix provides a self-contained account of these connections, making the framework accessible to readers with different backgrounds.
I appreciate the authors' effort in preparing the response and revision. However, I remain sceptical about the impact of this work, and will leave my rating as is.
Closing remarks on intellectual impact
Thank you for considering our response and revised manuscript. As the discussion period draws to a close, we would like to reiterate the theoretical results and the evidence across diverse network architectures that bear most directly on your question about the paper’s intellectual significance.
SBD establishes a general theoretical foundation for broadcast-based credit assignment. We show that, across a broad family of losses, the output score in principally chosen output coordinates should be orthogonal to hidden activity at the population optimum. This explains why the score is a natural teaching signal and gives us a criterion for designing alternatives: a candidate broadcast signal should preserve the same orthogonality at the optimum. Score vector expansion is one example of how to use this criterion. Three-factor rule emerges naturally from this principle, and we show how to enrich the teaching signal through score vector expansion (SVE) while preserving orthogonality. Under the stated assumptions, our Lyapunov analysis shows that the unregularized local ReLU rule drives its decorrelation objective to zero.
We demonstrate learning in CNNs, a ResNet, fully and locally connected recurrent networks, and a Vision Transformer. We also showcase the clear benefit of SVE.
The proposed SBD framework has direct implications for both machine learning and computational neuroscience. For machine learning, SBD lets selected layers update in parallel from a shared score without waiting for gradients to pass through intervening layers. This opens a path to training individual modules and to on-chip learning. For computational neuroscience, SBD turns a hypothesis about what a circuit learns into a prediction about its local plasticity. Given a candidate objective and an output representation, the theory specifies a teaching signal whose variation across trials can be tested against measured modulatory activity and synaptic changes.
Response to reviewer comments (Part I)
We thank the reviewer for their careful reading of the paper and thoughtful assessment. Below we provide our clarifications and provide information about relevant enhancements in the revised article.
We should first start with the observation that backpropagation (BP) remains as the accuracy reference, given the physical communication and computational resources are suitable for the generation of exact derivatives at individual nodes. SBD answers a question that BP's accuracy leaves open: when synapses are constrained to see only local activity and a broadcast teaching signal, what should that signal be for a given task loss, and why does plasticity driven by it assign credit? To answer this question, SBD establishes a principled framework for broadcast-based credit assignment and three-factor learning, connecting global task objectives to local synaptic plasticity. It identifies
Regarding the impact and future prospects for the specific fields:
Computational Neuroscience We can list
What next:
Falsifiable predictions and analysis tools for experiments:This framework generates concrete, falsifiable predictions for experimental investigations:
Each candidate behavioral objective defines a specific teaching signal (neuromodulatory factor) and predicts that synaptic plasticity progressively extinguishes correlations between local neural activity and broadcast signals as task proficiency is reached. These correlations act as a signature or reflector of local learning performance.
Broadcast enrichment through input-dependent modulation offers a new theoretical lens for analyzing top-down signaling.
Further directions Further exploration of SVE and the choice of activation nonlinearities are potential future directions.
Machine Learning For machine learning, SBD provides a theoretical basis for designing learning algorithms under constraints on feedback communication and local computation. Once the output score is available, selected layers or modules can compute their updates without waiting for derivatives to traverse the intervening network. This supports concrete directions