- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how to train recurrent neural networks in an online fashion, without using backpropagating error signals through long periods of times. The paper combines locally backpropagated errors (as in truncated backpropagation through time) with diagonal approximations to the sensitivity matrix of real-time recurrent learning (à la e-prop / RFLO). The paper provides analysis that this performs better than doing any of the two individually, on a diverse set of tasks and architectures.
Critical strengths
The paper is nicely written and well executed; from introduction everything makes sense.
Critical weaknesses
I am afraid that the algorithmic innovation of the papers are more limited than what is claimed. The idea that we can use truncated backpropagated error signals to couple with the eligibility traces of eprop rather some instantaneous errors is already present in Bellec et al. 2019 (variation 3 of the eprop algorithm, they additionally use synthetic gradients to estimate errors outside of that window) and to some extent in Schmidhuber 1992, Neural Computation (combines RTRL sensitivity with truncated backprop errors). In addition to that, the paper is motivated from an on-chip learning perspective. The truncated backprop errors that is added compared to eprop is what particularly hard to estimate on chip (which the authors acknowledge in the limitations section). It is unlikely that there is any practical setting in which this algorithm would be favored against e-prop or BPTT.
Questions for authors
Please let me know if my assessment of the contributions is unfair, in which case I am happy to revise it.
4 comments
[W.3]
It is unlikely that there is any practical setting in which this algorithm would be favored against e-prop or BPTT.
We have made the intended use case more specific in the revised manuscript. Compared with e-prop and HYPR, FBPTT additionally captures credit through recurrent connections within chunks. In the paper, this makes a large difference on several tasks; for example, on NeuroMorse HYPR remains at chance level (2.0%), whereas FBPTT reaches 62–69% (BRF) and 49–50% (SE-adLIF), and on sCIFAR with SE-adLIF FBPTT reaches 52–56% compared with 23% for HYPR (Tables 3 and 4 in the submitted manuscript). Compared with BPTT, the memory of FBPTT does not grow with the sequence length. We therefore position FBPTT as an option for long sequences with long-range dependencies under a fixed memory budget, in particular for networks whose internal dynamics can carry information over time, rather than as a general replacement for BPTT.
Questions for Authors
[Q.1]
Please let me know if my assessment of the contributions is unfair, in which case I am happy to revise it.
We hope that the revised framing, the explicit equivalence to e-prop 3, and the new experiments, which identify both the regime in which the forward-propagated neuron-internal credit is decisive and the regime in which it is not, make the contribution of the paper clearer, and we would be grateful if the reviewer would take them into account.
I thank the authors for their rebuttal. The clarifications and additional results strengthen the paper and better situate it within the literature.
However, the rebuttal has not changed my view on the method’s relevance to online learning:
- As the authors themselves acknowledge in the paper, the method has limited potential for on-chip learning, or, equivalently, for explaining learning in the brain, which is the other main motivation for online learning algorithms. The paper offers no insight that would help address these problems in the future, so it appears to be something of a dead end in that direction.
- On current hardware, memory is rarely a bottleneck, even for very long context lengths. And where memory is a concern, BPTT with gradient checkpointing offers cheap solution while still computing exact gradients.
I therefore maintain my score, and I leave it to the AC to weigh whether a solid experimental investigation outweighs a relatively narrow methodological contribution.
We thank the reviewer for considering our responses.
We note that we respectfully disagree with the reviewer regarding the statement "On current hardware, memory is rarely a bottleneck, even for long context lengths." After all, the whole point of truncated BPTT and also checkpointed BPTT is to overcome the memory demand of BPTT. In particular for large SNNs with thousands of simulation steps, memory demand is an important consideration.
Regarding gradient checkpointing, we note that it does not remove the limitations that motivate our approach. Checkpointed BPTT trades memory for recomputation: it requires at least one additional forward pass, and reducing memory further increases the compute cost, as parts of the forward pass must then be recomputed multiple times. For very long sequences, the cost of recomputation becomes substantial. Furthermore, memory still grows with sequence length: For checkpoints every -th step, memory cost in terms of sequence length is . More fundamentally, checkpointing does not change the offline nature of BPTT: gradients are only available after the full sequence (or truncation window) has been processed, so parameter updates cannot be made online as data streams in. In contrast, our method requires constant memory, independent of sequence length, which we believe is relevant for unbounded sequences where BPTT, with or without checkpointing, is not applicable.
We agree however that checkpoined BPTT deserves a discussion in the paper, which we will add to Discussion in the camera ready version in case of acceptance.
Weaknesses
[W.1]
Yes, the FBPTT gradient estimator is equivalent to e-prop 3 without the synthetic-gradient boundary term. We agree that the idea of combining truncated backpropagation with forward-propagated sensitivities goes back to Schmidhuber (1992). We now state this explicitly in the revised manuscript, give a proof of the equivalence in the appendix, and no longer present FBPTT as a new learning rule. The manuscript nevertheless makes two contributions, on which the revised version is focused.
The first is a boundary formulation of this estimator, which computes the same gradient as e-prop 3 in a more efficient way. In e-prop 3, the gradient contribution of a chunk is obtained by multiplying, at every time step, the learning signal with the eligibility trace, which contains the parameter influence from all previous chunks as well as from the current chunk. FBPTT handles these two parts separately: the contribution of the current chunk is computed exactly by standard truncated BPTT, and the forward-propagated sensitivity G~ enters only through the parameter influence of the previous chunks, once per chunk, via the boundary term δiλ+1iλ+1:(i+1)λHI,iλ+1G~iλ. Because G~ is needed only at chunk boundaries, it can be computed by a parallel associative scan, which reduces the number of sequential steps for this computation from O(λ) to O(logλ); the intermediate sensitivities within a chunk need not be computed explicitly, and only a single contraction with the error signal is performed per chunk instead of one per time step.
The second contribution is an evaluation of this estimator on modern spiking neuron models (BRF, SE-adLIF) as well as on GRU, LSTM and HORN networks on typical benchmark tasks, together with a gradient similarity analysis. Bellec et al. evaluated the variant without synthetic gradients only as an ablation on two LSTM tasks.
We have revised the manuscript accordingly. In the Introduction, we now discuss e-prop 3 and the hybrid BPTT-RTRL algorithm of Schmidhuber (1992) and state our contributions with respect to this prior work. The Related Work section contains a new paragraph on combinations of backward- and forward-propagation methods, including e-prop 3. In Methods, we added a new paragraph about the relation to e-prop 3 following Eq. (6), which explains the equivalence and the computational differences described above.
[W.2]
We agree that the backpropagation within chunks is difficult to implement on neuromorphic hardware with local, online learning rules, and we did not claim otherwise in our submitted manuscript. We have revised the motivation to clarify this more directly. FBPTT is not intended as an on-chip learning rule. Its use case is memory-bounded training of recurrent and spiking networks on long sequences on conventional hardware, where full BPTT is limited by memory that grows linearly with the sequence length, and where purely forward approximations such as e-prop and HYPR lose the credit that flows through recurrent connections.