acceptodds
Under review as a conference paper at ICLR 2027

Gradient-Aware Learnable Temporal Truncation for Back-Propagation Through Time in Spiking Neural Networks

Abstract

Spiking Neural Networks (SNNs) have demonstrated promising data-processing capabilities with low energy consumption. Back-Propagation Through Time (BPTT) enables training based on Spatio-Temporal Back-Propagation (STBP) by propagating errors through temporal dynamics. Conventional BPTT aggregates gradients calculated at each time step and updates network parameters using the aggregated gradients, even though the status of SNNs varies across time steps. This shortcoming causes two problems: temporal mismatch in gradient updates and high memory usage during the backward pass. Conventional studies have suggested Truncated BPTT (T-BPTT) to overcome these problems. However, they employ a predetermined truncation strategy that fails to account for temporal variations in gradient behavior, resulting in poor learning performance. To address these problems, we investigate how temporal truncation affects the gradients used to train SNNs and find that effective truncation configurations consistently exhibit smaller gradient dispersion. Guided by this finding, we propose Learnable Truncated BPTT (LT-BPTT), which jointly adjusts the number and lengths of truncated temporal segments. LT-BPTT generates candidate temporal partitions, evaluates their gradient dispersion using a probe mini-batch, and selects the partition with the lowest gradient dispersion. Extensive experiments across diverse SNN architectures and datasets demonstrate that LT-BPTT achieves the highest classification accuracy in most evaluated settings while reducing memory consumption compared with BPTT and benchmarks to improve learning performance of SNNs.

Primary area:
applications to neuroscience & cognitive science
Keywords:
SNNs, STBP, BPTT, gradients, truncation, adaptive

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gw7kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learnable Truncated BPTT (LT-BPTT), an adaptive temporal partitioning method for training spiking neural networks. Starting from full BPTT, the method generates candidate partitions and evaluates them on shared probe mini-batches using a combination of coordinate-wise gradient standard deviation and segment-level classification loss. The selected partition is used for subsequent training, preserving neuronal states across boundaries while truncating temporal gradients. Segment gradients are accumulated before a single optimizer update per mini-batch. Experiments cover multiple architectures and static and neuromorphic datasets, reporting accuracy improvements and reduced memory requirements relative to full BPTT.

Critical strengths
  1. Adaptive temporal partitioning addresses a relevant training trade-off. Jointly adjusting segment numbers and lengths offers more flexibility than a fixed truncation window. Preserving forward states while accumulating segment gradients into one update per mini-batch also makes the training procedure reasonably well controlled.

  2. The empirical evaluation spans several architectures and data modalities. Results include MLPs, convolutional networks, and a spiking transformer, with additional evaluations at different simulation lengths and network depths. Table 8 further supports the usefulness of combining gradient dispersion with the classification term rather than using either component alone.

Critical weaknesses
  1. The evidence does not sufficiently isolate the benefit of gradient-guided partition selection. LT-BPTT changes both temporal gradient paths and the supervision objective through segment-level losses. Table 8 evaluates the selection-score components, but does not establish how much improvement comes from adaptive search over well-tuned fixed or random partitions using identical segment losses. Table 7’s association between decreasing dispersion and increasing training accuracy may also reflect ordinary training progress. Appendix E appropriately acknowledges that lower dispersion does not guarantee better gradient alignment or accuracy, and its loss bound retains a separate truncation-discrepancy condition. Consequently, stronger controlled evidence is needed to establish the proposed statistic as a useful selection criterion.

  2. The reported efficiency results do not clearly establish the cost of the complete adaptive procedure. Table 3 measures controlled fixed-partition configurations on an MLP, while Table 10 does not explicitly account for evaluating multiple candidate partitions on probe mini-batches. Although Appendix C reports GPU timing and energy measurements, it is unclear whether these include the amortized search cost. Without this accounting and the selected partition trajectories, the practical accuracy–efficiency trade-off of adaptive LT-BPTT remains uncertain.

Questions for authors
  1. Can you compare LT-BPTT against tuned fixed partitions and random partition selection under identical segment-level supervision and optimizer-update schedules? This would clarify whether gradient-guided adaptation provides benefits beyond segmentation itself.

  2. Do the timing and energy measurements include candidate evaluation? Please specify the candidate budget, number of probe mini-batches, acceptance threshold, and monitored parameter set, and report end-to-end epoch time and peak memory including search.

  3. Can you report variability across the ten runs for the main accuracy comparisons? Some improvements over the strongest competing method are small, such as 44.39% versus 44.35% on DVS-CIFAR-100, making uncertainty important to the assessment.

1 comment
Reviewer_gw7kOpenReview ↗

Review

Thank you for the detailed response and additional experiments.

My main remaining concern is whether Table 8 isolates the benefit of gradient-guided partition selection. LT-BPTT achieves similar accuracy to the exhaustively tuned fixed-length baseline and higher mean accuracy than random truncation, but the comparison does not clearly specify whether all methods use the same segment-level loss and optimizer update schedule. Clarifying these controls would help determine how much of the observed difference can be attributed to the selection strategy. The end-to-end results also show a substantial search cost on DVS-Gesture-128, where GPU processing time increases from 1.48 to 7.80 seconds per mini-batch after amortizing the search cost. This limits the practical efficiency of LT-BPTT in that setting.

Overall, the revision improves the empirical evaluation, but the central concern about what drives the gains remains, so I maintain my original rating.

Reviewer_wauoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Learnable Truncated BPTT (LT-BPTT) for training spiking neural networks. A motivation study on three model–dataset pairs reports that the truncation length with the highest accuracy also has the smallest average coordinate-wise gradient standard deviation (Table 1). LT-BPTT starts from a single segment and, once per epoch, generates neighboring partitions by splitting, merging, or shifting segment boundaries, scores each on probe mini-batches by the gradient standard deviation plus a weighted segment-level cross-entropy, and adopts the best candidate if it improves the score by a threshold. Training then carries neuronal states across segments, detaches gradients at the boundaries, and accumulates length-weighted segment losses into one update per mini-batch. Experiments compare LT-BPTT with BPTT, TET, OTTT, SLTT, A-TBPTT, TMC, and rate-based BP on twelve datasets with T=4T=4 and T=8T=8, and report memory on an MLP, GPU energy and time, results on BrainScaleS-2, and an appendix analysis that bounds a one-step loss by the selection objective.

Critical strengths
  1. Broad evaluation with practical cost measurements. LT-BPTT is applied to four architecture families (MLP, VGG, ResNet, and a spiking transformer), static and neuromorphic datasets at two time-step settings, deeper variants (Table 6), and an on-chip deployment on BrainScaleS-2 (Tables 13–15), with per-batch GPU energy and time (Tables 11–12). The mechanism sits on top of standard BPTT and needs no change to the neuron model or architecture. This matters to my assessment because it shows the method is cheap to adopt across settings, which would make it useful if its benefit over simpler truncation were established.
  2. The appendix analysis is precise about its scope. Proposition E.4 gives an exact decomposition of the segment loss into the full-sequence loss plus a KL disagreement term, Corollary E.5 characterizes what an accepted split guarantees, and Sections E.2.2 and E.6 state plainly that the bounds do not imply better accuracy and do not cover Adam. This matters because the theoretical statements are checkable and correctly qualified, although the same qualifications limit how much they support the method (Weakness 2).
Critical weaknesses
  1. The reported gains are not attributable to the learnable partition search, and the comparisons are not controlled well enough to support the margins. Fixed-length T-BPTT, the most direct baseline, is missing from Table 2, yet Table 1 shows that a fixed kk already reaches 87.53% on VGG-11/CIFAR-10 (k=2k=2) and 71.86% on ResNet-18/CIFAR-100 (k=4k=4), against 87.98% and 72.16% for LT-BPTT. These 0.45- and 0.30-point differences are far inside the 95% confidence intervals of Table 1 (±1.98 and ±3.73), so most of the headline gain over the Baseline (e.g., +6.79 points on CIFAR-10) is already obtained by a fixed truncation length. The ablation points the same way: STD-only (83.08%) and Main-only (82.27%) on VGG-11/CIFAR-10 (Table 8) fall below fixed k=1k=1 and k=2k=2 (85.64% and 87.53%), although k=2k=2 also has the lowest dispersion in Table 1 and should be reachable by a dispersion-minimizing search. Table 2 reports 10-run means without any spread, while most margins over the runner-up (+0.04 on DVS-CIFAR-100, +0.16 on DVS-Gesture, +0.24 on F-MNIST, +0.93 on CIFAR-100, +1.19 on CIFAR-10) are smaller than the confidence intervals Table 1 reports for the same architectures. The F-MNIST Baseline in Table 2 (87.8%) equals the k=1k=1 entry of Table 1, whereas Table 1 reports 89.22% for full BPTT; with the latter, the margin of LT-BPTT over BPTT shrinks to 0.55 points. Finally, all methods sit well below their published results: in their original papers, with different architectures and time steps, OTTT and SLTT report above 93% on CIFAR-10 and above 96% on DVS-Gesture, against 84.27–86.79% and 80.91–85.76% here. Full BPTT on VGG-11 also shows a gradient standard deviation of 2.06×10−22.06\times10^{-2}, about 380 times the k=2k=2 value, which suggests unstable baseline training. This is consequential because the central claim is that adaptive selection outperforms existing methods, and the current evidence cannot separate it from a tuned fixed truncation length or from run-to-run noise.

  2. The dispersion criterion is weakly justified, the analysis does not connect it to truncation, and the efficiency claims overreach. The empirical premise rests on three model–dataset pairs (Table 1); for ResNet-18 the ordering depends on 9.77×10−59.77\times10^{-5} versus 1.05×10−41.05\times10^{-4} and on accuracies whose intervals overlap. Table 7 shows dispersion falling as training accuracy rises, which ordinary training also produces as gradients shrink, and it has no BPTT reference. LstdL_{\mathrm{std}} is a standard deviation across parameter coordinates and therefore scales with gradient magnitude; Appendix E.2.2 notes that it can drop under gradient rescaling without any change in the concentration measures (Γi\Gamma_i, steffs^{\mathrm{eff}}_t) that Section 4.2 uses to explain why dispersion harms credit assignment. In Theorem E.8, the truncation discrepancy BtrB_{\mathrm{tr}}, the quantity that actually differs between truncation choices and that A-TBPTT controls, enters as a constant that the objective neither estimates nor reduces, and the bound assumes SGD although most configurations use Adam (Section E.6.4). The efficiency claims also conflict with the paper's own data. The abstract and Section 6.2 state that LT-BPTT reduces memory compared with the benchmarks, but in Table 3 OTTT (3.57 MiB), SLTT (3.41 MiB), and rate-based BP (3.02 MiB) all use less than LT-BPTT (5.31–7.63 MiB). Table 3 measures a small MLP with fixed partitions, so the memory of the partitions selected in Table 2 is unknown, and because the search starts from a single full-length segment, peak memory over a run may equal that of BPTT. Candidate evaluation on probe batches is excluded from Tables 10–12. Lastly, the abstract motivates the method by the temporal mismatch of updating with aggregated gradients, yet LT-BPTT also accumulates all segment gradients into one update per mini-batch (Section 5.4). This is consequential because the selection criterion is the paper's methodological contribution and memory efficiency is one of its two headline benefits.

Questions for authors
  1. [Clarification] Why does the F-MNIST Baseline in Table 2 (87.8%) differ from the full-BPTT entry in Table 1 (89.22%)? Please report standard deviations or confidence intervals for Tables 2, 5, and 6, and state how hyperparameters were tuned for each comparison method, given that they are well below their published results. (Table 4 lists 1.00 training epochs for VGG, and the table notes refer to blank entries that do not exist.)
  2. [Requested analysis] Please add fixed T-BPTT with k∈{1,2,4}k\in\{1,2,4\} to Tables 2 and 5, trained with the same segment-level loss. Since Eq. 9 attaches a cross-entropy loss to each segment's averaged logits, which for k=1k=1 resembles the per-time-step loss of TET, please also separate the two ingredients: segment losses with full backpropagation, and truncation with the full-sequence loss.
  3. [Clarification] Which partitions does LT-BPTT actually select over training for each model, and how often does the selection change? What are the magnitudes of LstdL_{\mathrm{std}} and λmainLmain\lambda_{\mathrm{main}} L_{\mathrm{main}} during the search? With LstdL_{\mathrm{std}} around 10−510^{-5} to 10−410^{-4} (Table 1) and λmain=0.001\lambda_{\mathrm{main}}=0.001–0.01, the loss term may dominate JJ, and results with λmain=0\lambda_{\mathrm{main}}=0 would clarify this. Please also give QQ, the candidate limit, and the acceptance threshold.
  4. [Requested analysis] What are the peak memory and wall-clock time of complete LT-BPTT runs for VGG-11, ResNet-18, and the transformer, including candidate evaluation on probe batches and the initial full-length segment?
  5. [Suggestion for future work] Truncation matters most when TT is long. Experiments on long-horizon tasks such as SHD or SSC with T≥100T\geq100, where full-BPTT memory is a real constraint and fixed truncation biases credit assignment, would test whether adaptive partitioning is needed beyond a small grid over kk.
1 comment
AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer wauo,

We sincerely appreciate your time and effort in reviewing our paper. We have carefully addressed all your comments and provided detailed responses in our rebuttal. We would be grateful if you could kindly review our responses and share any further feedback or concerns.

Reviewer_5Kr1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes a method to adaptively decide the truncation size in T-BPTT. In order to achieve this, the method estimates each epoch the optimal (within a search space) truncated configuration by estimating gradients variance. By doing this the manuscript shows a simultaneous reduction in gradient variance and memory consumption with respect to BPTT.
Critical strengths
  • The idea of finding the optimal truncation length for training a SNN with T-BPTT is interesting and has potential implications on how to train SNNs efficiently.
Critical weaknesses
  • The proposed method is compared to other approaches without any statistical significance (e.g. Table 2), so it is impossible to determine if the accuracy improvement is simply noise.
  • The method builds under the assumption that accuracy and low gradient variance are highly correlated. However, there is no empirical evidence to show that the variance is reduced by the method. This is particularly important since the method re-select the partition for backpropagation every epoch so gradients change constantly.
Questions for authors
  • Could an empirical verification be done to show that the gradient variance of LT-BPTT is actually lower than T-BPTT at iso-memory usage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.