acceptodds
Under review as a conference paper at ICLR 2027

LRTB: Low-Rank Temporal Batch Normalization in spiking neural networks

Abstract

Batch normalization (BN) facilitates the training of deep spiking neural networks (SNNs), but temporally evolving neuronal activations pose additional challenges for statistical estimation and parameter adaptation. Existing temporal normalization methods may overlook between-step mean drift when estimating cumulative variance, while independently learned time-specific affine parameters do not explicitly exploit temporal structure. To address these limitations, we propose Low-Rank Temporal Batch Normalization (LRTB), which combines exact cumulative statistical estimation with dynamics-inspired low-rank temporal parameterization. LRTB accounts for both within-step variability and between-step mean drift to accurately estimate the variance of accumulated activations. It further represents time-varying affine parameters using a compact set of exponential basis functions inspired by leaky integrate-and-fire dynamics, enabling structured temporal parameter sharing with fewer learnable affine parameters. Experiments on CIFAR-10, CIFAR-100, DVS-CIFAR10, and ImageNet demonstrate improvements over reported temporal normalization baselines.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Temporal Batch Normalization, Low-Rank Parameterization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_RrLrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Low-Rank Temporal Batch Normalization (LRTB) for SNNs, which consists of two components. The first component improves cumulative variance estimation by considering both within-timestep variance and mean differences across timesteps. The second component uses a set of exponential basis functions to represent time-dependent normalization parameters, reducing the number of learnable affine parameters. Experiments on CIFAR10, CIFAR100, CIFAR10-DVS, and ImageNet show improvements over existing temporal normalization methods.

Critical strengths
  1. The paper identifies a valid issue in the cumulative variance estimation of existing temporal normalization methods and provides a mathematically correct solution.

  2. The proposed method achieves consistent performance improvements across different datasets and network architectures.

Critical weaknesses
  1. Most experiments are conducted with only four timesteps and a temporal rank of three. Under this setting, the low-rank constraint is relatively weak.

  2. Although the method reduces the number of learnable affine parameters, its practical computational benefits are not sufficiently demonstrated. The paper does not provide detailed comparisons of model size or inference latency.

  3. The ImageNet results of TAB are taken from the original paper, but LRTB uses different experimental settings on ImageNet. This makes it difficult to determine whether the improvement comes from LRTB itself.

Questions for authors
  1. How does LRTB perform with longer simulation horizons, such as T=10 or T=16?

  2. What are the actual parameter savings of LRTB, and how does its inference latency compare with existing normalization methods?

  3. How does LRTB perform on Spiking Transformer architectures?

4 comments
AuthorsOpenReview ↗

Response to Reviewer RrLr

We thank the reviewer for recognizing the mathematical correctness of the cumulative variance estimator and the improvements across datasets and architectures. We have added horizon and rank experiments, parameter and computational profiling, a controlled ImageNet comparison, and Spikformer evaluations.

Q1. Longer horizons and the effectiveness of the low-rank constraint

We extended the N-Caltech101 evaluation to T=8T=8 and T=16T=16, keeping r=3r=3.

Time stepsTAB (%)LRTB (%)Gain (pp)Temporal affine coefficient reduction
476.81779.239+2.42225.00%
878.89380.161+1.26862.50%
1679.46980.046+0.57781.25%

The temporal affine coefficient reduction is 1−r/T1-r/T relative to independent per-step coefficients; pp denotes percentage points.

With the same compact rank, LRTB outperforms TAB at every evaluated horizon. At T=16T=16, the temporal affine coefficient reduction reaches 81.25%, directly addressing the concern that the original T=4,r=3T=4,r=3 setting imposed only a mild low-rank constraint.

We also compared r=3r=3 and r=4r=4 at T=16T=16. Both achieve 80.046%, so r=3r=3 attains the same observed accuracy with 25% fewer temporal affine coefficients.

Q2. Computational efficiency

We profiled matched TAB and LRTB ResNet implementations at T=16T=16 on an NVIDIA A100 80GB GPU, using FP32, batch size 64, and synthetic 3×32×323\times32\times32 inputs. Before measurement, we performed 100 untimed warm-up executions using the same inputs and procedure. We then repeated the corresponding training or inference step 500 times, recording the elapsed time and peak GPU memory for each repetition. These repetitions were used for computational profiling and were separate from the model-training epochs.

MetricTABLRTBChange relative to TAB
Total parameters (M)14.89214.699-1.30%
Normalization parameters238,00044,976-81.10%
Peak training memory (MiB)79,52246,437-41.60%
Training time (ms/iteration)2160.542213.33+2.44%
Inference memory (MiB)4098.813681.93-10.17%

At T=16T=16, LRTB reduces peak training memory by 41.60%, inference memory by 10.17%, while training iteration time increases by only 2.44%.

Normalization parameters decrease by 81.10%, corresponding to approximately 1.30% fewer total model parameters. These measurements quantify the actual parameter savings and demonstrate practical inference and memory benefits in the evaluated configuration. We will include the results and profiling setup in the revised manuscript.

C3. Fair comparison on ImageNet-1K

We compared TAB and LRTB on ImageNet-1K under exactly the same training and evaluation conditions, with the normalization method as the only difference. Both methods use ResNet-34 at T=4T=4.

ImageNet-1K Top-1 accuracy (%)

MethodArchitectureTTTop-1
TAB, matched rerunResNet-34467.43
LRTBResNet-34468.75
Gain——+1.32 pp

Results are from one training run per method.

Under this controlled protocol, LRTB improves Top-1 accuracy by 1.32 percentage points. This comparison uses our rerun TAB baseline, providing more direct evidence of the normalization benefit than the original comparison against the published TAB result.

We will replace the original comparison with these matched results and include the complete training configuration in the revision.

Q3. Applicability to Spiking Transformer architectures

We additionally evaluated LRTB on Spikformer, comparing it with TAB on CIFAR-10 and CIFAR-100 under matched settings at T=4T=4.

Top-1 accuracy (%)

DatasetTABLRTBGain (pp)
CIFAR-1094.6095.01+0.41
CIFAR-10077.3478.26+0.92

Results are from one training run per configuration.

LRTB improves accuracy on both datasets, extending the empirical evidence beyond convolutional SNNs to the evaluated Spikformer architecture.

We will incorporate the horizon and rank experiments, computational profiling, controlled ImageNet results, and Spikformer comparisons into the revised manuscript.

Thank you for the detailed response and additional experiments. I appreciate the new results at longer timesteps, the controlled ImageNet comparison, the Spikformer experiments, and the efficiency measurements.

However, I still have some reservations. Although the low-rank parameterization shows benefits at T=16, the accuracy improvement over TAB becomes relatively small as T increases. Moreover, the reduction in total model parameters is only 1.30%, and inference latency is still not reported. I also believe the methodological novelty remains somewhat limited, as the variance correction is based on a standard statistical formula and the low-rank parameterization is relatively straightforward.

Overall, the additional experiments strengthen the paper, but I do not think they sufficiently change my assessment of its contribution. Therefore, I will keep my original rating of 2.

Thank you for pointing this out. We have added inference-latency measurements using the same TAB and LRTB ResNet implementations at T=16T=16 on an NVIDIA A100 80GB GPU, with FP32, batch size 64, and 3×32×323\times32\times32 inputs. Each method was evaluated in a fresh process under torch.inference_mode(), with 100 warm-up iterations followed by 500 synchronized forward passes. LRTB reduces the mean inference latency from 529.76 ms/batch for TAB to 305.76 ms/batch, a 42.28% reduction, while reducing inference peak memory by 10.17%.

The smaller accuracy gain at T=16T=16 may partly reflect model-performance saturation: both methods approach approximately 80% accuracy as the number of time steps increases, leaving less room for further improvement. Nevertheless, LRTB retains a positive gain while reducing temporal affine coefficients by 81.25%. We appreciate the reviewer’s careful assessment and the opportunity to clarify these practical efficiency results.

Thank you for the additional results and clarification. However, some of my previous concerns still remain, and I believe the methodological novelty is relatively limited. Therefore, I will keep my original rating unchanged.

Reviewer_gVknOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Low-Rank Temporal Batch Normalization (LRTB) for spiking neural networks (SNNs). LRTB estimates statistics by considering within-step variability and between-step mean drift. It also represents time-varying affine parameters using a compact set of exponential basis functions. To demonstrate the advantages of their method over other SNN-based BN methods, the authors conducted experiments on static and neuromorphic object recognition datasets.

Critical strengths
  1. The proposed method accounts for step-wise variance and incorporates the effect of variance caused by mean drift between different time steps.
  2. Ablation and comparative experiments demonstrate its effectiveness.
Critical weaknesses
  1. This paper claims that a low rank can reduce the number of affine parameters. However, it introduces fixed-basis matrix multiplication and retains independent time weights. It does not provide a complete comparison of the number of parameters, floating-point operations (FLOPs), and training/inference wall-clock time with other BN methods for SNNs. Therefore, it is impossible to determine the computational cost of achieving the accuracy gains. In particular, the computational cost of large ImageNet models has not been quantified.
  2. This paper claims that the exponential basis stems from the leakage kinetics of LIF neurons. However, this basis is merely a fixed, preset prior that is not connected to the neurons’ actual membrane potentials or time constants.
  3. All experiments were conducted at low time steps (T ≤ 6). However, the temporal variations observed at these time steps may not be significant, which fails to demonstrate the advantages of the proposed method under temporal variations. Therefore, the authors should demonstrate the method's effectiveness when the time step is scaled, as well as in long-term tasks, such as speech recognition.
  4. Temporal normalization can significantly impact the robustness of SNNs. The authors should provide additional details on how their proposed method affects the adversarial and noise robustness of SNNs.
  5. The performance metrics reported in this paper are significantly lower than those reported in existing studies of SNNs, such as [1, 2, 3]. Therefore, it is impossible to assess whether the proposed method truly offers a performance advantage. The authors should compare the proposed method with the most recent SNN studies.
[1] Temporal Representation Enhancement (TRE): Learning to Forget Dominant Patterns for Enhanced Temporal Spiking Features. CVPR. 2026.
[2] Stable Spike: Dual Consistency Optimization via Bitwise AND Operations for Spiking Neural Networks. CVPR. 2026.
[3] $A^2SG$:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks. ICML. 2026.
Questions for authors

Please see Weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer gVkn — Part 1/2

We thank the reviewer for recognizing the cumulative variance correction and the experimental ablations. We have added experiments addressing practical efficiency, neuronal-dynamics matching, simulation horizons, robustness, and applicability beyond convolutional SNNs.

C1. Computational efficiency

We profiled matched TAB and LRTB ResNet implementations at T=16T=16 on an NVIDIA A100 80GB GPU, using FP32, batch size 64, and synthetic 3×32×323\times32\times32 inputs. Before measurement, we performed 100 untimed warm-up executions using the same inputs and procedure. We then repeated the corresponding training or inference step 500 times, recording the elapsed time and peak GPU memory for each repetition. These repetitions were used for computational profiling and were separate from the model-training epochs.

MetricTABLRTBChange relative to TAB
Total parameters (M)14.89214.699-1.30%
Normalization parameters238,00044,976-81.10%
Peak training memory (MiB)79,52246,437-41.60%
Training time (ms/iteration)2160.542213.33+2.44%
Inference memory (MiB)4098.813681.93-10.17%

At T=16T=16, LRTB reduces peak training memory by 41.60% and inference memory by 10.17%, while training iteration time increases by only 2.44%. Normalization parameters decrease by 81.10%, with total-model parameter counts reported separately.

These measurements demonstrate practical inference and memory benefits in the evaluated configuration. The profiling covers the ResNet setup specified above, ImageNet-scale runtime and a complete FLOP breakdown are not included in these measurements.

C2. Connection between the exponential basis and neuronal dynamics

We directly tested this connection by varying the LIF neuronal leak and the basis leak over {0.7,0.8,0.9}\{0.7, 0.8, 0.9\}, using CIFAR-10, ResNet-19, T=4T=4, and r=3r=3. All configurations were trained and evaluated under exactly the same conditions, with the leak pairing as the only difference.

Top-1 accuracy (%)

Neuron leak / Basis leak0.70.80.9
0.796.0692.7590.08
0.895.8195.9792.08
0.990.7095.6996.13

The matched basis achieves the highest observed accuracy for each neuronal leak factor. Matched configurations average 96.05%, versus 92.85% for mismatched configurations.

The basis is fixed within each run but is constructed according to the neuronal leakage timescale. This experiment directly supports the importance of that choice. We will clarify that the exponential basis represents a dynamics-inspired prior over normalization trajectories rather than an explicit reconstruction of membrane potentials.

AuthorsOpenReview ↗

Response to Reviewer gVkn — Part 2/2

C3. Effectiveness at longer simulation horizons

We extended the evaluation to N-Caltech101 at T=4,8,16T=4,8,16, keeping r=3r=3.

Time stepsTAB (%)LRTB (%)Gain (pp)Temporal affine coefficient reduction
476.81779.239+2.42225.00%
878.89380.161+1.26862.50%
1679.46980.046+0.57781.25%

The temporal affine coefficient reduction is 1−r/T1-r/T relative to independent per-step coefficients; pp denotes percentage points.

LRTB improves over TAB at all three evaluated horizons. At T=16T=16, it retains an accuracy advantage while reducing temporal affine coefficients by 81.25%, demonstrating effectiveness under a substantially stronger temporal compression constraint.

These experiments extend the evaluation beyond the original T≤6T\leq6 settings. Long-duration sequence tasks such as speech recognition remain a separate extension of the current classification evaluation.

C4. Noise and adversarial robustness

In an additional CIFAR-10 evaluation using pretrained VGG9 models at T=4T=4, both TAB and LRTB exhibited substantial accuracy degradation under identical Gaussian-corrupted inputs (σ=0.15\sigma=0.15 and 0.200.20). LRTB focuses on temporal statistical estimation and affine parameterization under standard training, without a dedicated robustness objective. Future work will explore combining LRTB with noise-aware or adversarial training and systematically evaluate its robustness to input corruptions and adversarial attacks.

C5. Comparison with recent SNN methods

We appreciate the references to TRE, Stable Spike, and A²SG and will discuss these recent advances in the revised related work.

To the best of our knowledge, reproducible implementations of these three methods are not currently publicly available. Specifically, we could not identify official code releases for TRE and Stable Spike, while the public A²SG repository does not yet contain the implementation needed for reproduction. Consequently, reliable comparisons under matched experimental settings are currently infeasible.

Moreover, these methods primarily focus on temporal representation enhancement, consistency optimization, and surrogate gradient design, respectively, whereas LRTB specifically addresses temporal normalization. To isolate its contribution, we compare normalization methods within matched architectures and training settings, while retaining the published tdBN, TEBN, and TAB comparisons in the original manuscript as broader context.

We have strengthened this evidence with a controlled ImageNet-1K rerun and matched Spikformer experiments:

Top-1 accuracy (%)

DatasetArchitectureTTTABLRTBGain (pp)
ImageNet-1KResNet-34467.4368.75+1.32
CIFAR-10Spikformer494.6095.01+0.41
CIFAR-100Spikformer477.3478.26+0.92

These comparisons use one training run per configuration.

On ImageNet-1K, TAB and LRTB were evaluated under identical training and evaluation settings, with normalization as the only difference. The TAB result above is our matched rerun, replacing the original cross-publication comparison.

The Spikformer results further show that the benefit extends to the evaluated Spiking Transformer architecture. Together with the three-paired-seed CIFAR experiments, these results strengthen the evidence that replacing TAB with LRTB improves classification across datasets and architectures.

We will distinguish controlled reruns from published benchmark results, clarify the reproducibility status of these recent methods in the related work, and position the normalization contribution alongside recent advances in SNN architectures and training.

Reviewer_CsAmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Low-Rank Temporal Batch Normalization (LRTB) for spiking neural networks. LRTB contains two main components. First, it corrects cumulative temporal variance estimation by accounting for both within-step variance and between-step mean drift using the law of total variance. Second, it parameterizes time-varying BN scale and bias trajectories using a low-rank set of exponential temporal basis functions motivated by LIF dynamics. The method retains a learnable temporal importance weight and admits constant-memory updates of cumulative statistics. Experiments on CIFAR-10, CIFAR-100, DVS-CIFAR10, and ImageNet, together with component, basis, rank, and simulation-horizon ablations, show consistent improvements over reported temporal-normalization baselines

Critical strengths
  1. The paper identifies and cleanly fixes a concrete statistical issue in temporal accumulated normalization. The cumulative variance of activations over time contains both the average within-step variance and the variance induced by temporal changes in the step-wise means. LRTB explicitly includes the latter term, and the paper provides a useful empirical diagnosis showing that ignoring it can lead to increasingly large estimation errors in deeper layers. This part is mathematically clean, computationally inexpensive, and directly connected to the normalization mechanism.

  2. The low-rank temporal parameterization is simple, interpretable, and accompanied by relatively comprehensive ablations. Rather than independently learning affine parameters at every time step, LRTB represents the temporal trajectories using a shared exponential basis with channel-specific coefficients. The paper examines alternative basis families, different temporal ranks, the importance of the temporal weighting term, projection of unconstrained TAB parameters onto the proposed subspace, and different simulation horizons. The evaluation also covers both static and event-based datasets and includes ImageNet, which strengthens the empirical scope.

Critical weaknesses
  1. The evidence supporting the "dynamics-aligned low-rank" contribution and its practical advantage is not yet sufficiently strong. Most of the detailed basis/rank analysis is conducted at the very short horizon T=4T=4 with r=3r=3, where the temporal dimension is only four and the rank is already 7575% of the full temporal dimension. At T=2T=2, reff=2r_{\mathrm{eff}}=2, so there is no dimensional compression at all, and the longest evaluated horizon is only T=6T=6. Moreover, all main results are single fixed-seed runs, and the authors explicitly acknowledge that the reported differences do not establish statistical significance. This is important because several improvements over TAB are below one percentage point, and alternative bases such as wavelet or DCT are only around 0.20.2--0.40.4 points behind the exponential basis.

  2. The claim that the exponential basis is specifically "dynamics-aligned" is also not fully isolated. The basis is constructed using λLIF=0.9\lambda_{\mathrm{LIF}}=0.9, and the experiments use the same LIF leak factor. There is no experiment varying the neuron leak, using mismatched basis time constants, or considering another neuron model. Consequently, it remains unclear whether the gain comes specifically from alignment with neuronal dynamics or simply from imposing a smooth low-dimensional temporal regularizer. Finally, the paper emphasizes parameter efficiency, but does not quantify the reduction in total model parameters, training/inference memory, or latency; for the short horizons considered, the practical systems-level saving may be very small.

Questions for authors
  1. A closely related work, "Enhancing Learning of Spiking Neural Networks through Normalization with Time-Based Statistics Estimation" (ECAI 2025), also derives the overall temporal variance from within-step variances and between-step mean differences. Could the authors clarify the relationship between this work and the proposed exact cumulative temporal statistics?

  2. Could the authors rerun the closest TAB baseline and LRTB under exactly the same training pipeline for multiple random seeds and report mean and standard deviation on the main CIFAR-10/CIFAR-100 and preferably ImageNet or DVS-CIFAR10 settings? Since several improvements are below one percentage point, this would materially affect my confidence in the empirical advantage.

  3. Can the authors provide experiments at substantially longer simulation horizons, e.g., T=8T=8, T=16T=16, or larger, where r≪Tr \ll T and the low-rank constraint becomes genuinely nontrivial? The current T=4,r=3T=4,r=3 setting provides only modest temporal compression, while T=2T=2 provides none.

  4. How important is the claimed alignment between the exponential basis and the actual LIF dynamics? For example, what happens if the LIF leak factor is changed while keeping the basis fixed, or if the basis is deliberately constructed using a mismatched leak factor? Results across different leak factors or neuron models would help distinguish a genuinely dynamics-aligned representation from generic temporal smoothing/regularization.

  5. Could the authors report the actual reduction in normalization parameters as a fraction of total model parameters, as well as training/inference memory or runtime? This would clarify the practical significance of the "parameter-efficient" claim for the short simulation horizons considered.

  6. Since one of the main motivations for SNNs is efficient deployment on neuromorphic or resource-constrained hardware, could the authors discuss the deployment implications of LRTB more explicitly? In particular, although the low-rank temporal parameterization reduces the number of learnable affine parameters, LRTB still requires cumulative-statistics updates, temporal basis reconstruction, and the learnable temporal weight ω[t]\omega[t] at each simulation step. How do these operations affect inference latency, memory access, energy consumption, and hardware friendliness compared with TAB, TEBN, or standard time-specific BN? If possible, reporting deployment-oriented metrics such as parameter storage, MAC/addition overhead, runtime, or estimated energy on neuromorphic hardware would substantially strengthen the practical motivation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CsAm (Part 1/2)

We thank the reviewer for recognizing the correctness of the cumulative variance estimator, the interpretability of the temporal parameterization, and the breadth of the ablations. We have strengthened the evidence through paired-seed experiments, additional simulation horizons, neuron-basis matching tests, and computational profiling.

Q1. Relationship to Time-Based Statistics Estimation (TSE)

We thank the reviewer for highlighting Enhancing Learning of Spiking Neural Networks through Normalization with Time-Based Statistics Estimation (TSE, ECAI 2025). We will explicitly cite and discuss this work. The within-step/between-step variance decomposition follows the standard law of total variance; our contribution concerns its use in cumulative normalization and its combination with structured temporal affine parameters.

TSE estimates full-horizon statistics over 1:T1:T, combining observed statistics with estimates for unobserved steps obtained from recent training batches, and uses temporally shared affine parameters. LRTB instead computes exact pooled statistics over the observed prefix 1:t1:t and represents time-varying affine trajectories using a shared exponential basis with channel-specific coefficients. Both approaches can operate without accessing future activations of the current sequence.

The distinction therefore lies in the statistical target, estimation procedure, and affine parameterization.

For equal numbers of observations per step, using the population variance convention, LRTB computes

=\frac{1}{t}\sum_{s=1}^{t}\sigma_s^2 +\frac{1}{t}\sum_{s=1}^{t}(\mu_s-\mu_{1:t})^2.$$ The between-step mean-variation term corrects the omission in averaging per-step variances. The low-rank affine representation then provides a separate mechanism for structured temporal adaptation. The original controlled component ablation on CIFAR-10 with ResNet-19 at $T=4$ isolates these contributions: | Configuration | Top-1 (%) | |---|---:| | TAB | 94.76 | | TAB + exact cumulative statistics | 95.66 | | TAB + low-rank affine parameterization | 95.78 | | **LRTB: both components** | **96.13** | Both components improve the baseline independently, and their combination achieves the best result. We will make this contribution structure explicit while acknowledging the statistical foundation shared with TSE. ## Q2. Statistical reliability across random seeds We repeated the TAB/LRTB comparison using **three paired random seeds** on CIFAR-10 and CIFAR-100 with ResNet-19 at $T=4$. Within each paired comparison, the training pipeline was identical. **Top-1 accuracy (%), mean ± sample standard deviation** | Dataset | TAB | LRTB | |---|---:|---:| | CIFAR-10 | 94.33 ± 0.92 | **95.72 ± 0.68** | | CIFAR-100 | 76.76 ± 0.13 | **77.95 ± 0.09** | **Paired accuracy gains (percentage points)** | Dataset | Seed 1 | Seed 2 | Seed 3 | Mean paired gain | |---|---:|---:|---:|---:| | CIFAR-10 | +1.20 | +1.66 | +1.31 | **+1.39** | | CIFAR-100 | +1.17 | +1.24 | +1.18 | **+1.20** | LRTB improves over TAB in **all six paired comparisons**. These consistently positive gains strengthen the empirical evidence beyond the original single-run results. We will include both aggregate statistics and per-seed gains in the revision.
AuthorsOpenReview ↗

Response to Reviewer CsAm (Part 2/2)

Q3. Longer horizons and meaningful temporal compression

We extended the N-Caltech101 evaluation to T=8T=8 and T=16T=16, retaining r=3r=3.

Time stepsTAB (%)LRTB (%)Gain (pp)Temporal affine coefficient reduction
476.81779.239+2.42225.00%
878.89380.161+1.26862.50%
1679.46980.046+0.57781.25%

The temporal affine coefficient reduction is 1−r/T1-r/T relative to independent per-step coefficients; pp denotes percentage points.

At T=16T=16, the temporal affine representation uses only three basis components across sixteen time steps while retaining an accuracy advantage over TAB. The decreasing gain with TT reflects diminishing returns of independent per-step coefficients; the key point is that LRTB maintains its advantage with 81.25% fewer coefficients. The additional rank comparison gives 80.046% for both r=3r=3 and r=4r=4, so r=3r=3 achieves the same observed accuracy with fewer coefficients in this setting.

These experiments directly address the concern that the original short-horizon evaluation imposed only a weak low-rank constraint. We will distinguish the T=2T=2 full-dimensional setting from the compressed settings and add the new horizon and rank results.

Q4. Evidence for dynamics alignment

We independently varied the neuronal leak and the basis leak over {0.7,0.8,0.9}\{0.7, 0.8, 0.9\}, using CIFAR-10, ResNet-19, T=4T=4, and r=3r=3.

Top-1 accuracy (%)

Neuron leak / Basis leak0.70.80.9
0.796.0692.7590.08
0.895.8195.9792.08
0.990.7095.6996.13

For each neuronal leak factor, the matched basis gives the highest observed accuracy. Matched configurations average 96.05%, compared with 92.85% for mismatched configurations.

Unlike the original comparison between basis families, this experiment changes the relation between the neuron dynamics and the exponential basis while retaining the same basis family and rank. It therefore provides more direct evidence for selecting temporal basis constants according to neuronal leakage.

Q5. Computational efficiency

We profiled matched TAB and LRTB ResNet implementations at T=16T=16 on an NVIDIA A100 80GB GPU, using FP32, batch size 64, and synthetic 3×32×323\times32\times32 inputs. Before measurement, we performed 100 untimed warm-up executions using the same inputs and procedure. We then repeated the corresponding training or inference step 500 times, recording the elapsed time and peak GPU memory for each repetition. These repetitions were used for computational profiling and were separate from the model-training epochs.

MetricTABLRTBChange relative to TAB
Total parameters (M)14.89214.699-1.30%
Normalization parameters238,00044,976-81.10%
Peak training memory (MiB)79,52246,437-41.60%
Training time (ms/iteration)2160.542213.33+2.44%
Inference memory (MiB)4098.813681.93-10.17%

At T=16T=16, LRTB reduces peak training memory by 41.60% and inference memory by 10.17%, while training iteration time increases by only 2.44%.

The 81.10% reduction concerns normalization parameters; the corresponding total-model parameter reduction is approximately 1.30%. Reporting both quantities makes the scope of the parameter savings explicit. The memory measurements provide additional evidence of practical benefits in this evaluated configuration.

Q6. Deployment implications

LRTB retains the temporal importance weights and introduces cumulative-statistics computation and reconstruction of temporal affine coefficients. The reported parameter counts and GPU measurements evaluate the resulting implementation, providing deployment-relevant evidence beyond the number of learned coefficients alone.

At T=16T=16, LRTB reduces peak training memory by 41.60% and inference memory by 10.17%, while training iteration time increases by only 2.44%. These are GPU memory and runtime results, neuromorphic energy consumption and hardware-specific operation counts have not been measured.

We will revise the deployment discussion to distinguish parameter storage, measured GPU efficiency, and hardware-dependent energy consumption, and incorporate the new experiments into the manuscript.

Reviewer_CsAmOpenReview ↗

Follow-up on the Rebuttal

I thank the authors for their detailed responses and additional experiments, which have addressed several of my concerns. However, I still have some reservations regarding Q3 and Q6.

Q3: Although the authors extended the experiments to T=16T=16 with r=3r=3, the temporal dimension remains relatively small, and the low-rank structure is not sufficiently compelling. In typical low-rank methods, the rank is often substantially smaller than the original dimension. Moreover, the performance gains over TAB decrease as TT increases. Therefore, stronger evidence at substantially larger temporal dimensions is needed to demonstrate the effectiveness and scalability of the proposed low-rank parameterization.

Q6: The reported GPU memory savings are encouraging, but the practical benefits for neuromorphic deployment remain insufficiently evaluated, particularly regarding inference latency, computational overhead, and energy efficiency.

Overall, I appreciate the authors' efforts and additional evidence. Nevertheless, these concerns remain important to my assessment, and I have decided to maintain my original rating (2).

Response to Q3: Longer temporal dimensions

We agree that the accuracy gain decreases as TT increases. However, this trend may partly reflect saturation on N-Caltech101 rather than a failure of the low-rank representation. TAB improves from 76.817% at T=4T=4 to 78.893% at T=8T=8 and 79.469% at T=16T=16, while LRTB achieves 79.239%, 80.161%, and 80.046%, respectively. Thus, both methods approach approximately 80% accuracy, leaving limited room for further improvement at longer horizons.

Response to Q6: Inference and deployment efficiency

We additionally measured inference latency using the same TAB and LRTB ResNet implementations at T=16T=16 on an NVIDIA A100 80GB GPU, with FP32, batch size 64, and 3×32×323\times32\times32 inputs. Each method was evaluated in a fresh process under torch.inference_mode(), using 100 warm-up iterations followed by 500 synchronized forward passes. The measurement includes all operations executed by LRTB, including temporal affine reconstruction and temporal weighting.

LRTB reduces mean inference latency from 529.76 ms/batch to 305.76 ms/batch, a 42.28% reduction, and reduces inference peak memory by 10.17%. It also reduces normalization parameters by 81.10%, while the total model parameter reduction is 1.30%. During training, the additional computation results in only a 2.44% increase in iteration time.

These results quantify the practical latency, memory, and computational overhead of the evaluated GPU implementation. We have not measured energy consumption on neuromorphic hardware and therefore do not claim a hardware-specific energy reduction. Nevertheless, the reduced normalization-state storage, inference memory, and measured latency provide concrete deployment-relevant evidence beyond parameter counts alone.

Reviewer_uA1GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Low-Rank Temporal Batch Normalization (LRTB) for spiking neural networks (SNNs), addressing two core aspects of temporal normalization: (i) statistically consistent cumulative variance estimation across time that accounts for between-step mean drift, and (ii) a dynamics-aligned low-rank parameterization of time-varying BN affine parameters using a small set of exponential basis functions. Across CIFAR-10/100, DVS-CIFAR10, and ImageNet, LRTB yields consistent improvements over temporal normalization baselines (notably TAB) with ablations and diagnostics that isolate the contributions of exact cumulative statistics and the low-rank temporal parameterization.

Critical strengths

1. Simple and well-motivated technical design.

The paper identifies a limitation in existing cumulative temporal normalization, where averaging per-step variances ignores temporal mean drift. The proposed correction is mathematically sound and easy to implement. The exponential basis parameterization is also a reasonable way to capture temporal variations while reducing the number of learnable parameters.

2. Solid experimental validation.

The method is evaluated on four datasets, including ImageNet, and consistently improves over the reported TAB baseline under matched architectures and time steps. The component ablations show that both proposed techniques contribute to the improvement. The variance analysis and low-rank reconstruction results further support the motivation behind the method.

Critical weaknesses

1. Generalization across different neuronal dynamics.

The proposed exponential basis is motivated by LIF neuron dynamics, but the experiments mainly use a fixed neuron configuration. It would be useful to investigate how the method performs with different membrane time constants or neuron models, particularly since the temporal basis is fixed rather than adapted to individual layers. This would help establish the general applicability of the proposed parameterization.

2. Limited evaluation of longer temporal horizons and efficiency.

The experiments mainly focus on short simulation horizons (T = 2, 4, 6), where the benefit of low-rank parameterization is relatively limited. It remains unclear whether the method maintains its advantages at larger T. The paper also lacks runtime or memory comparisons with TAB, making it difficult to assess the practical efficiency benefits.

Questions for authors
  1. Sensitivity to neuronal dynamics: How sensitive is LRTB to the choice of time constants in the exponential basis? Have the authors evaluated different leak factors or layer-specific time constants? This would help clarify whether the proposed basis generalizes beyond the current neuron configuration.

  2. Longer simulation horizons: How does LRTB perform with longer simulation horizons (e.g., T = 16 or 32)? Does the preferred temporal rank increase with T? This would help assess the scalability of the proposed low-rank parameterization.

  3. Computational efficiency: Could the authors provide a comparison of training time and memory usage between LRTB and TAB? Since parameter efficiency is one of the motivations, understanding the actual computational overhead would help assess its practical benefits.

1 comment
AuthorsOpenReview ↗

# Response to Reviewer uA1G

We thank the reviewer for recognizing the motivation, mathematical soundness, and experimental validation of LRTB. We have added experiments on neuronal dynamics, simulation horizons, temporal rank, and practical computational costs.

Q1. Sensitivity to neuronal dynamics

We directly tested whether the temporal basis should be matched to the neuronal leak factor. On CIFAR-10 with ResNet-19, T=4T=4, and r=3r=3, we varied the LIF neuron leak factor and the leak factor used to construct the exponential basis over {0.7,0.8,0.9}\{0.7, 0.8, 0.9\} under exactly the same training and evaluation conditions for every configuration, with the leak pairing as the only difference.

Top-1 accuracy (%)

Neuron leak / Basis leak0.70.80.9
0.796.0692.7590.08
0.895.8195.9792.08
0.990.7095.6996.13

For each tested neuronal leak factor, the matched basis achieves the highest observed accuracy. The three matched configurations average 96.05%, compared with 92.85% across the six mismatched configurations.

This experiment directly supports choosing basis time constants according to neuronal leakage, extending the evidence beyond the original fixed-neuron setting. Within each run, LRTB uses a fixed temporal basis shared across layers. The basis provides a dynamics-inspired inductive bias for affine trajectories rather than reconstructing membrane potentials.

We will include this sensitivity study and clarify the basis construction in the revised manuscript.

Q2. Longer simulation horizons and temporal rank

We extended the evaluation to N-Caltech101 at T=4,8,16T=4,8,16, keeping the temporal rank fixed at r=3r=3.

Time stepsTAB (%)LRTB (%)Gain (pp)Temporal affine coefficient reduction
T=4T=476.81779.239+2.42225.00%
T=8T=878.89380.161+1.26862.50%
T=16T=1679.46980.046+0.57781.25%

The coefficient reduction is 1−r/T1-r/T, relative to independently learned affine coefficients at every time step. Here and throughout the response, pp denotes percentage points.

LRTB maintains an accuracy advantage at every evaluated horizon using the same compact rank. At T=16T=16, it improves over TAB by 0.577 percentage points while reducing temporal affine coefficients by 81.25%. Thus, the benefit persists when the low-rank constraint becomes substantially stronger.

To examine rank selection at T=16T=16, we additionally compared r=3r=3 and r=4r=4:

DatasetTTRankTop-1 (%)
N-Caltech10116380.046
N-Caltech10116480.046

At T=16T=16, r=3r=3 attains the same observed accuracy as r=4r=4 with 25% fewer temporal affine coefficients, supporting it as a compact choice for this setting.

Q3. Computational efficiency

We profiled matched TAB and LRTB ResNet implementations at T=16T=16 on an NVIDIA A100 80GB GPU, using FP32, batch size 64, and synthetic 3×32×323\times32\times32 inputs. Before measurement, we performed 100 untimed warm-up executions using the same inputs and procedure. We then repeated the corresponding training or inference step 500 times, recording the elapsed time and peak GPU memory for each repetition. These repetitions were used for computational profiling and were separate from the model-training epochs.

MetricTABLRTBChange relative to TAB
Total parameters (M)14.89214.699-1.30%
Normalization parameters238,00044,976-81.10%
Peak training memory (MiB)79,52246,437-41.60%
Training time (ms/iteration)2160.542213.33+2.44%
Inference memory (MiB)4098.813681.93-10.17%

At T=16T=16, LRTB reduces peak training memory by 41.60% and inference memory by 10.17%, while training iteration time increases by only 2.44%. These measurements demonstrate substantial memory savings in the evaluated configuration, providing practical evidence beyond normalization parameter savings.

We will include the complete measurements and profiling setup in the revision, distinguishing normalization parameter savings from total-model parameter counts and measured runtime and memory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.