acceptodds
Under review as a conference paper at ICLR 2027

Temporal-Aware Learning: Coordinating Supervision with the Temporal Evolution of Spiking Neural Networks

Abstract

The complex spatiotemporal dynamics of Spiking Neural Networks (SNNs) pose substantial challenges to direct training. In this work, we revisit this challenge from a temporal-evolution perspective by analyzing multi-granularity spiking activity, per-step predictions, and gradient relationships. We observe that spike firing fluctuations decrease, spatial firing patterns become more refined, and predictions become more accurate over time. Meanwhile, early-step gradients are less consistent with later-step gradients, whereas gradients among later steps become increasingly consistent. These observations suggest that temporal states contribute differently to optimization and uniform aggregation may underutilize coherent update tendency emerging at later steps. Unlike widely used objectives that supervise temporal states uniformly, we propose Temporal-Aware Learning (TAL), which considers the evolving states of SNNs. Our method incorporates quadratic temporal reweighting, later-step consistency regularization, and temporal target smoothing, to increase the contribution of locally coherent late-step gradients while softening early-step supervision. Experiments on static and neuromorphic datasets demonstrate that our method increases early–late gradient consistency, improves accuracy and accelerates convergence without additional inference overhead, with clear gains as temporal window increases within the evaluated range.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Event-based Vision, Neuromorphic Computing, Brain-inspired Computing, Optimization Objective

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5KMcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal-Aware Learning (TAL), a training objective designed to better coordinate supervision across different timesteps in SNNs. The authors observe that early temporal states exhibit unstable spike activity, while later states generally produce more accurate predictions and more consistent gradients. Based on these observations, TAL combines three components: quadratic temporal reweighting (QTR), later-step consistency regularization (LSCR), and temporal target smoothing (TTS). Experiments on static and neuromorphic datasets demonstrate improved classification accuracy and faster convergence without introducing additional inference overhead.

Critical strengths

1.The paper provides observations about the temporal evolution of SNNs, particularly the differences in gradient consistency between early and later timesteps. These observations offer a reasonable motivation for the proposed training strategy. 2.The proposed method achieves competitive performance across multiple static and neuromorphic datasets, including ImageNet-1K, and can be applied without modifying the inference architecture. 3.The paper includes extensive ablation studies on different temporal weighting strategies and hyperparameter settings, which help demonstrate the effectiveness of emphasizing later temporal states.

Critical weaknesses

1.According to Table 3, QTR alone already achieves most of the performance improvements. Adding LSCR and TTS only improves accuracy by another 0.30 and 0.18 percentage points on CIFAR10-DVS and N-Caltech101, respectively. This raises concerns about whether all three components are necessary. 2.The experimental evaluation could be strengthened by considering a wider range of network architectures, as only one architecture is evaluated on each dataset. 3.The discussion of related work could be more comprehensive. For example, TS-LIF [1] also investigates temporal gradient optimization, albeit from a different perspective. In addition, ETC [2] explores temporal consistency, which is closely related to the proposed LSCR component, but is not discussed. [1] Resolving the Timestep Scaling Paradox in Spiking Neural Networks with a Timestep-Scalable Neuron Model. ICML, 2026. [2] Improving Stability and Performance of Spiking Neural Networks through Enhancing Temporal Consistency. PR, 2025.

Questions for authors

1.Can the authors provide controlled comparisons with recent temporal optimization methods, particularly ETC, TMC, and DeepTAGE, under identical architectures and training settings? 2.Since QTR accounts for most of the performance gains, could the authors provide additional experiments to demonstrate the necessity of combining LSCR and TTS with QTR? 3.Why should more consistent gradients at later timesteps necessarily be more beneficial for optimization? 4.I would be willing to raise my score if the authors can adequately address these weaknesses and questions.

6 comments
AuthorsOpenReview ↗

Response to W1 & Q2: Necessity and complementary roles of TAL components.

We thank the reviewer for this important comment. We agree that QTR accounts for the majority of the accuracy improvement, and acknowledge that the additional gains from LSCR and TTS are relatively modest. However, the three components address different aspects of temporal supervision rather than serving as interchangeable mechanisms. Specifically, QTR adjusts the relative importance of temporal losses, TTS relaxes supervision on less mature early states, and LSCR constrains local prediction variations among later states. We therefore further examine whether LSCR and TTS provide meaningful refinements beyond QTR.

(1) Additional controlled component ablations

We conduct additional ablations on CIFAR10DVS under matched training settings, including the previously missing QTR+LSCR and QTR+TTS configurations. The results show that both components provide incremental improvements beyond QTR. Adding LSCR to QTR improves accuracy by 0.2%, while adding TTS improves it by 0.1%. Importantly, the component-removal comparisons further show that introducing LSCR into QTR+TTS improves accuracy from 86.6% to 86.8%, whereas introducing TTS into QTR+LSCR improves accuracy from 86.7% to 86.8%.

MethodAccuracyEarly–late gradient similarityLate-step logit variation
QTR86.5%0.0520.0042
QTR+LSCR86.7%0.0650.0009
QTR+TTS86.6%0.0760.0023
TAL86.8%0.0790.0005

(2) Distinct effects on temporal behavior

Beyond accuracy, we examine whether the additional components affect the temporal characteristics they are designed to address. For LSCR, adding it to QTR+TTS reduces late-step logit variation from 0.0023 to 0.0005 while improving accuracy from 86.6% to 86.8%. Although suppressing logit variation is consistent with the explicit regularization objective, the accompanying accuracy improvement indicates that this constraint does not merely reduce output variation at the expense of classification performance. For TTS, adding it to QTR+LSCR increases early–late gradient cosine similarity from 0.065 to 0.079, alongside an accuracy improvement from 86.7% to 86.8%. A similar trend is observed when adding TTS to QTR alone, where gradient similarity increases from 0.052 to 0.076. These results are consistent with TTS contributing to the coordination of early- and late-step supervision. Together, these observations indicate that LSCR and TTS provide distinct refinements, with the full TAL objective achieving the highest accuracy and gradient similarity and the lowest late-step logit variation among the evaluated configurations.

(3) Additional evidence supporting temporally differentiated designs

The temporal-boundary analyses in the original manuscript provide further evidence for the design choices underlying LSCR and TTS. For LSCR, Table 8 shows that starting consistency regularization too early substantially degrades accuracy. On CIFAR10DVS, accuracy increases from 80.5% at tlater=2t_{\mathrm{later}}=2 to 86.8% at tlater=5t_{\mathrm{later}}=5, remaining at 86.8% for later evaluated boundaries. This supports restricting consistency regularization to relatively mature temporal states rather than imposing it indiscriminately across the entire sequence. For TTS, Table 7 shows that extending target smoothing to later timesteps is not necessarily beneficial. On CIFAR10DVS, accuracy decreases from 86.8% at tearly=3t_{\mathrm{early}}=3 to 84.9% at tearly=4t_{\mathrm{early}}=4. This supports the motivation for relaxing supervision primarily at early timesteps while preserving stronger supervision at later stages. These sensitivity analyses complement the component ablations by showing that the temporal placement of the two mechanisms matters, rather than treating consistency regularization and target smoothing as uniformly applied auxiliary objectives.

(4) Clarification of the contributions

We agree that the results do not establish that all three components are strictly indispensable. Our intended claim is more specific: QTR provides the dominant performance gain, while LSCR and TTS offer complementary, temporally targeted refinements that can be combined to improve both accuracy and the corresponding temporal behavior metrics.

AuthorsOpenReview ↗

Response to W2: Generalization across network architectures

We thank the reviewer for this constructive suggestion. We agree that evaluating multiple architectures, particularly on the same dataset, is important for assessing the generalizability of TAL. Our original experiments primarily adopted standard dataset-specific backbones, which facilitate comparisons with existing methods but provide limited evidence of cross-architecture transferability. To address this concern, we conduct additional evaluations using different convolutional SNN architectures and an attention-based Spiking Transformer.

(1) Cross-architecture validation on visual benchmarks

We additionally evaluate TAL on ResNet20 for CIFAR10DVS, ResNet-18 for CIFAR10, and ResNet-19 for CIFAR100. Within each configuration, the baseline and TAL are evaluated under matched experimental settings, with only the supervision objective changed. TAL consistently improves accuracy across all three additional configurations.

表格

DatasetArchitectureBaselineTAL
CIFAR10DVSResNet2083.3%84.5%
CIFAR10ResNet-1895.72%96.53%
CIFAR100ResNet-1981.48%82.71%

(2) Extension to an attention-based Spiking Transformer

To further examine whether TAL generalizes beyond convolutional SNNs, we evaluate it on the Spiking Heidelberg Digits (SHD) dataset using TIM (IJCAI 2024), a Spikformer-based architecture equipped with a temporal interaction module. Under matched experimental settings, TAL achieves 86.9% accuracy, outperforming both the original TIM training objective (86.1%) and TET-style uniform temporal supervision (85.6%) by 0.8% and 1.3%, respectively. This result extends our evaluation to an attention-based Spiking Transformer and a non-visual auditory classification task. Importantly, TIM enhances temporal feature interactions within the network, whereas TAL coordinates timestep-specific supervision at the training-objective level. Their combination achieves improved performance without modifying the original attention mechanism, suggesting that architectural temporal modeling and temporally differentiated supervision can provide complementary benefits.

(3) Architectural applicability and generalization

TAL operates on timestep-specific predictions and losses without requiring backbone-specific architectural modifications or introducing additional inference overhead. The additional experiments, together with our original results, demonstrate consistent performance improvements across the evaluated VGG-based, ResNet-based, and Spiking Transformer architectures. In particular, the CIFAR10DVS experiments provide validation across different backbones on the same dataset, while the SHD experiment further extends the architectural and task coverage. Although these evaluations do not exhaustively cover all SNN architectures, they provide stronger empirical evidence that TAL's benefits are not confined to a particular network design.

AuthorsOpenReview ↗

Response to W3: Relationship to TS-LIF and ETC.

We thank the reviewer for highlighting these closely related studies. We agree that both TS-LIF and ETC are important to the discussion of temporal optimization in SNNs. In particular, ETC has already recognized that inconsistent temporal outputs can lead to conflicting optimization directions. We will explicitly acknowledge these contributions and clarify how TAL differs in its analysis and optimization formulation.

(1) Comparison with TS-LIF (ICML 2026)

TS-LIF investigates the timestep scaling paradox, attributing performance degradation at longer simulation horizons to vanishing temporal gradients and weakened cross-timestep dependencies. It addresses this problem by modifying neuronal dynamics through long-term memory reconsolidation and periodic temporal forgetting. The key distinction lies in the gradients being investigated. TS-LIF primarily studies temporal gradient propagation through neuronal states during BPTT, such as the derivatives of later membrane potentials with respect to earlier states. In contrast, TAL analyzes the directional relationships among timestep-specific supervision gradients with respect to shared network parameters. Accordingly, TS-LIF improves how temporal information and gradients propagate through neuronal dynamics, whereas TAL coordinates how supervision gradients from different timesteps contribute to parameter optimization. TAL does not modify neuronal state transitions or introduce additional temporal propagation paths. These approaches therefore address different aspects of temporal learning, although their potential complementarity remains to be evaluated empirically.

(2) Comparison with ETC (Pattern Recognition 2025)

We acknowledge that ETC is particularly relevant to our work. ETC identifies temporal output inconsistency as a source of optimization difficulties and introduces a consistency-based training objective. Our contribution is not the general observation that temporal gradients may conflict, nor the use of temporal consistency regularization itself. Instead, TAL further characterizes the stage-dependent structure of pairwise supervision-gradient relationships. As shown in Fig. 4, early–late gradients exhibit weak alignment, whereas gradients among later timesteps show substantially stronger consistency. These observations motivate differentiated treatment of temporal states rather than imposing uniform supervision or consistency constraints throughout the sequence. The two methods also differ in their overall training formulations. ETC combines sequence-level cross-entropy on temporally averaged outputs with an all-pair consistency objective based on temperature-scaled KL divergence between timestep-wise probability distributions, using stop-gradient targets. In contrast, TAL directly coordinates timestep-specific classification losses through quadratic temporal reweighting (QTR) and temporal target smoothing (TTS), while employing later-step consistency regularization (LSCR) as an auxiliary constraint. Thus, TAL explicitly differentiates both the relative importance and target confidence of temporal states. Regarding consistency regularization, ETC encourages agreement across different timestep pairs, whereas LSCR applies an L2L_2 penalty only to adjacent late-step logits, leaving early states unconstrained by this regularizer. This choice is motivated by the observed temporal asymmetry: early states undergo more substantial evolution, while later states exhibit stronger local consistency. Our boundary-sensitivity analysis in Table 8 further supports this design choice. On CIFAR10DVS, applying consistency regularization from timestep 2 yields 80.5% accuracy, whereas restricting it to timestep 5 or later achieves 86.8%. This suggests that the temporal placement of consistency constraints is important and that imposing them too early can interfere with useful temporal evolution.

(3) Empirical comparison and contribution

The additional controlled comparisons reported in our response to Q1 provide further empirical evidence. Under the evaluated matched configurations, the complete TAL objective achieves 86.8% versus 85.6% for ETC on CIFAR10DVS. These results support the effectiveness of the overall TAL objective, although they do not independently establish the superiority of LSCR over ETC's consistency regularizer. Overall, we position TAL as a temporally differentiated supervision framework motivated by the observed stage-dependent structure of supervision gradients, rather than as a new temporal consistency mechanism or a solution to temporal gradient vanishing. Its main contribution lies in coordinating temporal loss weighting, target confidence, and selective late-stage regularization according to the evolving characteristics of SNN states.

Reviewer_oAvUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes temporal-aware learning (TAL), which improves the performance of spiking neural networks (SNNs) by weighting time-step losses, applying consistency regularization, and smoothing labels. To demonstrate the effectiveness of their method, the authors conducted experiments on static and neuromorphic datasets.

Critical strengths
  1. This paper gives more weight to later time steps because they tend to be more stable and produce better results.
  2. Consistency regularization and label smoothing further improve the consistency and generalization of the SNN.
Critical weaknesses
  1. The proposed method lacks novelty. The analysis in this paper reveals that SNNs exhibit greater stability and better performance at later time steps. However, this is already common knowledge in the field of SNNs. Furthermore, since existing methods have already implemented consistency regularization [1] and label smoothing [2] for SNNs, the proposed method is not innovative.
  2. Rather than simply combining these three strategies to achieve performance improvements, the author should explain in detail why optimization should be based on time-weighted, consistency-regularized, and label-smoothed strategies.
  3. TAL requires many hyperparameters, making fine-tuning its performance tedious. The hyperparameter sensitivity experiments in the appendix, in particular, demonstrate that using inappropriate hyperparameters significantly degrades SNN performance compared to the baseline, further limiting the proposed method's generality.
  4. Since the proposed method is designed to address the temporal characteristics of SNNs, would it be effective for tasks with strong temporal dependencies, such as speech or long-video understanding?
  5. This paper lacks a discussion and comparison with the latest work on SNNs, such as the study published in 2026. Furthermore, Table 3 shows that the baseline used in this paper achieved an accuracy of 85.0% on CIFAR10DVS. This result outperforms most of the methods listed in Table 1 and calls into question the proposed method's performance advantage.
[1] Rethinking Spiking Neural Networks from an Ensemble Learning Perspective. ICLR. 2025.
[2] Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks. NeurIPS. 2024.
Questions for authors

Authors should focus on explaining the innovative and unique aspects of their work.

7 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Clarification of the novelty and distinct contributions of TAL.

We sincerely thank the reviewer for highlighting the importance of methodological novelty. We agree that improved prediction reliability at later timesteps, consistency regularization, and label smoothing have been explored in prior SNN studies. We would like to clarify that the central contribution of TAL is not the individual loss components or the observation that later predictions are more accurate, but a temporal supervision perspective motivated by the directional relationships among timestep-specific task-loss gradients. We elaborate on this distinction below.

(1) Core insight: From temporal prediction evolution to temporal gradient relationships

While previous studies have recognized that later temporal states generally yield more reliable predictions, prediction accuracy alone does not characterize how supervision at different timesteps interacts during optimization. Since SNN parameters are shared across timesteps, the gradients induced by different temporal losses jointly determine the parameter update. Their directional relationships therefore provide an additional perspective beyond prediction reliability or output stability. To investigate these relationships, we analyze pairwise cosine similarities among timestep-specific task-loss gradients. As shown in Fig. 4, under timestep-specific supervision, early–late gradients exhibit weak alignment (−0.010-0.010), whereas late–late gradients show substantially stronger consistency (0.340.34). A similar temporal asymmetry is observed using diagnostic per-step gradients under sequence-level supervision (−0.016-0.016 versus 0.740.74). Furthermore, Appendix A.4 (Fig. 11) shows that this temporal gradient structure persists across different training stages. These observations reveal a distinction between the predictive reliability of temporal states and their directional contributions to shared-parameter optimization. In particular, uniformly aggregating timestep-specific gradients does not explicitly distinguish their different directional relationships, potentially underutilizing the locally coherent gradient tendency among later states. This motivates coordinating temporal supervision rather than simply emphasizing later predictions based on their higher accuracy.

(2) Methodological contribution: Coordinating supervision according to temporal optimization characteristics

Motivated by the above gradient structure, TAL coordinates temporal supervision along three complementary dimensions. Quadratic Temporal Reweighting adjusts the relative contributions of timestep-specific classification losses, increasing the emphasis on later gradients while preserving non-zero supervision for earlier states. As analyzed in Eqs. (8)–(10), this changes the aggregate gradient composition toward the locally coherent late-step direction under the observed gradient relationships. Importantly, this is a local, conditional interpretation, rather than an assumption that later gradients are globally optimal. The quadratic schedule is a simple fixed instantiation of this gradient-informed principle, with its empirical suitability supported by Table 4.

Temporal Target Smoothing adjusts supervision strictness across temporal states. Rather than applying a fixed smoothing coefficient uniformly, TTS relaxes early-step targets and progressively restores one-hot supervision as temporal information accumulates. Later-step Consistency Regularization selectively constrains adjacent late-step predictions, where temporal states are relatively stable, without imposing the same additional consistency constraint on rapidly evolving early states. The distinction lies in how these mechanisms are coordinated to address different aspects of temporal optimization: QTR controls gradient contributions, TTS controls target confidence, and LSCR controls where prediction consistency is encouraged. We do not claim these individual regularization techniques are new; rather, TAL integrates them into a temporally differentiated supervision objective motivated by the observed gradient relationships.

AuthorsOpenReview ↗

Response to W1 & Q1: Clarification of the novelty and distinct contributions of TAL.

(3) Distinction from the two closely related studies

Compared with [1] (ICLR 2025): We acknowledge that [1] already exploits temporal directionality by guiding earlier temporal subnetworks using later ones. It studies SNNs from a temporal ensemble perspective and improves temporal subnetwork stability through membrane-potential smoothing and inter-subnetwork guidance. TAL addresses a different optimization question: how timestep-specific classification-loss gradients relate to one another when updating shared network parameters, and how their relative supervision contributions should be coordinated. Accordingly, TAL directly modifies timestep-specific task supervision through temporal weighting and target smoothing, together with selective late-stage output regularization. Thus, our distinction is not merely the use of a different consistency penalty or temporal asymmetry itself, but the gradient-relationship analysis and the resulting coordination of task-loss supervision.

Compared with [2] (NeurIPS 2024): This work investigates learnable membrane-potential initialization and proposes TET-S with a fixed label-smoothing coefficient across timesteps, as well as last-timestep supervision (LTS). In contrast, TAL retains supervision at all timesteps while progressively reallocating their contributions through QTR and adjusting target confidence through time-dependent smoothing. Rather than discarding early supervision or applying uniform smoothing, TAL differentiates temporal states according to their evolving optimization characteristics.

(4) Additional empirical evidence supporting the proposed distinction

We further provide controlled experiments to examine the effectiveness of temporally differentiated supervision beyond existing objectives. First, using the released implementation and matched training configurations of [2], TAL improves over TET-S on CIFAR10-DVS (85.7% →\rightarrow 86.3%) and N-Caltech101 (86.19% →\rightarrow 86.84%). With learnable membrane-potential initialization, replacing TET-S with TAL also improves accuracy from 86.4% to 87.2% and from 86.97% to 87.36%, respectively. These are single-run controlled comparisons against independently reproduced baselines, rather than claims of outperforming the best published results. Second, our additional component-wise analysis provides evidence of distinct temporal effects. Adding TTS to QTR improves early–late gradient similarity from 0.052 to 0.076, while adding LSCR to QTR reduces late-step logit variation from 0.0042 to 0.0009. Full TAL achieves the highest accuracy (86.8%) among the evaluated configurations. These results support the complementary roles of the proposed components, while acknowledging that QTR accounts for the largest accuracy gain.

In summary, TAL's contribution lies in identifying temporally asymmetric relationships among timestep-specific supervision gradients and using this optimization perspective to coordinate gradient contributions, supervision strictness, and prediction consistency. This differs from approaches primarily motivated by prediction stability, temporal subnetwork guidance, or uniform target smoothing.

AuthorsOpenReview ↗

Response to W2: why optimization should be based on time-weighted, consistency-regularized, and label-smoothed strategies.

We sincerely thank the reviewer for this insightful comment. We agree that simply combining temporal reweighting, consistency regularization, and label smoothing would not constitute sufficient methodological justification. Our design is instead motivated by the observation that different temporal states exhibit distinct optimization characteristics in terms of gradient relationships, prediction reliability, and output stability. To address these differences, we coordinate temporal supervision along three complementary dimensions. These components are not redundant tricks, but complementary mechanisms designed to address different aspects of temporal optimization.

(1) Why Quadratic Temporal Reweighting (QTR)? Allocating Temporal Optimization Contributions

Our gradient analysis reveals that early–late gradients exhibit weak directional alignment, whereas gradients among later timesteps show stronger mutual consistency (Fig. 4). Under such a gradient structure, uniformly weighting all timestep-specific losses may dilute the locally coherent gradient tendency among later states. Therefore, QTR progressively increases the relative contributions of later states while retaining non-zero supervision for all timesteps (Eqs. 8–10). This reweighting adjusts the composition of the aggregate gradient toward the locally coherent late-step direction. We adopt a quadratic schedule as a moderate convex design: it distinguishes later states more clearly than linear weighting without concentrating supervision as aggressively as exponential weighting. The effectiveness of this choice is further supported by the comparisons of different weighting schedules in Table 4 and Appendix A.3.

(2) Why Temporal Target Smoothing (TTS)? Adjusting the Strictness of Supervision

Early temporal states have relatively low prediction reliability because they have not yet fully integrated temporal information. Applying equally strict one-hot supervision to these less reliable states may exacerbate the mismatch between early-stage supervision and evolving temporal representations. Therefore, TTS applies a temporally varying smoothing coefficient exclusively to early states, progressively reducing the smoothing strength and restoring the original one-hot targets at later timesteps. As quantitatively verified in Table R1, adding TTS to QTR improves early–late gradient similarity from 0.052 to 0.076, supporting its role in mitigating temporal gradient misalignment. Thus, TTS adapts target supervision to the evolving reliability of temporal representations rather than applying uniform label smoothing.

(3) Why Later-step Consistency Regularization (LSCR)? Improving Late-stage Output Stability

Gradient consistency and prediction consistency characterize different aspects of temporal behavior. Although later-step gradients exhibit stronger directional consistency, adjacent late-step logits may still exhibit residual temporal variations. Meanwhile, early states are still undergoing substantial temporal evolution and should not be unnecessarily constrained by additional consistency regularization. Therefore, LSCR applies an L2 penalty exclusively to adjacent late-step logits, selectively regularizing local prediction variations after the temporal states become relatively stable. As shown in Table R1, adding LSCR to QTR reduces late-step logit variation from 0.0042 to 0.0009, supporting its effectiveness in improving late-stage prediction stability.

(4) Why Coordinate the Three Components?

The three components address complementary aspects of temporal supervision: QTR determines how much each timestep contributes to the overall optimization update (Contribution Allocation), TTS determines how strictly each timestep should be supervised (Target Confidence), and LSCR determines where additional prediction consistency should be imposed (Output Stability). They operate on different dimensions of temporal supervision rather than redundantly applying the same regularization. Our component-wise ablations demonstrate that TTS and LSCR provide complementary benefits beyond QTR, while the full TAL achieves the best overall results among the evaluated configurations, including the highest gradient similarity (0.079) and the lowest late-step logit variation (0.0005).

In summary, TAL is motivated by the mismatch between uniform supervision and the evolving optimization characteristics of SNN temporal states. Its three components coordinate temporal supervision through gradient contribution allocation, adaptive target confidence, and selective prediction consistency.

Reviewer_pPFwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes temporal SNN states through firing-rate variability, spatial spike patterns, per-timestep accuracy, and gradient cosine similarity. It proposes TAL, comprising quadratic temporal reweighting (QTR), later-step consistency regularization (LSCR), and temporal target smoothing (TTS). Experiments on static and neuromorphic datasets report improved accuracy and faster convergence measured in training epochs.

Critical strengths

Simple implementation without additional inference overhead. TAL modifies only the training objective and introduces no additional inference modules or trainable parameters.

Critical weaknesses
  1. Insufficient validation against closely related work. Shen et al. (2024) observed low early-timestep accuracy and proposed last-timestep supervision (LTS) and label-smoothed temporal efficient training (TET-S). However, the paper provides no direct comparison with these methods. Table 1 also cites their CIFAR10-DVS and N-Caltech101 results of 86.3% and 84.68%, while omitting the 87.8% and 87.86% reported for IMP+TET-S.
  2. Insufficient evidence linking observations to the proposed mechanisms. Spatially concentrated spike activity does not establish increased task-relevant information. It is also unclear whether low early-timestep accuracy or weak gradient alignment indicates harmful supervision. Although full TAL improves accuracy and gradient consistency, these gains are not attributed to the specific problems each component targets. Separate analyses are needed to determine whether LSCR reduces late-stage logit variation, and TTS mitigates early-stage gradient conflict.
  3. Limited validation across architectures and tasks. TAL is presented as a general SNN training strategy, but evaluation is restricted to VGG- and ResNet-based visual classification. Whether the observed temporal patterns and performance gains extend to Spiking Transformers or tasks beyond visual classification remains unverified.
  4. Insufficient reporting of statistical reliability. The number of independent runs and standard deviations are not reported. The main comparisons and ablations should report means and standard deviations across repeated runs.
  5. Insufficient isolation of component contributions. Most improvements in Table 3 come from QTR. Full TAL improves over QTR alone by 0.30 and 0.18 percentage points on CIFAR10DVS and N-Caltech101, respectively. The (\beta=0) setting in Table 9 corresponds to QTR+TTS, but QTR+LSCR is not reported. Ablations that remove each component from full TAL are needed.
  6. Incomplete specification of weighting functions and hyperparameters. Table 4 requires precise formulas and parameters for the sym-quadratic, exponential, and sym-exponential schedules. Although the role of (\epsilon_{\max}) in TTS is defined, its value and selection procedure are not reported.
Questions for authors
  1. How were the identical accuracies in Table 8 obtained? CIFAR10DVS achieves 86.8% for late-stage boundaries of 5, 6, and 7, while CIFAR10 achieves 96.90% for boundaries of 4 and 5. Were models trained independently for each setting? Please clarify whether these are single-run results or averages, and specify the rounding convention.
  2. What evaluation procedure was used for Observation 3? Figure 3 is described as reporting per-timestep accuracy. Please specify the model’s training horizon and checkpoint, and clarify whether accuracy is computed from individual (O(t)) or outputs averaged over timesteps (1) through (t). Analyzing intermediate states of a fixed-horizon model is valid, but it does not establish that early supervision contributes less. For CIFAR10DVS, both inputs and accumulated states vary over time. How were these effects distinguished from those of the supervision strategy?
5 comments
AuthorsOpenReview ↗

Response to W1: Comparison with closely related work

We thank the reviewer for highlighting the closely related work by Shen et al. (2024). We acknowledge that our original Table 1 omitted their best IMP+TET-S results. We clarify the complete published comparisons below and provide additional controlled experiments to evaluate TAL against TET-S.

(1) Clarification of published results

Shen et al. report 87.8% on CIFAR10-DVS at 48×48 with T=8T=8 and 87.1% at T=10T=10. At 128×128 with T=16T=16, their IMP+TET-S achieves 87.0%, whereas our original TAL achieves 87.4% at the same input resolution and simulation horizon, exceeding their reported result by 0.4%. On N-Caltech101, Shen et al. report 87.86% at 128×128 with T=16T=16. Importantly, the 87.8% result on CIFAR10-DVS was obtained at 48×48 with T=8T=8, whereas our 87.4% result uses 128×128 with T=16T=16. As detailed in their Appendix A.8, Shen et al. also employ different training configurations across input resolutions, including optimization settings, neuronal configurations, and data augmentation. Moreover, matching input resolution and simulation horizon alone does not ensure identical training protocols across studies. We therefore distinguish published reference comparisons from controlled evaluations that isolate the contribution of temporal supervision.

(2) Direct comparisons using Shen et al.'s implementation

To directly evaluate TAL against TET-S, we conduct additional experiments using the publicly released implementation of Shen et al. and their documented training configurations. We evaluate both conventional zero-initialized membrane potentials (without learnable IMP) and learnable IMP. Within each initialization setting, the network architecture, input resolution, simulation horizon, data processing, and training protocol are kept consistent, with only the supervision objective changed. The resulting single-run accuracies are summarized below. Under matched experimental settings, TAL outperforms TET-S both with and without learnable IMP. Without IMP, TAL improves accuracy by 0.60% and 0.65% on CIFAR10-DVS and N-Caltech101, respectively. With IMP, the corresponding improvements are 0.80% and 0.39%. These results provide direct empirical evidence that TAL's improvements over TET-S are not restricted to the use of learnable membrane-potential initialization.

MethodCIFAR10-DVS (48×48, T=8T=8)N-Caltech101 (128×128, T=16T=16)
TET-S85.7%86.19%
TAL86.3%86.84%
IMP+TET-S86.4%86.97%
IMP+TAL87.2%87.36%

(3) Clarification of reproduced and published results

We emphasize that the improvements in the controlled experiments are measured relative to our independently reproduced TET-S baselines, rather than the best published results. Although we use the publicly released code and follow the documented training configurations of Shen et al., our reproduced IMP+TET-S accuracies (86.4% and 86.97%) are lower than their originally reported best results (87.8% and 87.86%). We explicitly acknowledge this discrepancy and do not claim that our IMP+TAL experiments surpass these best published accuracies. Instead, these experiments isolate the effect of replacing TET-S with TAL under matched training conditions.

(4) Methodological distinction and relevance of LTS

Shen et al. primarily investigate learnable membrane-potential initialization and propose TET-S to improve temporal supervision on neuromorphic datasets. In contrast, our work systematically investigates the directional relationships among timestep-specific gradients, revealing weak early–late alignment and stronger late–late consistency. Motivated by these observations, TAL coordinates supervision through quadratic temporal reweighting, later-step consistency regularization, and time-dependent target smoothing. Unlike TET-S, which applies a fixed smoothing coefficient across timesteps, TAL explicitly differentiates both the relative contributions and target confidence of temporal states. Regarding LTS, Shen et al. primarily recommend this approach for static tasks. Their Appendix A.6 reports that LTS accuracy on CIFAR10-DVS decreases from 83.7% at T=4T=4 to 82.3% at T=16T=16, indicating its limitations for dynamic event-based inputs. We therefore focus our neuromorphic comparisons on TET-S, which is the more directly relevant supervision objective for these datasets.

Overall, our original TAL achieves 87.4% on CIFAR10-DVS at 128×128 with T=16T=16, compared with the 87.0% reported for IMP+TET-S at the same resolution and simulation horizon. More importantly, the additional controlled experiments show higher accuracy for TAL than TET-S under both conventional and learnable-IMP initialization settings. Together, these results provide complementary evidence for the effectiveness of temporally differentiated supervision, while distinguishing cross-study comparisons from controlled evaluations.

AuthorsOpenReview ↗

Response to W2 & W5.

Response to W2: Linking temporal observations to the proposed mechanisms.

We thank the reviewer for this important comment. We would like to clarify that our motivation is not based on spatial concentration or low early-step accuracy alone, nor does TAL assume that early supervision is inherently harmful. Instead, we jointly investigate spatial spike activity (Fig. 2), per-step prediction accuracy (Fig. 3), and, most importantly, pairwise temporal gradient relationships (Fig. 4). While Fig. 2 characterizes the evolution of spatial firing patterns, Figs. 3–4 further reveal improving prediction accuracy and a pronounced asymmetry in gradient relationships: early–late gradients exhibit weak alignment (−0.016-0.016/−0.010-0.010), whereas late–late gradients show substantially stronger consistency (�0��0�/�1��1�) under sequence-level/timestep-specific supervision. These observations motivate coordinating rather than eliminating early supervision. As analyzed in Eqs. (8)–(10), QTR adjusts the relative contributions of per-step gradients toward the locally coherent late-step tendency, without assuming that early gradients are uninformative or that later gradients are globally optimal.

Component-specific mechanism analysis. To directly verify whether LSCR and TTS address their intended temporal characteristics, we conduct additional controlled ablations on CIFAR10-DVS, comparing full TAL with variants that individually remove each component.

(1) LSCR: Late-stage prediction stability. Comparing QTR+TTS with full TAL, introducing LSCR reduces late-step logit variation from 0.0023 to 0.0005 (78.3% reduction), while improving accuracy from 86.6% to 86.8%. This provides direct empirical evidence that LSCR suppresses local fluctuations among later temporal outputs, consistent with its intended regularization role.

(2) TTS: Early–late gradient relationships. Comparing QTR+LSCR with full TAL, introducing TTS increases early–late gradient cosine similarity from 0.065 to 0.079. Similarly, adding TTS to QTR alone improves gradient similarity from 0.052 to 0.076. These results indicate that TTS contributes to improved early–late gradient alignment along the resulting optimization trajectories, supporting its role in coordinating early-stage supervision rather than merely applying uniform label smoothing. Together, these component-specific analyses establish clearer empirical connections between our proposed mechanisms and their targeted temporal behaviors. We will incorporate these results and refine the manuscript to distinguish descriptive observations from their optimization implications.

MethodAccuracyEarly-late gradient similarityLate-step logit variation
QTR86.5%0.0520.0042
QTR+LSCR86.7%0.0650.0009
QTR+TTS86.6%0.0760.0023
LSCR+TTS86.1%0.0440.0017
TAL86.8%0.0790.0005

Table: Additional ablation and temporal behavior analysis of TAL components on CIFAR10DVS.


Response to W5: Isolation of component contributions.

We thank the reviewer for this constructive suggestion. We acknowledge that QTR accounts for the majority of the accuracy improvement. However, the three components operate on a coupled temporal optimization process rather than as independent, additive mechanisms. Specifically, QTR adjusts the relative contributions of temporal losses, TTS modifies the early-step supervision targets within these weighted losses, and LSCR regularizes late-stage predictions. Their optimization effects can therefore partially overlap, explaining why their combined accuracy gains need not be additive.

Additional controlled ablations. To better isolate their contributions, we conduct additional experiments under matched training settings, including the previously missing QTR+LSCR configuration. QTR alone achieves 86.5%, while adding LSCR or TTS yields 86.7% and 86.6%, respectively. Combining all three components achieves the highest accuracy of 86.8%. More importantly, the component-removal analyses reveal distinct but complementary effects on temporal behavior: removing LSCR from full TAL increases late-step logit variation from 0.0005 to 0.0023, whereas removing TTS decreases early–late gradient similarity from 0.079 to 0.065. These results indicate that LSCR and TTS provide targeted refinements beyond their modest additional accuracy gains.

Clarification of Table 9. We clarify that β=0\beta=0 in the original Table 9 corresponds to QTR+TTS, which reported 86.5%, whereas the additional ablation reports 86.6%. The latter was obtained from an independent retraining under the same experimental protocol. This small difference is consistent with run-to-run training variability and does not change our main conclusion that QTR provides the dominant accuracy gain. We will clarify the experimental reporting to avoid ambiguity.

AuthorsOpenReview ↗

Response to W3 & W4

Response to W3

We thank the reviewer for this important comment. To directly address the concerns regarding Spiking Transformers and non-visual tasks, we additionally evaluate TAL on the SHD dataset using TIM (IJCAI 2024), a Spikformer-based architecture equipped with a Temporal Interaction Module. This extends our evaluation beyond convolutional SNNs and visual classification.

(1) Cross-architecture and cross-modal validation We integrate TAL into TIM without modifying its network architecture or attention mechanism. Under matched experimental settings, we compare three training objectives: the original TIM objective using label-smoothed cross-entropy (ε=0.1\varepsilon=0.1), achieving 86.1% accuracy; TET-style uniform temporal supervision, achieving 85.6%; and our proposed TAL, achieving 86.9%. TAL outperforms the original TIM objective by 0.8% and TET-style supervision by 1.3%. Notably, uniform timestep-wise supervision through TET decreases accuracy by 0.5% relative to the original TIM objective, whereas TAL achieves the highest accuracy among the three configurations. These results indicate that uniformly supervising all timesteps does not necessarily improve temporal learning, while coordinating supervision according to temporal evolution can provide additional benefits, even in an attention-based SNN.

(2) Orthogonality between temporal feature modeling and supervision optimization Spoken-digit recognition involves discriminative acoustic information that unfolds sequentially, such that early temporal states may be formed from incomplete acoustic evidence. This provides a natural motivation for temporally differentiated supervision. Importantly, TIM and TAL are orthogonal in their design and complementary in their roles: TIM enhances temporal feature interactions within Spiking Self-Attention, whereas TAL coordinates timestep-specific supervision at the optimization-objective level. The improvement of TAL over both the original TIM objective and TET-style supervision demonstrates that architectural temporal interaction modeling alone does not exhaust the benefits of temporal learning; how different temporal states are supervised also matters. These results support the complementary roles of temporal feature modeling and temporal supervision optimization.

(3) Broader applicability TAL operates on timestep-specific predictions and losses without requiring backbone-specific architectural modifications. Together with our existing results on static and neuromorphic visual benchmarks, the SHD experiment provides empirical evidence that TAL can improve an attention-based Spiking Transformer on a non-visual auditory sequence classification task, extending its demonstrated applicability beyond VGG/ResNet-based visual classification.


Response to W4

All runs follow the same architecture, data split, training protocol, and hyperparameter settings, with matched seeds for baseline–TAL comparisons.

(1) Statistical reliability of the main comparisons We report the mean accuracy and standard deviation across independent runs, as summarized below. These repeated evaluations provide additional evidence regarding the reliability of TAL's overall performance improvement, beyond the single-run results reported in the original manuscript.

DatasetBaselineTAL
CIFAR10DVS (T=10T=10)85.1% ±\pm 0.2%86.7% ±\pm 0.1%
N-Caltech101 (T=14T=14)83.07% ±\pm 0.13%86.06% ±\pm 0.07%
DVSGesture (T=20T=20)98.49% ±\pm 0.08%99.19% ±\pm 0.16%
CIFAR10 (T=6T=6)95.52% ±\pm 0.12%96.88% ±\pm 0.15%
CIFAR100 (T=6T=6)80.38% ±\pm 0.17%81.24% ±\pm 0.20%

Table: Accuracy (Mean ±\pm Std) across three independent training runs.

(2) Controlled component ablations For the component ablations, all variants are trained using the same fixed random seed, architecture, data split, training schedule, and hyperparameters, with only the selected loss components changed. This controlled protocol enables direct comparisons under matched conditions while reducing variability associated with differing experimental configurations. We clarify that these ablations are single-run controlled comparisons, rather than multi-seed statistical evaluations. Therefore, we do not interpret the small marginal accuracy gains as statistically established across random seeds. Instead, we complement the ablation results with the component-specific temporal behavior analyses presented earlier. In particular, LSCR reduces late-step logit variation from 0.0023 to 0.0005, while TTS improves early–late gradient similarity from 0.065 to 0.079 when added to QTR+LSCR. These observations provide additional evidence of their targeted effects on temporal optimization. We will clarify the distinction between multi-seed performance evaluations and fixed-seed component analyses in the revised manuscript to ensure transparent and reproducible reporting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.