acceptodds
Under review as a conference paper at ICLR 2027

STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation

Abstract

SNNs offer a route to efficient inference, but their performance still trails that of ANNs. ANN-to-SNN knowledge distillation helps narrow this gap, yet the original training data are often unavailable. Existing data-free knowledge distillation (DFKD) methods synthesize surrogate data using teacher-side priors, especially BN statistics, whose first two moments do not determine even a neuron's pre-reset threshold-crossing behavior. We propose Spike Tail-Aware Relational Synthesis (STARS), an add-on for BN-guided ANN-to-SNN DFKD that augments synthesis with Relational Consistency Alignment, matching cross-sample teacher–student relations, and Tail-Aware Regularization, matching soft exceedance probabilities over teacher-derived feature thresholds. The original teacher priors remain active during synthesis. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet across multiple ANN–SNN pairs show consistent gains over three DFKD baselines, reaching absolute accuracy gains of 4.6% on CIFAR-10 and 6.7% on CIFAR-100. Paired multi-seed and resource measurements further characterize reliability and training cost in the evaluated settings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Data-Free Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qDTYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses data-free knowledge distillation (DFKD) from ANN teachers to SNN students, where original training data are unavailable. The key observation is that existing BN-guided synthesis methods constrain only first and second moments of activations, which do not fully determine threshold-crossing behavior critical for SNNs. The paper proposes STARS, an add-on framework combining: Relational Consistency Alignment (RCA), that preserves cross-sample cosine similarities between teacher and student features and Tail-Aware Regularization (TAR), that matches soft exceedance probabilities over teacher-derived feature thresholds. The proposed method STARS have been evaluated on CIFAR-10, CIFAR-100, and Tiny-ImageNet with VGG and ResNet architectures, showing consistent improvements of 2-6.7% over three DFKD baselines (DeepInversion, CMI, NAYER).

Critical strengths

This paper fills a legitimate gap in existing DFKD methods and motivates the need for additional constraints. The proposed method augments existing BN-guided DFKD methods (DeepInversion, CMI, NAYER) without modifying their teacher priors, enabling fair isolation of its contribution. All three baselines show consistent improvements. Comprehensive experimental evaluation on multiple datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet) and ANN–SNN pairs). The paper shows strong empirical gains of up to 4.6% on CIFAR-10 and 6.7% on CIFAR-100 with NAYER+STARS are substantial in this setting. The method closes approximately 32-45% of the baseline-to-teacher accuracy gap on Tiny-ImageNet.

Critical weaknesses

Limited scope of theoretical justification: Equation (8) and subsequent analysis assume a static image repeated at each timestep (direct coding). This does not extend to event-based or temporally varying inputs, No theoretical guarantee that TAR's finite soft probes at selected quantiles actually improve the relevant tail quantities at the true neuron thresholds or lead to better SNN training.

TAR design choices appear ad-hoc: The quantile set {0.6, 0.7, 0.8, 0.9, 0.95} is specified without principled justification. Table 4 shows monotonic improvement with more quantiles up to M=5, but tests only these cumulative additions—not whether this set is optimal or whether diminishing returns exist at larger M. The softness parameter δ = 0.5 is set without ablation or justification. Pooling discards spatial and temporal order, reducing the method to layer-wise aggregate probabilities rather than preserving finer-grained spike dynamics.

Simulation length and neuron-model results (Tables 5–6) lack paired baselines. The paper states "they do not establish gains for every variant" (p. 8) but these negative statements weaken confidence in generalization. No ablation on layer-aggregation weights w_l; all set to 1/|S| uniformly without justification.

RCA matches cosine similarities of row-normalized features (Eq. 12–14). This is motivated as preserving "global semantic structure" but the connection to SNN performance is never established theoretically or empirically.

Novelty concerns: Relational matching is not new (AdaDFKD, Li et al., 2024, already studied inter-sample relationships; the paper cites this). STARS's contribution is combining RCA with TAR for SNNs, not relational matching itself. TAR is a straightforward application of sigmoid-smoothed survival functions, which is not conceptually novel, though applying it to SNN synthesis is new.

Questions for authors

On TAR hyperparameters: Why were the quantiles {0.6, 0.7, 0.8, 0.9, 0.95} chosen? Were other sets (e.g., deciles, percentiles) tested? How sensitive is the method to δ? Was δ = 0.5 chosen empirically or through ablation? On student dependence: If you train a student with STARS-synthesized data and then freeze the synthetic batch, does that batch transfer to a different student architecture? Can you provide a brief experiment?

On layer selection: Why are selected layers limited to stage finals and global pooling? Did you test intermediate layers or did a systematic search? Appendix Table 7 is cumulative; what is the minimum sufficient set?

On RCA specificity to SNNs: The ANN–ANN experiment (Appendix H) shows RCA gives +0.85%, much less than the +2.23–2.89% in ANN–SNN settings. Is RCA truly addressing an SNN-specific concern, or is it a general DFKD improvement? How would you untangle this?

On comparison with real-data KD: Could you provide a controlled fair comparison where you train an SNN student using NAYER+STARS with synthetic data versus using the same architecture and schedule but with real CIFAR data and standard KD? This would clarify the true cost of the data-free setting.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qDTY (Part 1 of 4)

W1. How does the direct-coding analysis support TAR’s threshold-related constraints and training benefits?

Thank you for recognizing the gap in BN-guided synthesis and asking how the tail constraint relates to spiking. In static-image ANN-to-SNN DFKD with direct coding, TAR constrains pooled feature tails that can inform the first-spike fraction under the conditions below. The equal-moment counterexample shows why this information is missing from BN moments. Repeating an image can still produce time-varying deep-layer currents.

We can also relate these pooled features to actual spiking under explicit conditions. At stage positions, teacher features are pooled pre-ReLU BN outputs, and student features average the actual pre-LIF currents over time and space. Both use xr=(zr−μT)/dTx_r=(z_r-\mu_T)/d_T, where μT\mu_T and vTv_T are the FP32 teacher population mean and variance over batch/channel entries, and dT=max⁡(vT,0)+10−5d_T=\sqrt{\max(v_T,0)+10^{-5}}. Normalizers, quantiles and teacher TAR targets are detached; student input gradients remain active. The additional post-activation global readout is a representation constraint, separate from the current argument below.

Fix a realized batch and pre-LIF stage with zero initial/reset voltage, τ>1\tau>1 and Vth>0V_{\mathrm{th}}>0; take 1≤t≤T1\le t\le T. Set α=1−1/τ\alpha=1-1/\tau, βs=1−αs\beta_s=1-\alpha^s and ut=(Vth/βt−μT)/dTu_t=(V_{\mathrm{th}}/\beta_t-\mu_T)/d_T. For spatial neuron ii, let Ii[k]I_i[k] be its actual input and zS,iz_{S,i} its time/spatial-pooled value, shared within its sample/channel. Define the leak-weighted prefix current and pooling residual by

Ai(s)=∑k=1sαs−kIi[k]∑k=1sαs−k,ei(t)=max⁡s≤t∣Ai(s)−zS,i∣dT.A_i(s)=\frac{\sum_{k=1}^{s}\alpha^{s-k}I_i[k]}{\sum_{k=1}^{s}\alpha^{s-k}},\qquad e_i(t)=\max_{s\le t}\frac{|A_i(s)-z_{S,i}|}{d_T}.

Let FS(t)F_S(t) be the actual fraction of neurons with a first spike by tt, sr(u)=Pr⁡(xr≥u)s_r(u)=\Pr(x_r\ge u) the empirical pooled-feature tail, and Rt(ϵ)=Pr⁡(ei(t)>ϵ)R_t(\epsilon)=\Pr(e_i(t)>\epsilon). For any ϵ≥0\epsilon\ge0, positive integration weights and increasing βs\beta_s give

sS(ut+ϵ)−Rt(ϵ)≤FS(t)≤sS(ut−ϵ)+Rt(ϵ).s_S(u_t+\epsilon)-R_t(\epsilon)\le F_S(t)\le s_S(u_t-\epsilon)+R_t(\epsilon).

Indeed, before the first reset the unreset voltage is βsAi(s)\beta_s A_i(s). On ei(t)≤ϵe_i(t)\le\epsilon, a pooled value above ut+ϵu_t+\epsilon forces a crossing by tt, while a value below ut−ϵu_t-\epsilon prevents one. The residual explicitly retains the temporal and spatial information pooling discards; the statement permits time-varying deep currents.

The next bound accounts for the distance from the effective threshold to a selected probe. For one of the MM selected quantile probes θm\theta_m, set Δ=∣ut−θm∣\Delta=|u_t-\theta_m|, pr(u)=E[σ((xr−u)/δ)]p_r(u)=\mathbb E[\sigma((x_r-u)/\delta)], and ℓ=M−1∑j[pS(θj)−pT(θj)]2\ell=M^{-1}\sum_j[p_S(\theta_j)-p_T(\theta_j)]^2. For ϵ≥0\epsilon\ge0, δ>0\delta>0 and η>0\eta>0,

∣FS(t)−sT(ut)∣≤Mℓ+Rt(ϵ)+∑r∈T,SPr⁡(∣xr−θm∣≤ϵ+Δ+η)+2e−η/δ.|F_S(t)-s_T(u_t)|\le\sqrt{M\ell}+R_t(\epsilon)+\sum_{r\in\\{T,S\\}}\Pr(|x_r-\theta_m|\le\epsilon+\Delta+\eta)+2e^{-\eta/\delta}.

This follows by splitting off the residual event and near-probe mass, bounding sigmoid/indicator error outside that region, and applying the triangle inequality. Here sT(ut)s_T(u_t) is the constant-current first-crossing reference induced by teacher pooled features, not observed ANN spikes. Probabilities are empirical on the fixed batch, with uniform sampling of sample/channel/spatial neurons on the student side.

TAR directly controls the soft discrepancy term. The other terms quantify the uncertainty in relating that control to first spikes.

To test whether TAR helps training, we compare RCA with RCA+TAR on CIFAR-100 ResNet-19 → S-ResNet-19, LIF T=4T=4, with matched initialization, synthesis and KD budgets (Table 1). Entries give top-1 accuracy from one run per setting.

Table 1. TAR added to RCA under matched budgets

MethodTop-1 (%)
NAYER+RCA76.18
NAYER+RCA+TAR78.42

With matched training budgets, adding TAR to RCA raises top-1 from 76.18% to 78.42% (Table 1). This isolates a classification benefit from adding the tail constraint on top of relational alignment. The first-spike analysis in revised Appendix B.4–B.5 explains when the pooled constraint is informative about spiking, through the pooling residual, probe distance and near-probe mass.

AuthorsOpenReview ↗

Response to Reviewer qDTY (Part 2 of 4)

W2 & Q1. What justifies TAR’s quantiles, softness and pooling, and does its synthetic data transfer across students?

Thank you for suggesting a transfer test with a freshly initialized student. The upper-tail placements perform best among the tested probe sets, and frozen STARS images improve both fresh students. Larger LIF inputs are more likely to contribute to threshold crossing, which motivates probing the upper tail. The default probes span the 60th to 95th percentiles, with closer spacing at the upper end. In the placement comparison, we fix M=5M=5, δ=0.5\delta=0.5 and TAR weight 1.

All arms retain RCA and compare the same teacher-normalized pooled features, with detached teacher targets and equal layer weights. The moment control adds squared differences in mean and population variance; the Wasserstein control takes the mean absolute gap between the sorted entries. The added distribution penalty has weight 1, following the manuscript setting; RCA also retains weight 1.

The CIFAR-100 experiments in Table 2 follow the manuscript’s NAYER training settings for ResNet-19 to S-ResNet-19 with four-step LIF. We apply the additional distribution losses during synthesis.

Table 2. Matched distribution controls and probe placement

ConstraintQuantilesTop-1 (%)
NAYER+RCA—76.18
+mean/variance—76.63
+sorted-feature Wasserstein—77.14
+TAR lower{0.05,0.1,0.2,0.3,0.4}76.81
+TAR all-range{0.1,0.3,0.5,0.7,0.9}77.11
+TAR upper-even{0.55,0.65,0.75,0.85,0.95}78.23
+TAR default{0.6,0.7,0.8,0.9,0.95}78.42

The default and evenly spaced upper-tail sets in Table 2 test two different grids within the upper tail.

A smaller δ\delta makes the sigmoid closer to a hard cutoff and concentrates its gradient near the probe. We use 0.5 as the default in teacher-normalized coordinates to retain gradients over a wider band. Table 3 varies softness, probe count and loss weights separately and reports all prespecified settings.

Table 3. One-factor sensitivity with fixed quantile endpoints for the count sweep

Softness δ\delta0.10.250.51
Top-1 (%)77.3977.9978.4278.03
Probe count MM5917
Top-1 (%)78.2378.3478.53
Loss weights (λrca,λtar)(\lambda_{\mathrm{rca}},\lambda_{\mathrm{tar}})(1, 1)(0.5, 1)(1, 0.5)(1, 2)(2, 1)
Top-1 (%)78.4277.9878.0278.5877.88

The fixed-range sweep uses the five-probe grid {0.60, 0.6875, 0.775, 0.8625, 0.95}. Both upper-tail grids in Table 2 use different locations.

Spatial averaging and student time averaging yield one value per image and channel. TAR compares the distribution across those BCBC values. This gives the teacher and student a common statistic for synthesis at the cost of spatial structure and spike timing within each feature map.

To test transfer, we collect images during the final 10 epochs of the NAYER and NAYER+STARS source runs with S-ResNet-19, using the same teacher, sampling rules and per-epoch quotas. We freeze each 50,000-image replay pool at the end of its source run. In Table 4, we train a new S-ResNet-19 and an S-ResNet-18 from scratch on the frozen pools. For each target architecture we match initialization, KD updates and augmentations.

Table 4. Frozen-pool transfer to fresh students

TargetFrozen NAYERFrozen STARS
New S-ResNet-1973.5978.15
S-ResNet-1870.5573.65

With five probes, the two upper-tail sets reach 78.23% and 78.42%, above the lower-tail and full-range sets at 76.81% and 77.11% (Table 2). The equal count isolates the effect of quantile placement. At fixed upper-tail endpoints, going from 5 to 17 probes adds 0.30 points, and δ=0.5\delta=0.5 gives the best accuracy in the tested softness sweep (Table 3). This supports five upper-tail probes at that softness as a practical default.

Replacing the frozen NAYER pool with the STARS pool raises accuracy from 73.59% to 78.15% for fresh S-ResNet-19 and from 70.55% to 73.65% for S-ResNet-18 (Table 4). The second gain shows that the synthesized data also benefit an architecture that did not participate in synthesis.

AuthorsOpenReview ↗

Response to Reviewer qDTY (Part 3 of 4)

W3 & Q2 & Q4. Are robustness, layer design and the gap to real-data KD supported by matched comparisons?

Thank you for suggesting matched robustness and real-data comparisons and for asking about layer selection. STARS improves over the paired NAYER baseline at every tested horizon and neuron setting. Uniform weights across the three stage ends and global readout perform best among the tested layer configurations. With the student update count and cosine schedule matched, STARS narrows the gap to real-data KD. Table 5 uses CIFAR-100 VGG-16 → S-VGG-16 and matches the teacher, initialization, schedule and update counts within each pair. Entries are top-1 accuracy (%).

Table 5. Paired horizon and neuron comparisons

LIF horizon TT248
NAYER65.8668.0968.52
+STARS72.4874.7975.25
Other neurons at T=4T=4IFPLIF
NAYER68.1768.92
+STARS74.8875.21

LIF uses a time constant of 2, IF has no leak, and PLIF starts at the LIF time constant and learns one decay parameter per module during KD. The leaky variants decay the input. All three use FP32 membrane states, threshold 1, a hard reset to zero after each spike, and an ATan surrogate with slope parameter 2 and detached reset. We initialize membrane state at zero before each independent forward and clear it after backward. Student BN updates only during KD and uses fixed running statistics during synthesis and evaluation.

We select stage finals to obtain one pre-activation correspondence per semantic stage, even when the networks have different block counts. The extra global position pools post-final teacher ReLU and student LIF outputs, with time averaging, and constrains a distinct representation. This practical rule avoids intermediate-block matching; it is not an exhaustive search. RN19 has three stage finals plus global pooling. Weights below are normalized to sum one.

Table 6. Layer weighting and subsets

Positions / weightsTop-1 (%)
All four, uniform78.42
All four, early-heavy (4,3,2,1)78.05
All four, late-heavy (1,2,3,4)78.33
Global only77.58
Stages 1,2,3 only78.03

For the real-data comparison, all arms use the same ResNet-19 teacher and matched four-step LIF S-ResNet-19 initializations. Across arms we use SGD (momentum 0.9, weight decay 0.0005, no Nesterov) and match 300,000 student updates with cosine scheduling over 300 active KD epochs. All arms use per-image KL at temperature 20 with squared-temperature scaling, detached teacher targets and time-averaged student logits. The real CE+KD arm adds real-label cross-entropy with weight 1.

Table 7. Matched real-data KD references

Data / objectiveTop-1 (%)
Synthetic NAYER73.95
Synthetic STARS78.42
Real KD-only78.88
Real CE+KD79.51

Real CE+KD adds label supervision and gives a separate reference, 1.09 points above STARS.

We report results from the final scheduled checkpoint of each run. The paired comparisons share the evaluation transform and the same 10,000 CIFAR-100 test images. Synthesis and KD in the data-free arms use synthetic inputs, and the settings are fixed before evaluation.

The 6.29–6.73 point gains in Table 5 show that STARS helps even at two time steps, and across each of the tested neuron models. On RN19, uniform stage-plus-global weighting reaches 78.42%, versus 78.03% for stages alone and 77.58% for the global readout alone (Table 6). This supports keeping both kinds of features for this pair. With the student update budget matched, the gap to real-image KD falls from 4.93 to 0.46 points (Table 7), so STARS recovers most of that gap in this run.

Reviewer_RdqEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data-free knowledge distillation from an artificial neural network (ANN) to a spiking neural network (SNN). Critically, the ANN’s original training data is unavailable so the paper uses a setup where synthetic images are generated to transfer predictions to the student. The authors argue that the statistics commonly used to generate synthetic data, which traditionally rely on BatchNorm statistics of the ANN, leave out information that may matter to the SNN. The authors propose a new method, STARS, that adds two loss components including relational component analysis (RCA) and tail-aware regularization (TAR). Experiments apply STARS to SNNs like DeepInversio, CMI, and NAYER on datasets like CIFAR-10, CIFAR-100 and Tiny ImageNet, reporting about 5% improvement on CIFAR-10 and about 7% improvement on CIFAR-100.

Critical strengths
  • Evidence and baselines: The improvements are substantial and consistent. The paper does a good job of evaluating multiple comparisons in Tables 1 and 2. The improvements over NAYER, the strongest baseline are consistent and useful.
  • Component ablations: TAR and RCA are ablated in Table 3 and Table 10. RCA is the more important component but TAR does add some performance.
  • Mathematical claims: The mathematical claims are scoped well. The assumptions for the claims seem applicable. The equal moment construction in Appendix B.2 seems correct.
Critical weaknesses
  • Explaining the tail objective: I can’t tell why TAR is useful or should help in the experimental applications the authors study. The theoretical argument makes a pretty important assumption that there are two distributions of a scalar current with equal moments. But, the experiments focus on two networks on the same synthetic inputs. The analysis doesn’t establish many important components. For example, why should the pooled teacher features help? Or why should the teacher quantiles capture the student’s relevant operating threshold of the SNN? Could the authors explain TAR and its design choices more clearly?
    • Why can’t I just have a teacher-student mean-variance penalty? Could this be compared under a controlled budget
  • Multi-seed comparisons are only used for some cases. It is imperative that this is applied across more experimental settings and eventually in the entire paper.
  • Comparison to AdaDFKD: The paper is fairly similar to AdaDFKD [1], even though it acknowledges this. The original AdaDFKD paper incorporates student-adaptive inter-sample relationships, which is the RCA loss. This is used in synthetic data generation and student training. This isn’t cosine relation matching but it's relevant to the contribution. Since RCA is providing most gains, I think AdaDFKD needs to be added as a baseline.
  • ANN to ANN: I was wondering about how STARS does when distilling ANN to ANN. But in Appendix H this is tested only with RCA. Why is this? TAR can be applied to ANN students without any problem. I think this will motivate and better explain what is happening with TAR while also saying whether this is more specific for TAR.
  • some writing is a bit repetitive (the same pre-activation rationale appears repetitive on page 5, BN-moment is restated several times)
Questions for authors
  • BatchNorm: How is BatchNorm implemented? Is it set to eval mode or train mode?
  • I’m not familiar with SNN literature but is there a measurement to connect TAR to spiking behavior?
  • Could the normalization and differentiation choices be explained more thoroughly?
6 comments
AuthorsOpenReview ↗

Response to Reviewer RdqE (Part 1 of 4)

W1 & Q2. Why do pooled teacher-tail statistics help the actual SNN, and how are they linked to spiking behavior?

Thank you for asking us to make the link between pooled features and spiking concrete. TAR aligns teacher and student feature tails on the same synthetic images, supplying information that mean and variance leave unspecified. The first-spike bound below explains when the pooled tails can constrain spiking in a direct-coded SNN.

At each stage end we take the teacher’s final BN output before ReLU and the student’s input XX before LIF. We average over space and, for the student, time, yielding one entry per image and channel. The extra global readout pools the final teacher ReLU output and the student LIF spikes averaged over time, before the classifier. Both sides use the same detached teacher normalization. TAR matches sigmoid-smoothed tail probabilities using quantiles 0.6, 0.7, 0.8, 0.9 and 0.95 at softness δ=0.5\delta=0.5. The teacher quantiles mark relative activation levels in each layer without real data. Pooling makes the ANN and SNN features comparable, while discarding spatial and temporal detail.

We compare alternative probe placements while keeping the count fixed (Table 1). Appendix B motivates tail matching through the equal-moment construction. To connect the implemented soft loss to empirical tails, let sr(u)=Pr⁡(xr≥u)s_r(u)=\Pr(x_r\ge u) for the normalized pooled entries of network rr, and let ℓ\ell be the mean squared soft-tail mismatch across the MM probes. At a selected probe θm\theta_m and for any ϵ>0\epsilon>0, we obtain

∣sS(θm)−sT(θm)∣≤Mℓ+∑r∈T,SPr⁡(∣xr−θm∣≤ϵ)+2e−ϵ/δ.|s_S(\theta_m)-s_T(\theta_m)|\le\sqrt{M\ell}+\sum_{r\in\\{T,S\\}}\Pr(|x_r-\theta_m|\le\epsilon)+2e^{-\epsilon/\delta}.

The bound separates soft-tail mismatch, near-probe probability mass and smoothing error.

The input X differs from the charged membrane H. Thresholding acts on H after integration, whereas stage TAR acts on averaged X and the extra readout penalty acts on pooled activation/spike outputs. Repeating an image preserves temporal integration, leak, and reset; upstream spikes also make deep-layer X[t]X[t] vary. The local constant-current first-crossing identity therefore motivates the missing information without equating pooled features with membrane trajectories.

We can also bound the first-spike fraction under LIF dynamics. At a fixed pre-LIF stage and batch, broadcast each image/channel pooled input to its spatial sites. With an equal number of sites per image/channel, uniform weighting over sites recovers the same pooled empirical tail. Let FS(t)F_S(t) be the fraction with a first spike by time tt. For zero initial, reset and resting voltage, a positive threshold and τ>1\tau>1, the constant-current threshold at that horizon in normalized coordinates is ut=(Vth/[1−(1−1/τ)t]−μT)/dTu_t=(V_{\mathrm{th}}/[1-(1-1/\tau)^t]-\mu_T)/d_T. Here μT\mu_T and dTd_T are the teacher mean and scale. For a≥0a\ge0, let Rt(a)R_t(a) be the fraction of neurons whose leak-weighted mean input in any prefix up to tt differs from their pooled input by more than adTa d_T. Then

sS(ut+a)−Rt(a)≤FS(t)≤sS(ut−a)+Rt(a).s_S(u_t+a)-R_t(a)\le F_S(t)\le s_S(u_t-a)+R_t(a).

For neurons outside the residual event, a sufficiently high pooled current guarantees crossing by tt. A sufficiently low one prevents crossing in every prefix. The bound uses actual time-varying currents. Teacher quantiles guide this comparison when a probe lies near utu_t and the residual and near-probe mass are small.

The corresponding spiking diagnostic is the fraction of neurons that have fired by each time step. Appendix B.5 gives the conditions under which pooled tails can bound it. With RCA and the synthesis budget fixed, TAR reaches 78.42% versus 76.63% for moment matching (Table 1). The comparison supports using tail information in synthesis to improve classification in this setting. Direct comparison of first-spike profiles remains future work.

AuthorsOpenReview ↗

Response to Reviewer RdqE (Part 2 of 4)

W2 & W5. How does TAR compare with mean–variance matching and with its full application to ANN students?

Thank you for suggesting a matched moment control and a full ANN ablation. With RCA and the synthesis budget held fixed, TAR outperforms mean–variance matching on the SNN. The full component comparison also shows a benefit from TAR for the ANN student.

We include NAYER as a reference in Table 1. The added-statistic arms retain RCA and use the CIFAR-100 NAYER setting in the paper (ResNet-19 to S-ResNet-19, four-step LIF). They use the same pooled features in teacher-normalized coordinates, detached teacher targets and equal layer weights. During synthesis, the moment control sums squared gaps in mean and population variance. The Wasserstein control averages absolute gaps between sorted entries. The added distribution penalty has weight 1, following the manuscript setting; RCA also retains weight 1.

Table 1. Matched distribution controls and probe placement

ConstraintQuantilesTop-1 (%)
NAYER—73.95
NAYER+RCA—76.18
+mean/variance—76.63
+sorted-feature Wasserstein—77.14
+TAR lower{0.05,0.1,0.2,0.3,0.4}76.81
+TAR all-range{0.1,0.3,0.5,0.7,0.9}77.11
+TAR upper-even{0.55,0.65,0.75,0.85,0.95}78.23
+TAR default{0.6,0.7,0.8,0.9,0.95}78.42

The placement controls hold the probe count at five, softness at 0.5 and TAR weight at 1. Both upper-tail grids score higher than the lower-tail and full-range grids in this run.

Table 2. One-factor sensitivity with fixed quantile endpoints for the count sweep

Softness δ\delta0.10.250.51
Top-1 (%)77.3977.9978.4278.03
Probe count MM5917
Top-1 (%)78.2378.3478.53
Loss weights (λrca,λtar)(\lambda_{\mathrm{rca}},\lambda_{\mathrm{tar}})(1, 1)(0.5, 1)(1, 0.5)(1, 2)(2, 1)
Top-1 (%)78.4277.9878.0278.5877.88

The count sweep uses equally spaced probes with endpoints 0.6 and 0.95. Its five-probe grid differs from both upper-tail grids in Table 1. The softness and RCA/TAR weight sweeps vary one setting at a time.

Appendix H reports the original RCA-only ANN experiment. We extend this comparison to all four RCA/TAR combinations, using the same RN19 teacher, matched ANN/SNN macro-architectures and matched synthesis/student budgets.

Table 3. ANN/SNN component comparison

StudentNAYER+RCA+TARFull STARS
ANN78.2679.1178.4479.32
SNN73.9576.1874.3678.42

At the tested unit loss weights, replacing moment matching with TAR raises SNN accuracy from 76.63% to 78.42% with RCA fixed (Table 1). This supports the tail constraint over the tested moment penalty on the same synthesis base. TAR also gives gains on both student types in Table 3. On its own it adds 0.18 points for the ANN and 0.41 for the SNN. On top of RCA it adds 0.21 and 2.24 points, respectively. The incremental gain from TAR is larger for this SNN, and the positive ANN result shows that the loss is useful there too.

AuthorsOpenReview ↗

Response to Reviewer RdqE (Part 3 of 4)

W3 & W4. How does STARS compare with AdaDFKD, and are its gains reliable across seeds and settings?

Thank you for suggesting the AdaDFKD comparison and broader seed coverage. RCA outperforms the AdaDFKD synthesis relation loss in both matched NAYER pairs, and STARS improves on the baseline at every seed across all 20 main settings. For the module comparison, we use the inter-sample relationship loss from AdaDFKD [1]’s Relationship Refinement Module (R²M), which uses a negative bank and progressively selects negatives by feature similarity. In Table 4, NAYER+AdaDFKD adds this synthesis loss to NAYER. The term uses the teacher’s and student’s final pooled features, with time averaging for the student. RCA uses cosine relations at the matched stages and global readout.

Within each architecture pair, we keep the NAYER generator and base synthesis losses, ANN checkpoint, SNN initialization, four simulation steps, and NAYER student KD objective and schedule fixed. The synthesis relation term uses the feature readouts described above. For each relation term, we compare runs with and without TAR. The two AdaDFKD arms share the original relation weight 0.1, temperature 0.07, bank capacity 12,288 and negative-selection schedule. Adding TAR to either relation loss tests its incremental contribution on the same base.

Table 4. Synthesis relation losses on a shared NAYER base (CIFAR-100 top-1, %). AdaDFKD denotes its R²M synthesis term.

Teacher → studentNAYER+AdaDFKDNAYER+AdaDFKD+TARNAYER+RCANAYER+RCA+TAR
ResNet-19 → S-ResNet-1972.0875.8976.1878.42
ResNet-34 → S-ResNet-1871.1474.3674.7876.97

NAYER+RCA scores 76.18% versus 72.08% and 74.78% versus 71.14% for NAYER+AdaDFKD in the two pairs. With TAR, RCA retains the lead under the feature choices described above. Adding TAR improves the AdaDFKD configurations by 3.81 and 3.22 points and the RCA configurations by 2.24 and 2.19 points. For context, the original NAYER results for the two pairs are 73.95% and 71.89%.

The revision extends three-seed reporting to all main DFKD settings in Table 5. We report top-1 mean and sample SD over paired seeds 0, 2026 and 42. We use the final checkpoints and match the evaluation split and transform within each pair. Tables 1–4 report single-run accuracy.

Table 5. Three-seed results across all main DFKD settings

SettingBase+STARS
CIFAR-10 VGG, DeepInversion66.85±0.0970.14±0.18
CIFAR-10 RN34, DeepInversion73.71±0.2077.04±0.98
CIFAR-10 RN19, DeepInversion77.24±0.7780.83±0.17
CIFAR-10 VGG, CMI78.77±1.2483.15±1.19
CIFAR-10 RN34, CMI82.87±0.6685.61±0.43
CIFAR-10 RN19, CMI85.16±0.2688.24±0.21
CIFAR-10 VGG, NAYER87.17±1.1492.49±0.52
CIFAR-10 RN34, NAYER90.50±0.3694.89±0.72
CIFAR-10 RN19, NAYER91.62±1.1095.78±0.22
CIFAR-100 VGG, DeepInversion35.64±0.5939.66±0.75
CIFAR-100 RN34, DeepInversion41.79±0.2645.99±0.21
CIFAR-100 RN19, DeepInversion41.93±0.3946.89±0.15
CIFAR-100 VGG, CMI55.89±0.3958.94±0.28
CIFAR-100 RN34, CMI57.42±1.1560.34±0.18
CIFAR-100 RN19, CMI60.24±0.4463.81±1.85
CIFAR-100 VGG, NAYER68.03±0.2774.59±0.40
CIFAR-100 RN34, NAYER71.91±0.2276.95±0.16
CIFAR-100 RN19, NAYER73.91±0.2178.78±0.52
Tiny-ImageNet, CMI57.51±0.3460.81±0.38
Tiny-ImageNet, NAYER61.21±0.4764.10±0.26

RCA's 4.10- and 3.64-point advantages on the common NAYER base favor it as the synthesis relation module for these pairs (Table 4). The gains of full STARS also reproduce across the main benchmark. All 60 within-seed comparisons across 20 settings favor STARS (Table 5), covering the tested datasets, architectures and baseline methods. The four RCA/TAR component variants for CIFAR-100 ResNet-19 → S-ResNet-19 are also repeated across three seeds (revised PDF, Appendix Table 20).

In the camera-ready version, we will report mean and sample SD over seeds 0, 42 and 2026 for every remaining experiment, including the remaining ablations.

Reviewer_jzxFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies data-free distillation from a pretrained ANN into a separately initialized SNN. STARS augments batch-normalization-guided synthesis with relational consistency alignment (RCA) and tail-aware regularization (TAR). RCA matches cross-sample relations, while TAR matches soft exceedance probabilities at teacher-derived quantiles. A local first-crossing counterexample motivates the tail constraints. Evaluation covers three CIFAR-10/100 baselines and two Tiny-ImageNet baselines, with NAYER gains reaching 4.6/6.7 percentage points on CIFAR-10/100. Validation includes ablations, paired seeds, and resource profiling.

Critical strengths

The consistent improvements are the strongest support for acceptance. All 18 CIFAR comparisons and both Tiny-ImageNet comparisons improve (Tables 1–2). All nine paired CIFAR-100 NAYER differences are positive (Table 8), supporting reliability within the tested configurations.

Both component ablations show TAR adding approximately 2.2 percentage points beyond RCA (Tables 3 and 10). The measured VGG configuration adds under 1% epoch time and no inference modules (Table 9). I think these complementary gains and low measured cost make STARS practically useful.

Critical weaknesses

The main concern is the distinctive methodological contribution. RCA gives larger standalone gains than TAR and has conceptual precedents in similarity-preserving KD (Tung and Mori, 2019). The cited AdaDFKD is not compared. As acknowledged, the local first-crossing analysis motivates without deriving the pooled, time-averaged TAR objective. Table 4 changes probe count and range together. Generic distribution-matching or equal-count placement controls would help isolate the distinctive value of TAR’s threshold-motivated probes.

Reproducibility details remain missing. Appendix F does not specify student checkpoint selection. TAR’s normalization formula, quantile-gradient handling, and numerical LIF parameters are unspecified. Please provide the surrogate-gradient configuration and optimizer choice, and explain how Algorithm 1 maps to generator updates. The ablations lack reported seed uncertainty and no λ/δ sensitivity study is shown.

Cited work: Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1365–1374, 2019.

Questions for authors
  1. Which rule selected the reported student checkpoints, and which data guided hyperparameter selection?

  2. With RCA and the synthesis budget fixed, how does TAR compare with student-side moment matching or equal-count probe sets differing only in placement?

  3. How does STARS compare with AdaDFKD under a matched ANN-to-SNN protocol?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jzxF (Part 1 of 4)

W1. What is new in the joint objective, and how does the local theory support the implemented TAR?

Thank you for highlighting the complementary gains from RCA and TAR and asking us to clarify the contribution of the joint objective. STARS pairs relational and tail constraints in the synthesis objective for static-image, direct-coded ANN-to-SNN transfer. The local analysis explains what BN moments miss and gives a conditional link from pooled feature tails to first spikes. Similarity-preserving KD [1] matches pairwise activation similarities, and AdaDFKD [2] uses adaptive relations in data-free learning. Our joint objective acts at semantic stages and a separate global readout and retains the base synthesis priors.

For the CIFAR-100 component comparison, we fix the RN19 teacher and match the ANN/SNN student architectures. Entries are top-1 accuracy (%) for one matched run per setting.

Table 1. ANN/SNN component comparison

StudentNAYER+RCA+TARFull STARS
ANN78.2679.1178.4479.32
SNN73.9576.1874.3678.42

Part 4 (Table 5) reports the SNN component ablation across three seeds.

The theory has two roles. First, for a constant local input aa before its first reset, first crossing by tt is a≥Vth/[1−(1−1/τ)t]a\ge V_{\mathrm{th}}/[1-(1-1/\tau)^t]. Equal means/variances can leave this probability different. This identifies information BN moments do not determine; repeated images do not make all deep currents constant.

Second, TAR directly matches sigmoid-smoothed survival probabilities of the actual pooled features. For a realized batch, let XrX_r be a uniformly sampled teacher-normalized pooled entry and sr(u)=Pr⁡(Xr≥u)s_r(u)=\Pr(X_r\ge u). With layer loss ℓ\ell and fixed probe θm\theta_m, for any ε>0\varepsilon>0, ∣sS(θm)−sT(θm)∣≤Mℓ+∑r∈T,SPr⁡!(∣Xr−θm∣≤ε)+2e−ε/δ.\left|s_S(\theta_m)-s_T(\theta_m)\right|\le\sqrt{M\ell}+\sum_{r\in\\{T,S\\}}\Pr\\!\left(|X_r-\theta_m|\le\varepsilon\right)+2e^{-\varepsilon/\delta}. The bound relates the optimized soft discrepancy to the hard tail through near-probe mass and smoothing error.

For the actual time-varying stage inputs, a first-spike bound makes the pooling gap explicit. Fix a batch and a pre-LIF stage with zero initial, reset and resting voltage, a positive threshold and τ>1\tau>1. Let ut=(Vth/[1−(1−1/τ)t]−μT)/dTu_t=(V_{\mathrm{th}}/[1-(1-1/\tau)^t]-\mu_T)/d_T be the constant-current threshold in teacher-normalized coordinates, and Rt(e)R_t(e) the fraction of spatial sites where any leak-weighted prefix mean differs from its image/channel pooled value by more than ee in normalized units. Each image/channel has the same number of spatial sites, so broadcasting its pooled value and weighting sites uniformly recovers the pooled tail. For e≥0e\ge0, the fraction FS(t)F_S(t) of neurons that have spiked by tt satisfies

sS(ut+e)−Rt(e)≤FS(t)≤sS(ut−e)+Rt(e).s_S(u_t+e)-R_t(e)\le F_S(t)\le s_S(u_t-e)+R_t(e).

The probe informs first spikes when the residual is small, the probe lies near utu_t, and little feature mass lies between the probe and the bracketed thresholds. The global readout is a separate representation constraint. Part 2 (Tables 2–3) compares the distribution losses and probe choices.

In the matched SNN, adding TAR to RCA raises top-1 from 76.18% to 78.42% (+2.24 points), compared with a 0.41-point gain for TAR alone (Table 1). The component contrast supports applying the tail constraint alongside the relational objective in synthesis for this SNN.

AuthorsOpenReview ↗

Response to Reviewer jzxF (Part 2 of 4)

W1 & W2 & Q2. Do matched distribution controls, equal-count probes and sensitivity tests isolate TAR’s added value?

Thank you for suggesting matched distribution controls and equal-count probes. At equal probe counts, both upper-tail placements outperform the lower-tail and full-range placements. Default TAR also scores above both generic distribution controls under the same synthesis budget. All arms retain RCA and compare the same teacher-normalized pooled features with detached teacher targets and equal layer weights. The moment control adds squared differences in mean and population variance. The Wasserstein control takes the mean absolute gap between sorted entries. RCA and the added distribution loss each have weight 1.

The CIFAR-100 experiments follow the manuscript’s NAYER training settings for ResNet-19 to S-ResNet-19 with four-step LIF, with the same synthesis budget across arms. We apply the additional distribution losses during synthesis.

Table 2. Matched distribution controls and probe placement

ConstraintQuantilesTop-1 (%)
NAYER+RCA—76.18
+mean/variance—76.63
+sorted-feature Wasserstein—77.14
+TAR lower{0.05,0.1,0.2,0.3,0.4}76.81
+TAR all-range{0.1,0.3,0.5,0.7,0.9}77.11
+TAR upper-even{0.55,0.65,0.75,0.85,0.95}78.23
+TAR default{0.6,0.7,0.8,0.9,0.95}78.42

We compare placements at M=5M=5, δ=0.5\delta=0.5 and TAR weight 1, using the same loss form. Both upper-tail placements perform similarly.

Table 3. One-factor sensitivity with fixed quantile endpoints for the count sweep

Softness δ\delta0.10.250.51
Top-1 (%)77.3977.9978.4278.03
Probe count MM5917
Top-1 (%)78.2378.3478.53
Loss weights (λrca,λtar)(\lambda_{\mathrm{rca}},\lambda_{\mathrm{tar}})(1, 1)(0.5, 1)(1, 0.5)(1, 2)(2, 1)
Top-1 (%)78.4277.9878.0278.5877.88

The count sweep uses equally spaced probes at fixed endpoints 0.6 and 0.95, so its five-probe grid differs from both upper-tail grids in Table 2. The weight sweep varies the RCA and TAR coefficients.

At unit loss weights with RCA and the synthesis budget fixed, default TAR reaches 78.42%, versus 76.63% for moment matching and 77.14% for sorted Wasserstein. This shows a larger incremental benefit from the tail objective than from either generic distribution control in this setting.

At five probes, the two upper-tail sets reach 78.23% and 78.42%, above the lower-tail and full-range sets at 76.81% and 77.11%. Holding the count fixed isolates a benefit from placing the probes in the upper tail.

Within the fixed upper-tail range, going from 5 to 17 probes adds 0.30 points. The tested softness sweep peaks at δ=0.5\delta=0.5, and equal loss weights reach 78.42%, 0.16 points below the best tested weighting. Together these sweeps support five upper-tail probes, δ=0.5\delta=0.5 and equal weights as a practical default for this setting.

AuthorsOpenReview ↗

Response to Reviewer jzxF (Part 3 of 4)

W1 & Q3. How does STARS compare with AdaDFKD under a matched ANN-to-SNN protocol?

Thank you for suggesting the matched AdaDFKD comparison. RCA gives higher top-1 accuracy than the AdaDFKD synthesis relation loss in both matched ANN-to-SNN pairs. For AdaDFKD [2], we add the inter-sample relationship loss from its Relationship Refinement Module (R²M) to NAYER. This loss uses a negative bank and progressively selects negatives by feature similarity. It uses the teacher’s and student’s final pooled features, averaging the student’s over time. RCA uses cosine relations at the matched stages and global readout.

Within each architecture pair, we keep the NAYER generator and base synthesis losses, ANN checkpoint, SNN initialization, four simulation steps, and NAYER student KD objective and schedule fixed. The synthesis relation term uses the feature readouts described above. For each relation term, we compare runs with and without TAR. The two AdaDFKD arms share the original relation weight 0.1, temperature 0.07, bank capacity 12,288 and negative-selection schedule. Adding TAR to either relation loss tests its incremental contribution on the same base.

Table 4. Synthesis relation losses on a shared NAYER base (CIFAR-100 top-1, %). AdaDFKD denotes its R²M synthesis term.

Teacher → studentNAYER+AdaDFKDNAYER+AdaDFKD+TARNAYER+RCANAYER+RCA+TAR
ResNet-19 → S-ResNet-1972.0875.8976.1878.42
ResNet-34 → S-ResNet-1871.1474.3674.7876.97

The original NAYER scores are 73.95% and 71.89% for the two pairs. Adding TAR improves the AdaDFKD configurations by 3.81 and 3.22 percentage points and the RCA configurations by 2.24 and 2.19 points.

On ResNet-19 → S-ResNet-19, NAYER+RCA reaches 76.18% versus 72.08% for NAYER+AdaDFKD. On ResNet-34 → S-ResNet-18, the scores are 74.78% versus 71.14%. RCA retains a 2.53- and 2.61-point lead when both configurations use TAR. The gains with and without TAR favor RCA as the synthesis relation module on the shared NAYER base for these two ANN-to-SNN DFKD pairs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We have uploaded a revised PDF with changes marked in blue, incorporating the clarifications, additional experiments and scope adjustments made in response to the reviews.

We thank all three reviewers for their constructive feedback. qDTY recognized the gap addressed by STARS, its retention of the original teacher priors, and the breadth of the evaluation. RdqE highlighted the consistent gains, useful component ablations and appropriately scoped local analysis. jzxF emphasized the complementary gains and practical value, including the low measured training overhead. Their questions helped us strengthen the following aspects.

1. Connecting TAR to spiking under a clear scope. STARS targets static-image ANN-to-SNN DFKD with direct coding. Repeated images do not imply constant deep-layer currents. Beyond the equal-moment counterexample, Appendix B.4–B.5 now relates soft-tail matching to empirical hard tails and bounds the first-spike fraction under time-varying LIF inputs. The bound explicitly accounts for pooling residuals, probe-to-threshold distance and near-probe mass. This gives a conditional connection to first spikes; classification benefits are evaluated through controlled experiments.

2. Isolating the value of tail constraints. With RCA, pooled features, synthesis/KD budgets and the manuscript's unit loss weights fixed, we compare TAR against teacher–student moment matching and sorted-feature Wasserstein matching.

Table 1. Distribution constraints added to NAYER+RCA (CIFAR-100, ResNet-19 → S-ResNet-19, seed 0).

Added constraintTop-1 (%)
None76.18
Mean/variance76.63
Sorted-feature Wasserstein77.14
TAR78.42

Default TAR leads mean/variance and sorted Wasserstein matching by 1.79 and 1.28 points on the shared base. At five probes, the upper-tail placements reach 78.23% and 78.42%, above the lower-tail and full-range sets at 76.81% and 77.11%. The matched count isolates the benefit of placement. Going from 5 to 17 probes at fixed endpoints adds 0.30 points. Together with the softness and weight sweeps, this supports the default in this setting (PDF Table 5 and Appendix Table 16).

3. Clarifying the contribution relative to relational KD. The revised paper credits similarity-preserving KD and AdaDFKD. With NAYER student training fixed, RCA leads AdaDFKD’s R²M synthesis loss by 4.10 and 3.64 points in the two ANN-to-SNN pairs. Both modules use their respective feature readouts and both gain from adding TAR (PDF Table 6). In the factorial comparison, TAR adds 0.21 points to RCA for the ANN and 2.24 points for the SNN (PDF Table 3). This supports the combination for the tested SNN and shows a smaller positive gain for the matched ANN.

4. Extending reliability beyond selected cases. PDF Tables 1–2 now report mean and sample SD over seeds 0, 2026 and 42 for all 20 main DFKD settings: DeepInversion, CMI and NAYER across all CIFAR-10/100 model pairs, plus CMI and NAYER on Tiny-ImageNet. All 60 within-seed comparisons favor STARS, showing a gain across the tested baselines, architectures and datasets. The repeated RN19 SNN component ablation also gives a TAR increment over RCA at each seed, by 2.24, 3.57 and 2.05 points (Appendix Table 20). For the camera-ready version, we will report mean and sample SD over seeds 0, 42 and 2026 for all remaining experiments, including the remaining ablations.

5. Testing transfer and robustness, and completing implementation details. Paired NAYER controls show gains of 6.29–6.73 points across the tested horizons and neuron models. Frozen STARS pools improve fresh S-ResNet-19 and S-ResNet-18 students by 4.56 and 3.10 points. With 300,000 student updates in each arm, the gap to real-image KD falls from 4.93 to 0.46 points at seed 0 (PDF Tables 7–8 and Appendix Table 18). We also report layer subsets and weightings. The revision specifies BN modes, shared teacher normalization, detached targets/quantiles, neuron and surrogate parameters, optimizer/update counts and final-checkpoint selection, and clarifies Algorithm 1's generator updates (Section 4 and Appendix J.1).

The individual responses give the full comparisons and derivations. We appreciate the reviewers' suggestions, which have made the empirical evidence, implementation and scope clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.