- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses data-free knowledge distillation (DFKD) from ANN teachers to SNN students, where original training data are unavailable. The key observation is that existing BN-guided synthesis methods constrain only first and second moments of activations, which do not fully determine threshold-crossing behavior critical for SNNs. The paper proposes STARS, an add-on framework combining: Relational Consistency Alignment (RCA), that preserves cross-sample cosine similarities between teacher and student features and Tail-Aware Regularization (TAR), that matches soft exceedance probabilities over teacher-derived feature thresholds. The proposed method STARS have been evaluated on CIFAR-10, CIFAR-100, and Tiny-ImageNet with VGG and ResNet architectures, showing consistent improvements of 2-6.7% over three DFKD baselines (DeepInversion, CMI, NAYER).
Critical strengths
This paper fills a legitimate gap in existing DFKD methods and motivates the need for additional constraints. The proposed method augments existing BN-guided DFKD methods (DeepInversion, CMI, NAYER) without modifying their teacher priors, enabling fair isolation of its contribution. All three baselines show consistent improvements. Comprehensive experimental evaluation on multiple datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet) and ANN–SNN pairs). The paper shows strong empirical gains of up to 4.6% on CIFAR-10 and 6.7% on CIFAR-100 with NAYER+STARS are substantial in this setting. The method closes approximately 32-45% of the baseline-to-teacher accuracy gap on Tiny-ImageNet.
Critical weaknesses
Limited scope of theoretical justification: Equation (8) and subsequent analysis assume a static image repeated at each timestep (direct coding). This does not extend to event-based or temporally varying inputs, No theoretical guarantee that TAR's finite soft probes at selected quantiles actually improve the relevant tail quantities at the true neuron thresholds or lead to better SNN training.
TAR design choices appear ad-hoc: The quantile set {0.6, 0.7, 0.8, 0.9, 0.95} is specified without principled justification. Table 4 shows monotonic improvement with more quantiles up to M=5, but tests only these cumulative additions—not whether this set is optimal or whether diminishing returns exist at larger M. The softness parameter δ = 0.5 is set without ablation or justification. Pooling discards spatial and temporal order, reducing the method to layer-wise aggregate probabilities rather than preserving finer-grained spike dynamics.
Simulation length and neuron-model results (Tables 5–6) lack paired baselines. The paper states "they do not establish gains for every variant" (p. 8) but these negative statements weaken confidence in generalization. No ablation on layer-aggregation weights w_l; all set to 1/|S| uniformly without justification.
RCA matches cosine similarities of row-normalized features (Eq. 12–14). This is motivated as preserving "global semantic structure" but the connection to SNN performance is never established theoretically or empirically.
Novelty concerns: Relational matching is not new (AdaDFKD, Li et al., 2024, already studied inter-sample relationships; the paper cites this). STARS's contribution is combining RCA with TAR for SNNs, not relational matching itself. TAR is a straightforward application of sigmoid-smoothed survival functions, which is not conceptually novel, though applying it to SNN synthesis is new.
Questions for authors
On TAR hyperparameters: Why were the quantiles {0.6, 0.7, 0.8, 0.9, 0.95} chosen? Were other sets (e.g., deciles, percentiles) tested? How sensitive is the method to δ? Was δ = 0.5 chosen empirically or through ablation? On student dependence: If you train a student with STARS-synthesized data and then freeze the synthetic batch, does that batch transfer to a different student architecture? Can you provide a brief experiment?
On layer selection: Why are selected layers limited to stage finals and global pooling? Did you test intermediate layers or did a systematic search? Appendix Table 7 is cumulative; what is the minimum sufficient set?
On RCA specificity to SNNs: The ANN–ANN experiment (Appendix H) shows RCA gives +0.85%, much less than the +2.23–2.89% in ANN–SNN settings. Is RCA truly addressing an SNN-specific concern, or is it a general DFKD improvement? How would you untangle this?
On comparison with real-data KD: Could you provide a controlled fair comparison where you train an SNN student using NAYER+STARS with synthetic data versus using the same architecture and schedule but with real CIFAR data and standard KD? This would clarify the true cost of the data-free setting.
5 comments
Response to Reviewer qDTY (Part 2 of 4)
W2 & Q1. What justifies TAR’s quantiles, softness and pooling, and does its synthetic data transfer across students?
Thank you for suggesting a transfer test with a freshly initialized student. The upper-tail placements perform best among the tested probe sets, and frozen STARS images improve both fresh students. Larger LIF inputs are more likely to contribute to threshold crossing, which motivates probing the upper tail. The default probes span the 60th to 95th percentiles, with closer spacing at the upper end. In the placement comparison, we fix , and TAR weight 1.
All arms retain RCA and compare the same teacher-normalized pooled features, with detached teacher targets and equal layer weights. The moment control adds squared differences in mean and population variance; the Wasserstein control takes the mean absolute gap between the sorted entries. The added distribution penalty has weight 1, following the manuscript setting; RCA also retains weight 1.
The CIFAR-100 experiments in Table 2 follow the manuscript’s NAYER training settings for ResNet-19 to S-ResNet-19 with four-step LIF. We apply the additional distribution losses during synthesis.
Table 2. Matched distribution controls and probe placement
| Constraint | Quantiles | Top-1 (%) |
|---|---|---|
| NAYER+RCA | — | 76.18 |
| +mean/variance | — | 76.63 |
| +sorted-feature Wasserstein | — | 77.14 |
| +TAR lower | {0.05,0.1,0.2,0.3,0.4} | 76.81 |
| +TAR all-range | {0.1,0.3,0.5,0.7,0.9} | 77.11 |
| +TAR upper-even | {0.55,0.65,0.75,0.85,0.95} | 78.23 |
| +TAR default | {0.6,0.7,0.8,0.9,0.95} | 78.42 |
The default and evenly spaced upper-tail sets in Table 2 test two different grids within the upper tail.
A smaller makes the sigmoid closer to a hard cutoff and concentrates its gradient near the probe. We use 0.5 as the default in teacher-normalized coordinates to retain gradients over a wider band. Table 3 varies softness, probe count and loss weights separately and reports all prespecified settings.
Table 3. One-factor sensitivity with fixed quantile endpoints for the count sweep
| Softness | 0.1 | 0.25 | 0.5 | 1 |
|---|---|---|---|---|
| Top-1 (%) | 77.39 | 77.99 | 78.42 | 78.03 |
| Probe count | 5 | 9 | 17 |
|---|---|---|---|
| Top-1 (%) | 78.23 | 78.34 | 78.53 |
| Loss weights | (1, 1) | (0.5, 1) | (1, 0.5) | (1, 2) | (2, 1) |
|---|---|---|---|---|---|
| Top-1 (%) | 78.42 | 77.98 | 78.02 | 78.58 | 77.88 |
The fixed-range sweep uses the five-probe grid {0.60, 0.6875, 0.775, 0.8625, 0.95}. Both upper-tail grids in Table 2 use different locations.
Spatial averaging and student time averaging yield one value per image and channel. TAR compares the distribution across those values. This gives the teacher and student a common statistic for synthesis at the cost of spatial structure and spike timing within each feature map.
To test transfer, we collect images during the final 10 epochs of the NAYER and NAYER+STARS source runs with S-ResNet-19, using the same teacher, sampling rules and per-epoch quotas. We freeze each 50,000-image replay pool at the end of its source run. In Table 4, we train a new S-ResNet-19 and an S-ResNet-18 from scratch on the frozen pools. For each target architecture we match initialization, KD updates and augmentations.
Table 4. Frozen-pool transfer to fresh students
| Target | Frozen NAYER | Frozen STARS |
|---|---|---|
| New S-ResNet-19 | 73.59 | 78.15 |
| S-ResNet-18 | 70.55 | 73.65 |
With five probes, the two upper-tail sets reach 78.23% and 78.42%, above the lower-tail and full-range sets at 76.81% and 77.11% (Table 2). The equal count isolates the effect of quantile placement. At fixed upper-tail endpoints, going from 5 to 17 probes adds 0.30 points, and gives the best accuracy in the tested softness sweep (Table 3). This supports five upper-tail probes at that softness as a practical default.
Replacing the frozen NAYER pool with the STARS pool raises accuracy from 73.59% to 78.15% for fresh S-ResNet-19 and from 70.55% to 73.65% for S-ResNet-18 (Table 4). The second gain shows that the synthesized data also benefit an architecture that did not participate in synthesis.
Response to Reviewer qDTY (Part 3 of 4)
W3 & Q2 & Q4. Are robustness, layer design and the gap to real-data KD supported by matched comparisons?
Thank you for suggesting matched robustness and real-data comparisons and for asking about layer selection. STARS improves over the paired NAYER baseline at every tested horizon and neuron setting. Uniform weights across the three stage ends and global readout perform best among the tested layer configurations. With the student update count and cosine schedule matched, STARS narrows the gap to real-data KD. Table 5 uses CIFAR-100 VGG-16 → S-VGG-16 and matches the teacher, initialization, schedule and update counts within each pair. Entries are top-1 accuracy (%).
Table 5. Paired horizon and neuron comparisons
| LIF horizon | 2 | 4 | 8 |
|---|---|---|---|
| NAYER | 65.86 | 68.09 | 68.52 |
| +STARS | 72.48 | 74.79 | 75.25 |
| Other neurons at | IF | PLIF |
|---|---|---|
| NAYER | 68.17 | 68.92 |
| +STARS | 74.88 | 75.21 |
LIF uses a time constant of 2, IF has no leak, and PLIF starts at the LIF time constant and learns one decay parameter per module during KD. The leaky variants decay the input. All three use FP32 membrane states, threshold 1, a hard reset to zero after each spike, and an ATan surrogate with slope parameter 2 and detached reset. We initialize membrane state at zero before each independent forward and clear it after backward. Student BN updates only during KD and uses fixed running statistics during synthesis and evaluation.
We select stage finals to obtain one pre-activation correspondence per semantic stage, even when the networks have different block counts. The extra global position pools post-final teacher ReLU and student LIF outputs, with time averaging, and constrains a distinct representation. This practical rule avoids intermediate-block matching; it is not an exhaustive search. RN19 has three stage finals plus global pooling. Weights below are normalized to sum one.
Table 6. Layer weighting and subsets
| Positions / weights | Top-1 (%) |
|---|---|
| All four, uniform | 78.42 |
| All four, early-heavy (4,3,2,1) | 78.05 |
| All four, late-heavy (1,2,3,4) | 78.33 |
| Global only | 77.58 |
| Stages 1,2,3 only | 78.03 |
For the real-data comparison, all arms use the same ResNet-19 teacher and matched four-step LIF S-ResNet-19 initializations. Across arms we use SGD (momentum 0.9, weight decay 0.0005, no Nesterov) and match 300,000 student updates with cosine scheduling over 300 active KD epochs. All arms use per-image KL at temperature 20 with squared-temperature scaling, detached teacher targets and time-averaged student logits. The real CE+KD arm adds real-label cross-entropy with weight 1.
Table 7. Matched real-data KD references
| Data / objective | Top-1 (%) |
|---|---|
| Synthetic NAYER | 73.95 |
| Synthetic STARS | 78.42 |
| Real KD-only | 78.88 |
| Real CE+KD | 79.51 |
Real CE+KD adds label supervision and gives a separate reference, 1.09 points above STARS.
We report results from the final scheduled checkpoint of each run. The paired comparisons share the evaluation transform and the same 10,000 CIFAR-100 test images. Synthesis and KD in the data-free arms use synthetic inputs, and the settings are fixed before evaluation.
The 6.29–6.73 point gains in Table 5 show that STARS helps even at two time steps, and across each of the tested neuron models. On RN19, uniform stage-plus-global weighting reaches 78.42%, versus 78.03% for stages alone and 77.58% for the global readout alone (Table 6). This supports keeping both kinds of features for this pair. With the student update budget matched, the gap to real-image KD falls from 4.93 to 0.46 points (Table 7), so STARS recovers most of that gap in this run.
Response to Reviewer qDTY (Part 1 of 4)
W1. How does the direct-coding analysis support TAR’s threshold-related constraints and training benefits?
Thank you for recognizing the gap in BN-guided synthesis and asking how the tail constraint relates to spiking. In static-image ANN-to-SNN DFKD with direct coding, TAR constrains pooled feature tails that can inform the first-spike fraction under the conditions below. The equal-moment counterexample shows why this information is missing from BN moments. Repeating an image can still produce time-varying deep-layer currents.
We can also relate these pooled features to actual spiking under explicit conditions. At stage positions, teacher features are pooled pre-ReLU BN outputs, and student features average the actual pre-LIF currents over time and space. Both use xr=(zr−μT)/dT, where μT and vT are the FP32 teacher population mean and variance over batch/channel entries, and dT=max(vT,0)+10−5. Normalizers, quantiles and teacher TAR targets are detached; student input gradients remain active. The additional post-activation global readout is a representation constraint, separate from the current argument below.
Fix a realized batch and pre-LIF stage with zero initial/reset voltage, τ>1 and Vth>0; take 1≤t≤T. Set α=1−1/τ, βs=1−αs and ut=(Vth/βt−μT)/dT. For spatial neuron i, let Ii[k] be its actual input and zS,i its time/spatial-pooled value, shared within its sample/channel. Define the leak-weighted prefix current and pooling residual by
Ai(s)=∑k=1sαs−k∑k=1sαs−kIi[k],ei(t)=maxs≤tdT∣Ai(s)−zS,i∣.
Let FS(t) be the actual fraction of neurons with a first spike by t, sr(u)=Pr(xr≥u) the empirical pooled-feature tail, and Rt(ϵ)=Pr(ei(t)>ϵ). For any ϵ≥0, positive integration weights and increasing βs give
sS(ut+ϵ)−Rt(ϵ)≤FS(t)≤sS(ut−ϵ)+Rt(ϵ).
Indeed, before the first reset the unreset voltage is βsAi(s). On ei(t)≤ϵ, a pooled value above ut+ϵ forces a crossing by t, while a value below ut−ϵ prevents one. The residual explicitly retains the temporal and spatial information pooling discards; the statement permits time-varying deep currents.
The next bound accounts for the distance from the effective threshold to a selected probe. For one of the M selected quantile probes θm, set Δ=∣ut−θm∣, pr(u)=E[σ((xr−u)/δ)], and ℓ=M−1∑j[pS(θj)−pT(θj)]2. For ϵ≥0, δ>0 and η>0,
∣FS(t)−sT(ut)∣≤Mℓ+Rt(ϵ)+∑r∈T,SPr(∣xr−θm∣≤ϵ+Δ+η)+2e−η/δ.
This follows by splitting off the residual event and near-probe mass, bounding sigmoid/indicator error outside that region, and applying the triangle inequality. Here sT(ut) is the constant-current first-crossing reference induced by teacher pooled features, not observed ANN spikes. Probabilities are empirical on the fixed batch, with uniform sampling of sample/channel/spatial neurons on the student side.
TAR directly controls the soft discrepancy term. The other terms quantify the uncertainty in relating that control to first spikes.
To test whether TAR helps training, we compare RCA with RCA+TAR on CIFAR-100 ResNet-19 → S-ResNet-19, LIF T=4, with matched initialization, synthesis and KD budgets (Table 1). Entries give top-1 accuracy from one run per setting.
Table 1. TAR added to RCA under matched budgets
With matched training budgets, adding TAR to RCA raises top-1 from 76.18% to 78.42% (Table 1). This isolates a classification benefit from adding the tail constraint on top of relational alignment. The first-spike analysis in revised Appendix B.4–B.5 explains when the pooled constraint is informative about spiking, through the pooling residual, probe distance and near-probe mass.