acceptodds
Under review as a conference paper at ICLR 2027

Beyond Firing Less: Activity Objectives Shape Population Organization in a Spike-Driven Transformer

Abstract

An activity objective specifies what training rewards. The statistic measured after training can move opposite to the objective's nominal direction, because training can reorganize which responses remain active. We study this mismatch in a Spike-Driven Transformer using per-input population firing-rate coefficient-of-variation regularization (PopFR-CV), together with mean-rate, lifetime-sparsity, and entropy-style objectives. PopFR-CV explicitly rewards greater firing-rate variation across activation coordinates for each input. Training produces the reverse measured response: evaluation CV falls by more than an order of magnitude, the equal-tensor silent-coordinate fraction increases by over 30 percentage points, mean firing decreases only modestly, and endpoint accuracy remains comparable. Controlled continuations from identical mature states show immediate trajectory separation followed by accuracy recovery. Paired CIFAR-100 endpoints localize the reorganization to strong activity withdrawal from monitored attention pathways; fully retained non-attention responses show little change in mean CV. The low-CV, enlarged-silence endpoint recurs across three static-image datasets within the studied Transformer. Other spiking systems and event data show distinct responses. These findings show that population participation is a distinct outcome of activity regularization that is not reliably predicted by mean firing or by the nominal direction of the optimized statistic.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Spiking Neural Networks, Spiking Transformers, Activity Regularization, Population Sparsity, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rtKDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This empirical study examines how activity objectives alter mean firing, within-input population firing-rate CV, and dataset-level silent-coordinate participation in a four-timestep Spike-Driven Transformer. Maximizing a per-input CV auxiliary statistic paradoxically produces much lower evaluated hard-spike CV and a larger silent set at competitive accuracy. The paper employs multi-seed matched comparisons, common-checkpoint continuations, tensor-aligned endpoint readouts, objective-component tests, per-input/lifetime Hoyer controls, validation-selected confirmation, and several cross-system boundary experiments.

Critical strengths
  1. The observation is unusually informative despite not being a new architectural block. On CIFAR-10, PopFR-CV shifts evaluation CV from 19.15 to 1.664 and the equal-tensor silent-coordinate fraction from 22.05% to 52.81%, while keeping mean accuracy comparable or better (Table 2). This makes the distinction between aggregate firing, per-response dispersion, and population participation tangible.
  2. Strong and candid experimental controls. Paired continuations restore a complete mature model/optimizer/random state, separating intervention from independently trained endpoints. The common-nonzero analysis localizes the aggregate CV shift primarily to attention-path response withdrawal, without claiming proof of functional dispensability. Further Hoyer, numerical-component, seed-level, and cross-system analyses disclose failures and unresolved mechanisms instead of suppressing them.
Critical weaknesses
  1. The main phenomenon is not yet a general property of spiking regularization. Its complete low-CV/large-silence/preserved-accuracy pattern is well reproduced within T=4 SDT on three static-image datasets, but not in the tested native SEW-ResNet, MNIST spiking MLP, or CIFAR10-DVS condition (Appendix D, Tables A12–A14). Because activation inventories, loss strengths, and optimization differ across these systems, those contrasts bound scope without identifying why the response is system-dependent. The authors openly acknowledge this limitation.
  2. The proposed interpretation remains descriptive rather than a validated causal mechanism or efficiency design rule. Attention-path withdrawal is strongly associated with the CV reversal, but an intervention preserving attention occupancy is not performed, and the influences of surrogate gradients, clipping, boundaries, and optimizer history are not uniquely resolved. Importantly, an energy model is already supplied: Appendix G, Table A21 estimates only about 4.1% (CIFAR-10) and 1.4% (CIFAR-100) lower arithmetic energy for PopFR-CV, excluding device overheads. The substantive contribution is population-organization insight, not demonstrated computational acceleration.
Questions for authors
  1. Can the authors provide a targeted, source-matched experiment or analysis that separates architecture from effective regularizer strength and response inventory in one additional system? A broad new benchmark sweep is not necessary to clarify the claimed scope.
  2. Does constraining occupancy in the monitored attention pathways during PopFR-CV adaptation attenuate or redirect the objective–outcome inversion?
  3. Which of the documented local-gradient/surrogate and objective-component diagnostics best rules out simple explanations of the inversion, and what remains unidentified?
1 comment

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, and I believe most of my concerns have been substantially addressed. In particular, the new intervention and branch-removal experiments strengthen the scientific contribution of the paper. However, some concerns remain regarding the generalizability of the findings and the precise mechanism underlying the observed phenomenon. Nevertheless, I believe the paper provides valuable insights into population dynamics in SNNs. Therefore, I will maintain my rating of 3 (Weak Acceptance).

Reviewer_hXoWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This study introduces a regularization loss into a Spike-Driven Transformer (SDT) to encourage higher Population CV. Surprisingly, the authors find that the resulting CV decreases after training. A lower CV suggests a more uniform distribution of firing activity, yet the experiments also show an increase in the number of silent neurons. To investigate these seemingly contradictory observations, the authors conduct further training experiments to identify where the changes occur within the network. They also perform various comparative experiments. However, the results reveal that the observed phenomenon is strongly dependent on the dataset and model architecture.

Critical strengths
  1. The paper identifies some counterintuitive phenomena.
Critical weaknesses
  1. Poor writing and unclear motivation. The abstract and introduction lack a clear logical structure, and the connection between the research question and the proposed methodology is difficult to understand. The introduction fails to establish the significance of the research question, and it is even difficult to determine what exactly the authors aim to investigate.

  2. Limited methodological scope and insufficiently focused experiments. The discrepancy between training objectives and observed outcomes should not be unique to SNNs. It is unclear why the authors restrict their investigation of this CV-related discrepancy to SNNs. Moreover, the phenomenon does not appear to be universal even within SNNs. This makes the findings highly limited in scope, and the paper fails to provide a satisfactory explanation for the observed phenomenon. Furthermore, the experimental design examines numerous minor influencing factors without being guided by a deeper theoretical insight. As a result, the study largely amounts to documenting a collection of empirical observations rather than establishing a concrete scientific finding.

Questions for authors
  1. Why using SNNs? CV can presumably be defined in a similar way for conventional artificial neural networks. Is there a specific reason why SNNs are particularly relevant to the phenomenon being investigated?

  2. What is the truly consistent and generalizable counterintuitive finding in this study? Which observations are specific to particular datasets or architectures? Which findings are genuinely important enough to be included in the main body of the paper?

Reviewer_JrvpOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how activity regularization changes firing patterns in a Spike-Driven Transformer. Its central observation is that an objective rewarding higher per-input firing-rate CV produces much lower measured CV, more silent coordinates. The full pattern recurs across 3 static-image datasets like CIFAR within previous studied architectures, with weaker or different responses elsewhere.

Critical strengths
  1. The paper distinguishes mean firing, within-response dispersion, and dataset-level participation. This reveals changes that would be missed by reporting firing rate alone.
  2. The common-state continuation is a strong control. Restoring the same model, optimizer, scheduler, random state, and data order helps attribute subsequent trajectory differences to the added objective.
  3. The authors investigate several alternative explanations through stabilization controls, related Hoyer objectives, and paired activity outputs. They also retain failed runs and report settings where the complete response does not recur.
  4. The manuscript appropriately distinguishes activity organization from computational savings. It does not assume that more silent coordinates automatically produce faster or more energy-efficient execution.
Critical weaknesses
  1. The Introduction moves too quickly from a brief motivation to the authors’ observations and contributions. Although related work was discussed in Section 2, the Introduction does not sufficiently explain what existing activity objectives have established, what remains unresolved, and why the proposed analysis addresses that gap. A clearer progression from previous studies to the research question would make the contribution easier to assess.
  2. The figures need substantial revision. Figure 1 combines objective definitions, aggregation axes, schematic distributions, and experimental previews, making it difficult to distinguish nominal expectations from measured outcomes. Figure 2 has crowded labels and overlapping conditions. Figure 3 compresses the central attention-path analysis into a small panel, while Figure 4 separates accuracy and mean firing from the CV and silence results by placing them in the appendix.
  3. The interpretation of the headline CV reduction depends strongly on how responses are aggregated. Entirely silent responses receive a stabilized CV value of one, and the large aggregate decrease is associated with attention-path withdrawal. The paired analyses address this issue, but this distinction should be central to the explanation rather than something readers must reconstruct.
  4. The choice of tensor weighting substantially changes the reported magnitude of increased silence. In the common-state experiment, the increase is approximately 32 percentage points with equal-tensor weighting and 15 points with coordinate weighting. Both support a change, but the main text should explain the choice of weighting and present both summaries prominently.
  5. The ending needs clearer synthesis and scope. The Discussion contains a concluding paragraph, but a more explicit conclusion should connect the research question, strongest finding, and practical implication. Important limitations are deferred to Appendix I; the main text should summarize the architecture-specific scope, unresolved mechanism, selection procedures, and absence of measured hardware benefits.
  6. The objective comparisons support different observed operating points more clearly than a general ranking. Exploratory coefficients have different tuning histories, and the matched Hoyer experiment controls initial displacement magnitude without matching update direction or subsequent strength.
  7. Also significant figures are inconsistent, which makes the overall paper less organized and scientific professional.
Questions for authors
  1. Can the Introduction more clearly connect prior studies to the specific unresolved question addressed here?
  2. Can an intervention that preserves attention-path participation test whether withdrawal mediates the CV inversion? How would this compare with deliberate attention suppression followed by adaptation?
  3. Why is equal-tensor weighting the principal summary, and do the main interpretations remain consistent under coordinate weighting and tensor-resolved reporting?
  4. Which system properties appear important/necessary for the complete response? e.g. the contrasting architecture, timestep, and event-data results would benefit from a more unified interpretation.
  5. Mentioned in the weakness, the manuscript would benefit from a clearer conclusion and a concise discussion of its principal limitations in the main text. The central figures should also provide a more integrated presentation of accuracy, firing, dispersion, and participation, with the most informative tensor-level results moved from the supplementary materials to the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.