acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Sparse Expansion for Continual Learning in Fly-Inspired Spiking Networks

Abstract

Artificial neural networks trained on a sequence of tasks tend to overwrite the knowledge of earlier tasks, whereas animals continue to acquire memories throughout life. The fruit-fly mushroom body motivates sparse expansion as one way to reduce this interference, but computational models usually freeze randomly sampled projection-neuron–Kenyon-cell connections. We study whether shaping these connections from unlabeled input improves later class-incremental learning in a spiking network. A global orthogonalizing update is nattractive at Kenyon-cell populations of thousands, where the residual would have to be accumulated over the whole population for every unit and the rule itself is non-local, so we apply it within small groups whose composition is redrawn during training; in expectation the pairwise interaction then becomes symmetric — the form of anti-Hebbian decorrelation — instead of the sequential recursion of the original rule. The learned connections are given the same number of inputs per cell and the same weight norm as the random baseline, frozen, and only a shared readout is trained. Across a synthetic odor stream and two image benchmarks, the learned mapping improves final class-incremental accuracy without labeled replay or test-time task identity. Feature measurements show lower class and task correlation on the image inputs, although active-set overlap increases slightly on the odor stream. The method combines an unsupervised feature stage with a biologically inspired spiking circuit, while requiring access to unlabeled inputs from future classes.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Continual Learning, Spiking Neural Networks, Unsupervised Representation Learning, Biologically Inspired Learning, Sparse Coding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4EXDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether learning sparse PN–KC connections benefits class-incremental learning in a fly-inspired SNN. It adapts Sanger’s hebbian rule to KC groups and derives a symmetric anti-Hebbian form. Pretraining uses unlabeled inputs from all future classes then a shared readout learns sequentially. Experiments on synthetic odors, MNIST, and CIFAR-100 show higher final accuracy and less negative backward transfer than random maps.

Critical strengths
  1. The main experiments improve final accuracy and BWT without labeled replay or additional readout regularization. This provides useful empirical evidence that unsupervised learning is beneficial for subsequent sequential learning.
  2. The learned expansion retains its advantage when SI is applied to the readout, and combining the two gives further gains on odor and MNIST. This supports its value as a modular front end that may be useful for a downstream continual-learning method.
Critical weaknesses
  1. The evaluation assumes access to unlabeled training examples from all future classes before the labeled task sequence begins. This deviates from standard continual learning settings and substantially weakens the motivation for the proposed approach. If we already have access to all data, there is no need for continual learning in the first place.
  2. The baselines are very limited, with almost no comparisons against alternative learning methods.
  3. Most critically, the proposed method is used entirely before continual learning begins and does not directly address the problem of continual learning across sequential tasks. It is better characterized as a pretraining method, yet comparisons with alternative pretraining approaches are missing.
Questions for authors

See weakness.

3 comments

We thank Reviewer 4EXD for noting our empirical soundness, zero-replay gains over random maps, and modularity with Synaptic Intelligence (SI). Below we report mean ± sample SD and address the pretraining framing (W2) and future-class exposure (W1) using final Average Accuracy (AAC) and Backward Transfer (BWT); alternative pretrainers (W2/W3) and joint-vs.-sequential decoupling (Exp C) follow in our next comment.

1. Unsupervised Pretraining Framing & Why Sequential Forgetting Remains Central

We agree with your characterization of offline Phase 1 (P1) in our frozen-map setting as an unsupervised sparse-expansion pretraining stage, and will adopt this framing explicitly in the Abstract, Introduction, and Discussion.

At the same time, unlabeled pretraining does not trivialize class-incremental learning (CIL): P1 observes no class labels, task identities, or task boundaries. As Experiment C (next comment) shows by comparing matched-budget joint readout (A_joint) against sequential CIL (A_seq = AAC), maps with high joint accuracy (RP and zero-bias k-sparse autoencoder, k-SAE) still collapse sequentially (forgetting gap D_map = A_joint - A_seq = 60.27-73.70 pp on MNIST). Unlabeled feature access alone therefore does not solve sequential forgetting; interference resistance in the sparse code drives CIL retention.

2. Eliminating Future-Class Exposure (Initial-Task-Only & Partial-Data Diagnostics)

We test strict initial-task-only restrictions (P1, preprocessing, and Kenyon-cell (KC) firing thresholds restricted strictly to Task 1; matched Random Projection RP uses identical Task-1 data) alongside partial-data class-pool sweeps (all CIFAR-100 pools match a 3.5k-update budget):

Dataset / SplitUnlabeled Pool (classes, n)RP AAC (%)Learned AAC (%)BWT (RP -> Learned)
MNIST 5x2Strict Task-1 (2, n=5)24.24±0.3744.35±3.15 (+20.11)-83.91->-21.79
MNIST 10x1Partial pool (2, n=5)25.28±0.2147.04±1.64 (+21.76)-76.74->-14.72
CIFAR-100 25x4Strict Task-1 (4, n=5)23.94±0.5626.91±0.81 (+2.97)-39.46->-29.55
CIFAR-100 25x4Partial pool (8 / 12 / 20, n=5)23.94±0.5629.89 / 31.57 / 33.03-> -27.7 / -24.3 / -22.5
Odor 5x22Strict Task-1 (22, n=5)64.45±5.5164.45±7.81-43.47->-42.39
Odor 10x11Partial pool (22 / 33, n=5)58.20±1.0664.32 / 73.64-> -38.2 / -26.7

Analysis: Multi-class Task-1 pools bring clear gains with zero future-class access:

  • Strict Task-1: clear gains on MNIST 5x2 (+20.11 pp) and CIFAR-100 (C100) 25x4 (+2.97 pp); Odor 5x22 improves in 4/5 seeds (+1.59 to +9.43 pp; mean 64.45% reflects one outlier seed).
  • Partial-pool scaling: MNIST {0,1} -> 10x1 generalizes well once >=2 classes appear in P1; C100 (4/8/12/20 classes, n=5) rises monotonically (26.91% -> 33.03%) toward Full 34.83%; Odor 10x11 climbs 58.20% (RP) -> 64.32% (22 classes) -> 73.64±3.87% (33 classes, within variance of Full 70.17±4.62%, n=10).
  • Diversity boundary: Minimal pools lacking class variance - single-digit {0} MNIST (22.34-26.51% vs. RP 33.93/32.23%, n=3) and 11-class Odor Task-1 (54.55% vs. RP 59.27%, n=5) - confirm P1 needs initial class diversity (>=2 digits / >=22 odors), not future classes (CIFAR-10 sweeps running).

3. Removing Offline P1 Entirely (Current-Task Online Plasticity)

Starting from untrained RP calibrated only on Task 1 (zero offline P1, zero future exposure) and updating PN-KC (projection neuron to KC) weights online strictly on current-task batches during CIL improves Odor (10x11, n=5, approx. 100 updates/task) AAC from 59.27±1.08% to 68.93±1.50% (+9.66 pp, BWT -44.60 -> -34.29 pp). High-update visual online streams (using lower learning rates tuned for approx. 10x more updates/task) are underway for our follow-up response.

Continuing our response on alternative unsupervised pretrainers (W2/W3) and separating joint feature quality from sequential forgetting via Experiment C (W1/W2):

4. Alternative Unsupervised Pretraining Baselines (Matched Constraints)

Within each benchmark, all pretrainers share identical network architecture, preprocessing, weight-sparsity constraints, threshold homeostasis, and sequential SGD readout settings (metrics: mean ± sample SD, final Average Accuracy AAC, and Backward Transfer BWT):

  1. Oja's rule (strictly synaptic local Hebbian ablation): achieves 68.64 ± 4.95% Odor (n=10; Sanger 70.17%), 63.48 ± 0.16% MNIST (n=5; Sanger 63.79%), and 34.82 ± 0.27% CIFAR-100 (C100, n=5; Sanger 34.83%), confirming local competitive plasticity + homeostasis captures the core benefit.
  2. Projection-neuron (PN) sample dictionary (n=5): non-iterative normalized PN exemplars yield 68.61 ± 3.19% Odor, 38.55 ± 0.26% MNIST, and 29.99 ± 0.41% C100, trailing Hebbian maps on vision.
  3. Spherical K-Means & zero-bias k-sparse autoencoder (k-SAE, n=5 per benchmark):
    • Odor: k-SAE reaches 56.80 ± 1.94% (below Random Projection RP 58.20%), whereas Spherical K-Means achieves 69.48 ± 2.97%, matching Hebbian maps on clustered synthetic odors.
    • MNIST & C100: Spherical K-Means gets 38.51 ± 0.32% / 32.43 ± 0.12%, and k-SAE drops to 10.98 ± 0.34% / 26.63 ± 0.53%, behind Oja/Sanger (63.48-63.79% / 34.82-34.83%) despite zero-bias leaky integrate-and-fire (LIF) compatibility.

5. Separating Feature Quality from Sequential Forgetting (Experiment C)

For each frozen map, we compare a matched-budget joint linear readout (A_joint, all classes mixed; 100/40/50 epochs on Odor/MNIST/C100 vs. 50/20/50 per task sequentially; not an unconstrained upper bound) against sequential class-incremental learning (CIL) (A_seq = AAC, with DAC denoting average diagonal per-task accuracy immediately after each task): D_map = A_joint - A_seq (paired mean from unrounded per-seed values).

BenchmarkMap (n=5 subset)A_jointDACA_seqBWT (pp)D_map (pp)
OdorRP97.18±0.3399.4358.20±1.06-45.8138.98±1.00
Odork-SAE97.34±0.6699.5756.80±1.94-47.5340.55±2.27
OdorSph. K-Means97.16±0.3999.7369.48±2.97-33.6127.68±2.68
OdorOja91.05±0.5296.6670.70±4.05-28.8420.34±4.45
OdorSanger91.66±0.4696.6670.45±5.00-29.1221.20±5.12
MNISTRP84.13±0.3094.9123.86±0.33-78.9460.27±0.18
MNISTk-SAE84.67±0.2998.4210.98±0.34-97.1673.70±0.47
MNISTSph. K-Means70.89±0.2589.1338.51±0.32-56.2432.37±0.42
MNISTSample Dict68.04±0.3787.0938.55±0.26-53.9229.49±0.60
MNISTOja64.98±0.1579.2863.48±0.16-17.561.50±0.16
MNISTSanger65.30±0.3079.6763.79±0.16-17.641.51±0.18
C100RP59.44±0.6364.1423.60±0.41-42.2335.84±0.81
C100k-SAE61.58±0.1972.0326.63±0.53-47.2934.95±0.44
C100Sph. K-Means60.54±0.2381.6132.43±0.12-51.2328.11±0.17
C100Sample Dict60.63±0.4680.6729.99±0.41-52.7930.63±0.69
C100Oja46.84±0.3170.8434.82±0.27-37.5112.02±0.55
C100Sanger46.84±0.3571.0034.83±0.49-37.6812.01±0.58

Takeaway (n=5):

  • High A_joint != low forgetting: RP and k-SAE reach higher A_joint (84.13%/84.67% MNIST; 59.44%/61.58% C100; 97.18%/97.34% Odor) yet collapse sequentially (23.86%/10.98% MNIST; D_map = 60.27 / 73.70 pp).
  • Highest A_seq & sharp D_map drop: Oja/Sanger lead A_seq (63.48-63.79% MNIST, well above RP+SI 33.95%; 34.82-34.83% C100) while cutting D_map to 1.50-1.51 pp (MNIST, vs. RP 60.27), 12.01-12.02 pp (C100, vs. 35.84), and 20.34-21.20 pp (Odor, vs. 38.98).
  • Driver: Lower cross-class active overlap (0.549->0.139-0.140 vs. k-SAE 0.618 MNIST; 0.570->0.342 vs. k-SAE 0.565 C100). We will include all baselines, framing, and protocols in the revision.

Thank you again for your detailed review (W1-W3). Here are the completed follow-up results on zero-offline-P1 online plasticity and partial-class pretraining (W1):

1. Zero-Offline-P1 Online Plasticity (W1, n=5)

Because visual continual streams execute about 10x more PN-KC updates per task than Odor, we keep dynamic KC threshold homeostasis active during Phase 2 (theta updated toward target 5% firing rate) to preserve KC sparsity under ongoing weight updates. Across all three benchmarks (n=5), online plasticity strictly on current-task batches consistently outperforms the matched Task-1 Random Projection (RP) control:

Benchmark (n=5)Task-1 RP AAC (%)Online Best AAC (%)GainBWT (RP -> Online)
Odor 10x1159.27±1.0868.93±1.50+9.66 pp-44.60 -> -34.29
MNIST 5x224.24±0.3737.06±0.27+12.82 pp-83.91 -> -62.04
CIFAR-100 25x423.94±0.5630.18±0.45+6.24 pp-39.46 -> -33.21

2. Completed CIFAR-10 Partial-Class Pretraining & Oja Baseline (W1, n=5)

  • Matching-schedule C10 Oja (n=5): reaches 74.59 ± 0.19% (10x1, BWT -27.56) and 74.74 ± 0.18% (5x2, BWT -28.71), matching Grouped Sanger (74.56 ± 0.18% / 74.55 ± 0.36%).
  • C10 5x2 partial-class pretraining (n=5 per pool): AAC scales monotonically with unlabeled class diversity before CIL (no future-class labels):
C10 5x2 Unlabeled Pool (n)AAC (%)BWT (pp)
Task-1 RP baseline (5)41.98±1.30-70.44
3 classes (Sanger, 5)49.65±1.53-58.98
4 classes (Sanger, 5)56.07±0.62-51.47
5 classes (Sanger, 5)63.11±0.75-42.72
Full 10 classes (Sanger, 5)74.55±0.36-28.91

We hope these n=5 results address your remaining concerns on future-class exposure and online adaptation, and we kindly invite your follow-up feedback before the discussion closes.

Reviewer_SWJUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper is motivated by the idea that projection neurons in the fly generate sparse activity in ‘Kenyon’ cells found in the fly mushroom body. They build a model for class incremental learning based on the idea that this projection could be learned by an unsupervised plasticity rule, which aids in continual learning.

The main difference between their model and the comparison is how the PN-to-KC weights are determined. As a control they use a fixed random projection. For the ‘learned map’ case these weights undergo unsupervised training. The rule they use to generate this learned map is a ‘grouped update’ rule (their equation 3).

In leaky integrate and fire networks they use sequential tasks to demonstrate their method (running the grouped update rule before beginning CIL) alleviates catastrophic forgetting better than the fixed random projection. They show this using synthetic odours, MNIST images and CIFAR-100 features from a pretrained transformer.

Critical strengths

They use a clearly expressed unsupervised plasticity rule & demonstrate it works on a range of tasks using spiking neural networks.

The results are clearly presented and I’m quite confident they are accurate.

The grouped update appears to be a novel contribution and there is some evidence that it does perform better than a local Oja’s rule.

Critical weaknesses

The authors themselves note in the conclusion that the learning rule itself is not a unique contribution (this confused me a little because the grouping does make it a new rule, which I don’t think is introduced elsewhere. They consider the grouping in the rule to be part of their ‘circuit-level design’).

The contribution is then the SNN model, the grouping design and the careful comparisons. Depending on the threshold this might not be considered novel enough.

In table 4.2 the Oja’s rule update is quite close to the grouped update. I think this is a hint the grouped update is better than Oja's rule but I would suggest more random seeds would be useful here.

Unlike Oja's rule, the grouped update is not local to a synapse (the authors do say this), which is a problem for biological plausibility, but not a problem for the CIL results.

The learning rule is expected to separate representations, so I’m not sure the result that it beats a fixed projection is particularly surprising, though it is good to see it working!

Questions for authors

Could you run the Oja’s rule comparison for MNIST and CIFAR-100 & run for more seeds in the odour task? It would be an interesting control. Showing the grouped update does better than just applying Oja’s rule updates would really strengthen the result.

A control where you use unlabeled inputs from another dataset and see if it aids CIL in a new one would be useful. Could you use Fashion-MNIST for the unlabeled inputs and then regular MNIST for the CIL? (Fashion-MNIST should work as a drop-in to the existing method, same pixel input shape, same number of classes). I don't think it's too concerning if this doesn't work though.

3 comments

We warmly thank Reviewer SWJU for the supportive evaluation - for recognizing our unsupervised rule's clarity, empirical consistency, and the grouped Sanger update rule as a methodological contribution - and for prompting multi-seed Oja and transfer controls. We report final Average Accuracy (AAC) and denote each circuit regime by (N_KC, K, WTA, od): KC count N_KC, projection-neuron fan-in K, top-WTA% Winner-Take-All sparsity, and intra-group orthogonalization decay od (16.67 Odor default vs. 2.5 moderate decay).

1. When Grouped Inhibition Helps Beyond Oja (Q1)

Core finding & Table 4.2 revision: Under default sparsity (N_KC=2000, K=6, 5% WTA), Oja and Grouped Sanger are statistically indistinguishable across benchmarks, while both cleanly beat Random Projection (RP):

  • Odor 10x11 (n=10): Oja 68.64 ± 4.95% vs. Sanger 70.17 ± 4.62% (+1.53 pp, p=0.395)
  • MNIST (n=5): 63.48% vs. 63.79%
  • Fashion-MNIST (n=5): 58.86% vs. 58.88%
  • CIFAR-100 (C100, n=5): 34.82% vs. 34.83%

We will revise Table 4.2 accordingly and delineate where intra-group inhibition contributes beyond Oja:

  • Why default sparsity ties: At 5% WTA, intra-group pairs co-activate in only about 9.2% of Odor batch steps (9.7% MNIST), so lateral subtraction rarely triggers and sparse Oja + homeostasis drives the main gain. Even lowering decay to od=2.5 (N_KC=2000: Oja 72.56 vs. Sanger 72.44%, -0.11 pp unrounded, n=2) or tightening capacity (od=2.5, N_KC=600: 75.97 vs. 77.56%; 250: 72.84 vs. 70.62%, n=2) at K=6, 5% WTA keeps both matched within small-sample noise.
  • High-overlap regimes (10-20% WTA): Raising fan-in K and WTA% lifts full-dataset KC pair co-activation to 0.92-1.00, lowers co-active weight cosine (0.150-0.183 vs. Oja 0.190 on 600, 20, 10%), and yields consistent Sanger gains in matched Odor windows plus an early trend on MNIST:
Regime (N_KC, K, WTA, od)nRP / Oja / Sanger AAC (%)Sanger - Oja & Signal
Odor Default (2000, 6, 5%, 16.67)1059.10 / 68.64 / 70.17+1.53 pp (p=0.395, tie; approx. 9.2% batch co-act.)
Odor High-ovlp (2000, 20, 10%, 2.5)239.83 / 41.99 / 45.23+3.24 pp (2/2; dataset co-act. approx. 0.92)
Odor High-ovlp (600, 20, 10%, 2.5)525.48 / 40.98 / 42.80+1.82 pp (trend p=0.148; weight cos 0.183<0.190)
Odor High-ovlp (600, 20, 10%, 16.67)525.48[1] / 40.98[1] / 44.55+3.57 pp (weight cos 0.150; +SI: 46.52%)
Odor Tight ovlp (250, 20, 20%, 2.5)514.55 / 18.34 / 21.73+3.39 pp (5/5, p=0.067)
MNIST High-ovlp (2000, 80, 10%, 2.5)320.96 / 57.15 / 58.67+1.52 pp (2/3, 57.15±2.51 vs. 58.67±0.24%)
  • Bounded regime: Outside those windows, overly strong decay (od=16.67) on tight, dense Odor (250, 20, 20%, 16.67) over-concentrates Sanger weights (effective fan-in 4.05 vs. Oja 6.86, dropping to 16.70% < 18.34%, -1.64 pp; restored to 6.18 and +3.39 pp at od=2.5), and tight high-overlap MNIST (600, 40, 10%, 2.5, n=3) favors Oja (55.29% vs. 50.26%). C100 probes are finishing ([1] RP has zero Phase 1 updates and Oja uses group size 1, so both are od-invariant).
  • Locality framing: We position sparse Hebbian plasticity + homeostasis (strictly local for Oja) as our primary continual-learning driver on sparse circuits, and Grouped Sanger (semi-local intra-group inhibition) as its targeted decorrelation extension under high KC co-activation.

2. Cross-Dataset Transfer (Q2) & Summary of New Controls

  • Cross-dataset transfer (freezing source PN-KC weights, recalibrating target Task-1 KC thresholds):
    • CIFAR-100 (C100) -> CIFAR-10 (C10) (n=5): 38.89 ± 0.38% vs. Task-1-calibrated C10 RP 32.35 ± 0.45% (+6.55 pp unrounded diff, 95% CI [5.90, 7.20], p < 1e-5; shared ViT features, vs. 74.56% in-domain C10).
    • Fashion-MNIST <-> MNIST: Raw pixels do not beat target RP under stroke mismatch.
  • Summary of added controls: Unified runs confirm consistent gains across all five benchmarks (General Response), positive stacking with Synaptic Intelligence (SI: 83.93% Odor, 64.83% MNIST, 36.27% C100), gains without future exposure (partial Phase 1 and Odor online updates), and sharp reduction of sequential forgetting gap (60.27 -> 1.50-1.51 pp on MNIST; details in pd96 and 4EXD).
Reviewer_SWJUOpenReview ↗

Improved soundness

Thank you to the authors for providing these results. The strengthened comparisons with Oja's rule are useful, even if it did not demonstrate that Grouped Sanger performed better overall. I appreciate the detailed investigation of when it may provide an additional benefit

Thank you for providing the cross-dataset transfer experiments.

I have raised the soundness score to 3.

We sincerely thank you for your thoughtful follow-up and for raising the Soundness score to 3. To complete the probe and follow-up experiments referenced earlier, we summarize the final multi-seed results below:

1. High-Overlap Comparison: Grouped Sanger vs. Oja ((N_KC, K, WTA%, od))

To clarify when Grouped Sanger adds value over local Oja, we probe regimes with higher PN-KC fan-in K and wider WTA% firing rates:

Regime (N_KC, K, WTA%, od)nRP AAC (%)Oja AAC (%)Sanger AAC (%)Sanger Advantage
Odor (600, 20, 10%, 16.67)525.48±1.2040.98±0.8144.55±2.76+3.57 pp
Odor (600, 20, 10%, 2.5)525.48±1.2040.98±0.8142.80±1.86+1.82 pp
Odor (250, 20, 20%, 2.5)514.55±0.8618.34±2.4221.73±3.03+3.39 pp
Odor (2000, 20, 10%, 2.5)540.34±2.2941.43±0.8942.57±2.59+1.14 pp
MNIST (2000, 80, 10%, 2.5)521.37±1.6557.61±1.8959.02±0.53+1.41 pp (3.6x lower SD)
C100 (600, 40, 10%, 2.5)512.11±0.5717.70±0.6217.96±0.47+0.26 pp (tighter SD)
C100 (2000, 80, 10%, 2.5)512.14±0.2223.17±0.3123.29±0.26+0.12 pp (tighter SD)
C10 (2000, 80, 10%, 2.5)330.51±0.4083.71±1.4284.24±0.26+0.53 pp (5.5x lower SD)

Increasing fan-in connectivity K and KC firing rate activates Grouped Sanger's intra-group orthogonalization: it translates into direct accuracy gains on raw Odor and pixel MNIST, while on ViT-embedded CIFAR-100/CIFAR-10 features it primarily tightens multi-seed stability (cutting cross-seed SD by up to 5.5x).

2. Completed CIFAR-10 Benchmarks & Visual Online Plasticity (n=5)

  • CIFAR-10 Oja (n=5): matches Grouped Sanger closely across both splits: 74.59 ± 0.19% (10x1, BWT -27.56, vs. Sanger 74.56 ± 0.18%) and 74.74 ± 0.18% (5x2, BWT -28.71, vs. Sanger 74.55 ± 0.36%).
  • CIFAR-10 5x2 partial-class pretraining (n=5): scales monotonically from 49.65 ± 1.53% (3 classes) -> 56.07 ± 0.62% (4 classes) -> 63.11 ± 0.75% (5 classes), vs. Task-1 RP 41.98 ± 1.30% and full 10-class 74.55 ± 0.36%.
  • Zero-offline-P1 visual online plasticity (n=5): because visual streams perform about 10x more PN-KC updates per task than Odor, keeping dynamic KC threshold homeostasis active during Phase 2 preserves KC sparsity and rescues visual online learning:
Benchmark (n=5)Task-1 RP AAC (%)Online Best AAC (%)GainBWT (RP -> Online)
MNIST 5x224.24±0.3737.06±0.27+12.82 pp-83.91 -> -62.04
CIFAR-100 25x423.94±0.5630.18±0.45+6.24 pp-39.46 -> -33.21

(Together with Odor 10x11 68.93 ± 1.50% vs. RP 59.27 ± 1.08%, all three core benchmarks achieve significant n=5 online gains.)

Thank you again for your valuable suggestions throughout the discussion.

Reviewer_pd96OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies data-dependent sparse PN–KC expansion rule for class-incremental learning of SNNs. A grouped generalized Hebbian update is applied to unlabeled inputs before the task sequence; the sparse map is then frozen while a shared readout learns sequentially without labeled replay or test-time task identity. Compared with fixed random projections, final AAC and BWT improves.

Critical strengths
  1. The performance is not bad for such kind of methods.
  2. Well-motivated design with helpful analyses.
Critical weaknesses
  1. The unlabeled pretraining stage accesses future-class training inputs. This assumption is stated, but an initial-task-only variant would help assess a broader continual-learning setting.
  2. Oja and grouping controls are most developed on the odor stream. A small Oja v.s.grouped-GHA comparison would better contextualize the particularly large image gains.
Questions for authors
  1. Could the authors report the effect of restricting unsupervised pretraining to the initial task?
  2. Could the authors add an Oja comparison on one image benchmark, and report approximate Phase-1 runtime or memory cost?
2 comments

We thank Reviewer pd96 for recognizing our biological design, empirical gains, and interpretability analyses, and for two insightful questions. Benchmark tables and Phase 1 (P1) cost appear in General Response. Values report mean ± sample SD (metrics: final Average Accuracy AAC and Backward Transfer BWT).

1. Unsupervised Pretraining on Partial Data (Q1a)

We evaluated strict initial-task-only limits (unlabeled P1, preprocessing, and Kenyon-cell (KC) firing thresholds restricted strictly to Task 1; matched Random Projection RP uses identical Task-1 data) alongside partial-class pretraining before class-incremental learning (CIL; CIFAR-100 pools match a 3.5k-update budget):

Dataset / SplitUnlabeled Pool (classes, n)RP AAC (%)Learned AAC (%)BWT (RP -> Learned)
MNIST 5x2Strict Task-1 (2, n=5)24.24±0.3744.35±3.15 (+20.11)-83.91->-21.79
MNIST 10x1Partial pool (2, n=5)25.28±0.2147.04±1.64 (+21.76)-76.74->-14.72
CIFAR-100 25x4Strict Task-1 (4, n=5)23.94±0.5626.91±0.81 (+2.97)-39.46->-29.55
CIFAR-100 25x4Partial pool (8 / 12 / 20, n=5)23.94±0.5629.89 / 31.57 / 33.03-> -27.7 / -24.3 / -22.5
Odor 5x22Strict Task-1 (22, n=5)64.45±5.5164.45±7.81-43.47->-42.39
Odor 10x11Partial pool (22 / 33, n=5)58.20±1.0664.32 / 73.64-> -38.2 / -26.7

Analysis: Multi-class unlabeled pools bring clear gains with zero future-class exposure:

  • MNIST {0,1}: Strict Task-1 5x2 recovers about 59% of Full 5x2 Sanger gain (+20.11 vs. +34.34 pp); partial {0,1} 10x1 gains +21.76 pp (47.04% vs. 25.28%).
  • CIFAR-100 (C100): scales monotonically across 4/8/12/20 classes (26.91% -> 33.03%) toward Full 34.83%.
  • Odor: 10x11 climbs 58.20% (RP) -> 64.32±2.49% (22 classes) -> 73.64±3.87% (33 classes, within variance of Full 70.17±4.62%, n=10); 5x22 Strict Task-1 improves in 4/5 seeds (+1.59 to +9.43 pp; mean 64.45% reflects one outlier seed).
  • Diversity boundary: Narrow pools lacking cross-class variance - single-digit {0} MNIST (22.34-26.51% vs. RP 33.93/32.23%, n=3) and 11-class Odor Task-1 (54.55% vs. RP 59.27%, n=5) - confirm P1 needs minimal class diversity (>=2 digits / >=22 odors), not future classes (CIFAR-10 (C10) sweeps running).

2. Zero Offline P1: Current-Task Online Plasticity (Q1b)

Initializing from untrained RP calibrated only on Task 1 (zero offline P1, zero future exposure) and updating PN-KC weights online strictly on current-task batches improves Odor (10x11, n=5, about 100 updates/task) AAC from 59.27±1.08% to 68.93±1.50% (+9.66 pp; BWT -44.60 -> -34.29 pp). High-update visual online streams (executing about 10x more updates/task and requiring lower learning rates) are underway for our follow-up response.

3. Image Oja vs. Grouped Sanger, Mechanism Analysis, and Compute Cost (Q2)

Under identical schedules (full table in General Response Sec. 2):

  • MNIST (400ep, n=5): RP = 23.86±0.33%, Oja = 63.48±0.16%, Grouped Sanger = 63.79±0.16%.
  • CIFAR-100 (500ep, n=5): RP = 23.60±0.41%, Oja = 34.82±0.27%, Grouped Sanger = 34.83±0.49%.
  • Fashion-MNIST (200ep, n=5): RP = 39.73±1.20%, Oja = 58.86±0.28%, Grouped Sanger = 58.88±0.31% (5x2: Oja 59.12%, Sanger 59.07%).
  • CIFAR-10 (250ep, n=5): RP = 33.60±0.41%, Grouped Sanger = 74.56±0.18% (matching-schedule Oja is underway for our next update).

Mechanism & Compute: Circuits are parameterized by KC count N_KC (2,000 Odor, 15,680 MNIST, 15,360 C100), projection-neuron (PN) fan-in K, and top-WTA% Winner-Take-All sparsity:

  • Default 5% WTA: intra-group KC pairs co-activate in only about 9.69% of MNIST batch steps (9.24% Odor), so lateral subtraction rarely fires; sparse Oja + homeostasis already lowers cross-class active overlap (0.549 -> 0.139 on MNIST, vs. Sanger 0.140) and drives the core gain despite divergent weight rows.
  • Higher overlap (higher fan-in K, wider WTA%): engages intra-group competition, yielding +3.2-3.6 pp Sanger gains in Odor (10-20% WTA) and edging Sanger ahead on MNIST (N_KC=2000, K=80, 10% WTA, n=3: 58.67% vs. 57.15%, 2/3; RP 20.96%); C100 probes & C10 Oja are finishing.
  • Compute cost: 0.38-7.08 GiB; 45-52m Odor, 10.5-12.2h MNIST/C100 (details in General Response Sec. 1).

Thank you again for your constructive review. Here are the completed follow-up results for the visual online-plasticity, CIFAR-10 (C10), and high-overlap experiments:

1. Zero-Offline-P1 Online Plasticity (Q1b)

Because visual continual streams execute about 10x more PN-KC updates per task than Odor, we keep dynamic KC threshold homeostasis active during Phase 2 (theta updated toward target 5% firing rate) to preserve KC sparsity under ongoing weight updates. Across all three benchmarks (n=5), online plasticity strictly on current-task batches consistently outperforms the matched Task-1 Random Projection (RP) control:

Benchmark (n=5)Task-1 RP AAC (%)Online Best AAC (%)GainBWT (RP -> Online)
Odor 10x1159.27±1.0868.93±1.50+9.66 pp-44.60 -> -34.29
MNIST 5x224.24±0.3737.06±0.27+12.82 pp-83.91 -> -62.04
CIFAR-100 25x423.94±0.5630.18±0.45+6.24 pp-39.46 -> -33.21

2. Completed CIFAR-10 Oja & Partial-Class Unsupervised Pretraining (Q1a, Q2)

  • Matching-schedule C10 Oja (n=5): reaches 74.59 ± 0.19% (10x1, BWT -27.56) and 74.74 ± 0.18% (5x2, BWT -28.71), matching Grouped Sanger (74.56 ± 0.18% / 74.55 ± 0.36%).
  • C10 5x2 partial-class pretraining: AAC scales monotonically with unlabeled class diversity before CIL (no future-class labels):
C10 5x2 Unlabeled Pool (n)AAC (%)BWT (pp)
Task-1 RP baseline (5)41.98±1.30-70.44
3 classes (Sanger, 5)49.65±1.53-58.98
4 classes (Sanger, 5)56.07±0.62-51.47
5 classes (Sanger, 5)63.11±0.75-42.72
Full 10 classes (Sanger, 5)74.55±0.36-28.91

3. High-Overlap Probing: Grouped Sanger vs. Oja (Q2)

To probe where Grouped Sanger provides an edge over Oja, we increase PN-KC fan-in K and relax WTA% firing rates ((N_KC, K, WTA%, od)):

Regime (N_KC, K, WTA%, od)nRP AAC (%)Oja AAC (%)Sanger AAC (%)Sanger Advantage
Odor (600, 20, 10%, 16.67)525.48±1.2040.98±0.8144.55±2.76+3.57 pp
Odor (250, 20, 20%, 2.5)514.55±0.8618.34±2.4221.73±3.03+3.39 pp (5/5)
Odor (2000, 20, 10%, 2.5)540.34±2.2941.43±0.8942.57±2.59+1.14 pp
MNIST (2000, 80, 10%, 2.5)521.37±1.6557.61±1.8959.02±0.53+1.41 pp (3.6x lower SD)
C100 (2000, 80, 10%, 2.5)512.14±0.2223.17±0.3123.29±0.26+0.12 pp (tighter SD)
C10 (2000, 80, 10%, 2.5)330.51±0.4083.71±1.4284.24±0.26+0.53 pp (5.5x lower SD)

Increasing fan-in connectivity and KC firing rate activates Grouped Sanger's intra-group orthogonalization: it yields consistent accuracy gains on raw Odor and pixel MNIST, and substantially lowers cross-seed variance on ViT-embedded CIFAR features.

We kindly invite your follow-up feedback on these completed results before the discussion window closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the AC and Reviewers pd96, SWJU, 4EXD. We clarify Phase 1 (P1) as unsupervised sparse-expansion pretraining before frozen class-incremental learning (CIL), reporting mean ± sample SD (AAC: final average accuracy; BWT: backward transfer; RP: random projection; SI: Synaptic Intelligence).

1. Vectorized Framework

A vectorized GPU pipeline preserving all biological mechanics enables n=5-10 controls across five benchmarks:

  • P1 speed & cost (A800, excl. ep-1 JIT; 10%-scale -> vectorized):
    • Odor (1500ep): 21.40 -> 1.95 s/ep (0.38 GiB, full 45-52 min)
    • MNIST (400ep): 236.59 -> 9.23 s/ep (7.08 GiB, full 10.5-10.9 h)
    • CIFAR-100 (C100, 500ep): 181.08 -> 7.56 s/ep (6.94 GiB, full 11.3-12.2 h; threshold cooling + Phase 2 readout 9-10 min)
  • Unified suite: All added runs use this pipeline, kept separate from paper Tables 4.1-4.4 for matched comparisons (protocols in appendix).

2. Multi-Seed Benchmarks (Vectorized Pipeline)

Benchmark / Split (n)RP AAC (%)Oja (%)Sanger (%)BWT (RP->Sanger)
Odor .030 10x11 (10)59.10±1.3368.64±4.9570.17±4.62-44.89->-29.55
Odor .030 5x22 (5)64.64±4.1173.02±5.0975.64±2.11-43.35->-25.37
MNIST 10x1 (5)23.86±0.3363.48±0.1663.79±0.16-78.94->-17.64
MNIST 5x2 (5)23.61±0.3659.78±2.8157.95±2.32-83.87->-29.38
Fashion-MNIST 10x1 (5)39.73±1.2058.86±0.2858.88±0.31-48.48->-15.60
Fashion-MNIST 5x2 (5)34.61±0.8959.12±0.4359.07±0.34-65.71->-16.96
CIFAR-100 25x4 (5)23.60±0.4134.82±0.2734.83±0.49-42.23->-37.68
CIFAR-100 10x10 (5)24.14±0.3634.73±0.2234.60±0.28-42.06->-38.60
CIFAR-10 10x1 (5)33.60±0.41-74.56±0.18-73.78->-27.58
CIFAR-10 5x2 (5)41.64±0.44-74.55±0.36-70.75->-28.91
Odor .030 10x11+SI (5)80.98±0.9083.16±2.2583.93±1.78-19.95->-11.82
MNIST 10x1+SI (5)33.95±0.1964.60±0.2564.83±0.28-64.22->-13.77
CIFAR-100 25x4+SI (5)26.02±0.4136.34±0.4036.27±0.33-38.33->-35.27

Across all five datasets and splits, sparse Hebbian plasticity + homeostasis beats RP and stacks with SI; Oja captures the primary gain, while Grouped Sanger adds gains in higher-overlap regimes (see SWJU).

3. Added Experiments & Key Findings

  1. Partial-Data P1 & Online Updates (pd96, 4EXD):
    • Partial-data P1 (zero future classes):
      • MNIST 5x2: 2 unlabeled classes beat matched RP (44.35% vs. 24.24%).
      • C100: rises monotonically from 26.91% (4 classes) to 33.03% (20 classes) vs. RP 23.94%.
      • Odor 10x11: 22/33 classes reach 64.32% / 73.64% vs. RP 58.20% (CIFAR-10 (C10) sweeps running).
    • Zero-P1 online plasticity: Current-task updates lift Odor 10x11 by +9.66 pp (68.93% vs. Task-1 RP 59.27%; visual streams underway).
  2. Oja vs. Sanger Scope (pd96, SWJU):
    • Default 5% Winner-Take-All (WTA) sparsity (9-10% intra-group batch co-activation): Oja + homeostasis matches Sanger.
    • Higher overlap (higher projection-neuron (PN) fan-in K, 10-20% WTA): yields +3.2-3.6 pp Sanger gains in Odor and +1.52 pp on MNIST (Kenyon-cell count N_KC=2000, K=80, 10% WTA, ortho-decay 2.5, n=3; C10/C100 finishing).
  3. Transfer (SWJU):
    • C100 -> C10: Freezing C100 maps beats target RP by +6.55 pp (p<1e-5, shared ViT features).
    • Fashion-MNIST <-> MNIST: Raw pixels do not transfer due to stroke mismatch.
  4. Joint-vs.-Sequential & Pretrainers (4EXD): Matched-budget joint readout (A_joint) vs. sequential CIL (A_seq = AAC) isolates forgetting (D_map = A_joint - A_seq):
    • High-A_joint collapse (MNIST): RP (84.13%) and zero-bias k-sparse autoencoder (k-SAE, 84.67%) collapse sequentially to 23.86% / 10.98% (D_map = 60.27 / 73.70 pp), while Oja/Sanger keep 63.48-63.79%.
    • D_map reduction vs. RP:
      • MNIST: 60.27 -> 1.50-1.51 pp (active overlap 0.549 -> 0.139-0.140).
      • C100: 35.84 -> 12.01-12.02 pp (K-Means 28.11, k-SAE 34.95).
      • Odor: 38.98 -> 20.34-21.20 pp (K-Means 27.68, k-SAE 40.55).
    • Pretrainers (MNIST / C100): Oja/Sanger (63.48-63.79% / 34.82-34.83%) beat PN dictionary (38.55% / 29.99%), K-Means (38.51% / 32.43%), and k-SAE (10.98% / 26.63%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.