- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies whether learning sparse PN–KC connections benefits class-incremental learning in a fly-inspired SNN. It adapts Sanger’s hebbian rule to KC groups and derives a symmetric anti-Hebbian form. Pretraining uses unlabeled inputs from all future classes then a shared readout learns sequentially. Experiments on synthetic odors, MNIST, and CIFAR-100 show higher final accuracy and less negative backward transfer than random maps.
Critical strengths
- The main experiments improve final accuracy and BWT without labeled replay or additional readout regularization. This provides useful empirical evidence that unsupervised learning is beneficial for subsequent sequential learning.
- The learned expansion retains its advantage when SI is applied to the readout, and combining the two gives further gains on odor and MNIST. This supports its value as a modular front end that may be useful for a downstream continual-learning method.
Critical weaknesses
- The evaluation assumes access to unlabeled training examples from all future classes before the labeled task sequence begins. This deviates from standard continual learning settings and substantially weakens the motivation for the proposed approach. If we already have access to all data, there is no need for continual learning in the first place.
- The baselines are very limited, with almost no comparisons against alternative learning methods.
- Most critically, the proposed method is used entirely before continual learning begins and does not directly address the problem of continual learning across sequential tasks. It is better characterized as a pretraining method, yet comparisons with alternative pretraining approaches are missing.
Questions for authors
See weakness.
3 comments
Continuing our response on alternative unsupervised pretrainers (W2/W3) and separating joint feature quality from sequential forgetting via Experiment C (W1/W2):
4. Alternative Unsupervised Pretraining Baselines (Matched Constraints)
Within each benchmark, all pretrainers share identical network architecture, preprocessing, weight-sparsity constraints, threshold homeostasis, and sequential SGD readout settings (metrics: mean ± sample SD, final Average Accuracy AAC, and Backward Transfer BWT):
- Oja's rule (strictly synaptic local Hebbian ablation): achieves 68.64 ± 4.95% Odor (n=10; Sanger 70.17%), 63.48 ± 0.16% MNIST (n=5; Sanger 63.79%), and 34.82 ± 0.27% CIFAR-100 (C100, n=5; Sanger 34.83%), confirming local competitive plasticity + homeostasis captures the core benefit.
- Projection-neuron (PN) sample dictionary (n=5): non-iterative normalized PN exemplars yield 68.61 ± 3.19% Odor, 38.55 ± 0.26% MNIST, and 29.99 ± 0.41% C100, trailing Hebbian maps on vision.
- Spherical K-Means & zero-bias k-sparse autoencoder (k-SAE, n=5 per benchmark):
- Odor: k-SAE reaches 56.80 ± 1.94% (below Random Projection
RP58.20%), whereas Spherical K-Means achieves 69.48 ± 2.97%, matching Hebbian maps on clustered synthetic odors. - MNIST & C100: Spherical K-Means gets 38.51 ± 0.32% / 32.43 ± 0.12%, and k-SAE drops to 10.98 ± 0.34% / 26.63 ± 0.53%, behind Oja/Sanger (63.48-63.79% / 34.82-34.83%) despite zero-bias leaky integrate-and-fire (LIF) compatibility.
- Odor: k-SAE reaches 56.80 ± 1.94% (below Random Projection
5. Separating Feature Quality from Sequential Forgetting (Experiment C)
For each frozen map, we compare a matched-budget joint linear readout (A_joint, all classes mixed; 100/40/50 epochs on Odor/MNIST/C100 vs. 50/20/50 per task sequentially; not an unconstrained upper bound) against sequential class-incremental learning (CIL) (A_seq = AAC, with DAC denoting average diagonal per-task accuracy immediately after each task): D_map = A_joint - A_seq (paired mean from unrounded per-seed values).
| Benchmark | Map (n=5 subset) | A_joint | DAC | A_seq | BWT (pp) | D_map (pp) |
|---|---|---|---|---|---|---|
| Odor | RP | 97.18±0.33 | 99.43 | 58.20±1.06 | -45.81 | 38.98±1.00 |
| Odor | k-SAE | 97.34±0.66 | 99.57 | 56.80±1.94 | -47.53 | 40.55±2.27 |
| Odor | Sph. K-Means | 97.16±0.39 | 99.73 | 69.48±2.97 | -33.61 | 27.68±2.68 |
| Odor | Oja | 91.05±0.52 | 96.66 | 70.70±4.05 | -28.84 | 20.34±4.45 |
| Odor | Sanger | 91.66±0.46 | 96.66 | 70.45±5.00 | -29.12 | 21.20±5.12 |
| MNIST | RP | 84.13±0.30 | 94.91 | 23.86±0.33 | -78.94 | 60.27±0.18 |
| MNIST | k-SAE | 84.67±0.29 | 98.42 | 10.98±0.34 | -97.16 | 73.70±0.47 |
| MNIST | Sph. K-Means | 70.89±0.25 | 89.13 | 38.51±0.32 | -56.24 | 32.37±0.42 |
| MNIST | Sample Dict | 68.04±0.37 | 87.09 | 38.55±0.26 | -53.92 | 29.49±0.60 |
| MNIST | Oja | 64.98±0.15 | 79.28 | 63.48±0.16 | -17.56 | 1.50±0.16 |
| MNIST | Sanger | 65.30±0.30 | 79.67 | 63.79±0.16 | -17.64 | 1.51±0.18 |
| C100 | RP | 59.44±0.63 | 64.14 | 23.60±0.41 | -42.23 | 35.84±0.81 |
| C100 | k-SAE | 61.58±0.19 | 72.03 | 26.63±0.53 | -47.29 | 34.95±0.44 |
| C100 | Sph. K-Means | 60.54±0.23 | 81.61 | 32.43±0.12 | -51.23 | 28.11±0.17 |
| C100 | Sample Dict | 60.63±0.46 | 80.67 | 29.99±0.41 | -52.79 | 30.63±0.69 |
| C100 | Oja | 46.84±0.31 | 70.84 | 34.82±0.27 | -37.51 | 12.02±0.55 |
| C100 | Sanger | 46.84±0.35 | 71.00 | 34.83±0.49 | -37.68 | 12.01±0.58 |
Takeaway (n=5):
- High
A_joint!= low forgetting: RP and k-SAE reach higherA_joint(84.13%/84.67% MNIST; 59.44%/61.58% C100; 97.18%/97.34% Odor) yet collapse sequentially (23.86%/10.98% MNIST;D_map= 60.27 / 73.70 pp). - Highest
A_seq& sharpD_mapdrop: Oja/Sanger leadA_seq(63.48-63.79% MNIST, well above RP+SI 33.95%; 34.82-34.83% C100) while cuttingD_mapto 1.50-1.51 pp (MNIST, vs. RP 60.27), 12.01-12.02 pp (C100, vs. 35.84), and 20.34-21.20 pp (Odor, vs. 38.98). - Driver: Lower cross-class active overlap (0.549->0.139-0.140 vs. k-SAE 0.618 MNIST; 0.570->0.342 vs. k-SAE 0.565 C100). We will include all baselines, framing, and protocols in the revision.
Thank you again for your detailed review (W1-W3). Here are the completed follow-up results on zero-offline-P1 online plasticity and partial-class pretraining (W1):
1. Zero-Offline-P1 Online Plasticity (W1, n=5)
Because visual continual streams execute about 10x more PN-KC updates per task than Odor, we keep dynamic KC threshold homeostasis active during Phase 2 (theta updated toward target 5% firing rate) to preserve KC sparsity under ongoing weight updates. Across all three benchmarks (n=5), online plasticity strictly on current-task batches consistently outperforms the matched Task-1 Random Projection (RP) control:
| Benchmark (n=5) | Task-1 RP AAC (%) | Online Best AAC (%) | Gain | BWT (RP -> Online) |
|---|---|---|---|---|
Odor 10x11 | 59.27±1.08 | 68.93±1.50 | +9.66 pp | -44.60 -> -34.29 |
MNIST 5x2 | 24.24±0.37 | 37.06±0.27 | +12.82 pp | -83.91 -> -62.04 |
CIFAR-100 25x4 | 23.94±0.56 | 30.18±0.45 | +6.24 pp | -39.46 -> -33.21 |
2. Completed CIFAR-10 Partial-Class Pretraining & Oja Baseline (W1, n=5)
- Matching-schedule C10 Oja (
n=5): reaches74.59 ± 0.19%(10x1,BWT -27.56) and74.74 ± 0.18%(5x2,BWT -28.71), matching Grouped Sanger (74.56 ± 0.18%/74.55 ± 0.36%). - C10
5x2partial-class pretraining (n=5per pool):AACscales monotonically with unlabeled class diversity before CIL (no future-class labels):
C10 5x2 Unlabeled Pool (n) | AAC (%) | BWT (pp) |
|---|---|---|
| Task-1 RP baseline (5) | 41.98±1.30 | -70.44 |
| 3 classes (Sanger, 5) | 49.65±1.53 | -58.98 |
| 4 classes (Sanger, 5) | 56.07±0.62 | -51.47 |
| 5 classes (Sanger, 5) | 63.11±0.75 | -42.72 |
| Full 10 classes (Sanger, 5) | 74.55±0.36 | -28.91 |
We hope these n=5 results address your remaining concerns on future-class exposure and online adaptation, and we kindly invite your follow-up feedback before the discussion closes.
We thank Reviewer 4EXD for noting our empirical soundness, zero-replay gains over random maps, and modularity with Synaptic Intelligence (
SI). Below we reportmean ± sample SDand address the pretraining framing (W2) and future-class exposure (W1) using final Average Accuracy (AAC) and Backward Transfer (BWT); alternative pretrainers (W2/W3) and joint-vs.-sequential decoupling (Exp C) follow in our next comment.1. Unsupervised Pretraining Framing & Why Sequential Forgetting Remains Central
We agree with your characterization of offline Phase 1 (P1) in our frozen-map setting as an unsupervised sparse-expansion pretraining stage, and will adopt this framing explicitly in the Abstract, Introduction, and Discussion.
At the same time, unlabeled pretraining does not trivialize class-incremental learning (CIL): P1 observes no class labels, task identities, or task boundaries. As Experiment C (next comment) shows by comparing matched-budget joint readout (
A_joint) against sequential CIL (A_seq = AAC), maps with high joint accuracy (RPand zero-biask-sparse autoencoder,k-SAE) still collapse sequentially (forgetting gapD_map = A_joint - A_seq = 60.27-73.70 ppon MNIST). Unlabeled feature access alone therefore does not solve sequential forgetting; interference resistance in the sparse code drives CIL retention.2. Eliminating Future-Class Exposure (
Initial-Task-Only&Partial-DataDiagnostics)We test strict initial-task-only restrictions (P1, preprocessing, and Kenyon-cell (KC) firing thresholds restricted strictly to Task 1; matched Random Projection
RPuses identical Task-1 data) alongside partial-data class-pool sweeps (all CIFAR-100 pools match a 3.5k-update budget):Analysis: Multi-class Task-1 pools bring clear gains with zero future-class access:
{0,1} -> 10x1generalizes well once>=2classes appear in P1; C100 (4/8/12/20 classes, n=5) rises monotonically (26.91% -> 33.03%) toward Full 34.83%; Odor 10x11 climbs 58.20% (RP) -> 64.32% (22 classes) -> 73.64±3.87% (33 classes, within variance of Full 70.17±4.62%, n=10).{0}MNIST (22.34-26.51% vs. RP 33.93/32.23%, n=3) and 11-class Odor Task-1 (54.55% vs. RP 59.27%, n=5) - confirm P1 needs initial class diversity (>=2digits />=22odors), not future classes (CIFAR-10 sweeps running).3. Removing Offline P1 Entirely (
Current-Task Online Plasticity)Starting from untrained RP calibrated only on Task 1 (zero offline P1, zero future exposure) and updating PN-KC (projection neuron to KC) weights online strictly on current-task batches during CIL improves Odor (
10x11, n=5, approx. 100 updates/task) AAC from 59.27±1.08% to 68.93±1.50% (+9.66 pp, BWT -44.60 -> -34.29 pp). High-update visual online streams (using lower learning rates tuned for approx. 10x more updates/task) are underway for our follow-up response.