acceptodds
Under review as a conference paper at ICLR 2027

Continual Learning with Growing Self-Organizing Maps and Synthetic Replay

Abstract

This work presents a generative continual learning framework based on growing self-organizing maps (GSOMs) that are augmented with learned distributional statistics as well as encoder–decoder models for class-incremental learning. The proposed approach enables exemplar-free replay using distributional statistical memory, which eliminates the need to store raw data. Each GSOM unit maintains its own mean, variance, and covariance estimates, which are subsequently used to generate synthetic samples for replay; in encoder–decoder configurations, these samples are then decoded back into the input space (via ancestral sampling) for subsequent training. Our method is fully unsupervised, as it does not rely on explicit task boundaries or class labels during training. Results across multiple benchmarks show that the proposed approach achieves performance competitive even with supervised state-of-the-art memory-based methods while consistently outperforming memory-free approaches. In several settings, our framework matches or exceeds existing baselines, particularly in challenging single-class incremental scenarios. We also provide baseline results for single-class incremental TinyImageNet and MiniImageNet, offering a useful reference for future work. This work highlights the effectiveness of an unsupervised, adaptive, topology-driven neural form of statistical replay as a scalable, flexible approach to continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Lifelong Learning, Self-Organizing Maps, Growing Self-Organizing Maps, Class Incremental Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y21rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines growing self-organizing maps with per-unit Gaussian statistics to support label-free, task-boundary-free representation learning and synthetic replay. A normalized Mahalanobis novelty score, together with quantization error and structural gates, controls map growth. Synthetic samples are generated from unit statistics, either directly in input space or through learned decoders, and mixed with incoming batches. Configurations include a from-scratch VAE and pretrained ResNet-18 or CLIP encoders, with global or local adaptation. Experiments cover split and single-class sequences on MNIST, CIFAR-10/100, TinyImageNet and MiniImageNet. Classification uses a post-hoc majority-label assignment to units. The paper reports multi-run accuracy, growth/replay ablations, generative-quality diagnostics and forgetting analyses.

Critical strengths

A coherent integration of adaptive topology and statistical replay. The model allocates new units to poorly represented regions and uses local distributional summaries to rehearse older regions without retaining an exemplar buffer in the learner. This is a useful direction for unsupervised continual representation learning, and the paper acknowledges the limitations of Gaussian replay and per-unit encoder–decoder capacity.

Ablations demonstrate that replay and the growth rule have meaningful effects. In Table 5, adding replay improves frozen-ResNet final accuracy from 25.11% to 58.00% on CIFAR-10 and from 8.89% to 35.67% on CIFAR-100. Combined error/surprisal growth also improves over either criterion alone in the VAE setting. The primary 20-run results, replay-volume sensitivity and reconstruction-versus-ancestral sample analysis are useful experimental strengths.

Critical weaknesses

Section 3 states that units receive the majority label of mapped training samples after training. The supplied mnist_gsom_paper.py is explicit: lines 814–816 recompute unit labels from all previously seen training examples and labels after each class, and lines 842–844 repeat this over the complete training set for final evaluation; label_units at lines 672–679 performs the assignment. This is not test-label leakage, and the unsupervised representation updates can still be valid. However, the evaluated prediction function receives retrospective labeled access to historical raw data that a continually deployed classifier would not have under the stated no-storage setting. Reassigning labels after map changes can also repair the classifier retrospectively. Consequently, comparison with supervised continual classifiers does not yet establish an equivalent end-to-end continual classification result. The paper must separate representation quality under an offline labeling oracle from classification with a specified feasible calibration/label budget, and disclose whether the same protocol applies to the other datasets.

Unsupervised, task-free learning is meaningful, but comparisons do not adequately isolate the benefit of growing topology and Gaussian replay. Table 2 includes DCM (2024), yet coverage of recent relevant approaches is limited; CURL, separately grouped as unsupervised and task-free, is reported only on MNIST. Many published baseline results differ in supervision, pretraining, calibration access, or resources; cross-study significance tests cannot resolve these differences. Simple controls using the same frozen encoder with online k-means, an incremental Gaussian mixture, or prototype clustering are missing. The fixed-SOM ablation uses task-aware replay whereas GSOM is task-free (Appendix F.1), confounding the effect of growth. Costs also matter: full 512-dimensional covariance requires roughly 1 MiB per unit in float32, before decoder and replay storage; Appendix F.5 reports a 7.5:1 replay-to-real ratio at 956 units. Avoiding an exemplar buffer therefore does not establish resource efficiency. Representative recent comparisons and simple strong controls with matched encoder, calibration, and total resource budgets are needed to demonstrate practical value.

Questions for authors

Evaluation clarification: Is full historical training data used to label the current map after every evaluation checkpoint for all reported configurations, as in the supplied MNIST code? Please specify the label budget, data access and whether labels affect only scoring or a maintained classifier. How do the key results change when the mapping is learned causally with an explicit bounded calibration budget, and when the baselines receive the same calibration access?

Experimental clarification: Can you compare a representative recent method under the same protocol, or explain the incompatibilities? Please prioritize same-encoder online clustering and a fixed-size SOM with identical batch-level replay. Match calibration access and total storage/compute, including covariance, decoders, and replay memory. When does GSOM's added complexity provide a clear practical benefit?

Reproducibility clarification: Please reconcile the statistical and metric definitions. In Section 2.3, the log-determinant is unit-dependent, so dropping it gives a Mahalanobis novelty criterion rather than an equivalent negative log-likelihood under a shared threshold. Appendix B’s initialization correction is not generally an unbiased centered-variance estimator, particularly with nonzero initial variance. Also, Table 19 reports negative forgetting for some tasks; under max-over-observed-accuracies forgetting, this quantity cannot be negative. Please state the exact estimators/metric equations used and provide the corresponding natural-image experiment implementation. These are requests to verify reported results, rather than future-work suggestions.

6 comments
AuthorsOpenReview ↗

Rebuttal Response

Thank you for the careful review, including the code inspection. We confirm that the submitted evaluation used retrospective labeling and now report causal labeling for both frozen ResNet-18 and the VAE learned from the stream. On Split CIFAR-10 with the learned VAE, GSOM reaches 48.01 ± 0.78% under causal labeling versus 40.06 ± 1.61% for k-means with per-centroid Gaussian replay. We also clarify the statistical definitions and provide the implementation details below.

W1/Q1

Yes: in every configuration and dataset, unit labels were recomputed at each evaluation checkpoint (after each task) from all labeled training data seen so far, and over the full training set for the final accuracy. Labels only affect scoring: they never enter training, and no classifier is maintained. But this does give the predictor retrospective access to past labeled data. We now report a causal protocol (frozen ResNet-18). The causal protocol maintains label assignments from incoming labeled examples and propagated replay labels without revisiting past real examples.

GSOM labelingS-CIFAR-10S-CIFAR-100
Oracle (submitted)58.0035.67
Causal: incoming-batch labels + replay votes (3 runs)55.33 ± 1.0231.95 ± 0.89
Causal, 200 labels/class (4%)51.89 ± 1.11–
Causal, 10 labels/class (0.2%)39.78 ± 2.07–
  • Additionally, we also experimented with a bounded-budget causal labeling protocol where only 200 or 10 randomly chosen labels per class are available, and only these are used to label the SOM units.

  • For frozen ResNet-18, causal accuracy is 55.33 / 31.95% on Split CIFAR-10 / CIFAR-100. On CIFAR-10, using 200 labels per class gives 51.89%, or 93.8% of the full-label causal accuracy; using 10 labels per class gives 39.78%. The purity results below describe post-hoc class separation: | Purity after class | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |---|---|---|---|---|---|---|---|---|---|---| | GSOM + VAE | 1.000 | 0.932 | 0.884 | 0.857 | 0.843 | 0.834 | 0.829 | 0.825 | 0.823 | 0.821 | | GSOM + CLIP | 1.000 | 0.968 | 0.946 | 0.931 | 0.920 | 0.913 | 0.908 | 0.904 | 0.902 | 0.900 |

  • Purity decreases more slowly later in the sequence and ends at 0.821 for VAE and 0.900 for CLIP.

We also test causal labeling with a VAE learned from the stream, where changing representations make the label-maintenance problem more challenging.

VAE learned from the streamReplayOracle (%)Causal (%)Drop (pp)
Streaming k-meansNo39.98 ± 1.1134.23 ± 0.585.75
Streaming k-means + per-centroid Gaussian replayYes45.11 ± 1.0540.06 ± 1.615.05
GSOM without topology (no grid)Yes42.91 ± 1.9234.01 ± 2.018.90
GSOMYes50.11 ± 0.8948.01 ± 0.782.10

GSOM exceeds k-means with per-centroid Gaussian replay by 7.95 points and topology-free GSOM by 14.00 points under causal labeling. Its oracle-to-causal drop is 2.10 points. Thus, the advantage in this experiment persists without retrospective access to historical labeled examples.

AuthorsOpenReview ↗

Part 2 - Rebuttal Response

W2/Q2

EncoderModelLabels in trainingS-CIFAR-10
Frozen ResNet-18Online k-means (MacQueen, 1967; K = 956) / growing k-means, no replayno68.9 / 70.4
Nearest class meanyes77.2
Fixed SOM 40×40 (task-free, same replay)no51.7
GSOM without topologyno53.2
GSOMno58.0
VAE trained in the stream (200 epochs/task, 3 runs)Online k-means, no replayno40.0 ± 1.1
Online k-means + Gaussian replay per centroidno45.1 ± 1.1
GSOM without topologyno42.9 ± 1.9
GSOMno50.1
  • When the encoder learns from the stream, GSOM wins: +10.1 over k-means, and +5.0 even when k-means gets the same Gaussian replay (same VAE training). Without its topology GSOM drops to 42.9, below k-means with replay, so the topology is what puts GSOM ahead.

  • The task-free fixed SOM with identical replay loses everywhere (VAE, ResNet, CLIP, also at GSOM's size) by 1.3–6.6 points, so the earlier task-aware confound did not create GSOM's advantage.

  • Gaussian-component control: k-means with per-centroid covariance and Gaussian replay provides a simple hard-assignment local-Gaussian baseline. It scores 45.11% under oracle labeling and 40.06% under causal labeling.

  • Recent methods: DCM and CN-DPM train supervised classifiers and store samples. Running them under our unlabeled, exemplar-free protocol would mean removing both, which changes the methods. CURL is the closest unsupervised task-free method but reports only MNIST (92.59, vs 92.11 for GSOM + VAE).

  • On frozen features, simple prototype methods classify better, even without labels or replay: a frozen encoder leaves little to forget and labeled class means are known to be strong here (Janson et al., 2022; Zhou et al., 2023);

  • Cross-study tests: We agree, Welch tests against reported numbers cannot settle differences in supervision, pretraining or setup. We will present them only as context and base our claims on the same-protocol comparisons above. -Cost. We store each covariance as an upper triangle, so it takes about 0.53 MB per unit at d = 512, rather than the 1 MiB of a full matrix. Full covariance takes 508 / 617 MB on CIFAR-10 / 100, more than a 5k-image buffer (15.4 MB); a diagonal covariance takes 5.9 / 7.1 MB at 46.11 / 26.11 accuracy, and low-rank is future work. The VAE decoder (9.3M parameters in total for the VAE) is shared by all units, not stored per unit; the per-unit encoder–decoder variant is exploratory and we will mark it as such.

  • Training time and the replay ratio. Following A-GEM (Chaudhry et al., 2019, Table 7), we time single-pass training (each example seen once by GSOM and decoder):

MethodSettingTraining time
EWCsingle pass, CIFAR-100 (20×5)250 s
A-GEMsingle pass, CIFAR-100 (20×5)449 s
GEMsingle pass, CIFAR-100 (20×5)5238 s
Ours, no replaysingle pass, CIFAR-100 (10×10)601 s
Ours, with replaysingle pass, CIFAR-100 (10×10)780 s

In this separate single-pass timing experiment, enabling replay increases runtime from 601 to 780 seconds, an increase of 179 seconds (29.8%). The baseline times are as reported by Chaudhry et al. on their hardware and split, so they are context only. GEM and A-GEM store raw exemplars; we replay from unit statistics. Our reported accuracies use a multi-epoch protocol, so these times show relative cost, not the cost of the reported runs.

AuthorsOpenReview ↗

Part 3 - Rebuttal Resposne

Q3

  • Log-determinant: correct, it differs across units. We will call the criterion a normalized Mahalanobis novelty score, not an NLL. It measures distance relative to each unit's own spread, so wide or narrow Gaussians are not pushed to grow or held back.
  • Bias correction: the mean and covariance start at 0 and the variance at a small constant (1e-4; the appendix's "ones" is an error we will correct), with β = 1 − α. It removes start-up bias for the mean and covariance but is not an unbiased variance estimator; we will call it a heuristic and give the exact updates (Appendix B). It adds 0.3–1.1 points.
  • Negative forgetting: our metrics follow Lopez-Paz & Ranzato (2017). The "forgetting" column is the negative per-task backward transfer, R_{j,j} − R_{T,j}, which is negative when accuracy on a task rises by the end of the stream.
  • Code: We will update the supplementary material to include the natural image code implementation.

We hope these results address your concerns, and we are happy to run any further analysis that would help.

References

  • Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018). Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV).
  • Janson, P., Zhang, W., Aljundi, R., & Elhoseiny, M. (2022). A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428.
  • Lee, S., Ha, J., Zhang, D., & Kim, G. (2020). A neural Dirichlet process mixture model for task-free continual learning. In International Conference on Learning Representations (ICLR).
  • MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (Vol. 1, pp. 281–297).
  • Rao, D., Visin, F., Rusu, A. A., Teh, Y. W., Pascanu, R., & Hadsell, R. (2019). Continual unsupervised representation learning. In Advances in Neural Information Processing Systems (NeurIPS).
  • Ye, F., & Bors, A. G. (2024). Online task-free continual generative and discriminative learning via dynamic cluster memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26202–26212).
  • Zhou, D.-W., Cai, Z.-W., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2023). Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338.
Reviewer_foamOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an unsupervised and task-free continual learning framework based on Growing Self-Organizing Maps (GSOMs) and synthetic replay. The proposed method extends Self-Organizing Maps (SOMs) by dynamically adding neurons when the existing architecture is insufficient to represent incoming data in a continual learning setting. Each neuron maintains statistical information about the input vectors assigned to it, including their mean, variance, and covariance. These statistics are then used to generate synthetic samples without explicitly storing real samples. The authors propose a new growth criterion that goes beyond traditional quantization-error-based approaches. Specifically, they introduce a surprisal-driven growth mechanism that combines accumulated quantization error with Mahalanobis-distance-based surprisal, along with additional conditions such as minimum activation counts and cooldown periods. The method is evaluated on several benchmark datasets, including MNIST, CIFAR-10, CIFAR-100, TinyImageNet, and MiniImageNet, using class-incremental learning protocols without providing task identities to the model. The results show improvements in classification accuracy for certain datasets and configurations, although the improvements are not consistent across all experimental settings

Critical strengths
  1. The paper is clearly written, well-organized, and easy to follow.
  2. The paper addresses an important and relevant problem of unsupervised and task-free continual learning.
  3. The proposed method is evaluated on multiple benchmark datasets, and ablation studies are provided to investigate the contributions of different components.
Critical weaknesses
  1. Limited novelty: The novelty of the proposed approach appears limited, as SOMs have previously been explored in continual learning settings, and GSOMs have also been investigated in the context of continual learning. The proposed growth criterion appears to be an incremental extension of existing approaches. 2.Modest and inconsistent improvements: Although the proposed method achieves improvements over existing approaches in certain experimental settings, these improvements are generally modest and not consistent across all datasets and configurations. 3.Experimental comparison with fixed-size SOMs: The paper uses the same fixed-size SOM architecture across different datasets, whereas GSOMs are initialised with different capacities depending on the dataset or experimental configuration. This raises concerns about the fairness of the comparison and makes it difficult to assess the benefits of adaptive growth. 4.Insufficient evidence supporting adaptive growth: Dynamic network growth is one of the main contributions of the paper. However, fixed-size SOMs outperform GSOMs in several reported experiments. The experimental results do not convincingly establish the benefits of adaptive growth in terms of classification accuracy, knowledge retention, or memory efficiency.
Questions for authors
  1. Could the authors clarify the differences between the proposed framework and earlier SOM-based and GSOM based continual learning methods, particularly those employing statistical memory and synthetic replay?
  2. Why is the fixed-size SOM configured with 1,600 neurons across all datasets? Have the authors considered tuning the SOM architecture separately for each dataset?
  3. Section 2.1 describes the growth condition using an OR criterion, whereas Equation (5) uses an AND condition. Could the authors clarify which condition is implemented?
  4. Could the authors include an ablation including OR instead of And with different criteria functions mentions?
  5. Could the authors calculate FID or any other similar metrics to see how close are generated synthetic samples to the real sample?
6 comments
AuthorsOpenReview ↗

Rebuttal response

Thank you for the careful review and specific questions. We added task-free fixed-SOM controls, an AND-versus-OR ablation, and prototype baselines. In the learned-VAE Split CIFAR-10 experiment, GSOM also retains its advantage under causal labeling: 48.01% versus 40.06% for k-means with per-centroid Gaussian replay. We distinguish these controlled comparisons from comparisons with published results that use different protocols.

W1/Q1

We agree SOM’s have been studied in CL scenarios and we provide the following table to state the difference between our work and the related works. We will update the related-work section on the paper with the following additional, new works. Summary (accuracy from our paper's no-pretraining table; † = from its ResNet-18-backbone table; – = not reported):

MethodGrowsNo labelsTask-freeNo stored samplesS-MNISTS-CIFAR-10S-CIFAR-100
SOMLP✗partly✓ (claimed)✓–––
DendSOM✗✓✗✓95.2846.79–
c-SOM✗✓✗✓–––
GDM (GWR)✓✗✗✓–––
CURL✓✓✓✗92.59†––
CN-DPM✓✗✓✗93.8146.98–
DCM✓✗✓✗95.12†50.02†22.03†
GSOM + VAE✓✓✓✓92.1150.1113.81

Our main contribution is one set of per-unit statistics both decides where to grow and generates replay, on a growing topology. The closest prior work, c-SOM, also replays from per-unit mean and variance. GSOM differs in four ways: (i) it grows, using a surprisal-and-error rule; (ii) it keeps a full covariance, which adds 11.9 points over a diagonal one; (iii) it replays in a learned latent space and decodes to images, which lets it scale beyond MNIST; and (iv) it is task-free, while c-SOM uses task-dependent learning schedules. The growth rule is not a small change: replacing the standard GSOM error rule with our AND rule adds 5.2 / 2.9 points on CIFAR-10 / 100 (Q4), and removing the topology costs 4.9–8.6 points.

W2

The submitted tables mixed different supervision, task aware/free and pretraining. We will narrow the claim to methods that learn their representation from the stream without labels, task boundaries or stored samples. GSOM outperforms the tested fixed-SOM and topology-free controls in the reported settings. In the learned-VAE Split CIFAR-10 comparison, it exceeds k-means with per-centroid Gaussian replay by 5.00 points under oracle labeling and 7.95 under causal labeling.

Same encoder, same streamMargin
vs fixed SOM (task-free, same replay)+1.3 to +6.6
vs GSOM without topology (same growth and replay)+4.9 to +8.6
vs streaming k-means, same VAE trained in the stream+10.1
and k-means with GSOM's Gaussian replay, same VAE trained in the stream+5.0

GSOM's accuracy is also unchanged when classes arrive one at a time (58.00 → 58.21), while task-aware baselines drop (e.g. LwF 18.53 → 10.05).

W3, W4, Q2

40×40 is larger than every final GSOM map (≤ 1408 units). The different GSOM initial sizes (2×2, 5×5, 7×7) are tiny compared with the final maps (e.g. 376 / 956 / 1167 units on MNIST / CIFAR-10 / CIFAR-100 with VAE); growth, not initialization, sets the capacity. We additionally tested a 31×31 fixed map, approximately matching GSOM’s final CIFAR-10 size; it performs below 40×40 in the reported VAE and CLIP comparisons. This is a size-sensitivity comparison, rather than a comprehensive per-dataset architecture search. On CIFAR-100, GSOM (1167 units) sits between the two sizes and beats both. The submitted fixed SOM used task boundaries, whereas GSOM did not. We ran task-free fixed SOM with the same replay, and it loses everywhere:

Fixed 40×40, task-aware (submitted)Fixed 40×40, task-freeFixed 31×31, task-freeGSOM
C10 / C100, VAE53.01 / 14.6347.36 / 12.5245.26 / 12.2250.11 / 13.81
C10 / C100, CLIP81.24 / 67.1976.31 / 62.7672.20 / 60.2078.84 / 65.01
C10 / C100, ResNet-18–51.72 / 29.84–58.00 / 35.67

Growth also helps beyond accuracy:

  • Memory and replay: GSOM uses 21–40% fewer units than the 40×40 map, so it stores and replays proportionally less.
  • *Unit quality: its units are purer than those of the fixed SOM in the paper (0.82 vs 0.58 with VAE; 0.90 vs 0.78 with CLIP), and purity levels off as classes arrive instead of falling steadily (VAE 0.93 → 0.82, CLIP 0.97 → 0.90 from class 2 to 10).
  • Retention: a classifier trained only on GSOM's replay keeps at 79–84% of the real-data reference accuracy on the oldest classes over five tasks; this supports retained discriminative information but does not rule out cumulative error (details in our response to Reviewer ebZy).
AuthorsOpenReview ↗

Part 2 - Rebuttal Response

Q3/Q4

We use AND, as in Eq. 5; We will fix the "or" in Sec. 2.1.

We ran OR as a new variant (3 runs) and list it with the paper's single-criterion ablation (VAE-GSOM):

Growth ruleS-MNISTS-CIFAR-10S-CIFAR-100Units (C10 / C100)
Error only (standard GSOM)89.42 ± 1.2144.87 ± 1.0310.92 ± 0.88–
Surprisal only91.26 ± 1.0448.63 ± 0.9512.97 ± 0.96–
Error OR surprisal–48.55 ± 2.0712.31 ± 1.101271 / 1519
Error AND surprisal (Eq. 5)92.11 ± 0.9850.11 ± 0.8913.81 ± 1.05956 / 1167

OR grows the map about 30% larger but is no more accurate than surprisal alone, and it is below AND on both datasets. Requiring both signals permits growth when normalized Mahalanobis deviation and accumulated quantization error exceed their thresholds, subject to the remaining gates. This rule has the highest mean accuracy among the tested alternatives.

Q5 We have reported the FID scores on Appendix H.1. On Split Cifar-10:

Replay sampling gives FID 100.0 (ResNet-18) / 98.0 (CLIP), vs 94.0 / 92.0 when reconstructing real images; CN-DPM reports 95.23 for reconstruction. Gaussian-sampled decodings have FID approximately six points higher than real-image reconstructions in these settings. This measures an additional generation-quality gap; it does not establish class coverage or long-term retention.

We hope these results address your concerns, and we are happy to run any further analysis that would help.

References

  • Bashivan, P., Schrimpf, M., Ajemian, R., Rish, I., Riemer, M., & Tu, Y. (2019). Continual learning with self-organizing maps. arXiv preprint arXiv:1904.09330.
  • Lee, S., Ha, J., Zhang, D., & Kim, G. (2020). A neural Dirichlet process mixture model for task-free continual learning. In International Conference on Learning Representations (ICLR).
  • Li, Z., & Hoiem, D. (2018). Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2935–2947.
  • Marsland, S., Shapiro, J., & Nehmzow, U. (2002). A self-organising network that grows when required. Neural Networks, 15(8–9), 1041–1058.
  • Parisi, G. I., Tani, J., Weber, C., & Wermter, S. (2018). Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Frontiers in Neurorobotics, 12, 78.
  • Pinitas, K., Chavlis, S., & Poirazi, P. (2021). Dendritic self-organizing maps for continual learning. arXiv preprint arXiv:2110.13611.
  • Rao, D., Visin, F., Rusu, A. A., Teh, Y. W., Pascanu, R., & Hadsell, R. (2019). Continual unsupervised representation learning. In Advances in Neural Information Processing Systems (NeurIPS).
  • Vaidya, H., Desell, T., & Ororbia, A. (2021). Reducing catastrophic forgetting in self organizing maps with internally-induced generative replay. arXiv preprint arXiv:2112.04728.
  • Ye, F., & Bors, A. G. (2024). Online task-free continual generative and discriminative learning via dynamic cluster memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26202–26212).

We would like to thank you again for your review. We're hoping you can respond to our feedback so we can provide additional information or perform any other additional results that may help within the rebuttal period.

Reviewer_ebZyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the GSOM memory capacity allocation mechanism, using the local distribution statistics stored in each unit to generate replay samples, allowing continual learning without saving the original samples, or using training labels and task boundaries. The authors evaluated this method on benchmarks and got pretty good results. Overall, the method is reasonably designed, and experiments show that growth and replay are indeed useful, but the independent contribution of the topological structure, as well as why statistical memory remains effective over long-term updates, hasn’t been fully verified.

Critical strengths
  1. This method is built around unlabeled, task-free data streams and has a pretty complete design. GSOM units organize inputs through competitive learning, growth rules decide when to increase capacity, and local statistics are used to generate historical samples. The representation learning and map updates during continual learning don’t rely on class labels or task IDs, so its task-free and unsupervised training properties have a practical basis. Even though these features have appeared separately before, implementing them into a growable local statistical memory framework still has some value.
  2. Using the same set of local distribution statistics for both capacity allocation and memory replay makes the method more consistent and therefore more reasonable. Compared to just using prototype distances, the deviation after covariance normalization takes into account the scale and directional differences of different local regions, providing richer info for growth decisions. These statistics can also be used directly for sampling and replay. This shared design is one of the more meaningful parts of the paper, with potential value in linking 'where more representation capacity is needed' with 'how to retain existing knowledge.'
  3. In the experiments reported in the paper, this method achieved pretty good performance.
Critical weaknesses
  1. The novelty of the proposed framework in the text hasn’t been fully reflected in the unique advantages of GSOM’s mechanisms. The key difference from existing statistical replay methods isn’t just saving the mean and covariance, but using units that can grow and maintain topological structure to organize these statistics. However, it’s still unclear how much of the performance gain comes from the topological neighborhood structure versus simply increasing local Gaussian components, introducing full covariance, or using better feature representations. A dynamic prototype/local distribution model without topological constraints might also achieve similar memory organization and replay.
  2. Surprisal measures local statistical deviation, but it doesn’t directly determine whether a new representation unit is needed. The authors hope this criterion allocates new capacity when the current representations are insufficient, but Mahalanobis distance actually judges whether a sample deviates from an existing unit’s Gaussian ellipsoid. They aren’t exactly the same: a unit with large covariance may absorb new patterns that should be represented separately, while tail samples from old distributions may get high scores without necessarily warranting new units.
  3. Growth, capacity, and replay are interdependent, and the actual benefits of the core mechanism haven’t been fully verified. Since the paper generates replay samples per unit, adding new units not only increases model capacity but also increases replay, changing the ratio of new to old data in training. So, the accuracy improvements from the growth criterion may include effects from capacity, memory partitioning, and training weights changes. The fixed SOM baseline uses a replay process with known task boundaries, which differs from the GSOM training protocol, making it impossible to independently verify how much gain is from adaptive growth.
  4. The analysis of generation quality isn’t sufficient to prove that local statistical memory can maintain old knowledge long-term. The authors compare FID between real latent reconstructions and Gaussian-sampled decodings, which helps assess extra loss from sampling, but similar overall generation quality doesn’t mean old classes, rare patterns, and their discriminative structures are preserved. Especially in cycles where synthetic samples further participate in statistical updates, errors from local Gaussian approximations can be repeatedly propagated.
Questions for authors

Please kindly refer to the weaknesses.

6 comments
AuthorsOpenReview ↗

Rebuttal Response

We thank the reviewer for the very thoughtful and technically detailed feedback. Below, we address each weakness that the reviewer has pointed out.

W1

We agree that the contribution is the organization of statistical memory on a growing map. The topology determines where new units are inserted and couples nearby prototype updates; it also provides a two-dimensional representation that can be inspected after training. We therefore compare GSOM with a version that removes the grid and neighborhood updates while retaining statistical replay.

We ran three new experiments to test whether these matter.

(a) We kept the encoder fixed (frozen ResNet-18, Split CIFAR-10) and removed one part of GSOM per run. The run removes the topology entirely: no grid and no neighborhood updates, with each new unit placed at its triggering sample. It keeps the same surprisal rule, full covariance, statistics and replay. This is the model of just growing local Gaussians without topology.

ModelTopologyGrowthFull cov.UnitsAcc. (%)
GSOM without topology✗✓✓120353.15 ± 0.75
Fixed SOM 40×40 (task-free, same replay)✓✗✓160051.72 ± 1.56
GSOM, diagonal covariance✓✓✗95646.11 ± 2.01
GSOM✓✓✓95658.00 ± 1.31

The topology-free variant produces 1,203 units versus 956 for GSOM, yet its mean accuracy is 4.85 percentage points lower. Thus, having more units is not sufficient to explain the advantage in this comparison. Replacing full covariance with diagonal covariance lowers mean accuracy by 11.89 points.

(b) The topology matters even more on a harder dataset and without pretraining (new runs: 3 each; Welch p < 0.02):

ResNet-18, S-CIFAR-10ResNet-18, S-CIFAR-100VAE (no pretraining), S-CIFAR-10
GSOM without topology (no grid)53.15 ± 0.7527.12 ± 0.7842.91 ± 1.92
GSOM58.00 ± 1.3135.67 ± 1.9650.11 ± 0.89

(c) We repeated the replay-only test from W4 on both models: a linear classifier is trained only on each model's replay and tested on real images of the oldest classes (task 1) after all 5 tasks. The ratio compares it with an identical classifier trained on real data (3 runs).

After task 5 (frozen ResNet-18, Split CIFAR-10)Replay-only acc. on task-1 classesRatio to real data
GSOM without topology (no grid)59.76 ± 2.050.68
GSOM69.6 ± 1.70.80

The replay-trained classifier scores 9.84 percentage points higher with GSOM than with topology-free GSOM on the oldest classes after task 5. This provides evidence that GSOM’s replay retains more useful discriminative information in this probe, in addition to its advantage in the main classification evaluation.

The frozen-ResNet comparisons control the input representation. We also test a VAE learned from the stream, where the representation changes during training. Streaming k-means generates replay from its own per-centroid statistics using the same Gaussian-sampling procedure as GSOM. The table below reports both retrospective oracle labeling and causal labeling, with the reported VAE schedule of 200 epochs per task.

VAE learned from the streamReplayOracle (%)Causal (%)Drop (pp)
Streaming k-meansNo39.98 ± 1.1134.23 ± 0.585.75
Streaming k-means + per-centroid Gaussian replayYes45.11 ± 1.0540.06 ± 1.615.05
GSOM without topology (no grid)Yes42.91 ± 1.9234.01 ± 2.018.90
GSOMYes50.11 ± 0.8948.01 ± 0.782.10

Under causal labeling, GSOM reaches 48.01 ± 0.78%, compared with 40.06 ± 1.61% for k-means with per-centroid Gaussian replay and 34.01 ± 2.01% for topology-free GSOM: advantages of 7.95 and 14.00 percentage points. The oracle-to-causal drop is 2.10 points for GSOM, versus 5.05 and 8.90 points for these controls. The advantage therefore persists without retrospective labeling in this experiment. These results support the contribution of the combined topological organization and growth mechanism; they do not separately identify neighborhood adaptation, growth placement, or label propagation as the cause.

However, with a frozen pretrained encoder, the picture changes. The encoder is frozen, so its features do not drift; however, the clustering and label assignments can still change during learning, and simple prototypes do better on Split CIFAR-10: online k-means, which uses no labels, reaches 68.9% vs 58.0% for GSOM (on Split CIFAR-100 the two are level, 35.4 vs 35.7). Class means computed with labels are known to be strong on frozen pretrained features (Janson et al., 2022).

AuthorsOpenReview ↗

Part 2 - Rebuttal Response

W2

We agree. This is why surprisal never triggers growth alone: Eq. 5 also requires high accumulated error, a minimum hit count, a cooldown and a decaying growth probability, and the surprisal threshold is the 92nd percentile of recent scores.

  • Tail samples: a high novelty score alone does not trigger growth. Accumulated error and the structural gates must also be satisfied. These conditions reduce sensitivity to isolated deviations, but do not guarantee that an old-distribution tail sample cannot trigger growth.
  • A broad unit absorbing a new pattern: you are right that our rule does not catch this directly. A unit that is wide in every direction gives low surprisal, so the AND rule blocks growth there. (Mahalanobis distance still flags a new pattern lying along a direction where the unit has little variance.)

To see how often this case hurts, we tested an OR variant that permits either high accumulated error or high novelty, subject to the other gates. It produces 1,271 / 1,519 units on CIFAR-10 / CIFAR-100, compared with 956 / 1,167 for AND, but obtains lower mean accuracy: 48.55 / 12.31 versus 50.11 / 13.81%. This supports AND among the tested rules; it does not measure how frequently broad units suppress useful growth. A broad covariance can reduce the Mahalanobis score and block growth despite high error, which we will state as a limitation. The error-only and novelty-only results are 44.87 / 10.92 and 48.63 / 12.97%, respectively.

W3

We agree. The submitted fixed SOM also used task boundaries, so the comparison was not fair. We reran the fixed SOM task-free, with exactly GSOM's replay, at two sizes: 40×40, larger than any GSOM map, and 31×31, about GSOM's size on CIFAR-10. This removes the task-boundary mismatch and compares two fixed capacities with adaptive growth under the same replay procedure. Total replay volume can still differ because it depends on the number of active units.

Task-free, same replayFixed 40×40Fixed 31×31 (≈ GSOM size)GSOM
CIFAR-10 / 100, VAE47.36 / 12.5245.26 / 12.2250.11 / 13.81
CIFAR-10 / 100, CLIP76.31 / 62.7672.20 / 60.2078.84 / 65.01
CIFAR-10 / 100, ResNet-1851.72 / 29.84–58.00 / 35.67

GSOM wins in every setting (+1.3 to +6.6) while using 21–40% fewer units than the 40×40 map.

  • The 31×31 map, at about GSOM's size, does worse than 40×40, so GSOM's gain does not come from more units. W1 gives a second check: without topology, GSOM grows 26% more units (1203 vs 956) and still scores 4.85 points lower.
  • Replay volume and the new-to-old ratio: growth increases replay when K is fixed. In the reported ablations, increasing the map from 921 to 1,621 units raises accuracy by 0.77 points, whereas increasing K from 1 to 2 lowers it by 5.89 points. These results show that additional replay is not consistently beneficial. They do not fully separate topology, capacity, and replay weighting; that requires a fixed total replay budget.
  • Growth also divides memory more cleanly. GSOM's units are purer than those of the fixed 40×40 SOM in the paper (0.82 vs 0.58 with VAE; 0.90 vs 0.78 with CLIP), although GSOM uses fewer active units(956 vs 1225 with VAE; 1034 vs 1325 with CLIP).

Together, the fixed-map and topology-free controls support a benefit from how memory is organized and expanded.

AuthorsOpenReview ↗

Part 3 - Rebuttal Repsonse

W4 In GSOM, replayed samples go back into training and update the unit statistics, so the loop you describe is real. FID cannot show whether old classes stay separable, so we designed a direct test. After each task, we train a linear classifier only on GSOM's replay and test it on real images of the oldest classes (task 1). The classifier never sees a real old image, so its accuracy measures what GSOM's memory keeps. As a reference, we train an identical classifier on real data (frozen ResNet-18, Split CIFAR-10, 3 runs):

After task12345
Trained on replay only82.1 ± 1.178.6 ± 2.475.8 ± 0.573.1 ± 1.069.6 ± 1.7
Trained on real data98.094.893.792.587.3
Ratio (replay ÷ real)0.840.830.810.790.80
  • A classifier trained only on replay still separates the oldest classes, at 79–84% of what real data gives. -The replay-only accuracy decreases from 82.1 to 69.6%, while the real-data reference decreases from 98.0 to 87.3%. The ratio ranges from 0.79 to 0.84 and ends at 0.80. This shows that replay retains useful information for recognizing the oldest classes across the tested five-task sequence. The reference provides context for the changing evaluation difficulty, but the ratio does not isolate or eliminate that effect.
  • The topology matters here too as without it, the ratio after task 5 falls to 0.68 (W1c).

The replay-only classifier remains below the real-data reference. This gap reflects the replay pipeline and cannot be attributed solely to the Gaussian approximation. The experiment covers five tasks with a frozen encoder and does not directly test rare-mode coverage or rule out cumulative error over longer streams.

We hope these results address your concerns, and we are happy to run any further analysis that would help.

References

  • Janson, P., Zhang, W., Aljundi, R., & Elhoseiny, M. (2022). A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Rebuttal Response

Thank you for the review and specific concerns. We now evaluate causal labeling with both frozen ResNet-18 and a VAE learned from the stream. In the learned-VAE Split CIFAR-10 experiment, GSOM reaches 48.01 ± 0.78%, compared with 40.06 ± 1.61% for k-means with per-centroid Gaussian replay. We also report the memory costs and acknowledge that full covariance is not memory-efficient at 512 dimensions

W1/Q1

The original evaluation used historical labeled training examples to assign unit labels retrospectively. Labels did not enter the representation or map updates, but this evaluation access is unavailable to a deployed classifier without stored historical data. We therefore evaluate causal label assignments from incoming labeled examples and replay-label propagation, without revisiting historical real examples. The first table reports frozen ResNet-18; the second reports the VAE learned from the stream.

GSOM labelingS-CIFAR-10S-CIFAR-100
Original (all past labels)58.0035.67
Causal (3 runs)55.33 ± 1.0231.95 ± 0.89
Causal, 200 labels/class (4%)51.89 ± 1.11–
Causal, 10 labels/class (0.2%)39.78 ± 2.07–
  • Additionally, we also experimented with a bounded-budget causal labeling protocol where only 200 or 10 randomly chosen labels per class are available, and only these are used to label the SOM units.

  • For frozen ResNet-18, causal accuracy is 55.33 / 31.95% on Split CIFAR-10 / CIFAR-100. On CIFAR-10, using 200 labels per class gives 51.89%, or 93.8% of the full-label causal accuracy; using 10 labels per class gives 39.78%. The purity results below describe post-hoc class separation: | Purity after class | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |---|---|---|---|---|---|---|---|---|---|---| | GSOM + VAE | 1.000 | 0.932 | 0.884 | 0.857 | 0.843 | 0.834 | 0.829 | 0.825 | 0.823 | 0.821 | | GSOM + CLIP | 1.000 | 0.968 | 0.946 | 0.931 | 0.920 | 0.913 | 0.908 | 0.904 | 0.902 | 0.900 |

  • Purity decreases more slowly later in the sequence and ends at 0.821 for VAE and 0.900 for CLIP.

We also test causal labeling with a VAE learned from the stream, where changing representations make the label-maintenance problem more challenging.

VAE learned from the streamReplayOracle (%)Causal (%)Drop (pp)
Streaming k-meansNo39.98 ± 1.1134.23 ± 0.585.75
Streaming k-means + per-centroid Gaussian replayYes45.11 ± 1.0540.06 ± 1.615.05
GSOM without topology (no grid)Yes42.91 ± 1.9234.01 ± 2.018.90
GSOMYes50.11 ± 0.8948.01 ± 0.782.10

GSOM exceeds k-means with per-centroid Gaussian replay by 7.95 points and topology-free GSOM by 14.00 points under causal labeling. Its oracle-to-causal drop is 2.10 points. Thus, the advantage in this experiment persists without retrospective access to historical labeled examples.

AuthorsOpenReview ↗

Part 2 - Rebuttal Response

W2/Q2

ResNet-18 (d = 512)UnitsFull cov.Diagonal
CIFAR-10 / 100956 / 1161508 / 617 MB5.9 / 7.1 MB
TinyImageNet1312697 MB8.1 MB
Acc. C10 / C10058.00 / 35.6746.11 / 26.11
  • Memory scaling: each unit stores d(d+1)/2 + 3d floats, so memory grows linearly with the number of units and quadratically with d. At d = 128 (VAE), 956 units take 33 MB. In the reported configurations, TinyImageNet uses approximately 13% more units than CIFAR-100 despite having twice as many training images.

  • Equal-budget comparison: full covariance at d = 512 (508 MB) is more than storing the whole CIFAR-10 training set (154 MB). The diagonal variant (5.9 MB) is about the size of a 2k-image buffer. We agree that an equal-memory comparison would clarify the accuracy–storage trade-off, and we will run these experiments and report the results during the rebuttal period. Our objective is continual learning without retaining historical real examples; we do not claim lower memory usage than exemplar-based methods. Where retaining such examples is restricted, even a smaller exemplar buffer does not satisfy this constraint. GSOM instead retains distributional summaries for generating replay, accepting the associated storage cost.

  • Compute scaling: per sample, finding the closest unit costs O(units × d), and the covariance update grows with d².

  • Training time and the replay ratio. Following A-GEM (Chaudhry et al., 2019, Table 7), we time single-pass training (each example seen once by GSOM and decoder):

MethodSettingTraining time
EWCsingle pass, CIFAR-100 (20×5)250 s
A-GEMsingle pass, CIFAR-100 (20×5)449 s
GEMsingle pass, CIFAR-100 (20×5)5238 s
Ours, no replaysingle pass, CIFAR-100 (10×10)601 s
Ours, with replaysingle pass, CIFAR-100 (10×10)780 s

In this separate single-pass timing experiment, enabling replay increases runtime from 601 to 780 seconds, an increase of 179 seconds (29.8%). The baseline times are as reported by Chaudhry et al. on their hardware and split, so they are context only. GEM and A-GEM store raw exemplars; we replay from unit statistics. Our reported accuracies use a multi-epoch protocol, so these times show relative cost, not the cost of the reported runs.

AuthorsOpenReview ↗

Part 3 - Rebuttal Response

Q3

We ran prototype baselines on the same frozen features, stream and unit labeling as GSOM (Split CIFAR-10, 3 runs each). Streaming and growing k-means use no labels in training and have about as many units as GSOM. Nearest class mean (NCM, the classifier of SimpleCIL) needs labels to compute its class means.

Frozen encoderMethodLabels in trainingS-CIFAR-10
ResNet-18Streaming / growing k-meansno68.9 / 70.4
NCMyes77.2
Fixed SOM (task-free, same replay)no51.7
GSOMno58.0
CLIP ViT-B/32Streaming / growing k-meansno82.5 / 86.5
NCMyes89.5
GSOMno75.2
VAE trained in the stream (no pretraining)Streaming k-means: no replay / with GSOM's replayno40.0 / 45.1
GSOMno50.1
  • On frozen features, simple prototypes beat GSOM on CIFAR-10 with both encoders, and NCM, which uses labels, is best. A frozen encoder leaves little to forget, which matches Janson et al. (2022) and Zhou et al. (2023). On Split CIFAR-100 with ResNet-18, GSOM and streaming k-means are level (35.7 vs 35.4; NCM 53.5). We will report all of these and narrow our claim.
  • When the VAE learns from the stream on Split CIFAR-10, GSOM reaches 48.01% under causal labeling, versus 34.23% for k-means without replay, 40.06% for k-means with per-centroid Gaussian replay, and 34.01% for topology-free GSOM. The corresponding advantages are 13.78, 7.95, and 14.00 points. The advantage therefore survives causal labeling in this experiment; we do not claim superiority across all encoders or datasets.
  • GSOM beats the task-free fixed SOM with identical replay on every encoder (+1.3 to +6.6).
  • Recent image-encoder-only methods: SLCA (Zhang et al., 2023) and ADAM-Adapter (Zhou et al., 2023) reach 67.6 and 65.5 final accuracy on Split CIFAR-100 (10 tasks) with CLIP, as reported by Huang et al. (2024). Both use labels and train task by task, so we list them as context only.

We hope these results address your concerns, and we are happy to run any further analysis that would help.

References

  • Chaudhry, A., Ranzato, M., Rohrbach, M., & Elhoseiny, M. (2019). Efficient lifelong learning with A-GEM. In International Conference on Learning Representations (ICLR).
  • Huang, L., Cao, X., Lu, H., & Liu, X. (2024). Class-incremental learning with CLIP: Adaptive representation adjustment and parameter fusion. In European Conference on Computer Vision (ECCV).
  • Janson, P., Zhang, W., Aljundi, R., & Elhoseiny, M. (2022). A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428.
  • Zhang, G., Wang, L., Kang, G., Chen, L., & Wei, Y. (2023). SLCA: Slow learner with classifier alignment for continual learning on a pre-trained model. In IEEE/CVF International Conference on Computer Vision (ICCV).
  • Zhou, D.-W., Cai, Z.-W., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2023). Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338.

We would like to thank you again for your review. We're hoping you can respond to our feedback so we can provide additional information or perform any other additional results that may help within the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.