- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines growing self-organizing maps with per-unit Gaussian statistics to support label-free, task-boundary-free representation learning and synthetic replay. A normalized Mahalanobis novelty score, together with quantization error and structural gates, controls map growth. Synthetic samples are generated from unit statistics, either directly in input space or through learned decoders, and mixed with incoming batches. Configurations include a from-scratch VAE and pretrained ResNet-18 or CLIP encoders, with global or local adaptation. Experiments cover split and single-class sequences on MNIST, CIFAR-10/100, TinyImageNet and MiniImageNet. Classification uses a post-hoc majority-label assignment to units. The paper reports multi-run accuracy, growth/replay ablations, generative-quality diagnostics and forgetting analyses.
Critical strengths
A coherent integration of adaptive topology and statistical replay. The model allocates new units to poorly represented regions and uses local distributional summaries to rehearse older regions without retaining an exemplar buffer in the learner. This is a useful direction for unsupervised continual representation learning, and the paper acknowledges the limitations of Gaussian replay and per-unit encoder–decoder capacity.
Ablations demonstrate that replay and the growth rule have meaningful effects. In Table 5, adding replay improves frozen-ResNet final accuracy from 25.11% to 58.00% on CIFAR-10 and from 8.89% to 35.67% on CIFAR-100. Combined error/surprisal growth also improves over either criterion alone in the VAE setting. The primary 20-run results, replay-volume sensitivity and reconstruction-versus-ancestral sample analysis are useful experimental strengths.
Critical weaknesses
Section 3 states that units receive the majority label of mapped training samples after training. The supplied mnist_gsom_paper.py is explicit: lines 814–816 recompute unit labels from all previously seen training examples and labels after each class, and lines 842–844 repeat this over the complete training set for final evaluation; label_units at lines 672–679 performs the assignment. This is not test-label leakage, and the unsupervised representation updates can still be valid. However, the evaluated prediction function receives retrospective labeled access to historical raw data that a continually deployed classifier would not have under the stated no-storage setting. Reassigning labels after map changes can also repair the classifier retrospectively. Consequently, comparison with supervised continual classifiers does not yet establish an equivalent end-to-end continual classification result. The paper must separate representation quality under an offline labeling oracle from classification with a specified feasible calibration/label budget, and disclose whether the same protocol applies to the other datasets.
Unsupervised, task-free learning is meaningful, but comparisons do not adequately isolate the benefit of growing topology and Gaussian replay. Table 2 includes DCM (2024), yet coverage of recent relevant approaches is limited; CURL, separately grouped as unsupervised and task-free, is reported only on MNIST. Many published baseline results differ in supervision, pretraining, calibration access, or resources; cross-study significance tests cannot resolve these differences. Simple controls using the same frozen encoder with online k-means, an incremental Gaussian mixture, or prototype clustering are missing. The fixed-SOM ablation uses task-aware replay whereas GSOM is task-free (Appendix F.1), confounding the effect of growth. Costs also matter: full 512-dimensional covariance requires roughly 1 MiB per unit in float32, before decoder and replay storage; Appendix F.5 reports a 7.5:1 replay-to-real ratio at 956 units. Avoiding an exemplar buffer therefore does not establish resource efficiency. Representative recent comparisons and simple strong controls with matched encoder, calibration, and total resource budgets are needed to demonstrate practical value.
Questions for authors
Evaluation clarification: Is full historical training data used to label the current map after every evaluation checkpoint for all reported configurations, as in the supplied MNIST code? Please specify the label budget, data access and whether labels affect only scoring or a maintained classifier. How do the key results change when the mapping is learned causally with an explicit bounded calibration budget, and when the baselines receive the same calibration access?
Experimental clarification: Can you compare a representative recent method under the same protocol, or explain the incompatibilities? Please prioritize same-encoder online clustering and a fixed-size SOM with identical batch-level replay. Match calibration access and total storage/compute, including covariance, decoders, and replay memory. When does GSOM's added complexity provide a clear practical benefit?
Reproducibility clarification: Please reconcile the statistical and metric definitions. In Section 2.3, the log-determinant is unit-dependent, so dropping it gives a Mahalanobis novelty criterion rather than an equivalent negative log-likelihood under a shared threshold. Appendix B’s initialization correction is not generally an unbiased centered-variance estimator, particularly with nonzero initial variance. Also, Table 19 reports negative forgetting for some tasks; under max-over-observed-accuracies forgetting, this quantity cannot be negative. Please state the exact estimators/metric equations used and provide the corresponding natural-image experiment implementation. These are requests to verify reported results, rather than future-work suggestions.
6 comments
Part 2 - Rebuttal Response
W2/Q2
| Encoder | Model | Labels in training | S-CIFAR-10 |
|---|---|---|---|
| Frozen ResNet-18 | Online k-means (MacQueen, 1967; K = 956) / growing k-means, no replay | no | 68.9 / 70.4 |
| Nearest class mean | yes | 77.2 | |
| Fixed SOM 40×40 (task-free, same replay) | no | 51.7 | |
| GSOM without topology | no | 53.2 | |
| GSOM | no | 58.0 | |
| VAE trained in the stream (200 epochs/task, 3 runs) | Online k-means, no replay | no | 40.0 ± 1.1 |
| Online k-means + Gaussian replay per centroid | no | 45.1 ± 1.1 | |
| GSOM without topology | no | 42.9 ± 1.9 | |
| GSOM | no | 50.1 |
-
When the encoder learns from the stream, GSOM wins: +10.1 over k-means, and +5.0 even when k-means gets the same Gaussian replay (same VAE training). Without its topology GSOM drops to 42.9, below k-means with replay, so the topology is what puts GSOM ahead.
-
The task-free fixed SOM with identical replay loses everywhere (VAE, ResNet, CLIP, also at GSOM's size) by 1.3–6.6 points, so the earlier task-aware confound did not create GSOM's advantage.
-
Gaussian-component control: k-means with per-centroid covariance and Gaussian replay provides a simple hard-assignment local-Gaussian baseline. It scores 45.11% under oracle labeling and 40.06% under causal labeling.
-
Recent methods: DCM and CN-DPM train supervised classifiers and store samples. Running them under our unlabeled, exemplar-free protocol would mean removing both, which changes the methods. CURL is the closest unsupervised task-free method but reports only MNIST (92.59, vs 92.11 for GSOM + VAE).
-
On frozen features, simple prototype methods classify better, even without labels or replay: a frozen encoder leaves little to forget and labeled class means are known to be strong here (Janson et al., 2022; Zhou et al., 2023);
-
Cross-study tests: We agree, Welch tests against reported numbers cannot settle differences in supervision, pretraining or setup. We will present them only as context and base our claims on the same-protocol comparisons above. -Cost. We store each covariance as an upper triangle, so it takes about 0.53 MB per unit at d = 512, rather than the 1 MiB of a full matrix. Full covariance takes 508 / 617 MB on CIFAR-10 / 100, more than a 5k-image buffer (15.4 MB); a diagonal covariance takes 5.9 / 7.1 MB at 46.11 / 26.11 accuracy, and low-rank is future work. The VAE decoder (9.3M parameters in total for the VAE) is shared by all units, not stored per unit; the per-unit encoder–decoder variant is exploratory and we will mark it as such.
-
Training time and the replay ratio. Following A-GEM (Chaudhry et al., 2019, Table 7), we time single-pass training (each example seen once by GSOM and decoder):
| Method | Setting | Training time |
|---|---|---|
| EWC | single pass, CIFAR-100 (20×5) | 250 s |
| A-GEM | single pass, CIFAR-100 (20×5) | 449 s |
| GEM | single pass, CIFAR-100 (20×5) | 5238 s |
| Ours, no replay | single pass, CIFAR-100 (10×10) | 601 s |
| Ours, with replay | single pass, CIFAR-100 (10×10) | 780 s |
In this separate single-pass timing experiment, enabling replay increases runtime from 601 to 780 seconds, an increase of 179 seconds (29.8%). The baseline times are as reported by Chaudhry et al. on their hardware and split, so they are context only. GEM and A-GEM store raw exemplars; we replay from unit statistics. Our reported accuracies use a multi-epoch protocol, so these times show relative cost, not the cost of the reported runs.
Part 3 - Rebuttal Resposne
Q3
- Log-determinant: correct, it differs across units. We will call the criterion a normalized Mahalanobis novelty score, not an NLL. It measures distance relative to each unit's own spread, so wide or narrow Gaussians are not pushed to grow or held back.
- Bias correction: the mean and covariance start at 0 and the variance at a small constant (1e-4; the appendix's "ones" is an error we will correct), with β = 1 − α. It removes start-up bias for the mean and covariance but is not an unbiased variance estimator; we will call it a heuristic and give the exact updates (Appendix B). It adds 0.3–1.1 points.
- Negative forgetting: our metrics follow Lopez-Paz & Ranzato (2017). The "forgetting" column is the negative per-task backward transfer, R_{j,j} − R_{T,j}, which is negative when accuracy on a task rises by the end of the stream.
- Code: We will update the supplementary material to include the natural image code implementation.
We hope these results address your concerns, and we are happy to run any further analysis that would help.
References
- Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018). Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV).
- Janson, P., Zhang, W., Aljundi, R., & Elhoseiny, M. (2022). A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428.
- Lee, S., Ha, J., Zhang, D., & Kim, G. (2020). A neural Dirichlet process mixture model for task-free continual learning. In International Conference on Learning Representations (ICLR).
- MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (Vol. 1, pp. 281–297).
- Rao, D., Visin, F., Rusu, A. A., Teh, Y. W., Pascanu, R., & Hadsell, R. (2019). Continual unsupervised representation learning. In Advances in Neural Information Processing Systems (NeurIPS).
- Ye, F., & Bors, A. G. (2024). Online task-free continual generative and discriminative learning via dynamic cluster memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26202–26212).
- Zhou, D.-W., Cai, Z.-W., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2023). Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338.
Rebuttal Response
Thank you for the careful review, including the code inspection. We confirm that the submitted evaluation used retrospective labeling and now report causal labeling for both frozen ResNet-18 and the VAE learned from the stream. On Split CIFAR-10 with the learned VAE, GSOM reaches 48.01 ± 0.78% under causal labeling versus 40.06 ± 1.61% for k-means with per-centroid Gaussian replay. We also clarify the statistical definitions and provide the implementation details below.
W1/Q1
Yes: in every configuration and dataset, unit labels were recomputed at each evaluation checkpoint (after each task) from all labeled training data seen so far, and over the full training set for the final accuracy. Labels only affect scoring: they never enter training, and no classifier is maintained. But this does give the predictor retrospective access to past labeled data. We now report a causal protocol (frozen ResNet-18). The causal protocol maintains label assignments from incoming labeled examples and propagated replay labels without revisiting past real examples.
Additionally, we also experimented with a bounded-budget causal labeling protocol where only 200 or 10 randomly chosen labels per class are available, and only these are used to label the SOM units.
For frozen ResNet-18, causal accuracy is 55.33 / 31.95% on Split CIFAR-10 / CIFAR-100. On CIFAR-10, using 200 labels per class gives 51.89%, or 93.8% of the full-label causal accuracy; using 10 labels per class gives 39.78%. The purity results below describe post-hoc class separation: | Purity after class | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |---|---|---|---|---|---|---|---|---|---|---| | GSOM + VAE | 1.000 | 0.932 | 0.884 | 0.857 | 0.843 | 0.834 | 0.829 | 0.825 | 0.823 | 0.821 | | GSOM + CLIP | 1.000 | 0.968 | 0.946 | 0.931 | 0.920 | 0.913 | 0.908 | 0.904 | 0.902 | 0.900 |
Purity decreases more slowly later in the sequence and ends at 0.821 for VAE and 0.900 for CLIP.
We also test causal labeling with a VAE learned from the stream, where changing representations make the label-maintenance problem more challenging.
GSOM exceeds k-means with per-centroid Gaussian replay by 7.95 points and topology-free GSOM by 14.00 points under causal labeling. Its oracle-to-causal drop is 2.10 points. Thus, the advantage in this experiment persists without retrospective access to historical labeled examples.