- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces IPA, a hebbian adaptation mechanism. The motivation is clear: Hebbian plasticity allows NNs to adapt without backprop, but designing hebbian rules requires black-box optimizations of the parameters. Then, the question is if we can analytically construct such Hebbian rules from a small labeled support set and be competitive with few-shot and continual learning methods. The paper is difficult to follow as it condenses a lot of information, metrics and values into plain text. The benchmarks evaluated are broad but the results are relatively small compared to the baselines.
Critical strengths
-
Motivation is clear and the method is interpretable. The central idea of analytically constructing Hebbian plasticity rules rather than relying on black-box optimization is interesting and has the potential to impact the literature. The method follows naturally from its motivation.
-
Theoretical contribution is meaningful and limitation is acknowledged. For instance, Proposition 3.1 is useful and well explained.
-
There is considerable experimental breadth. The paper evaluates the method across few-shot classification, meta-learning, and few-shot class-incremental learning, using different architectures, baselines, and ablations. While, I still have concerns about some of the comparisons and their interpretation, this breadth is a positive aspect, suggesting that the method is not exclusively desgined to just fit one experiment.
-
The method has practical advantages at deployment. IPA operates on individual unlabeled samples without requiring gradient updates, replay buffers, or access to an entire query batch. The paper also provides measurements of computational and memory requirements.
Critical weaknesses
Major
-
The experiments do not fully isolate the contribution of Hebbian plasticity. The method proposed, IPA-Approx (Sec. 4.3), combines the prototype-based initialization of the classifier, the hebbian updates and the confidence-weight of the class prototypes, while NCM-r uses hard nearest-centroid assignments but such comparison changes both the classifier adaptation and the prototype refinement strategy. Then, we see in Table 1 that NCM-r underperforms NCM on CIFAR-100. This seems like it inflates the benefit of IPA-Approx. I'd recommend introducing an ablation performing the same soft-prototype as IPA-Approx without the hebbian updates.
-
The meta-learning comparison does not control for training conditions. Figure 3 results on cross-domain and cross-body are interesting. It shows an improvements over ANILr in 23/32 comparisons and on the transferability of IPA-trained backbones for 11/16 comparisons. However, these findings are not sufficient evidence to interpret it as if the Hebbian head encourages the transferable representations as stated in Lines 353-355. In particular, Section 4.2 mentions that Meta-IPA uses additional unlabeled episode data during meta-training, whereas MAML and ANILr adapt using only labeled support sets. If that is the case, could the authors clarify why the improvements are solely attributed to the hebbian adaptation mechanism? An ablation where Meta-IPA does not use additional unlabeled data would help establish a stronger claim about the representation-transfer advantage.
Minor
-
The paper is hard to parse. There is a lot of information, results, math notation and citations embedded in the text. See 4.1 as an example. I understand the necessity of explaining and citing it all. However, lots of parts, as they stand right now, are extremely difficult to parse and understand the main claim.
-
The experimental presentation is dense. Consider breaking down Fig. 3 into smaller figures or a different representation to improve readability.
-
The practical impact and relevance of results is not sufficiently discussed.
Questions for authors
-
What improvement shown in the results is uniquely attributed to the Hebbian update? I understand it is important to completely isolate the contribution in order to evaluate such claims.
-
Could the meta-learning results be reproduced under comparable training budgets? (See Major Weakness 2) Also, how much of the cross-domain improvement of Fig. 3 is due to the learned representations?
-
Meta-learning learning rate (Section 4.2, p. 6; Table 6, p. 22). Could the authors clarify which value was used for the results in Figure 3 and Table 7? Section 4.2 reports a meta-learning rate of (10^{-2}) for Meta-IPA, whereas Table 6 specifies (10^{-3}). This paragraph of Sec 4.2 is also a good example for the Minor Weakness 1.
-
Omniglot results (Table 7, p. 24). I'm curious to understand what might have caused Meta-IPA to achieve 98.98% accuracy in the Omniglot 20-way 1-shot setting but just 95.71% with 5 shots. Could the authors explain why that could have happened?
3 comments
Thank you to the authors for your rebuttal. My concerns have only been minimally addressed. I still have concerns about the viability of contribution to Hebbian plasticity as the isolation has not been clarified. I do not see the revision pdf and cannot make judgements on the improvement of readability and presentation either.
I will be keeping my Weak Rejection (2) rating. I hope the authors take the feedback time to improve the manuscript and to provide more concrete evidence.
Thank you
Thank you for the follow-up. We have uploaded a preliminary revision of the PDF. Changes are marked in red (we did not highlight every change). It restructures §4.1 into a setup and a results paragraph, splits Figure 3 into a cross-domain figure in the main text and a cross-body figure in the appendix, and states the practical relevance in the deployment-cost paragraph of §4.3 (one sample at a time, a single pass, 250 KiB of state, no replay buffer and no gradients).
On the effect size/outperformance issue. We are mostly presenting a novel method for test-time adaptation. The Hebbian head is what lets IPA adapt one sample at a time with no gradients, replay or query batch. It never falls below plain NCM, and it adds clearly where the refinement fails (self-supervised features, imbalanced or contaminated streams).
On the isolation of the Hebbian update. The control soft-NCM-r differs from IPA-Approx in one component only. It uses the same prototype construction, the same confidence-weighted refinement, temperature, running counts, sample order and budget, and removes the plastic head, so that the assignments come from the current prototypes instead of the Hebbian weights (Algorithm 2). The difference between the two is therefore due to the Hebbian update alone. It is positive on ResNet-12 (+0.21 on CIFAR-100, +0.28 on MiniImageNet, p < 1e-3) and in all nine combinations of three self-supervised backbones and three datasets (+0.53 to +6.04, p < 1e-4). Within the head, setting β = 0 removes the BC term and keeps the rest of the update, which removes most of the gain on DINO features (69.53 against 67.54 on CIFAR-100, last column of Table 2, and 74.18 against 69.81 on CUB200, Table 21). Table 2 in §4.3 now reports this decomposition step by step, from NCM over the prototype construction and the refinement to the head and the BC term, and Table 20 repeats it for the further backbones.
Where your comments are addressed in the preliminary revision (line numbers of the PDF).
- W1 / Q1, soft-NCM-r as the matched control (lines 418–420, Algorithm 2 at line 1280), contribution of the head (lines 456–461, Table 2 at lines 432–441), the BC term and β (lines 463–473, Table 21 at line 1757), stream conditions and further backbones (Section C.19 from line 1693, Tables 18–20).
- W2, support-only meta-training and the claim limited to what was measured (lines 356–360, Appendix C.5 at lines 1166–1170).
- Q2, training budgets (lines 304–305 and 1162–1164).
- Q3, lr_meta = 1e-2 for Meta-IPA (line 303, line 1161 and Table 7 at line 1244).
- Q4, Omniglot 1- versus 5-shot (lines 1150–1153 and from line 1398).
- M1, §4.1 split into a setup paragraph (line 257) and a results paragraph (line 268).
- M2, cross-domain Figure 3 in the main text (lines 340–342) and cross-body Figure 12 in the appendix (line 1395).
- M3, practical relevance (lines 475–480).
Review response LPjD
Thank you for the detailed review and the presentation suggestions. We will try to improve the text and considered your feedback helpful in improving the manuscript overall.
W1 / Q1. The soft-NCM-r ablation is presented in the common response. Here we show, that uniquely attributable to the Hebbian update are: head isolation +0.21 (CIFAR-100/ResNet-12), +2.37 (CIFAR-100/DINO), +0.91 (MiniImageNet/DINO) and +0.28 (MiniImageNet/ResNet-12); Robustness at test-time adaptation under skewed streams IPA-Approx (DINO) 100-fold class imbalance 67.45, against 64.37 for the control and 66.04 for NCM.
W2. Meta-IPA adapts on unlabeled episode images during meta-training; MAML and ANILr adapt on the labeled support only. We ran the ablation you suggest on MiniImageNet 5-way 5-shot. We meta-trained Meta-IPA twice with identical seeds (N=10 per condition), tasks and budget (1,000 meta-steps of 32 tasks), once with the inner loop on unlabeled episode images and once on the support embeddings only, and evaluated both on 600 held-out episodes per run (mean ± std over 5 seeds).
We would like to point out that the lower number of meta-updates reduce the effect of possible data imbalance; and the gradient graph is not retained over time (i.e., unlabeled data never enter a loss, and no gradient flows through their features). Unlabeled data only drives head dynamics defined by the Hebbian rules conditioned on the support, so the two effects cannot be separated. Support-only meta-training is as accurate, so the extra unlabeled data is not what drives Meta-IPA's results. At evaluation Meta-IPA adapts on the support only, like MAML and ANILr.
Q2. Meta-IPA runs under a lower meta-budget (1000 meta-steps of 32 tasks) with 100 unlabeled Hebbian inner steps; MAML and ANILr run 10000 meta-steps with 5 gradient inner steps, their standard protocol. Meta-IPA thus receives a tenth of the body updates. (The "100 meta-steps" of the submitted §4.2 and Table 6 was an error for 1,000.) On how much of the cross-domain gain is due to the learned representation, the cross-body swap shows that the bodies carry part of it.
Q3. We follow the best performing sweep setup, our apologies for the confusion: Meta-IPA meta_lr 1e-2 in Figure 3 and Table 7. For the meta benchmarks we follow the original meta_lr 1e-3 for MAML and ANILr. The 1e-3 for Meta-IPA in Table 6 is a typo.
Q4. Omniglot is the only dataset with 20 images per character. At 1-shot, four unlabeled images per character remain after the support and the fifteen queries, and the inner loop adapts on those. At 5-shot none remain, so it adapts on the support embeddings without labels. The Hebbian loop then sees only the images that already define the prototypes. We hypothesise that this weakens the meta-learning signal for the body. The inversion does not occur on CUB200 or MiniImageNet, where a separate unlabeled set exists at both shot counts.
M1–M3. We will restructure §4.1 into a setup paragraph and a results paragraph with the full grid in the appendix, redesign Figure 3 into a cross-domain figure in the main text and a cross-body figure in the appendix, and add with a paragraph on practical relevance, adaptation one sample at a time in a single pass, 250 KiB of state, no replay buffer and no gradients.