- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses the continual learning problem inspired by the brain; it introduces the Efficient Coding Regularizer (ECR), which penalizes response energy while encouraging diverse, high-effective-rank representations.
Critical strengths
1- The idea is interesting. 2. They have lots of qualitative results; especially in the appendix, it may help some readers. 3. Candid about negative results: ECR alone fails in CIL, and gains on CIFAR-100-C are small.
Critical weaknesses
There are some minor errors that the authors need to proofread the paper carefully again, such as
- High overlap and repetition between the abstract and introduction.
- Repeating the acronyms ECR and CL; you need to define them once at first appearance. I mean, Efficient Coding Regularizer (ECR) appears in 107, 154, and....
- There are some other minor typos and wording problems; with careful reading, you can identify them.
- Seed counts vary (3, 5, or 20) across experiments. Please make them consistent.
- Table 6 has 3 OOD ratios against 5 in Table 5.
- EFC task forgetting is exactly 1.5 ± 0.2 on both CIFAR-100 and Tiny-ImageNet in Table 1.
- Table 1 and Table 4 define “current / global accuracy” differently.
- Fig. 15 includes a Nullspace + Efficient Head + OOD configuration, but no table reports it.
I have some major issues with this paper.
- The title and abstract overclaim. Your title has “Brain-inspired”. However, nothing neuroscientific enters the method beyond motivation. I read the paper carefully i was expecting to see something realted to you claims in title and abstract.
For general covariance, minimizing γ·tr(C) − λ·Σlog λᵢ drives every eigenvalue to λ/γ, which is 0.5 with the paper’s values. ECR is therefore a whitening loss with a fixed target scale. There is no noise model and no information term.
“Improves CL”: in Table 1, ECR alone is worse than standard training on global accuracy (4.4 vs 4.7) and forgetting (89.5 vs 85.8). The best CIL result (Nullspace + 0.05 OOD) uses no ECR.
Head-level ECR on CIFAR-100 gives 24.0±1.0 → 21.8±3.1 forgetting over 3 seeds, which is within noise, and accuracy is slightly lower (29.6 → 28.0; 28.0 → 27.3 over 20 seeds). Only the Tiny-ImageNet gap is clear, and that is also 3 seeds. There is no control for a plain logit-norm penalty, so the effect can’t be attributed to efficient coding.
The cortical-column claim in the abstract and conclusion has no supporting analysis and should be removed.
- The baselines are weak. DIL results compare only against standard training, and in Fig. 2 retention is 0.2–0.5 for both methods, so both forget heavily. There is no comparison with plasticity methods (L2-init, shrink-and-perturb, continual backprop, weight decay, leaky or CReLU activations, LayerNorm), nor with decorrelation or effective-rank regularizers. Dying ReLUs at 8–16 units may be fixed by an activation choice. How do you explain this? I know you mainly focused on DIL, but in CIL you could have also added some SOTA baselines such as: WSN: Forget-free Continual Learning with Winning Subnetworks, Haeyong Kang, ICML 2022 SNV: Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?, Vahedifar, ICML 2026, DCNet: On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning Tianqi Wang, arXiv preprint arXiv:2501.15454
3. The method section equations are not well justified or motivated.
For example, line 161: seeing "We center the responses as Rc = R − 1 N 11⊤R and compute their covariance as CR = 1N R⊤R." is sudden and not well defined. It is ambiguous for the first reader why you defined them like this and what gain they provide.
-
The metrics are near-circular, and the headline numbers are best-case. Effective rank and dead fraction are close to what log-det optimizes. “Up to 19.3 pp / 4.5× / 86 pp” are maxima over 12 widths. The “26% fewer epochs” figure depends on an unreported early-stopping criterion (max epochs, patience, and learning rate are not given for the MLP). Collapsed standard runs likely hit the epoch cap. The extra gradient-norm backward pass for η* is not counted in cost.
-
The decodability evidence is contradicted by the appendix. App. I admits Fig. 1’s examples are selected to illustrate the effect. At matched k in Table 7, standard training is better on CIFAR-100 at block 6 for all k (e.g., 19.6 vs 18.9 dB at k = 384), and it also wins at blocks 3 and 6 for k = 384 on Flowers and PACS. Fig. 1 uses block 6, so “across datasets” fails. The “matched variance” test (I.2) gives ECR far more components (e.g., k = 342 vs 72), so it is not a fair budget. In the noise test (I.3), it’s unclear whether noise is scaled globally or per direction. If global, low-variance PCs in the standard model are swamped, this is an artifact of the spectrum's shape rather than robustness. The main text places this analysis in the MLP PMNIST subsection (“we freeze the networks…”), but it runs on a ViT with 20 tasks that is never introduced in the main text. Am I right? Please let me know if I am missing anything.
-
The CIL comparison is not like-for-like.
OOD data (CIFAR-10 for CIFAR-100, ImageNet for Tiny-ImageNet) has semantic overlap with the task classes and provides negative evidence about other classes. Beating EFC with external data qualifies the “exemplar-free” claim. Zero task forgetting is true by construction (Eq. 5). Where ECR is applied in the “Efficient” CIL row is not specified.
Questions for authors
Please see my weakness section.
13 comments
Major issues:
We thank the reviewer for their detailed concern. We would like to clarify that the whitening-like behavior identified by the reviewer is not unrelated to our neuroscience motivation. In fact, it is one of the properties predicted by the efficient-coding theory that inspired ECR.
(1.1) Regarding the neuroscience motivation and mathematical interpretation of ECR, our method is motivated by the efficient-coding principle, which seeks to represent information while limiting the energy required for neural responses. A relevant theoretical formulation is provided by Tavoni (2026), A Unified Theory of Context-Conditioned Efficient and Predictive Coding. This work describes efficient coding as a balance between response entropy and response energy, and shows that the optimal representation has uncorrelated dimensions with equal second moments. Specifically, Equation (44) derives
This is directly relevant to the reviewer's observation. The fact that ECR favors a whitening-like representation is not an unexpected property of our objective. Rather, it is consistent with the optimal representation derived from efficient-coding theory. Our regularizer follows the efficient-coding principle by combining a response-energy penalty with a diversity term. Under a Gaussian approximation, diversity is related to response entropy. The energy term limits response magnitude, while the diversity term prevents the representation from concentrating in only a few dimensions. The reviewer correctly observes that our objective favors equal covariance eigenvalues with a target scale of approximately 0.5. However, this scale also has a theoretical interpretation. The coefficient ratio used in ECR corresponds to an energy coefficient of 1 in the formulation of Tavoni (2026), for which Equation (44) gives a target second moment of 0.5. Our coefficients were also evaluated empirically, so we do not claim that the theory uniquely determines their numerical values. Instead, the theory explains why this type of solution is desirable and how its scale relates to the balance between energy and diversity.
In our CL setting, ECR is optimized alongside the task loss. As the task loss becomes small, the regularizer continues to encourage representations toward this balance. We do not expect every trained network to reach the exact theoretical optimum, since the task, network architecture, and finite-sample estimates also influence the learned responses.
We therefore agree with the reviewer regarding the whitening interpretation, but we believe this observation supports its connection to efficient coding rather than contradicting it. We also recognize that our method does not reproduce the biological circuitry or explicitly estimate mutual information.
(1.2) Regarding the claim that ECR improves CL, we would like to clarify that our work specifically refers to small networks, rather than claiming that ECR universally improves all CL settings. Our abstract also explicitly acknowledges that ECR alone does not prevent forgetting in class-incremental learning, and that its clearest benefits are observed in smaller networks and simpler tasks. Our main focus is domain-incremental learning under limited network capacity, where ECR improves acquisition and retention relative to standard training, reduces task-learning collapse, and maintains more active and diverse representations. On CIFAR-100-C, the improvements are smaller but remain positive. We included class-incremental learning to examine whether these benefits extend to a more challenging setting, and explicitly reported that ECR alone is insufficient. As we clarified in our previous response regarding OOD suppression, the improvement in the best class-incremental configuration comes primarily from addressing cross-task classifier responses through OOD training, rather than from ECR. Therefore, the results in Table 1 illustrate a limitation that we already acknowledge in the paper, rather than contradicting our claim about the benefits of efficient coding in small networks.
(1.3) Regarding the head-level ECR results, we have now extended the comparison in Table 1 to five paired seeds, and the results remain consistent with our original findings. The CIFAR-100 trend is also observed in our 20-seed results in Table 4, so it is not limited to the original three seeds. We acknowledge the small reduction in global accuracy on CIFAR-100, but our purpose in applying ECR at the task-head level was to regulate classifier responses, rather than claim that it alone resolves class-incremental learning. As clarified in our previous response, OOD suppression addresses the additional problem of competition between task heads.
(1.4) Regarding the logit-norm control, we have already examined the individual contributions of ECR's energy and diversity terms in DIL. To address the reviewer's concern specifically in CIL, we are now extending this analysis with a matched head-level logit-norm baseline, using five paired seeds. This will allow us to examine whether the observed effect comes from response-energy regulation alone or from the combination of energy and diversity.
Rebuttal
We sincerely thank the reviewer for recognizing the interest for our qualitative analyses provided in the appendix and our transparent reporting of negative results. We also appreciate the detailed concerns regarding the presentation, experimental comparisons, and theoretical motivation of ECR. Below, we address the concerns for which we currently have completed analyses, following the order of the review. Some additional experimental runs are still in progress, and we plan to provide a follow-up response tomorrow to address the remaining points.
W1 & W2 & W3: We have revised the text, specifically regarding repetitions in the abstract and introduction, typos and wording problems, and the repetitions of abbreviations.
W4: We have made the results in Table 1 to have the same number of seeds of 5. Unfortunately, we didn’t have enough time to make these 20 seeds for the rebuttal, but we will make all of the results with 3 or 5 seeds to have 20 seeds when submitting the camera-ready version.
W5: The complete results will be provided in our next response.
W6: We have checked our results again, and the initial three-seed results were correct. However, with the consideration of the current 5 seeds that we have, for CIFAR-100, it became 1.6 ± 0.2, which is also close to the previous version.
W7: Thanks for bringing this out. We have matched the wording of Table 4 to Table 1.
W8: In the following table, we have included the results of “Nullspace + Efficient Head + 0.05 OOD”, using five paired seeds on 20-task class-incremental CIFAR-100. We can see the results are close for Nullspace + 0.05 OOD and Nullspace + Efficient Head + 0.05 OOD, but a bit better for Nullspace + 0.05 OOD. One possible explanation is that OOD suppression already helps distinguish between task heads by reducing their responses to out-of-task inputs, while ECR additionally constrains the magnitude and diversity of head responses. When used together, these objectives may not be fully complementary, as the additional regularization could limit the separation between correct and competing class predictions. Although further analysis would be needed to confirm this mechanism, our results suggest that the benefits of ECR and OOD suppression are not necessarily additive. This is consistent with our broader finding that efficient coding can be beneficial in certain settings but does not universally improve performance when combined with other continual learning mechanisms. Our motivation for including OOD suppression was to investigate a limitation that became apparent in the class-incremental setting. While Nullspace protects previously learned classifiers from changes, it does not prevent a classifier from responding strongly to inputs belonging to other tasks.