acceptodds
Under review as a conference paper at ICLR 2027

Brain-Inspired Efficient Coding Improves Continual Learning in Small Networks

Abstract

Humans and animals continually learn new tasks and stimuli, but artificial neural networks often struggle to do so without disrupting prior knowledge. A successful continual learning (CL) approach must preserve what has already been learned while using its representational capacity efficiently enough to support future learning. Inspired by efficient coding in information theory and sensory neuroscience, which proposes that neural systems balance response diversity against metabolic cost, we introduce the Efficient Coding Regularizer (ECR), which penalizes response energy while encouraging diverse, high-effective-rank representations. We then study ECR across CL problems of increasing complexity. We first consider domain-incremental learning, where labels remain fixed as the input distribution changes. On permuted MNIST, ECR improves both acquisition of new tasks and retention of previous ones, reduces the rate of task-learning collapse by up to 19.3 percentage points, requires approximately 26% fewer epochs per task, and maintains up to the representation diversity (effective rank) of standard training. On CIFAR-100, ECR continues to preserve representational diversity in the corrupted setting but yields smaller and less consistent accuracy gains, and in class-incremental learning, it alone does not prevent forgetting. We therefore develop a nullspace-based approach that protects previous classifiers from destructive updates, providing a foundation on which ECR can act. In this context, applying ECR to the task heads constrains their response scale and reduces global forgetting while keeping accuracy comparable. Together, these results show that efficient-coding principles can preserve representational capacity and regulate network responses, with the clearest benefits in smaller networks and simpler tasks. In larger networks and more complex tasks, efficient coding works best when paired with explicit constraints. Beyond machine learning, these findings inspire neuroscience by suggesting that efficient coding operates most effectively within smaller neural circuits, such as cortical columns, a possibility that remains to be tested.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
efficient coding, continual learning, class-incremental learning, domain-incremental learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pLzmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the continual learning problem inspired by the brain; it introduces the Efficient Coding Regularizer (ECR), which penalizes response energy while encouraging diverse, high-effective-rank representations.

Critical strengths

1- The idea is interesting. 2. They have lots of qualitative results; especially in the appendix, it may help some readers. 3. Candid about negative results: ECR alone fails in CIL, and gains on CIFAR-100-C are small.

Critical weaknesses

There are some minor errors that the authors need to proofread the paper carefully again, such as

  1. High overlap and repetition between the abstract and introduction.
  2. Repeating the acronyms ECR and CL; you need to define them once at first appearance. I mean, Efficient Coding Regularizer (ECR) appears in 107, 154, and....
  3. There are some other minor typos and wording problems; with careful reading, you can identify them.
  4. Seed counts vary (3, 5, or 20) across experiments. Please make them consistent.
  5. Table 6 has 3 OOD ratios against 5 in Table 5.
  6. EFC task forgetting is exactly 1.5 ± 0.2 on both CIFAR-100 and Tiny-ImageNet in Table 1.
  7. Table 1 and Table 4 define “current / global accuracy” differently.
  8. Fig. 15 includes a Nullspace + Efficient Head + OOD configuration, but no table reports it.

I have some major issues with this paper.

  1. The title and abstract overclaim. Your title has “Brain-inspired”. However, nothing neuroscientific enters the method beyond motivation. I read the paper carefully i was expecting to see something realted to you claims in title and abstract.

For general covariance, minimizing γ·tr(C) − λ·Σlog λᵢ drives every eigenvalue to λ/γ, which is 0.5 with the paper’s values. ECR is therefore a whitening loss with a fixed target scale. There is no noise model and no information term.

“Improves CL”: in Table 1, ECR alone is worse than standard training on global accuracy (4.4 vs 4.7) and forgetting (89.5 vs 85.8). The best CIL result (Nullspace + 0.05 OOD) uses no ECR.

Head-level ECR on CIFAR-100 gives 24.0±1.0 → 21.8±3.1 forgetting over 3 seeds, which is within noise, and accuracy is slightly lower (29.6 → 28.0; 28.0 → 27.3 over 20 seeds). Only the Tiny-ImageNet gap is clear, and that is also 3 seeds. There is no control for a plain logit-norm penalty, so the effect can’t be attributed to efficient coding.

The cortical-column claim in the abstract and conclusion has no supporting analysis and should be removed.

  1. The baselines are weak. DIL results compare only against standard training, and in Fig. 2 retention is 0.2–0.5 for both methods, so both forget heavily. There is no comparison with plasticity methods (L2-init, shrink-and-perturb, continual backprop, weight decay, leaky or CReLU activations, LayerNorm), nor with decorrelation or effective-rank regularizers. Dying ReLUs at 8–16 units may be fixed by an activation choice. How do you explain this? I know you mainly focused on DIL, but in CIL you could have also added some SOTA baselines such as: WSN: Forget-free Continual Learning with Winning Subnetworks, Haeyong Kang, ICML 2022 SNV: Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?, Vahedifar, ICML 2026, DCNet: On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning Tianqi Wang, arXiv preprint arXiv:2501.15454

3. The method section equations are not well justified or motivated.

For example, line 161: seeing "We center the responses as Rc = R − 1 N 11⊤R and compute their covariance as CR = 1N R⊤R." is sudden and not well defined. It is ambiguous for the first reader why you defined them like this and what gain they provide.

  1. The metrics are near-circular, and the headline numbers are best-case. Effective rank and dead fraction are close to what log-det optimizes. “Up to 19.3 pp / 4.5× / 86 pp” are maxima over 12 widths. The “26% fewer epochs” figure depends on an unreported early-stopping criterion (max epochs, patience, and learning rate are not given for the MLP). Collapsed standard runs likely hit the epoch cap. The extra gradient-norm backward pass for η* is not counted in cost.

  2. The decodability evidence is contradicted by the appendix. App. I admits Fig. 1’s examples are selected to illustrate the effect. At matched k in Table 7, standard training is better on CIFAR-100 at block 6 for all k (e.g., 19.6 vs 18.9 dB at k = 384), and it also wins at blocks 3 and 6 for k = 384 on Flowers and PACS. Fig. 1 uses block 6, so “across datasets” fails. The “matched variance” test (I.2) gives ECR far more components (e.g., k = 342 vs 72), so it is not a fair budget. In the noise test (I.3), it’s unclear whether noise is scaled globally or per direction. If global, low-variance PCs in the standard model are swamped, this is an artifact of the spectrum's shape rather than robustness. The main text places this analysis in the MLP PMNIST subsection (“we freeze the networks…”), but it runs on a ViT with 20 tasks that is never introduced in the main text. Am I right? Please let me know if I am missing anything.

  3. The CIL comparison is not like-for-like.

OOD data (CIFAR-10 for CIFAR-100, ImageNet for Tiny-ImageNet) has semantic overlap with the task classes and provides negative evidence about other classes. Beating EFC with external data qualifies the “exemplar-free” claim. Zero task forgetting is true by construction (Eq. 5). Where ECR is applied in the “Efficient” CIL row is not specified.

Questions for authors

Please see my weakness section.

13 comments

We sincerely thank the reviewer for recognizing the interest for our qualitative analyses provided in the appendix and our transparent reporting of negative results. We also appreciate the detailed concerns regarding the presentation, experimental comparisons, and theoretical motivation of ECR. Below, we address the concerns for which we currently have completed analyses, following the order of the review. Some additional experimental runs are still in progress, and we plan to provide a follow-up response tomorrow to address the remaining points.

W1 & W2 & W3: We have revised the text, specifically regarding repetitions in the abstract and introduction, typos and wording problems, and the repetitions of abbreviations.

W4: We have made the results in Table 1 to have the same number of seeds of 5. Unfortunately, we didn’t have enough time to make these 20 seeds for the rebuttal, but we will make all of the results with 3 or 5 seeds to have 20 seeds when submitting the camera-ready version.

W5: The complete results will be provided in our next response.

W6: We have checked our results again, and the initial three-seed results were correct. However, with the consideration of the current 5 seeds that we have, for CIFAR-100, it became 1.6 ± 0.2, which is also close to the previous version.

W7: Thanks for bringing this out. We have matched the wording of Table 4 to Table 1.

W8: In the following table, we have included the results of “Nullspace + Efficient Head + 0.05 OOD”, using five paired seeds on 20-task class-incremental CIFAR-100. We can see the results are close for Nullspace + 0.05 OOD and Nullspace + Efficient Head + 0.05 OOD, but a bit better for Nullspace + 0.05 OOD. One possible explanation is that OOD suppression already helps distinguish between task heads by reducing their responses to out-of-task inputs, while ECR additionally constrains the magnitude and diversity of head responses. When used together, these objectives may not be fully complementary, as the additional regularization could limit the separation between correct and competing class predictions. Although further analysis would be needed to confirm this mechanism, our results suggest that the benefits of ECR and OOD suppression are not necessarily additive. This is consistent with our broader finding that efficient coding can be beneficial in certain settings but does not universally improve performance when combined with other continual learning mechanisms. Our motivation for including OOD suppression was to investigate a limitation that became apparent in the class-incremental setting. While Nullspace protects previously learned classifiers from changes, it does not prevent a classifier from responding strongly to inputs belonging to other tasks.

MethodGlobal accuracy ↑ (%)Forgetting ↓ (pp)Initial old-task accuracy ↑ (%)Final old-task accuracy ↑ (%)
Nullspace29.35 ± 2.3724.00 ± 0.8452.88 ± 2.3928.88 ± 2.39
Nullspace + Efficient Head27.82 ± 2.8321.49 ± 2.2148.92 ± 1.9127.43 ± 3.02
Nullspace + 0.05 OOD39.13 ± 1.7214.93 ± 0.7653.88 ± 2.4538.95 ± 2.02
Nullspace + Efficient Head + 0.05 OOD37.24 ± 1.3015.43 ± 1.2952.57 ± 2.4637.14 ± 1.71

Major issues:

We thank the reviewer for their detailed concern. We would like to clarify that the whitening-like behavior identified by the reviewer is not unrelated to our neuroscience motivation. In fact, it is one of the properties predicted by the efficient-coding theory that inspired ECR.

(1.1) Regarding the neuroscience motivation and mathematical interpretation of ECR, our method is motivated by the efficient-coding principle, which seeks to represent information while limiting the energy required for neural responses. A relevant theoretical formulation is provided by Tavoni (2026), A Unified Theory of Context-Conditioned Efficient and Predictive Coding. This work describes efficient coding as a balance between response entropy and response energy, and shows that the optimal representation has uncorrelated dimensions with equal second moments. Specifically, Equation (44) derives

Ce=E[ee⊤]=12γtheoryI.C_e=\mathbb{E}[ee^\top]=\frac{1}{2\gamma_{\mathrm{theory}}}I.

This is directly relevant to the reviewer's observation. The fact that ECR favors a whitening-like representation is not an unexpected property of our objective. Rather, it is consistent with the optimal representation derived from efficient-coding theory. Our regularizer follows the efficient-coding principle by combining a response-energy penalty with a diversity term. Under a Gaussian approximation, diversity is related to response entropy. The energy term limits response magnitude, while the diversity term prevents the representation from concentrating in only a few dimensions. The reviewer correctly observes that our objective favors equal covariance eigenvalues with a target scale of approximately 0.5. However, this scale also has a theoretical interpretation. The coefficient ratio used in ECR corresponds to an energy coefficient of 1 in the formulation of Tavoni (2026), for which Equation (44) gives a target second moment of 0.5. Our coefficients were also evaluated empirically, so we do not claim that the theory uniquely determines their numerical values. Instead, the theory explains why this type of solution is desirable and how its scale relates to the balance between energy and diversity.

In our CL setting, ECR is optimized alongside the task loss. As the task loss becomes small, the regularizer continues to encourage representations toward this balance. We do not expect every trained network to reach the exact theoretical optimum, since the task, network architecture, and finite-sample estimates also influence the learned responses.

We therefore agree with the reviewer regarding the whitening interpretation, but we believe this observation supports its connection to efficient coding rather than contradicting it. We also recognize that our method does not reproduce the biological circuitry or explicitly estimate mutual information.

(1.2) Regarding the claim that ECR improves CL, we would like to clarify that our work specifically refers to small networks, rather than claiming that ECR universally improves all CL settings. Our abstract also explicitly acknowledges that ECR alone does not prevent forgetting in class-incremental learning, and that its clearest benefits are observed in smaller networks and simpler tasks. Our main focus is domain-incremental learning under limited network capacity, where ECR improves acquisition and retention relative to standard training, reduces task-learning collapse, and maintains more active and diverse representations. On CIFAR-100-C, the improvements are smaller but remain positive. We included class-incremental learning to examine whether these benefits extend to a more challenging setting, and explicitly reported that ECR alone is insufficient. As we clarified in our previous response regarding OOD suppression, the improvement in the best class-incremental configuration comes primarily from addressing cross-task classifier responses through OOD training, rather than from ECR. Therefore, the results in Table 1 illustrate a limitation that we already acknowledge in the paper, rather than contradicting our claim about the benefits of efficient coding in small networks.

(1.3) Regarding the head-level ECR results, we have now extended the comparison in Table 1 to five paired seeds, and the results remain consistent with our original findings. The CIFAR-100 trend is also observed in our 20-seed results in Table 4, so it is not limited to the original three seeds. We acknowledge the small reduction in global accuracy on CIFAR-100, but our purpose in applying ECR at the task-head level was to regulate classifier responses, rather than claim that it alone resolves class-incremental learning. As clarified in our previous response, OOD suppression addresses the additional problem of competition between task heads.

(1.4) Regarding the logit-norm control, we have already examined the individual contributions of ECR's energy and diversity terms in DIL. To address the reviewer's concern specifically in CIL, we are now extending this analysis with a matched head-level logit-norm baseline, using five paired seeds. This will allow us to examine whether the observed effect comes from response-energy regulation alone or from the combination of energy and diversity.

Reviewer_3rYEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an Efficient Coding Regularizer (ECR) that penalizes response energy while encouraging diverse, high-effective-rank representations, with its overall weight adapted by gradient-norm balancing. On domain-incremental PMNIST, ECR improves acquisition and retention, reduces task-learning collapse by up to 19.3 percentage points, requires roughly 26% fewer epochs per task, and reaches up to 4.5× the effective rank of standard training; on CIFAR-100-C (40 corruption tasks, ResNet-18) the accuracy gains are smaller and less consistent. For exemplar-free class-incremental learning the paper adds a classifier-nullspace feature update, applies ECR to the task heads to constrain their response scale, and separately suppresses head responses with unlabeled external images. Validation centres on the PMNIST width sweep (20 seeds), the CIFAR-100-C domain sequence, and the CIFAR-100 / Tiny-ImageNet class-IL comparisons, with additional decoding, relearning and transformer experiments.

Critical strengths
  1. The PMNIST diagnosis is thorough and interpretable. The width sweep with 20 seeds jointly characterizes acquisition, retention, collapse rate, inactive units and effective rank. This provides useful empirical evidence about how these quantities co-evolve under capacity constraints, while the reported negative results clarify the limits of effective rank as a performance proxy.
  2. Negative results and the internal control are handled well. The authors report that higher effective rank does not guarantee higher accuracy, and that ECR alone is insufficient in class-IL; they also report the direct Nullspace → Nullspace + head-level ECR comparison, which properly supports the narrower claim that head-level ECR reduces forgetting.
Critical weaknesses
  1. The regime carrying the headline numbers is compared only against standard training. The key PMNIST and CIFAR-100-C results lack a comparison with continual backpropagation (a published method for the same failure mode), and there is no λ = 0 (energy-only) ablation. Consequently neither the value of ECR relative to existing methods nor the necessity of the log-determinant term is established. This most affects my assessment.
  2. Practical computational cost and the adaptive-weight rule are under-specified. The ~26% epoch reduction is supported by Fig. 6, but ECR adds a covariance log-determinant and an additional backward pass, and neither wall-clock time nor total computation is reported; the η update is described only as "toward η*", without step size or update frequency.
Questions for authors
  1. In the existing narrow-network PMNIST setting, with comparable tuning and evaluation, how does ECR compare with continual backpropagation and with an energy-only (λ = 0) control?
  2. What is the exact update rule moving η toward η*, including the update frequency and any step-size or smoothing parameters?
  3. What are the paired runtimes or total computation of ECR and standard training under the same hardware and stopping criterion, including the gradient-norm estimation overhead?
4 comments

Dear Reviewer 3rYE,

We sincerely thank the reviewer for recognizing our thorough and interpretable analysis of continual learning in small networks, careful reporting of negative results, and the internal controls supporting the proposed method. We also appreciate the reviewer's constructive concerns regarding comparisons with existing methods, the contributions of the energy and diversity terms, and the practical computational cost of ECR. Below, we provide additional results and discussion to address your concerns.

To address these concerns, we provide new comparisons with Continual Backpropagation (CBP) on both PMNIST and CIFAR-100-C, together with Energy-only and Diversity-only ablations to clarify the complementary roles of the two regularization terms. We also provide the exact adaptive-weight update rule, paired wall-clock measurements, and additional relearning-time results. These analyses further clarify how ECR balances acquisition, retention, and representational activity, while also addressing the implementation and efficiency questions raised in the review.

In our analysis, the PMNIST comparisons use FC1 = 256, varying FC2 widths, 50 tasks, and five seeds. The CIFAR-100-C comparisons use the same 40-task ResNet setup as in the paper, with five seeds for Standard, CBP, and ECR.

W1 & Q1: Comparison with Continual Backpropagation and ECR Component Ablations

ECR vs. Continual Backpropagation (CBP). For PMNIST, we tuned CBP using a six-configuration grid search over replacement rates {10−510^{-5}, 10−410^{-4}, 10−310^{-3}} and maturity thresholds {100, 1000}, across FC2 widths of 8, 16, and 32 with five seeds per configuration (90 runs in total). The best configuration, selected using a predefined criterion combining validation acquisition and retention, was a replacement rate of 10−510^{-5} and a maturity threshold of 1000. The reported results use the same five seeds evaluated during hyperparameter selection. For CIFAR-100-C, we use CBP's published reference configuration without additional tuning. The results show that ECR achieves higher retention than CBP across all three tested PMNIST widths, with comparable acquisition. We note that CBP primarily focuses on maintaining plasticity rather than directly addressing forgetting, whereas our approach considers both acquisition and retention.

We have also provided Energy-only (λ=0\lambda = 0) and Diversity-only (γ=0\gamma = 0) ablations to better clarify the contributions of both terms, particularly in controlling dead neurons and activation magnitude (measured by total hidden RMS). The results in the following table show that the energy and diversity terms have complementary roles. The energy term discourages large activations and helps keep response magnitudes controlled, while the diversity term encourages the network to maintain a broader set of representation directions. This distinction is visible in the results. Diversity-only can achieve high acquisition and low dead-neuron fractions, but its activation RMS is higher than full ECR. Energy-only training reduces activation magnitude but does not explicitly encourage representation diversity. These results support the complementary roles of both terms, with the combined objective providing a balance between acquisition, retention, representational diversity, and controlled activation magnitude. In the following table, best values for acquisition, retention, and dead-neuron fraction at each width are bolded.

FC2MethodAcquisition ↑ (%)Retention ↑ (%)FC2 Dead neurons ↓ (%)Total hidden RMS
8Standard77.4 ± 27.023.8 ± 6.845.0 ± 30.618.3 ± 4.1
8CBP93.5 ± 2.025.9 ± 3.016.8 ± 11.014.3 ± 1.6
8Energy-only (λ=0\lambda = 0)85.9 ± 13.025.8 ± 5.24.0 ± 8.92.04 ± 0.83
8Diversity-only (γ=0\gamma = 0)96.0 ± 0.134.0 ± 2.50.0 ± 0.0174 ± 5.4
8ECR95.6 ± 1.029.7 ± 1.10.5 ± 1.11.76 ± 0.29
16Standard89.6 ± 7.325.3 ± 2.759.8 ± 14.316.5 ± 3.0
16CBP96.6 ± 0.130.6 ± 2.225.0 ± 2.810.7 ± 0.74
16Energy-only (λ=0\lambda = 0)95.7 ± 0.933.0 ± 2.21.4 ± 0.91.75 ± 0.40
16Diversity-only (γ=0\gamma = 0)96.2 ± 0.027.3 ± 1.80.0 ± 0.0124 ± 4.6
16ECR96.4 ± 0.132.6 ± 3.15.0 ± 8.22.77 ± 0.77
32Standard89.1 ± 14.525.3 ± 3.374.5 ± 14.915.2 ± 3.3
32CBP96.8 ± 0.128.8 ± 1.450.4 ± 7.99.49 ± 0.87
32Energy-only (λ=0\lambda = 0)96.6 ± 0.141.4 ± 2.70.5 ± 0.60.970 ± 0.067
32Diversity-only (γ=0\gamma = 0)96.2 ± 0.018.5 ± 0.90.0 ± 0.087.9 ± 1.7
32ECR96.7 ± 0.144.4 ± 1.90.0 ± 0.01.74 ± 0.23

We additionally evaluated CBP on CIFAR-100-C using the same 40-task setup and five matched seeds. ECR achieves higher acquisition, retention, and final accuracy compared with CBP. In particular, ECR improves final accuracy by 1.71 percentage points (95% CI: [0.88, 2.55]), with higher accuracy across all five seeds. While CBP completely avoids dead neurons in the measured hidden layer, ECR also maintains a very low fraction compared with standard training.

MethodAcquisition ↑ (%)Retention ↑ (%)Final accuracy ↑ (%)Dead neurons ↓ (%)
Standard63.55 ± 0.1655.83 ± 0.3055.51 ± 0.571.91 ± 0.95
CBP63.58 ± 0.1455.97 ± 0.1954.79 ± 0.230.00 ± 0.00
ECR64.36 ± 0.1857.09 ± 0.2156.51 ± 0.750.53 ± 0.60

W2 & Q2: Adaptive-Weight Update Rule and Implementation Details

We compute the target multiplier as

η∗=clip⁡[0.01,100](ρ∥∇θsLtask∥2max⁡(∥∇θsLECR∥2,10−12))\eta^{\ast} = \operatorname{clip}_{[0.01,100]}\left(\rho \frac{\lVert \nabla_{\theta_s}\mathcal{L}_{\mathrm{task}} \rVert_2}{\max\left(\lVert \nabla_{\theta_s}\mathcal{L}_{\mathrm{ECR}} \rVert_2, 10^{-12}\right)}\right)

where ρ=0.1\rho = 0.1, and θs\theta_s comprises the shared FC1 and FC2 weights and biases. The multiplier is initialized to 1. At the first update, it is set to η∗\eta^{\ast}; subsequent updates use log-space smoothing,

log⁡η←0.9log⁡η+0.1log⁡η∗.\log\eta \leftarrow 0.9\log\eta + 0.1\log\eta^{\ast}.

followed by clipping to [0.01,100][0.01,100]. Updates occur every 100 optimizer steps, beginning at step 100, and the multiplier persists across task boundaries. At each update, we separately compute the gradient magnitudes of the task loss and ECR loss with respect to the shared network parameters, using the existing forward computation. This introduces additional gradient calculations without requiring additional forward passes. We will clarify these implementation details in the revised manuscript.

W2 & Q3: Computational Efficiency, Wall-Clock Time, and Relearning

We recovered paired wall-clock measurements from the training process's per-task checkpoint timestamps. For each seed, standard training and ECR were executed consecutively on the same GPU, with identical stopping criteria. The reported epochs and wall-clock times are averaged across the five seeds. On the NVIDIA L40S, for FC1=256 and varying FC2 widths, ECR is 13–17% faster. We have also provided the number of epochs needed to train for these specific architectures as another way of comparison, showing ECR is able to reduce the number of epochs needed for training 19–23%. These results are summarized in the following table:

FC2Standard epochsECR epochsStandard time (h)ECR time (h)
16184614991.951.70
32162112421.791.48
64150511861.701.46
128141510871.721.44

These timings include ECR loss computation, gradient-norm estimation, evaluation, and checkpointing, and cover tasks 2–50.

For an additional timing analysis, we have also extended the timing report in Fig. 24 of the supplementary in the following table. The following report includes the number of optimization steps and wall-clock time required for each network to recover its initial best accuracy on a previously learned task. Across all four datasets, ECR reaches this accuracy in fewer steps and less time than standard training. Notably, on PACS, standard training did not reach the recovery target within the measured window, even though the observed training time already exceeded that required by ECR. Therefore, the reported PACS values for standard training represent lower bounds, and the actual difference may be larger. The following experiments were done on an NVIDIA GeForce RTX 5090.

DatasetStandard iterationsECR iterationsStandard time (s)ECR time (s)
PMNIST36.1315.401.3350.595
CIFAR-10017.409.400.8770.461
Flowers-10212.447.670.2390.148
PACS≥274.40*72.87≥5.842*2.227

Conclusion

We thank the reviewer again for their constructive feedback. In response to the main concerns, we have provided direct comparisons with CBP on PMNIST and CIFAR-100-C, Energy-only and Diversity-only ablations, the exact adaptive-weight update rule, and paired wall-clock and relearning-time measurements. These additional results strengthen the empirical evidence for ECR, clarify the complementary roles of its regularization terms, and demonstrate its ability to balance acquisition, retention, and representational activity while maintaining practical computational efficiency. We believe these findings address the reviewer's concerns and strengthen the paper's contributions. We respectfully ask the reviewer to reconsider their assessment in light of this additional evidence.

Thank you for the extra experiments and detailed responses. The new comparisons and ablations largely address my main concerns and support ECR's benefits in the small-network domain-incremental settings. I am raising my score to 3.

Dear reviewer 3rYE,

We sincerely thank you again for your constructive review and for taking the time to examine our additional results and reconsider your assessment. Your questions about computational efficiency, implementation details, and comparisons with existing methods were particularly helpful. They encouraged us to provide stronger experimental evidence and make our methodology clearer. We truly appreciate your engagement throughout the rebuttal and will continue to improve the manuscript based on your feedback.

Reviewer_ueLsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the Efficient Coding Regularizer (ECR), which combines an activation-energy penalty with a log-determinant covariance reward, using adaptive gradient balancing to control its contribution to supervised training. The strongest results occur in small networks on permuted MNIST, where ECR improves acquisition and retention while reducing representation collapse. On CIFAR-100-C, increased representation diversity produces less consistent accuracy gains.

Critical strengths
  1. Convincing evidence of improved learning under constrained capacity, supported by controlled, multi-seed experiments.

  2. A clear separation between preserving previous classifiers and preventing cross-task prediction errors.

Critical weaknesses
  1. The experiments do not sufficiently isolate the distinctive contribution of ECR, especially beyond PMNIST. The domain-incremental comparison is primarily against standard training. The hyperparameter sweeps retain both regularization terms and therefore do not establish whether the proposed energy–diversity combination is necessary, whether one component explains most of the improvement, or whether adaptive gradient balancing is itself important.

  2. The interpretation of the representation diagnostics is stronger than the measurements currently justify. Effective rank is computed from the eigenvalues of C_R+\epsilon I, rather than the unregularized covariance. From this definition, shrinking C_R toward zero makes the normalized spectrum approach uniformity and the measured effective rank approach D, even without creating additional informative directions. This is a potential confound because ECR explicitly penalizes response energy.

Questions for authors

The following questions must be addressed for this paper:

  1. Can you report energy-only, log-determinant-only, and full-ECR ablations, together with fixed versus adaptive regularization weighting? These comparisons would be most informative on the 50-task, narrow-network PMNIST setting. Does the combined objective outperform a matched variance/covariance regularizer applied to the same supervised task?

  2. Can the accuracy tables include all four combinations of head-level ECR on/off and OOD suppression on/off on the same Nullspace backbone, with paired-seed differences? Figure 15 already includes diagnostics for the combined configuration, but the corresponding global-accuracy results are absent from Tables 1 and 4. Please also define the forgetting metric precisely and clarify whether its reduction reflects better preservation of previously achieved performance or lower earlier accuracy.

  3. Can effective rank be recomputed from the unregularized covariance spectrum, alongside total variance and sensitivity to \epsilon? How much of the reported rank advantage remains after eliminating dependence on the absolute activation scale?

5 comments

We sincerely thank the reviewer for recognizing our evidence of improved learning under constrained capacity and our clear distinction between preserving previous classifiers and preventing cross-task prediction errors. We also appreciate the reviewer's constructive concerns regarding the individual contributions of ECR's energy and diversity terms, the interpretation of effective rank, and the role of ECR in class-incremental learning.

To address these concerns, we provide Energy-only and Diversity-only ablations, fixed and adaptive ECR comparisons, and variance/correlation regularization results. We also recompute effective rank using the unregularized covariance and provide all four combinations of head-level ECR and OOD suppression on the same Nullspace backbone, using five paired seeds. These additional analyses clarify the complementary roles of ECR's components, support the robustness of our effective-rank findings, and further identify where efficient coding provides benefits and where additional mechanisms are needed.

Below, we address each concern with the corresponding results and discussion. For the PMNIST experiments reported here, we use 50 tasks, five seeds, and networks with FC1 fixed at 256 and varying FC2 widths.

W1 & Q1: ECR Ablations and Regularizer Comparisons

In the following table, we compare standard training, energy-only, diversity-only, and full ECR with fixed and adaptive regularization weighting (η). The results show that the energy and diversity terms have complementary roles. The energy term discourages large activations and helps keep response magnitudes controlled, while the diversity term encourages the network to maintain a broader set of representation directions. This distinction is visible in the results. Diversity-only can achieve high acquisition and low dead-neuron fractions, but its activation RMS is higher than that of full ECR. Energy-only training reduces activation magnitude but does not encourage representation diversity. Full ECR balances these two objectives, providing competitive acquisition and retention while maintaining controlled activation magnitudes. These results support the complementary contributions of both terms, although their relative benefits vary across network widths.

The results also clarify the contribution of adaptive η. Full ECR with adaptive weighting achieves the highest mean retention at FC2 = 32, 64, and 128, while fixed weighting performs better at the two narrowest widths. These findings suggest that adaptive weighting is particularly beneficial at larger network widths.

In each network configuration, we have bolded the best comparable values for readability:

FC2MethodAcquisition ↑ (%)Retention ↑ (%)FC2 Dead fraction ↓ (%)Total hidden RMS
8Standard77.4 ± 27.023.8 ± 6.845.0 ± 30.618.3±4.1
8Energy only85.9 ± 13.025.8 ± 5.24.0 ± 8.92.04±0.83
8Diversity only96.0 ± 0.134.0 ± 2.50.0 ± 0.0174±5.4
8Full ECR, fixed η96.2 ± 0.031.6 ± 1.21.0 ± 1.02.41±0.096
8Full ECR, adaptive η95.6 ± 1.029.7 ± 1.10.5 ± 1.11.76±0.29
16Standard89.6 ± 7.325.3 ± 2.759.8 ± 14.316.5±3.0
16Energy only95.7 ± 0.9**33.0 ± 2.21.4 ± 0.91.75±0.40
16Diversity only96.2 ± 0.027.3 ± 1.80.0 ± 0.0124±4.6
16Fixed η96.5 ± 0.136.5 ± 1.31.0 ± 0.72.53±0.14
16Adaptive ECR96.4 ± 0.132.6 ± 3.15.0 ± 8.22.77±0.77
32Standard89.1 ± 14.525.3 ± 3.374.5 ± 14.915.2±3.3
32Energy only96.6 ± 0.141.4 ± 2.70.5 ± 0.60.970±0.067
32Diversity only96.2 ± 0.018.5 ± 0.90.0 ± 0.087.9±1.7
32Fixed η96.7 ± 0.040.9 ± 1.40.5 ± 0.32.50±0.067
32Adaptive ECR96.7 ± 0.144.4 ± 1.90.0 ± 0.01.74±0.23
64Standard96.3 ± 0.429.1 ± 2.981.5 ± 4.811.4±1.7
64Energy only96.7 ± 0.047.1 ± 0.90.3 ± 0.20.881±0.040
64Diversity only96.2 ± 0.016.5 ± 0.20.0 ± 0.069.2±2.5
64Fixed η96.9 ± 0.042.3 ± 1.21.9 ± 0.72.70±0.071
64Adaptive ECR96.8 ± 0.148.7 ± 0.80.0 ± 0.11.79±0.087
128Standard96.8 ± 0.030.0 ± 1.786.1 ± 1.98.77±0.72
128Energy only96.7 ± 0.150.2 ± 1.81.1 ± 0.40.878±0.051
128Diversity only96.0 ± 0.116.7 ± 0.40.0 ± 0.055.9±1.2
128Fixed η97.0 ± 0.045.1 ± 1.74.4 ± 1.62.75±0.096
128Adaptive ECR96.9 ± 0.050.6 ± 0.60.2 ± 0.21.85±0.12

We also investigated the variance/covariance regularizer suggested by the reviewer. We tested several variance targets and covariance coefficients to find a suitable configuration. However, we found that the variance term remained inactive during most training updates, even after adjusting its target. As a result, the regularizer was effectively driven by the covariance term alone, rather than a meaningful combination of both terms. We therefore did not consider these pilot results a sufficiently calibrated comparison. Instead, we provide five-seed results using a variance/correlation regularizer, where correlation is the normalized form of covariance. This normalization reduces the influence of activation scale, allowing us to examine redundancy between neurons more independently of their response magnitudes. Although this is not identical to the suggested variance/covariance objective, it allows us to examine how ECR compares with another approach to controlling representational redundancy under the same supervised learning setting. Our work is motivated by the efficient-coding principle of balancing response cost and representation diversity, and alternative regularizers may also encourage related properties. Here, you can see our results for both objectives, instead of energy/diversity.

FC2MethodAcquisition ↑ (%)Retention ↑ (%)FC2 Dead neurons ↓ (%)Gradient ratio
8Full adaptive ECR95.6 ± 1.029.7 ± 1.10.5 ± 1.10.108
8Var + Corr (0.3, 0.3)96.1 ± 0.135.7 ± 2.711.0 ± 6.30.096
8Var + Corr (0.1, 1.0)94.5 ± 2.531.4 ± 5.024.0 ± 6.40.098
16Full adaptive ECR96.4 ± 0.132.6 ± 3.15.0 ± 8.20.108
16Var + Corr (0.3, 0.3)96.4 ± 0.034.0 ± 1.98.4 ± 3.40.090
16Var + Corr (0.1, 1.0)96.1 ± 0.133.6 ± 2.735.8 ± 6.00.093
32Full adaptive ECR96.7 ± 0.144.4 ± 1.90.0 ± 0.00.106
32Var + Corr (0.3, 0.3)96.6 ± 0.033.9 ± 1.615.8 ± 1.50.085
32Var + Corr (0.1, 1.0)96.4 ± 0.132.0 ± 1.434.4 ± 3.60.086

The values in parentheses denote the two regularization coefficients, respectively. The final column reports the achieved ratio between the regularization and supervised gradient norms, with a target of 0.10. The variance/correlation regularizer achieves higher retention at FC2 = 8 and 16, while ECR achieves higher retention at FC2 = 32 and maintains lower dead-neuron fractions across all three widths. These findings suggest that alternative regularization objectives can also provide substantial benefits, while highlighting differences in how the objectives affect acquisition, retention, and representational activity.

W2 & Q3: Effective-Rank Correction and Total Variance

We thank the reviewer for pointing out the potential influence of ε on the effective-rank calculation. We originally included ε for numerical stability. However, we agree that adding εI to the covariance could increase measured effective rank when the covariance eigenvalues are very small. Following your suggestion, we recomputed effective rank directly from the unregularized covariance spectrum and compared it with the calculation using Equation (9) of the submitted manuscript. While we can see that ECR total variance is lower than the one for standard, even with removing the ε, the difference between actual values and the ratio between ECR and standard is small and negligible, and the effective-rank advantage of ECR remains after removing ε. We will update the manuscript reported effective ranks based on the corrected equation.

FC2Standard original rank (eq (9))ECR original rank (eq (9))Standard raw rankECR raw rankOriginal ECR/StandardCorrected ECR/ StandardStandard total varianceECR total variance
83.47 ± 0.616.38 ± 0.503.47 ± 0.626.39 ± 0.501.8391.83933,998 ± 24,03027.9 ± 11.7
163.58 ± 1.157.62 ± 1.383.58 ± 1.157.63 ± 1.382.1312.13328,853 ± 12,130172.6 ± 110.7
323.99 ± 1.8414.61 ± 1.464.00 ± 1.8414.66 ± 1.473.6603.66927,573 ± 10,366103.5 ± 36.5
645.06 ± 1.0119.51 ± 1.455.07 ± 1.0119.63 ± 1.463.8523.87216,042 ± 4,800193.7 ± 21.4
1285.72 ± 0.4825.34 ± 3.465.73 ± 0.4825.64 ± 3.504.4284.4749,528 ± 1,796345.9 ± 44.1

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear Reviewers and AC,

As the rebuttal period comes to an end, we would like to thank all reviewers for their valuable comments and suggestions, which have helped us improve our work. We understand that some reviewers may also have their own submissions and papers to work on, and we truly appreciate the considerable time and expertise they have devoted to our work. While we have addressed their concerns to the best of our ability, we will continue to address their suggestions when preparing the final version. Science moves forward through all of our contributions, and we thank the reviewers for recognizing our work as part of this process.

We would like to summarize our main contributions and the additional evidence provided during the discussion.

In this paper, we introduced the Efficient Coding Regularizer (ECR), inspired by the efficient-coding principle of balancing response energy and representation diversity. Our results show that ECR improves acquisition and retention in domain-incremental learning, particularly in small networks with limited representational capacity.

Importantly, Reviewer ueLs recognized our "convincing evidence of improved learning under constrained capacity, supported by controlled, multi-seed experiments," while Reviewer 3rYE described our PMNIST analysis as "thorough and interpretable." In their follow-up, Reviewer 3rYE further stated that "the new comparisons and ablations largely address my main concerns and support ECR's benefits in the small-network domain-incremental settings." We sincerely appreciate this assessment.

We are also grateful that Reviewer pLzm recognized the interest of our idea and acknowledged the value of our qualitative analyses. In their follow-up, the reviewer also noted that "the added CBP/S&P, LeakyReLU/LayerNorm, timing, and Var+Corr results address several of my concerns." These additional comparisons have strengthened the experimental evaluation of ECR and helped us present its contributions more clearly.

We would also like to summarize our main updates during the rebuttal:

  1. Paper scope and motivation: We revised the title to focus on domain-incremental learning in small networks, clarified the theoretical motivation of ECR, and made the limitations of our approach more explicit.

  2. Additional baselines: We conducted new comparisons with Continual Backpropagation (CBP), Shrink and Perturb (S&P), Winning Subnetworks (WSN), L2-init, and variance/correlation regularization. We also started additional experiments with SNV, DCNet, CReLU, and weight decay.

  3. Activation and normalization choices: We evaluated LeakyReLU and LayerNorm, both independently and combined with ECR. These experiments showed that ECR can provide additional retention benefits even when these methods are used.

  4. ECR ablations: We compared energy-only, diversity-only, and full ECR, as well as fixed and adaptive regularization weighting, to better understand the contribution of each component.

  5. Representation analysis: We recomputed effective rank using the unregularized covariance and reported total variance to verify that the observed improvements were not caused by the numerical stabilization term.

  6. Class-incremental learning: We added comparisons of head-level ECR and OOD suppression, including all four configurations, and conducted additional logit-norm control experiments on CIFAR-100 and Tiny-ImageNet.

  7. Computational efficiency: We clarified the adaptive-weight update rule, reported wall-clock training times, and provided additional relearning-time measurements and early-stopping details.

  8. Presentation and experimental details: We revised several sections for clarity, updated tables with additional seeds, clarified the dead-fraction measurements and forgetting metrics, and provided further explanations of the decoding experiments and their limitations.

We are glad that we were able to address many of the reviewers' concerns. We will continue to refine the presentation of our contributions and improve the clarity of the manuscript in the camera-ready version.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.