acceptodds
Under review as a conference paper at ICLR 2027

PSCAD: Parameter Space Continuous Attractor Dynamics in Continual Learning

Abstract

Continual learning models acquire new knowledge while preserving previously learned information. Dynamic architecture methods allocate task-specific parameters to protect previously learned knowledge from direct overwriting, but leave the growing collection of task updates without explicit coordination. Inspired by the separation of human brain between knowledge acquisition during wakefulness and memory consolidation during sleep, we introduce Parameter-Space Continuous Attractor Dynamics (PSCAD), a parameter-consolidation mechanism for rehearsal-free continual learning that complements data-driven training process. After task-specific learning, the mechanism treats accumulated task parameters as dynamical states and refines them through CANN-inspired dynamics. The dynamics combine affinity-weighted interactions among task parameters, adaptation-based regulation, and associative retrieval from a learnable Hopfield memory. We establish sufficient conditions for Lyapunov energy dissipation and convergence to the equilibrium set of the continuous-time system with fixed memory. Experiments on ImageNet-R and ImageNet-A show consistent improvements in final average accuracy across all evaluated settings, together with gains in average anytime accuracy in most settings. Ablations further support the contribution of each consolidation component. The findings demonstrate the effectiveness of parameter-space consolidation through CANN-inspired dynamics in continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Continuous Attractor Dynamics, Hopfield Networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CqtAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper proposes a new architecture-based approach for class-incremental learning, PSCAD. The strategy treats accumulated task parameters as a dynamical system and refines it via CANN-experimental dynamics.

Critical strengths
  1. the architecture-based strategy is an interesting approach for continual learning.
  2. the presentation of this paper is fine.
Critical weaknesses
  1. My main concern lies in the experiment of this paper which does not appear to be rigorous and carefully designed. Experiments are conducted with a single random seed where no standard deviations are reported.

  2. PROL by Ma'sum et al is proposed for online continual learning (single-pass). How do you guarantee the fairness of the comparison? You need to compare it under a single pass learning mode as well.

  3. It is not clear whether your approach utilizes a pretrained network or not. This should be clearly explained. It is mentioned that SD-LoRA is the main competitor. Do you develop your method based on it? If so, the novelty of your approach should be explained.

Questions for authors

Please answer the three major concerns.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CqtA

Thank you for recognizing the value of PSCAD and our theoretical analysis. We address your concerns about the theoretical analysis, experimental evidence and implementation details below. Table numbers refer to the revised manuscript.

  1. Missing standard deviations. We now report means and standard deviations over three independent runs with different random seeds, including the main results and component ablations. These statistics make the variability explicit; we do not interpret small mean improvements alone as evidence of statistical significance.

Table 1. Results on ImageNet-R. Values are mean ± standard deviation.

MethodN=5 Acc ↑N=5 AAA ↑N=10 Acc ↑N=10 AAA ↑N=20 Acc ↑N=20 AAA ↑
Full Fine-Tuning64.92 ± 0.8775.57 ± 0.5060.57 ± 1.0672.31 ± 1.0949.95 ± 1.3165.32 ± 0.84
L2P73.04 ± 0.7176.94 ± 0.4171.26 ± 0.4476.13 ± 0.4668.97 ± 0.5174.16 ± 0.32
DualPrompt69.99 ± 0.5772.24 ± 0.4168.22 ± 0.2073.81 ± 0.3965.23 ± 0.4571.30 ± 0.16
CODA-Prompt76.63 ± 0.2780.30 ± 0.2874.05 ± 0.4178.14 ± 0.3969.38 ± 0.3373.95 ± 0.63
InfLoRA76.95 ± 0.2381.81 ± 0.1474.75 ± 0.6480.67 ± 0.5569.89 ± 0.5676.68 ± 0.57
SD-LoRA (Reported)79.15 ± 0.2083.01 ± 0.4277.34 ± 0.3582.04 ± 0.2475.26 ± 0.3780.22 ± 0.72
SD-LoRA (Reproduced)77.74 ± 0.1082.02 ± 1.0975.77 ± 0.4781.21 ± 1.0372.98 ± 1.1479.47 ± 1.52
PLAN77.79 ± 0.2481.93 ± 0.6375.25 ± 0.4280.41 ± 0.5671.06 ± 0.4277.93 ± 0.56
HippoTune77.16 ± 0.2881.04 ± 0.3774.85 ± 0.1779.92 ± 0.2274.06 ± 0.2579.33 ± 0.49
PSCAD79.98 ± 0.3184.00 ± 0.2877.64 ± 0.3383.01 ± 0.8975.43 ± 0.0680.69 ± 0.88

2.Comparison with PROL. Thank you for pointing out this protocol mismatch. We agree that PROL targets single-pass online continual learning, whereas our experiments use multiple training epochs per task. The data-access constraints differ, making the original comparison inappropriate. We have therefore removed PROL from the main comparison table and withdrawn the associated comparative claims.

3.Pretrained backbone and novelty beyond SD-LoRA. We use a pretrained ViT-B/16, initialized from an ImageNet-21K checkpoint further fine-tuned on ImageNet-1K, as specified in Section 5.1. PSCAD is built on SD-LoRA, which provides the task-specific parameter-learning stage. Our contribution is an additional parameter-space consolidation mechanism: accumulated task parameters are treated as interacting dynamical states and refined through affinity-weighted interactions, adaptation, and Hopfield memory retrieval, then combined with their original values through residual connections. The memory is learned from the current-task loss by differentiating through consolidation, without storing or replaying historical training examples.

The novelty therefore lies in explicitly reorganizing previously learned task parameters through a memory-guided dynamical process, beyond the task-specific learning provided by SD-LoRA. We further characterize this process through continuous-time and discrete-time energy-dissipation analyses under stated conditions. The reproduced SD-LoRA comparisons and component ablations evaluate the contribution of this additional consolidation mechanism.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your time and constructive feedback. We have posted our responses and updated the manuscript with additional experiments and clarifications addressing your concerns.

We would appreciate your feedback on whether these additions resolve your main concerns. If any issues remain, we would be happy to clarify them. If you find that your concerns have been addressed, we would be grateful if you could reconsider your assessment and rating accordingly.

Thank you for considering our response.

My comments have been addressed. I agree to increase my score to weak accept.

Reviewer_qj8nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Parameter-Space Continuous Attractor Dynamics (PSCAD) for rehearsal-free class-incremental learning. Built on SD-LoRA, PSCAD treats accumulated task-specific parameters as dynamical states and consolidates them through potential-based interactions, adaptation, and Hopfield memory retrieval. The evolved parameters are combined with the original parameters through a residual connection. The memory matrix is updated using current-task data by differentiating through the consolidation process. The authors construct a Lyapunov function and establish energy dissipation for the continuous-time system with fixed memory under specified conditions, together with convergence to the equilibrium set under a stricter condition. Experiments on ImageNet-R and ImageNet-A show higher final average accuracy than the reported SD-LoRA baseline in all evaluated configurations, although average anytime accuracy decreases on ImageNet-A.

Critical strengths
  1. A concrete mechanism for parameter-space consolidation. PSCAD explicitly allows accumulated task parameters to interact and evolve under guidance from persistent memory. This offers a worthwhile direction for rehearsal-free continual learning. The higher final accuracy than the reported SD-LoRA baseline across all four configurations in Tables 1–2 provides preliminary empirical support.

  2. A relatively complete theoretical treatment with explicit assumptions. Pages 4–5 and Appendix A provide the energy dissipation identity, a lower bound, compactness of sublevel sets, and convergence to the equilibrium set. This analysis helps characterize the proposed system under fixed memory, beyond an exclusively empirical description of its dynamics.

Critical weaknesses
  1. The modest improvements are not supported by sufficient uncertainty reporting or controlled comparisons. Relative to SD-LoRA, the final accuracy improvements on ImageNet-R are 0.83, 0.64, and 0.17 percentage points, and the improvement on ImageNet-A is 0.67 points. Although the authors report averages over three runs, they provide no standard deviations, per-seed results, or paired differences. Consequently, it is unclear whether these improvements are reliable relative to run-to-run variation. The ablation differences in Table 3 are similarly small, particularly the AAA differences of 0.03–0.19 points. Moreover, page 6 states that baseline results are taken from previous publications, without presenting a rerun of the direct SD-LoRA backbone under identical settings. Table 4 shows approximately twice the GFLOPs per batch and 3.85 times the trainable parameter count. Comparisons under matched settings and budgets, controls for additional capacity, and simpler parameter-processing baselines are therefore important for attributing the gains to the proposed dynamics. The lower AAA on ImageNet-A also limits a broad claim of improved continual-learning performance.

  2. The implemented algorithm is insufficiently specified, and the theoretical properties are not adequately connected to implementation or knowledge retention. Sections 3–4 do not sufficiently specify the evolved parameter representation, task-affinity computation, dynamical hyperparameters, memory initialization, or the repeated execution of Stages II–III. These details are essential for assessing the meaning of parameter-space distances and reproducing the main results. The theorem concerns continuous-time dynamics with fixed memory, whereas the implementation uses finite-step discretization, residual mixing, and learned memory. The continuous-time condition alone does not establish stability or energy dissipation for the discrete updates in Equations (10)–(11), and the single illustrative trajectory in Figure 2 does not establish a general connection. More importantly, decreasing parameter energy does not imply decreasing previous-task loss. Before/after consolidation results on previous tasks, forgetting measurements, and comparisons with simple smoothing or direct retrieval would help establish why these dynamics preserve knowledge.

Questions for authors
  1. Are all SD-LoRA results in Tables 1–2 taken from prior publications? Could you rerun SD-LoRA using identical data splits, task orders, seeds, and training settings, and report per-seed results and paired differences for the main comparisons and ablations?
  2. Please specify the actual representation of θi\theta_i, the computation of wijw_{ij}, all dynamical hyperparameters, memory initialization, and the complete multi-epoch execution of Stages II–III. If the states are LoRA factors, how are equivalent factor representations handled when computing distances and retrieval?
  3. What time constants, step size, and iteration count are used in the experiments? Do they satisfy the theoretical conditions? Could you report energy, parameter-norm, and previous-task accuracy changes under the actual training configuration?
  4. Under matched training budgets, how does PSCAD compare with simple parameter smoothing, residual rescaling, or direct Hopfield retrieval? Do the FLOP measurements in Table 4 include the full dynamical unrolling and backpropagation through memory?

Additional Comments

  1. Make N/KN/K and w/wiw/w_i consistent between Algorithm 1 and the main text.
  2. Clarify how the gradient-update notation in Equation (13) relates to the AdamW optimizer used experimentally.
  3. Provide precise definitions of Acc and AAA and the evaluation protocol.
  4. Calibrate the description of the overhead against Table 4, distinguishing per-batch computation from total training cost.
  5. In Figure 2, the gap relative to the final recorded energy is not the error relative to the true limiting energy. The exponential-fit interpretation should therefore be qualified.
  6. Resolve the missing citation “(?)” on page 1.
3 comments
AuthorsOpenReview ↗

Response to Reviewer qj8n (Part 1/2)

Thank you for recognizing the value of PSCAD and our theoretical analysis. We address your concerns about the theoretical analysis, experimental evidence and implementation details below. Table numbers refer to the revised manuscript. We provide our response in two parts due to the character limit.

  1. Missing standard deviations. We now report means and standard deviations over three independent runs with different random seeds, including the main results and component ablations. These statistics make the variability explicit; we do not interpret small mean improvements alone as evidence of statistical significance.

  2. Small differences in the ablation study. We have extended the ablation study to ImageNet-R with 10 tasks, where the performance gaps are larger than in the original 5-task setting. This provides additional evidence for the contributions of the proposed components beyond a single task configuration.

Table 4. Component ablations on ImageNet-R. Values are mean ± standard deviation.

MethodN=5 Acc ↑N=5 AAA ↑N=10 Acc ↑N=10 AAA ↑
Full Model79.98 ± 0.3184.00 ± 0.2877.64 ± 0.3383.01 ± 0.89
Without Interaction79.76 ± 0.3683.97 ± 0.2577.26 ± 0.4182.44 ± 1.00
Without Adaptation79.74 ± 0.2783.90 ± 0.3077.44 ± 0.1582.35 ± 1.04
Without Hopfield79.75 ± 0.1783.81 ± 0.4677.06 ± 0.0882.25 ± 1.10
  1. No controlled rerun of SD-LoRA. We have rerun SD-LoRA under the same experimental settings as PSCAD and report the reproduced results alongside the originally published results in Table 1. Relative to the reproduced baseline, PSCAD improves final accuracy by 2.24, 1.87, and 2.45 percentage points on ImageNet-R with 5, 10, and 20 tasks, respectively; the corresponding AAA gains are 1.98, 1.80, and 1.22 points. Both sets of baseline results remain visible for transparency.

Selective Results of Table 1. Results on ImageNet-R. Values are mean ± standard deviation.

MethodN=5 Acc ↑N=5 AAA ↑N=10 Acc ↑N=10 AAA ↑N=20 Acc ↑N=20 AAA ↑
SD-LoRA (Reported)79.15 ± 0.2083.01 ± 0.4277.34 ± 0.3582.04 ± 0.2475.26 ± 0.3780.22 ± 0.72
SD-LoRA (Reproduced)77.74 ± 0.1082.02 ± 1.0975.77 ± 0.4781.21 ± 1.0372.98 ± 1.1479.47 ± 1.52
PSCAD79.98 ± 0.3184.00 ± 0.2877.64 ± 0.3383.01 ± 0.8975.43 ± 0.0680.69 ± 0.88
  1. Simpler alternatives, training budgets, and FLOP accounting. We added an equal-epoch comparison with parameter smoothing, residual rescaling, and direct Hopfield retrieval on ImageNet-R with 5 tasks (Table 6). PSCAD performs best among the evaluated consolidation strategies under the same epoch budget. The FLOP measurements include the full dynamical unrolling and backpropagation through the memory module. As clarified in Section 5.5, the reported GFLOPs sum the forward and backward costs of one batch from each training stage after warm-up.

Table 6. Comparison with simpler consolidation strategies under the same epoch budget on ImageNet-R (N=5). Values are mean ± standard deviation.

MethodAcc ↑AAA ↑
PSCAD79.98 ± 0.3184.00 ± 0.28
Parameter Smoothing79.36 ± 0.2183.44 ± 0.73
Residual Rescaling78.77 ± 0.0183.18 ± 0.72
Direct Hopfield Retrieval78.69 ± 0.2983.12 ± 1.03
  1. Lower AAA on ImageNet-A. We acknowledge the limitation in AAA on ImageNet-A. Nevertheless, it improves final accuracy from 55.96 to 56.63 on this dataset and improves both Acc and AAA across all three ImageNet-R configurations. These results support the value of PSCAD in improving continual-learning performance, while showing that its benefits can vary across all datasets and metrics. We have clarified this limitation in Section 5.2.

Our responses to the remaining concerns continue in Part 2/2.

AuthorsOpenReview ↗

Response to Reviewer qj8n (Part 2/2)

This continues our response in Part 1/2. Below, we address the remaining concerns about the theoretical analysis, experimental evidence and implementation details.

  1. Implementation details and equivalent LoRA representations. The state θi\theta_i concatenates the flattened LoRA factors introduced for task ii across the adapted layers:

θi=concat⁡ℓ(vec⁡(Ai(ℓ)),vec⁡(Bi(ℓ))).\theta_i=\operatorname{concat}_{\ell}\left(\operatorname{vec}(A_i^{(\ell)}),\operatorname{vec}(B_i^{(\ell)})\right).

Task affinities wijw_{ij} are computed as

wij=max⁡(0,θi⊤θj∥θi∥2∥θj∥2)(i≠j),wii=0.w_{ij}=\max\left(0,\frac{\theta_i^\top\theta_j}{\|\theta_i\|_2\|\theta_j\|_2}\right)\quad(i\ne j),\qquad w_{ii}=0.

The bandwidth and interaction strength are calibrated from the parameter states:

at=max⁡(0.5 median⁡i<j∥θi−θj∥2,10−12),a_t=\max\left(0.5\,\operatorname{median}_{i<j}\|\theta_i-\theta_j\|_2,10^{-12}\right),

J0=0.01∗∥−θ−V+Iext∥2∥Ω(J0=1)(θ)∥2,J_0=0.01 * \frac{\|-\theta-V+I_{\mathrm{ext}}\|_2}{\|\Omega^{(J_0=1)}(\theta)\|_2},

where IextI_{\mathrm{ext}} is the Hopfield retrieval input and Ω(J0=1)\Omega^{(J_0=1)} is the interaction force evaluated with unit coupling strength.

The remaining settings are ρ=0.5\rho=0.5, β=1\beta=1, m=0.5m=0.5, τs=0.6\tau_s=0.6, τz=1.0\tau_z=1.0, Δs=0.01\Delta s=0.01 and K=5K=5. The memory matrix uses the default Kaiming-uniform weight initialization of nn.Linear.

For each task, Stage I runs for E1E_1 epochs, followed by E2E_2 epochs of Stage II–III: consolidation is unrolled, and the current-task loss is backpropagated through it to update memory using AdamW. Algorithm 1 specifies these loops. We use E1=E2=15E_1=E_2=15 on ImageNet-R and E1=E2=10E_1=E_2=10 on ImageNet-A.

We acknowledge that our factor-space representation does not account for equivalent LoRA factorizations. We adopt this representation to keep the state dimension manageable by avoiding explicit construction of the dense weight-update matrices.

  1. Connecting continuous-time theory to discrete updates. We agree that the original continuous-time theorem alone does not establish discrete-time stability. We have added Theorem 2 and its proof in Appendix A.4 for the sequential discrete updates in Eqs. (14)–(15). With memory, affinities, and dynamical parameters fixed within a trajectory, it provides an explicit sufficient step-size condition, Qh≻0Q_h\succ0, guaranteeing energy decrease at non-equilibrium states and bounded iterates. Convergence to the equilibrium set follows if the same fixed system is iterated indefinitely.

  2. Experimental settings and before/after consolidation results. We use τs=0.6\tau_s=0.6, τz=1.0\tau_z=1.0, Δs=0.01\Delta s=0.01 and K=5K=5 internal steps per consolidation. Thus, τs>mτz\tau_s>m\tau_z holds (0.6>0.50.6>0.5).

Table 3 reports before/after consolidation measurements on ImageNet-R with 5 tasks. PSCAD reduces the Lyapunov energy with a smaller change in the LoRA parameter norm than parameter smoothing or direct Hopfield retrieval, while achieving the highest post-consolidation old-task accuracy. These results support its ability to refine parameters while preserving previously learned knowledge.

Table 3. Changes before and after consolidation on ImageNet-R (N=5). Values are mean ± standard deviation.

MethodΔE\Delta EΔ∥θLoRA∥2\Delta\lVert\theta_{\mathrm{LoRA}}\rVert_2Old-task Acc beforeOld-task Acc after
PSCAD-15.832 ± 3.323-0.0723 ± 0.001578.568 ± 0.35378.648 ± 0.401
Parameter Smoothing-14.544 ± 1.689-0.3478 ± 0.036478.341 ± 0.47578.439 ± 0.425
Direct Hopfield Retrieval-42.056 ± 2.665-0.3344 ± 0.076678.439 ± 0.54278.010 ± 0.455
  1. Additional comments. We have revised the notation in Algorithm 1, clarified the AdamW update, defined Acc and AAA , clarified that the computational cost is measured per batch, and corrected the missing citation. Figure 2 now identifies the reference as the final recorded energy.
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your time and constructive feedback. We have posted our responses and updated the manuscript with additional experiments and clarifications addressing your concerns.

We would appreciate your feedback on whether these additions resolve your main concerns. If any issues remain, we would be happy to clarify them. If you find that your concerns have been addressed, we would be grateful if you could reconsider your assessment and rating accordingly.

Thank you for considering our response.

Reviewer_6QhrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Parameter-Space Continuous Attractor Dynamics (PSCAD) for rehearsal-free continual learning. It refines task-specific parameters using continuous attractor dynamics, adaptation, and Hopfield memory. The method builds on SD-LoRA and aims to preserve previously learned knowledge without storing old training data.

Critical strengths
  1. The paper explores knowledge consolidation directly in parameter space rather than using stored training samples.
  2. PSCAD combines parameter interactions, adaptation, and Hopfield memory within a unified framework.
  3. The paper provides a Lyapunov-based analysis of energy dissipation and convergence under fixed-memory conditions.
  4. PSCAD improves final average accuracy over its SD-LoRA backbone across all evaluated task settings.
Critical weaknesses
  1. PSCAD improves accuracy by only 0.17 percentage points on ImageNet-R with 20 tasks as mentioned in Table 1. The results are averaged over three runs without reporting standard deviations or statistical significance.
  2. The stability analysis assumes continuous-time dynamics with fixed memory shown in Theorem 1. It does not establish stability for the implemented discrete updates or the complete memory-learning procedure.
  3. PSCAD modifies previously learned task parameters during consolidation in Section 4. The paper does not establish whether these modifications consistently preserve knowledge from earlier tasks.
  4. Component ablations are reported only for ImageNet-R with five tasks shown in Table 3. The improvements are small. More extensive experiments are needed to establish each component's contribution.
  5. In Table 4, PSCAD nearly doubles the reported GFLOPs and quadruples trainable parameters compared with SD-LoRA. The small accuracy gains do not clearly justify this additional cost.
  6. Experiments cover only ImageNet-R and ImageNet-A. Direct forgetting measurements and evaluations on longer task sequences are needed to support the knowledge-retention claims.
Questions for authors
  1. Can the authors demonstrate that the theoretical stability conditions also hold for the implemented discrete consolidation process?
  2. How are the task affinity weights computed? How does the method prevent changes to previously learned parameters from increasing forgetting?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6Qhr

Thank you for recognizing the value of PSCAD and our theoretical analysis. We address your concerns about the theoretical analysis, experimental evidence and implementation details below. Table numbers refer to the revised manuscript.

  1. Missing standard deviations. We now report means and standard deviations over three independent runs with different random seeds, including the main results and component ablations. These statistics make the variability explicit; we do not interpret small mean improvements alone as evidence of statistical significance.

2.Connecting continuous-time theory to discrete updates. We agree that the original continuous-time theorem alone does not establish discrete-time stability. We have added Theorem 2 and its proof in Appendix A.4 for the sequential discrete updates in Eqs. (14)–(15). With memory, affinities, and dynamical parameters fixed within a trajectory, it provides an explicit sufficient step-size condition, Qh≻0Q_h\succ0, guaranteeing energy decrease at non-equilibrium states and bounded iterates. Convergence to the equilibrium set follows if the same fixed system is iterated indefinitely.

3.Preserving knowledge when modifying previous task parameters. PSCAD combines the evolved parameters with the original task parameters through a residual connection after the internal consolidation steps:

θ~i=(1−wi)θi+wiθi(sK).\widetilde{\theta}_i=(1-w_i)\theta_i+w_i\theta_i(s_K).

This mixing retains the contribution of the original parameters and controls the magnitude of the consolidation-induced change. To directly examine this issue, we added before/after consolidation measurements on ImageNet-R with 5 tasks (Table 3). PSCAD reduces the Lyapunov energy with a smaller change in the LoRA parameter norm than parameter smoothing or direct Hopfield retrieval, while achieving the highest post-consolidation old-task accuracy.

Table 3. Changes before and after consolidation on ImageNet-R (N=5). Values are mean ± standard deviation.

MethodΔE\Delta EΔ∥θLoRA∥2\Delta\lVert\theta_{\mathrm{LoRA}}\rVert_2Old-task Acc beforeOld-task Acc after
PSCAD-15.832 ± 3.323-0.0723 ± 0.001578.568 ± 0.35378.648 ± 0.401
Parameter Smoothing-14.544 ± 1.689-0.3478 ± 0.036478.341 ± 0.47578.439 ± 0.425
Direct Hopfield Retrieval-42.056 ± 2.665-0.3344 ± 0.076678.439 ± 0.54278.010 ± 0.455

4.Small differences in the ablation study. We have extended the ablation study to ImageNet-R with 10 tasks, where the performance gaps are larger than in the original 5-task setting. This provides additional evidence for the contributions of the proposed components beyond a single task configuration.

Table 4. Component ablations on ImageNet-R. Values are mean ± standard deviation.

MethodN=5 Acc ↑N=5 AAA ↑N=10 Acc ↑N=10 AAA ↑
Full Model79.98 ± 0.3184.00 ± 0.2877.64 ± 0.3383.01 ± 0.89
Without Interaction79.76 ± 0.3683.97 ± 0.2577.26 ± 0.4182.44 ± 1.00
Without Adaptation79.74 ± 0.2783.90 ± 0.3077.44 ± 0.1582.35 ± 1.04
Without Hopfield79.75 ± 0.1783.81 ± 0.4677.06 ± 0.0882.25 ± 1.10

5.Simpler alternatives with same training budgets. To assess whether PSCAD's additional computational cost is justified, we added an equal-epoch comparison with parameter smoothing, residual rescaling, and direct Hopfield retrieval on ImageNet-R with 5 tasks (Table 6). PSCAD performs best among the evaluated consolidation strategies under the same epoch budget, supporting the value of parameter-space consolidation mechanism.

Table 6. Comparison with simpler consolidation strategies under the same epoch budget on ImageNet-R (N=5). Values are mean ± standard deviation.

MethodAcc ↑AAA ↑
PSCAD79.98 ± 0.3184.00 ± 0.28
Parameter Smoothing79.36 ± 0.2183.44 ± 0.73
Residual Rescaling78.77 ± 0.0183.18 ± 0.72
Direct Hopfield Retrieval78.69 ± 0.2983.12 ± 1.03

6.Longer task sequences and direct forgetting measurements. We extended the evaluation to ImageNet-R with 40 sequential tasks and directly measured forgetting. SD-LoRA and PSCAD achieve forgetting scores of 9.18 ± 0.94 and 9.03 ± 0.41, respectively (lower is better). PSCAD has lower mean forgetting under this longer sequence.

7.Task-affinity weights definition. The affinity weight wijw_{ij} measures similarity between the flattened, concatenated LoRA-factor representations of tasks ii and jj:

wij=max⁡(0,θi⊤θj∥θi∥2∥θj∥2)(i≠j),wii=0.w_{ij}=\max\left(0,\frac{\theta_i^\top\theta_j}{\|\theta_i\|_2\|\theta_j\|_2}\right)\quad(i\ne j),\qquad w_{ii}=0.

These symmetric, nonnegative weights modulate the interaction between task parameter states: positively aligned states receive stronger affinity weights, while nonpositively aligned states receive zero weight.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your time and constructive feedback. We have posted our responses and updated the manuscript with additional experiments and clarifications addressing your concerns.

We would appreciate your feedback on whether these additions resolve your main concerns. If any issues remain, we would be happy to clarify them. If you find that your concerns have been addressed, we would be grateful if you could reconsider your assessment and rating accordingly.

Thank you for considering our response.

I thank the authors for their detailed responses. The discrete-time stability analysis and extended ablation studies strengthen the paper. However, several concerns remain.

  1. The new stability theorem assumes fixed memory within a consolidation trajectory. It remains unclear whether the experimentally used step size satisfies the proposed stability condition.
  2. In Table 3, parameter smoothing achieves a slightly larger accuracy improvement after consolidation than PSCAD. Comparisons from identical checkpoints would better establish the knowledge-retention benefits.
  3. The equal-epoch comparison does not ensure equal computational budgets. The additional computational cost remains insufficiently justified by the marginal performance improvements.
  4. The 40-task experiment shows only a 0.15 percentage-point reduction in forgetting. This does not provide sufficiently strong evidence of improved long-term knowledge retention.
  5. The task-affinity measure uses flattened LoRA factors. Equivalent LoRA factorizations can have different parameter similarities. The authors should clarify how this ambiguity is handled.

Overall, I appreciate the authors' efforts. I retain my original rating of weak rejection.

AuthorsOpenReview ↗

Response to Reviewer 6Qhr (Round 2)

We thank the reviewer for the careful follow-up and for acknowledging the strengthened stability analysis and ablation studies. Below, we address the remaining technical concerns.

1.Stability condition and the experimental step size.

We have now explicitly checked the proposed step-size condition on ImageNet-R with N=5N=5, using three random seeds. For each task and seed, we reconstructed consolidation trajectories from saved parameter and memory snapshots before and after memory training, yielding 30 trajectories in total. Each trajectory keeps the memory fixed throughout its internal updates and uses the implemented step size h=0.01h=0.01.

Under the actual dynamical parameters (τs,τz,m)=(0.6,1.0,0.5)(\tau_s,\tau_z,m)=(0.6,1.0,0.5), the sufficient condition Qh≻0Q_h\succ0 reduces to ℓt<19.497487\ell_t<19.497487. Across the checked trajectories, the maximum value of the curvature bound ℓt\ell_t was 0.1689800.168980, and the minimum eigenvalue of QhQ_h was 0.0765413>00.0765413>0. Thus, the experimentally used step size satisfies the sufficient condition for discrete-time energy dissipation for all 30 checked trajectories on ImageNet-R with N=5N=5.

2.Interpretation of the consolidation comparison.

We agree that parameter smoothing produces a slightly larger immediate increase in old-task accuracy in Table 3: 0.098 percentage points, compared with 0.080 for PSCAD. However, these changes should be interpreted in the context of the sequential training protocol.

Each method alternates the same task-specific learning procedure with its consolidation procedure throughout the task sequence. Consequently, the old-task accuracy measured before consolidation at a later task already reflects the effects of consolidation at earlier tasks. Although the task-specific learning procedure is shared, the resulting model states need not be identical because their preceding consolidation histories differ. PSCAD’s higher pre-consolidation accuracy is therefore consistent with a cumulative benefit of integrating PSCAD with task-specific learning.

Table 3 shows that PSCAD maintains the highest old-task accuracy both before and after consolidation along its own training trajectory. Complementing this observation, Table 6 reports higher final accuracy and average anytime accuracy for PSCAD than for parameter smoothing under the same number of training epochs: 79.98 versus 79.36, and 84.00 versus 83.44, respectively. These results support the effectiveness of the parameter-space consolidation of PSCAD.

3.Computational budget and additional cost.

We agree that equal epochs alone do not guarantee equal computational budgets. However, profiling the simpler consolidation baselines using the same protocol as Table 4 yields 4672.92, 4672.92, and 4672.99 GFLOPs for parameter smoothing, residual rescaling, and direct Hopfield retrieval, respectively, compared with 4673.36 GFLOPs for PSCAD—a difference of less than 0.01%. Thus, although our experimental design explicitly matches training epochs, the measured computation is also closely matched for these implementations. Under this comparison, PSCAD achieves highest Acc and AAA, which supports its utility.

Table 6: Comparison with simpler consolidation strategies under the same number of training epochs on ImageNet-R (N = 5).

MethodAcc ↑AAA ↑GFLOPs ↓
PSCAD79.98 (0.31)84.00 (0.28)4673.36
Parameter Smoothing79.36 (0.21)83.44 (0.73)4672.92
Residual Rescaling78.77 (0.01)83.18 (0.72)4672.92
Direct Hopfield Retrieval78.69 (0.29)83.12 (1.03)4672.99

4.Evidence from the 40-task experiment and other task configurations

We acknowledge that the 0.15-percentage-point reduction in forgetting in the 40-task experiment is modest. Nevertheless, PSCAD still improves upon the baseline in this longer task sequence, providing additional evidence of its benefit for knowledge retention.

The broader evaluation nevertheless shows consistent benefits across the ImageNet-R task configurations. Relative to the reproduced SD-LoRA baseline, PSCAD improves final accuracy by 2.24, 1.87, and 2.45 percentage points with 5, 10, and 20 tasks, respectively; the corresponding AAA gains are 1.98, 1.80, and 1.22 points. These results indicate improvements in both final performance and performance throughout the learning sequence.

5.Factorization dependence of task affinity

We acknowledge that the current task-affinity measure is not invariant to equivalent LoRA factorizations. We use the factor-space representation to keep the state dimension manageable and avoid explicitly constructing dense weight-update matrices. Accordingly, the resulting affinity should be interpreted as similarity within the chosen parameterization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.