acceptodds
Under review as a conference paper at ICLR 2027

Neural Consolidation Process: Selective Tagging and Validated Capture for Continual Learning

Abstract

Continual learning in deep neural networks is severely bottlenecked by catastrophic forgetting. While the human brain elegantly resolves this stability-plasticity dilemma through systems memory consolidation, i.e., gradually transferring short-term memory into stable neocortical traces, mimicking this in artificial networks without destructive overwriting remains an open challenge. In this paper, we propose **Neural Consolidation Process (NeuralCP)**, a novel continual learning framework inspired by the biological Synaptic Tagging and Capture (STC) mechanism. NeuralCP models short-term memory using a lightweight LoRA module to rapidly encode task-specific knowledge. Instead of accumulating task-specific adapters or relying on rigid parameter-space regularization, NeuralCP employs function-space attribution to explicitly tag modules responsible for critical representation shifts. We then introduce a replay-validated capture mechanism, utilizing a minimal calibration set strictly to filter for highly beneficial and consistently recallable memory tags. Finally, these validated tags are consolidated into the backbone via residual distillation within a stable-plastic subspace decomposition, dedicatedly protecting previously consolidated knowledge while maintaining future adaptability. Crucially, NeuralCP introduces zero additional parameters or overhead at inference. Extensive evaluations across eight sequential tasks demonstrate that NeuralCP promotes average continual learning scores by **22.72, 7.39, and 8.05** points on Qwen3-0.6B, 4B, and 8B respectively, setting a new state-of-the-art. Besides, NeuralCP exhibits consistent improvements and scalability on visual Transformers.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, PEFT, LLM Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ft9COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Neural Consolidation Process (NeuralCP), a biologically inspired continual learning framework designed to mitigate catastrophic forgetting while avoiding the increasing inference-time parameter overhead associated with task-specific adapters. Inspired by the Synaptic Tagging and Capture mechanism, NeuralCP separates learning into three main stages, including a temporary LoRA adapter which learns the incoming task, a replay-validation mechanism which evaluates the utility and consistency of the resulting memory tags using a small held-out calibration set, and lastly validated modules which are consolidated into the backbone through residual distillation and stable–plastic subspace decomposition.

Critical strengths

Two critical strengths exist in this paper:

  1. The proposed separation between temporary adaptation, selective validation, and permanent consolidation provides an intuitive framework for balancing stability and plasticity.

  2. Function-space attribution provides a meaningful mechanism for selective updating. This proposed mechanism combines hidden-state differences and loss-gradient sensitivity to identify task-relevant modules. This is a reasonable alternative to relying exclusively on parameter magnitudes or parameter-space importance measures which adopted by many previous research works.

Critical weaknesses

Two critical weaknesses of this paper I would like point out are:

  1. The novelty of several proposed components requires clearer justification. Although the biological motivation is interesting, the technical components primarily combine established ideas, including LoRA-based task adaptation, activation/gradient attribution, held-out validation, knowledge distillation, and orthogonal subspace projection. In particular, Section 3.3 and Appendix E explicitly show that the proposed residual logit distillation objective is algebraically equivalent to ordinary teacher–student logit matching. Consequently, residual distillation itself does not constitute a substantially different optimization objective.

  2. The replay-validation mechanism does not convincingly establish that historical knowledge is protected. A central claim is that replay validation identifies memory traces that should be retained. However, Section 3.2 defines the replay set R_t as a subset of the current task D_t, rather than using examples from previously learned tasks. Moreover, the validation residual in Eq. (8) is computed between the current short-term adapter and the current backbone, even when evaluating historical tags. This raises a fundamental concern: a historical tag may be rejected simply because it is not activated by current-task examples, despite remaining important for earlier tasks. Conversely, a tag may pass validation because it is useful for the current task without guaranteeing that consolidating the corresponding module preserves previous-task performance. The method therefore appears to validate current-task consistency more directly than historical knowledge retention. The authors should clarify how historical tags are meaningfully reactivated and validated without access to historical task samples and whether the proposed mechanism genuinely provides protection beyond the stable–plastic projection.

Questions for authors
  1. What are the key methodological innovations relative to existing selective distillation and subspace-based continual learning approaches? This question is related to the first weakness, as I pointed out in the Critical Weakness box.

  2. How does replay validation assess the usefulness of historical tags without using historical data? This question is related to the second weakness, as I pointed out in the Critical Weakness box.

  3. How are historical memory tags used during consolidation? The distillation objective seems to use the current short-term teacher rather than the stored tag representations. Does the low-rank tag basis U_{t,m} influence consolidation directly, or is it used only to update the stable memory bank? What additional benefit does retaining historical tags provide beyond maintaining the stable subspace itself?

  4. How does NeuralCP handle long-term capacity saturation? Another important question in continual learning field should be always addressed is how the method decide which old directions or memories to retain or discard, when the capacity is saturated, or the stable-subspace rank reaches r_{max}. Could the authors provide experiments involving substantially longer task sequences, together with measurements of stable ranks, plastic capacity, and forgetting?

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Ft9C (1/2)

We sincerely thank the reviewers for their meticulous and constructive review. We are pleased that the reviewers recognized NeuralCP's simplicity and novel selective update mechanism. We have added additional experiments and explanations to answer your questions.

W1 & Q1. Methodological contribution

The innovation lies not in the components themselves, but in our definition of stored content. EWC, O-LoRA, and LoRA-Mixer retain all updates, regardless of whether they are useful, informative, or noisy, and then process them. We consider which parts of the updates should be internalized into the backbone model and propose a standard process (rapid adaptation, attribution, validation, and consolidation) for implementation without incurring any additional inference overhead, which is crucial in real-world deployment scenarios. Regarding residual distillation, we will further refine its distinctions based on your suggestions to avoid misinterpreting it as a new loss function. To facilitate better understanding, we summarize the differences in the following diagram:

ApproachRepresentative methodMain mechanism
Parameter regularizationEWCPenalize changes to parameters important for previous tasks
Subspace-based learningO-LoRAReduce interference through orthogonal task updates
Modular learningLoRA-MixerRetain task-specific adapters and route inputs among experts
Validated selective consolidationNeuralCPSelect and validate candidate modules, consolidate into the backbone, and discard the temporary adapter

NeuralCP turns temporary task adaptation into selectively consolidated knowledge within a single backbone. By jointly deciding what to write and which directions to protect, it supports continual adaptation with zero additional inference parameters or routing overhead, without accumulating task-specific adapters. Component ablations demonstrate the value of this coordinated design. We will reinforce this section in the abstract and introduction.

W2 & Q2. Can historical tags be validated without historical data?

Eq. (8) compares the current adapter's residual against the stored prototype, so what the gate tests is whether the current task re-derives a stored direction. It does not re-measure old-task accuracy, and we will state this boundary explicitly in §3.2. Reactivation is observed, not induced. We never replay old inputs: a historical tag is re-examined only when the current task selects its module and drives the residual along the stored direction, which is the second activation STC requires for capture. A tag the current task does not re-derive is not re-captured, but it is not forgotten either — it stays in BmB_m and keeps constraining updates through Eq. (11). Rejection costs a write, never a memory.

Operationally, an accepted historical tag keeps its module in ScaptS_{\text{capt}} (Eq. 10): the tag acts as a credential for the module, while what is written is still the current residual. That asymmetry is what makes it safe to be strict. We will also stop implying that the gate demonstrates retention: retention comes from projection and from limiting the write to modules that pass the gate — the gate decides where we may write, the projection decides what cannot be overwritten. To see whether this check buys anything beyond projection, we run four tag-admission policies on Qwen3-0.6B over the five-task sequence. Training settings and the stable–plastic projection are identical in all four rows; only the rule for admitting a tag differs. The results are shown below:

MetricAccept all tagsNew-tag validation onlyCurrent-task gate (ours)Balanced past + current gate
Final CL avg.70.0773.8675.4275.68
Old-task avg.69.0273.4074.0774.30

The results show that historical tags can be revalidated against the current residual without any historical examples, and that a gate using historical examples offers little advantage over ours. The stored prototype supplies the historical reference and the current residual supplies the evidence, so no past data is needed on either side.

AuthorsOpenReview ↗

Rebuttal for Reviewer Ft9C (2/2)

Q3. How are historical tags used during consolidation?

Historical tags decide where the write happens, not what is written. At task tt the distillation target is always the current short-term teacher; a historical tag is used only as an admission credential for its module. When it passes validation against the current residual, its module stays in ScaptS_{\text{capt}} and may receive the write; when it fails, that module is not written.

Ut,mU_{t,m} does not affect the current consolidation; it only updates the stable bank. The target at task tt is the current short-term teacher, and Ut,mU_{t,m} is merged into BmB_m only after consolidation finishes. BmB_m is what constrains the next task's update through Eq. (11), so Ut,mU_{t,m} shapes the next write, not this one. In short, it takes no part in how the current update is computed — only in how the next one is constrained.

The extra benefit is a write-or-not check on each module, which the bank cannot provide. After orthonormalization and truncation, the bank records which directions are occupied but not which task produced them or whether they are still in use; the stored prototype is what allows a direction to be re-identified in the current forward pass. That is what lets us decide at task tt whether a module should receive a write, whereas projection only says where a write may not go. To test whether this adds anything beyond the bank, we run three validation policies on Qwen3-0.6B over the same five tasks, keeping the backbone, task order and projection identical and changing only the rule for admitting a tag. The results are as follows.

MetricNo validationNew tags validated, historical notFull (ours)
Five-task avg.70.0773.8675.42

Validating new tags is worth 3.79 points (70.07 → 73.86), and re-validating historical tags a further 1.56 (73.86 → 75.42).

Q4. Capacity saturation and longer task sequences

What happens when the bank is full? Each accepted tag brings up to eight directions with it, and a module's bank can hold 32. When a module is full, later directions are simply not added to it, and earlier ones are never evicted. Full modules also receive a lower score under the capacity penalty in Eq. (4), so later tasks tend to choose modules that still have room. The protected directions occupy only a small part of each module's input space, so most of it stays free for new updates.

Ten-task test. On Qwen3-0.6B we add QQP, MNLI, BoolQ, HellaSwag and STSB after the original five tasks. STSB uses Spearman correlation on a 0–100 scale, and the other tasks use accuracy. The results are as follows.

EvaluationBaseSD-LoRABiLoRANeuralCP
Five-task average after task 552.7065.5768.8275.42
Ten-task average after task 1052.1061.8064.1072.90

NeuralCP keeps its lead on the longer sequence as well. We evaluate the original five tasks again after task 10, so the change is not confounded by the difficulty of the added tasks. The results are as follows.

Original five-task averageSD-LoRABiLoRANeuralCP
After task 565.5768.8275.42
After task 1060.3162.9571.36
Change−5.26−5.87−4.06

All three methods lose some accuracy, but NeuralCP loses the least. Its advantage therefore comes from retaining old tasks, not only from learning new ones faster.

Rank and remaining capacity. For modules with nonempty banks we report the average number of protected directions and the share of input dimensions the projection leaves free.

Task boundary1510
Mean stable rank81428
Share of input dimensions free≥99%≥98%≥97%

These figures describe the geometry of the constraints, not the functional capacity. How much information the model can actually absorb should ultimately be reflected in its performance on real-world tasks. Based on the results of the ten tasks mentioned above, NeuralCP maintaining its accuracy even with a doubling of task sequences is the best proof.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Reviewer_ri8COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NeuralCP, a continual learning framework inspired by the biological Synaptic Tagging and Capture (STC) mechanism. The central idea is to reframe replay from “data rehearsal / gradient mixing” into a strict validation gate: each incoming task first trains a transient LoRA adapter as short-term memory; functional attribution on hidden-state shifts identifies task-salient modules and produces compact memory tags; a minimal held-out replay set validates whether these tags remain beneficial and consistently recallable; only validated tags are consolidated into the backbone via residual distillation under a stable-plastic subspace decomposition. After consolidation, the transient LoRA is discarded, so the final model introduces zero additional inference parameters or latency. The paper evaluates NeuralCP on eight sequential language tasks (GLUE, BoolQ, HellaSwag, GSM8K) with Qwen3-0.6B/4B/8B and LLaMA3.2-1B, and on ImageNet-R class-incremental learning with ViT-B/16 under 5-task and 10-task splits. It reports state-of-the-art average accuracy against Merged-LoRA, O-LoRA, SD-LoRA, EWC-LoRA, BiLoRA, and LoRA-Mixer, plus ablations on functional attribution, memory tagging, replay validation, and stable-plastic decomposition, and an efficiency comparison against LoRA-Mixer.

Critical strengths
  1. The replay-as-validation-gate framing is a consequential conceptual shift. Reframing replay from “rehearse old data via gradient updates” to “strictly validate whether stored memory tags remain beneficial and consistently recallable” is the paper’s most distinctive idea. It directly addresses the question of which task-induced changes deserve durable retention, rather than only how to constrain global updates. This framing is what makes the rest of the pipeline coherent, and it is the main reason the paper is worth discussing.

  2. The transient-LoRA / zero-inference-overhead design is practically meaningful and empirically supported. By treating LoRA as temporary short-term memory and consolidating validated changes into the backbone, NeuralCP avoids the ever-growing adapter bank of modular methods. The comparison against LoRA-Mixer (higher average accuracy, no additional parameters after consolidation, lower inference time and higher throughput) provides concrete evidence that this design choice has practical value, not just conceptual appeal.

Critical weaknesses

1.Key design choices lack ablation and sensitivity analysis, weakening the soundness of the SOTA claims. The product fusion , the calibration-set size and sampling, the stable-basis rank cap are central to the method, yet the main text provides no ablation or sensitivity analysis for them. The tag SVD rank is only studied in Appendix A, and the relative weighting of  and  is not ablated at all. Without these, it is difficult to tell how much of the gain comes from the proposed selective-consolidation principle versus specific hyperparameter choices. 2. The replay validation gate suffers from a distributional mismatch, potentially invalidating the "validation" claim. NeuralCP repurposes replay as a strict validation gate to decide whether a stored memory tag remains beneficial. However, the validation is performed on a held-out set , which consists exclusively of current-task data. Using current-task data to validate historical tags introduces a severe distributional bias: historical tags might be rejected simply because they do not activate on the new task, even if they are critical for preserving old-task knowledge. Conversely, the new task could spuriously activate old tags. Since this gate is the core mechanism that prevents noise from being consolidated, the paper does not analyze whether it reliably identifies modules essential for old-task retention rather than just current-task adaptation.

Questions for authors
  1. Fusion of attribution signals. Can you provide an ablation comparing against additive fusion, max fusion, and using or alone? This would clarify whether the product form is essential or whether the gains are robust to the fusion choice.
  2. Calibration set sampling and statistical significance. How is the calibration set sampled from (random, stratified, fixed seed)? How sensitive are the attribution scores, tag acceptance rates, and final accuracy to calibration-set size and composition? Appendix A studies replay-capture data size, but the calibration set used for functional attribution is a separate quantity and is not analyzed. Additionally, given the multi-stage and stochastic nature of NeuralCP, are the reported results averaged over multiple random seeds? If so, please report the standard deviations or confidence intervals, particularly for the main comparisons with the strongest baselines (e.g., BiLoRA, EWCLoRA), to clarify whether the reported gains are statistically significant.
  3. Validation on current-task data (Distributional mismatch). The replay validation set is drawn entirely from the current task . Could you provide analysis or empirical evidence on how often old, important tags are mistakenly rejected because they don't trigger on the new task? Since the goal is to preserve old knowledge, why not validate on a small, balanced mixture of past and current data?
  4. Consolidation without task labels (Noise amplification). The consolidation objective (Eq. 14-15) distills the short-term adapter into the backbone using only inputs x, without any task labels y. If the transient LoRA overfits the current task, this label-free distillation could consolidate spurious representations or noise. Do you have any analysis or mitigation (e.g., early stopping, consistency regularization) to prevent the backbone from absorbing noise during this write-back stage?
7 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer ri8C (1/2)

We sincerely thank the reviewers for their meticulous and constructive review. We are pleased that the reviewers recognized the innovativeness and significant practical implications of NeuralCP. We have added additional experiments and explanations to address your questions.

W1 & Q1 & Q2. Design Ablations, Calibration Sensitivity, and Statistical Variability

Ablation of key design choices. We report the sensitivity of stable rank caps and SVD rank to five tasks on Qwen3-0.6B, with all other settings held constant. The results are as follows:

ParameterValuesAverage CL Score
Stable Rank Cap rmax⁡r_{\max}8 / 16 / 32 / 6472.51 / 74.36 / 75.42 / 74.88
Tag SVD Rank4 / 8 / 16 / 2473.28 / 75.42 / 75.91 / 75.34

The defaults (tag SVD rank 8 and rmax⁡=32r_{\max}=32) are optimal or near-optimal. Every tested rank configuration exceeds BiLoRA’s 68.82, supporting robustness across the evaluated settings.

Attribution Signal Fusion. We compared product fusion with no attribution, either signal alone, and other fusion rules. The results are as follows:

Selection ScoreNoneActivation Shift OnlyGradient OnlyMaximumAdditive, α=0.5Product
Average CL Score67.1171.6570.4272.3673.2875.42

Maximum and additive fusion both outperform either signal alone, supporting the benefit of combining activation shifts with task-loss sensitivity. Product fusion favors modules strong in both signals and exceeds the best tested additive variant by 2.14 points. The additive-weight sweep gives 72.61, 73.28, and 72.94 at α=0.3\alpha=0.3, 0.50.5, and 0.70.7, respectively; product fusion remains best across these comparisons.

Calibration Sampling and Statistical Variability. The default attribution set is sampled randomly with a fixed seed. We independently varied its size while keeping the validation set at 256 samples, and the results are as follows:

Attribution Samples64128256512
Mean CL Score73.9074.8075.4275.50
Tag Acceptance Rate71%66%62%60%

Accuracy improves with sample size, with diminishing gains beyond 256 samples, while the acceptance rate decreases, showing the effect on filtering. For composition sensitivity, stratified sampling achieves 75.38 versus 75.42 for uniform random sampling. Two independent 256-sample draws select the same 11 of 12 modules, supporting selection stability. we provide standard deviations over three seeds for NeuralCP and the two baselines named by the reviewer. The results are as follows:

MethodsNeuralCPBiLoRAEWC-LoRA
Mean ± Standard Deviation75.42 ± 0.2968.82 ± 0.3567.88 ± 0.41

NeuralCP’s 6.60-point advantage over BiLoRA substantially exceeds the observed seed variability. We will include these analyses in §4.2 and add mean ± std to Table 1 in the revision.

W2 & Q3. Does current-task data validate historical tags correctly?

Historical-tag validation. We compare four validation regimes on Qwen3-0.6B over the five-task sequence, using the same training settings, attribution procedure, and subspace-projection rule. The current-task gate uses 256 validation samples drawn from the current task and disjoint from the 256-sample calibration set; the balanced gate uses validation samples from both past and current tasks. We evaluate all tasks after completing the sequence. The results are:

Validation regimeFinal CL avg.Old-task avg.
Accept all tags (no gate)70.0769.02
Validate new tags only; accept all historical tags73.8673.40
Current-task validation75.4274.07
Balanced past + current validation75.6874.30

The balanced control scores 0.26 and 0.23 points higher, a margin comparable to the seed-to-seed variation reported in W1 (±0.29), and it requires keeping a pool of past-task examples. Since NeuralCP is designed to need no stored history, we do not adopt it. Rejection is harmless either way. Existing stable directions keep constraining later gradients, so a rejected tag costs a write, not a memory.

AuthorsOpenReview ↗

Rebuttal for Reviewer ri8C (2/2)

Q4. Label-free write-back and noise amplification

Noise control during consolidation. The distillation loss does not directly use labels, but both teacher training and write-back selection are supervised: the validation gate requires a label-based task-loss gain and consistent residual directions across batches. To test the mechanism, we vary the LoRA adaptation epochs on Qwen3-0.6B over the five tasks while fixing consolidation to a single epoch. The results are as follows:

EpochsTeacher train lossTeacher validation lossAcc, w/o validationAcc, NeuralCP
10.420.5870.0775.42
30.270.6468.3275.08
50.180.7666.8174.46

Teacher training loss falls while validation loss rises, so the adapter does overfit as training lengthens. NeuralCP nonetheless degrades by only 0.96 points across this range, against 3.26 points without validation, indicating that the gate mitigates performance degradation associated with teacher overfitting.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Reviewer_VFKwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NeuralCP proposes a continual learning framework structured as a progression from transient LoRA adapters to consolidated backbone weights. The pipeline has four stages: train a LoRA module on the current task; use function-space attribution on hidden-state shifts to tag modules responsible for task-salient changes; validate tagged modules against a minimal calibration set to filter beneficial and recallable tags; consolidate validated tags into the backbone via residual distillation within a stable-plastic subspace decomposition. The method claims zero additional parameters at inference. Evaluated on eight sequential tasks across Qwen3-0.6B, 4B, and 8B, reporting average CL score improvements of 22.72, 7.39, and 8.05 points respectively.

Critical strengths

The use of replay for validation rather than training is the most interesting design choice. Standard replay methods mix old and new gradients directly. NeuralCP instead uses a small calibration set as a filter: tagged modules are tested for whether they are beneficial and consistently recallable before consolidation. This repurposing of replay from a training signal to a selection criterion is a genuinely novel idea and addresses a real failure mode of naive replay. If the ablation confirms that this validation step is essential, it is a meaningful contribution.

The zero-inference-overhead property is practically important. Many continual learning methods for LLMs accumulate adapters or maintain external memory, increasing serving cost linearly with the number of tasks. NeuralCP consolidates into the backbone and discards the LoRA module, which is the right design for deployment. The stable-plastic subspace decomposition, if well-implemented, provides a principled way to protect consolidated knowledge without freezing parameters entirely.

Critical weaknesses

The method is a complex pipeline with many interacting components, and the paper does not adequately isolate their individual contributions. The framework involves LoRA training, function-space attribution, tag selection, replay validation, residual distillation, and subspace decomposition. Each introduces hyperparameters and design choices. The ablation study needs to systematically remove each component and report the effect. Without this, it is impossible to determine whether the gains come from the full pipeline or from one or two dominant components. The 22.72-point improvement on Qwen3-0.6B is large enough to raise the question of whether the baselines are appropriately configured.

The biological framing is overextended relative to what the method actually implements. Synaptic Tagging and Capture is a specific hypothesis about protein synthesis, synaptic tags, and capture of plasticity-related proteins. NeuralCP does not implement any of these mechanisms. The LoRA module is not analogous to a synaptic tag in any mechanistic sense. The "consolidation" step is residual distillation, not systems-level memory transfer. The biological language adds rhetorical weight but does not correspond to the algorithm. The paper would be stronger if it presented the method on its engineering merits and used the biological analogy sparingly, or if it developed the analogy with greater mechanistic fidelity.

Questions for authors

What is the size of the "minimal calibration set" used for validated capture? How does performance degrade as this set shrinks to zero? At what point does the method become equivalent to unvalidated consolidation?

How is the stable-plastic subspace decomposition computed? Is it recomputed at each task boundary, or is it fixed after the first task? How sensitive is performance to the dimensionality of the stable subspace?

The 22.72-point improvement on Qwen3-0.6B is unusually large. What are the absolute CL scores of the baselines? Are the baselines using the same LoRA rank and training budget?

For the function-space attribution, what specific attribution method is used? How does it compare against simpler alternatives such as gradient magnitude or activation magnitude per module?

3 comments
AuthorsOpenReview ↗

Rebututal for Reviewer VFKw (1/2)

We sincerely thank the reviewers for their meticulous and constructive review. We are pleased that the reviewers recognized NeuralCP's innovation, efficiency, and practicality. We have added additional experiments and explanations to address your concerns.

W1 & Q3. Component ablation, and the size of the gain on Qwen3-0.6B

We address the two points in turn: first whether the gain comes from the full pipeline or from one or two components, then whether the baselines are configured comparably. Table 4 in the paper describes our ablation process for each stage using the same scheme. All runs used the Qwen3-0.6B backbone network and the five-task sequence from the original paper, and the results are shown below.

VariantsARC-ESST-2RTEMRPCQNLIMean
No Functional Attribution58.8773.4560.2172.4670.5867.11
No Memory Tagging60.1576.2356.8869.7571.1966.84
No Replay Validation63.2879.8859.3773.5574.2870.07
No Stable Plastic Decomposition61.3375.9360.6572.4168.7067.80
Complete NeuralCP67.3484.6365.3478.9680.8375.42

The results show that each component plays a crucial role, and removing any stage results in a performance penalty. The replay-validation stage, which you asked about specifically, reduces the mean score from 75.42 to 70.07 when removed. We also vary the LoRA rank while keeping all other settings fixed:

LoRA rank4816
Mean73.8575.4274.97

NeuralCP remains effective across the tested LoRA ranks, supporting its robustness to adapter capacity. Appendix A provides additional sensitivity analyses of tag SVD rank, replay-capture data size, capture and recall thresholds, and the consolidation learning rate.

The second point concerns the magnitude of the 22.72-point gain. It is measured against the unadapted base model, whose initially weak task performance leaves substantial room for supervised adaptation. Standard LoRA fine-tuning on individual tasks, without continual learning, also produces substantial improvements:

MethodARC-ESST-2RTEMRPCQNLI
Base60.6957.1153.4342.6549.62
+LoRA (single-task)83.1694.5082.3186.5293.61

The CL baselines likewise improve over the base model by 6.57–16.12 points. Our main comparison is with these adapted baselines: NeuralCP improves over the strongest baseline, BiLoRA, by 6.60 points. The final CL score is the average performance across all five tasks, evaluated after completing the task sequence. The absolute scores are:

MethodBaseMerged-LoRAO-LoRASD-LoRAEWC-LoRABiLoRANeuralCP
Avg. CL score52.7059.2764.2765.5767.8868.8275.42

All LoRA baselines and NeuralCP use the same training data, task order, and LoRA rank (r=8r=8). They also share α=16\alpha=16, dropout 0.05, learning rate 2×10−42\times10^{-4}, weight decay 0.05, batch size 16, and one LoRA adaptation epoch per task. We evaluate all methods using lm-evaluation-harness: 0-shot accuracy for the classification tasks and 5-shot exact match for GSM8K.

W2. Biological framing.

Thank you for highlighting this distinction. We will clarify the scope of the STC analogy: it motivates selective retention of useful task-induced changes at a conceptual level. In NeuralCP, LoRA provides temporary task adaptation, memory tags summarize selected functional changes, and replay validation determines their eligibility for write-back. Residual distillation then integrates the validated teacher updates into the backbone, while stable–plastic decomposition protects previously learned knowledge.

We will shorten the biological discussion, replace “closely mimics” with “conceptually inspired by,” and describe consolidation explicitly as selective write-back through residual distillation. This will make the algorithm’s engineering contribution more prominent: selective integration of task-induced changes, protection of prior knowledge, and no additional inference parameters or routing overhead after consolidation.

AuthorsOpenReview ↗

Rebuttal for reviewer VFKw (2/2)

Q1. Calibration set size

In the paper, we default to 256 samples per task, with ∣Ct∣=256|\mathcal C_t|=256 used for attribution and label statistics, and ∣Rt∣=256|\mathcal R_t|=256 used for capture validation. The samples for each task are disjoint and randomly drawn. With zero validation samples, we keep the attribution set at 256 and bypass the gate, recovering the unvalidated variant (70.07). This performance degradation also illustrates the importance of the validation process. We ablated the model's performance under different data sizes, with the results shown below:

Validation Samples0128256384
Mean70.0775.2175.4275.88

It's worth noting that the performance curve is flat from 128 to 384, therefore 256 is a safety margin, not an adjusted optimum.

Q2. Stable–plastic subspace

The subspace is maintained module-wise and updated incrementally at task boundaries. Each module stores a row-orthonormal stable basis BmB_m; its orthogonal complement defines the plastic subspace. During consolidation, BmB_m is held fixed, and gradients are projected using I−Bm⊤BmI-B_m^\top B_m.

Each accepted tag contributes a residual basis obtained from the top-8 right singular vectors of the calibration-batch residual matrix. After consolidation, accepted bases are appended to the corresponding stable bank, followed by QR orthonormalization and truncation to rmax⁡r_{\max}. Modules without accepted tags remain unchanged. Thus, the basis evolves throughout the task sequence rather than being fixed after the first task. This update requires no historical training examples. We evaluate sensitivity to the stable rank cap on Qwen3-0.6B over the five tasks, with all other settings fixed:

rmax⁡r_{\max}8163264
Avg. CL score72.5174.3675.4274.88

A small cap of 8 reduces performance by 2.91 points relative to 32, while increasing the cap from 32 to 64 changes the score by only 0.54 points. The cap controls both the retained stable directions and the occupancy-based conflict penalty. We will add these results and clarify the update schedule in §3.3.

Q4. Attribution method and simpler alternatives

Our attribution combines adapter-induced activation shifts with task-loss gradient magnitude. For each candidate module we measure how much its hidden activations shift under the adapter (activation shift) and how sensitive the task loss is to its output (gradient magnitude), normalize both across modules, and multiply them. The product requires a module to be both strongly perturbed and functionally important, rather than either alone; the score is then penalized when a module's stable subspace is already full, and the top-k modules are selected. Based on your suggestion, we ablated each alternative, the results are as follows::

Selection scoreNone (all modules)Gradient onlyActivation shift onlyAdditive, α=0.5Product (ours)
Mean67.1170.4271.6573.2875.42

Either factor alone clearly beats no selection, but the product of the two reaches 75.42 , 2.1 points above additive fusion and 3.8 above the best single factor. So the benefit comes from requiring both conditions rather than from either signal by itself.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair,

We thank you for handling our submission, and Reviewers VFKw, Ft9C, and ri8C for careful, technically engaged reviews. We submitted point-by-point responses with new experiments or analyses for every major concern. Reviewer ri8C engaged with our response and confirmed that the additional ablations and statistical analysis addressed their main concerns. Reviewers VFKw and Ft9C did not comment after our response, and no further discussion took place. We ask you to weigh this when deciding, and to assess our responses directly where they differ from the reviews.

Strengths

Conceptual reframing. Replay becomes a validation gate rather than a training signal: only changes that are both beneficial and consistently recallable are written back, which addresses a real failure mode of naive replay.

Deployment cost. Consolidating into the backbone and discarding the adapter gives zero additional inference parameters or latency, unlike adapter banks that grow with each task.

Evidence. Eight sequential language tasks on Qwen3-0.6B/4B/8B and LLaMA3.2-1B, plus ImageNet-R, with component ablations and an efficiency comparison against LoRA-Mixer.

Common concerns and how we addressed them

Component isolation and the size of the gain. Removing attribution, tagging, replay validation, or subspace decomposition each lowers the five-task average, and the validation gate alone accounts for the larger part of the gap. All baselines match data, task order, rank, and budget; the headline figure is relative to the unadapted base, and NeuralCP still leads the strongest baseline by 6.60 points. This directly addresses reviewer VFKw's W1 & Q3, and reviewer ri8C's W1.

Hyperparameter sensitivity and statistical significance. We now report sweeps over calibration size, rank caps, and fusion rules: product fusion beats additive, max, and single-signal variants, and three-seed runs place the margin over the strongest baseline far outside seed variance. This directly addresses reviewer VFKw's Q1 & Q4, and reviewer ri8C's W1 & Q2.

Whether current-task data can validate historical tags. We compare four admission rules: the balanced past-and-current gate is within seed noise and requires stored history, so we keep the gate that needs none. A historical tag decides where a write may occur, while the stable subspace protects what cannot be overwritten. This directly addresses reviewer ri8C's W2 & Q3, and reviewer Ft9C's W2 & Q2–Q3.

Novelty relative to existing components, and the biological framing. We position the contribution as validated selective consolidation, state explicitly that the residual objective is algebraically equivalent to logit matching, and present STC as conceptual inspiration rather than mechanism. This directly addresses reviewer VFKw's W2, and reviewer Ft9C's W1 & Q1.

Capacity saturation, longer sequences, and label-free write-back. On a ten-task sequence NeuralCP keeps its lead and forgets less than the baselines; stable ranks stay far below the cap with almost all input dimensions free; and performance degrades only slightly when the teacher overfits, where the ungated variant collapses. This directly addresses reviewer Ft9C's Q4, and reviewer ri8C's Q4.

We will revise the manuscript accordingly and are glad to provide any further material.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.