- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Neural Consolidation Process (NeuralCP), a biologically inspired continual learning framework designed to mitigate catastrophic forgetting while avoiding the increasing inference-time parameter overhead associated with task-specific adapters. Inspired by the Synaptic Tagging and Capture mechanism, NeuralCP separates learning into three main stages, including a temporary LoRA adapter which learns the incoming task, a replay-validation mechanism which evaluates the utility and consistency of the resulting memory tags using a small held-out calibration set, and lastly validated modules which are consolidated into the backbone through residual distillation and stable–plastic subspace decomposition.
Critical strengths
Two critical strengths exist in this paper:
-
The proposed separation between temporary adaptation, selective validation, and permanent consolidation provides an intuitive framework for balancing stability and plasticity.
-
Function-space attribution provides a meaningful mechanism for selective updating. This proposed mechanism combines hidden-state differences and loss-gradient sensitivity to identify task-relevant modules. This is a reasonable alternative to relying exclusively on parameter magnitudes or parameter-space importance measures which adopted by many previous research works.
Critical weaknesses
Two critical weaknesses of this paper I would like point out are:
-
The novelty of several proposed components requires clearer justification. Although the biological motivation is interesting, the technical components primarily combine established ideas, including LoRA-based task adaptation, activation/gradient attribution, held-out validation, knowledge distillation, and orthogonal subspace projection. In particular, Section 3.3 and Appendix E explicitly show that the proposed residual logit distillation objective is algebraically equivalent to ordinary teacher–student logit matching. Consequently, residual distillation itself does not constitute a substantially different optimization objective.
-
The replay-validation mechanism does not convincingly establish that historical knowledge is protected. A central claim is that replay validation identifies memory traces that should be retained. However, Section 3.2 defines the replay set R_t as a subset of the current task D_t, rather than using examples from previously learned tasks. Moreover, the validation residual in Eq. (8) is computed between the current short-term adapter and the current backbone, even when evaluating historical tags. This raises a fundamental concern: a historical tag may be rejected simply because it is not activated by current-task examples, despite remaining important for earlier tasks. Conversely, a tag may pass validation because it is useful for the current task without guaranteeing that consolidating the corresponding module preserves previous-task performance. The method therefore appears to validate current-task consistency more directly than historical knowledge retention. The authors should clarify how historical tags are meaningfully reactivated and validated without access to historical task samples and whether the proposed mechanism genuinely provides protection beyond the stable–plastic projection.
Questions for authors
-
What are the key methodological innovations relative to existing selective distillation and subspace-based continual learning approaches? This question is related to the first weakness, as I pointed out in the Critical Weakness box.
-
How does replay validation assess the usefulness of historical tags without using historical data? This question is related to the second weakness, as I pointed out in the Critical Weakness box.
-
How are historical memory tags used during consolidation? The distillation objective seems to use the current short-term teacher rather than the stored tag representations. Does the low-rank tag basis U_{t,m} influence consolidation directly, or is it used only to update the stable memory bank? What additional benefit does retaining historical tags provide beyond maintaining the stable subspace itself?
-
How does NeuralCP handle long-term capacity saturation? Another important question in continual learning field should be always addressed is how the method decide which old directions or memories to retain or discard, when the capacity is saturated, or the stable-subspace rank reaches r_{max}. Could the authors provide experiments involving substantially longer task sequences, together with measurements of stable ranks, plastic capacity, and forgetting?
5 comments
Rebuttal for Reviewer Ft9C (2/2)
Q3. How are historical tags used during consolidation?
Historical tags decide where the write happens, not what is written. At task the distillation target is always the current short-term teacher; a historical tag is used only as an admission credential for its module. When it passes validation against the current residual, its module stays in and may receive the write; when it fails, that module is not written.
does not affect the current consolidation; it only updates the stable bank. The target at task is the current short-term teacher, and is merged into only after consolidation finishes. is what constrains the next task's update through Eq. (11), so shapes the next write, not this one. In short, it takes no part in how the current update is computed — only in how the next one is constrained.
The extra benefit is a write-or-not check on each module, which the bank cannot provide. After orthonormalization and truncation, the bank records which directions are occupied but not which task produced them or whether they are still in use; the stored prototype is what allows a direction to be re-identified in the current forward pass. That is what lets us decide at task whether a module should receive a write, whereas projection only says where a write may not go. To test whether this adds anything beyond the bank, we run three validation policies on Qwen3-0.6B over the same five tasks, keeping the backbone, task order and projection identical and changing only the rule for admitting a tag. The results are as follows.
| Metric | No validation | New tags validated, historical not | Full (ours) |
|---|---|---|---|
| Five-task avg. | 70.07 | 73.86 | 75.42 |
Validating new tags is worth 3.79 points (70.07 → 73.86), and re-validating historical tags a further 1.56 (73.86 → 75.42).
Q4. Capacity saturation and longer task sequences
What happens when the bank is full? Each accepted tag brings up to eight directions with it, and a module's bank can hold 32. When a module is full, later directions are simply not added to it, and earlier ones are never evicted. Full modules also receive a lower score under the capacity penalty in Eq. (4), so later tasks tend to choose modules that still have room. The protected directions occupy only a small part of each module's input space, so most of it stays free for new updates.
Ten-task test. On Qwen3-0.6B we add QQP, MNLI, BoolQ, HellaSwag and STSB after the original five tasks. STSB uses Spearman correlation on a 0–100 scale, and the other tasks use accuracy. The results are as follows.
| Evaluation | Base | SD-LoRA | BiLoRA | NeuralCP |
|---|---|---|---|---|
| Five-task average after task 5 | 52.70 | 65.57 | 68.82 | 75.42 |
| Ten-task average after task 10 | 52.10 | 61.80 | 64.10 | 72.90 |
NeuralCP keeps its lead on the longer sequence as well. We evaluate the original five tasks again after task 10, so the change is not confounded by the difficulty of the added tasks. The results are as follows.
| Original five-task average | SD-LoRA | BiLoRA | NeuralCP |
|---|---|---|---|
| After task 5 | 65.57 | 68.82 | 75.42 |
| After task 10 | 60.31 | 62.95 | 71.36 |
| Change | −5.26 | −5.87 | −4.06 |
All three methods lose some accuracy, but NeuralCP loses the least. Its advantage therefore comes from retaining old tasks, not only from learning new ones faster.
Rank and remaining capacity. For modules with nonempty banks we report the average number of protected directions and the share of input dimensions the projection leaves free.
| Task boundary | 1 | 5 | 10 |
|---|---|---|---|
| Mean stable rank | 8 | 14 | 28 |
| Share of input dimensions free | ≥99% | ≥98% | ≥97% |
These figures describe the geometry of the constraints, not the functional capacity. How much information the model can actually absorb should ultimately be reflected in its performance on real-world tasks. Based on the results of the ten tasks mentioned above, NeuralCP maintaining its accuracy even with a doubling of task sequences is the best proof.
We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.
Dear reviewer:
Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.
Sincerely,
Authors
Rebuttal for Reviewer Ft9C (1/2)
We sincerely thank the reviewers for their meticulous and constructive review. We are pleased that the reviewers recognized NeuralCP's simplicity and novel selective update mechanism. We have added additional experiments and explanations to answer your questions.
W1 & Q1. Methodological contribution
The innovation lies not in the components themselves, but in our definition of stored content. EWC, O-LoRA, and LoRA-Mixer retain all updates, regardless of whether they are useful, informative, or noisy, and then process them. We consider which parts of the updates should be internalized into the backbone model and propose a standard process (rapid adaptation, attribution, validation, and consolidation) for implementation without incurring any additional inference overhead, which is crucial in real-world deployment scenarios. Regarding residual distillation, we will further refine its distinctions based on your suggestions to avoid misinterpreting it as a new loss function. To facilitate better understanding, we summarize the differences in the following diagram:
NeuralCP turns temporary task adaptation into selectively consolidated knowledge within a single backbone. By jointly deciding what to write and which directions to protect, it supports continual adaptation with zero additional inference parameters or routing overhead, without accumulating task-specific adapters. Component ablations demonstrate the value of this coordinated design. We will reinforce this section in the abstract and introduction.
W2 & Q2. Can historical tags be validated without historical data?
Eq. (8) compares the current adapter's residual against the stored prototype, so what the gate tests is whether the current task re-derives a stored direction. It does not re-measure old-task accuracy, and we will state this boundary explicitly in §3.2. Reactivation is observed, not induced. We never replay old inputs: a historical tag is re-examined only when the current task selects its module and drives the residual along the stored direction, which is the second activation STC requires for capture. A tag the current task does not re-derive is not re-captured, but it is not forgotten either — it stays in Bm and keeps constraining updates through Eq. (11). Rejection costs a write, never a memory.
Operationally, an accepted historical tag keeps its module in Scapt (Eq. 10): the tag acts as a credential for the module, while what is written is still the current residual. That asymmetry is what makes it safe to be strict. We will also stop implying that the gate demonstrates retention: retention comes from projection and from limiting the write to modules that pass the gate — the gate decides where we may write, the projection decides what cannot be overwritten. To see whether this check buys anything beyond projection, we run four tag-admission policies on Qwen3-0.6B over the five-task sequence. Training settings and the stable–plastic projection are identical in all four rows; only the rule for admitting a tag differs. The results are shown below:
The results show that historical tags can be revalidated against the current residual without any historical examples, and that a gate using historical examples offers little advantage over ours. The stored prototype supplies the historical reference and the current residual supplies the evidence, so no past data is needed on either side.