- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DyCapMoE, a continual learning method that aims to address the redundant parameter expansion observed in existing approaches. Specifically, DyCapMoE employs Hard-Concrete optimization to prune unnecessary experts and rank columns during task training. Additionally, it constructs a feature mean as a task anchor to provide a task prior during inference. Experimental results on four benchmarks demonstrate the effectiveness of DyCapMoE.
Critical strengths
- Experimental results suggest that the proposed DyCapMoE achieves promising performance when task-level priors are used, particularly demonstrating nearly zero forgetting on previously learned tasks.
Critical weaknesses
-
The paper lacks a quantitative analysis demonstrating the extent of parameter redundancy in existing approaches, which could provide stronger motivation for the proposed method. Furthermore, the idea of reducing redundant parameters in continual learning has already been explored in [1]. As a result, the proposed approach appears to be another implementation of an existing idea, adapted to more modern architectures such as LLMs. Overall, the paper does not provide sufficient motivation or novel insights to clearly distinguish its contributions from prior work.
-
The observation motivating sign-aware coupling is interesting; however, the paper lacks comparisons with alternative approaches, such as post-hoc pruning and naive multiplicative gating. As a result, it remains unclear to what extent the proposed sign-aware coupling contributes to the overall performance improvements.
[1] 2019 NeurIPS Compacting, Picking and Growing for Unforgetting Continual Learning
Questions for authors
Besides the weakness shown in the above section, please also see the following questions:
Q1: What is the model size after applying the proposed method? Does DyCapMoE achieve a more compact model compared to existing approaches? More importantly, to what extent do existing methods suffer from parameter redundancy, and how effectively does the proposed method address this issue?
Q2: What is the accuracy of the task router? Given that the proposed method achieves nearly zero forgetting, does this imply that the performance on previously learned tasks remains largely unchanged when the task is correctly identified? Furthermore, since the proposed task prior is based on a simple feature mean and cosine similarity classifier, does this suggest that task-level inference is relatively straightforward? If so, how much of the reported performance improvement can be attributed to accurate task identification rather than the proposed continual learning mechanism?
3 comments
Response to Reviewer 1FMf (2/3)
W2. Sign-aware coupling and post-hoc pruning
The observation motivating sign-aware coupling is interesting; however, the paper lacks comparisons with alternative approaches, such as post-hoc pruning and naive multiplicative gating. As a result, it remains unclear to what extent the proposed sign-aware coupling contributes to the overall performance improvements.
(1) Sign-aware coupling. Thanks for your review. Table d reports the sign-aware coupling against naive multiplication. Only the gate–logit coupling differs; the initial capacity, Hard-Concrete gates, training schedule, and the mask on closed gates are the same, and the task router is disabled.
Table d (Table 23 in the revised paper). Coupling comparison on UCIT; task router disabled.
| Coupling | Retained cap. (%) | MFN (%) | BWT (%) |
|---|---|---|---|
| Sign-aware | 23.48 | 68.03 | -8.28 |
| Naive multiplication | 28.47 | 58.37 | -13.83 |
We find that sign-aware coupling raises MFN by 9.66 while retaining less capacity in this control.
(2) Post-hoc pruning. Table e reports post-hoc pruning, in which the 25%, 50%, or 75% of each task's new experts with the highest routing utilization on 2,000 current-task training examples are retained. Old experts remain frozen; later tasks inherit the pruned pool. In the retrain variant, the pruned model is trained for 100 more steps on the data of the current task after each prune (global batch size 64, i.e., 6,400 training examples).
Table e (Table 24 in the revised paper). Learned retention and task-end pruning on UCIT; task router disabled.
| Procedure | Retained cap. (%) | MFN (%) | BWT (%) |
|---|---|---|---|
| IncMoE-LoRA (Table 4) | 100 | 66.27 | -9.91 |
| Prune | 75 | 65.22 | -9.28 |
| Prune + retrain | 75 | 66.18 | -8.75 |
| Prune | 50 | 61.36 | -10.43 |
| Prune + retrain | 50 | 64.47 | -9.93 |
| Prune | 25 | 59.74 | -10.27 |
| Prune + retrain | 25 | 62.18 | -9.34 |
| DyCapMoE (ECap+RCap) | 23.48 | 68.03 | -8.28 |
Retained capacity is relative to the initial budget. Post-hoc pruning removes whole experts; DyCapMoE also prunes rank directions. DyCapMoE outperforms all tested post-hoc variants at a lower retained capacity, including those with retraining. These procedures differ in timing, selection rule, and granularity.
Q1. Deployed model size and measured over-allocation
Q1: What is the model size after applying the proposed method? Does DyCapMoE achieve a more compact model compared to existing approaches? More importantly, to what extent do existing methods suffer from parameter redundancy, and how effectively does the proposed method address this issue?
(1) Deployed size. The deployed model keeps the frozen LLaVA-1.5-7B backbone. Adapter sizes after eight CoIN tasks are:
| Method | Adapter parameters (M) |
|---|---|
| Fixed-budget IncMoE-LoRA | 1425.0 |
| PCLR (reported) | 766.7 |
| DyCapMoE | 457.8 |
Counts exclude the backbone and include LoRA factors and router columns for IncMoE-LoRA and DyCapMoE (Appendix J); the PCLR count is the value reported in its paper, as cited in Appendix L. DyCapMoE uses 67.9% fewer adapter parameters than fixed-budget IncMoE-LoRA, and about 40% fewer than PCLR.
(2) Reduced capacity. In the tested IncMoE-LoRA setting on UCIT, removing 25% of each task's new experts lowers MFN from 66.27 to 65.22; retraining for 100 steps per task raises it to 66.18, only 0.09 below the unpruned reference. Removing 75% gives 62.18 after retraining (Table e).
(3) Effectiveness of learned retention. From the same initial rank budget of 32, DyCapMoE retains an average total rank of 7.51, 76.5% less rank, while improving MFN from 66.27 to 68.03. Fixed rank 8 reaches 64.70 (Table 4). Learning what to retain thus improves accuracy at comparable or lower retained rank. This rank reduction is obtained during training; the CoIN storage reduction above counts actual adapter parameters. After training, about 25% of the new experts can be removed with almost no loss in MFN, since the fixed-budget model already relies on the rest. When retention is learned during training, 76.5% of the allocated rank is not needed on UCIT, and about 69% on CoIN and 81% on DCL. The redundancy of the fixed budget is therefore in the allocation, and it is removed by learning the allocation during training. Post-hoc pruning also uses a preset retention rate and adds a retraining stage after every task, and with retraining its MFN remains below that of DyCapMoE at every tested rate. DyCapMoE learns the retention of each task within the same training run (+2.2% training time on CoIN, Appendix J).
The same trend appears in O-LoRA, which has no expert pool: adding RCap retains 31.56% of its initial rank budget of 32 while increasing UCIT MFN from 66.24 to 66.89, with the task router disabled (Table 14 of the submission). This supports the capacity–accuracy benefit beyond IncMoE-LoRA.
Response to Reviewer 1FMf (3/3)
Q2. Router accuracy and attribution of the gains
Q2: What is the accuracy of the task router? Given that the proposed method achieves nearly zero forgetting, does this imply that the performance on previously learned tasks remains largely unchanged when the task is correctly identified? Furthermore, since the proposed task prior is based on a simple feature mean and cosine similarity classifier, does this suggest that task-level inference is relatively straightforward? If so, how much of the reported performance improvement can be attributed to accurate task identification rather than the proposed continual learning mechanism?
(1) Task-router accuracy. CoIN routing accuracy is 89.9%, averaged equally across eight tasks. The simple prototype works well on average, while GQA and VQAv2 remain difficult at 58.8% and 76.4%. Predicted routing gives 65.40 MFN, compared with 66.32 for oracle task IDs (Appendix K).
(2) Forgetting under correct routing. A correctly identified old-task input reproduces its end-of-task layer outputs under Proposition A.2. The mask retains experts from the correct task and earlier tasks; their weights, router columns, and gate values are frozen. Fixed-budget experts receive the same protection with the same mask.
(3) Gains from task routing and proposed dynamic capacity learning. The table below reports fixed and learned capacity with and without the prototype task router. Initial total rank budgets match: 64 on CoIN/DCL and 32 on UCIT. The results are in Tables 4 and 21 of the revised paper.
Capacity and task routing: MFN (%).
| Benchmark | Capacity | Without task router | With task router |
|---|---|---|---|
| CoIN | Fixed budget | 51.37 | 62.58 |
| CoIN | Learned budget (ours) | 58.22 | 65.40 |
| UCIT | Fixed budget | 66.27 | 73.75 |
| UCIT | Learned budget (ours) | 68.03 | 74.91 |
| DCL | Fixed budget | 62.83 | 65.16 |
| DCL | Learned budget (ours) | 67.72 | 67.40 |
On CoIN and UCIT, the task router gives the larger gain in MFN, since it removes most of the forgetting. The fixed budget benefits in the same way, so this gain is due to task identification. The gain from capacity learning is the difference between the two budgets. Without the task router, all experts in the pool are routing candidates for old-task inputs, and the learned budget is higher than the fixed budget by 6.85 MFN on CoIN, 1.76 on UCIT and 4.89 on DCL. With the same task router, it is higher by 2.82, 1.16 and 2.24, with less stored rank. On DCL, the learned budget has almost no forgetting without the task router, so enabling it does not improve MFN.
We have revised the paper accordingly. The changes that respond to the reviews are marked in red, and we have also improved the writing, notation, tables and figures throughout the paper for clarity and precision. The method and the results reported in the submission are unchanged.
Thank you again for your time and consideration. If our clarifications and the additional results address your concerns, we would be grateful if you would consider raising your score.
Response to Reviewer 1FMf (1/3)
Thank you for your careful review and for recognizing the promising performance with task-level priors, nearly zero forgetting on previously learned tasks, and the interesting observation motivating sign-aware coupling. We address each of the concerns below.
W1. Parameter redundancy and distinction from CPG
(1) Retained capacity and accuracy. Thanks for your feedback. Existing incremental MoE-LoRA methods typically fix the number of new experts and their rank before each task. The capacity that a task uses varies widely: with learned retention, tasks keep from about 11% to 66% of the same allocation on CoIN (Fig. 3a in the revised paper) and from about 15% to 35% on UCIT (Fig. 3b). No single budget suits all tasks, and choosing a budget for each task by probing or sweeps would require additional training. We therefore learn the retained capacity together with the task loss. On UCIT with the task router disabled, DyCapMoE starts from the same total rank budget of 32 as IncMoE-LoRA, retains an average total rank of 7.51 (23.48%), and reaches 68.03 MFN. Fixed ranks 32 and 8 reach 66.27 and 64.70 MFN, respectively (Table 4). The saving concerns capacity retained after training; the initial allocation is unchanged. The same holds on CoIN and DCL: DyCapMoE keeps 31.15% and 18.80% of the allocated rank and reaches a higher MFN than every fixed budget (58.22 against 51.37 for the full budget on CoIN, and 67.72 against 62.83 on DCL; Table 4 in the revised paper).
We also report how much capacity can be removed from IncMoE-LoRA by pruning each task's new experts before the next task, with and without retraining for 100 steps on the current task. Later tasks inherit the pruned pool; task routing is disabled. The table below is a subset of Table e in W2.
We find that retraining largely restores accuracy at 75% retention, while removing more experts reduces accuracy even after retraining. The fixed-budget model already relies on most of its new experts after training, so little of the budget can be removed afterwards. When retention is learned during training, the same tasks are learned with 23.48% of the budget at a higher MFN. The redundancy is therefore in the allocation, as discussed in Q1.
(2) Difference from CPG. Pruning and growth have established precedents [1–4]. In MoE-LoRA, capacity gates also change token routing: a gate intended to suppress an expert can increase its routing probability. Under naive multiplication, reducing a gate from 1 to 0.5 changes a logit of −2 to −1 and increases the expert's softmax probability, with other logits fixed. Our sign-aware coupling prevents this reversal by gating only the positive part of the logit and excluding experts whose gate is exactly zero from the top-k selection. On UCIT without task routing, this change raises MFN by 9.66 compared with naive multiplication under the same training configuration (Table d in W2).
CPG [1] learns masks to reuse frozen old weights, expands if an accuracy target is unmet, and selects retained new capacity through repeated pruning and retraining. DyCapMoE uses standard Hard-Concrete gates to learn new expert and rank retention during task training using the task loss and sparsity penalties. Sign-aware coupling handles the interaction between expert suppression and routing during this optimization. Units with zero deterministic gates are removed at task end. Table e in W2 compares this procedure with post-hoc pruning and retraining. We have added this comparison with CPG to Sec. 2 of the revised paper.
[1] Compacting, Picking and Growing for Unforgetting Continual Learning.
[2] Lifelong Learning with Dynamically Expandable Networks.
[3] Reinforced Continual Learning.
[4] DER: Dynamically Expandable Representation for Class Incremental Learning.