acceptodds
Under review as a conference paper at ICLR 2027

How Much to Expand, and How to Route: Dynamic-Capacity Mixture-of-Experts for Continual Learning

Abstract

Continual learning (CL) adapts large pre-trained models to a stream of new tasks without revisiting earlier data. A common architecture adds a group of LoRA experts per task, freezes old experts, and routes each token through a learned Mixture-of-Experts (MoE) router. Freezing prevents weight overwriting, but forgetting persists through routing drift as old-task tokens shift toward newer experts. Two decisions govern this framework: how much capacity each task adds, and how the accumulated experts are routed at inference. Existing methods fix the first by hand on both the expert-count and rank axes, and address the second with per-token routing alone or with trained task predictors. We propose DyCapMoE, which makes both decisions data-driven. Dynamic Expert Capacity (ECap) attaches a Hard-Concrete gate to each new expert with a sign-aware coupling that keeps gate closure monotone under softmax routing, and Dynamic Rank Capacity (RCap) applies the same mechanism to each expert's rank. Gates that reach zero are pruned at task end, so each task retains only the capacity it uses. A prototype task router provides the task-level inference prior from by-products of the forward pass without trainable parameters. Across four CL benchmarks on LVLMs and LLMs, the learned budget retains under a third of the allocated rank at equal or better final accuracy, the task router brings forgetting to near zero, and the full model attains the best final accuracy on all four benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, MoE

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1FMfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DyCapMoE, a continual learning method that aims to address the redundant parameter expansion observed in existing approaches. Specifically, DyCapMoE employs Hard-Concrete optimization to prune unnecessary experts and rank columns during task training. Additionally, it constructs a feature mean as a task anchor to provide a task prior during inference. Experimental results on four benchmarks demonstrate the effectiveness of DyCapMoE.

Critical strengths
  • Experimental results suggest that the proposed DyCapMoE achieves promising performance when task-level priors are used, particularly demonstrating nearly zero forgetting on previously learned tasks.
Critical weaknesses
  • The paper lacks a quantitative analysis demonstrating the extent of parameter redundancy in existing approaches, which could provide stronger motivation for the proposed method. Furthermore, the idea of reducing redundant parameters in continual learning has already been explored in [1]. As a result, the proposed approach appears to be another implementation of an existing idea, adapted to more modern architectures such as LLMs. Overall, the paper does not provide sufficient motivation or novel insights to clearly distinguish its contributions from prior work.

  • The observation motivating sign-aware coupling is interesting; however, the paper lacks comparisons with alternative approaches, such as post-hoc pruning and naive multiplicative gating. As a result, it remains unclear to what extent the proposed sign-aware coupling contributes to the overall performance improvements.

[1] 2019 NeurIPS Compacting, Picking and Growing for Unforgetting Continual Learning

Questions for authors

Besides the weakness shown in the above section, please also see the following questions:

Q1: What is the model size after applying the proposed method? Does DyCapMoE achieve a more compact model compared to existing approaches? More importantly, to what extent do existing methods suffer from parameter redundancy, and how effectively does the proposed method address this issue?

Q2: What is the accuracy of the task router? Given that the proposed method achieves nearly zero forgetting, does this imply that the performance on previously learned tasks remains largely unchanged when the task is correctly identified? Furthermore, since the proposed task prior is based on a simple feature mean and cosine similarity classifier, does this suggest that task-level inference is relatively straightforward? If so, how much of the reported performance improvement can be attributed to accurate task identification rather than the proposed continual learning mechanism?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1FMf (1/3)

Thank you for your careful review and for recognizing the promising performance with task-level priors, nearly zero forgetting on previously learned tasks, and the interesting observation motivating sign-aware coupling. We address each of the concerns below.


W1. Parameter redundancy and distinction from CPG

The paper lacks a quantitative analysis demonstrating the extent of parameter redundancy in existing approaches, which could provide stronger motivation for the proposed method. Furthermore, the idea of reducing redundant parameters in continual learning has already been explored in [1]. As a result, the proposed approach appears to be another implementation of an existing idea, adapted to more modern architectures such as LLMs. Overall, the paper does not provide sufficient motivation or novel insights to clearly distinguish its contributions from prior work.

(1) Retained capacity and accuracy. Thanks for your feedback. Existing incremental MoE-LoRA methods typically fix the number of new experts and their rank before each task. The capacity that a task uses varies widely: with learned retention, tasks keep from about 11% to 66% of the same allocation on CoIN (Fig. 3a in the revised paper) and from about 15% to 35% on UCIT (Fig. 3b). No single budget suits all tasks, and choosing a budget for each task by probing or sweeps would require additional training. We therefore learn the retained capacity together with the task loss. On UCIT with the task router disabled, DyCapMoE starts from the same total rank budget of 32 as IncMoE-LoRA, retains an average total rank of 7.51 (23.48%), and reaches 68.03 MFN. Fixed ranks 32 and 8 reach 66.27 and 64.70 MFN, respectively (Table 4). The saving concerns capacity retained after training; the initial allocation is unchanged. The same holds on CoIN and DCL: DyCapMoE keeps 31.15% and 18.80% of the allocated rank and reaches a higher MFN than every fixed budget (58.22 against 51.37 for the full budget on CoIN, and 67.72 against 62.83 on DCL; Table 4 in the revised paper).

We also report how much capacity can be removed from IncMoE-LoRA by pruning each task's new experts before the next task, with and without retraining for 100 steps on the current task. Later tasks inherit the pruned pool; task routing is disabled. The table below is a subset of Table e in W2.

New experts retainedMFN without retraining (%)MFN with retraining (%)
100% (Table 4 reference)66.27—
75%65.2266.18
50%61.3664.47
25%59.7462.18

We find that retraining largely restores accuracy at 75% retention, while removing more experts reduces accuracy even after retraining. The fixed-budget model already relies on most of its new experts after training, so little of the budget can be removed afterwards. When retention is learned during training, the same tasks are learned with 23.48% of the budget at a higher MFN. The redundancy is therefore in the allocation, as discussed in Q1.

(2) Difference from CPG. Pruning and growth have established precedents [1–4]. In MoE-LoRA, capacity gates also change token routing: a gate intended to suppress an expert can increase its routing probability. Under naive multiplication, reducing a gate from 1 to 0.5 changes a logit of −2 to −1 and increases the expert's softmax probability, with other logits fixed. Our sign-aware coupling prevents this reversal by gating only the positive part of the logit and excluding experts whose gate is exactly zero from the top-k selection. On UCIT without task routing, this change raises MFN by 9.66 compared with naive multiplication under the same training configuration (Table d in W2).

CPG [1] learns masks to reuse frozen old weights, expands if an accuracy target is unmet, and selects retained new capacity through repeated pruning and retraining. DyCapMoE uses standard Hard-Concrete gates to learn new expert and rank retention during task training using the task loss and sparsity penalties. Sign-aware coupling handles the interaction between expert suppression and routing during this optimization. Units with zero deterministic gates are removed at task end. Table e in W2 compares this procedure with post-hoc pruning and retraining. We have added this comparison with CPG to Sec. 2 of the revised paper.

[1] Compacting, Picking and Growing for Unforgetting Continual Learning.

[2] Lifelong Learning with Dynamically Expandable Networks.

[3] Reinforced Continual Learning.

[4] DER: Dynamically Expandable Representation for Class Incremental Learning.

AuthorsOpenReview ↗

Response to Reviewer 1FMf (2/3)

W2. Sign-aware coupling and post-hoc pruning

The observation motivating sign-aware coupling is interesting; however, the paper lacks comparisons with alternative approaches, such as post-hoc pruning and naive multiplicative gating. As a result, it remains unclear to what extent the proposed sign-aware coupling contributes to the overall performance improvements.

(1) Sign-aware coupling. Thanks for your review. Table d reports the sign-aware coupling against naive multiplication. Only the gate–logit coupling differs; the initial capacity, Hard-Concrete gates, training schedule, and the mask on closed gates are the same, and the task router is disabled.

Table d (Table 23 in the revised paper). Coupling comparison on UCIT; task router disabled.

CouplingRetained cap. (%)MFN (%)BWT (%)
Sign-aware23.4868.03-8.28
Naive multiplication28.4758.37-13.83

We find that sign-aware coupling raises MFN by 9.66 while retaining less capacity in this control.

(2) Post-hoc pruning. Table e reports post-hoc pruning, in which the 25%, 50%, or 75% of each task's new experts with the highest routing utilization on 2,000 current-task training examples are retained. Old experts remain frozen; later tasks inherit the pruned pool. In the retrain variant, the pruned model is trained for 100 more steps on the data of the current task after each prune (global batch size 64, i.e., 6,400 training examples).

Table e (Table 24 in the revised paper). Learned retention and task-end pruning on UCIT; task router disabled.

ProcedureRetained cap. (%)MFN (%)BWT (%)
IncMoE-LoRA (Table 4)10066.27-9.91
Prune7565.22-9.28
Prune + retrain7566.18-8.75
Prune5061.36-10.43
Prune + retrain5064.47-9.93
Prune2559.74-10.27
Prune + retrain2562.18-9.34
DyCapMoE (ECap+RCap)23.4868.03-8.28

Retained capacity is relative to the initial budget. Post-hoc pruning removes whole experts; DyCapMoE also prunes rank directions. DyCapMoE outperforms all tested post-hoc variants at a lower retained capacity, including those with retraining. These procedures differ in timing, selection rule, and granularity.


Q1. Deployed model size and measured over-allocation

Q1: What is the model size after applying the proposed method? Does DyCapMoE achieve a more compact model compared to existing approaches? More importantly, to what extent do existing methods suffer from parameter redundancy, and how effectively does the proposed method address this issue?

(1) Deployed size. The deployed model keeps the frozen LLaVA-1.5-7B backbone. Adapter sizes after eight CoIN tasks are:

MethodAdapter parameters (M)
Fixed-budget IncMoE-LoRA1425.0
PCLR (reported)766.7
DyCapMoE457.8

Counts exclude the backbone and include LoRA factors and router columns for IncMoE-LoRA and DyCapMoE (Appendix J); the PCLR count is the value reported in its paper, as cited in Appendix L. DyCapMoE uses 67.9% fewer adapter parameters than fixed-budget IncMoE-LoRA, and about 40% fewer than PCLR.

(2) Reduced capacity. In the tested IncMoE-LoRA setting on UCIT, removing 25% of each task's new experts lowers MFN from 66.27 to 65.22; retraining for 100 steps per task raises it to 66.18, only 0.09 below the unpruned reference. Removing 75% gives 62.18 after retraining (Table e).

(3) Effectiveness of learned retention. From the same initial rank budget of 32, DyCapMoE retains an average total rank of 7.51, 76.5% less rank, while improving MFN from 66.27 to 68.03. Fixed rank 8 reaches 64.70 (Table 4). Learning what to retain thus improves accuracy at comparable or lower retained rank. This rank reduction is obtained during training; the CoIN storage reduction above counts actual adapter parameters. After training, about 25% of the new experts can be removed with almost no loss in MFN, since the fixed-budget model already relies on the rest. When retention is learned during training, 76.5% of the allocated rank is not needed on UCIT, and about 69% on CoIN and 81% on DCL. The redundancy of the fixed budget is therefore in the allocation, and it is removed by learning the allocation during training. Post-hoc pruning also uses a preset retention rate and adds a retraining stage after every task, and with retraining its MFN remains below that of DyCapMoE at every tested rate. DyCapMoE learns the retention of each task within the same training run (+2.2% training time on CoIN, Appendix J).

The same trend appears in O-LoRA, which has no expert pool: adding RCap retains 31.56% of its initial rank budget of 32 while increasing UCIT MFN from 66.24 to 66.89, with the task router disabled (Table 14 of the submission). This supports the capacity–accuracy benefit beyond IncMoE-LoRA.

AuthorsOpenReview ↗

Response to Reviewer 1FMf (3/3)

Q2. Router accuracy and attribution of the gains

Q2: What is the accuracy of the task router? Given that the proposed method achieves nearly zero forgetting, does this imply that the performance on previously learned tasks remains largely unchanged when the task is correctly identified? Furthermore, since the proposed task prior is based on a simple feature mean and cosine similarity classifier, does this suggest that task-level inference is relatively straightforward? If so, how much of the reported performance improvement can be attributed to accurate task identification rather than the proposed continual learning mechanism?

(1) Task-router accuracy. CoIN routing accuracy is 89.9%, averaged equally across eight tasks. The simple prototype works well on average, while GQA and VQAv2 remain difficult at 58.8% and 76.4%. Predicted routing gives 65.40 MFN, compared with 66.32 for oracle task IDs (Appendix K).

(2) Forgetting under correct routing. A correctly identified old-task input reproduces its end-of-task layer outputs under Proposition A.2. The mask retains experts from the correct task and earlier tasks; their weights, router columns, and gate values are frozen. Fixed-budget experts receive the same protection with the same mask.

(3) Gains from task routing and proposed dynamic capacity learning. The table below reports fixed and learned capacity with and without the prototype task router. Initial total rank budgets match: 64 on CoIN/DCL and 32 on UCIT. The results are in Tables 4 and 21 of the revised paper.

Capacity and task routing: MFN (%).

BenchmarkCapacityWithout task routerWith task router
CoINFixed budget51.3762.58
CoINLearned budget (ours)58.2265.40
UCITFixed budget66.2773.75
UCITLearned budget (ours)68.0374.91
DCLFixed budget62.8365.16
DCLLearned budget (ours)67.7267.40

On CoIN and UCIT, the task router gives the larger gain in MFN, since it removes most of the forgetting. The fixed budget benefits in the same way, so this gain is due to task identification. The gain from capacity learning is the difference between the two budgets. Without the task router, all experts in the pool are routing candidates for old-task inputs, and the learned budget is higher than the fixed budget by 6.85 MFN on CoIN, 1.76 on UCIT and 4.89 on DCL. With the same task router, it is higher by 2.82, 1.16 and 2.24, with less stored rank. On DCL, the learned budget has almost no forgetting without the task router, so enabling it does not improve MFN.


We have revised the paper accordingly. The changes that respond to the reviews are marked in red, and we have also improved the writing, notation, tables and figures throughout the paper for clarity and precision. The method and the results reported in the submission are unchanged.

Thank you again for your time and consideration. If our clarifications and the additional results address your concerns, we would be grateful if you would consider raising your score.

Reviewer_V7ArOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses a routing drift problem in continual learning (CL) with groups of LoRA experts. Specifically, it proposes DyCapMoE, which uses a hard-concrete gate to control the activation of specific experts; and a prototype task router to accumulate expert knowledge at inference. Experiments on four CL benchmarks and multiple backbones show the effectiveness of the proposed model.

Critical strengths
  1. The model addresses an important routing drift problem in CL with MoE. The design of the model is supported by theoretical analysis.

  2. The experiments are conducted on multiple benchmarks and backbones, with results supporting the effectiveness of the model. The effect of each component in the model is thoroughly discussed in the ablation study.

Critical weaknesses
  1. The effect of the task router is highly influenced by the task boundary. In Table 19, the router performance decreases when the task boundary is blurred by 30%. In this case, can the proposed model work in class-incremental scenarios, when there's no boundary between tasks?

  2. In Table 2, the model's MFT is lower than the baselines on UCIT. Does this mean that DyCapMoE sacrifices some model capacity with the Hard-Concrete sparsity penalties?

Questions for authors

Please see the weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer V7Ar (1/2)

Thank you for your careful review and for recognizing the importance of routing drift, the theoretical support for our design, and the evaluation across benchmarks and backbones with detailed ablations. We address each of the concerns below.


W1. Task boundaries

The effect of the task router is highly influenced by the task boundary. In Table 19, the router performance decreases when the task boundary is blurred by 30%. In this case, can the proposed model work in class-incremental scenarios, when there's no boundary between tasks?

(1) Training boundaries and task IDs. Thanks for the insightful review. Our current protocol uses identifiable training stages to determine when to add and freeze expert groups. These stages can contain overlapping task data, and no task ID is provided at inference. In real-world scenarios, the only signal available is that a new batch of data has arrived. It is not known in advance whether the batch is a new task or how much capacity it requires. DyCapMoE treats the batch as a new stage and adds a group of experts, and the ECap and RCap gates decide how much of this group is kept. When the batch is close to earlier data, little new capacity is kept (about 11% for VQAv2 on CoIN, Fig. 3a in the revised paper).

(2) Effect of task overlap. With 30% overlap in Table 19, the task boundaries are already heavily blurred, since each predefined stage also contains samples from the other tasks. A test sample no longer corresponds to a single stage, and predicting its task ID is not the same problem as in the original setting. The routing accuracy is still measured against the original task of each sample, and it drops from 89.9% to 85.6% (table below). This drop has little effect on the final accuracy, since our router stays within 0.42 MFN of oracle task IDs (0.92 without overlap). When a sample is sent to another stage, that stage has also been trained on data from the same task, so the sample can often still be answered correctly. The model without the router also forgets less in this setting, because later stages contain data from earlier tasks. The router therefore has less forgetting to remove, and its gain becomes smaller (from 7.18 to 0.74 MFN). DyCapMoE still works in this setting with a smaller contribution from the router, which is stated in the limitations of the submission (Sec. 5).

Effect of task overlap on CoIN (subset of Table 19).

OverlapRouter accuracy (%)MFN without router (%)MFN with router (%)MFN with oracle task IDs (%)
0%89.958.2265.4066.32
30%85.662.2162.9563.37

(3) Class-incremental learning and streams without boundaries. Class-incremental learning with classes arriving in stages follows the same protocol. Our benchmarks are instruction-tuning tasks, and we have not evaluated class-incremental benchmarks. A continual learning stream without task boundaries would require additional mechanisms to decide when to expand the expert pool, train new expert groups, and form task anchors. One possible design is to add an expert group every fixed number of training steps, where we expect the stored parameters to grow slowly since ECap and RCap keep only the capacity that is used. Another is to monitor the entropy of the token routing during training and to add experts when it indicates that the existing ones are not sufficient. We leave this extension, including automated capacity expansion, as an interesting direction for future work.

AuthorsOpenReview ↗

Response to Reviewer V7Ar (2/2)

W2. UCIT fine-tuning accuracy and sparsity

In Table 2, the model's MFT is lower than the baselines on UCIT. Does this mean that DyCapMoE sacrifices some model capacity with the Hard-Concrete sparsity penalties?

(1) Effect of sparsity on new-task learning. Thanks for your feedback. Although some baselines in Table 2 have higher MFT, adding ECap/RCap does not lower MFT in the matched UCIT comparison. Table 22 in the revised paper reports the MFT of fixed-budget IncMoE-LoRA without sparsity gates and of ECap+RCap, in the setting of Table 4. Both start from total rank 32 and use the same backbone, optimizer, and training schedule, with the task router disabled.

Capacity and accuracy on UCIT; task router disabled.

MethodStored rankMFT (%)MFN (%)
Fixed-budget IncMoE-LoRA (no sparsity gates)3274.5366.27
DyCapMoE (ECap+RCap)7.51 (23.48%)74.9368.03

ECap+RCap retains 76.5% less rank while MFT increases from 74.53 to 74.93. Thus, the tested capacity reduction preserves average accuracy when each task is first learned. MFT is also not lowered on CoIN and DCL, where ECap+RCap has a higher MFT than the full fixed budget (Table 22). On DCL, DyCapMoE has the highest MFT in Table 2.

(2) Baselines with a higher MFT. The fixed-budget model without gates has an MFT of 74.53, which is already below the 76.89 of LoRA. The lower MFT therefore does not come from the sparsity gates, and ECap+RCap even raises it slightly to 74.93. LoRA trains one adapter on each new task without any constraint to keep earlier tasks, so it fits the current task better. MFT measures the accuracy right after each task is learned and does not reflect later forgetting.

LoRA also forgets much more, with a BWT of −18.78 and a final accuracy of 61.24, against −8.28 and 68.03 for DyCapMoE without the task router (Table 2). The other baselines with a higher MFT on UCIT (O-LoRA, SEFE and DISCO) also forget more and end with a lower MFN than DyCapMoE in the same block of Table 2 (Sec. 4.2). These baselines learn each task slightly better at first, whereas DyCapMoE forgets less and ends with a higher final accuracy.


We have revised the paper accordingly. The changes that respond to the reviews are marked in red, and we have also improved the writing, notation, tables and figures throughout the paper for clarity and precision. The method and the results reported in the submission are unchanged.

Thank you again for your time and consideration. If our clarifications and the additional results address your concerns, we would be grateful if you would consider raising your score.

Reviewer_xbE3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a novel Mixture-of-Experts approach to adapt large pre-trained models in Continual Learning settings. The main contribution lies in a dynamic mechanism that decides, after training on each new task, how to prune both the expert pool and the rank directions for each surviving expert. The method includes a prototype-based router, that select the most similar expert via cosine similarity.

Critical strengths

● I appreciate the fact that the proposed method is conceptually simple, without any unnecessary complicacy. The authors clearly identified specific issues, and the algorithm precisely attacks those aspects. ● The method is shown to be highly efficient, reducing the allocated budget while keeping similar accuracy results.

Critical weaknesses

● The authors provide ablation studies to understand the impact of their contribution in Table 4. However, since it is the main point of the paper, I think those results are not enough to prove the point. For example, the authors could consider showing the results of those experiments also on the other benchmarks, instead of limiting it only to a single one.

Questions for authors

● Can you provide results for the fixed-budget and task router configuration?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xbE3

Thank you for your careful review. We appreciate your recognition that DyCapMoE is conceptually simple, addresses the identified problems, and reduces the allocated budget while maintaining similar accuracy. We address the cross-benchmark ablations and the fixed-budget configuration with task routing below.


W1. Capacity ablations across benchmarks

The authors provide ablation studies to understand the impact of their contribution in Table 4. However, since it is the main point of the paper, I think those results are not enough to prove the point. For example, the authors could consider showing the results of those experiments also on the other benchmarks, instead of limiting it only to a single one.

Thank you for your suggestion. We report the capacity ablation of Table 4 on CoIN and DCL with the task router disabled (Tables a and b). The results follow the same trend as on UCIT. On both benchmarks, ECap+RCap keeps under a third of the allocated rank and reaches a higher MFN and a better BWT than every fixed budget, including the full one. ECap alone already gives a higher MFN than every fixed budget, and adding RCap lowers the stored rank and improves MFN further.

Table a. Capacity ablation on CoIN (task router disabled).

MethodStored rank ↓MFN ↑BWT ↑
Fixed budget (IncMoE-LoRA)
r=16r{=}161649.43-12.53
r=32r{=}323251.16-13.66
r=64r{=}646451.37-13.97
Learned budget (DyCapMoE, allocated r=64r{=}64)
+ ECap21.40 (33.43%)56.84-10.23
+ RCap (ours)19.94 (31.15%)58.22-9.26

Table b. Capacity ablation on DCL (task router disabled).

MethodStored rank ↓MFN ↑BWT ↑
Fixed budget (IncMoE-LoRA)
r=16r{=}161661.19-3.16
r=32r{=}323262.28-3.05
r=64r{=}646462.83-3.38
Learned budget (DyCapMoE, allocated r=64r{=}64)
+ ECap13.91 (21.73%)66.54-0.23
+ RCap (ours)12.03 (18.80%)67.720.43

Q1. Fixed budget with the task router

Can you provide results for the fixed-budget and task router configuration?

Thank you for your question. We report the fixed-budget model with the same prototype task router on CoIN, UCIT and DCL (Table c). The task router also reduces forgetting for the fixed-budget model, since old experts are frozen and later-task experts are masked in both cases (Prop. A.2). DyCapMoE obtains a higher MFN than the fixed-budget models on all three benchmarks while storing less rank. For DyCapMoE on DCL, forgetting is already close to zero without the task router (Sec. 4.3), so enabling it does not improve MFN.

Table c. Fixed and learned capacity with the prototype task router enabled.

BenchmarkCapacity settingRankMFN ↑BWT ↑
CoINFixed budget3261.83-0.87
CoINFixed budget6462.58-0.68
CoINDyCapMoE (ours)19.9465.40-0.30
UCITFixed budget1672.16-0.59
UCITFixed budget3273.75-0.64
UCITDyCapMoE (ours)7.5174.910.03
DCLFixed budget3264.38-0.27
DCLFixed budget6465.16-0.35
DCLDyCapMoE (ours)12.0367.400.03

We have revised the paper accordingly. The changes that respond to the reviews are marked in red, and we have also improved the writing, notation, tables and figures throughout the paper for clarity and precision. The method and the results reported in the submission are unchanged.

Thank you again for your time and consideration. If our clarifications and the additional results address your concerns, we would be grateful if you would consider raising your score.

Thank you for the detailed response. The additional capacity ablations on CoIN and DCL (Tables a, b) confirm the trend observed on UCIT, and the fixed-budget + task-router results (Table c) address my question.

These results also clarify the relative contribution of the two components: most of the forgetting reduction on CoIN and UCIT comes from the prototype task router, which benefits the fixed-budget model similarly, while learned capacity adds roughly 1–3 MFN points on top of it (with substantially fewer stored parameters). I would encourage the authors to state this decomposition explicitly in the main text, to report variance across seeds (given that some gaps, e.g. +1.16 on UCIT with the router, are small), and to include a fixed-budget baseline with the router at a rank comparable to the one retained by DyCapMoE. It would also be worth discussing the observation that reducing capacity alone improves MFN without the router, which suggests that the capacity mechanism partially acts as a mitigation of routing drift.

My concerns have been addressed, and I maintain my positive assessment.

Thank you for your time and thoughtful feedback! We are glad the additional results addressed your concerns. We will revise the manuscript with great care and appreciate your positive assessment.

Comments on the paper

1 comment
Area_Chair_ufDDOpenReview ↗

[Reminder] Please Join the Discussion and Update Your Feedback

Dear Reviewers,

The rebuttal period is coming to an end. Please read the authors' responses and update your reviews if appropriate. It would be helpful to share your thoughts as soon as possible to allow enough time for a productive discussion. Thank you!

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.