acceptodds
Under review as a conference paper at ICLR 2027

MoCE: Mixture of Communitified Experts for Continual Vision-Language Learning under Task-Complexity Flux

Abstract

Class-incremental learning (CIL) with vision-language models is often formulated as a balanced sequence of class increments, but deployed systems rarely receive such regular updates. Some increments contain only a few closely related classes, while others introduce many visually diverse or semantically distant classes. We refer to this variation as uneven class-incremental complexity. It exposes a weakness of standard sparse Mixture-of-Experts (MoE) adapters: a fixed top- router activates the same number of isolated experts for every input and every increment, even when the amount of expert knowledge needed by the current update changes. We propose MoCE, a Mixture of Communitified Experts for CLIP-based class-incremental learning. MoCE decouples the number of executed experts from the number of involved experts. It represents adapters as nodes in an input-conditioned learnable graph, lets each node exchange messages with its neighbors, and then selects sparse community anchors. A selected anchor is therefore not a single isolated expert; it is the representative of a graph community whose state has already absorbed information from related experts. As a result, MoCE can execute a small anchor set for efficiency while involving a variable number of neighboring experts in routing, and it can optionally increase the anchor budget for increments with more classes. This gives the router a flexible capacity response to uneven task complexity without dense expert execution. Experiments under even and uneven CIL schedules show that MoCE improves over standard MoE adapters, with especially consistent gains when the number of arriving classes varies across increments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mixture of expert, graph neural network, fluid continual learning, uneven complexity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rzrXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates class-incremental learning for vision-language models under uneven task complexity, where the number and diversity of newly introduced classes may vary across increments. The authors identify a limitation of conventional fixed top-kk sparse Mixture-of-Experts (MoE) routing, which constrains the number of executed experts regardless of the complexity of the incoming task. To address this issue, they propose MoCE, a Mixture of Communitified Experts framework that decouples the set of experts participating in routing from the subset ultimately selected for execution. Specifically, MoCE models adapter experts as nodes in an input-conditioned graph, performs lightweight message passing among neighboring experts, and uses the resulting graph-enhanced representations to select sparse community anchors. The framework further presents an Adaptive Anchor Budget, allowing increments with larger numbers of classes to activate more anchor experts while maintaining sparse computation. Experiments on CIFAR100 and TinyImageNet-100 under both balanced and uneven class-incremental schedules demonstrate consistent improvements over standard MoE adapters and several continual-learning baselines.

Critical strengths
  1. The paper is well written, clearly organized, and easy to follow.

  2. The proposed method is conceptually simple while achieving substantial and consistent performance improvements across both balanced and uneven class-incremental learning settings.

  3. The distinction between executed experts and experts involved in the routing process is a novel and well-motivated design choice. This formulation allows inter-expert communication to influence routing decisions without requiring dense expert execution, thereby providing greater routing flexibility while preserving sparse computation.

Critical weaknesses
  1. The formulation of the input-conditioned adjacency matrix is not sufficiently motivated. In particular, all N2N^2 pairwise expert relationships are predicted directly from the input representation hbh_b, without explicitly conditioning on the identities or representations of the two experts whose relationship is being estimated. As a result, it is unclear whether the learned adjacency matrix captures meaningful inter-expert relationships.

  2. The motivation for MoCE is less compelling once the number of newly introduced classes is explicitly available as prior information through the Adaptive Anchor Budget. If CtC_t is already used to increase the number of executed experts for more complex increments, a natural and considerably simpler baseline would be a standard MoE with an adaptive top-kk routing budget. The current experiments do not clearly establish whether graph-based community routing provides benefits beyond this simpler form of adaptive expert allocation.

  3. Despite the substantial performance gains obtained by modifying the routing mechanism, the paper provides limited mechanistic analysis of why the proposed routing strategy is effective. More quantitative and qualitative analyses of expert specialization, routing diversity, community structure, expert co-activation, or the relationship between routing patterns and task complexity would substantially strengthen the claim that the observed improvements arise from meaningful community-based expert coordination.

  4. Relatedly, it is difficult to disentangle the contribution of graph-based routing from the effect of simply involving or executing a larger number of experts through the Adaptive Anchor Budget. A key ablation comparing at least (1) standard MoE with fixed top-kk, (2) standard MoE with adaptive top-kk, (3) MoCE with fixed kk, and (4) MoCE with adaptive kk is missing. Without such a comparison, the incremental benefit of the proposed community-based routing mechanism remains unclear.

  5. The proposed MoCE formulation appears fairly general and is not inherently specific to class-incremental learning. The core idea, performing graph-based inter-expert communication before sparse expert selection, could seemingly be applied to standard MoE architectures in non-continual settings as well. Apart from the Adaptive Anchor Budget, the paper does not clearly identify which aspects of the method are specifically motivated by the challenges of class-incremental learning or why those challenges particularly require the proposed community-based routing mechanism.

Questions for authors

N/A

2 comments

Comment 1 response: We appreciate the reviewer's observation that the adjacency matrix is predicted directly from the input representation without explicitly conditioning each edge on expert-specific embeddings. Our formulation is motivated by the assumption that different inputs may require different organizations of expert information. A fixed expert topology may not capture such input-dependent variations. In the current implementation, the adjacency prediction module maps the 768-dimensional input representation through a hidden layer of size 512 to N2N^2 adjacency logits. Thus, the adjacency predictor generates an input-conditioned matrix in which different output coordinates correspond to different expert pairs.

We acknowledge that explicitly incorporating the representations of experts ii and jj into adjacency prediction is a reasonable alternative. Such a formulation could potentially capture complementary aspects of expert relationships. Our current approach instead prioritizes direct, input-conditioned prediction of expert connectivity, allowing the graph organization to respond to different samples without requiring a fixed topology. We will clarify this design motivation and explicitly distinguish the current input-conditioned adjacency predictor from an expert-embedding-conditioned alternative. We clarified the motivation and formulation of input-conditioned expert relationships in Section 3.3, including how we construct expert nodes and pairwise adjacency logits. Additional architecture and adjacency-selection details are provided in Appendix A.2.

Comment 2 response: We agree that standard MoE with an adaptive top-k budget is an essential baseline for determining whether graph-based routing contributes improvements beyond adaptive expert allocation. To address this concern, we implemented an additional standard MoE baseline (Section 4.5, Table 4). The corresponding adaptive-budget policy is described in Appendix A.2.

Comment 3 response: We thank the reviewer for suggesting further investigation into the routing mechanism. We analyzed expert selection, class-conditioned utilization, routing diversity, graph adjacency, and their relationships with incremental task size. (1) Expert selection and routing diversity. Using the N=16 MoCE adaptive-routing statistics, we examined how different inputs were assigned to anchor experts. This reveals an important distinction between the nominal top-k budget and the number of experts receiving substantial weight. (2) Expert specialization and input-level relationships. We also conducted a separate input-level analysis with a completed N=8 MoCE model. See Section 4.6 (Figure 2), this example illustrates the coexistence of shared and input-specific expert selections. Additional class-conditioned routing analysis (Appendix A.6.2) showed substantial variation in expert preferences across classes. (3) Graph connectivity. We also examined the stage-level adjacency statistics (Appendix A.6.1). The results confirm that the graph incorporates off-diagonal expert relationships. We included these observations and explicitly distinguish descriptive routing evidence from causal claims about why graph message passing improves accuracy.

Comment 4 response: We agree that this ablation is necessary. We conducted the requested four-way experiment and mentioned the findings in comment 2.

Comment 5 response: We appreciate the reviewer raising this concern. We agree, and we might apply it to more general learning tasks, such as multi-task and cross-domain learning, and even reinforcement learning in future work. However, our work is motivated by a particular challenge in continual learning: successive class increments can introduce substantially different demands on the organization and use of expert knowledge. Conventional class-incremental learning experiments often divide classes into equally sized increments. In practical settings, however, the number and composition of incoming classes may vary between stages. Our design therefore considers two complementary mechanisms. First, input-conditioned graph construction allows the organization of expert relationships to vary with the incoming samples. Second, the adaptive anchor budget permits the number of selected experts to depend on the size of the corresponding class increment. We also evaluated MoCE under both even and uneven class-incremental settings. While the observed difference is larger in the uneven setting, these experiments differ in the number of stages and do not isolate task-size heterogeneity. The original formal comparisons also combine graph routing with different execution-budget behavior. The empirical results demonstrate MoCE's applicability to uneven class-incremental learning, while the additional four-way ablation provides a more direct examination of the contributions of graph-based coordination and adaptive execution.

AuthorsOpenReview ↗

Follow-Up on Our Response to Your Comments

We would like to kindly follow up on our response to your comments. We have addressed the questions and concerns raised in your review and hope that our clarifications have been helpful.

If you have any remaining concerns or questions, please let us know. We would be happy to provide further clarification or address them promptly.

Thank you again for your time and thoughtful feedback.

Reviewer_w5v7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies uneven class-incremental learning, where different incremental stages introduce varying numbers of new classes. To address this challenge, the authors propose MoCE, a graph-based routing mechanism for sparse Mixture-of-Experts (MoE). MoCE constructs an input-dependent graph with experts as nodes, allowing neighboring experts to exchange information before selecting a subset of experts for execution. This aims to improve expert selection while adapting execution capacity to the number of newly introduced classes and maintaining sparse computation. Extensive experiments on two datasets, CIFAR-100 and TinyImageNet-100, demonstrate improvements over standard sparse MoE and several continual-learning baselines under both even and uneven class-incremental settings. Overall, the paper is well written and easy to follow, with generally clear mathematical formulations and notation, although its organization could be improved.

The paper addresses a practical problem with an interesting approach and encouraging empirical results. However, unclear descriptions of experimental settings and hyperparameters, along with insufficient ablations to isolate the contributions of the proposed components, limit the strength of the empirical evidence. Considering the author resolved my concerns, I increase the score from Weak Accept to Clear Accept.

Critical strengths

S1. Practical and relevant problem setting. The paper introduces and addresses uneven class-incremental learning, a practical and relevant setting where different incremental stages introduce varying numbers of new classes.

S2. Interesting and effective methodology. The proposed graph-based routing mechanism enables information exchange among experts while maintaining sparse execution, distinguishing between experts involved in routing decisions and those selected for execution. This provides a promising approach to improving expert selection and adapting execution capacity to varying task sizes.

S3. Clear mathematical formulation. The proposed method, MoCE, is presented with clear mathematical formulations and well-defined notations, making the overall methodology straightforward to understand and follow.

S4. Convincing empirical results. The authors provide extensive experiments on two datasets under both even and uneven class-incremental settings, demonstrating consistent improvements over standard MoE and several continual-learning baselines. The reported computational cost also suggests that the proposed method is relatively efficient.

Critical weaknesses

W1. Unclear presentation of experimental settings and ablations. Several important experimental settings and hyperparameters are not clearly introduced or sufficiently explained in the main paper (e.g., the identity-bias coefficient α\alpha in Section 4.6, the top-mm selection in Section 3, and the uneven task schedules U1–U8), making the experiments difficult to follow without referring to the appendix. Additionally, the four uneven 20-step schedules (U5–U8) are unclear, as each is described with only five increments in Appendix A.2. The authors should address these issues by improving the organization of the paper, briefly summarizing key experimental settings and ablations in the main text, and providing appropriate references to the appendix.

W2. Limited ablations of the main components. Although the appendix provides several hyperparameter ablations, the individual contributions of graph-based routing and adaptive anchor execution are not clearly isolated. In particular, the standard MoE baseline uses a fixed number of executed experts, while MoCE can adapt this number based on the task size. Additional comparisons between MoE with adaptive execution and MoCE with fixed execution would help clarify whether the improvements arise from graph-based routing, adaptive execution, or their combination.

W3. Limited evidence of meaningful expert relationships. Although the proposed graph enables information exchange among experts by retaining the top-mm connections for each expert before selecting a subset for execution, it remains unclear whether the learned connections capture meaningful relationships between expert specializations or simply increase the capacity of the routing mechanism. Additional analysis, comparisons with a stronger router without graph message passing, or further discussion would help justify the proposed design.

Questions for authors

Q1. Could the authors clarify some experimental settings and hyperparameters, particularly the identity-bias coefficient α\alpha, top-mm neighbor selection, and uneven task schedules U1–U8? Specifically, why are U5–U8 described as 20-step schedules when only five increments are listed in Appendix A.2?

Q2. Although the appendix provides several hyperparameter ablations and routing analyses, could the authors further isolate the contributions of graph-based routing and adaptive anchor execution, as these represent two distinct improvements over standard sparse MoE? In particular, comparisons between MoE with adaptive top-kk execution and MoCE with fixed top-kk execution would help clarify whether the reported improvements arise from graph-based routing, adaptive execution, or their combination.

Q3. Could the authors provide additional evidence or further discussion on whether the learned graph connections capture meaningful relationships between expert specializations, rather than simply increasing the capacity of the routing mechanism? For example, a comparison with a stronger, parameter-matched router without graph message passing could help clarify the benefit of the proposed graph-based design.

6 comments

W1 / Q1 response: We appreciate the reviewer's observation that several experimental configurations were insufficiently explained in the main paper. We clarify the key parameters as follows.

(1) Identity-mixing coefficient. The coefficient controls the contribution of the identity matrix to the final adjacency matrix: A=I+(1-alpha)Ã. In the analyzed configuration (alpha= 0.7), this preserves a substantial self-connection contribution while allowing off-diagonal information exchange.

(2) Graph-neighbor selection. The implementation first constructs an input-conditioned adjacency matrix, symmetrizes it, and applies row-wise softmax normalization. When graph_adj_top_m is smaller than N, the highest-ranked m entries per row are retained and renormalized. Diagonal entries compete with off-diagonal entries during this operation; self-connections are subsequently incorporated through identity mixing. The formal uneven-schedule experiments use N=8 and graph_adj_top_m=5. The additional N=16 ablation uses graph_adj_top_m=9.

(3) Adaptive anchor budget. The number of permitted anchor experts depends on the number of classes in the corresponding increment: kt=clip(2+[Ct−10]/5,1,N)kt=clip(2+[Ct-10]/5, 1, N)

(4) Here, increment refers to the step number—that is, the number of tasks per class in an even setting. For instance, in a 20-step setting (20 increments), each class contains 20 tasks; given 100 total tasks, this corresponds to 5 classes. The term "increment" is inherited from the even-task setting in class-incremental learning (CIL), where every class has an equal number of tasks. However, in uneven settings, the number of tasks differs across classes, meaning the increment takes on multiple values within a single experiment. We retain the term "increment" here solely to maintain consistent terminology with the even-task setting. We have updated the term in Section 4.2.

W2 / Q2 response: We agree that it is important to distinguish the contribution of graph-based expert coordination from the effect of increasing the execution budget. We conducted an additional four-way ablation using (N=16), ViT-B/16, the uneven CIFAR-100 schedule, and three random seeds. The four configurations are: (1) Standard MoE with fixed (k=2). (2) Standard MoE with adaptive k. (3) MoCE with fixed (k=2). (4) MoCE with adaptive k. We added the requested four-way comparison in Section 4.5 (Table 4). These results indicate that the strongest final-accuracy gains arise when graph-based coordination and adaptive execution are combined.

W3 / Q3 response: Analysis of Learned Expert Relationships. We thank the reviewer for requesting additional evidence of the relationships learned between experts. To examine the behavior of the expert graph, We added an input-level expert-relationship analysis in Section 4.6 (Figure 2). Additional evidence is provided in Appendix A.6.1 (Figure 15) for stage-level adjacency, Appendix A.6.2 (Figure 16) for class-conditioned expert selection, Appendix A.6.3 (Figure 17) for effective routing sparsity, and Appendix A.6.4 (Table 7) for stage-level routing diversity.

we analyzed input-level anchor selections and learned expert connections using a separately trained (N=8) MoCE model. For this analysis, one CIFAR-100 test image was selected from each of the 100 classes, producing 4,950 unique image pairs. Across these pairs, all shared at least one selected expert in four or more of the 12 vision layers, with a median overlap of eight layers. Additionally, 1,410 pairs exhibited at least one layer in which the two inputs selected distinct experts that were connected as strongest learned neighbors. As an illustrative example, we examined a streetcar–tractor image pair. The two inputs selected the same expert in seven of the 12 layers. At layer 8, they selected different experts, expert 6 and expert 7, which were connected through a learned graph edge with weights 0.27 and 0.29. This example demonstrates that their graph representations can incorporate information from one another despite their distinct anchor selections.

We also examined class-conditioned expert utilization. For example, oak tree and tulip exhibited highly overlapping selection profiles, whereas worm displayed a substantially different distribution of selected experts. These findings provide descriptive evidence that the routing mechanism is input- and class-dependent.

The available (N=16) routing logs also show highly concentrated gating weights, suggesting that the adaptive budget does not necessarily translate into multiple experts receiving substantial execution weight. We will therefore present these analyses as evidence of routing and expert-relationship behavior.

Thank you for your response and the additional experiments. I found parts of the rebuttal hard to follow and had difficulty locating the changes in the revised manuscript. Please highlight the changes in the revised PDF so reviewers can identify them quickly.

Please also summarize the key results and takeaways directly in your OpenReview reply, using Markdown for readability. In particular, please include the following:

  1. W1/Q1 — CIL settings. Summarize the uniform and non-uniform settings, the key hyperparameters, and the main results. Please show how the five listed increments map onto the 20 steps for one of U5–U8.
  2. W2/Q2 — Four-way ablation. Report the results for all four configurations with insights here.
  3. W3/Q3 — Expert relationships. Briefly state the main finding from the new analysis and what it tells us about the learned relationships between experts, rather than pointing to different parts of the (revised) manuscript.

Thank you for your suggestions. We have highlighted the revised sections in blue.

W1/Q1. CIL settings

A class is a group of tasks. A uniform step is the number of tasks in every class. The 10-step setting has 10 classes of 10 tasks. The 20-step setting has 5 classes of 20 tasks, [20,20,20,20,20][20,20,20,20,20]: tasks 1--20, 21--40, 41--60, 61--80, and 81--100. The 50-step setting has 2 classes of 50 tasks, [50,50][50,50].

Uneven schedules have 10 classes, so they use the same 10-class name. Their class numbers differ. The main schedule is [28,19,13,11,8,6,5,4,3,3][28,19,13,11,8,6,5,4,3,3].

All runs use ViT-B/16, learning rate 0.0010.001, weight decay 00, batch size 3232, and 15 epochs per class. Graph runs use 1 GNN layer, hidden size 512512, λ=100\lambda=100, and α=0.7\alpha=0.7. The budget is k=2k=2, or k=clip(2+⌊(Ct−10)/5⌋,1,N)k=\mathrm{clip}(2+\lfloor(C_t-10)/5\rfloor,1,N), where CtC_t is the current class number. Evaluation uses that class's router.

On Tiny-ImageNet with N=8N=8, uniform 20-step gives MoE 81.99/87.6781.99/87.67 and MoCE 86.50/90.9486.50/90.94 (final/average). Uniform 50-step gives MoE 85.22/88.7585.22/88.75 and MoCE 87.76/90.1187.76/90.11. On the uneven class numbers above, CIFAR-100 gives MoE 76.40/83.2776.40/83.27 and MoCE 80.59/86.8680.59/86.86, and Tiny-ImageNet gives MoE 79.88/85.0679.88/85.06 and MoCE 83.03/86.9683.03/86.96.

W2/Q2. Four-way ablation

This is a separate CIFAR-100 study: N=16N=16, seeds 42/43/44, class numbers [28,19,13,11,8,6,5,4,3,3][28,19,13,11,8,6,5,4,3,3]. It crosses the graph with the budget. Means over three seeds:

ConfigurationFinalAverage
MoE, k=276.5082.90
MoE, adaptive77.7884.18
MoCE, k=276.2683.79
MoCE, adaptive80.6884.99

The adaptive budget raises final accuracy in both models. At k=2k=2, the graph changes final accuracy by +0.50+0.50, −1.33-1.33, and +0.13+0.13. With the adaptive budget, the changes are +2.72+2.72, +1.87+1.87, and +4.12+4.12, and forgetting drops by 2.32.3--2.62.6 points on every seed. The final-accuracy interaction is positive on all three seeds, mean +3.14+3.14.

W3/Q3. Expert relationships

On N=16N=16 MoCE, seed 42, layer 11, the largest gate holds at least 99.5%99.5\% of the weight, and from the fifth class on each saved image has one gate above 0.0010.001. Oak tree and tulip have selection cosine 0.9990.999: both use expert 12 on nearly every image and expert 15 on 70%70\% and 76%76\%. Worm, cosine 00 with oak tree, uses expert 6 on 62%62\% and expert 10 on 28%28\%. The self-edge remains 0.7000.700 because A=0.7I+0.3AA=0.7I+0.3A, and the mean off-diagonal weight remains 0.0200.020. Streetcar and tractor share an expert on 7 of 12 layers; at layer 8 they use experts 7 and 6, edge 0.270.27. The same neighbor pattern occurs in 50 of 200 same-superclass pairs.

Reviewer_GkYgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoCE, a graph-enhanced mixture-of-experts framework for class-incremental vision-language learning under uneven class-incremental complexity. MoCE first decouples the number of executed experts from the number of involved experts and constructs an input-conditioned graph to exchange routing information among experts. It then selects sparse community anchors by using residual logit mixing and can optionally increase the anchor budget for increments with more classes. Experiments on CIFAR100 and TinyImageNet-100 show accuracy improvements over standard MoE under both even and uneven class-incremental schedules.

Critical strengths
  1. The paper is generally well structured, and the overall method is easy to follow.
  2. The paper provides extensive evaluations across multiple class-incremental schedules, expert configurations, and two CLIP backbones.
Critical weaknesses
  1. The claimed flexible expert involvement is contradicted by the default configuration. With N=8 and m=8 (Appendix A.3), every expert retains all graph neighbors, so the involved routing set cannot expand or contract as claimed.
  2. The main comparisons in Table 2 and Table 5 are unfair, as the standard MoE uses a fixed k=2, while MoCE uses a task-complexity-adaptive anchor budget that can execute more experts (e.g., k=8 when C_t=46 in U1 with N=8, according to Appendix A.2 and A.3), making it impossible to attribute the reported gains specifically to graph-community routing.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Clarification on Graph-Neighbor Selection and Flexible Expert Involvement

Comment 1 response: We thank the reviewer for identifying an ambiguity in our description of the graph-neighbor configuration. We agree that the distinction between the total number of experts, retained graph neighbors, and executed anchor experts requires clarification. In our implementation, these quantities are controlled separately: N denotes the total number of experts and graph nodes. m is graph_adj_top_m, and it determines the maximum number of adjacency entries retained per row before identity mixing. k denotes the maximum number of anchor experts selected for execution. In the formal uneven-schedule experiments with N=8, the stored configuration uses graph_adj_top_m=5, rather than m=8. In our additional N=16 ablation, this parameter is set to 9. Specifically, the adjacency matrix is constructed through input-conditioned prediction, symmetrization, and row-wise normalization. The top-m operation retains the highest-ranked adjacency entries, including diagonal entries as candidates. The final matrix incorporates an identity-mixing term:A=I+(1−alpha)A~A=I+(1-alpha)Ã, where =0.7 . This produces input-conditioned weighted expert relationships while preserving a fixed total expert pool. We acknowledge that when m = N, the top-m operation does not sparsify the graph. This occurs in some experimental configurations and should be distinguished from the sparse-neighborhood settings. Our intended flexibility concerns the input-dependent organization and weighting of expert relationships, rather than expansion or contraction of the total expert pool or the fixed per-row neighbor budget. We revised the corresponding descriptions in Section 3 and Appendix A.2 to clarify these distinctions and correct the inconsistent notation.

Comment 2 response: We agree that comparing standard MoE with fixed k=2 against MoCE with an adaptive execution budget does not independently establish the contribution of graph-based expert coordination. To address this concern, we conducted an additional four-way ablation in Section 4.5 (Table 4) comparing standard MoE and MoCE under both fixed and adaptive execution budgets. The experiments use CIFAR-100, ViT-B/16, N=16 experts, the uneven incremental schedule [28,19,13,11,8,6,5,4,3,3], and three random seeds (42, 43, and 44). The adaptive budget is determined by the number of classes in the corresponding task: $ kt=clip(2+[Ct-10]/5, 1, N)$. Adaptive execution improves final accuracy in both architectures. For standard MoE, final accuracy increases from 76.50% to 77.78%. When (k=2) is fixed, MoCE does not consistently outperform standard MoE. Its mean final accuracy is 76.26%, compared with 76.50% for standard MoE. Under the same adaptive budget, MoCE achieves 80.68% final accuracy, exceeding adaptive MoE by 2.90 percentage points, while reducing forgetting from 9.70 to 7.18. These findings suggest that combining graph-based routing and adaptive execution is more effective in this setting. We also clarified the adaptive anchor-budget policy in Appendix A.2 (Figure 4).

Thank you for the clarification and the additional experiments. The four-way ablation is helpful and partially addresses my concern about the execution-budget confound. However, several issues remain.

  1. The correction to m=5m=5 (N=8N=8) in Appendix A.2 resolves my previous concern regarding a fully connected default graph. However, input-dependent connectivity and weights alone do not support Section 3.4's claim that the number of involved experts adapts to task complexity.

  2. While Table 4 shows gains under the adaptive budget, Section 4.5 notes remaining differences in router-logit regularization, and the original uneven-schedule comparisons remain confounded by differing execution budgets. I am not clear about the contribution of graph-based routing.

Thanks for your comments.

Section 3.4 defines two quantities. The involved count is ninv=∣Ub∣n_{\mathrm{inv}}=|\mathcal{U}b|, the selected anchors plus their neighbors. The anchor budget ktk_t is set by the class number. Input-dependent edges show that the neighbors can change with the image. They do not show that ninvn{\mathrm{inv}} grows with the class number. Across seeds 42–44, the effective count stays about 1.8–2.8 as the class number falls from 28 to 3, and the largest gate still holds at least 99% of the weight. The part of Section 3.4 that follows task complexity is ktk_t.

The graph contribution is the matched comparison in Table 4. At fixed k=2k=2, MoCE reaches 76.26 and MoE reaches 76.50. Under the same adaptive budget, MoCE reaches 80.68, and MoE reaches 77.78, a +2.90 final-accuracy difference, with forgetting of 7.18 versus 9.70. Section 4.5 is correct that the router-logit term remains only on the graph path, so this difference includes that regularizer. The original uneven comparisons use different budgets and are not this result.

We appreciate your feedback. And looking forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.