acceptodds
Withdrawn from ICLR 2027

Rex-MoE: Knowledge Reuse in Expandable Mixture-of-Experts for Continual Learning of Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning (CL) requires models to learn new tasks sequentially while preserving previously acquired knowledge. Expandable mixture-of-experts (MoE) provides a natural framework by introducing new experts for incoming tasks while preserving historical experts to retain knowledge. However, existing methods typically reuse historical experts only as whole modules, which limits the flexibility to reuse and recombine useful components of acquired knowledge when learning new tasks. By reusing these components while keeping historical experts fixed, models could better balance new-task learning and knowledge retention. In this paper, we propose knowledge Reuse in expandable MoE (Rex-MoE), which enables flexible knowledge reuse during expert learning while coordinating the contributions of new and historical experts. For expert learning, Rex-MoE factorizes each adaptation into expandable shared bases and a task-specific transformation coupling historical and new bases. This structure allows new tasks to reuse historical bases without rewriting the transformations among them. For expert integration, Rex-MoE learns the relative contributions of new and historical experts while constraining router updates to limit interference. Extensive experiments on CL benchmarks demonstrate state-of-the-art performance of Rex-MoE.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Mixture-of-Experts, Low-Rank Adaptation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YVi1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Rex-MoE studies continual LLM adaptation without replay or test-time task identities. It expands shared orthonormal input/output bases and learns task-specific transformations between historical and new directions. Historical expert parameters remain frozen. A constrained routing mechanism combines the experts. Experiments on two 15-task benchmarks report improved performance and reduced forgetting.

Critical strengths
  1. Clear construction. The factorization makes the reuse of historical input/output directions explicit and preserves historical expert weight matrices.

  2. Consistent results. Rex-MoE improves average performance over the selected GainLoRA baseline in all eight reported backbone/order settings. The evaluation includes generation and classification with encoder-decoder and decoder-only models.

  3. Relevant controls. The random-basis experiment tests the value of learned historical directions. Reduced-expansion variants address parameter-budget differences, and the routing ablations support the contribution of historical experts.

Critical weaknesses
  1. Much of the underlying approach is established. O-LoRA and InfLoRA already use orthogonal adaptation for continual learning, and Rex-MoE adopts StelLA's three-factor parameterization and Stiefel optimization. The paper acknowledges these predecessors. C-ADA (ECCV 2024) also expands adapter parameters while freezing historical blocks and encouraging orthogonality. TRGP (ICLR 2022) combines orthogonal updates with learned transformations of protected historical subspaces. The paper needs to establish what substantial capability its particular combination adds.

  2. The closest architectural precedents are missing. MoBLoRA (Chen et al., ACL 2026) already uses shared orthonormal input/output bases, task-specific mixing matrices, orthogonal expansion, and zero-padding to preserve historical transformations. MoRAM (Lu et al., first posted June 2025) already reuses accumulated rank-one components instead of whole adapters. I did not find either discussed or compared in the manuscript. Rex-MoE's remaining differences include its restricted core, historical-new cross-couplings, and routing. Their value needs a direct comparison with these methods and ablations that isolate those differences.

  3. The need for continual weight adaptation is poorly motivated. The experiments assume that the LLM must update its parameters for each incoming task. They do not show why a capable frozen model using task instructions or in-context examples is inadequate. LFPT5 uses learned prompt tuning and does not answer this question. The paper should identify the intended deployment and define what information competing methods may retain. It should then compare against the strongest prompting alternative allowed by those constraints.

  4. The resource trade-off is incomplete. Bases, cores, and routers accumulate across tasks. The parameter controls and inference FLOPs are useful, but the paper does not establish the full training, storage, and per-query costs for an intended workload. The cumulative basis also has a finite dimensional limit. A 15-task evaluation gives limited evidence about the method's operating range.

  5. The routing guarantee is narrower than model preservation. Frozen expert matrices do not ensure unchanged predictions when routing weights and intermediate representations change. The margin constraint applies to a batch-averaged, linearized quantity on current-task inputs. Historical router-subspace construction and the initialization of new routers also need clarification.

Questions for authors

See weaknesses

Reviewer_tz7FOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Rex-MoE, an expandable MoE framework for continual learning. It introduces a knowledge-reusing expert that builds on historical low-rank bases and a constrained routing mechanism to coordinate new and previous experts. The method is evaluated on two continual learning benchmarks with three backbone models.

Critical strengths

The paper is generally clear and easy to follow.

The results are consistent across several task orders, and the ablation studies provide some support for the proposed components.

Critical weaknesses
  1. I find the novelty rather limited. The main ideas, including low-rank subspace learning, expandable experts, and constrained routing, have already been explored in prior work. I am not convinced that putting them together offers enough novelty.

  2. The experiments are quite small in scale. The largest model is Llama-2-7B, and the others are T5-Large and Qwen3-1.7B. I would like to see whether the method still works well on larger, more recent LLMs.

  3. I am also concerned about the training cost. The method requires additional orthogonalization and constrained updates, but there is no comparison of GPU-hours, training time, or peak memory with the baselines. Inference FLOPs alone do not tell us how expensive the method is to train.

  4. I do not fully understand how the proposed routing constraint prevents forgetting. It is applied to new-task samples, but the paper does not clearly explain why this should preserve routing behavior on previous tasks.

  5. The experiments only cover two benchmarks with 15 tasks each. Since the method keeps adding experts, I think longer task sequences are needed to see how well it actually scales.

Questions for authors
  1. What is the main novelty of Rex-MoE compared with existing methods such as StelLA [1] and GainLoRA [2]? Have the authors considered comparing against a combination of these approaches?

  2. How much additional training cost does Rex-MoE introduce? Could the authors provide GPU-hours, training time, and peak memory usage compared with the strongest baselines?

  3. Could the authors explain why constraining the routing margin on new-task samples helps prevent forgetting on previous tasks? Is there any direct evidence that it preserves historical routing behavior?

[1] StelLA: Subspace Learning in Low-Rank Adaptation Using Stiefel Manifold, NeurIPS 2025.

[2] Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models, NeurIPS 2025。

Reviewer_bmw6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Rex-MoE, an MoE framework for continual learning of LLMs, where LoRA modules serve as experts. The framework consists of two main components: (i) a knowledge reuse mechanism that incrementally expands shared bases, allowing new experts to reuse subspaces learned from previous tasks; and (ii) a constrained routing mechanism that coordinates the contributions of new and historical experts while mitigating interference. Experiments across multiple continual learning benchmarks and model backbones demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper is well-written and easy to follow.

  • The idea of learning an incrementally expanding shared subspace to facilitate knowledge reuse is interesting.

  • The proposed method demonstrates strong empirical results on the reported benchmarks.

Critical weaknesses
  • The discussion of existing MoE-based continual learning methods remains limited.

  • I have significant concerns about the soundness of the proposed KRE mechanism as the number of tasks increases.

    • In L210, the cumulative rank is defined as Rt=tr≤min⁡(m,n)R_t = tr \leq \min(m,n). However, since mm and nn are fixed by the backbone architecture, this condition cannot hold indefinitely as the number of tasks increases.
    • Furthermore, L220–222 state that newly introduced basis directions are projected onto the orthogonal complements of historical bases. As the cumulative bases expand, the dimension of the remaining orthogonal complement decreases. Eventually, there will be no dimension left to introduce rr new orthogonal basis vectors, making the proposed expansion procedure impossible.
    • Since the experiments consider only 15 tasks, this limitation may not arise in the evaluated settings. Nevertheless, it raises concerns about the applicability of the method to substantially longer task sequences.
  • According to Section 4.1.2, the new expert introduced at task tt contains (2t−1)r2(2t-1)r^2 trainable core parameters, resulting in a total of T2r2T^2r^2 core parameters after TT tasks. Therefore, the total number of core parameters grows quadratically with the number of tasks. Although the paper provides parameter-count comparisons and an inference FLOPs analysis, these experiments are limited to short sequences of tasks.

  • Appendix C.2 states that each expert has a two-layer query MLP and a corresponding key, both shared across all Transformer layers. However, the paper does not sufficiently motivate this design choice or evaluate its effect compared with layer-specific routing. Furthermore, it is unclear whether the query MLP remains trainable across tasks. If so, updating it could potentially lead to forgetting.

Questions for authors

Please refer to the weaknesses discussed above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.