- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Rex-MoE studies continual LLM adaptation without replay or test-time task identities. It expands shared orthonormal input/output bases and learns task-specific transformations between historical and new directions. Historical expert parameters remain frozen. A constrained routing mechanism combines the experts. Experiments on two 15-task benchmarks report improved performance and reduced forgetting.
Critical strengths
-
Clear construction. The factorization makes the reuse of historical input/output directions explicit and preserves historical expert weight matrices.
-
Consistent results. Rex-MoE improves average performance over the selected GainLoRA baseline in all eight reported backbone/order settings. The evaluation includes generation and classification with encoder-decoder and decoder-only models.
-
Relevant controls. The random-basis experiment tests the value of learned historical directions. Reduced-expansion variants address parameter-budget differences, and the routing ablations support the contribution of historical experts.
Critical weaknesses
-
Much of the underlying approach is established. O-LoRA and InfLoRA already use orthogonal adaptation for continual learning, and Rex-MoE adopts StelLA's three-factor parameterization and Stiefel optimization. The paper acknowledges these predecessors. C-ADA (ECCV 2024) also expands adapter parameters while freezing historical blocks and encouraging orthogonality. TRGP (ICLR 2022) combines orthogonal updates with learned transformations of protected historical subspaces. The paper needs to establish what substantial capability its particular combination adds.
-
The closest architectural precedents are missing. MoBLoRA (Chen et al., ACL 2026) already uses shared orthonormal input/output bases, task-specific mixing matrices, orthogonal expansion, and zero-padding to preserve historical transformations. MoRAM (Lu et al., first posted June 2025) already reuses accumulated rank-one components instead of whole adapters. I did not find either discussed or compared in the manuscript. Rex-MoE's remaining differences include its restricted core, historical-new cross-couplings, and routing. Their value needs a direct comparison with these methods and ablations that isolate those differences.
-
The need for continual weight adaptation is poorly motivated. The experiments assume that the LLM must update its parameters for each incoming task. They do not show why a capable frozen model using task instructions or in-context examples is inadequate. LFPT5 uses learned prompt tuning and does not answer this question. The paper should identify the intended deployment and define what information competing methods may retain. It should then compare against the strongest prompting alternative allowed by those constraints.
-
The resource trade-off is incomplete. Bases, cores, and routers accumulate across tasks. The parameter controls and inference FLOPs are useful, but the paper does not establish the full training, storage, and per-query costs for an intended workload. The cumulative basis also has a finite dimensional limit. A 15-task evaluation gives limited evidence about the method's operating range.
-
The routing guarantee is narrower than model preservation. Frozen expert matrices do not ensure unchanged predictions when routing weights and intermediate representations change. The margin constraint applies to a batch-averaged, linearized quantity on current-task inputs. Historical router-subspace construction and the initialization of new routers also need clarification.
Questions for authors
See weaknesses