acceptodds
Under review as a conference paper at ICLR 2027

Zero-Drift Expert Pool: Factorized Routing for Stable Pre-Trained-Model Continual Learning

Abstract

Continual learning with pre-trained models (PTMs) keeps the backbone frozen, adds lightweight adapters, and stores no exemplars. A standard way to scale such a learner is to grow a pool of experts, one per task, and route inputs among them. The obstacle is routing stability: any scheme that normalizes scores over the expert set (softmax, sparsemax, normalized sigmoid) changes every old gate when a new expert is appended, re-mixing previously frozen features. We prove that any normalized router drifts, that this drift cannot be eliminated via regularization, and that factorized per-expert gates eliminate gate-level drift. Building on this, we introduce Zero-Drift Expert Pool (ZDEP): factorized sigmoid gates with hard top-1 routing so appended experts do not affect existing gates, a validation-based controller that adds or reuses an expert, and a modular classifier decoupled from the router. On CIFAR-100, ImageNet-100 (10 tasks each), and ImageNet-R (5 tasks) with an ImageNet-21k ViT-B/16 backbone, ZDEP reaches 0.902, 0.935, and 0.782 average accuracy, surpassing RanPAC (0.874/0.892/0.660), FeCAM (0.851/0.901/0.381), SimpleCIL, and CL-LoRA. A learned softmax router (SoftMoE-style) achieves only 0.358 on CIFAR-100; by contrast ZDEP holds routing drift at , a nine-order-of-magnitude difference.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning, Pre-trained Models (PTMs), Mixture of Experts (MoE), Zero-Drift Routing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3BbXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors of the paper operate on rehearsal-free class-incremental learning with a frozen ViT-B/16 and a growing pool of LoRA experts. They argue that routers that normalize scores over the expert set (softmax and similar) shift every old gate when a new expert is added. Gates that each depend only on their own expert (“factorized” gates) do not shift, and therefore propose the method ZDEP.

Critical strengths

Clear design principle. Showing that renormalization over a growing expert set rescales old gates is simple and easy to check. Building the method around frozen, factorized gates is principled and easy to implement.

Practical properties. No exemplars, frozen backbone, about 2M trainable parameters, a simple growth rule, and code is promised.

Critical weaknesses

The theoretical results look mostly trivial to me, and the main “if and only if” claim is too strong.

Proposition 1: This is just a basic mathematical result. Dividing by a larger sum reduces the old gates' values. The claim that the result is “tight when all old scores are equal” is confusing, because it describes equality rather than a bound. Also, Sparsemax is listed as a normalized method, but it does not follow the formula used in the proposition. Therefore, the proposition does not cover Sparsemax as written.

Proposition 2: This result is almost self-evident. If a gate does not depend on other experts, adding new experts will not affect it.

Proposition 3: The argument is circular. The old parameters are frozen, and the method is required to remain unchanged whenever a new expert is added. Therefore, the conclusion that the gates cannot depend on other experts is already implied by the assumptions.

Proposition 4: The reasoning is unclear and inconsistent. It first claims that the gradients are zero for all parameters, but then says routing only works if the key is frozen.

The abstract also contradicts the theoretical analysis. It claims that freezing the keys means the highest-scoring gate remains unchanged. This is incorrect because adding new gates can change which gate has the highest score. Section 3.1 also acknowledges that a new gate can outperform the previous winner.

The paper does not measure its main claim. The experiments do not report routing accuracy, how often old samples are assigned to different experts after new experts are added, or how many experts the controller actually selects for each dataset.

There may also be label or task-ID leakage during inference. The abstract and Section 1 state that the method commits to a winning expert for each class after every task and reuses this assignment during inference. If test-time routing relies on these class-specific assignments, the method may require the test sample's class or task label. This would give the method access to information that may not be available in a real-world setting.

Overall, the paper is poorly presented and unfortunately needs significant improvement in both presentation and technical content. The writing and overall presentation could be more polished and structured, as the current version makes it difficult to follow the main contributions. The experimental evaluation would benefit from stronger and more comprehensive baselines, as well as a more carefully designed experimental setup. Finally, the theoretical analysis needs to be strengthened to provide more meaningful insights and better support the paper's central claims.

Questions for authors

None.

Reviewer_g3SyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ZDEP grows a pool of frozen LoRA experts with independent sigmoid gates, a validation-based expansion rule, and a modular classifier. The paper argues that factorization characterizes gate invariance under pool expansion and reports improved class-incremental accuracy over several baselines.

Critical strengths

The distinction between changes in old gate values and changes in the selected expert is explicitly recognized in Section 3.1. The experiments include a shared ridge-head control, normalization controls, and an analysis showing that measured gate drift need not track forgetting.

Critical weaknesses

Proposition 3 does not follow from the stated assumptions. Append-only invariance implies independence from experts added later, but does not rule out dependence on earlier experts. For example, g1=σ(q⊤θ1)g_1=\sigma(q^\top\theta_1) and ge=σ(q⊤θe+q⊤θ1)g_e=\sigma(q^\top\theta_e+q^\top\theta_1) for e≥2e\geq2 are differentiable and append-invariant with frozen old parameters, but are not factorized as defined. The proof's step of adding an arbitrary expert last needs an additional order-invariance or permutation-equivariance assumption. This undermines the central necessary-and-sufficient claim as stated. Frozen old gates do not imply a frozen top-1 winner. Adding a gate valued 0.90.9 changes the winner even if the previous winning gate remains 0.80.8. The abstract and Appendix A nevertheless attribute invariant argmax to frozen keys, while the abstract also describes a committed per-class winner. These are different inference procedures. A class-committed route requires an explanation of how the relevant class is determined without its test label; a global argmax requires measuring route-switch rates.

Questions for authors

What additional assumption excludes the append-invariant, non-factorized counterexample to Proposition 3? Is inference a global argmax, a committed per-class expert assignment, or evaluation of class-specific experts? Does it ever use true task or class labels?

Reviewer_hGLJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies expandable-expert continual learning on frozen pre-trained backbones. Its central theoretical claim is that any router normalizing gate scores over the expert set (softmax, sparsemax, normalized sigmoid) necessarily drifts when experts are appended, and that factorized gates are necessary and sufficient for zero drift. Based on this, ZDEP combines factorized sigmoid gates with hard top-1 routing, a validation-based expand-vs-reuse controller, and a modular classifier, reporting 0.902/0.935/0.782 average accuracy on CIFAR-100/ImageNet-100/ImageNet-R with a frozen ImageNet-21k ViT-B/16, above RanPAC, FeCAM, SimpleCIL, and CL-LoRA. A dose–response study claims gate drift and forgetting are decoupled, with residual forgetting confined to the classifier head.

Critical strengths
  • The proof that normalized routers must drift (Proposition 1) and that factorization is sufficient (Proposition 2) is elementary but stated precisely, and the paper is commendably careful in distinguishing gate-level invariance from feature-level stability rather than overclaiming a blanket guarantee.
  • Five seeds with shuffled task orders, fixed hyperparameters across datasets, and head-matched ablations show attention to experimental hygiene.
Critical weaknesses
  • The zero-drift argument only works because routing is, by the paper's own description, committed per class after each task and reused at inference. That is not a router in any meaningful sense: it is a per-class lookup of a frozen expert, i.e., the method reduces to one LoRA adapter per task, selected by a frozen per-class assignment. The genuinely hard part of expandable-expert CL, deciding, from the input alone, which expert an old sample belongs to at test time, is sidestepped rather than solved.
  • Since the paper's thesis is explicitly regularized normalized routing is inferior to factorized routing, the absence of even one shared-benchmark comparison against StaR-MoE leaves the main empirical claim unsupported.
  • The accuracy gains are then better explained by added per-task adapter capacity plus a strong ridge head, neither of which requires the routing formalism.
  • The manuscript needs substantial improvement.
Questions for authors

please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.