PACE: Probabilistic Expert Assignment for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning requires models to acquire new capabilities while retaining previously learned ones. Expert-based methods mitigate catastrophic forgetting through parameter isolation, but expert allocation often relies on heuristic or method-specific criteria, while inference-time selection requires a separate routing mechanism. We propose **Probabilistic Assignment for Continual Experts (PACE)**, a router-free method that provides a probabilistic mechanism for selective expert sharing, built from a Chinese Restaurant Process prior and closed-form parametric empirical-Bayes evidence in a frozen multimodal feature space. Before adaptation, PACE determines whether an arriving task should reuse an existing LoRA expert or create a new one, after which only the assigned expert is updated. At inference, the same task statistics enable training-free selection of a single expert, eliminating routing drift and expert interference. We further provide a theoretical analysis of expert sharing, showing that cross-task interference is confined to tasks sharing the same expert and establishing sufficient conditions under which sharing improves generalization over task-wise isolation. Experiments across four continual multimodal benchmarks, CoIN, UCIT, TriGap, and MLLM-CL, demonstrate strong performance and knowledge retention across diverse task sequences. Further analyses show that effective expert sharing depends critically on which tasks are grouped together: PACE preserves task-specific response behavior, avoids highly interfering assignments, and reduces computation through selective adaptation and single-expert generation.