GeoSpec: Geometry-Guided Expert Specialization for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning requires multimodal large language models to acquire new tasks sequentially while preserving previously learned capabilities. Progressive expert-based methods can isolate past experts from subsequent updates, but two challenges remain: how to adapt each new expert to the functional demands of the current task, and how to reliably access the accumulated experts without test-time task identities. We introduce GeoSpec, a progressive MoE-LoRA framework that addresses these challenges through geometry-guided expert adaptation and compatibility-guided routing. For each arriving task, Task-Conditioned Geometric Adaptation (TCGA) derives a task-conditioned local-output geometry from current-task activations and uses this geometry consistently to shape both expert updates and their fixed-rank realization. At inference, Compatibility-Guided Expert Routing combines task-prototype affinity with discriminative task evidence learned from answer-free, task-labeled support inputs, providing complementary signals for expert selection under task ambiguity. GeoSpec achieves the highest final AVG among the compared methods on all three evaluated MCIT benchmarks, reaching 71.34 on CoIN, 75.97 on UCIT, and 49.70 on TriGap.