DualMem-Evo: Dual-Memory Evolution for Replay-Free Continual Adaptation of Vision-Language Models

Vision-language models (VLMs) have demonstrated remarkable transferability, yet their deployment in dynamic environments requires continuous adaptation to emerging tasks without access to previous training data. This setting poses a fundamental challenge: how to acquire new knowledge while preserving previously learned semantic representations. Existing continual adaptation approaches typically rely on either shared adaptation modules, which suffer from severe interference, or isolated task experts, which limit knowledge transfer across tasks. In this work, we propose DualMem-Evo, a dual-memory continual adaptation framework that views lifelong learning as a process of knowledge evolution rather than static knowledge preservation. Our framework introduces two complementary memories: a semantic evolution memory that continuously accumulates transferable cross-task knowledge, and an episodic expert memory that preserves task-specific expertise. To enable effective memory reuse under a strict no-replay constraint, we develop a Prototype-Guided Expert Evolution mechanism that dynamically retrieves, interacts with, and consolidates historical expert memory modules through cross-modal semantic prototypes and low-rank memory fusion. Furthermore, we introduce functional memory separation and stability-aware semantic regulation to maintain representation diversity while preventing semantic drift during incremental adaptation. The empirical results demonstrate that our framework achieves the state-of-the-art performance.