Learning to Compose Prompts with Hebbian Pathways for Continual Learning

Parameter-efficient continual learning adapts a frozen pre-trained model with small modules, such as prompts, that hold the knowledge acquired over the task sequence. However, allocating modules to each task avoids interference but prevents reuse and enlarges the pool with every task, whereas sharing modules across tasks enables reuse but exposes the shared modules to interference from later tasks. This paper proposes a new method of Learning to Compose (L2C) that represents the task knowledge as a composition of prompts selected at multiple layers of a vision transformer, thereby lowering the unit of sharing and protecting from tasks to prompts. Under this representation, a new task can be expressed as a new composition that reuses the prompts of earlier tasks. L2C determines which prompts to protect from a selection history that is decoupled from the task loss, in which a Hebbian rule accumulates the selection probabilities of prompts chosen together at adjacent layers. The history serves as a prior that stabilizes compositions already in use and determines which prompts are frozen or re-initialized orthogonally to the frozen keys, recovering capacity without enlarging the pool. Experiments on three benchmarks with 5 to 40 tasks show that L2C attains the highest average incremental accuracy and the lowest forgetting against ten baselines in all nine settings, while training fewer than 0.2% of the backbone parameters.