CAR: Contextual Adapter Routing for Rehearsal-Free Class-Incremental Learning

Modern rehearsal-free class-incremental learning (CIL) increasingly builds on pretrained models augmented with task-specialized prompts, adapters, or other modular components. While substantial effort has focused on how such modules are learned and stored, inference is still typically performed independently for each test sample, discarding contextual evidence that may be available across nearby observations. We introduce Contextual Adapter Routing (CAR), a frozen context-conditioned inference strategy that exploits a small unlabeled coherent test context without replay, task labels, retraining, or test-time parameter updates. CAR observes that task-specialized modules contain weak but useful task-discriminative information in their predictive uncertainty: although this signal is noisy for individual samples, aggregating it across a coherent context yields substantially more reliable module preferences. CAR converts this contextual preference into a soft task prior while retaining image-specific all-class prediction, thereby improving routing without replacing the underlying continual learner. Across multiple CIL benchmarks and modular learners, CAR improves average continual and final-stage accuracy by up to $11.14$ and $13.49$ percentage points, respectively. Controlled analyses show that the gain depends on contextual coherence: distinct coherent observations provide substantially stronger evidence than same-image augmentations or unsupervised grouping of shuffled samples. We further study unreliable contexts and use routing confidence to fall back toward native sample-wise inference when contextual evidence is weak. Theoretically, when observations share a local shifted distribution and the appropriate module has a positive expected entropy margin, the probability of routing error decreases exponentially with context size. Together, these results establish frozen context-conditioned inference as a complementary design axis for modular continual learning: preserve native sample-wise inference when reliable context is unavailable, while exploiting coherent unlabeled context when it is informative.