Stream-Aware Prompt Inference via Simplex Search for Class-Incremental Learning

Class-incremental learning (CIL) requires models to acquire new classes while retaining previously learned knowledge continually. With the emergence of powerful pre-trained models (PTMs), particularly Vision Transformers, CIL has increasingly shifted toward parameter-efficient adaptation, where frozen backbones are augmented with lightweight prompts or adapters. While these approaches have advanced rehearsal-free CIL, their inference mechanisms are predominantly designed for individual samples or task-specific routing, conventionally assuming an independent test stream. In practical deployments, however, incoming observations may exhibit local statistical coherence, sharing latent context such as task, domain, environment, or scene, while independent and coherent observations may also be interleaved. Our experiments reveal that such stream structure provides an additional source of information for CIL at inference time, yet it remains largely unexplored in prompt-based CIL. We introduce a stream-aware prompt inference paradigm that exploits local coherence when present while remaining effective for uncorrelated streams. We realize this through ARC (Anchor-Residual Composition), which decomposes each task prompt into a shared anchor and low-rank task residuals, yielding a linearly composable prompt space. Building on this representation, SPS (Simplex Prompt Search) performs entropy-driven optimization over convex combinations of task residuals, adapting inference to the incoming stream. For uncorrelated streams, SPS performs sample-wise optimization, while locally coherent streams enable batch-level optimization that exploits shared statistical structure. Across diverse distribution-shift CIL benchmarks, ARC consistently outperforms existing prompt- and adapter-based methods under both independent and locally coherent streams. Further analysis shows that the effectiveness of stream-aware prompt composition is governed by the geometry of the learned prompt space, highlighting stream structure and prompt geometry as complementary dimensions of inference in continual learning.