Locally Correct, Globally Wrong: Task Identity Recovery for Replay-Free Multi-Source Class-Incremental Learning

Replay-free multi-source class-incremental learning requires models to learn from changing data sources without revisiting past data. Beyond forgetting, two challenges arise. First, independently trained source-specific classifiers may produce scores that are not jointly calibrated, so a global decision over all classes can be wrong even when the corresponding local classifier is correct. Second, new sources may introduce new classes or repeat existing label sets, requiring the model to decide whether to create a new task branch or adapt an existing one. We address this problem with HiDMoA (Hierarchical Decoupled Mixture-of-Adapters). The key idea is to introduce an intermediate task level, where tasks are characterized by source-specific feature distributions. These distributional representations enable two-stage classification: first recovering the input's task identity, then classifying the input within that task's label space, thereby avoiding direct comparisons of scores across independently trained source-specific classifiers. Specifically, HiDMoA uses one source fVAE (feature variational autoencoder) per source for reconstruction-based task identification, while each task branch combines task-branch adapters and a local cosine head on a frozen pretrained backbone for within-task classification. The same reconstruction evidence determines whether a source with a repeated label set joins an existing task branch or starts a new one. Reused task branches are updated through replay-free distillation. On an 8-source, 50-class industrial surface-defect stream, HiDMoA achieves 93.1%, 93.4%, and 92.6% Last Macro-F1 across three source orders, outperforming all evaluated baselines in each order. Task-identity accuracy reaches 99.72%, bringing Last Macro-F1 within one point of an oracle given the true task identity. On an 8-source, 135-class general multi-source stream, HiDMoA-ResNet-50 achieves 95.4% average Macro-F1 and leads at seven of eight evaluation steps. On a repeated-label stream, HiDMoA achieves 95.8% Last Macro-F1, compared with 85.4% for the strongest evaluated baseline.