Historical Supervision Reactivation with Dynamic Heterogeneous Memory for Long-Horizon Class-Incremental Learning

Rehearsal-free prompt-based methods adapt pretrained models to new classes by training a small set of prompt parameters, yet can still forget old classes during subsequent learning. We investigate whether a limited history of examples can alleviate this forgetting by restoring direct supervision. We introduce Historical Supervision Reactivation (HSR), which enables losses on stored images to update trainable prompts and the classifier. To sustain this training process within limited storage, we further introduce Dynamic Heterogeneous Memory (DHM). DHM protects historical images and reserves replay quotas to retain their update paths and opportunities for use, while managing the remaining examples through a classification-stability-guided lifecycle from FP32 images to FP16 images and compact features. Our controlled study evaluates 23 configurations across three seeds on CIFAR-100 and ImageNet-R, for 69 runs with ten tasks and three training epochs per task. In supervision-path comparisons, HSR improves mean final accuracy by 1.84 and 3.15 percentage points over image replay with historical prompt gradients blocked. Under the same memory caps, DHM improves accuracy by 1.99 and 1.01 points over a fixed heterogeneous buffer. Historical image supervision mitigates forgetting in prompt-based models, and DHM further improves accuracy and forgetting under a fixed memory budget.