PatchBELL: Incremental Learning via Statistical Modeling of Learned Representations

Class-incremental learning (CIL) aims to continuously incorporate new classes while preserving previously learned knowledge. However, updating model parameters on new-class examples often causes catastrophic forgetting on previously learned classes. While recent approaches increasingly leverage pretrained encoders, they mainly rely on image-level embeddings and overlook the statistical structure of patch-level representations. We introduce a probabilistic approach to CIL directly in the frozen representation space, based on class-conditional distributions of patch-level embeddings. We show that these patch populations exhibit an approximately Gaussian structure and formalize a class-conditional likelihood model. Patch-level modeling provides orders of magnitude more samples, enabling reliable class estimation from limited data. However, class-conditional patch modeling introduces two challenges: independently modeled class likelihoods are not directly comparable, and many patches capture non-discriminative content. To address these challenges, we propose PatchBELL, which represents each class with fixed-size statistics and classifies images using comparable likelihoods and confidence-weighted patch evidence. The same confidence scores naturally provide spatial explanations. New classes are incorporated in a single pass, without replay or modification of stored class models, eliminating forgetting by design and remaining robust to incremental task composition. Because PatchBELL requires neither parameter adaptation nor language supervision, it scales naturally to large foundation models and works with both vision-language and vision-only encoders. Experiments across standard CIL benchmarks and multiple foundation models demonstrate state-of-the-art performance with substantially lower incremental update cost.