acceptodds
Under review as a conference paper at ICLR 2027

PatchBELL: Incremental Learning via Statistical Modeling of Learned Representations

Abstract

Class-incremental learning (CIL) aims to continuously incorporate new classes while preserving previously learned knowledge. However, updating model parameters on new-class examples often causes catastrophic forgetting on previously learned classes. While recent approaches increasingly leverage pretrained encoders, they mainly rely on image-level embeddings and overlook the statistical structure of patch-level representations. We introduce a probabilistic approach to CIL directly in the frozen representation space, based on class-conditional distributions of patch-level embeddings. We show that these patch populations exhibit an approximately Gaussian structure and formalize a class-conditional likelihood model. Patch-level modeling provides orders of magnitude more samples, enabling reliable class estimation from limited data. However, class-conditional patch modeling introduces two challenges: independently modeled class likelihoods are not directly comparable, and many patches capture non-discriminative content. To address these challenges, we propose PatchBELL, which represents each class with fixed-size statistics and classifies images using comparable likelihoods and confidence-weighted patch evidence. The same confidence scores naturally provide spatial explanations. New classes are incorporated in a single pass, without replay or modification of stored class models, eliminating forgetting by design and remaining robust to incremental task composition. Because PatchBELL requires neither parameter adaptation nor language supervision, it scales naturally to large foundation models and works with both vision-language and vision-only encoders. Experiments across standard CIL benchmarks and multiple foundation models demonstrate state-of-the-art performance with substantially lower incremental update cost.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
class-incremental learning, continual learning, representation learning, foundation models, probabilistic modeling, patch representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aeznOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PATCH BELL, a statistical approach to class-incremental learning. It keeps the visual encoder frozen and models each class using Gaussian distributions of patch features, with Jacobian correction and entropy-based aggregation. New classes are added without gradient-based training or replaying old data. The results are strong on ImageNet-R and ImageNet100, but the likelihood formulation and the overall computational cost need further investigation.

Critical strengths

The update mechanism is simple and avoids modifying old class models, preventing forgetting caused by parameter updates.

The method achieves strong results on ImageNet-R and ImageNet100. The ablations also show that both Jacobian correction and entropy weighting help.

It avoids repeated optimization and replay, while naturally producing spatial confidence maps for explaining predictions.

Critical weaknesses

Each class uses a different PCA subspace, with (k=500<d=1024) in the main experiments. It is unclear whether the Jacobian correction alone makes the resulting likelihoods directly comparable. Better accuracy does not, by itself, validate this probabilistic interpretation.

Old class statistics remain unchanged, but adding new classes can still change entropy weights and predictions. The method prevents forgetting caused by parameter updates, but does not necessarily guarantee stable accuracy on old classes.

The method avoids gradient-based training, but still requires covariance estimation and eigendecomposition. The paper focuses on update FLOPs and does not sufficiently evaluate model storage.

Fig. 6 shows several examples with seemingly reasonable confidence maps, but failure cases are missing. Since the patch weight measures how discriminative a patch is across classes rather than whether it supports the correct class, high-confidence regions may still be irrelevant or misleading. This raises concerns about the faithfulness of the explanations and their potential impact on classification accuracy.

Every test patch must be evaluated against all class models, so the main classification cost grows roughly linearly with the number of classes. The paper does not report actual inference latency, throughput, or peak memory, making its scalability claims difficult to assess.

The main experiments use DINOv3 ViT-L/16, while the baselines use different backbones and training procedures. This makes it difficult to isolate the gains from the proposed method itself. PATCH BELL also does not outperform every baseline on every benchmark.

Questions for authors

How does the Jacobian correction make likelihoods comparable when each class uses a different truncated PCA subspace? Could the authors validate this using a shared subspace or an explicit model of the discarded dimensions?

Could the authors report old-class accuracy across incremental stages to verify whether keeping class statistics unchanged also preserves recognition performance?

Could the authors report inference latency, throughput, peak memory, and model storage at different class scales, ideally compared with a simple classifier using the same backbone?

Without gradient-based training, how much time is spent on covariance estimation and eigendecomposition, and how much do these steps contribute to the total update cost?

Could the authors provide some failure cases and analyze when PATCH BELL assigns high confidence to irrelevant or misleading patches, such as background regions or other objects? Can the method make correct predictions while relying on the wrong patches, and how often do such patches lead to misclassification? What is their impact on explanation faithfulness and classification accuracy, and can these cases be detected or mitigated?

At this stage, I still have several concerns that I hope the authors can address, particularly regarding the likelihood formulation, the overall computational cost, and the failure cases of patch-based explanations. I would appreciate further clarification and supporting evidence. Depending on the authors’ responses and any additional results, I am open to revising my score.

2 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for recognizing the efficiency of the incremental update and the empirical contribution of the Jacobian and entropy components. We have revised the theoretical claims, expanded the resource analysis, added matched-backbone comparisons, and included qualitative failure cases for the confidence maps.

W1, Q1. Jacobian correction with truncated, class-specific subspaces.

We agree with the mathematical distinction. For full-dimensional whitening (k=dk=d) with nonsingular covariance, Equation (3) is the Gaussian log-density, including the covariance log-determinant, and likelihoods are comparable in the common feature space. For k<dk<d, the class-specific projections are non-invertible: correcting the retained covariance volume does not account for the discarded dimensions or define an exact full-space density. We have made this limitation explicit in Section 3.2 and Appendix A.3 and now describe the default k=500k=500 score as a computational approximation, not an exact change-of-variables likelihood. Empirically, the whitening-dimension sweep (Figure 4a and Appendix Table 10) gives 87.28% Last at k=500k=500 and 86.90% at the full k=1024k=1024. These results support the truncated formulation as an efficient approximation that preserves the classification performance of the exact full-dimensional model.

W2, Q2. Preservation of old knowledge versus old-class recognition.

We have refined our claims throughout the manuscript to distinguish parameter-induced forgetting from changes in recognition accuracy. PATCHBELL guarantees that previously learned class models remain unchanged, independently of class order or task partitioning (Tables 4 and 8). However, adding new classes introduces additional competing likelihoods and changes entropy-based confidence weights, potentially affecting old-class predictions. Thus, PATCHBELL eliminates forgetting caused by parameter updates, not the inherent difficulty of distinguishing an expanding set of classes.

W3, W5, W6, Q3, Q4. Matched-backbone fairness, computational cost, inference scaling, and model storage.

We agree that matched-backbone comparisons are essential. We therefore added Table 2, evaluating PATCHBELL and leading baselines under identical encoders and protocols. For the limited rebuttal period we report ImageNet100 results.

With CLIP ViT-L/14, PATCHBELL achieves accuracy comparable to MG-CLIP, with 8.3× lower update FLOPs and 17.6× lower peak GPU memory. While inference requires more FLOPs, class-conditional evaluations are independent and can be parallelized efficiently on GPUs. We report FLOPs rather than wall-clock time to provide a hardware-independent measure of computational cost.

Importantly, backbone flexibility is a key advantage of PATCHBELL that matched-backbone comparisons alone do not capture. Most competing methods depend on language representations and cannot operate on other powerful vision-only encoders such as DINOv3. In contrast, PATCHBELL requires neither text supervision nor backbone-specific adaptation, allowing it to exploit advances in visual foundation models directly. With the same DINOv3 backbone, PATCHBELL substantially outperforms RSIAT (87.28% vs. 70.86% Last accuracy). Notably, RSIAT's performance degrades substantially when transferring from its original pretrained backbone to CLIP ViT-L/14 or DINOv3, highlighting its sensitivity to backbone choice. While additional hyperparameter tuning might improve transfer, it would incur substantial computational overhead, particularly given RSIAT's already high update cost.

Together, these results demonstrate competitive accuracy at substantially lower update cost under matched backbones and broad compatibility across foundation models. We have accordingly refined the positioning of PATCHBELL relative to existing methods throughout the revised manuscript, from the abstract and introduction to the related work, method sections, and experimental evaluation.

Table 2 in the updated paper:

MethodAvg ↑Last ↑Update PFLOPs/task ↓Inference GFLOPs/image ↓Peak allocated GPU (GiB) ↓Peak reserved GPU (GiB) ↓
CLIP ViT-L/14
CLAP82.0881.2611.512162.0563.3053.486
MG-CLIP90.3383.4017.414162.43033.10134.299
RSIAT71.5757.18188.009163.64325.75230.475
PATCH BELL89.1582.362.107188.2771.8802.047
DINOv3 ViT-L/16 (vision-only)
RSIAT85.1670.86145.826126.94516.41516.902
PATCH BELL92.2587.281.634145.7801.2201.273
AuthorsOpenReview ↗

Response 2

W3, W5, W6, Q3, Q4. (continued).

Computational cost. To isolate the computational overhead of PATCHBELL, Appendix Table 16 decomposes the total cost into frozen encoder computation and PATCHBELL-specific operations, for both CLIP ViT-L/14 and DINOv3 ViT-L/16. We report FLOPs per incremental task during updates and per image during inference. For each setting, we compare the complete PATCHBELL pipeline against encoder-only feature extraction, with the difference measuring the additional computation introduced by our method. The results show negligible update overhead (approximately 0.5%) and moderate inference overhead (approximately 16% relative to encoder computation). While inference cost grows linearly with the number of classes, class-conditional likelihood evaluations are independent and can be parallelized on GPUs, reducing their impact on practical latency.

Table 16 in the updated paper:

BackboneConfigurationUpdate PFLOPs/task ↓Inference GFLOPs/image ↓
CLIP ViT-L/14Full PATCHBELL2.107188.277
Encoder only2.097162.024
PATCHBELL overhead0.01026.253
DINOv3 ViT-L/16Full PATCHBELL1.634145.780
Encoder only1.626125.680
PATCHBELL overhead0.00820.100

Storage requirements. PATCHBELL stores only fixed-size class statistics, with memory growing linearly in the number of classes. Table 17 quantifies these requirements across backbones and class counts, showing that even with 1,000 classes and a 1,024-dimensional backbone, total storage is only 1.91 GiB. Importantly, these statistics are stored in CPU memory and do not require persistent GPU allocation. As demonstrated in Table 2, PATCHBELL has the lowest peak GPU memory consumption among all compared methods, confirming that its statistical modeling introduces modest memory requirements despite maintaining independent models for each class.

Table 17 in the updated paper:

BackboneFeature dimension100 classes200 classes1,000 classes
CLIP ViT-B/167680.143 GiB0.287 GiB1.433 GiB
CLIP ViT-L/141,0240.191 GiB0.382 GiB1.911 GiB
DINOv3 ViT-B/167680.143 GiB0.287 GiB1.433 GiB
DINOv3 ViT-L/161,0240.191 GiB0.382 GiB1.911 GiB

W4, Q5. Confidence-map faithfulness and failure cases.

We thank the reviewer for raising this important distinction. Our confidence maps measure how discriminative each patch is across classes, rather than whether it supports the ground-truth class. We therefore added representative failure cases in Figures 7 and 8, illustrating two limitations: confidence concentrating on a salient but potentially irrelevant object in multi-object scenes, and high-confidence regions appearing on both the target object and irrelevant background areas.

While directly quantifying the frequency and classification impact of such failures would require additional region-level annotations, we provide quantitative evidence of explanation faithfulness through deletion and insertion experiments (Appendix Tables 18 and 19). These evaluations assess whether highlighted regions genuinely influence classification, both for PATCHBELL and an independent classifier. Across 5,000 images, our confidence maps remain competitive with dedicated attribution methods, demonstrating that they generally identify meaningful, decision-relevant regions.

We emphasize that PATCHBELL is not proposed as a dedicated explainability method. The confidence maps are an intrinsic by-product of classification, providing informative spatial explanations without additional training or attribution computation. The new failure cases clarify their limitations, while the quantitative evaluations demonstrate their practical value.

Final note: We thank the reviewer again for their constructive and detailed feedback, which has helped us strengthen the theoretical framing, computational evaluation, and reliability of the confidence maps. We hope the revised manuscript and additional evidence adequately address the main concerns and support a positive reassessment of our work.

Reviewer_U92SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an exemplar-free class-incremental classifier built on frozen visual encoders. Each class is represented by a Gaussian fitted to its patch features, using class-specific PCA whitening and a covariance-volume correction. At inference, cross-class entropy supplies patch weights, and weighted log-likelihoods determine the image label. Class models are estimated independently, so the final model is invariant to class order and task partitioning. Experiments cover ImageNet-R, ImageNet100 and ImageNet-1K, several encoder families, data imbalance, component ablations and spatial attribution. The main DINOv3 ViT-L/16 configuration reports final accuracies of 89.05%, 87.28% and 81.62% on the three benchmarks.

Critical strengths

A simple incremental mechanism with a useful structural guarantee. Frozen representations and independently stored class statistics avoid modifying old class models. The invariant final accuracy across task partitions and class orders is consistent with this construction (Tables 3 and 7). This is practically attractive when repeated adaptation is expensive; the guarantee is about unchanged class models and the final set of models, rather than unchanged predictions as new competitors arrive.

Substantial empirical and component analysis. The method is strong on ImageNet-R and ImageNet100, transfers across six encoder configurations (Table 9), and includes meaningful controls. On the same DINOv3 representation, the full method improves ImageNet100 final accuracy from 85.60% for CLS Gaussian likelihood to 87.28% (Table 12). The Jacobian ablation and the independent-classifier attribution evaluation provide further evidence that the design has value beyond a bare patch-voting baseline.

Critical weaknesses

Equation (3) computes class-conditional likelihoods using different class-specific rank-(k) PCA projections. Although the Jacobian correction accounts for covariance differences within the retained subspaces, it does not address the class-dependent dimensions discarded during projection. As acknowledged in Appendix A.3, these transformations are non-invertible, and the paper does not sufficiently establish whether the resulting likelihood scores are directly comparable across classes. This issue is particularly important because likelihood comparability is a central assumption underlying the proposed probabilistic formulation. The Gaussianity analysis in Table 13 also provides limited evidence for the assumed joint Gaussian distributions.

The main performance comparisons use different pretrained backbones for PatchBELL and competing methods, making it difficult to distinguish improvements from the proposed classifier versus those from stronger pretrained representations. As shown in Table 9, switching PatchBELL from CLIP ViT-L/14 to DINOv3 ViT-L/16 improves ImageNet100 final accuracy from 82.36% to 87.28%, exceeding its 3.68-point advantage over RSIAT in Table 1. Although Table 12 includes a same-backbone CLS Gaussian baseline, comprehensive matched-backbone comparisons with leading CIL methods are still lacking. This limits the strength of the evidence supporting the claimed methodological superiority.

Questions for authors

Clarification affecting soundness: What common observed-space probability model makes Equation (3) comparable when every class uses a different truncated PCA subspace? Can you supply the derivation, or compare the default with a common projection and a residual-noise model while keeping the encoder fixed? Please also specify the sample counts and held-out protocol for Table 13.

Evidence affecting the recommendation: Can you compare PatchBELL and strong statistical baselines under identical encoder checkpoints, class orders, and data subsets on all three benchmarks? Please report uncertainty for the 87.28% versus 85.60% comparison, and explain how hyperparameters were selected without tuning on final evaluation labels.

Practical clarification: What are measured prediction latency and stored-statistics memory at 100, 200, and 1,000 classes, alongside update time? Class-wise transforms make the straightforward inference cost scale as O(CPdk), and immutable old models do not make the class-dependent entropy weights invariant when classes are added. Please distinguish model-parameter retention from old-class accuracy retention. These are requests to substantiate current claims, rather than suggestions for unrelated future work.

7 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the careful assessment and for distinguishing the invariance of stored class statistics from the behavior of predictions as new classes arrive. The revision strengthens the mathematical qualification, adds controlled backbone comparisons, and reports more detailed resource requirements.

W1, Q1. Likelihood comparability under class-specific truncated PCA, and Gaussianity evidence.

We agree with the mathematical distinction. For full-dimensional whitening (k=dk=d) with nonsingular covariance, Equation (3) is the Gaussian log-density, including the covariance log-determinant, and likelihoods are comparable in the common feature space. For k<dk<d, the class-specific projections are non-invertible: correcting the retained covariance volume does not account for the discarded dimensions or define an exact full-space density. We have made this limitation explicit in Section 3.2 and Appendix A.3 and now describe the default k=500k=500 score as a computational approximation, not an exact change-of-variables likelihood. Empirically, the whitening-dimension sweep (Figure 4a and Appendix Table 10) gives 87.28% Last at k=500k=500 and 86.90% at the full k=1024k=1024. These results support the truncated formulation as an efficient approximation that preserves the classification performance of the exact full-dimensional model.

Regarding Gaussianity, Appendix B.5 now clarifies the evaluation protocol. For each ImageNet100 class, we uniformly sample N=10,000N=10{,}000 patch representations, using at most ten spatial patches per image. Patch locations are sampled uniformly, and a class-specific full-dimensional PCA-whitening transform is fitted to the sampled population.

We evaluate Gaussianity before and after whitening using the Anderson--Darling and D'Agostino--Pearson tests at 5%5\% significance over all 1,0241{,}024 representation coordinates and 256 fixed random one-dimensional projections per class. Each distribution contains 10,00010{,}000 scalar samples. Figure 5 reports class-wise test statistics, while Table 15 aggregates pass rates across all 100 classes, providing empirical evidence for approximate Gaussian structure along both individual coordinates and random projections.

W2, Q2. Matched-backbone baselines, variability, and hyperparameters.

We agree that matched-backbone comparisons are essential. We therefore added Table 2, evaluating PATCHBELL and leading baselines under identical encoders and protocols. For the limited rebuttal period we report ImageNet100 results.

With CLIP ViT-L/14, PATCHBELL achieves accuracy comparable to MG-CLIP, with 8.3× lower update FLOPs and 17.6× lower peak GPU memory. While inference requires more FLOPs, class-conditional evaluations are independent and can be parallelized efficiently on GPUs. We report FLOPs rather than wall-clock time to provide a hardware-independent measure of computational cost.

Importantly, backbone flexibility is a key advantage of PATCHBELL that matched-backbone comparisons alone do not capture. Most competing methods depend on language representations and cannot operate on other powerful vision-only encoders such as DINOv3. In contrast, PATCHBELL requires neither text supervision nor backbone-specific adaptation, allowing it to exploit advances in visual foundation models directly. With the same DINOv3 backbone, PATCHBELL substantially outperforms RSIAT (87.28% vs. 70.86% Last accuracy). Notably, RSIAT's performance degrades substantially when transferring from its original pretrained backbone to CLIP ViT-L/14 or DINOv3, highlighting its sensitivity to backbone choice. While additional hyperparameter tuning might improve transfer, it would incur substantial computational overhead, particularly given RSIAT's already high update cost.

Together, these results demonstrate competitive accuracy at substantially lower update cost under matched backbones and broad compatibility across foundation models. We have accordingly refined the positioning of PATCHBELL relative to existing methods throughout the revised manuscript, from the abstract and introduction to the related work, method sections, and experimental evaluation.

Table 2 in the updated paper:

MethodAvg ↑Last ↑Update PFLOPs/task ↓Inference GFLOPs/image ↓Peak allocated GPU (GiB) ↓Peak reserved GPU (GiB) ↓
CLIP ViT-L/14
CLAP82.0881.2611.512162.0563.3053.486
MG-CLIP90.3383.4017.414162.43033.10134.299
RSIAT71.5757.18188.009163.64325.75230.475
PATCH BELL89.1582.362.107188.2771.8802.047
DINOv3 ViT-L/16 (vision-only)
RSIAT85.1670.86145.826126.94516.41516.902
PATCH BELL92.2587.281.634145.7801.2201.273
AuthorsOpenReview ↗

Response 2

W2, Q2. (continued)

Regarding variability, Appendix Table 8 reports results across five random class orders. PATCHBELL achieves 87.28 ± 0.00% Last accuracy, confirming its invariance to class order, while Avg accuracy is 92.12 ± 0.66%, reflecting differences in intermediate task compositions. Importantly, both PATCHBELL and the CLS Gaussian baseline are deterministic and class-order invariant, so their Last accuracy comparison (87.28% vs. 85.60%) is unaffected by class ordering.

Regarding hyperparameter selection, the main parameter is the number of retained whitening dimensions kk. The dimension sweep (Appendix Table 10) demonstrates strong robustness across a wide range of dimensions. We select k=500k=500 as an intermediate value, noy by selecting the configuration that maximizes test accuracy or minimizes computational cost.

Q3. Inference cost, storage, and the meaning of forgetting.

We agree that computational efficiency should be evaluated beyond incremental updates. We have therefore substantially expanded our resource analysis. Table 2 (discussed above in W2) now reports both update and inference FLOPs, alongside measured peak GPU memory, while Appendix Tables 16 and 17 provide a detailed breakdown of computational costs and persistent storage requirements.

Computational cost. To isolate the computational overhead of PATCHBELL, Appendix Table 16 decomposes the total cost into frozen encoder computation and PATCHBELL-specific operations, for both CLIP ViT-L/14 and DINOv3 ViT-L/16. We report FLOPs per incremental task during updates and per image during inference. For each setting, we compare the complete PATCHBELL pipeline against encoder-only feature extraction, with the difference measuring the additional computation introduced by our method. The results show negligible update overhead (approximately 0.5%) and moderate inference overhead (approximately 16% relative to encoder computation). While inference cost grows linearly with the number of classes, class-conditional likelihood evaluations are independent and can be parallelized on GPUs, reducing their impact on practical latency.

Table 16 in the updated paper:

BackboneConfigurationUpdate PFLOPs/task ↓Inference GFLOPs/image ↓
CLIP ViT-L/14Full PATCHBELL2.107188.277
Encoder only2.097162.024
PATCHBELL overhead0.01026.253
DINOv3 ViT-L/16Full PATCHBELL1.634145.780
Encoder only1.626125.680
PATCHBELL overhead0.00820.100

Storage requirements. PATCHBELL stores only fixed-size class statistics, with memory growing linearly in the number of classes. Table 17 quantifies these requirements across backbones and class counts, showing that even with 1,000 classes and a 1,024-dimensional backbone, total storage is only 1.91 GiB. Importantly, these statistics are stored in CPU memory and do not require persistent GPU allocation. As demonstrated in Table 2, PATCHBELL has the lowest peak GPU memory consumption among all compared methods, confirming that its statistical modeling introduces modest memory requirements despite maintaining independent models for each class.

Table 17 in the updated paper:

BackboneFeature dimension100 classes200 classes1,000 classes
CLIP ViT-B/167680.143 GiB0.287 GiB1.433 GiB
CLIP ViT-L/141,0240.191 GiB0.382 GiB1.911 GiB
DINOv3 ViT-B/167680.143 GiB0.287 GiB1.433 GiB
DINOv3 ViT-L/161,0240.191 GiB0.382 GiB1.911 GiB

Forgetting and model stability. We have also refined our claims throughout the manuscript to distinguish parameter-induced forgetting from changes in recognition accuracy. PATCHBELL guarantees that previously learned class models remain unchanged, independently of class order or task partitioning (Tables 4 and 8). However, adding new classes introduces additional competing likelihoods and changes entropy-based confidence weights, potentially affecting old-class predictions. Thus, PATCHBELL eliminates forgetting caused by parameter updates, not the inherent difficulty of distinguishing an expanding set of classes.

Final note: We thank the reviewer again for their constructive feedback, which has helped us strengthen the mathematical formulation, experimental evaluation, and clarity of our claims. We hope the revisions and additional evidence address the main concerns and encourage a positive reassessment of our work.

We thank the reviewer again for their thoughtful feedback. We have provided detailed responses to the concerns and requests, supported by additional experiments and clarifications in the revised manuscript. We would appreciate your reassessment and welcome any further questions.

Reviewer_b7t5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a PATCHBELL (Patch-Based Entropy-weighted Log-Likelihood) approach for non-replay class-incremental learning that operates in the frozen representation space of visual foundation models. The proposed approach models class-conditional distributions at the patch level to scale effectively and prevent forgetting by design. The authors formalize an approximately Gaussian class-conditional patch-likelihood model, equipped with PCA whitening and a log-determinant Jacobian volume correction, to make class-likelihoods directly comparable. To handle background noise, they propose a cross-class entropy weighting scheme that isolates discriminative regions, yielding robust classification scores and intrinsic spatial explanation maps. It significantly outperforms the compared approaches.

Critical strengths

The paper proposes a simple, intuitive, training-free CIL method with a clear mechanism. By modeling independent class statistics in a single pass without replay or parameter updates, PATCHBELL eliminates catastrophic forgetting by design.

The paper reports whitening-dimension sweeps, backbone and layer sweeps, Gaussianity tests, the imbalance behavior of the Jacobian term, and the per-patch confidence maps with quantitative results.

Significantly outperforms the closest method, RSIAT, on two out of three benchmarks

Critical weaknesses

The proposed method needs to be analyzed further on settings with a large number of classes (around 1K) since its performance improvement vanishes on ImageNet-1K.

The comparison may not be fair since PatchBELL uses a DINOv3 ViT-L/16 trained on LVD-1689M, while most baselines (RAPF, MG-CLIP, CLAP, PROOF, RSIAT’s CLIP setting) use CLIP, probably ViT-B/16, and DINOv3 is far stronger on ImageNet-like domains. The authors should clarify this issue

Like other non-replay frozen-encoder methods, PATCHBELL assumes the pretrained foundation model's representation space is already sufficiently informative for newly encountered classes. In domains with extreme visual novelty or severe distribution shifts where the frozen encoder lacks capacity, closed-form statistical modeling alone may hit a performance ceiling.

Questions for authors

The authors should explain whether the method has any vulnerabilities with CIL settings with a large number of classes (around 1K)

The authors should have compared the methods with the same backbone.

The authors should explain how distribution shift would affect the performance

7 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for recognizing the simplicity of PATCH BELL, its single-pass incremental updates, and the strength of the empirical analysis. We have revised the paper to better clarify our claims and added controlled comparisons and a new domain-transfer experiment. We address the three concerns below.

W1. Performance with a large number of classes.

We acknowledge that increasing the number of classes introduces additional competition between class-conditional likelihoods, affecting the entropy-based patch confidence maps and potentially classification performance. While PATCHBELL remains competitive on ImageNet-1K (81.62% Last accuracy), we agree that this behavior warrants further investigation. We are therefore conducting additional experiments with varying numbers of classes, from 10 to 1,000, to systematically evaluate the impact on classification accuracy. We will incorporate these results into the revised manuscript by the end of the rebuttal period.

W2. Fairness of the backbone comparison.

We agree that matched-backbone comparisons are essential. We therefore added Table 2, evaluating PATCHBELL and leading baselines under identical encoders and protocols. For the limited rebuttal period we report ImageNet100 results.

With CLIP ViT-L/14, PATCHBELL achieves accuracy comparable to MG-CLIP, with 8.3× lower update FLOPs and 17.6× lower peak GPU memory. While inference requires more FLOPs, class-conditional evaluations are independent and can be parallelized efficiently on GPUs. We report FLOPs rather than wall-clock time to provide a hardware-independent measure of computational cost.

Importantly, backbone flexibility is a key advantage of PATCHBELL that matched-backbone comparisons alone do not capture. Most competing methods depend on language representations and cannot operate on other powerful vision-only encoders such as DINOv3. In contrast, PATCHBELL requires neither text supervision nor backbone-specific adaptation, allowing it to exploit advances in visual foundation models directly. With the same DINOv3 backbone, PATCHBELL substantially outperforms RSIAT (87.28% vs. 70.86% Last accuracy). Notably, RSIAT's performance degrades substantially when transferring from its original pretrained backbone to CLIP ViT-L/14 or DINOv3, highlighting its sensitivity to backbone choice. While additional hyperparameter tuning might improve transfer, it would incur substantial computational overhead, particularly given RSIAT's already high update cost.

Together, these results demonstrate competitive accuracy at substantially lower update cost under matched backbones and broad compatibility across foundation models. We have accordingly refined the positioning of PATCHBELL relative to existing methods throughout the revised manuscript, from the abstract and introduction to the related work, method sections, and experimental evaluation.

Table 2 in the updated paper:

MethodAvg ↑Last ↑Update PFLOPs/task ↓Inference GFLOPs/image ↓Peak allocated GPU (GiB) ↓Peak reserved GPU (GiB) ↓
CLIP ViT-L/14
CLAP82.0881.2611.512162.0563.3053.486
MG-CLIP90.3383.4017.414162.43033.10134.299
RSIAT71.5757.18188.009163.64325.75230.475
PATCH BELL89.1582.362.107188.2771.8802.047
DINOv3 ViT-L/16 (vision-only)
RSIAT85.1670.86145.826126.94516.41516.902
PATCH BELL92.2587.281.634145.7801.2201.273
AuthorsOpenReview ↗

Response 2

W3. Robustness to distribution shift.

We agree that the effectiveness of frozen-encoder methods depends on the quality of their pretrained representations. To evaluate robustness beyond natural images, we added a medical-imaging experiment on PathMNIST-64 (Appendix B.3.2, Table 9), using identical backbones and incremental protocols across all methods. PATCHBELL achieves the highest Avg and Last accuracy, outperforming all competing methods. Notably, while the other non-replay methods struggle to transfer to this substantially different visual domain, PATCHBELL maintains strong performance. These results demonstrate its robustness to a significant domain shift, although settings where the frozen encoder fails to capture essential discriminative information remains a limitation.

Table 9 in the updated paper:

MethodBackboneAvg ↑Last ↑
CLAPCLIP ViT-L/1488.9383.19
MG-CLIPCLIP ViT-L/1479.4253.06
RSIATCLIP ViT-L/1472.3643.45
PATCHBELLCLIP ViT-L/1489.5286.46

Final note: We thank the reviewer for their constructive feedback, which has helped us strengthen both the experimental evaluation and the framing of our contributions. The revised manuscript incorporates these improvements, and we hope the additional evidence and clarifications will support a more positive reassessment of our work.

AuthorsOpenReview ↗

Follow up on W1

Following our previous response, we have completed two additional experiments to better characterize the impact of increasing the number of classes on PATCHBELL.

1. Scalability from 10 to 1,000 classes.

We added a systematic evaluation on ImageNet-1K, varying the number of classes from 10 to 1,000 using three nested random class orders, with and without Jacobian correction.

As expected, classification accuracy decreases as the number of classes grows, reflecting the increasing difficulty of the task. However, this degradation is gradual, with no sharp decline or performance collapse even at 1,000 classes. Also, the Jacobian correction consistently improves accuracy, with its contribution increasing from 1.20 percentage points at 10 classes to 3.92 points at 1,000 classes. This highlights the growing importance of likelihood correction as the number of class-specific models increases. Overall, PATCHBELL remains competitive at 1,000 classes, achieving 81.62% Last accuracy. These results are reported in the newly added Table 22 of the revised manuscript.

Table 22 in the updated paper (results are mean ± standard deviation (%) over three nested class orders):

Total classesClasses/taskWithout Jacobian AvgWithout Jacobian LastWith Jacobian AvgWith Jacobian Last
10198.62 ± 0.3497.67 ± 0.3199.51 ± 0.2998.87 ± 0.50
1001093.92 ± 0.3791.12 ± 0.2696.09 ± 0.0793.78 ± 0.20
2002091.47 ± 0.4488.33 ± 0.7994.02 ± 0.1791.21 ± 0.58
3003089.84 ± 0.5186.40 ± 0.6692.55 ± 0.2589.42 ± 0.44
5005087.25 ± 0.4482.61 ± 0.4890.31 ± 0.1986.17 ± 0.46
7007085.32 ± 0.3680.35 ± 0.3788.56 ± 0.1483.96 ± 0.45
1,00010083.01 ± 0.2277.70 ± 0.00386.45 ± 0.1281.62 ± 0.00

2. Impact on previously learned classes.

To distinguish the effect of increasing class competition from forgetting of previously learned classes, we additionally evaluated accuracy on a fixed set of first-task classes throughout the incremental process on ImageNet100, averaged over five random class orders.

As the number of classes increases from 10 to 100, average first-task accuracy decreases from 97.40% to 87.32%, demonstrating that the expanding class set makes classification more challenging. Importantly, first-task accuracy remains comparable to overall accuracy throughout the incremental process (87.32% vs. 87.28% at the final task). This indicates that early classes do not suffer disproportionate degradation relative to the full set of classes. So while adding classes increases classification difficulty, previously learned classes remain competitive with newly introduced ones, consistent with our goal of mitigating catastrophic forgetting.

Table 21 in the updated paper:

TaskSeen classesAll seen classes (%)First-task classes (%)
11097.40 ± 1.7797.40 ± 1.77
22096.12 ± 1.3695.80 ± 2.07
33094.61 ± 1.2694.60 ± 2.17
44093.80 ± 0.9693.36 ± 2.12
55092.44 ± 1.9092.00 ± 2.73
66091.75 ± 1.7090.80 ± 2.28
77090.71 ± 1.5190.12 ± 2.34
88089.32 ± 1.0388.44 ± 2.06
99087.80 ± 0.1487.60 ± 2.87
1010087.28 ± 0.0087.32 ± 2.83

Together, these experiments provide a more complete picture of PATCHBELL's behavior as the number of classes increases: classification becomes more challenging, the Jacobian correction becomes increasingly beneficial, and previously learned classes remain competitive with the overall class population.

We have incorporated both experiments into the revised manuscript and hope these additional results address the reviewer's concern regarding scalability to a large number of classes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We thank the reviewers for their constructive feedback. In response, we revised the manuscript and added controlled comparisons, new experiments, and clarifications. These additions address the main concerns and strengthen the paper.

1. Matched-backbone comparisons and efficiency. Under identical backbones and protocols (Table 2), PatchBELL achieves 82.36% final accuracy with CLIP ViT-L/14 versus 83.40% for MG-CLIP, with 8.3× fewer update FLOPs and 17.6× lower peak GPU memory. With DINOv3, PatchBELL achieves 87.28% versus 70.86% for RSIAT. These comparisons address the concern that gains stem solely from different backbones.

We also report inference computation and storage: inference cost grows with class count, and class statistics require 1.91 GiB for 1,000 classes. ImageNet-1K matched-backbone comparisons were infeasible during rebuttal due to competitors’ training time. They are underway and will be included in the camera-ready version. We also added a matched-backbone medical-domain comparison (point 4 below).

2. Likelihood comparability. We clarified the distinction between the probabilistic model in Section 3, which defines comparable full-dimensional Gaussian likelihoods, and its efficient implementation in PatchBELL (Section 4), which uses truncated scores as an approximation.

A new controlled comparison of full, truncated, and isotropic-residual formulations (Table 24) directly assesses this approximation. The isotropic-residual model defines a full-dimensional Gaussian and closely matches the full model, with 0.995 patch-ranking and 0.996 entropy-map correlation. All three achieve similar accuracy (86.90–87.28%). Thus, the method remains effective with valid full-dimensional likelihoods, while the experiment quantifies the approximation’s effects. We also clarified that Gaussianity tests provide empirical evidence, not proof of joint Gaussianity.

3. Scalability and old-class accuracy. Classification becomes harder as new classes introduce more competitors. A new study from 10 to 1,000 classes shows gradual accuracy degradation without any collapse (Table 22). A second experiment tracks first-task classes as new ones are added (Table 21). Their final accuracy remains comparable to overall accuracy (87.32% vs. 87.28%), showing that old classes remain as competitive as new classes, consistent with avoiding the forgetting PatchBELL targets.

4. Domain transfer. In a matched-backbone evaluation on the medical dataset PathMNIST-64, PatchBELL achieves the highest average and final accuracy among compared methods. While the two leading non-replay methods perform poorly on images from this domain, PatchBELL remains effective. As with any frozen-model based method, performance still depends on the representation quality.

5. Confidence maps. We added qualitative failure cases and a localization analysis on ImageNet100 (Table 20). We also clarified that patch confidence measures discrimination across classes, not necessarily support for the correct class. We also analyzed an additional signal for unreliable predictions (Table 23).

6. Reproducibility. The supplementary materials now include the full implementation, an ImageNet100 runner, confidence-map output code, and 20 additional qualitative examples. Code to reproduce all remaining experiments will be released upon publication.

Reviewer assessments. Reviewers aezn and b7t5 raised their scores after considering our responses and experiments. Reviewer U92S acknowledged the improvements and clarified their main remaining concern about likelihood comparability. We added a controlled likelihood experiment and clarified the matched-backbone comparisons, but received no further response. We respectfully ask that the final assessment considers the revised manuscript and substantive evidence, including the additions that have not been reassessed.

The revised paper presents a strong, practical approach to class-incremental learning: it adds classes without replay or parameter adaptation, combining competitive accuracy with substantially lower update costs. PatchBELL is encoder-agnostic, applicable to vision-only and vision-language models of different sizes, requiring only a single encoder pass per image of a new class. The new analyses strengthen its framing and empirical support, establishing PatchBELL as a scalable and flexible solution for class-incremental learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.