- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PATCH BELL, a statistical approach to class-incremental learning. It keeps the visual encoder frozen and models each class using Gaussian distributions of patch features, with Jacobian correction and entropy-based aggregation. New classes are added without gradient-based training or replaying old data. The results are strong on ImageNet-R and ImageNet100, but the likelihood formulation and the overall computational cost need further investigation.
Critical strengths
The update mechanism is simple and avoids modifying old class models, preventing forgetting caused by parameter updates.
The method achieves strong results on ImageNet-R and ImageNet100. The ablations also show that both Jacobian correction and entropy weighting help.
It avoids repeated optimization and replay, while naturally producing spatial confidence maps for explaining predictions.
Critical weaknesses
Each class uses a different PCA subspace, with (k=500<d=1024) in the main experiments. It is unclear whether the Jacobian correction alone makes the resulting likelihoods directly comparable. Better accuracy does not, by itself, validate this probabilistic interpretation.
Old class statistics remain unchanged, but adding new classes can still change entropy weights and predictions. The method prevents forgetting caused by parameter updates, but does not necessarily guarantee stable accuracy on old classes.
The method avoids gradient-based training, but still requires covariance estimation and eigendecomposition. The paper focuses on update FLOPs and does not sufficiently evaluate model storage.
Fig. 6 shows several examples with seemingly reasonable confidence maps, but failure cases are missing. Since the patch weight measures how discriminative a patch is across classes rather than whether it supports the correct class, high-confidence regions may still be irrelevant or misleading. This raises concerns about the faithfulness of the explanations and their potential impact on classification accuracy.
Every test patch must be evaluated against all class models, so the main classification cost grows roughly linearly with the number of classes. The paper does not report actual inference latency, throughput, or peak memory, making its scalability claims difficult to assess.
The main experiments use DINOv3 ViT-L/16, while the baselines use different backbones and training procedures. This makes it difficult to isolate the gains from the proposed method itself. PATCH BELL also does not outperform every baseline on every benchmark.
Questions for authors
How does the Jacobian correction make likelihoods comparable when each class uses a different truncated PCA subspace? Could the authors validate this using a shared subspace or an explicit model of the discarded dimensions?
Could the authors report old-class accuracy across incremental stages to verify whether keeping class statistics unchanged also preserves recognition performance?
Could the authors report inference latency, throughput, peak memory, and model storage at different class scales, ideally compared with a simple classifier using the same backbone?
Without gradient-based training, how much time is spent on covariance estimation and eigendecomposition, and how much do these steps contribute to the total update cost?
Could the authors provide some failure cases and analyze when PATCH BELL assigns high confidence to irrelevant or misleading patches, such as background regions or other objects? Can the method make correct predictions while relying on the wrong patches, and how often do such patches lead to misclassification? What is their impact on explanation faithfulness and classification accuracy, and can these cases be detected or mitigated?
At this stage, I still have several concerns that I hope the authors can address, particularly regarding the likelihood formulation, the overall computational cost, and the failure cases of patch-based explanations. I would appreciate further clarification and supporting evidence. Depending on the authors’ responses and any additional results, I am open to revising my score.
2 comments
Response 2
W3, W5, W6, Q3, Q4. (continued).
Computational cost. To isolate the computational overhead of PATCHBELL, Appendix Table 16 decomposes the total cost into frozen encoder computation and PATCHBELL-specific operations, for both CLIP ViT-L/14 and DINOv3 ViT-L/16. We report FLOPs per incremental task during updates and per image during inference. For each setting, we compare the complete PATCHBELL pipeline against encoder-only feature extraction, with the difference measuring the additional computation introduced by our method. The results show negligible update overhead (approximately 0.5%) and moderate inference overhead (approximately 16% relative to encoder computation). While inference cost grows linearly with the number of classes, class-conditional likelihood evaluations are independent and can be parallelized on GPUs, reducing their impact on practical latency.
Table 16 in the updated paper:
| Backbone | Configuration | Update PFLOPs/task ↓ | Inference GFLOPs/image ↓ |
|---|---|---|---|
| CLIP ViT-L/14 | Full PATCHBELL | 2.107 | 188.277 |
| Encoder only | 2.097 | 162.024 | |
| PATCHBELL overhead | 0.010 | 26.253 | |
| DINOv3 ViT-L/16 | Full PATCHBELL | 1.634 | 145.780 |
| Encoder only | 1.626 | 125.680 | |
| PATCHBELL overhead | 0.008 | 20.100 |
Storage requirements. PATCHBELL stores only fixed-size class statistics, with memory growing linearly in the number of classes. Table 17 quantifies these requirements across backbones and class counts, showing that even with 1,000 classes and a 1,024-dimensional backbone, total storage is only 1.91 GiB. Importantly, these statistics are stored in CPU memory and do not require persistent GPU allocation. As demonstrated in Table 2, PATCHBELL has the lowest peak GPU memory consumption among all compared methods, confirming that its statistical modeling introduces modest memory requirements despite maintaining independent models for each class.
Table 17 in the updated paper:
| Backbone | Feature dimension | 100 classes | 200 classes | 1,000 classes |
|---|---|---|---|---|
| CLIP ViT-B/16 | 768 | 0.143 GiB | 0.287 GiB | 1.433 GiB |
| CLIP ViT-L/14 | 1,024 | 0.191 GiB | 0.382 GiB | 1.911 GiB |
| DINOv3 ViT-B/16 | 768 | 0.143 GiB | 0.287 GiB | 1.433 GiB |
| DINOv3 ViT-L/16 | 1,024 | 0.191 GiB | 0.382 GiB | 1.911 GiB |
W4, Q5. Confidence-map faithfulness and failure cases.
We thank the reviewer for raising this important distinction. Our confidence maps measure how discriminative each patch is across classes, rather than whether it supports the ground-truth class. We therefore added representative failure cases in Figures 7 and 8, illustrating two limitations: confidence concentrating on a salient but potentially irrelevant object in multi-object scenes, and high-confidence regions appearing on both the target object and irrelevant background areas.
While directly quantifying the frequency and classification impact of such failures would require additional region-level annotations, we provide quantitative evidence of explanation faithfulness through deletion and insertion experiments (Appendix Tables 18 and 19). These evaluations assess whether highlighted regions genuinely influence classification, both for PATCHBELL and an independent classifier. Across 5,000 images, our confidence maps remain competitive with dedicated attribution methods, demonstrating that they generally identify meaningful, decision-relevant regions.
We emphasize that PATCHBELL is not proposed as a dedicated explainability method. The confidence maps are an intrinsic by-product of classification, providing informative spatial explanations without additional training or attribution computation. The new failure cases clarify their limitations, while the quantitative evaluations demonstrate their practical value.
Final note: We thank the reviewer again for their constructive and detailed feedback, which has helped us strengthen the theoretical framing, computational evaluation, and reliability of the confidence maps. We hope the revised manuscript and additional evidence adequately address the main concerns and support a positive reassessment of our work.
Response 1
We thank the reviewer for recognizing the efficiency of the incremental update and the empirical contribution of the Jacobian and entropy components. We have revised the theoretical claims, expanded the resource analysis, added matched-backbone comparisons, and included qualitative failure cases for the confidence maps.
W1, Q1. Jacobian correction with truncated, class-specific subspaces.
We agree with the mathematical distinction. For full-dimensional whitening (k=d) with nonsingular covariance, Equation (3) is the Gaussian log-density, including the covariance log-determinant, and likelihoods are comparable in the common feature space. For k<d, the class-specific projections are non-invertible: correcting the retained covariance volume does not account for the discarded dimensions or define an exact full-space density. We have made this limitation explicit in Section 3.2 and Appendix A.3 and now describe the default k=500 score as a computational approximation, not an exact change-of-variables likelihood. Empirically, the whitening-dimension sweep (Figure 4a and Appendix Table 10) gives 87.28% Last at k=500 and 86.90% at the full k=1024. These results support the truncated formulation as an efficient approximation that preserves the classification performance of the exact full-dimensional model.
W2, Q2. Preservation of old knowledge versus old-class recognition.
We have refined our claims throughout the manuscript to distinguish parameter-induced forgetting from changes in recognition accuracy. PATCHBELL guarantees that previously learned class models remain unchanged, independently of class order or task partitioning (Tables 4 and 8). However, adding new classes introduces additional competing likelihoods and changes entropy-based confidence weights, potentially affecting old-class predictions. Thus, PATCHBELL eliminates forgetting caused by parameter updates, not the inherent difficulty of distinguishing an expanding set of classes.
W3, W5, W6, Q3, Q4. Matched-backbone fairness, computational cost, inference scaling, and model storage.
We agree that matched-backbone comparisons are essential. We therefore added Table 2, evaluating PATCHBELL and leading baselines under identical encoders and protocols. For the limited rebuttal period we report ImageNet100 results.
With CLIP ViT-L/14, PATCHBELL achieves accuracy comparable to MG-CLIP, with 8.3× lower update FLOPs and 17.6× lower peak GPU memory. While inference requires more FLOPs, class-conditional evaluations are independent and can be parallelized efficiently on GPUs. We report FLOPs rather than wall-clock time to provide a hardware-independent measure of computational cost.
Importantly, backbone flexibility is a key advantage of PATCHBELL that matched-backbone comparisons alone do not capture. Most competing methods depend on language representations and cannot operate on other powerful vision-only encoders such as DINOv3. In contrast, PATCHBELL requires neither text supervision nor backbone-specific adaptation, allowing it to exploit advances in visual foundation models directly. With the same DINOv3 backbone, PATCHBELL substantially outperforms RSIAT (87.28% vs. 70.86% Last accuracy). Notably, RSIAT's performance degrades substantially when transferring from its original pretrained backbone to CLIP ViT-L/14 or DINOv3, highlighting its sensitivity to backbone choice. While additional hyperparameter tuning might improve transfer, it would incur substantial computational overhead, particularly given RSIAT's already high update cost.
Together, these results demonstrate competitive accuracy at substantially lower update cost under matched backbones and broad compatibility across foundation models. We have accordingly refined the positioning of PATCHBELL relative to existing methods throughout the revised manuscript, from the abstract and introduction to the related work, method sections, and experimental evaluation.
Table 2 in the updated paper: