- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SAEF is a semantic-guided adaptive expert forest method for class-incremental learning (CIL). They aim to improve knowledge sharing between tasks. The method is built on a pre-trained backbone to extract features. Then, for each task, an adapter is trained to discriminate between the classes in the current task. While other previous methods use a selection mechanism at test time to select the appropriate adapter and isolate the rest, SAEF organizes these adapters into a hierarchical forest based on their semantics.
Their algorithm consists of three steps:
- Group tasks into clusters using CLIP text embeddings.
- Build a balanced tree in each cluster. Each expert captures shared knowledge related to the classes in that cluster.
- At inference time, use predictive entropy to select the appropriate path in the tree and identify the relevant experts for classification.
The method is evaluated on four benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet. They show improved results compared to prompt-based and adapter-based baselines, although some relevant baselines are missing. They also provide an extensive ablation study to assess the importance of each component.
Critical strengths
- Combining adapter-based methods with hierarchical knowledge sharing to organize and fuse adapters for CIL.
- Demonstrating improved performance compared to adapter-based and prompt-based methods.
- Providing a comprehensive ablation study.
- The method is clearly described in three stages, and the figures are well presented.
Critical weaknesses
- The method does not compare with these models: TUNA (1) and DAF (2). TUNA uses task-specific adapters, entropy-based selection, and a fused universal adapter, making it especially close to SAEF's motivation. The differences between TUNA and SAEF should also be explained in the related work section. DAF addresses the same knowledge-integration problem. These papers seem to have experimental settings similar to those of this paper, with slightly better results on some datasets, questioning the claim that this model is state-of-the-art.
- SAEF creates additional merged experts and combines their predictions. Figure 4 compares flat and hierarchical structures. However, the hierarchical structure can have more experts, and its advantage may be due to this reason. A fair comparison should include the same number of experts across all methods.
- Some ablation studies show very small changes (0.1–0.3%), such as those in Tables A7–A9. This shows that some of the design decisions have little impact on performance and might not be necessary.
(1) TUNA: Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning (ICCV 2025)
(2) DAF: Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning
Questions for authors
- Can we have a hybrid clustering approach (text + vision) and include it in the ablation study? Please also include random clustering to assess the importance of semantic grouping.
- How calibrated are expert entropies across nodes? Did you test any per-expert temperature calibration or normalization?
6 comments
Rebuttal by Authors (2/2)
Response 4 (Hybrid and random clustering ablations).
We thank the reviewer for this suggestion. We have expanded Appendix N, Table A8(a)–(b), with hybrid and random clustering and separate comparisons of grouping and merge criteria. Table A8(b) first fixes conceptual clusters and varies the merge criterion:
| Merging Criterion / Prototype | ImageNet-R (%) | ImageNet-R (%) |
|---|---|---|
| Visual cosine (Default) | 84.54 | 80.02 |
| Visual Euclidean | 84.57 | 80.25 |
| Text cosine | 84.35 | 79.63 |
| Text Euclidean | 84.34 | 79.63 |
| Hybrid (0.5) cosine | 84.55 | 79.99 |
| Hybrid (0.5) Euclidean | 84.55 | 79.99 |
| Adapter cosine | 84.43 | 79.83 |
| Adapter Euclidean | 84.43 | 79.87 |
| Random | 84.11 | 79.47 |
Table A8(a) then varies the clustering source while retaining the subsequent tree construction and inference:
| Clustering Feature / Grouping | ImageNet-R (%) | ImageNet-R (%) |
|---|---|---|
| Text + K-Means (Default) | 84.54 | 80.02 |
| Visual + K-Means | 84.25 | 79.25 |
| Hybrid (0.5) + K-Means | 84.55 | 79.98 |
| Random grouping | 84.17 | 79.57 |
The core innovation is the semantic hierarchy of usable prediction experts and their adaptive multi-level integration. Hybrid clustering closely matches the text default, visual clustering remains effective, and visual merge criteria exceed random pairing. These results show that the hierarchical framework is robust across prototype sources and similarity metrics, making its components replaceable while preserving effective knowledge reuse. We explain this distinction in the added Core Novelty and Significance of SAEF discussion in Appendix N.
Response 5 (Entropy calibration sensitivity).
We sincerely thank the reviewer for raising this important technical question. In our implementation (detailed in Appendix C.2), to prevent entropy magnitude inflation caused by the expanding label space across incremental tasks, we compute predictive entropy over the top- sorted softmax probabilities with , which acts as an implicit calibration and normalization across hierarchy nodes:
To assess whether performance is sensitive to this calibration, we evaluated an uncalibrated baseline that computes standard Shannon entropy directly over all seen classes without top- truncation, yielding 84.50% average accuracy and 80.01% final accuracy on ImageNet-R. This negligible difference (84.50% vs. 84.54%) confirms that SAEF's entropy-guided tree routing and confidence-based weighting are inherently stable and largely insensitive to calibration or class-count scaling, as the relative confidence order between candidate child nodes remains consistent. We will include this calibration sensitivity analysis in Appendix C.2 of the revised manuscript.
We thank the reviewer again and welcome any further questions.
Response to Rebuttal: Remaining Concerns
Thanks to the authors for the detailed response and the new experiments. Some of my concerns are addressed. TUNA and DAF are now discussed in the related work. The hybrid and random clustering results are added in Table A8. The matched-capacity comparison is added in Table A16. But some concerns remain.
1. Comparison with TUNA and DAF
The new comparison is only on ImageNet-R, where SAEF is better than both methods. But my concern was about the other datasets. The baseline numbers in Table 1 are exactly the same as Table 1 of TUNA (same backbone, seed 1993 and splits). DAF also uses the same setting. So we can compare with their reported results directly:
| Avg / Final (%) | CIFAR-100 | ImageNet-R | ImageNet-A | ObjectNet |
|---|---|---|---|---|
| SAEF (Table 1) | 94.53 / 90.60 | 84.54 / 80.02 | 73.39 / 64.91 | 76.38 / 66.58 |
| TUNA (ICCV'25, Table 1) | 94.44 / 90.74 | 84.22 / 79.42 | 73.78 / 64.78 | 76.46 / 66.32 |
| DAF (NeurIPS'26, Table 1) | 94.58 / 91.15 | 84.01 / 79.63 | 72.06 / 62.54 | 76.11 / 66.37 |
SAEF is not the best in 4 of the 8 numbers: CIFAR-100 (both metrics), ImageNet-A average and ObjectNet average. Where SAEF is better, the gap is between 0.13% and 0.60%. But the paper still says SAEF has the highest average and final accuracy on all four benchmarks (Sec. 4.2), that it is SOTA on all four benchmarks (App. N), and that it "consistently outperforms recent competitors" (Figure 4 caption).
Please add TUNA, DAF, MiN and RAA to Table 1 for all four datasets, and change these claims. Based on the results, SAEF is competitive with recent methods, but it is not state-of-the-art on all benchmarks.
2. The gains from the hierarchy and semantic guidance are small
Thanks for adding Table A16. But if we look at it together with Tables A1, A5 and A8, the gains are close to the noise between seeds.
- SAEF is better than Flat-all by 0.47/0.35 and better than Random tree by 0.43/0.55. These are single-seed numbers, and Table A1 reports a std of 0.25/0.42 on ImageNet-R.
- Leaf + Root is very close to TUNA's design (one task adapter and one universal adapter). It already gets 84.34/79.72 in Table A5. So the intermediate levels, which are the new part compared to TUNA, add only 0.20/0.30.
- Random grouping (84.17/79.57) and random merging (84.11/79.47) are within about 0.5% of the default. ViT-visual clustering is even worse than random grouping in final accuracy (79.25 vs 79.57). The default is also not the best option, since Visual Euclidean gets 84.57/80.25. To me this looks like noise, not a clear trend.
The authors say these small differences show robustness. But if random choices are within 0.5% of the semantic ones, then the semantic part is not the main reason for the gain. This was my point about Tables A7–A9.
Can you run SAEF, Random tree, Flat-all, Leaf + Root and Random grouping with the different seeds of Table A1 and report mean ± std? If the gaps are not stable, the claims about semantic guidance should be toned down.
Summary
The rebuttal addresses my points about related work, hybrid/random clustering and matched capacity. The comparison with TUNA/DAF on the other datasets and the SOTA claims are still open. The new results also show that the semantic hierarchy adds less than the paper claims. I keep my score for now. To keep it, I want to see the claims in Sec. 4.2, App. N and Figure 4 corrected, and TUNA and DAF added to Table 1.
Rebuttal by Authors (1/2)
Thank you for your careful review and constructive suggestions. We address your concerns below. The corresponding revisions are highlighted in light blue.
Response 1 (Comparison and relationship with TUNA and DAF).
We thank the reviewer for this suggestion. SAEF's distinction is its multi-level hierarchy of usable experts. As clarified in Section 2.2, TUNA [1] combines task-specific and universal adapters through entropy-based selection, while DAF [2] constructs a global adapter. SAEF additionally builds shared internal experts and fuses predictions along selected paths.
Section 4.2, Figure 4 adds comparisons on ImageNet-R using supervised IN21K, MAE [3], and DINO [4] ViT-B/16 weights. Each cell reports average/final accuracy (%):
SAEF exceeds TUNA and DAF on both metrics across all three settings. These comparisons support the benefit of progressively shared internal experts across diverse pre-trained representations.
[1] Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning,ICCV 2025.
[2] Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning, NeurIPS 2026.
[3] Masked Autoencoders Are Scalable Vision Learners, CVPR 202.
[4] Emerging properties in self-supervised vision transformers, ICCV 2021.
Response 2 (Matched expert capacity).
We thank the reviewer for this comment. The benefit of SAEF comes from organizing and selectively using experts at multiple levels. As discussed in Appendix J, throughput and deployment latency are more pressing constraints than lightweight adapter storage alongside a frozen backbone [1, 2]. We have added equal-capacity comparisons in Appendix V, Table A16, with a corresponding discussion in Section 4.4, to isolate this structural benefit.
All variants store 2T−1 adapters and share Gaussian prototype classifier calibration. Random tree retains conceptual clustering, height-aware balancing, and adaptive inference, randomizing only within-cluster merges. Flat-all entropy-weights the same leaf and merged adapters; matched-capacity MOS [3] uses the same expanded pool.
SAEF improves average/final accuracy by 0.43/0.55 points over Random tree and 0.47/0.35 points over Flat-all. Expanding MOS's adapter pool yields 82.83/77.91, compared with 82.96/77.93 for standard MOS: simply adding adapters provides no gain. The key is how accumulated knowledge is organized and used. SAEF turns merged adapters into a hierarchy of prediction experts and selectively fuses their outputs, achieving higher accuracy at equal storage with logarithmic path queries instead of evaluating all 19 adapters.
[1] Computationally budgeted continual learning: What does matter? CVPR. 2023.
[2] Real-time evaluation in online continual learning: A new hope. CVPR. 2023.
[3] Mos: Model surgery for pre-trained model-based class-incremental learning. AAAI. 2025.
Response 3 (Core contribution and component robustness).
We thank the reviewer for this perspective. Our core innovation is a semantics-guided hierarchy in which every internal node is itself a usable prediction expert. This enables selective knowledge reuse at multiple levels, beyond flat adapter routing or a single shared module. We have added the Core Novelty and Significance of SAEF discussion in Appendix N to distinguish this contribution from the replaceable merging operators, distance metrics, and prototype extractors evaluated in Tables A7–A9. Their small performance differences demonstrate robustness across these choices.
These individual choices are modular components rather than the core contribution. The consistent results in Tables A7–A9 demonstrate that SAEF's hierarchical framework remains robust across alternative merging operators, similarity metrics, and prototype sources.