acceptodds
Under review as a conference paper at ICLR 2027

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

Abstract

Class-Incremental Learning (CIL) requires models to learn new classes without forgetting old ones. A common method is to freeze a pre-trained model and train a new, lightweight adapter for each task. While this prevents forgetting, it treats the learned knowledge as a simple, unstructured collection and fails to use the relationships between tasks. To this end, we propose the Semantic-guided Adaptive Expert Forest (SAEF), a new method that organizes adapters into a structured hierarchy for better knowledge sharing. SAEF first groups tasks into conceptual clusters based on their semantic relationships. Then, within each cluster, it builds a balanced expert tree by creating new adapters from merging the adapters of similar tasks. At inference time, SAEF finds and activates a set of relevant experts from the forest for any given input. The final prediction is made by combining the outputs of these activated experts, weighted by how confident each expert is. Experiments on several benchmark datasets show that SAEF achieves SOTA performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, Pre-Trained Models, Parameter-Efficient Adapters

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ojmkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAEF is a semantic-guided adaptive expert forest method for class-incremental learning (CIL). They aim to improve knowledge sharing between tasks. The method is built on a pre-trained backbone to extract features. Then, for each task, an adapter is trained to discriminate between the classes in the current task. While other previous methods use a selection mechanism at test time to select the appropriate adapter and isolate the rest, SAEF organizes these adapters into a hierarchical forest based on their semantics.

Their algorithm consists of three steps:

  1. Group tasks into clusters using CLIP text embeddings.
  2. Build a balanced tree in each cluster. Each expert captures shared knowledge related to the classes in that cluster.
  3. At inference time, use predictive entropy to select the appropriate path in the tree and identify the relevant experts for classification.

The method is evaluated on four benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet. They show improved results compared to prompt-based and adapter-based baselines, although some relevant baselines are missing. They also provide an extensive ablation study to assess the importance of each component.

Critical strengths
  1. Combining adapter-based methods with hierarchical knowledge sharing to organize and fuse adapters for CIL.
  2. Demonstrating improved performance compared to adapter-based and prompt-based methods.
  3. Providing a comprehensive ablation study.
  4. The method is clearly described in three stages, and the figures are well presented.
Critical weaknesses
  1. The method does not compare with these models: TUNA (1) and DAF (2). TUNA uses task-specific adapters, entropy-based selection, and a fused universal adapter, making it especially close to SAEF's motivation. The differences between TUNA and SAEF should also be explained in the related work section. DAF addresses the same knowledge-integration problem. These papers seem to have experimental settings similar to those of this paper, with slightly better results on some datasets, questioning the claim that this model is state-of-the-art.
  2. SAEF creates additional merged experts and combines their predictions. Figure 4 compares flat and hierarchical structures. However, the hierarchical structure can have more experts, and its advantage may be due to this reason. A fair comparison should include the same number of experts across all methods.
  3. Some ablation studies show very small changes (0.1–0.3%), such as those in Tables A7–A9. This shows that some of the design decisions have little impact on performance and might not be necessary.

(1) TUNA: Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning (ICCV 2025)

(2) DAF: Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning

Questions for authors
  1. Can we have a hybrid clustering approach (text + vision) and include it in the ablation study? Please also include random clustering to assess the importance of semantic grouping.
  2. How calibrated are expert entropies across nodes? Did you test any per-expert temperature calibration or normalization?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for your careful review and constructive suggestions. We address your concerns below. The corresponding revisions are highlighted in light blue.


Response 1 (Comparison and relationship with TUNA and DAF).

We thank the reviewer for this suggestion. SAEF's distinction is its multi-level hierarchy of usable experts. As clarified in Section 2.2, TUNA [1] combines task-specific and universal adapters through entropy-based selection, while DAF [2] constructs a global adapter. SAEF additionally builds shared internal experts and fuses predictions along selected paths.

Section 4.2, Figure 4 adds comparisons on ImageNet-R using supervised IN21K, MAE [3], and DINO [4] ViT-B/16 weights. Each cell reports average/final accuracy (%):

MethodIN21K Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)MAE Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)DINO Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)
SAEF (Ours)84.542 / 80.0251.393 / 40.4278.603 / 73.95
TUNA [1]84.175 / 79.5849.775 / 39.4877.006 / 70.80
DAF [2]83.974 / 79.4836.292 / 28.1573.341 / 67.37

SAEF exceeds TUNA and DAF on both metrics across all three settings. These comparisons support the benefit of progressively shared internal experts across diverse pre-trained representations.

[1] Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning,ICCV 2025.
[2] Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning, NeurIPS 2026.
[3] Masked Autoencoders Are Scalable Vision Learners, CVPR 202.
[4] Emerging properties in self-supervised vision transformers, ICCV 2021.


Response 2 (Matched expert capacity).

We thank the reviewer for this comment. The benefit of SAEF comes from organizing and selectively using experts at multiple levels. As discussed in Appendix J, throughput and deployment latency are more pressing constraints than lightweight adapter storage alongside a frozen backbone [1, 2]. We have added equal-capacity comparisons in Appendix V, Table A16, with a corresponding discussion in Section 4.4, to isolate this structural benefit.

All variants store 2T−12T-1 adapters and share Gaussian prototype classifier calibration. Random tree retains conceptual clustering, height-aware balancing, and adaptive inference, randomizing only within-cluster merges. Flat-all entropy-weights the same leaf and merged adapters; matched-capacity MOS [3] uses the same expanded pool.

MethodImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
SAEF (Ours)84.5480.02
Random tree84.1179.47
Flat-all experts84.0779.67
MOS [3] (matched capacity)82.8377.91

SAEF improves average/final accuracy by 0.43/0.55 points over Random tree and 0.47/0.35 points over Flat-all. Expanding MOS's adapter pool yields 82.83/77.91, compared with 82.96/77.93 for standard MOS: simply adding adapters provides no gain. The key is how accumulated knowledge is organized and used. SAEF turns merged adapters into a hierarchy of prediction experts and selectively fuses their outputs, achieving higher accuracy at equal storage with logarithmic path queries instead of evaluating all 19 adapters.

[1] Computationally budgeted continual learning: What does matter? CVPR. 2023.
[2] Real-time evaluation in online continual learning: A new hope. CVPR. 2023.
[3] Mos: Model surgery for pre-trained model-based class-incremental learning. AAAI. 2025.


Response 3 (Core contribution and component robustness).

We thank the reviewer for this perspective. Our core innovation is a semantics-guided hierarchy in which every internal node is itself a usable prediction expert. This enables selective knowledge reuse at multiple levels, beyond flat adapter routing or a single shared module. We have added the Core Novelty and Significance of SAEF discussion in Appendix N to distinguish this contribution from the replaceable merging operators, distance metrics, and prototype extractors evaluated in Tables A7–A9. Their small performance differences demonstrate robustness across these choices.

These individual choices are modular components rather than the core contribution. The consistent results in Tables A7–A9 demonstrate that SAEF's hierarchical framework remains robust across alternative merging operators, similarity metrics, and prototype sources.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Response 4 (Hybrid and random clustering ablations).

We thank the reviewer for this suggestion. We have expanded Appendix N, Table A8(a)–(b), with hybrid and random clustering and separate comparisons of grouping and merge criteria. Table A8(b) first fixes conceptual clusters and varies the merge criterion:

Merging Criterion / PrototypeImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
Visual cosine (Default)84.5480.02
Visual Euclidean84.5780.25
Text cosine84.3579.63
Text Euclidean84.3479.63
Hybrid (0.5) cosine84.5579.99
Hybrid (0.5) Euclidean84.5579.99
Adapter cosine84.4379.83
Adapter Euclidean84.4379.87
Random84.1179.47

Table A8(a) then varies the clustering source while retaining the subsequent tree construction and inference:

Clustering Feature / GroupingImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
Text + K-Means (Default)84.5480.02
Visual + K-Means84.2579.25
Hybrid (0.5) + K-Means84.5579.98
Random grouping84.1779.57

The core innovation is the semantic hierarchy of usable prediction experts and their adaptive multi-level integration. Hybrid clustering closely matches the text default, visual clustering remains effective, and visual merge criteria exceed random pairing. These results show that the hierarchical framework is robust across prototype sources and similarity metrics, making its components replaceable while preserving effective knowledge reuse. We explain this distinction in the added Core Novelty and Significance of SAEF discussion in Appendix N.


Response 5 (Entropy calibration sensitivity).

We sincerely thank the reviewer for raising this important technical question. In our implementation (detailed in Appendix C.2), to prevent entropy magnitude inflation caused by the expanding label space across incremental tasks, we compute predictive entropy over the top-MM sorted softmax probabilities with M=∣Ct∣M=|\mathcal C_t|, which acts as an implicit calibration and normalization across hierarchy nodes:

To assess whether performance is sensitive to this calibration, we evaluated an uncalibrated baseline that computes standard Shannon entropy directly over all seen classes without top-MM truncation, yielding 84.50% average accuracy and 80.01% final accuracy on ImageNet-R. This negligible difference (84.50% vs. 84.54%) confirms that SAEF's entropy-guided tree routing and confidence-based weighting are inherently stable and largely insensitive to calibration or class-count scaling, as the relative confidence order between candidate child nodes remains consistent. We will include this calibration sensitivity analysis in Appendix C.2 of the revised manuscript.


We thank the reviewer again and welcome any further questions.

Reviewer_ojmkOpenReview ↗

Response to Rebuttal: Remaining Concerns

Thanks to the authors for the detailed response and the new experiments. Some of my concerns are addressed. TUNA and DAF are now discussed in the related work. The hybrid and random clustering results are added in Table A8. The matched-capacity comparison is added in Table A16. But some concerns remain.

1. Comparison with TUNA and DAF

The new comparison is only on ImageNet-R, where SAEF is better than both methods. But my concern was about the other datasets. The baseline numbers in Table 1 are exactly the same as Table 1 of TUNA (same backbone, seed 1993 and splits). DAF also uses the same setting. So we can compare with their reported results directly:

Avg / Final (%)CIFAR-100ImageNet-RImageNet-AObjectNet
SAEF (Table 1)94.53 / 90.6084.54 / 80.0273.39 / 64.9176.38 / 66.58
TUNA (ICCV'25, Table 1)94.44 / 90.7484.22 / 79.4273.78 / 64.7876.46 / 66.32
DAF (NeurIPS'26, Table 1)94.58 / 91.1584.01 / 79.6372.06 / 62.5476.11 / 66.37

SAEF is not the best in 4 of the 8 numbers: CIFAR-100 (both metrics), ImageNet-A average and ObjectNet average. Where SAEF is better, the gap is between 0.13% and 0.60%. But the paper still says SAEF has the highest average and final accuracy on all four benchmarks (Sec. 4.2), that it is SOTA on all four benchmarks (App. N), and that it "consistently outperforms recent competitors" (Figure 4 caption).

Please add TUNA, DAF, MiN and RAA to Table 1 for all four datasets, and change these claims. Based on the results, SAEF is competitive with recent methods, but it is not state-of-the-art on all benchmarks.

2. The gains from the hierarchy and semantic guidance are small

Thanks for adding Table A16. But if we look at it together with Tables A1, A5 and A8, the gains are close to the noise between seeds.

  • SAEF is better than Flat-all by 0.47/0.35 and better than Random tree by 0.43/0.55. These are single-seed numbers, and Table A1 reports a std of 0.25/0.42 on ImageNet-R.
  • Leaf + Root is very close to TUNA's design (one task adapter and one universal adapter). It already gets 84.34/79.72 in Table A5. So the intermediate levels, which are the new part compared to TUNA, add only 0.20/0.30.
  • Random grouping (84.17/79.57) and random merging (84.11/79.47) are within about 0.5% of the default. ViT-visual clustering is even worse than random grouping in final accuracy (79.25 vs 79.57). The default is also not the best option, since Visual Euclidean gets 84.57/80.25. To me this looks like noise, not a clear trend.

The authors say these small differences show robustness. But if random choices are within 0.5% of the semantic ones, then the semantic part is not the main reason for the gain. This was my point about Tables A7–A9.

Can you run SAEF, Random tree, Flat-all, Leaf + Root and Random grouping with the different seeds of Table A1 and report mean ± std? If the gaps are not stable, the claims about semantic guidance should be toned down.

Summary

The rebuttal addresses my points about related work, hybrid/random clustering and matched capacity. The comparison with TUNA/DAF on the other datasets and the SOTA claims are still open. The new results also show that the semantic hierarchy adds less than the paper claims. I keep my score for now. To keep it, I want to see the claims in Sec. 4.2, App. N and Figure 4 corrected, and TUNA and DAF added to Table 1.

Reviewer_FobgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies class-incremental learning with pre-trained models without replaying original samples. The motivation is that existing collections of task adapters cannot fully exploit task relationships. SAEF uses CLIP class-name embeddings to cluster tasks, then merges adapters into a balanced expert tree based on visual prototypes, and additionally constructs a global root. At inference, SAEF proposes selecting paths in each tree according to lower prediction entropy, fusing predictions from global, internal, and leaf nodes. The paper reports outperforming selected competing methods on four benchmarks, and supports the role of hierarchical reuse through ablation, long-sequence, and forgetting analyses.

Critical strengths
  1. The problem is clearly defined and practically valuable. Considering adapter reuse, cross-task recognition, and inference cost simultaneously is more complete than focusing only on parameter-level forgetting.
  2. The design of internal nodes as actual prediction experts is worth studying. Compared with a tree structure that only has routing meaning, multi-level expert fusion provides a concrete and implementable knowledge combination scheme.
  3. The reported performance covers multiple difficult datasets. In Table 1, it leads on both metrics across four datasets, with final accuracy advantages of 2.68/2.96 percentage points on ImageNet-A/ObjectNet, respectively.
  4. The appendix experiments are relatively rich and include some honest limitations. They already include five seeds, no CLIP, fusion replacement, 50/100 tasks, and forgetting rate; it also acknowledges MagMax equivalence, linear storage growth, internal coverage not equaling classification accuracy, and theoretical approximation conditions.
Critical weaknesses
  1. There is no strict causal link between motivation and method. The motivation of this paper is reasonable, it is intuitive that tasks with similar semantics share visual features. However, the implementation of SAEF does not seem to fully meet the requirements of the motivation. There are mainly 3 issues: 1) Similar category names do not mean that the visual objects themselves are similar; 2) The process of obtaining prototypes via CLIP more reflects the prior distribution during CLIP training. Because the fine-grainedness of tasks is inconsistent, visual objects may be semantically consistent, but the semantic vectors output by CLIP are not necessarily similar; 3) Semantically similar objects also cannot guarantee that two adapters are suitable for merging, as this is affected by multiple factors, such as task difficulty, sample distribution, and sample quality. For example, there may be two similar categories, but one of them has some salient visual feature and is easier to distinguish, while the other does not; after merging, the features of one of the categories may degrade.

  2. The experimental tasks are combinations of random categories, with no specific associations within a task. Averaging the embeddings of all classes in a task dilutes the internal multimodal structure. In the setting of 20 classes per task, the means of different tasks may all be close to the semantic center of the full data. Therefore, intuitively, SAEF should be more sensitive to task combinations, and this kind of tree structure should be more suitable for settings with strong associations among classes within a task.

  3. The method does not completely isolate irrelevant concepts. Eq. (8) merges all cluster roots to form a global expert, and Eq. (11) activates, for each input, the global root and one path in all K trees. Therefore, selectivity is mainly reflected within each tree, and mixing across concept trees always exists. Low-entropy weights can attenuate some contributions, but they do not guarantee that the weights of irrelevant experts are small.

  4. Existing ablations cannot uniquely identify the contribution of the semantic structure. Table A8 shows no text clustering gain on ObjectNet, and only 0.09/0.13 percentage points on CIFAR-100. On the other hand, the final improvement on ImageNet-A reaches 4.01, indicating that external text priors may be quite important for this dataset. In Table A5, the full path improves only 0.20/0.30 over leaf plus global root, while in Table 2, the average accuracy of flat and SAEF differs by 0.09. Only after adding random trees and an equal-compute flat entropy ensemble can it be determined whether the gains come from the “semantic hierarchy” or from multi-expert ensembling, classifier calibration, and a larger storage budget. In addition, this also raises concerns about comparison fairness due to the introduction of CLIP’s external knowledge priors.

  5. The comparison does not sufficiently cover recent pre-trained model-based class-incremental learning methods. Comparisons with more recent methods, such as MiN [1], RAA [2], and Fly-CL [3], are still missing. All three address class-incremental learning with pre-trained models. In particular, RAA also investigates adapter combination, while Fly-CL provides an efficiency-oriented comparison. The latest comparison method in the main text is MOS (AAAI 2025), but its actual submission time should be 2024, and its performance has a certain gap compared with the latest CIL methods. Therefore, for the comparison experiments, please compare at least one baseline method from within the past year. The following methods can be used as references.

    [1] Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning, NeurIPS 2025.

    [2] Random Amalgamation of Adapters for Flatter Loss Landscapes: Towards Class-Incremental Learning with Better Stability, AAAI 2026.

    [3] Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning, ICLR 2026.

  6. Generalizability across pretrained backbones is insufficiently validated. Although the main experiments and Appendix I cover ImageNet-21K and ImageNet-1K pretraining, respectively, both use ViT-B/16 and pretraining setups that have been used for several years. The evaluation therefore remains limited to one backbone architecture. Experiments with other architectures and newer pretrained weights would help determine whether SAEF's gains persist across different pretrained representations.

  7. The anonymous implementation differs from the manuscript in several core methodological choices. There are mainly 6 differences: 1) Sec. 3.2 selects the number of clusters automatically using the silhouette score, whereas the provided training procedure disables automatic selection and uses a fixed cluster count; 2) Eq. (5) computes visual task prototypes using the first-task adapter A1, whereas the implementation uses each task's own adapter; 3) Eq. (6) selects nodes for merging based on cosine similarity, whereas the implementation uses Euclidean distance between prototypes; 4) Eq. (2) defines the classification loss over all seen classes, whereas the adapter-training code restricts it to the current task's classes; 5) Eq. (10) computes entropy over all seen classes, whereas the implementation sums only over the largest M probabilities without renormalization, where M is the number of classes in the current task; 6) the manuscript's standard evaluation excludes optional early exit, whereas the provided evaluation procedure enables it by default. These discrepancies affect the training objective, forest construction, and expert-selection procedure, raising concerns about which method was actually evaluated.

Questions for authors
  1. Referring to W1, what is the causal link between the paper's motivation and method? Is there any theoretical derivation or empirical evidence showing that the similarity between CLIP-based task prototypes predicts visual similarity and the benefits of merging the corresponding adapters?
  2. How sensitive is SAEF to task composition, particularly when comparing randomly grouped classes with semantically coherent groups?
  3. How does SAEF suppress irrelevant concepts when the global expert and a path from every tree are activated? Can irrelevant experts receive high weights despite low predictive entropy?
  4. Can you isolate the contribution of the semantic hierarchy using random-tree and flat entropy-ensemble baselines with matched computation, storage, classifier calibration, and access to CLIP priors?
  5. How does SAEF compare with at least one recent baseline, such as MiN, RAA, or Fly-CL under the same evaluation protocol?
  6. How generalizable is SAEF under other model architectures/pretrained weights?
  7. What is the actual code implementation used in the paper? Is there an error in the paper's description, or an error in the implementation of the anonymous code?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Thank you for your careful review and constructive suggestions. We address your concerns below. The corresponding revisions are highlighted in light green.


Response 1 (Connecting motivation to the method).

We thank the reviewer for this question. Since old data is unavailable for joint training and adapter weights naturally reside in the input visual subspace, SAEF directly achieves visual feature sharing in weight space—using conceptual clustering to isolate unrelated tasks, visual similarity to merge aligned adapters without conflict, and hierarchical routing to activate shared coarse-to-fine features. Conceptual clustering is a structural innovation that organizes tasks into coherent domains before hierarchical merging; this design is independent of the choice of CLIP. We address the three issues separately.

Response to Issue (1): Category names versus visual similarity. Stage 1 provides coarse conceptual grouping, and Stage 2 selects merge pairs using visual similarity within each cluster. Visual prototypes can also directly guide Stage 1. We use text by default because language supplies high-level conceptual anchors that are less affected by style, texture, and background variations under domain shift. We have expanded Appendix N, Table A8(a)–(b) to separate these two stages and explain this choice.

Response to Issue (2): CLIP priors and semantic granularity. The expanded Table A8(a) compares text, visual, hybrid, and random clustering:

Clustering Feature / MethodImageNet-R Aˉ\bar{A} (%)ImageNet-R ATA_T (%)
Text + K-means (Ours)84.5480.02
Visual + K-means84.2579.25
Hybrid (0.5 Text + 0.5 Visual) + K-means84.5579.98
Random grouping84.1779.57

Text (84.54%) outperforms Visual (84.25%), and both surpass Random grouping (84.17%). This confirms the effectiveness of semantic priors for conceptual clustering, while demonstrating that SAEF remains highly robust to the specific choice of clustering feature.

Response to Issue (3): Compatibility of merged adapters. We have added visual-, text-, adapter-weight-, hybrid-, and randomly guided merging comparisons in Table A8(b). Visual cosine reaches 84.54/80.02, versus 84.43/79.83 for adapter-weight cosine and 84.11/79.47 for random merging. Figure 5(c)–(d) and Appendix L, Table A5 further show useful internal-node predictions: Leaf + Root exceeds Leaf Only (84.34% versus 83.97%), the full path reaches 84.54%, and pruning intermediate nodes reduces accuracy. These results support the value of merging even when related tasks differ in difficulty or feature salience: merged experts can contribute complementary discriminative information, while the retained leaf experts preserve task-specific specialization. Conceptual grouping, visually guided merging, and adaptive fusion thus make shared and specialized knowledge jointly useful.

Merge CriterionImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
Visual Cosine (SAEF Default)84.5480.02
Visual Euclidean84.5780.25
Text Cosine84.3579.63
Text Euclidean84.3479.63
Hybrid (0.5 Text + 0.5 Visual) Cosine84.5579.99
Hybrid (0.5 Text + 0.5 Visual) Euclidean84.5579.99
Adapter Weight Cosine84.4379.83
Adapter Weight Euclidean84.4379.87
Random Merging84.1179.47

Response 2 (Sensitivity to task composition).

We thank the reviewer for this observation. SAEF is effective under both random and semantically coherent task compositions because its bottom-up construction uses relative pairwise prototype rankings rather than absolute prototype variance. Even when random 20-class task means approach the dataset mean, relatively similar task pairs remain candidates for aligned knowledge sharing. Under the prototype-to-parameter approximations in Appendix E.4, greater relative similarity favors smaller merge-induced structural deviation (Property 3).

We have added Appendix W, Table A17, comparing random assignment with CLIP- and ViT-based class grouping on ImageNet-R:

MethodRandom Grouping Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)CLIP Grouping Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)ViT Grouping Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)
SAEF (Ours)84.54 / 80.0284.41 / 80.2783.67 / 80.22
MOS82.96 / 77.9382.68 / 77.5481.08 / 77.72

SAEF exceeds MOS on both metrics under every partition, with final accuracy between 80.02% and 80.27%. SAEF performs consistently well with both semantically homogeneous and heterogeneous task compositions. Relative prototype rankings guide compatible merges in either setting, allowing effective hierarchical sharing across different degrees of intra-task semantic coherence.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Response 3 (Cross-tree sharing and irrelevant experts).

We thank the reviewer for this critical question. Continual-learning tasks share transferable visual knowledge, so SAEF is designed for confidence-weighted cross-domain integration. An expert can provide useful visual primitives—such as shapes or textures—even when it is associated with a different task.

To clarify this distinction, we explicitly evaluated expert-selection accuracy in Section 4.4, Table 3, and expanded Appendix Q. At T=10T=10, exact leaf-selection accuracy is 53.27% for learned gating, 69.32% for flat lowest-entropy selection, and 54.52% for SAEF. Notably, although SAEF does not have the highest exact expert-selection accuracy, it achieves the highest classification performance (84.54/80.02, versus 83.47/78.67 for flat selection). This is because the hierarchy supports correct classification beyond exact leaf retrieval: shallow decisions guide retrieval into relevant semantic regions, and deeper experts refine the prediction. Even when the exact leaf is missed, internal experts can retain knowledge from the ground-truth task's neighborhood, while related leaves contribute transferable capabilities. This explains the gap between leaf-selection and classification performance.

We have also added aggregate fusion weights (Eq. (12)) for target and non-target paths on ImageNet-R (T=10T=10) in Appendix Q:

DatasetRelevant weightIrrelevant weight
ImageNet-R0.630.37

Crucially, entropy routing selects the lowest-entropy branch within each tree, restricting non-target retrieval to selected semantic subtrees and favoring visually compatible experts. These experts can provide complementary feature information and constructive regularization, with the target path retaining the larger average weight. Figure 5(c) and Appendix L, Table A5 show that Full Adaptive Path exceeds both Single Tree and Leaf Only. The benefit comes from structured integration: semantic routing makes non-target knowledge useful, and multi-level fusion preserves effective predictions even when exact leaf retrieval fails.


Response 4 (Controlling capacity, computation, calibration, and priors).

We thank the reviewer for requesting these controls. We have added matched-capacity controls in Appendix V, Table A16, discussed them in Section 4.4, and expanded the prior-source ablations in Appendix N, Table A8(a).

1. External CLIP priors. As detailed in Response 1, frozen-ViT clustering alone achieves 84.25/79.25 on ImageNet-R, exceeding standard MOS at 82.96/77.93. This establishes an effective visual-only version and separates conceptual clustering from the choice of external text features.

2. Storage, calibration, and ensembling (Section 4.4, Appendix J & Appendix V, Table A16). The benefit of SAEF comes from organizing and selectively using experts at multiple levels. As discussed in Appendix J, deployment latency is a more pressing constraint than lightweight adapter storage [1, 2]. To isolate our structural benefit at equal capacity (2T−1=192T-1=19 adapters) and shared Gaussian prototype calibration (Appendix C.3), we compare against: Random tree (randomizing only within-cluster merge pairs), Flat-all experts (entropy-weighting all leaf and merged adapters), and matched-capacity MOS [3] (using the same expanded pool).

MethodImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
SAEF (Ours)84.5480.02
Random tree84.1179.47
Flat-all experts84.0779.67
MOS [3] (matched capacity)82.8377.91

SAEF improves average/final accuracy by 0.43/0.55 points over Random tree and 0.47/0.35 points over Flat-all, whereas expanding MOS (82.83/77.91 vs. standard 82.96/77.93) provides no gain. The key is how accumulated knowledge is organized and used: SAEF achieves higher accuracy at equal storage with logarithmic path queries instead of evaluating all 19 adapters.

[1] Computationally budgeted continual learning: What does matter? CVPR. 2023.
[2] Real-time evaluation in online continual learning: A new hope. CVPR. 2023.
[3] Mos: Model surgery for pre-trained model-based class-incremental learning. AAAI. 2025.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Response 5 (Comparison with recent CIL methods).

We thank the reviewer for recommending these methods. The revised evaluation includes MiN [1], RAA [2], and Fly-CL [3] under the same exemplar-free protocol. We have added these comparisons to Section 4.2, Figure 4, using supervised IN21K, MAE [4], and DINO [5] ViT-B/16 weights on ImageNet-R:

MethodIN21K Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)MAE Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)DINO Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)
SAEF (Ours)84.542 / 80.0251.393 / 40.4278.603 / 73.95
Fly-CL [3]73.641 / 67.1735.851 / 26.3068.997 / 61.82
RAA [2]84.413 / 79.8739.653 / 34.0577.340 / 71.03
MiN [1]84.484 / 80.2733.908 / 25.8378.043 / 73.53

SAEF achieves the highest average accuracy across the three settings. On MAE, its margins over Fly-CL, RAA, and MiN are 15.54, 11.74, and 17.48 points, respectively. The extended comparison demonstrates the benefit of hierarchical reuse alongside recent noise-based, adapter-combination, and efficiency-oriented approaches.

[1] Masked Autoencoders Are Scalable Vision Learners, CVPR 2022.
[2] Emerging properties in self-supervised vision transformers, ICCV 2021.
[3] Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning, NeurIPS 2025.
[4] Random Amalgamation of Adapters for Flatter Loss Landscapes: Towards Class-Incremental Learning with Better Stability, AAAI 2026.
[5] Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning, ICLR 2026.


Response 6 (Generalization across pre-trained representations).

We thank the reviewer for this question. The revised experiments test transfer across supervised and self-supervised representations. Alongside ImageNet-1K results in Appendix I, we have added multi-backbone comparisons to Section 4.2, Figure 4: supervised IN21K, masked autoencoding with MAE [1], and self-distillation with DINO [2], using ViT-B/16 on ImageNet-R:

MethodIN21K Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)MAE Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)DINO Aˉ\bar{\mathcal{A}} / AT\mathcal{A}_T (%)
SAEF (Ours)84.542 / 80.0251.393 / 40.4278.603 / 73.95
Fly-CL [5]73.641 / 67.1735.851 / 26.3068.997 / 61.82
RAA [4]84.413 / 79.8739.653 / 34.0577.340 / 71.03
MiN [3]84.484 / 80.2733.908 / 25.8378.043 / 73.53

SAEF reaches 51.393/40.42 with MAE and 78.603/73.95 with DINO, with the highest average accuracy among the compared methods in each setting. These results establish robustness across distinct pre-training paradigms within ViT-B/16.

[1] Masked Autoencoders Are Scalable Vision Learners, CVPR 2022.
[2] Emerging properties in self-supervised vision transformers, ICCV 2021.
[3] Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning, NeurIPS 2025.
[4] Random Amalgamation of Adapters for Flatter Loss Landscapes: Towards Class-Incremental Learning with Better Stability, AAAI 2026.
[5] Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning, ICLR 2026.


Response 7 (Implementation and manuscript alignment).

We thank the reviewer for carefully examining our code. We have aligned the anonymous repository and manuscript with the evaluated benchmark configuration. The earlier repository contained development and ablation settings; the updated version uses the reported configuration. We corrected two descriptions: Eq. (2) now computes cross-entropy over current-task classes Ct\mathcal C_t, and Eq. (10) sums the top-MM softmax probabilities at their original values, where M=∣Ct∣M=|\mathcal C_t|. The other four evaluated choices are automatic silhouette-based cluster selection (Section 3.2), visual prototypes from the shared first-task adapter A1A_1 (Eq. (5)), cosine-based merge pairing (Eq. (6)), and standard full traversal with τe=0\tau_e=0; Section 4.4, Table 2, and Appendix Q, Table A11 separately study optional early exit. We also revised Appendix D, Algorithms 1–2, to state construction, early exit, and cached routing explicitly. We have updated the supplementary materials accordingly.


We thank the reviewer again and welcome any further questions.

Reviewer_2qFtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Semantic-guided Adaptive Expert Forest (SAEF) for exemplar-free class-incremental learning with a frozen pretrained ViT and task-specific adapters. At each incremental stage, SAEF clusters observed tasks using CLIP text prototypes and constructs balanced expert trees by merging adapters according to visual-prototype similarity. During inference, it follows low-entropy branches in each tree and fuses predictions from the global root and selected path experts. The main contribution is hierarchical expert organization and adaptive reuse.

Critical strengths
  1. Internal nodes provide predictive evidence alongside hierarchical routing, enabling shared and task-specific knowledge to contribute jointly.
  2. SAEF achieves the highest average and final accuracy among the methods in Table 1 on all four benchmarks. On ObjectNet, final accuracy exceeds MOS by 2.96 percentage points.
Critical weaknesses
  1. The benefit of semantic organization is not fully isolated. SAEF stores approximately 2T−12T-1 adapters, nearly twice the number of task-specific adapters. Matched-budget comparisons with random balanced trees and flat ensembles are needed to distinguish semantic structure from additional experts and ensembling.
  2. The theoretical justification remains largely heuristic. The gradient-subspace argument for Property 1 does not by itself establish the claimed property of optimized adapter weights. The relationship between visual-prototype similarity and parameter-subspace similarity is assumed in Appendix E.4 without empirical validation.
  3. The efficiency analysis contains unresolved inconsistencies. Algorithm 2 accepts τe\tau_e but omits the entropy-based early-exit condition. Appendix G also assumes that the second tree has an average search depth of approximately one, which requires justification, particularly without early exit.
Questions for authors
  1. How much improvement remains over random balanced trees and flat entropy-weighted ensembles under matched storage and inference budgets?
  2. Can you empirically validate the proposed relationship between visual-prototype similarity, parameter conflicts, and post-merge performance?
  3. How is early exit implemented, and what are the measured depths and forward-pass counts for each tree? Please clarify whether routing outputs are cached and reconcile these measurements with Table 2.
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for your careful review and constructive suggestions. We address your concerns below. The corresponding revisions are highlighted in light yellow.


Response 1 (Isolating semantic organization).

We thank the reviewer for this suggestion. The benefit of SAEF comes from organizing and selectively using experts at multiple levels. As discussed in Appendix J, throughput and deployment latency are more pressing constraints than lightweight adapter storage alongside a frozen backbone [1, 2]. We have added equal-capacity comparisons in Appendix V, Table A16, with a corresponding discussion in Section 4.4, to isolate this structural benefit.

All variants store 2T−12T-1 adapters and share Gaussian prototype classifier calibration. Random tree keeps conceptual clustering, height-aware balancing, and adaptive inference, randomizing only within-cluster merge pairs. Flat-all experts entropy-weights the same leaf and merged adapters, while matched-capacity MOS [3] uses the same expanded pool.

MethodImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
SAEF (Ours)84.5480.02
Random tree84.1179.47
Flat-all experts84.0779.67
MOS [3] (matched capacity)82.8377.91

SAEF improves average/final accuracy by 0.43/0.55 points over Random tree and 0.47/0.35 points over Flat-all. Expanding MOS's adapter pool yields 82.83/77.91, compared with 82.96/77.93 for standard MOS: simply adding adapters provides no gain. The key is how the accumulated knowledge is organized and used. SAEF turns merged adapters into a hierarchy of prediction experts and selectively fuses their outputs, achieving higher accuracy at equal storage with logarithmic path queries instead of evaluating all 19 adapters.

[1] Computationally budgeted continual learning: What does matter? CVPR. 2023.
[2] Real-time evaluation in online continual learning: A new hope. CVPR. 2023.
[3] Mos: Model surgery for pre-trained model-based class-incremental learning. AAAI. 2025.


Response 2 (Theory and empirical validation of merging).

We thank the reviewer for raising this point. Our core innovation is a semantics-guided hierarchy in which every internal node is a usable prediction expert. The merge criterion is a modular design choice, and SAEF remains robust across similarity metrics and merging operators. Section 3.3 and Appendix E provide analytical motivation for the default visual criterion under explicit approximations; the new experiments test its practical effectiveness and the framework's robustness.

We have added a direct empirical comparison in Appendix N, Table A8(b): conceptual clustering, balancing, and inference remain fixed while the merge-pair criterion varies. We also explicitly distinguish the hierarchical contribution from individual operator choices in the new Core Novelty and Significance of SAEF discussion in Appendix N.

Merge CriterionImageNet-R Aˉ\bar{\mathcal{A}} (%)ImageNet-R AT\mathcal{A}_T (%)
Visual Cosine (SAEF Default)84.5480.02
Visual Euclidean84.5780.25
Text Cosine84.3579.63
Text Euclidean84.3479.63
Hybrid (0.5 Text + 0.5 Visual) Cosine84.5579.99
Hybrid (0.5 Text + 0.5 Visual) Euclidean84.5579.99
Adapter Weight Cosine84.4379.83
Adapter Weight Euclidean84.4379.87
Random Merging84.1179.47

Visual similarity matches or slightly exceeds adapter-weight similarity and improves over random merging, supporting it as a practical merge criterion. Together with Appendices M and O, these experiments show stable performance across merging operators, metrics, and prototype extractors. The evidence supports visual similarity for the default construction and establishes hierarchical knowledge reuse as a robust contribution across component choices.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Response 3 (Early exit, caching, and efficiency measurements).

We thank the reviewer for identifying these ambiguities. We have revised Algorithm 2 in Appendix D to explicitly include the early-exit condition and cached routing outputs. Traversal stops at a leaf or when Hnode<τeH_{\mathrm{node}}<\tau_e. Outputs and entropies from both children are cached and reused for fusion; each evaluated expert requires one forward pass.

We have updated Appendix G to count both trees explicitly using measured depths and expanded Table 2 in Section 4.4 with per-tree depths, forward counts, query-based speedups, and wall-clock times. With path depth counted in visited nodes, each tree requires 2dˉk−12\bar d_k-1 evaluations, giving 1+∑k(2dˉk−1)1+\sum_k(2\bar d_k-1) including the global expert. Table 2 reports final-stage inference measurements on CIFAR-100:

ConfigurationAˉ\bar{\mathcal{A}} (%) ↑\uparrowTree 1 Depth ↓\downarrowTree 1 Forward ↓\downarrowTree 2 Depth ↓\downarrowTree 2 Forward ↓\downarrowTheo. Speedup ↑\uparrowTime (s) ↓\downarrow
Flat Structure (Baseline)94.44--------1.00$\times$1747.76
SAEF (τe=0.0\tau_e = 0.0)94.534.007.004.467.921.26$\times$1506.69
SAEF (τe=0.5\tau_e = 0.5)94.491.712.421.822.643.30$\times$700.33
SAEF (τe=1.0\tau_e = 1.0)94.241.201.401.221.445.21$\times$588.85
SAEF (τe=2.0\tau_e = 2.0)93.961.001.001.001.006.67$\times$324.25

At τe=1.0\tau_e=1.0, SAEF achieves a measured 2.97$\times$ speedup with only a 0.29-point accuracy decrease relative to full traversal. These measurements show that upper-level experts often suffice for confident predictions: cached adaptive traversal delivers a substantial measured speedup while preserving nearly all of the full-path accuracy.


We thank the reviewer again and welcome any further questions.

Reviewer_2qFtOpenReview ↗

Response to Rebuttal

I’d like to thank the author for their response, which has largely cleared up my questions regarding efficiency and the number of additional adapters. I still have the following two concerns:

  1. The derivation for Property 1 only demonstrates the subspace properties of the gradient. It does not seem to directly imply that the optimized weights possess the same properties.
  2. Based on the experimental results, there was no significant difference in performance between randomly selecting trees and selecting those with the highest visual similarity. This makes me wonder whether the degree of similarity among visual prototypes for different tasks is distinguishable.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Dear Reviewers:

Thank you for your careful reviews and constructive suggestions. The revised manuscript strengthens the evidence for semantic hierarchical reuse, adds recent comparisons, and clarifies the evaluated implementation. Our point-by-point responses explain each revision and its supporting results.

  • Reviewer 2qFt: the corresponding revisions are highlighted in light yellow.
  • Reviewer Fobg: the corresponding revisions are highlighted in light green.
  • Reviewer ojmk: the corresponding revisions are highlighted in light blue.

The additional experiments address the following concerns:

  • Reviewers 2qFt, Fobg, and ojmk — Isolating hierarchical reuse. Equal-capacity comparisons share adapter count and classifier calibration. SAEF exceeds Random tree by 0.43/0.55 points and Flat-all experts by 0.47/0.35 points in average/final accuracy on ImageNet-R, supporting structured expert reuse at equal storage (Appendix V, Table A16).
  • Reviewers 2qFt, Fobg, and ojmk — Semantic priors and merge compatibility. New visual, text, hybrid, and random controls distinguish clustering from merge pairing. Visual-only clustering remains competitive, while language provides high-level conceptual anchors under domain shift. Visual merge criteria match or slightly exceed adapter-weight criteria. These results support CLIP as a replaceable prior and visual similarity as a practical merge criterion (Appendix N, Table A8).
  • Reviewers Fobg and ojmk — Recent methods and pre-training. Comparisons with MiN, RAA, Fly-CL, TUNA, and DAF cover IN21K, MAE, and DINO ViT-B/16 weights. SAEF achieves the highest average accuracy across these settings, supporting transfer across supervised and self-supervised representations (Section 4.2, Figure 4).
  • Reviewer 2qFt — Inference efficiency. Measured depths and forward counts account for both trees. Optional early exit reduces final-stage inference time from 1747.76 s to 588.85 s on CIFAR-100 (Table 2; Appendix G).
  • Reviewer ojmk — Entropy sensitivity. Full entropy and top-M entropy differ by only 0.04/0.01 points in average/final accuracy on ImageNet-R, supporting robustness to entropy truncation (Appendix C.2).
  • Reviewer Fobg — Task composition and cross-tree sharing. SAEF exceeds MOS under random and semantically coherent task partitions. Path ablations show the benefit of combining shared and specialized predictions, and measured weights show a dominant target-path contribution of 0.63, alongside 0.37 from other paths (Figure 5(c)–(d); Appendices L, Q, and W).

In addition, we have made the following major revisions to the manuscript:

  • Reviewers 2qFt, Fobg, and ojmk — Hierarchical reuse. Added equal-capacity MOS, Flat-all, and Random tree controls with shared calibration (Section 4.4; Appendix V, Table A16).
  • Reviewers 2qFt, Fobg, and ojmk — Clustering and merging. Added hybrid/random clustering and alternative merge-criterion comparisons (Appendix N, Table A8(a)–(b)).
  • Reviewers Fobg and ojmk — Recent methods and pre-training. Added comparisons with MiN, RAA, Fly-CL, TUNA, and DAF using IN21K, MAE, and DINO weights (Section 4.2, Figure 4).
  • Reviewer 2qFt — Inference efficiency. Added measured depths, forward counts, and final-stage inference times for both trees (Section 4.4, Table 2; Appendix G).
  • Reviewer Fobg — Cross-tree sharing. Added target/non-target path fusion-weight measurements (Appendix Q).
  • Reviewer Fobg — Task composition. Added comparisons under random and semantically coherent task partitions (Appendix W, Table A17).
  • Reviewer Fobg — Main-text presentation. Moved expert-selection results and path interpretation from the appendix into Section 4.4, Table 3.
  • Reviewer ojmk — Entropy sensitivity. Added full-entropy versus top-M sensitivity results (Appendix C.2).

Together, these revisions support SAEF's central contribution: a hierarchy that selectively reuses shared and task-specific knowledge with efficient inference and stable component choices.

We thank the reviewers again and welcome any further questions.

Best regards,

The Authors of the Paper

Dear Reviewers,

Thank you for your efforts so far. As we are halfway through the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.