acceptodds
Under review as a conference paper at ICLR 2027

DualTrack: Accessible Knowledge Transfer for Continual Learning under Long-Tailed Distributions

Abstract

Long-tailed continual learning requires models to acquire new classes from imbalanced task streams while retaining previously observed classes without replaying historical images. The challenge extends beyond catastrophic forgetting: few-shot classes provide unreliable representation estimates, and added experts can make historical knowledge inaccessible even when their parameters remain unchanged. We present DualTrack, a framework that couples stable coordinates, statistical routing, protected adaptation, and tail-class transfer. A frozen ViT-B/16 backbone is augmented with early shared orthogonal LoRA modules and late general/specific LoRA branches. An independent routing stream maintains class-conditional diagonal Gaussian statistics to decide whether a new class should reuse an existing expert or receive additional capacity. When an expert absorbs a class, rank expansion preserves its initial function, freezes the old rank prefix, and trains only newly allocated directions; historical gradient-direction projection further restricts updates to directions compatible with prior adaptation. Finally, image-free class prototypes are transferred to tail classes using visual similarity, cross-layer route overlap, and a count-adaptive shrinkage coefficient. We evaluate on long-tailed continual-learning benchmarks and compare DualTrack with replay, regularization, prompt, expandable-expert, and prototype-based baselines under matched ViT backbones.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, long-tailed learning, knowledge transfer, class-incremental learning, low-rank adaptation, expert routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QNJeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DualTrack, a replay-free framework for long-tailed class-incremental learning. It introduces a dual-stream architecture that separates Gaussian-based expert routing from LoRA adaptation. The framework further incorporates adaptive expert expansion and prototype-based knowledge transfer to improve the learning of tail classes while preserving historical knowledge. Experiments on six benchmarks show competitive performance. Ablation studies and robustness evaluations provide additional validation of the proposed method.

Critical strengths
  1. The paper provides a relatively comprehensive experimental evaluation across six benchmarks, demonstrating competitive performance against multiple representative continual learning baselines. The ablation studies offer useful evidence for several major components.

  2. The proposed route-aware prototype transfer is a reasonable approach to addressing unreliable feature estimates for tail classes. By incorporating visual similarity, expert-route overlap, and class-frequency-dependent weighting, the method enables knowledge transfer from historical classes without storing previous images.

Critical weaknesses
  1. The paper's presentation and writing could be improved. Figure 1 contains excessive information and is difficult to follow, while Figure 3 uses a somewhat informal visual style and contains a spelling error "Guassian router". The ordering of baselines in Table 1 is also somewhat confusing and lacks a clear rationale, organizing them chronologically by publication year would make the comparison more intuitive. In addition, the Ethics Statement section is left empty, and some descriptions in the appendix are repetitive, suggesting that the manuscript would benefit from more careful proofreading and editing.

  2. The discussion of related work is insufficient in two aspects. First, the subsection Continual learning and parameter-efficient adaptation only covers works published in or before 2024, overlooking recent advances. Although DualPrompt and CL-LoRA are discussed elsewhere, the distinctions between DualTrack and existing parameter-efficient adaptation strategies are not sufficiently clarified. Second, given the central role of prototype-based prediction and tail-class knowledge transfer in this paper, relevant prototype-based continual learning methods, such as LADA [1] DIA [2], and GPA [3], should also be discussed. A more thorough comparison with these approaches especial the prototype subcomponents would help better establish the novelty and technical contributions of DualTrack.

  3. The evaluation of long-tailed learning remains limited in terms of imbalance severity and dataset diversity. All six main benchmarks use the same imbalance ratio of 10, which is relatively mild compared with more challenging long-tailed recognition settings. Experiments on more challenging or naturally imbalanced datasets, such as Places-LT or iNaturalist, under suitable class-incremental protocols would provide stronger evidence for the method's generalizability. More detailed head-, medium-, and tail-class improvements would also help clarify its effectiveness under severe imbalance.

[1] LADA: Scalable Label-Specific CLIP Adapter for Continual Learning. ICML 2025

[2] Dynamic Integration of Task-Specific Adapters for Class Incremental Learning. CVPR 2025

[3] A Tiny Change, A Giant Leap: Long-Tailed Class-Incremental Learning via Geometric Prototype Alignment. ICCV 2025

Questions for authors
  1. Section 4 reports a prototype-fusion weight of 1.0, while Appendices B.5 and B.9 describe a neutral prototype-only inference mode in which Specific expert outputs are disabled. Could the authors clarify the exact inference configurations used for the main results? In particular, how does expert routing directly contribute to the final predictions when neutral prototype-only inference is adopted?

  2. Appendices B.3 and B.9 indicate that the reference configuration uses old_lr_mult = 0, implying that eligible historical experts receive no effective parameter updates even when historical gradient projection is applied. Please clarify whether gradient-direction protection is actually active in the reported experiments and provide evidence for its contribution?

9 comments

We thank Reviewer QNJe for the valuable comments. In the revised manuscript, we have added recent related models to the main comparison table, included performance overhead experiments on additional datasets, provided more detailed ablation studies, and introduced experiments under more extreme imbalance scenarios. Furthermore, we have simplified the framework diagram in the main text, retaining only an intuitive illustration of the core mechanisms and data flow. Meanwhile, we have provided more detailed and explicit explanations of our model's mechanisms in the appendix to reduce potential misinterpretations. By demonstrating the rationality of our architectural design with experimental data, we have further elevated the rigor of the paper.

Q1:Presentation Issues

Regarding the presentation of the paper, Figure 1 clarifies the data flow so that readers can follow the entire training process. Figure 3 illustrates how routing decisions are made during training and inference and how they differ between the two stages. We have improved the color scheme. (Spelling errors.) The revised main table will list the models by year. We have also added an ethics statement and clarified the descriptions in the appendix.

Q2:Insufficient Related Work

We have added more recent models, including TUNA and CoRE. We could not make direct performance comparisons with methods such as LADA, DIA, and GPA because the code was not sufficiently clear.But we found more recent research on extending expert models in class-incremental scenarios[1, 2, 3]: CoRE, TUNA, and SEMA.

DatasetLong-tail settingMethodFinal accuracy (%)Wall-clock time (min)Peak process GPU memory (MiB)Final trainable parameters (M)
CUB200CUB-LTTUNA83.8337.244,8943.959
CUB200CUB-LTCaRE83.1962.3721,8162.152
CUB200CUB-LTSEMA69.1441.004,8300.154
CIFAR-100IR=10TUNA85.45136.892,2423.806
CIFAR-100IR=10CaRE85.57335.5514,8481.906
CIFAR-100IR=10SEMA81.3788.127,1760.077
CIFAR-100IR=100TUNA85.18111.392,2503.806
CIFAR-100IR=100CaRE85.09312.3515,0301.906
CIFAR-100IR=100SEMA78.1083.817,1300.077
TinyImageNetIR=10TUNA87.5187.054,1923.959
TinyImageNetIR=10CaRE88.00701.2715,0142.152
TinyImageNetIR=10SEMA84.38432.527,8280.154
TinyImageNetIR=100TUNA87.1881.754,1923.959
TinyImageNetIR=100CaRE87.53587.3615,0902.152
TinyImageNetIR=100SEMA82.80282.819,6920.154
CUB200CUB-LTDual86.2716.443,4400.369
CIFAR-100IR=10Dual85.0161.371,9800.319
CIFAR-100IR=100Dual84.7952.331,9800.319
TinyImageNetIR=10Dual88.2757.233,2800.369
TinyImageNetIR=100Dual87.7350.763,2800.369

Note: CUB-LT uses linear long-tailed sampling with a minimum retention ratio of 0.1. After discrete sampling, the maximum and minimum class counts are 30/2, yielding an actual ratio of 15:1.

Q3:Limited Evaluation

We have added a more challenging 1:100 imbalance setting and have also conducted experiments on Places-LT, a naturally imbalanced dataset. We report the results by head, middle, and tail groups:

DatasetMethodSeedH 1--3 (per task; mean)M 4--6 (per task; mean)T 8--10 (per task; mean)Mean across all 10 tasks
cifar100DualTrack199385.0 / 85.9 / 86.7; 85.8086.3 / 84.9 / 74.6; 81.9385.2 / 87.0 / 87.2; 86.4884.73
cifar100cl_lora199368.7 / 63.8 / 63.2; 65.2376.8 / 85.0 / 69.8; 77.2084.0 / 86.5 / 87.8; 86.1076.03
cifar100keeplora199391.4 / 75.3 / 72.1; 79.6081.8 / 79.2 / 74.6; 78.5381.1 / 73.9 / 79.0; 78.0078.78
cifar100CaRE199385.7 / 87.4 / 89.1; 87.4086.6 / 84.8 / 72.4; 81.2782.2 / 85.5 / 87.4; 85.0385.09
cifar100SEMA199383.6 / 75.7 / 88.7; 82.6780.4 / 80.6 / 49.1; 70.0374.5 / 84.8 / 84.4; 81.2378.10
cifar100TUNA199385.6 / 86.2 / 89.2; 87.0086.7 / 84.0 / 74.1; 81.6084.3 / 86.5 / 87.3; 86.0385.18
tinyimagenetCaRE199387.6 / 88.9 / 85.8; 87.4385.5 / 87.6 / 86.0; 86.3786.7 / 90.5 / 85.9; 87.7087.53
tinyimagenetSEMA199382.3 / 79.7 / 78.3; 80.1075.0 / 84.5 / 78.4; 79.3082.9 / 92.2 / 86.6; 87.2382.80
tinyimagenetTUNA199388.0 / 88.7 / 85.0; 87.2384.3 / 86.9 / 85.0; 85.4086.1 / 91.3 / 86.1; 87.8387.18
tinyimagenetDualTrack199388.6 / 90.2 / 86.7; 88.5085.5 / 87.6 / 86.2; 86.4088.2 / 90.7 / 85.6; 88.1687.73
MethodFull-set final task-macro (%)Incremental task-macro mean (%)Train time (min)Peak allocated GPU memory (MiB)Trainable parameters (M)
DualTrack46.919856.6976264.512,4490.447
TUNA45.500256.4520207.763,5294.213
CaRE46.087256.6488986.989,8562.553
cl_lora35.132446.4400283.188,7630.276
kp_lora38.796047.4512210.444,8510.983

Q4:Inference Configuration and Gradient Protection

The ablation results for the prototype fusion weight are reported below for CUB200 with seed=2026.

SettingFull-set micro (%)Full-set class-macro (%)
inference, final features: Specific scale=036.243336.9232
inference, final features: Specific scale=0.586.226086.9065
inference, final features: Specific scale=177.873778.0313
Training with Specific residual on86.226086.9065
Training with Specific residual off31.291030.7290
Prototype fusion weightFull-set microFull-set class macroFull-set task mean
0.070.6489%70.8898%70.8056%
0.576.3048%76.6063%76.5482%
1.085.9860%86.3065%86.9103%

We apologize for the confusion caused by the appendix and will clarify the inference procedure and the prototype fusion weight. Classification uses two paths. One computes cosine similarities to the stored class prototypes, and the other uses a linear classification head. The two outputs are added to obtain the final prediction vector, as follows:

Reviewer_AKbtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors explore long tailed class incremental learning using retrained transformer. The suggest that for tail classes, weak evidence means route selection will be uncertain while unrestricted sharing improves transfer but exposes historical paths to representation drift. Their hypothesis is that stability and accessibility should be coupled through a shared coordinate system between tail transfer and expert routing. Towards this they propose the DUAL TRACk framework which is a replacement free framework that couples stable coordinates, statistical routing, protected adaptation and tail class transfer. The authors experiment across 6 major datasets and establish the efficacy of their approach.

Critical strengths
  1. The paper combines several strong and intuitive concepts like statistical routing, early shared LoRA, late general LoRA and late specific LoRA modules into design components which together construct the DUAL TRACK framework.
  2. Each component introduced in the framework serves a specific purpose, which is easy to understand intuitively, and sufficient effort has been put into properly justifying each of these components empirically through ablations like that in Tables 3,4 and 5.
  3. Further the multi seed results also sufficiently establish the reliability of the proposed framework.
  4. Regardless of the number of components contributing to the framework, it only optimises a mere 3% of the total trainable parameters of MoE-Adapters. Thus, making it a competitive parameter efficient method among the lowest of the PEFT baselines.
Critical weaknesses
  1. Ablation section could further include reversed configuration of the LoRA modules along with all shared, all specific configurations etc. Reversal ablation could strengthen the design choice of the LoRA configuration.
  2. What ensures that the General LoRA and the specific LoRA does not end up learning redundant information? Is there a specific constraint imposed on these in order to ensure that they actually differ.
  3. Including a sensitivity analysis with regard to the choice of the split point, that's which layer separates the early LoRAs and late LoRAs could help further justify the current design.
  4. To ensure that the routing logic works optimally, reporting the routing accuracy is recommended. This would give us more insight on whether the current Gaussian routing algorithm is helping us to its optimum or could be improved further.
Questions for authors

Check the Critical Weaknesses section

5 comments

We thank Reviewer AKbt for the valuable comments. In the revised manuscript, we have added recent related models to the main comparison table, included performance overhead experiments on additional datasets, provided more detailed ablation studies, and introduced experiments under more extreme imbalance scenarios. Furthermore, we have simplified the framework diagram in the main text, retaining only an intuitive illustration of the core mechanisms and data flow. Meanwhile, we have provided more detailed and explicit explanations of our model's mechanisms in the appendix to reduce potential misinterpretations. By demonstrating the rationality of our architectural design with experimental data, we have further elevated the rigor of the paper.

Q1&Q3:Architectural Layout

We have completed the requested layout-reversal ablation. We tested architectures with the expert pool in the first 3 layers or the last 3 layers (the original layout), as well as fully shared and fully specialized layouts.

General LoRA SettingMicro Acc. (%)Class-Macro Acc. (%)Task-Mean Acc. (%)
Frozen after Task 181.791581.891681.8746
Trainable throughout81.826081.906581.9103
LayoutSeedIncremental task-macro (%)Final task-macro (%)Final-stage optimizer parameters
all_shared199376.5933.53221384
all_shared202465.5418.55221384
all_shared340775.8133.39221384
all_specific199379.3571.69510152
all_specific202478.4270.30396488
all_specific340780.4571.61457928
reversed199381.4974.27338120
reversed202481.3173.17322760
reversed340780.9072.64310472
original199386.2776.50331976
original202486.6578.63322760
original340785.3069.96316616

The results indicate that the original layout is the best among the configurations tested. It allows the model to learn both general and specialized representations of high-level image features: general LoRA captures commonalities across tasks, while special LoRA captures differences. Using only general LoRA prevents the model from learning distinctions among individual tasks and classes. Applying special LoRA across all 12 layers makes the task decomposition too fine-grained, and the large number of routing decisions at inference impairs fitting. Placing the expert pool in the first three layers is also inefficient because shallow layers mainly learn low-level visual features such as color, texture, and edges. A single general LoRA module suffices to distinguish shallow features within a dataset.

Q2:Redundant Information Processing

Redundant information can still be useful. Recent research suggests, for example, that background information can help a model learn image styles and types, thereby aiding classification. We therefore did not explicitly design general and special LoRA to process separate types of information. The purpose of general LoRA is to keep the model from becoming overly specialized in response to differences in imagery across tasks. We expect general LoRA to absorb redundant information from different tasks and stabilize learning across the model.

Table 1: Effect of General LoRA training strategies on overall classification performance.

General LoRA SettingMicro Acc. (%)Class-Macro Acc. (%)Task-Mean Acc. (%)
Frozen after Task 181.791581.891681.8746
Trainable throughout81.826081.906581.9103

Q4:Routing accuracy

Routing accuracy across datasets is reported below.

For CUB-200-Imbalance, we used seed 1993 and 10 tasks, with 5,794 test images per group. On B, the native TUNA/CaRE selectors and the SEMA diagnostic passed the complete per-image validation. On C, training was completed for Original and Reversed, but only aggregate evidence was available. These layouts did not pass the full validation because per-image records were missing. Test labels were used only for scoring. Classification accuracy and routing metrics are reported separately.

MethodMeasurementMicro / full path (%)Task macro (%)Acceptance
OriginalFull expert-path match60.5109N/AAggregate only; per-image records missing
ReversedFull expert-path match1.6914N/AAggregate only; per-image records missing
TUNANative minimum-entropy task selector13.082512.9804Accepted
CaRENative minimum-entropy outer task selector12.547512.5316Accepted
LayoutLayerMatch (%)
OriginalLayer_1074.3700
OriginalLayer_1173.2654
OriginalLayer_968.2430
ReversedLayer_07.8012
ReversedLayer_110.6144
ReversedLayer_213.3759

Per-task selection accuracy for TUNA/CaRE (only outer task adapter selection was evaluated for CaRE):

Task (1-based)Test imagesTUNA (%)CaRE (%)
159325.801029.5110
25533.616616.0940
35611.96089.8039
45931.18044.0472
55850.51285.6410
65803.96559.4828
75802.758616.5517
85701.228110.1754
95963.187915.6040
1058385.59188.4048

For SEMA, we report class routing consistency separately. We used 3,198 training images to identify the modal expert for each class and 5,794 test images for evaluation. Each of layers 9/10/11 was assigned 1 expert, and only 1 dominant expert was observed in each layer. Both layerwise and full-path consistency were 100%. This is a degenerate single-expert case and does not indicate correct task routing. Autonomous task routing accuracy is recorded as N/A for MoE-Adapters (two shared experts, top-k=2) and DSEF (all-expert or oracle implementations).

Most current expert expansion models do not pursue routing consistency as an end in itself. Instead, they aim to make different experts learn different types of knowledge and to select experts based on the content of the input image. When an image differs substantially in style, this design lets the model select an expert with relevant knowledge based on the style distinctions it has learned, rather than following a fixed routing path. Such content-dependent routing can improve robustness.

The layout ablation supports the chosen configuration, especially for incremental accuracy, though the final-task results show high variance across seeds. The effect of the split point has not been evaluated, and it remains unclear whether the general and specific LoRA modules learn distinct or complementary information. The reported routing accuracy ( around 60% full-path match, reported only in aggregate.) Although relative to other methods, this is a better accuracy, this also leaves room for improvement, and an oracle-routing comparison would help assess how much routing errors affect performance. Overall, I appreciate the authors clarifications and maintain my score.

Reviewer_Pqa5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies class-incremental learning (CIL) for long-tailed distributions where new classes come from imbalanced task streams resulting in unreliable few-shot class representations. The authors propose a replay-free method to tackle representation stability, expert accessibility, and tail-estimation uncertainty. The method uses a dual-track ViT adapter which is augmented with shared LoRA modules in the early transformer blocks and specific or expert LoRA modules in the late transformer blocks. A class-conditional gaussian router is used to determine expert reuse, isolation and capacity expansion. The last step involves route-aware prototype transfer for few-shot classes. The authors validate the proposed method across six long-tailed CIL benchmarks.

Critical strengths
  1. The proposed method is well explained with illustrations and details.
Critical weaknesses
  1. Limited novelty: The proposed method combines several components already used in existing works for CIL. The paper is not well positioned against several existing CIL works [a,b,c] which uses similar approaches for expert routing. These relevant methods specifically TUNA [b] is also not included in the method comparisons. Expert selection using gaussian distributions was proposed in [d]. The authors do not attribute the components of the proposed method to existing works.

  2. The paper does not provide details of the additional complexity and capacity of the proposed method with respect to existing methods for bigger benchmarks. Table 2 shows results only for CIFAR10. Can the authors provide more insights into how much the capacity expansion affects in settings with more classes and bigger benchmarks like ImageNet-A, CUB200, Core50?

  3. Several related papers for long-tailed CIL [e,f,g] are ignored in the discussions and comparisons. The setting of long-tailed class-incremental learning is not motivated in the paper.

[a] Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts. ICML 2026.

[b] Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning. ICCV 2025.

[c] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning. CVPR 2025.

[d] Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts. ICML 2025.

[e] Dynamic Adapter Tuning for Long-Tailed Class-Incremental Learning. WACV 2025.

[f] A Tiny Change, A Giant Leap: Long-Tailed Class-Incremental Learning via Geometric Prototype Alignment. ICCV 2025.

[g] Long-tail CIL via Independent Sub-prototype Construction. CVPR 2024.

Questions for authors
  1. Refer to the weaknesses for questions.

  2. Can the authors point out what is the novel component in the proposed method with respect to existing works?

8 comments

We thank Reviewer Pqa5 for the valuable comments.

In the revised manuscript, we have added recent related models to the main comparison table, included performance overhead experiments on additional datasets, provided more detailed ablation studies, and introduced experiments under more extreme imbalance scenarios. Furthermore, we have simplified the framework diagram in the main text, retaining only an intuitive illustration of the core mechanisms and data flow. Meanwhile, we have provided more detailed and explicit explanations of our model's mechanisms in the appendix to reduce potential misinterpretations. By demonstrating the rationality of our architectural design with experimental data, we have further elevated the rigor of the paper.

Q1:Limited novelty

We have added comparisons with the CIL methods you suggested [1, 2, 3]: CoRE, TUNA, and SEMA. The table below reports results on CUB200, cifar100, and Tiny img under the severe imbalance setting of ρ=0.01\rho=0.01.

DatasetLong-tail settingMethodFinal accuracy (%)Wall-clock time (min)Peak process GPU memory (MiB)Final trainable parameters (M)
CUB200CUB-LTTUNA83.8337.244,8943.959
CUB200CUB-LTCaRE83.1962.3721,8162.152
CUB200CUB-LTSEMA69.1441.004,8300.154
CIFAR-100IR=10TUNA85.45136.892,2423.806
CIFAR-100IR=10CaRE85.57335.5514,8481.906
CIFAR-100IR=10SEMA81.3788.127,1760.077
CIFAR-100IR=100TUNA85.18111.392,2503.806
CIFAR-100IR=100CaRE85.09312.3515,0301.906
CIFAR-100IR=100SEMA78.1083.817,1300.077
TinyImageNetIR=10TUNA87.5187.054,1923.959
TinyImageNetIR=10CaRE88.00701.2715,0142.152
TinyImageNetIR=10SEMA84.38432.527,8280.154
TinyImageNetIR=100TUNA87.1881.754,1923.959
TinyImageNetIR=100CaRE87.53587.3615,0902.152
TinyImageNetIR=100SEMA82.80282.819,6920.154
CUB200CUB-LTDual86.2716.443,4400.369
CIFAR-100IR=10Dual85.0161.371,9800.319
CIFAR-100IR=100Dual84.7952.331,9800.319
TinyImageNetIR=10Dual88.2757.233,2800.369
TinyImageNetIR=100Dual87.7350.763,2800.369

Note: CUB-LT uses linear long-tailed sampling with a minimum retention ratio of 0.1. After discrete sampling, the maximum and minimum class counts are 30/2, yielding an actual ratio of 15:1.

We have added a more challenging 1:100 imbalance setting and have also conducted experiments on Places-LT, a naturally imbalanced dataset. We report the results by head, middle, and tail groups:

DatasetMethodSeedH 1--3 (per task; mean)M 4--6 (per task; mean)T 8--10 (per task; mean)Mean across all 10 tasks
cifar100DualTrack199385.0 / 85.9 / 86.7; 85.8086.3 / 84.9 / 74.6; 81.9385.2 / 87.0 / 87.2; 86.4884.73
cifar100cl_lora199368.7 / 63.8 / 63.2; 65.2376.8 / 85.0 / 69.8; 77.2084.0 / 86.5 / 87.8; 86.1076.03
cifar100keeplora199391.4 / 75.3 / 72.1; 79.6081.8 / 79.2 / 74.6; 78.5381.1 / 73.9 / 79.0; 78.0078.78
cifar100CaRE199385.7 / 87.4 / 89.1; 87.4086.6 / 84.8 / 72.4; 81.2782.2 / 85.5 / 87.4; 85.0385.09
cifar100SEMA199383.6 / 75.7 / 88.7; 82.6780.4 / 80.6 / 49.1; 70.0374.5 / 84.8 / 84.4; 81.2378.10
cifar100TUNA199385.6 / 86.2 / 89.2; 87.0086.7 / 84.0 / 74.1; 81.6084.3 / 86.5 / 87.3; 86.0385.18
tinyimagenetCaRE199387.6 / 88.9 / 85.8; 87.4385.5 / 87.6 / 86.0; 86.3786.7 / 90.5 / 85.9; 87.7087.53
tinyimagenetSEMA199382.3 / 79.7 / 78.3; 80.1075.0 / 84.5 / 78.4; 79.3082.9 / 92.2 / 86.6; 87.2382.80
tinyimagenetTUNA199388.0 / 88.7 / 85.0; 87.2384.3 / 86.9 / 85.0; 85.4086.1 / 91.3 / 86.1; 87.8387.18
tinyimagenetDualTrack199388.6 / 90.2 / 86.7; 88.5085.5 / 87.6 / 86.2; 86.4088.2 / 90.7 / 85.6; 88.1687.73
MethodFull-set final task-macro (%)Incremental task-macro mean (%)Train time (min)Peak allocated GPU memory (MiB)Trainable parameters (M)
DualTrack46.919856.6976264.512,4490.447
TUNA45.500256.4520207.763,5294.213
CaRE46.087256.6488986.989,8562.553
cl_lora35.132446.4400283.188,7630.276
kp_lora38.796047.4512210.444,8510.983

These results show that Dual (ours) remains competitive with state-of-the-art methods and achieves better knowledge transfer for tail classes under imbalance.

The novelty of our model lies in its use of adaptive expansion and LoRA rank increases to transfer knowledge. In conventional expert expansion methods such as TUNA [2], the parameter count grows linearly with the number of tasks, and routing becomes more complex. This directly affects model performance across tasks. With adaptive expansion, the parameter count of our model does not grow linearly with the number of tasks, and routing does not become excessively complex as experts accumulate. In imbalanced settings, our fusion approach also increases sample diversity for tail classes, demonstrating the model's ability to transfer knowledge.

TUNA [2] likewise combines general and specialized components. In TUNA, however, the universal adapter is obtained by fusing specialized adapters at inference time, which differs from how the general and specialized LoRA modules are trained in our method. In reproducing TUNA, we also found that the experimental implementation did not use the universal adapter. Instead, it used a residual connection that combined the adapter for the most recent task with a low-rank adapter. You also noted the relevance of DCE [4] to this work. We agree that Gaussian feature modeling has been studied before and that the initial manuscript did not discuss or credit this work adequately. We will cite it in the revision and clarify that our contribution concerns how class-conditional statistics connect layerwise expert selection, constrained expert reuse, and capacity allocation, rather than Gaussian modeling itself.

Gaussian statistics play different roles in DCE and DualTrack. DCE uses the Gaussian statistics of each class in each domain to generate balanced pseudo-features for training an MLP expert selector. At prediction time, that selector produces the expert fusion coefficients. The Gaussian routing module in DualTrack directly computes the matching energy between an input and the registered class distributions, then uses those energies to score the experts. This statistical routing module therefore does not require Gaussian sampling to train an MLP selector. Specifically, at layer ll, each expert kk is associated with a set of registered classes Cl,k\mathcal C_{l,k}, and its score is determined by the best-matching class region:

δl,k(z)=min⁡c∈Cl,kdl,c(z),ql,k(z)=softmaxk[−δl,k(z)/τr]\delta _{l,k}(z) = \min _{c \in \mathcal{C} _{l,k}} d _{l,c}(z), \quad q _{l,k}(z) = \mathrm{softmax} _k [ -\delta _{l,k}(z)/\tau _r ]

where dl,cd_{l,c} is the matching energy under a diagonal Gaussian. This design preserves distinct regions for the classes associated with each expert and avoids scoring an expert that covers both head and tail classes using a single pooled center. The statistical estimates for classes with few samples may still be inaccurate.

The same class-conditional statistics also support expert assignment for new classes. DualTrack determines whether to reuse an existing expert based on distributional coverage, the margin between candidates, class capacity, and the per-task quota for absorbing classes. When the criteria are met, it triggers a controlled increase in LoRA rank; it can also allocate a new expert when the quota allows. In addition, the routing stream excludes direct feedback from the Specific residual, so expert adaptation does not directly change the coordinates used for subsequent routing. Updates to the shared module can still shift these coordinates. Our claimed technical distinction is thus the connection among statistical matching, separation of the routing coordinates, and constrained expert reuse. We do not claim to be the first to use Gaussian distributions for expert selection.

Q2:Resource Consumption Analysis

We have added experiments on computational cost for ImageNet-A and CUB200.

CUB200-Imbalance

MethodTime (min)Peak process GPU memory (MiB)Final trainable parameters (M)
CL-LoRA30.6841680.276
DCE13.8243020.023
DER++57.66402885.952
DSEF5.21467421.264
KeepLoRA14.4183600.983
MoE-Adapters19.4439044.027
RanPAC16.2611802.000
StarPrompt87.6864023.328
DualTrack16.4434400.369

Comments on the paper

2 comments

Dear Reviewers,

The authors have provided their rebuttals to the reviewers' comments. Please take a moment to review the responses and continue the discussion on any remaining concerns, unresolved issues, or points that may require further clarification.

Your further input and discussion are highly appreciated.

Regards,

AC for this paper

Dear AC,

The innovations of our model mainly lie in the following two aspects:

1. We incorporate LoRA and the idea of LoRA-based dimension expansion into continual learning, to alleviate the issue that the routing selection becomes intractable when the number of experts grows horizontally with an unlimited sequence of tasks for expandable expert models. ​ 2. We introduce general experts and special experts to capture universal and task-specific features from the datasets, respectively.

Extensive ablation experiments have demonstrated that our architecture can reduce routing complexity and boost the model’s learning capability. This work provides a new perspective and practical paradigm for the research of expandable expert models in continual learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.