acceptodds
Under review as a conference paper at ICLR 2027

Distribution-Conditioned Task Routing for Class-Incremental Learning

Abstract

Parameter-efficient adaptation enables continual learners to acquire task-specific knowledge through compact model updates while maintaining strong within-task performance. However, class-incremental inference requires each input to be classified among all classes seen so far without access to its task identity. For learners equipped with task-specific parameter-efficient modules, this introduces a critical task-routing challenge beyond catastrophic forgetting. We study post-hoc task routing without retraining the learner or introducing a separately trained router. Such training-free inference-time calibration remains comparatively underexplored in parameter-efficient class-incremental learning. We identify three sources of routing error—feature-level, task-level, and class-level misalignment—and propose Feature Distribution Calibration (FDC). Its three components address these misalignments: Task Subspace Filtering (TSF) suppresses feature components outside each task's principal subspace, Residual Likelihood Calibration (RLC) evaluates the typicality of its subspace residual, and Prototype Affinity Calibration (PAC) measures compatibility with the task's class prototypes. Experiments demonstrate plug-and-play applicability to eight parameter-efficient class-incremental methods using a shared encoder. With one component configuration selected per method across all five benchmarks, FDC improves final accuracy in all 40 method–dataset pairs by 4.39 percentage points on average. Enabling all components improves 35 of the 40 pairs, with an average gain of 4.45 points. When applied to a simple baseline, FDC achieves strong overall performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D3jPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feature Distribution Calibration (FDC) for class-incremental learners sharing an adapted encoder. Task Subspace Filtering, Residual Likelihood Calibration, and Prototype Affinity Calibration use stored feature statistics to calibrate task-head competition without additional gradient-based optimization.

Critical strengths

The routing problem is clearly motivated, and comparisons with and without FDC reuse the same checkpoints and test examples.

Critical weaknesses
  1. RLC’s Gaussian approximation needs stronger validation beyond standardization. Comparing final-encoder features with historical statistics also raises calibration concerns.
  2. TSF uses centered PCA but filters uncentered features with unchanged biases. Please justify this choice against mean-preserving residual attenuation.
  3. FDC is evaluated only at final checkpoints under a fixed class order. Please report intermediate-stage performance across multiple class orders.
  4. Improve figure readability.
Questions for authors
  1. Do the selective-configuration gains persist when components are chosen on independent validation data rather than the reported evaluation results?
  2. Can the authors quantify residual-distribution drift and its impact on routing accuracy for earlier tasks?
  3. Does FDC outperform direct prototype- or residual-based routing under matched checkpoints, statistics, and inference budgets, given its roughly 1.9× dual-view latency?
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer D3jP (Part 1)

We sincerely thank Reviewer D3jP for the constructive comments and for recognizing the clear motivation of the routing problem and the controlled comparisons using the same checkpoints and test examples. We have directly tested the Gaussian approximation in RLC and the suggested mean-preserving variant of TSF. We have also added experiments using class orders 1993/1994/1995, focusing on the cost-efficient ET single-view version. We provide pointwise responses below.

W1: RLC’s Gaussian approximation needs stronger validation beyond standardization...

A1: we agree that standardization alone does not establish Gaussianity. We therefore directly tested more flexible distributions, including skew-normal and Beta, with parameters fitted from training residuals.

Own-residual modelET ACC change: original τ (pp)ET ACC change: training-refitted τ (pp)
Skew-normal−0.078+0.017
Beta−0.045+0.022

These replacements bring little average improvement, even when the calibration scale is refitted. This supports the practical adequacy of the Gaussian approximation in these experiments. We address the separate concern about historical statistics through the residual-drift and routing analysis in Q2 below.

W2: TSF uses centered PCA but filters uncentered features with unchanged biases. Please justify this choice against mean-preserving residual attenuation.

A2: We tested the mean-preserving variant of TSF, keeping the attenuation strength and calibration scales unchanged. This comparison examines whether preserving the task mean improves performance under the same attenuation and calibration settings.

DatasetMean-preserving FDC − current FDC, ET (pp)
CIFAR-100−0.240 ± 0.265
CUB-200−1.173 ± 0.200
ImageNet-R−0.067 ± 0.076
ImageNet-A−0.461 ± 0.287
OmniBenchmark−0.128 ± 0.198
Mean−0.414

Mean-preserving attenuation reduces the mean accuracy on all five datasets, by 0.414 pp overall and most strongly on CUB-200. These results support retaining the current TSF formulation in the evaluated settings. We conjecture that attenuating the task-dependent mean component also helps balance cross-task competition; this accuracy comparison alone does not establish the mechanism.

Q1: Do the selective-configuration gains persist when components are chosen on independent validation data...

A3: We will use full-component FDC as the main result in the paper and provide leave-one-dataset-out component selection as a supplementary result. For selection, we choose the TSF/PAC/RLC configuration with the largest average gain on the other four datasets, then evaluate it on the held-out dataset. Each of the five datasets is held out in turn for every learner and class order, giving 8 learners × 5 held-out datasets × 3 orders = 120 evaluation pairs per view.

ConfigurationET mean gain ± SD (pp)ET improvedDual mean gain ± SD (pp)Dual improved
Full FDC+3.060 ± 0.059107/120+4.287 ± 0.015106/120
Leave-one-dataset-out+3.482 ± 0.127109/120+4.525 ± 0.163109/120

Held-out selection gives a slightly larger average gain, while the proportion of improved pairs remains similar.

W3: FDC is evaluated only at final checkpoints under a fixed class order...

A4: We now report cumulative AIA at every task, rather than only the endpoint. AIA is the average of the accuracies measured after each task learned so far, with each evaluation covering all classes seen at that stage. We divide each sequence into five equal intervals (four tasks per interval for 20-task sequences; two for 10-task sequences). Each entry averages the per-task cumulative AIA gains within that interval.

DatasetAIA gain (pp)0–20%20–40%40–60%60–80%80–100%
CIFAR-100FDC − baseline+0.15+1.05+1.63+1.93+2.15
FDC (refreshed statistics) − baseline+0.27+1.39+2.35+2.90+3.25
CUB-200FDC − baseline+1.37+3.52+4.58+5.12+5.66
FDC (refreshed statistics) − baseline+1.50+3.76+4.88+5.52+6.16
ImageNet-RFDC − baseline+0.97+1.63+2.12+2.44+2.71
FDC (refreshed statistics) − baseline+1.01+1.92+2.51+2.88+3.22
ImageNet-AFDC − baseline+2.81+4.33+5.13+5.52+5.77
FDC (refreshed statistics) − baseline+2.70+4.34+5.23+5.68+5.97
OmniBenchmarkFDC − baseline+0.52+1.47+1.95+2.30+2.66
FDC (refreshed statistics) − baseline+0.82+2.53+3.48+4.25+4.91

These results show that FDC's gains persist during incremental learning, and the interval-averaged AIA gains increase as more tasks are learned on all five datasets.

Feature drift does occur, and refreshing PAC/RLC statistics using historical training images under the current encoder further improves performance overall. Nevertheless, FDC remains effective with historical statistics despite this drift. Refreshing requires revisiting old training data; approximating it without retaining old data is a direction for future work.

AuthorsOpenReview ↗

Author Response to Reviewer D3jP (Part 2)

Q3: Does FDC outperform direct prototype- or residual-based routing under matched checkpoints, statistics, and inference budgets, given its roughly 1.9× dual-view latency?

A5: We first tried simple logit and temperature calibration controls under the same three-order protocol:

MethodMean final ACC ± SD (%)
Maximum logit70.541 ± 0.682
Training-statistics logit z-score70.124 ± 0.826
Maximum softmax, T=164.449 ± 0.253
Maximum softmax, T=267.550 ± 0.154
Maximum softmax, T=469.561 ± 0.227
FDC ET75.739

FDC ET exceeds these simple controls by at least 5.198 pp on average.

We then directly compare with subspace residuals, likelihood-based scoring, and prototype similarity, using ViM-style scoring [1], ConjNorm [2], prototype classification [3], shared-covariance Mahalanobis [4], Mahalanobis++ [5], and FeCAM [6].

ET scoring methodStandalone ACC (%)With FDC (%)Gain over standalone (pp)
ViM-style residual scoring [1]73.87175.907+2.036
ConjNorm [2]69.78375.681+5.898
Prototype similarity [3]76.54777.032+0.485
Shared-covariance Mahalanobis [4]76.37077.038+0.668
Mahalanobis++ [5]75.75776.871+1.114
FeCAM, PILOT-ViT scoring [6]75.59776.767+1.170
FDC alone75.739——

These are scoring adaptations on the same Cumulative-LoRA checkpoints. For their fusion with FDC, we add the scoring signal to the final FDC class scores and take a global argmax, using a common coefficient of 1. Our point is that although some standalone geometric classifiers are already strong, adding FDC further improves them.

In other words, FDC provides task-conditioned calibration that is complementary to the evaluated post-hoc scoring adaptations. Its distinction lies in how statistical evidence is used: TSF filters head responses according to task-specific feature support, while RLC and PAC interpret residuals and prototype affinities relative to task-specific training statistics.

We further compared FDC with all six scoring adaptations on InfLoRA, InfLoRA-CA, EWC-LoRA, SD-LoRA, Split-LoRA, LoRA-Sub, RSIAT, and LoDA, across five datasets and three class orders (120 evaluations).Each method was evaluated independently using the same learner checkpoints and a single encoder view. The table reports gains over the original learner and the number of evaluations in which accuracy decreased:

MethodMean gain over original learner (pp)Degraded evaluations
FDC ET+3.06012/120
ViM-style ET−2.15056/120
ConjNorm ET−1.83268/120
Prototype ET+2.70532/120
Shared Mahalanobis ET+2.47428/120
Mahalanobis++ ET+2.72532/120
FeCAM ET+1.74635/120

Our fusion results are consistent with FDC's complementary role. Across 120 matched learner–dataset–order evaluations, adding FDC improves the overall mean accuracy of all six evaluated scoring adaptations. For Prototype, Shared Mahalanobis, Mahalanobis++, and FeCAM scoring, fusion exceeds both the standalone scorer and standalone FDC, with gains of 0.606–0.802 percentage points over FDC. These results support empirical complementarity rather than statistical independence or universal improvement for every learner. Since the current fusion combines complete score functions, it does not by itself isolate task-level calibration from the contribution of the retained classifier.

We also agree that the extra view has a substantial cost. ET provides most of the improvement, while dual requires a second encoder view.

ConfigurationMean final ACC (%)Gain over baseline (pp)Encoder views
Cumulative baseline70.541—1
FDC ET75.739+5.1981
FDC dual76.817+6.2762

ET retains 82.8% of the dual version's gain; dual adds 1.078 pp over ET.

For convenience, we summarize the inference-time measurements already reported in Appendix C.2, Table 7 of the submitted paper:

DatasetBaseline (ms/batch)FDC ET (ms/batch)FDC dual (ms/batch)
CIFAR-100170.5173.3 (+1.7%)323.1 (+89.5%)
CUB-200171.3174.1 (+1.6%)324.2 (+89.3%)
ImageNet-R170.5171.7 (+0.7%)320.1 (+87.7%)
ImageNet-A171.0172.5 (+0.8%)321.5 (+88.0%)
OmniBenchmark169.9171.3 (+0.8%)319.2 (+87.9%)

References

[1] ViM: Out-of-Distribution With Virtual-Logit Matching. CVPR 2022.

[2] ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection. ICLR 2024.

[3] iCaRL: Incremental Classifier and Representation Learning. CVPR 2017.

[4] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS 2018.

[5] Mahalanobis++: Improving OOD Detection via Feature Normalization. ICML 2025.

[6] FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning*. NeurIPS 2023. (We use the pretrained-ViT scoring implementation in PILOT)

AuthorsOpenReview ↗

Author Response to Reviewer D3jP (Part 3)

Q2: Can the authors quantify residual-distribution drift and its impact on routing accuracy for earlier tasks?

A6: To quantify residual drift directly, we measured residuals on a fixed set of training samples from earlier tasks: the mean residual rises from 0.2505 at task arrival to 0.2943 under the final encoder with the old PCA, and falls to 0.2512 after rebuilding PCA.

We also isolated RLC refresh while keeping PAC and the other components fixed:

RLC-only oracle refreshET ACC change (pp)ET routing accuracy change (pp)
Distribution statistics only; original PCA and τ+0.179+0.186
PCA and distribution statistics; original τ+0.451+0.474
PCA, distribution statistics, and τ+0.580+0.629

W4: Improve figure readability.

A7: We will improve figure readability.

Reviewer_PM6GOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper examines task routing in parameter-efficient class-incremental learners that use a shared final encoder and separately trained task heads. The authors argue that poor competition between heads, rather than loss of within-task discrimination alone, accounts for a substantial part of the final error. Feature Distribution Calibration (FDC) is a post-hoc procedure that leaves the learned model unchanged. Task Subspace Filtering (TSF) removes feature components outside a candidate task's principal subspace; Residual Likelihood Calibration (RLC) compares task-specific residual distributions; and Prototype Affinity Calibration (PAC) provides class-level similarity evidence.

Critical strengths

The paper isolates a meaningful inference bottleneck. Figure 2 compares final-model predictions with and without the true task identity while holding the shared encoder fixed. The substantial gap supports treating cross-head routing separately from within-task recognition. Figures 3–4 further connect routing confusion with subspace residuals and prototype affinity. The motivation is therefore backed by concrete diagnostics rather than an assumption that any additional calibration must help.

Post-hoc correction across several existing learners is practically useful. FDC does not retrain the LoRA modules or classification heads. Table 3 evaluates Cumulative-LoRA and eight other PEFT-CIL methods. On Cumulative-LoRA, the five final-accuracy gains are +4.28, +5.97, +3.58, +10.88, and +6.35 percentage points. The authors also report a +4.45-point average gain when all components are enabled. These results indicate that the stored statistics can provide useful routing evidence beyond a single learner's training objective.

Critical weaknesses

The headline 40/40 result is affected by evaluation-informed component selection. Section 5.3 selects components for each learner according to their measured standalone gains. Appendix G explicitly states, “This selection uses the reported evaluation results.” Table 3 then reports improvements for all 40 method–dataset pairs using configurations chosen with those same five datasets. This is not an independent test of a previously fixed configuration. The paper also reports that enabling all components improves 35/40 pairs, with an average gain of 4.45 points, compared with 4.39 points for selective configurations. The fixed-configuration result is relevant and should be reported prominently, including its failures; it does not validate the stronger 40/40 claim. Independent component selection, for example through leave-one-dataset-out evaluation or held-out validation, is necessary before the claimed plug-and-play reliability can be established.

The validity and cost of the stored statistics require closer examination. Section 4 constructs task-specific PCA bases, residual models, and class prototypes at the time each task is learned. At final inference, those statistics are applied to an encoder that has subsequently accumulated additional LoRA updates. The paper does not directly measure the drift between representations under EtE_t and ETE_T, leaving open when historical PCA and prototype evidence becomes stale. RLC also relies on compact own/foreign residual models without showing how well they capture heterogeneous foreign tasks or remain calibrated as tasks age. The probability-chain-rule discussion in Eqs. (5–7) motivates a decomposition; it does not derive the particular FDC scores as an optimal or identified correction. Finally, dual-view FDC has a substantial runtime cost: Appendix Table 7 reports 170.5 to 320.1 ms/batch (+87.7%) on ImageNet-R, with a similar near-doubling on the other benchmarks. The term “modest overhead” should be tied to the exact configuration needed for the reported gain, rather than to storage cost alone.

Questions for authors
  1. Can the authors select all components without using the test results of the target dataset and report leave-one-dataset-out or held-out performance? Which five method–dataset pairs degrade with the fixed full-component configuration, and why?
  2. How do task-specific PCA subspaces, prototypes, and residual-likelihood calibration drift between EtE_t and the final ETE_T? Please report routing accuracy and calibration error as a function of task age.
  3. At comparable storage and inference cost, how much of the gain remains over simple logit/temperature calibration, prototype routing, and Mahalanobis/FeCAM-style scoring? Please include single-view and dual-view accuracy–latency trade-offs.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer PM6G (Part 1)

We sincerely thank Reviewer PM6G for the constructive and insightful comments, and we are glad that Reviewer recognize our paper to be well motivated, practical, and easy to follow. We have added experiments using class orders 1993/1994/1995. We focus on the cost-efficient ET single-view version when answering your questions. To fully address Reviewer’s concerns, we provide pointwise responses below.

W1 & Q1: The headline 40/40 result is affected by evaluation-informed component selection. Section 5.3 selects components for each learner according to their measured standalone gains...

A1: Thank you for your suggestion. We will use full-component FDC as the main result in the paper and provide leave-one-dataset-out component selection as a supplementary result. For selection, we choose the TSF/PAC/RLC configuration with the largest average gain on the other four datasets, then evaluate it on the held-out dataset. Each of the five datasets is held out in turn for every learner and class order, giving 8 learners × 5 held-out datasets × 3 orders = 120 evaluation pairs per view.

ConfigurationET mean gain ± SD (pp)ET improvedDual mean gain ± SD (pp)Dual improved
Full FDC+3.060 ± 0.059107/120+4.287 ± 0.015106/120
Leave-one-dataset-out+3.482 ± 0.127109/120+4.525 ± 0.163109/120

Held-out selection gives a slightly larger average gain, while the proportion of improved pairs remains similar.

The ET failures are concentrated in LoRA-Sub and RSIAT:

Learner / datasetOrders with a decreaseMean full-FDC ET change over three orders (pp)
LoRA-Sub / CUB-2003/3−2.474
LoRA-Sub / ImageNet-A2/3−0.680
LoRA-Sub / CIFAR-1001/3+0.667
RSIAT / CUB-2003/3−2.516
RSIAT / ImageNet-A2/3−0.219
RSIAT / OmniBenchmark1/3−0.206

Both LoRA-Sub and RSIAT explicitly constrain feature relationships between old and new classes during training. LoRA-Sub uses historical class prototypes as negatives in its triplet loss, while RSIAT combines representation alignment with an orthogonality loss involving old-class prototypes. TSF may alter the distances and angles learned under these objectives, while PAC/RLC may shift task scores against the separation already learned by the classifier.

W2 & Q2: The validity and cost of the stored statistics require closer examination. Section 4 constructs task-specific PCA bases, residual models, and class prototypes at the time each task is learned. At final inference, those statistics are applied to an encoder that has subsequently accumulated additional LoRA updates...

A2: Thank you for your suggestion. We now report cumulative AIA at every task, rather than only the endpoint. AIA is the average of the accuracies measured after each task learned so far, with each evaluation covering all classes seen at that stage. We divide each sequence into five equal intervals (four tasks per interval for 20-task sequences; two for 10-task sequences). Each entry averages the per-task cumulative AIA gains within that interval.

DatasetAIA gain (pp)0–20%20–40%40–60%60–80%80–100%
CIFAR-100FDC − baseline+0.15+1.05+1.63+1.93+2.15
FDC (refreshed statistics) − baseline+0.27+1.39+2.35+2.90+3.25
CUB-200FDC − baseline+1.37+3.52+4.58+5.12+5.66
FDC (refreshed statistics) − baseline+1.50+3.76+4.88+5.52+6.16
ImageNet-RFDC − baseline+0.97+1.63+2.12+2.44+2.71
FDC (refreshed statistics) − baseline+1.01+1.92+2.51+2.88+3.22
ImageNet-AFDC − baseline+2.81+4.33+5.13+5.52+5.77
FDC (refreshed statistics) − baseline+2.70+4.34+5.23+5.68+5.97
OmniBenchmarkFDC − baseline+0.52+1.47+1.95+2.30+2.66
FDC (refreshed statistics) − baseline+0.82+2.53+3.48+4.25+4.91

These results show that FDC's gains persist during incremental learning, and the interval-averaged AIA gains increase as more tasks are learned on all five datasets.

Feature drift does occur, and refreshing PAC/RLC statistics using historical training images under the current encoder further improves performance overall. Nevertheless, FDC remains effective with historical statistics despite this drift. Refreshing requires revisiting old training data; approximating it without retaining old data is a direction for future work.

We recommend ET as the default: it uses a single encoder view and adds only 0.7–1.7% inference latency and 3.7–5.9 MiB peak GPU memory over the baseline on Cumulative-LoRA across the five datasets (Appendix C.2, Table 7). Dual uses a second encoder view and takes 1.88–1.90× the baseline inference time, making it an optional choice when additional accuracy justifies the cost.

AuthorsOpenReview ↗

Author Response to Reviewer PM6G (Part 2)

Q3.: At comparable storage and inference cost, how much of the gain remains...

A3: Thank you for your suggestion. To address your concern, we first tried simple logit and temperature calibration controls under the same three-order protocol:

MethodMean final ACC ± SD (%)
Maximum logit70.541 ± 0.682
Training-statistics logit z-score70.124 ± 0.826
Maximum softmax, T=164.449 ± 0.253
Maximum softmax, T=267.550 ± 0.154
Maximum softmax, T=469.561 ± 0.227
FDC ET75.739

FDC ET exceeds these simple controls by at least 5.198 pp on average. All logit controls use the adapted encoder's original task heads.

We then directly compare against subspace residuals, likelihood-based scoring, and prototype similarity, using ViM-style scoring [1], ConjNorm [2], prototype classification [3], shared-covariance Mahalanobis [4], Mahalanobis++ [5], and FeCAM [6].

ET scoring methodStandalone ACC (%)With FDC (%)Gain over standalone (pp)
ViM-style residual scoring [1]73.87175.907+2.036
ConjNorm [2]69.78375.681+5.898
Prototype similarity [3]76.54777.032+0.485
Shared-covariance Mahalanobis [4]76.37077.038+0.668
Mahalanobis++ [5]75.75776.871+1.114
FeCAM, PILOT-ViT scoring [6]75.59776.767+1.170
FDC alone75.739——

Above are scoring adaptations on the same Cumulative-LoRA checkpoints. For their fusion with FDC, we add the scoring signal to the final FDC class scores and take a global argmax, using a common coefficient of 1. Our point is that although some standalone geometric classifiers are already strong, adding FDC further improves them.

In other words, FDC provides task-conditioned calibration that is complementary to the evaluated post-hoc scoring adaptations. Its distinction lies in how statistical evidence is used: TSF filters head responses according to task-specific feature support, while RLC and PAC interpret residuals and prototype affinities relative to task-specific training statistics.

We further compared FDC with all six scoring adaptations on InfLoRA, InfLoRA-CA, EWC-LoRA, SD-LoRA, Split-LoRA, LoRA-Sub, RSIAT, and LoDA, across five datasets and three class orders (120 evaluations).Each method was evaluated independently using the same learner checkpoints and a single encoder view. The table reports gains over the original learner and the number of evaluations in which accuracy decreased:

MethodMean gain over original learner (pp)Degraded evaluations
FDC ET+3.06012/120
ViM-style ET−2.15056/120
ConjNorm ET−1.83268/120
Prototype ET+2.70532/120
Shared Mahalanobis ET+2.47428/120
Mahalanobis++ ET+2.72532/120
FeCAM ET+1.74635/120

Our fusion results are consistent with FDC's complementary role. Across 120 matched learner–dataset–order evaluations, adding FDC improves the overall mean accuracy of all six evaluated scoring adaptations. For Prototype, Shared Mahalanobis, Mahalanobis++, and FeCAM scoring, fusion exceeds both the standalone scorer and standalone FDC, with gains of 0.606–0.802 percentage points over FDC. These results support empirical complementarity rather than statistical independence or universal improvement for every learner. Since the current fusion combines complete score functions, it does not by itself isolate task-level calibration from the contribution of the retained classifier.

We agree that the extra view has a substantial cost. ET provides most of the improvement, while dual requires a second encoder view.

ConfigurationMean final ACC (%)Gain over baseline (pp)Encoder views
Cumulative baseline70.541—1
FDC ET75.739+5.1981
FDC dual76.817+6.2762

ET retains 82.8% of the dual version's gain; dual adds 1.078 pp over ET.

For convenience, we summarize the inference-time measurements already reported in Appendix C.2, Table 7 of the submitted paper:

DatasetBaseline (ms/batch)FDC ET (ms/batch)FDC dual (ms/batch)
CIFAR-100170.5173.3 (+1.7%)323.1 (+89.5%)
CUB-200171.3174.1 (+1.6%)324.2 (+89.3%)
ImageNet-R170.5171.7 (+0.7%)320.1 (+87.7%)
ImageNet-A171.0172.5 (+0.8%)321.5 (+88.0%)
OmniBenchmark169.9171.3 (+0.8%)319.2 (+87.9%)

References

[1] ViM: Out-of-Distribution With Virtual-Logit Matching. CVPR 2022.

[2] ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection. ICLR 2024.

[3] iCaRL: Incremental Classifier and Representation Learning*. CVPR 2017.

[4] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS 2018.

[5] Mahalanobis++: Improving OOD Detection via Feature Normalization. ICML 2025.

[6] FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning*. NeurIPS 2023. (We use the pretrained-ViT scoring implementation in PILOT)

AuthorsOpenReview ↗

Following up before the discussion period closes

Dear Reviewer PM6G,

Thank you for reviewing our paper and for your constructive feedback. As the author–reviewer discussion deadline approaches, we would appreciate your feedback on whether our responses have addressed your concerns. We would be happy to provide further clarification or address any remaining questions.

Best regards, The Authors of Paper 15149

Reviewer_HydrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-hoc task routing in class-incremental learning. The proposed method, FDC, uses task subspaces, residual distributions, and class prototypes to calibrate predictions from existing task heads without further gradient-based training. The method is evaluated on five datasets and several continual learning methods, showing consistent average improvements over the original predictions.

Critical strengths
  1. The paper addresses a useful problem. Even when within-task recognition remains strong, competition between task heads can substantially hurt class-incremental accuracy. Improving this without retraining the model has practical value.

  2. The experiments cover several methods and datasets, with three training seeds. Comparing predictions before and after calibration using the same checkpoints makes the contribution of FDC easier to assess.

  3. The ablations show that the components provide useful and partly complementary improvements. Importantly, enabling all components without method-specific selection still gives a positive average gain for every evaluated method. The paper also reports memory and inference costs.

Critical weaknesses
  1. The component configurations used in the main plug-and-play results are selected using the reported evaluation results. Using one configuration across datasets does not fully remove this concern. The full-component results already provide useful evidence, so these should be the main comparison, while component selection should use separate validation data.

  2. The components are closely related to established ideas such as subspace residuals, likelihood-based scoring, and prototype similarity. The ablations are helpful, but direct comparisons with simpler statistical routing methods under the same feature and data conditions would make the methodological contribution clearer.

  3. The adapted-view statistics are collected when each task is learned, while inference uses the final encoder. It is not fully clear how sensitive the method is to the resulting feature drift. Experiments with different task orders or longer sequences would help, especially since the current evaluation uses a fixed class order.

  4. Dual-view calibration requires an additional encoder forward pass and nearly doubles inference latency in the reported experiments. The adapted-only version is much cheaper and already performs well, so the practical trade-off between the two versions deserves more emphasis.

Questions for authors

See weakenesses

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Hydr (Part 1)

We sincerely thank Reviewer Hydr for the constructive and insightful comments, and we are glad that Reviewer recognize our paper to be practical from both problem and results. We have added experiments using class orders 1993/1994/1995. We focus on the cost-efficient ET single-view version when answering your questions. Unless stated otherwise, results are averaged over the five datasets and three class orders; gains are in percentage points (pp). To fully address Reviewer’s concerns, we provide pointwise responses below.

W1: The component configurations used in the main plug-and-play results are selected using the reported evaluation results. Using one configuration across datasets does not fully remove this concern. The full-component results already provide useful evidence...

A1: Thank you for your suggestion. We will use full-component FDC as the main result in the paper and provide leave-one-dataset-out component selection as a supplementary result. For component selection, we choose the TSF/PAC/RLC configuration with the largest average gain on the other four datasets, then evaluate it on the held-out dataset. Each of the five datasets is held out in turn for every learner and class order, giving 8 learners × 5 held-out datasets × 3 orders = 120 evaluation pairs per view.

ConfigurationET mean gain ± SD (pp)ET improvedDual mean gain ± SD (pp)Dual improved
Full FDC+3.060 ± 0.059107/120+4.287 ± 0.015106/120
Leave-one-dataset-out+3.482 ± 0.127109/120+4.525 ± 0.163109/120

Held-out selection gives a slightly larger average gain, while the proportion of improved pairs remains similar.

W2. The components are closely related to established ideas such as subspace residuals, likelihood-based scoring, and prototype similarity. The ablations are helpful, but direct comparisons...

A2: Thank you for your suggestion. We now directly compare with subspace residuals, likelihood-based scoring, and prototype similarity, using ViM-style scoring [1], ConjNorm [2], prototype classification [3], shared-covariance Mahalanobis [4], Mahalanobis++ [5], and FeCAM [6].

ET scoring methodStandalone ACC (%)With FDC (%)Gain over standalone (pp)
ViM-style residual scoring [1]73.87175.907+2.036
ConjNorm [2]69.78375.681+5.898
Prototype similarity [3]76.54777.032+0.485
Shared-covariance Mahalanobis [4]76.37077.038+0.668
Mahalanobis++ [5]75.75776.871+1.114
FeCAM, PILOT-ViT scoring [6]75.59776.767+1.170
FDC alone75.739——

Above are scoring adaptations on the same Cumulative-LoRA checkpoints. For their fusion with FDC, we add the scoring signal to the final FDC class scores and take a global argmax, using a common coefficient of 1. Our point is that although some standalone geometric classifiers are already strong, adding FDC further improves them.

In other words, FDC provides task-conditioned calibration that is complementary to the evaluated post-hoc scoring adaptations. Its distinction lies in how statistical evidence is used: TSF filters head responses according to task-specific feature support, while RLC and PAC interpret residuals and prototype affinities relative to task-specific training statistics. In particular, the RLC/PAC correction is shared across classes within each task, allowing it to modify cross-task competition without changing the within-task class ranking of a fixed score function.

We further compared FDC with all six scoring adaptations on InfLoRA, InfLoRA-CA, EWC-LoRA, SD-LoRA, Split-LoRA, LoRA-Sub, RSIAT, and LoDA, across five datasets and three class orders (120 evaluations).Each method was evaluated independently using the same learner checkpoints and a single encoder view. The table reports gains over the original learner and the number of evaluations in which accuracy decreased:

MethodMean gain over original learner (pp)Degraded evaluations
FDC ET+3.06012/120
ViM-style ET−2.15056/120
ConjNorm ET−1.83268/120
Prototype ET+2.70532/120
Shared Mahalanobis ET+2.47428/120
Mahalanobis++ ET+2.72532/120
FeCAM ET+1.74635/120

Our fusion results are consistent with FDC's complementary role. Across 120 matched learner–dataset–order evaluations, adding FDC improves the overall mean accuracy of all six evaluated scoring adaptations. For Prototype, Shared Mahalanobis, Mahalanobis++, and FeCAM scoring, fusion exceeds both the standalone scorer and standalone FDC, with gains of 0.606–0.802 percentage points over FDC. These results support empirical complementarity rather than statistical independence or universal improvement for every learner. Since the current fusion combines complete score functions, it does not by itself isolate task-level calibration from the contribution of the retained classifier.

AuthorsOpenReview ↗

Author Response to Reviewer Hydr (Part 2)

W3: The adapted-view statistics are collected when each task is learned, while inference uses the final encoder. It is not fully clear how sensitive the method is to the resulting feature drift. Experiments with different task orders or longer sequences would help, especially since the current evaluation uses a fixed class order.

A3: Thank you for your suggestion. To address your concern, we now report cumulative AIA at every task, rather than only the endpoint. AIA is the average of the accuracies measured after each task learned so far, with each evaluation covering all classes seen at that stage. We divide each sequence into five equal intervals (four tasks per interval for 20-task sequences; two for 10-task sequences). Each entry averages the per-task cumulative AIA gains within that interval (using class order 1993/1994/1995).

DatasetAIA gain (pp)0–20%20–40%40–60%60–80%80–100%
CIFAR-100FDC − baseline+0.15+1.05+1.63+1.93+2.15
FDC (refreshed statistics) − baseline+0.27+1.39+2.35+2.90+3.25
CUB-200FDC − baseline+1.37+3.52+4.58+5.12+5.66
FDC (refreshed statistics) − baseline+1.50+3.76+4.88+5.52+6.16
ImageNet-RFDC − baseline+0.97+1.63+2.12+2.44+2.71
FDC (refreshed statistics) − baseline+1.01+1.92+2.51+2.88+3.22
ImageNet-AFDC − baseline+2.81+4.33+5.13+5.52+5.77
FDC (refreshed statistics) − baseline+2.70+4.34+5.23+5.68+5.97
OmniBenchmarkFDC − baseline+0.52+1.47+1.95+2.30+2.66
FDC (refreshed statistics) − baseline+0.82+2.53+3.48+4.25+4.91

These results show that FDC's gains persist during incremental learning, and the interval-averaged AIA gains increase as more tasks are learned on all five datasets.

Feature drift does occur, and refreshing PAC/RLC statistics using historical training images under the current encoder further improves performance overall. Nevertheless, FDC remains effective with historical statistics despite this drift. Refreshing requires revisiting old training data; approximating it without retaining old data is a direction for future work.

W4: Dual-view calibration requires an additional encoder forward pass and nearly doubles inference latency in the reported experiments. The adapted-only version is much cheaper and already performs well, so the practical trade-off between the two versions deserves more emphasis.

A4: We agree that the extra view has a substantial cost. ET provides most of the improvement, while dual requires a second encoder view.

ConfigurationMean final ACC (%)Gain over baseline (pp)Encoder views
Cumulative baseline70.541—1
FDC ET75.739+5.1981
FDC dual76.817+6.2762

ET retains 82.8% of the dual version's gain; dual adds 1.078 pp over ET.

For convenience, we summarize the inference-time measurements already reported in Appendix C.2, Table 7 of the submitted paper:

DatasetBaseline (ms/batch)FDC ET (ms/batch)FDC dual (ms/batch)
CIFAR-100170.5173.3 (+1.7%)323.1 (+89.5%)
CUB-200171.3174.1 (+1.6%)324.2 (+89.3%)
ImageNet-R170.5171.7 (+0.7%)320.1 (+87.7%)
ImageNet-A171.0172.5 (+0.8%)321.5 (+88.0%)
OmniBenchmark169.9171.3 (+0.8%)319.2 (+87.9%)

These measurements use Cumulative-LoRA's seed-1 configurations on one NVIDIA A100 80GB PCIe, FP32, and batch size 64, with the same final checkpoint across variants. ET adds only 0.7–1.7% latency, whereas dual takes 1.88–1.90× the baseline latency. We therefore emphasize ET as the cost-efficient default and present dual as an optional accuracy–latency trade-off.

References

[1] ViM: Out-of-Distribution With Virtual-Logit Matching. CVPR 2022.

[2] ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection. ICLR 2024.

[3] iCaRL: Incremental Classifier and Representation Learning*. CVPR 2017. (Nearest-class-mean reference; our cosine scoring comparison uses the shared checkpoints and training class means, without iCaRL training or exemplar selection.)

[4] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS 2018.

[5] Mahalanobis++: Improving OOD Detection via Feature Normalization. ICML 2025.

[6] FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning*. NeurIPS 2023. (We use the pretrained-ViT scoring implementation in PILOT: https://github.com/sun-hailong/LAMDA-PILOT/blob/main/models/fec)

Dear Authors,

Thanks for the detailed answers to my questions. I claim here that my concerns were succesfully addressed hence I maintain my positive score. I am not sure that I can raise my score to 4 for this paper but this is due to coarse score division of current year's ICLR. Just for the record and ACs, if score division was 'weak accept, accept, strong accept', I think I would give accept instead of weak accept after rebuttal.

Thank you for your positive assessment and constructive feedback. We are glad that our responses have addressed your concerns. We will incorporate the additional experimental results, analyses, and clarifications presented in our rebuttal into the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Request for Reviewer Feedback

Dear Reviewers,

Thank you again for your constructive feedback. We have provided detailed responses and additional experiments addressing your concerns. For ease of reference, we summarize the main updates below.

  1. Full-component results and held-out validation (Hydr W1; PM6G W1/Q1; D3jP Q1). We will make full-component FDC the main comparison. Across eight learners (InfLoRA, InfLoRA-CA, EWC-LoRA, SD-LoRA, Split-LoRA, LoRA-Sub, RSIAT, and LoDA), five datasets, and three class orders, full-component ET improves accuracy in 107/120 evaluations, with an average gain of 3.060 percentage points. For leave-one-dataset-out component selection, each of the five datasets is held out in turn, with components selected on the other four. This achieves an average gain of 3.482 percentage points, improving 109/120 evaluations. We also provide a preliminary analysis of the failures concentrated in LoRA-Sub and RSIAT.

  2. Comparisons with and complementarity to established statistical scoring methods (Hydr W2; PM6G Q3; D3jP Q3). We compared FDC with six scoring methods covering subspace residuals (ViM-style), density estimation (ConjNorm), prototype similarity (Prototype), and covariance-aware distances (Shared Mahalanobis, Mahalanobis++, and FeCAM), using the same frozen learner checkpoints. All methods use a single encoder view from the adapted encoder (ET). Across the 120 evaluations, FDC achieves the largest average gain and the fewest accuracy decreases. Applying FDC on top of each of the six scoring methods further improves them. For Prototype, Shared Mahalanobis, Mahalanobis++, and FeCAM, adding FDC improves mean accuracy by 0.961–1.970 percentage points over the respective scoring method, while also exceeding standalone FDC by 0.606–0.802 points. Simple logit and temperature calibration were also tested on Cumulative-LoRA, with neither improving accuracy over the baseline.

  3. Performance throughout incremental learning and feature drift (Hydr W3; PM6G W2/Q2; D3jP W3/Q2). We added task-by-task cumulative AIA results across class orders 1993/1994/1995. The interval-averaged cumulative AIA gains remain positive and increase steadily across the five learning intervals on all five datasets. FDC with historical statistics retains 80.6% of the cumulative AIA gain achieved by refreshing both PAC and RLC, averaged over the final 20% of tasks (+3.790 vs. +4.702 percentage points over the baseline). Refreshing RLC alone adds only 0.451 points in classification accuracy and 0.474 points in routing accuracy

  4. Gaussian approximation and mean-preserving TSF (D3jP W1/W2). We tested alternative residual distributions, including skew-normal and Beta, but found no appreciable average accuracy improvement, even after refitting the calibration scale. We therefore retain the Gaussian model for now. Besides, the suggested mean-preserving TSF variant reduces accuracy by 0.414 percentage points overall. We suspect that attenuating task-specific mean components helps reduce bias in cross-task competition.

  5. Single-view accuracy–latency trade-off (Hydr W4; PM6G Q3; D3jP Q3). We emphasize the single-view version using only the adapted encoder (ET) as the cost-efficient default. On Cumulative-LoRA, it retains 82.8% of the dual-view accuracy gain, with only 0.7–1.7% latency overhead in the submitted paper’s timing measurements. Adding the frozen pretrained encoder as a second view improves accuracy by a further 1.078 percentage points, but requires 1.88–1.90× baseline latency.

The experimental details and results are provided in our individual responses. We would appreciate your feedback on whether these additions address your concerns and would be happy to clarify any remaining questions during the discussion period.

Thank you for your time and consideration.

The Authors of Paper 15149

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.