acceptodds
Under review as a conference paper at ICLR 2027

Locally Correct, Globally Wrong: Task Identity Recovery for Replay-Free Multi-Source Class-Incremental Learning

Abstract

Replay-free multi-source class-incremental learning requires models to learn from changing data sources without revisiting past data. Beyond forgetting, two challenges arise. First, independently trained source-specific classifiers may produce scores that are not jointly calibrated, so a global decision over all classes can be wrong even when the corresponding local classifier is correct. Second, new sources may introduce new classes or repeat existing label sets, requiring the model to decide whether to create a new task branch or adapt an existing one. We address this problem with HiDMoA (Hierarchical Decoupled Mixture-of-Adapters). The key idea is to introduce an intermediate task level, where tasks are characterized by source-specific feature distributions. These distributional representations enable two-stage classification: first recovering the input's task identity, then classifying the input within that task's label space, thereby avoiding direct comparisons of scores across independently trained source-specific classifiers. Specifically, HiDMoA uses one source fVAE (feature variational autoencoder) per source for reconstruction-based task identification, while each task branch combines task-branch adapters and a local cosine head on a frozen pretrained backbone for within-task classification. The same reconstruction evidence determines whether a source with a repeated label set joins an existing task branch or starts a new one. Reused task branches are updated through replay-free distillation. On an 8-source, 50-class industrial surface-defect stream, HiDMoA achieves 93.1%, 93.4%, and 92.6% Last Macro-F1 across three source orders, outperforming all evaluated baselines in each order. Task-identity accuracy reaches 99.72%, bringing Last Macro-F1 within one point of an oracle given the true task identity. On an 8-source, 135-class general multi-source stream, HiDMoA-ResNet-50 achieves 95.4% average Macro-F1 and leads at seven of eight evaluation steps. On a repeated-label stream, HiDMoA achieves 95.8% Last Macro-F1, compared with 85.4% for the strongest evaluated baseline.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
class-incremental learning, task-identity inference, adapter-based continual learning, feature-space generative routing, industrial defect recognition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ytf9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage class-incremental learning (CIL) framework called HiDMoA especially for multi-source image classification tasks. The first stage determines the task identity based on reconstruction errors from source-specific VAEs, and the second stage use the task-specific classifier with associated task-branch adapters to predict the label of the input. Empirical evaluations on two benchmarks supports the efficacy of the proposed method.

Critical strengths

The main novelty is the strategy to determine the task identity based on source-specific feature VAE (fVAE). Such reconstruction error-based strategy has not been applied for CIL before.

Critical weaknesses
  1. The two-stage CIL strategy is overall not novel and “formalize how task-identification errors contribute to global classification risk” has been essentially analyzed in the earlier work (e.g., Kim et al., 2022).
  2. The application of the proposed source-specific fVAE is very limited. It implicitly assumes that different sources (datasets) are quite distinct and the feature distributions are clearly separated from each other in the shared encoder output (feature) space. However, such assumption is too strict to be applicable in practice. When different sources contain visually similar but different classes, such fVAE strategy would be likely ineffective to determine the task identity.
  3. The empirical evaluations are quite limited and not convincing. First, for each benchmark, the sources are so different from each other such that multi-source CIL becomes an easier rather than more challenging CIL problem. Mult-source benchmark dataset where at least part of sources are visually similar to each other should be inlcued for empirical evaluation, e.g., by separating certain original dataset into multiple sources at the class level (i.e., without label repeat). Second, one crucial ablation study is missing, i.e., replacing the proposed fVAE by a simple multi-source classifier based on feature replay in the encoder’s output feature space, where Gaussian distribution can be estimated for each source during CIL for feature replay. Third, when multiple sources of datasets contain different classes but share the same task branch, how could the proposed strategy work? Note that the methodology description and relevant empirical evaluations assume multiple sources share repeated class labels when sharing the same task branch. Fourth, the proposed method includes multiple hyper-parameters, but relevant sensitivity studies are missing.
Questions for authors
  1. Please clarify the really novel part for the proposed method by comparing closely related work.
  2. How would the proposed method work effectively if multiple sources datasets contain visually similar but semantically different classes of images?
  3. Please provide missing empirical evaluations to support the efficacy and necessity of the proposed fVAE strategy.
9 comments
AuthorsOpenReview ↗

Response to Reviewer 3 — Ytf9 — Weakness 1 and Question 1 — Task-selection mechanism and evidence (Part 1/3)

Thank you for the suggestions on related work and more challenging evaluations. We summarize the revisions and new results in response to your comments below.

We build on the two-stage view of Kim et al. (2022) and do not claim it as new. The novel parts of HiDMoA are the following. First, the evidence for task selection: source models are learned separately from the class scores, and our analysis shows that the class scores of fixed local heads cannot in general identify the task, however they are calibrated. Second, the organization of classifiers: persistent source models are separated from task branches, so that compatible sources can share a classifier, a mechanism that Expert Gate names as future work. Third, the method is replay-free, whereas the closest task-identity methods, MORE and TPL, rely on stored samples.

Part 1 addresses task selection; Part 2 covers source-fVAE modelling and router controls; Part 3 covers classifier sharing and complete-method comparisons.

  1. Relative to Kim et al. (2022), the new element is the evidence used for task selection. Independently learned local heads can classify their own inputs correctly while a larger score from an unrelated head wins the global decision. Without previous-source samples for joint score alignment, effective within-task classification alone leaves this selection problem unresolved. Within the framework of Kim et al.’s A Theoretical Study on Solving Continual Learning (2022), our contribution concerns how source evidence is learned and retained, and how it determines which classifier should process an input.

    Kim et al. establish a probabilistic decomposition into within-task prediction and task-identity prediction and relate the corresponding cross-entropy losses to OOD detection. Their analysis treats task-identity prediction as given and links it to OOD detection; ours asks which evidence can support task selection once the local heads are fixed, and differs in three respects. First, our fixed-head analysis characterizes how task selection affects classification risk under disjoint labels (Proposition 2). Second, it identifies a limit of score-based selection: per-branch monotone recalibration selects a branch from the top scores of the local heads; when these scores leave task identity ambiguous, no recalibration can resolve the ambiguity (Appendix B.2), whereas separate source evidence can use information beyond these scores. Third, Appendix B.1 (Eq. 9) permits several branches to contain the same label and decomposes the classification error accordingly. This preserves the source-to-branch organization, whereas Remark 3 of Kim et al. assigns recurring classes to unique tasks.

    The Appendix B.2 construction has perfect local heads but identical top scores for two equally likely sources, giving 1/2 error to selection based only on that statistic; a separating source representation permits zero error with the same heads. Proposition 2 and Sec. 4 isolate this information limitation from Proposition 1’s additional A5/A6 conditions. The existing Table 2 tests the decision rules with classification branches fixed: Last Macro-F1 is 75.65 with global maximum, 86.53 with fitted affine calibration, and 92.47 with source-fVAE routing. The 5.94-point gain over the fitted calibration locates a benefit in branch selection.

    Fixed-classifier selection rules (Table 2). Scores are percentages, mean ± sample SD over seeds 42/43/44.

    Fixed-head decision ruleLast Macro-F1Avg Macro-F1
    Global maximum75.65±2.1584.04±2.88
    Task-wise affine calibration86.53±1.3591.59±0.28
    Source-fVAE routing92.47±0.4495.39±0.23

    We have revised the first contribution in the Introduction to state these distinctions. In practice, MORE and TPL use a 600-sample replay memory in our benchmark (Table 20), whereas HiDMoA uses separately retained source models without replay and, with the same DeiT-Small classifier, exceeds TPL and MORE by 7.4 and 8.6 Last Macro-F1 points averaged over the three industrial orders (Table 1).

AuthorsOpenReview ↗

Response to Reviewer 3 — Ytf9 — Weakness 1 and Question 1 — Source-fVAE modelling and router controls (Part 2/3)

  1. Source fVAEs learn input-distribution evidence separately from local class scores. Each source is fitted independently in the same frozen routing feature space and retains its own fVAE. This avoids allowing a larger source to dominate a pooled source-fitting objective and preserves earlier source representations without revisiting their inputs. Both deterministic AE and fVAE use reconstruction evidence; our fVAE learns input-dependent latent means and variances, trains reconstruction over sampled latent neighbourhoods, and averages reconstruction error over the posterior at inference. The common-prior KL term regularizes this representation. Gaussian-FR already models covariance, so the controlled comparison tests the full source-model design rather than attributing every gain to variance alone (Sec. 3.2 and Appendix D.9).

    Table 3 fixes classification branches and uses the same frozen ResNet-18 routing features. On CR7, AE, Gaussian-FR, and source-fVAE achieve 76.87, 90.02, and 95.12 Task-ID accuracy and 73.45, 89.79, and 94.47 Last Macro-F1, respectively. Source-fVAE has the highest mean on both metrics across all six streams. The joint comparison of Task-ID accuracy and F1 directly examines the task-identification benefit independently of classifier capacity; its magnitude varies across streams.

    Fixed-classifier router controls (Table 3). The first three numeric columns are Last Macro-F1; the last three are Last Task-ID Accuracy, all in percent, mean ± sample SD over seeds 42/43/44. Both metrics are evaluated at the final session. “Source-fVAE” corresponds to the task-fVAE router when each source has its own task branch.

    StreamSessionsAE F1Gaussian-FR F1Source-fVAE F1AE Task-IDGaussian-FR Task-IDSource-fVAE Task-ID
    NEU391.41±2.5899.01±0.5799.51±0.2191.48±2.5999.01±0.5799.51±0.21
    CR7373.45±3.3689.79±0.5694.47±0.2476.87±2.5190.02±0.5895.12±0.41
    Fashion-MNIST586.94±0.3386.72±0.5089.97±0.2289.79±0.2489.70±0.4092.74±0.16
    MNIST594.79±0.1495.19±0.4897.39±0.0995.40±0.0795.75±0.4298.03±0.12
    EuroSAT588.63±0.7293.12±0.1693.34±0.5489.82±0.6894.19±0.0394.36±0.62
    Steel, order 1892.35±0.7091.91±0.9092.47±0.4499.68±0.0699.43±0.2699.72±0.00
AuthorsOpenReview ↗

Response to Reviewer 3 — Ytf9 — Weakness 1 and Question 1 — Source-to-task organization and complete-method evidence (Part 3/3)

  1. Persistent source models are separated from selectively shared classification branches. A source describes an input population, while a task branch supplies a label-prediction function; these need not correspond one-to-one. HiDMoA retains one fVAE per source while deciding whether an arriving source needs a new classifier. Sources with identical label sets may share a branch after passing the reconstruction-compatibility test. Their source models remain separate after sharing, and the reused branch is updated with current-source supervision and a pre-update teacher on current inputs. Expert Gate uses reconstruction-related evidence to select a parent and a transfer procedure for a new expert. It adds a new expert for every task and names “a mechanism to decide when to merge tasks with high relatedness degree rather than adding a new expert model” as future work (Aljundi et al., 2017, Sec. 5). In HiDMoA, the same evidence also determines classifier creation or reuse, with a shared frozen backbone and local adapter and head updates (Secs. 2–3 and Appendix G).

    Table 4 isolates allocation on the six-source repeated-label steel stream, holding the classifier and router configuration fixed and retaining six source fVAEs for every policy. Adaptive allocation reaches 96.01±0.66 Last Macro-F1, compared with 95.22±1.35 for always-new and 95.31±0.21 for always-reuse, using 4/4/5 branches instead of six. Old-source and current-source accuracy are 97.92 and 95.24, compared with 97.52 and 93.54 for always-new. This supports selective sharing with fewer classifiers and higher mean classification performance. Scores are percentages, mean ± sample SD over seeds 42/43/44; old/current denote previously arriving and current sources.

    Steel allocation control (Table 4):

    PolicyLast Macro-F1Old-source AccCurrent-source AccBranches (42/43/44)
    Always new95.22±1.3597.52±0.5393.54±2.576/6/6
    Same-label always reuse95.31±0.2197.57±0.1793.88±1.023/3/3
    Adaptive, γ=196.01±0.6697.92±0.3095.24±1.564/4/5
  2. As a complete method, HiDMoA exceeds Expert Gate by 10.1–10.8 Last Macro-F1 points. Complete-method comparisons evaluate the combined source-modelling and classifier-organization design. In Sec. 5.2, Table 1, HiDMoA and the Expert Gate adaptation use DeiT-Small classification backbones and frozen ResNet-18 routing features on the eight-source, 50-class industrial stream. HiDMoA exceeds Expert Gate by 10.1–10.8 Last Macro-F1 points across three source orders. Expert Gate retains independent full classifiers, whereas HiDMoA specializes a shared frozen backbone through adapters. These complete-method results complement the fixed-head and source-model controls in Parts 1 and 2 and the allocation-policy control above. Scores are percentages, mean ± standard deviation over three seeds.

    Industrial stream — direct Expert Gate comparison (Table 1):

    MethodOrder 1 Last F1Order 2 Last F1Order 3 Last F1
    Expert Gate83.0±1.3282.6±2.5882.1±1.19
    HiDMoA93.1±0.4093.4±0.2092.6±0.60
Reviewer_oJhLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies replay-free learning from a sequence of labeled visual sources, with source boundaries observed during training and no task identity supplied at test time. HiDMoA separates source modeling from task-local classification. Each source has a feature VAE trained in a shared frozen routing space, while each task branch has adapters and a local cosine classifier on a frozen classification backbone. Reconstruction errors select the branch at inference. Sources with identical label sets may share a branch if their reconstruction-error ratio passes a compatibility threshold; the reused branch is updated with distillation on current-source inputs.

Critical strengths
  1. The paper addresses a meaningful failure mode: independently trained classifiers can be locally accurate while their global score comparison selects an inappropriate branch. The separation between source evidence and class scores is coherent.
  2. Industrial-stream improvements are substantial and consistent across three source orders, with means and standard deviations over three seeds. The repeated-label results are also encouraging.
Critical weaknesses
  1. The contribution beyond reconstruction-based expert routing needs clearer isolation Expert Gate, which the paper appropriately cites, already uses feature autoencoders for test-time expert selection and reconstruction-based relatedness for transfer. The proposed compatibility ratio therefore uses closely related evidence, although its purpose here is to decide whether sources should share a classifier branch. The distinctive contribution appears to be the separation of persistent source models from potentially shared task branches, together with its analysis and implementation. Its benefit is harder to isolate on the main disjoint-label benchmarks, where every source necessarily creates a new branch. A comparison with an Expert Gate-style deterministic feature-autoencoder router would be particularly helpful.
  2. Evidence for reconstruction routing under less separable real sources remains limited Figure 4 and Appendix D provide convincing evidence that reconstruction routing works well on the industrial stream. The common frozen representation and metric also make the scores meaningful to compare. They do not, however, establish that the correct source will have the lowest expected reconstruction error: independently fitted models can differ in reconstruction difficulty or generalization to other sources. Appendix B.3 and the limitations section correctly acknowledge this distinction. The general stream includes visually heterogeneous datasets such as MNIST, EuroSAT, and Food11, where source identity may provide a relatively strong cue. The Gaussian simulation usefully varies task overlap, but its decoupled router uses raw-input mean prototypes rather than the proposed feature VAEs. It consequently supports the two-stage principle more directly than the behavior of this particular router under overlap.
Questions for authors
  1. What benefit is specific to the VAE source model beyond deterministic reconstruction-based expert routing, and how should the task-formation contribution be distinguished from reconstruction-based relatedness in Expert Gate?
  2. How does the implemented router behave when sources have similar acquisition characteristics, beyond the prototype-based Gaussian simulation?
  3. Do adaptive branch assignments improve the old-source/new-source tradeoff over simple allocation policies, and what do the repeated-label results show about preservation after merging?
  4. Could the complete inference-cost accounting and the missing feature-VAE configuration details be specified?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2 — oJhL — Weakness 1 and Questions 1, 3 — Source modelling and task formation (Part 1/2)

Thank you for highlighting the separation of persistent source models from potentially shared task branches. We added three controlled experiments: a comparison of source models with the classifiers fixed, including the Expert Gate-style autoencoder router you proposed (Table 3); classifier sharing on a repeated-label stream (Table 4); and assignment under real domain shifts (Table 12).

Tabulated scores are percentages, mean ± sample SD over seeds 42/43/44. “Old” and “current” denote previously arriving and current sources.

  1. Source populations and prediction functions need not correspond one-to-one. HiDMoA retains one fVAE per source, and sources with identical labels may share a classifier after the reconstruction-compatibility test while their source evidence remains distinct. The compatibility ratio indeed uses evidence closely related to the relatedness of Expert Gate; the difference lies in the decision it drives. Expert Gate selects a parent and a transfer rule for a new expert, whereas HiDMoA also decides whether a new classifier is needed. Expert Gate leaves this decision open: its conclusion names “a mechanism to decide when to merge tasks with high relatedness degree rather than adding a new expert model” as future work (Aljundi et al., 2017, Sec. 5). A reused branch is updated with current-source supervision and distillation from its pre-update copy (Secs. 3.2–3.4; G).

  2. With the classifiers fixed, replacing the autoencoder router with source-fVAE raises Task-ID accuracy and Macro-F1 on all six streams. The fVAE extends AE routing by modelling the variation of the reconstruction. A deterministic AE evaluates a single latent code, which can also reconstruct a similar competing source well. The fVAE learns input-dependent posterior means and variances, is trained over sampled latent neighbourhoods, and averages the reconstruction error over them, so that the expected error reflects both the fit of the mean reconstruction and its variation. The KL term regularizes the posterior location and scale toward a common standard-normal prior under unequal source sizes and class frequencies, while the local classifiers resolve the classes. Gaussian-FR already models covariance; the fVAE adds a nonlinear latent model and decoder.

    Table 3 fixes classifiers and uses common frozen ResNet-18 features. On CR7, the gains over AE and Gaussian-FR are 18.25 and 5.10 Task-ID points and 21.02 and 4.68 F1 points. Raising β from 0 to 0.01 improves F1 from 88.28 to 89.97 on Fashion-MNIST and from 91.59 to 93.34 on EuroSAT (D.7, Table 10). D.9 specifies AE fitting and random-only branch initialization; steel AE matches fVAE depth and the 128-dimensional bottleneck. Complete Expert Gate separately trains full classifiers (Table 1; G).

    Table 3: fixed-classifier controls. The first three metric columns are final-session Macro-F1 and the last three are final-session Task-ID accuracy. Source-fVAE is task-fVAE when each source has its own branch.

    StreamSessionsAE F1Gaussian-FR F1Source-fVAE F1AE Task-IDGaussian-FR Task-IDSource-fVAE Task-ID
    NEU391.41±2.5899.01±0.5799.51±0.2191.48±2.5999.01±0.5799.51±0.21
    CR7373.45±3.3689.79±0.5694.47±0.2476.87±2.5190.02±0.5895.12±0.41
    Fashion-MNIST586.94±0.3386.72±0.5089.97±0.2289.79±0.2489.70±0.4092.74±0.16
    MNIST594.79±0.1495.19±0.4897.39±0.0995.40±0.0795.75±0.4298.03±0.12
    EuroSAT588.63±0.7293.12±0.1693.34±0.5489.82±0.6894.19±0.0394.36±0.62
    Steel, order 1892.35±0.7091.91±0.9092.47±0.4499.68±0.0699.43±0.2699.72±0.00
AuthorsOpenReview ↗

Response to Reviewer 2 — oJhL — Weakness 1 and Questions 1, 3 — Source modelling and task formation (Part 2/2)

  1. On the repeated-label steel stream, adaptive assignment improves both old-source and current-source accuracy over the simple policies. The repeated-label steel stream isolates classifier sharing. The main disjoint-label benchmarks create one branch per source. Here NEU, XSDD, and Magnetic each form two class-stratified complementary sources, and all policies keep the same classifier and router settings and all six fVAEs. At γ=1, adaptive allocation improves mean F1 with 4/4/5 branches instead of six; relative to always-new, old-source and current-source accuracy increase by 0.40 and 1.70 points, and both also exceed same-label always reuse. This supports preservation after sharing on this controlled stream (Table 4; D.10, Table 12).

    Table 4: steel allocation. Branch counts list seeds 42/43/44.

    PolicyLast Macro-F1Old-source AccCurrent-source AccBranches (42/43/44)
    Always new95.22±1.3597.52±0.5393.54±2.576/6/6
    Same-label always reuse95.31±0.2197.57±0.1793.88±1.023/3/3
    Adaptive, γ=196.01±0.6697.92±0.3095.24±1.564/4/5
  2. Under real domain shifts, adaptive assignment avoids the loss of old-source accuracy caused by forced reuse. PACS and VLCS test whether independent predictors are retained under domain shifts. Each stream has four domains with shared labels; all policies use the same classifier and router settings and retain one fVAE per source. At γ=1, adaptive allocation keeps four branches, matches always-new and exceeds forced reuse by 12.29 (PACS) and 3.46 (VLCS) Last Macro-F1 points. Old-source accuracy is 95.29 and 84.11, against 74.83 and 78.22 under forced reuse. No domains merge at γ=1.

    Table 12: real-domain old/current-source tradeoff (D.10).

    StreamPolicyLast Macro-F1Old-source AccCurrent-source AccBranches (42/43/44)
    PACSAlways new95.49±0.4195.29±0.9494.62±0.554/4/4
    PACSSame-label always reuse83.20±5.7374.83±8.2294.68±0.391/1/1
    PACSAdaptive, γ=195.49±0.4195.29±0.9494.62±0.554/4/4
    VLCSAlways new83.79±1.0784.11±0.6878.82±0.734/4/4
    VLCSSame-label always reuse80.33±2.0078.22±0.6680.79±1.581/1/1
    VLCSAdaptive, γ=183.79±1.0784.11±0.6878.82±0.734/4/4
AuthorsOpenReview ↗

Response to Reviewer 2 — oJhL — Weakness 2 and Question 2 — Routing between similar sources

A common representation and metric make scores comparable, but do not guarantee the correct source has the lowest reconstruction error. On class-disjoint sessions from single datasets with shared acquisition characteristics, source-fVAE achieves 92.74–99.51% Last Task-ID accuracy and outperforms AE and Gaussian-FR on all five streams (Table 3).

  1. Independent fitting retains source structure on a common measurement scale. All fVAEs share frozen feature coordinates, architecture, latent width, prior, objective and per-dimension metric (Sec. 3.2; G). Each is fitted to its own source and frozen. The bottleneck and the KL term constrain its representation, and posterior averaging evaluates reconstruction over its learned latent neighbourhood. Distinguishable class mixtures can remain under shared acquisition styles, so a source-specialized model can better reconstruct its own inputs.

    In B.3, expected reconstruction equals a common scaling of model log-likelihood plus source-dependent variational corrections. If the correct model’s likelihood advantage dominates the absolute correction difference, reconstruction retains its source preference despite independent fitting. Table 3 tests this mechanism with fVAEs beyond the prototype simulation.

  2. Fixed-classifier tests on five single-dataset streams. NEU and CR7 have three class-disjoint sessions; MNIST, Fashion-MNIST, and EuroSAT have five. Sessions retain the acquisition pipeline and broad appearance of one dataset, reducing cross-dataset style cues. Sec. 5.3.2, Table 3 holds classification branches fixed and fits AE, Gaussian-FR, and source-fVAE on the same frozen ResNet-18 features. Source-fVAE has the highest mean Last Task-ID accuracy and Last Macro-F1 on all five streams.

    Its Last Task-ID accuracy is 99.51±0.21 on NEU, 95.12±0.41 on CR7, 92.74±0.16 on Fashion-MNIST, 98.03±0.12 on MNIST, and 94.36±0.62 on EuroSAT. On CR7, the corresponding Last Macro-F1 is 94.47±0.24; AE reaches 76.87±2.51 Task-ID and 73.45±3.36 F1, and Gaussian-FR reaches 90.02±0.58 Task-ID and 89.79±0.56 F1. Fashion-MNIST has the lowest source-fVAE Task-ID accuracy among these five streams and reaches 89.97±0.22 F1, compared with 89.70±0.40 Task-ID and 86.72±0.50 F1 for Gaussian-FR. These fixed-classifier results directly examine task identification and its classification consequences, while also showing the remaining routing difficulty.

    Table 3 reports percentages, mean ± sample SD over seeds 42/43/44; D.9 specifies protocols. Complete-method and shared-label domain comparisons are in F.3 (Table 18) and F.4 (Table 19). These tests reduce acquisition-style cues without quantifying distributional overlap.

  3. Optional sparse Top-k inference supplements Top-1 on ambiguous inputs. In Appendix D.6, Table 9, a small relative reconstruction-error gap retains the runner-up branch; otherwise, inference keeps Top-1. Relative to the reported Top-1 references, mean Last Macro-F1 is higher on PACS, VLCS, EuroSAT, and MNIST, while Fashion-MNIST remains close (89.97 versus 89.94). Only 0.45–9.49% of final-test inputs activate the second branch across the five streams.

    Validation-calibrated adaptive Top-k (Appendix D.6, Table 9). Last Macro-F1 (%), mean ± sample SD over seeds 42/43/44. The primary comparison returns one maximum-cosine class; test usage is the mean fraction activating the second branch. Thresholds are selected on validation data only.

    StreamTop-1Adaptive Top-kTest Top-2 (%)
    PACS95.49±0.4195.69±0.397.02%
    VLCS83.79±1.0783.87±1.149.49%
    Fashion-MNIST89.97±0.2289.94±0.153.34%
    EuroSAT93.34±0.5493.49±0.533.40%
    MNIST97.39±0.0997.75±0.100.45%

    PACS/VLCS use source-fVAEs with four separate classifier branches; Fashion-MNIST, EuroSAT, and MNIST use task-fVAE router controls with random-only, fixed classification branches. Maximum-cosine selection compares class scores within the retained candidate set; restricting that set limits cross-branch competition, but does not eliminate possible score mismatch between the candidate heads.

  4. The remaining boundary is source identifiability and the consequence of a wrong branch choice. Appendix B.3 states the required positive expected-score margin relative to sampling error. More posterior samples reduce estimation noise, but cannot repair an incorrect expected ranking when source distributions overlap. With disjoint label sets, an incorrect branch excludes the correct class; with shared labels, source confusion need not prevent a correct semantic prediction. The reported Task-ID and F1 therefore describe complementary aspects of routing behavior. The prototype simulation in Appendix E illustrates the two-stage principle, and Appendix I records the remaining overlap limitation.

Reviewer_Pi61OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiDMoA for replay-free multi-source class-incremental learning. It uses feature VAEs to identify the task of each test sample, then applies task-specific adapters and local classifiers. Sources with matching label sets can share a task branch, which is updated through distillation. Experiments on industrial and general multi-source benchmarks show improvements over several existing methods.

Critical strengths
  1. Clear motivation. The paper shows that independently trained local classifiers may produce scores that are not comparable across tasks. Table 2 supports the benefit of separating task identification from classification.

  2. Strong experimental results. HiDMoA achieves 92.6–93.4% Last Macro-F1 across three source orders on the industrial benchmark. The routing ablation also reports 99.72% task-ID accuracy, close to oracle performance.

Critical weaknesses
  1. Limited novelty. HiDMoA shares several ideas with Expert Gate, including reconstruction-based routing, expert reuse, and knowledge transfer. The theoretical contribution is also limited, as Proposition 1 relies on assumptions A5–A6 about the relative risk and complexity of the compared methods.

  2. Baseline comparisons. The baselines use different backbones, while HiDMoA also uses a separate ResNet-18 for routing. On the general benchmark, HiDMoA-DeiT achieves 93.3% average Macro-F1, slightly below TPL's 93.6%. The stronger result of 95.4% uses ResNet-50 without a matched-backbone comparison. This makes it difficult to judge the source of the gains.

  3. Inference scalability. HiDMoA evaluates every stored source fVAE at test time. Although Table 6 shows that fewer Monte Carlo samples can work well, routing cost still grows with the number of sources. The paper does not measure inference latency on longer streams.

  4. Limited source-merging evaluation. The repeated-label experiment splits existing datasets into random halves. It does not test merging across real domain shifts. The method also requires identical label sets and lacks a direct comparison with always creating a new branch.

Questions for authors
  1. What is the main difference between HiDMoA and Expert Gate? Can the authors explain how assumptions A5–A6 apply to the proposed model?

  2. Can the authors provide comparisons using the same backbone and pretraining settings, including the additional routing network? A direct comparison with Expert Gate would also be useful.

  3. How does inference latency change as more sources are added? Can the authors compare the default routing setup with a smaller sampling budget on longer streams?

  4. Does source merging improve performance over always creating a new branch? How does the method handle sources with the same labels but different distributions, and how sensitive is it to the merging threshold?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1 — Pi61 — Weakness 1 and Question 1 — Methodological contribution (Part 1/2)

Thank you for the constructive questions. We address your concerns and describe the corresponding revisions below.

  1. Both methods route inputs using source-specific reconstruction models. HiDMoA further addresses three questions left open by Expert Gate.

    (1) When should a new source share an existing classifier? Expert Gate uses reconstruction-based relatedness to select a parent expert and transfer rule (fine-tuning or LwF), then retains a separate full classifier per task (Aljundi et al., 2017, Sec. 1). Its authors identify the alternative as future work: “a mechanism to decide when to merge tasks with high relatedness degree rather than adding a new expert model, seems an interesting research direction” (Sec. 5). HiDMoA provides such a mechanism. It retains one fVAE per source together with a source-to-task mapping, so that sources with identical labels can share a compatible classifier while keeping distinct source models; input populations and prediction functions need not correspond one-to-one. On the six-source steel stream, sharing reduces the number of classifiers from six to four or five and attains 96.01 Last Macro-F1, compared with 95.22 when every source receives its own classifier (Table 4). On PACS and VLCS, whose domains differ in appearance, HiDMoA keeps the classifiers separate, whereas forced sharing lowers Last Macro-F1 by 12.29 and 3.46 points, respectively (Table 12).

    (2) Why does selection by class scores fail, and can calibration repair it? Expert Gate reports this failure empirically: “just using the output of the highest scoring expert is no guarantee for success as neural networks can erroneously give high confidence scores” (Sec. 1); its most confident expert reaches 48.7% average accuracy, against 67.3% for its gate (its Table 2). Our analysis explains why. When the top scores of the local heads do not reveal which source an input comes from, no rescaling of these scores can recover it (Proposition 2; Appendix B.2). The experiments confirm this: with the classifiers fixed, Last Macro-F1 is 75.65 under the global maximum, 86.53 after affine calibration, and 92.47 with source routing (Table 2).

    (3) How should a source be represented for routing? Expert Gate uses “a one layer under-complete autoencoder” (Sec. 1), which maps each input to a single code. HiDMoA represents each source by a feature VAE and scores an input by its reconstruction error averaged over posterior samples, so that the score reflects the distribution of the source rather than a single code. With the classifiers and routing features fixed, source-fVAE has higher mean Task-ID accuracy and Last Macro-F1 than the autoencoder on all six streams, with Task-ID gains of 2.63–18.25 points on the five single-dataset streams (Table 3). On CR7, Task-ID accuracy rises from 76.87 to 95.12 and Last Macro-F1 from 73.45 to 94.47; the gains over Gaussian-FR are 5.10 and 4.68 points, respectively. The KL term also contributes: raising β from 0 to 0.01 increases Last Macro-F1 from 88.28 to 89.97 on Fashion-MNIST and from 91.59 to 93.34 on EuroSAT (D.7, Table 10), which supports regularization of the source model. In addition, Expert Gate stores a full network of about 220 MB for every task (Sec. 4.2), whereas HiDMoA trains only adapters and a local head on a shared frozen backbone, which reduces the classifier weights that require gradients and optimizer states (Secs. 2–3; Appendix G).

    Table 3: fixed-classifier router controls. Both metrics are final-session percentages, mean ± sample SD over seeds 42/43/44. The first three numeric metric columns are Last Macro-F1; the last three are Last Task-ID accuracy. Source-fVAE is task-fVAE when each source has its own task branch.

    StreamSessionsAE F1Gaussian-FR F1Source-fVAE F1AE Task-IDGaussian-FR Task-IDSource-fVAE Task-ID
    NEU391.41±2.5899.01±0.5799.51±0.2191.48±2.5999.01±0.5799.51±0.21
    CR7373.45±3.3689.79±0.5694.47±0.2476.87±2.5190.02±0.5895.12±0.41
    Fashion-MNIST586.94±0.3386.72±0.5089.97±0.2289.79±0.2489.70±0.4092.74±0.16
    MNIST594.79±0.1495.19±0.4897.39±0.0995.40±0.0795.75±0.4298.03±0.12
    EuroSAT588.63±0.7293.12±0.1693.34±0.5489.82±0.6894.19±0.0394.36±0.62
    Steel, order 1892.35±0.7091.91±0.9092.47±0.4499.68±0.0699.43±0.2699.72±0.00
  2. As a complete method, HiDMoA outperforms Expert Gate on every stream we tested. With the same DeiT-Small classifiers and frozen ResNet-18 routing features, HiDMoA exceeds Expert Gate by 10.1–10.8 Last Macro-F1 points across the three industrial orders (Table 1; per-order results appear in our response to Weakness 2) and by 2.61–20.16 points across the six within-dataset streams (Table 18). On PACS, Last Macro-F1 rises from 92.70 to 95.49 and Task-ID accuracy from 89.65 to 97.50; on VLCS, they rise from 80.82 to 83.79 and from 71.09 to 83.16 (Table 19).

AuthorsOpenReview ↗

Response to Reviewer 1 — Pi61 — Weakness 1 and Question 1 — Justification and scope of A5–A6 (Part 2/2)

The theoretical contribution does not rest on A5–A6. These two assumptions are used only for the ordering in Proposition 1, whose role is to position the three classes of predictors. The main results hold without them, and the experiments support both assumptions for HiDMoA.

  1. Four results hold without A5–A6.

    • Proposition 2. With the same local heads, the risk difference between calibrated score-based selection and separate routing is determined entirely by the inputs on which the two rules select different branches (Eq. 7). It requires only disjoint label sets and fixed heads (A1–A4), which hold by construction on the two main benchmarks.

    • Limit of calibration (Appendix B.2). Any per-branch recalibration selects a branch from the top scores of the local heads. If these scores leave the task ambiguous, calibration cannot remove the resulting error. In our constructive example this error is 1/2 with perfect local heads, whereas a router with separate source evidence makes no error.

    • Risk decomposition for repeated labels (Appendix B.1). The classification error equals the routing error plus the local error, also when label sets overlap (Eq. 9). Assigning several sources to one branch does not raise the routing error above the source-identification error (Corollary 2).

    • Conditions for correct routing (Appendix B.3). Reconstruction routing is correct when the reconstruction margin exceeds the Monte Carlo error, which yields an explicit sampling budget (Proposition 5; Corollary 4).

  2. A5–A6 formalize when separate source evidence offers a useful tradeoff between risk and model complexity. Their motivation is the division between identifying an input’s source and predicting its class within a local branch.

    A5 describes an information advantage for source-based selection. It requires that a router using separate source evidence makes fewer selection errors than the best rule based only on the top scores of the local heads. Its condition, εˉ+ηR≤emin⁡cal\bar\varepsilon+\eta_{\mathcal R}\le e^{\mathrm{cal}}_{\min}, is appropriate when the local heads classify well within their own tasks, their maximum scores leave ambiguity about task identity, and the frozen source features retain information that resolves this ambiguity. This is an average-risk condition; it does not require perfect separation of every source or correct routing of every input. Appendix B.3 gives reconstruction-margin and Monte Carlo conditions for correct fVAE routing, and the B.2 example above gives a regime in which A5 holds.

    The experiments are consistent with A5. On the industrial stream, source routing identifies the task with 99.72% accuracy and brings Last Macro-F1 within one point of the oracle given the true task (Figure 4). On PACS and VLCS, Task-ID accuracy is 97.50% and 83.16% for HiDMoA, against 96.79% and 76.24% for TPL, which predicts the task from OOD and likelihood-based scores (Table 19). In the controlled simulation, separate routing exceeds the calibrated global classifier by 20.0–26.5 accuracy points in all four settings (Table 13).

    A6 expresses the intended allocation of function flexibility. Affine calibration is the least flexible of the three classes, joint learning is the most flexible, and source routing lies between them. Calibration adjusts 2K2K coefficients but cannot recover source information absent from the top scores of the local heads. HiDMoA gives source selection a nonlinear model in a frozen feature space, with fixed hidden and latent dimensions; the joint class additionally permits changes to the recognition representation and class decision. On the eight-source industrial stream, for example, affine calibration adjusts 16 coefficients, whereas joint learning would adapt the backbone together with a 50-class head. This division motivates a router with enough flexibility to distinguish sources within a constrained representation. A6 itself remains an assumption on the function classes.

  3. The fixed-head experiment tests the selection rules directly. Table 2 keeps the local heads fixed and changes only the selection rule: Last Macro-F1 rises from 75.65 with global maximum selection and 86.53 with affine calibration to 92.47 with source routing. The 5.94-point gain over fitted calibration supports the use of separate source evidence. Table 3 in Part 1 of this response further tests source identification with fixed classifiers. Sec. 4 and Appendix B specify the assumptions and independent conclusions, and D.11 connects them to the ablations.

    Fixed-classifier selection rules (Table 2). Scores are percentages, mean ± sample SD over seeds 42/43/44.

    Fixed-head decision ruleLast Macro-F1Avg Macro-F1
    Global maximum75.65±2.1584.04±2.88
    Task-wise affine calibration86.53±1.3591.59±0.28
    Source-fVAE routing92.47±0.4495.39±0.23
AuthorsOpenReview ↗

Response to Reviewer 1 — Pi61 — Weakness 2 and Question 2 — Matched-backbone and Expert Gate comparisons

The gains are not explained by the backbone. We address your points in turn: matched classifier backbones, the additional routing encoder, the direct comparison with Expert Gate, and the source of the gains.

  1. On the industrial stream, the comparison with the task-identity baselines already uses the same classifier backbone. In Table 1, HiDMoA, BUILD, MORE, TPL, and Expert Gate all use DeiT-Small classifiers; the revised table lists the backbone of every method. Averaged over the three source orders, HiDMoA exceeds TPL by 7.4 and MORE by 8.6 Last Macro-F1 points. Moreover, MORE and TPL use a 600-sample replay memory (Appendix G, Table 20), whereas HiDMoA stores no previous samples. The remaining baselines keep the backbones of their original implementations: ResNet-18, the PEC CNN, and, for SEMA, ViT-B/16, which is larger than DeiT-Small.

  2. With matched ResNet-50 classifiers, HiDMoA retains its lead on the general stream. The comparison must separate the learning strategy from the quality of the pretrained representation. A stronger classifier alone could improve local recognition without validating source-based selection. We therefore match ImageNet-1K V2 ResNet-50 classifiers, pretrained weights, source order, splits, and seeds 42/43/44 on the 8-source, 135-class general stream. HiDMoA gains 4.8 Last and 4.8 Avg Macro-F1 points over TPL, and 5.6 and 7.5 points over MORE. This supports an advantage beyond the classifier-backbone choice. Sec. 5.2 and Appendix F.2, Table 16 report the comparison and retain the DeiT result (93.3 versus TPL’s 93.6 Avg Macro-F1). With DeiT-Small, the two methods are close: after the final source, HiDMoA reaches 94.1 against 93.8 for TPL, and the 0.3-point difference in the average arises at the first step (92.6 against 95.7 on CIFAR-10 alone), where no task selection is involved. HiDMoA obtains this result without the replay memory that TPL uses. Appendix G specifies method-specific optimization: at most 50 epochs with patience 5 for HiDMoA, and 40 epochs with patience 15 for TPL and MORE.

    General stream — matched ResNet-50 classifiers (Table 16):

    Method (ResNet-50)Last Macro-F1Avg Macro-F1
    HiDMoA95.9±0.0095.4±0.10
    TPL91.1±0.5190.6±0.17
    MORE90.3±0.7087.9±0.22
  3. The additional ResNet-18 routing encoder is included in all reported costs and does not explain the gains. The encoder is frozen, shared by all sources, and used only to select the task branch; classification relies on the DeiT-Small branch alone. Including this encoder and all source-fVAE scores, the forward cost of HiDMoA is 18.37G FLOPs per input, compared with 71.70G for TPL and 71.71G for MORE (Appendix H, Table 21). Expert Gate uses the same frozen ResNet-18 routing features and DeiT-Small classifiers and is 10.1–10.8 points lower (item 4), and Table 3 varies only the source model on these features.

  4. In a direct comparison, HiDMoA exceeds Expert Gate by 10.1–10.8 Last Macro-F1 points. The Expert Gate comparison tests an alternative way of organizing the same recognition problem. Both methods use DeiT-Small classifiers and frozen ResNet-18 routing features on the 8-source, 50-class industrial stream. Expert Gate learns and retains an independent full classifier per source through relatedness-based transfer; HiDMoA specializes a shared frozen backbone through adapters and permits compatible sources to share a branch. The gain holds for each of the three orders (Sec. 5.2, Table 1). The fixed-classifier router control in Table 3 then isolates the choice between AE and fVAE separately from these complete-method differences. Scores are percentages, mean ± standard deviation over three seeds.

    Industrial stream — direct Expert Gate comparison (Table 1):

    MethodOrder 1 Last F1Order 2 Last F1Order 3 Last F1
    Expert Gate83.0±1.3282.6±2.5882.1±1.19
    HiDMoA93.1±0.4093.4±0.2092.6±0.60
  5. The source of the gains is isolated with the classifiers held fixed. Table 2 keeps the same local heads and changes only the decision rule: Last Macro-F1 is 75.65 with the global maximum, 86.53 with affine calibration, and 92.47 with source routing. The gain therefore comes from task selection, not from the backbone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.