- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a two-stage class-incremental learning (CIL) framework called HiDMoA especially for multi-source image classification tasks. The first stage determines the task identity based on reconstruction errors from source-specific VAEs, and the second stage use the task-specific classifier with associated task-branch adapters to predict the label of the input. Empirical evaluations on two benchmarks supports the efficacy of the proposed method.
Critical strengths
The main novelty is the strategy to determine the task identity based on source-specific feature VAE (fVAE). Such reconstruction error-based strategy has not been applied for CIL before.
Critical weaknesses
- The two-stage CIL strategy is overall not novel and “formalize how task-identification errors contribute to global classification risk” has been essentially analyzed in the earlier work (e.g., Kim et al., 2022).
- The application of the proposed source-specific fVAE is very limited. It implicitly assumes that different sources (datasets) are quite distinct and the feature distributions are clearly separated from each other in the shared encoder output (feature) space. However, such assumption is too strict to be applicable in practice. When different sources contain visually similar but different classes, such fVAE strategy would be likely ineffective to determine the task identity.
- The empirical evaluations are quite limited and not convincing. First, for each benchmark, the sources are so different from each other such that multi-source CIL becomes an easier rather than more challenging CIL problem. Mult-source benchmark dataset where at least part of sources are visually similar to each other should be inlcued for empirical evaluation, e.g., by separating certain original dataset into multiple sources at the class level (i.e., without label repeat). Second, one crucial ablation study is missing, i.e., replacing the proposed fVAE by a simple multi-source classifier based on feature replay in the encoder’s output feature space, where Gaussian distribution can be estimated for each source during CIL for feature replay. Third, when multiple sources of datasets contain different classes but share the same task branch, how could the proposed strategy work? Note that the methodology description and relevant empirical evaluations assume multiple sources share repeated class labels when sharing the same task branch. Fourth, the proposed method includes multiple hyper-parameters, but relevant sensitivity studies are missing.
Questions for authors
- Please clarify the really novel part for the proposed method by comparing closely related work.
- How would the proposed method work effectively if multiple sources datasets contain visually similar but semantically different classes of images?
- Please provide missing empirical evaluations to support the efficacy and necessity of the proposed fVAE strategy.
9 comments
Response to Reviewer 3 — Ytf9 — Weakness 1 and Question 1 — Source-fVAE modelling and router controls (Part 2/3)
-
Source fVAEs learn input-distribution evidence separately from local class scores. Each source is fitted independently in the same frozen routing feature space and retains its own fVAE. This avoids allowing a larger source to dominate a pooled source-fitting objective and preserves earlier source representations without revisiting their inputs. Both deterministic AE and fVAE use reconstruction evidence; our fVAE learns input-dependent latent means and variances, trains reconstruction over sampled latent neighbourhoods, and averages reconstruction error over the posterior at inference. The common-prior KL term regularizes this representation. Gaussian-FR already models covariance, so the controlled comparison tests the full source-model design rather than attributing every gain to variance alone (Sec. 3.2 and Appendix D.9).
Table 3 fixes classification branches and uses the same frozen ResNet-18 routing features. On CR7, AE, Gaussian-FR, and source-fVAE achieve 76.87, 90.02, and 95.12 Task-ID accuracy and 73.45, 89.79, and 94.47 Last Macro-F1, respectively. Source-fVAE has the highest mean on both metrics across all six streams. The joint comparison of Task-ID accuracy and F1 directly examines the task-identification benefit independently of classifier capacity; its magnitude varies across streams.
Fixed-classifier router controls (Table 3). The first three numeric columns are Last Macro-F1; the last three are Last Task-ID Accuracy, all in percent, mean ± sample SD over seeds 42/43/44. Both metrics are evaluated at the final session. “Source-fVAE” corresponds to the task-fVAE router when each source has its own task branch.
Stream Sessions AE F1 Gaussian-FR F1 Source-fVAE F1 AE Task-ID Gaussian-FR Task-ID Source-fVAE Task-ID NEU 3 91.41±2.58 99.01±0.57 99.51±0.21 91.48±2.59 99.01±0.57 99.51±0.21 CR7 3 73.45±3.36 89.79±0.56 94.47±0.24 76.87±2.51 90.02±0.58 95.12±0.41 Fashion-MNIST 5 86.94±0.33 86.72±0.50 89.97±0.22 89.79±0.24 89.70±0.40 92.74±0.16 MNIST 5 94.79±0.14 95.19±0.48 97.39±0.09 95.40±0.07 95.75±0.42 98.03±0.12 EuroSAT 5 88.63±0.72 93.12±0.16 93.34±0.54 89.82±0.68 94.19±0.03 94.36±0.62 Steel, order 1 8 92.35±0.70 91.91±0.90 92.47±0.44 99.68±0.06 99.43±0.26 99.72±0.00
Response to Reviewer 3 — Ytf9 — Weakness 1 and Question 1 — Source-to-task organization and complete-method evidence (Part 3/3)
-
Persistent source models are separated from selectively shared classification branches. A source describes an input population, while a task branch supplies a label-prediction function; these need not correspond one-to-one. HiDMoA retains one fVAE per source while deciding whether an arriving source needs a new classifier. Sources with identical label sets may share a branch after passing the reconstruction-compatibility test. Their source models remain separate after sharing, and the reused branch is updated with current-source supervision and a pre-update teacher on current inputs. Expert Gate uses reconstruction-related evidence to select a parent and a transfer procedure for a new expert. It adds a new expert for every task and names “a mechanism to decide when to merge tasks with high relatedness degree rather than adding a new expert model” as future work (Aljundi et al., 2017, Sec. 5). In HiDMoA, the same evidence also determines classifier creation or reuse, with a shared frozen backbone and local adapter and head updates (Secs. 2–3 and Appendix G).
Table 4 isolates allocation on the six-source repeated-label steel stream, holding the classifier and router configuration fixed and retaining six source fVAEs for every policy. Adaptive allocation reaches 96.01±0.66 Last Macro-F1, compared with 95.22±1.35 for always-new and 95.31±0.21 for always-reuse, using 4/4/5 branches instead of six. Old-source and current-source accuracy are 97.92 and 95.24, compared with 97.52 and 93.54 for always-new. This supports selective sharing with fewer classifiers and higher mean classification performance. Scores are percentages, mean ± sample SD over seeds 42/43/44; old/current denote previously arriving and current sources.
Steel allocation control (Table 4):
Policy Last Macro-F1 Old-source Acc Current-source Acc Branches (42/43/44) Always new 95.22±1.35 97.52±0.53 93.54±2.57 6/6/6 Same-label always reuse 95.31±0.21 97.57±0.17 93.88±1.02 3/3/3 Adaptive, γ=1 96.01±0.66 97.92±0.30 95.24±1.56 4/4/5 -
As a complete method, HiDMoA exceeds Expert Gate by 10.1–10.8 Last Macro-F1 points. Complete-method comparisons evaluate the combined source-modelling and classifier-organization design. In Sec. 5.2, Table 1, HiDMoA and the Expert Gate adaptation use DeiT-Small classification backbones and frozen ResNet-18 routing features on the eight-source, 50-class industrial stream. HiDMoA exceeds Expert Gate by 10.1–10.8 Last Macro-F1 points across three source orders. Expert Gate retains independent full classifiers, whereas HiDMoA specializes a shared frozen backbone through adapters. These complete-method results complement the fixed-head and source-model controls in Parts 1 and 2 and the allocation-policy control above. Scores are percentages, mean ± standard deviation over three seeds.
Industrial stream — direct Expert Gate comparison (Table 1):
Method Order 1 Last F1 Order 2 Last F1 Order 3 Last F1 Expert Gate 83.0±1.32 82.6±2.58 82.1±1.19 HiDMoA 93.1±0.40 93.4±0.20 92.6±0.60
Response to Reviewer 3 — Ytf9 — Weakness 1 and Question 1 — Task-selection mechanism and evidence (Part 1/3)
Thank you for the suggestions on related work and more challenging evaluations. We summarize the revisions and new results in response to your comments below.
We build on the two-stage view of Kim et al. (2022) and do not claim it as new. The novel parts of HiDMoA are the following. First, the evidence for task selection: source models are learned separately from the class scores, and our analysis shows that the class scores of fixed local heads cannot in general identify the task, however they are calibrated. Second, the organization of classifiers: persistent source models are separated from task branches, so that compatible sources can share a classifier, a mechanism that Expert Gate names as future work. Third, the method is replay-free, whereas the closest task-identity methods, MORE and TPL, rely on stored samples.
Part 1 addresses task selection; Part 2 covers source-fVAE modelling and router controls; Part 3 covers classifier sharing and complete-method comparisons.
Relative to Kim et al. (2022), the new element is the evidence used for task selection. Independently learned local heads can classify their own inputs correctly while a larger score from an unrelated head wins the global decision. Without previous-source samples for joint score alignment, effective within-task classification alone leaves this selection problem unresolved. Within the framework of Kim et al.’s A Theoretical Study on Solving Continual Learning (2022), our contribution concerns how source evidence is learned and retained, and how it determines which classifier should process an input.
Kim et al. establish a probabilistic decomposition into within-task prediction and task-identity prediction and relate the corresponding cross-entropy losses to OOD detection. Their analysis treats task-identity prediction as given and links it to OOD detection; ours asks which evidence can support task selection once the local heads are fixed, and differs in three respects. First, our fixed-head analysis characterizes how task selection affects classification risk under disjoint labels (Proposition 2). Second, it identifies a limit of score-based selection: per-branch monotone recalibration selects a branch from the top scores of the local heads; when these scores leave task identity ambiguous, no recalibration can resolve the ambiguity (Appendix B.2), whereas separate source evidence can use information beyond these scores. Third, Appendix B.1 (Eq. 9) permits several branches to contain the same label and decomposes the classification error accordingly. This preserves the source-to-branch organization, whereas Remark 3 of Kim et al. assigns recurring classes to unique tasks.
The Appendix B.2 construction has perfect local heads but identical top scores for two equally likely sources, giving 1/2 error to selection based only on that statistic; a separating source representation permits zero error with the same heads. Proposition 2 and Sec. 4 isolate this information limitation from Proposition 1’s additional A5/A6 conditions. The existing Table 2 tests the decision rules with classification branches fixed: Last Macro-F1 is 75.65 with global maximum, 86.53 with fitted affine calibration, and 92.47 with source-fVAE routing. The 5.94-point gain over the fitted calibration locates a benefit in branch selection.
Fixed-classifier selection rules (Table 2). Scores are percentages, mean ± sample SD over seeds 42/43/44.
We have revised the first contribution in the Introduction to state these distinctions. In practice, MORE and TPL use a 600-sample replay memory in our benchmark (Table 20), whereas HiDMoA uses separately retained source models without replay and, with the same DeiT-Small classifier, exceeds TPL and MORE by 7.4 and 8.6 Last Macro-F1 points averaged over the three industrial orders (Table 1).