- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether new task accuracy in continual learning reliably reflects the ability of inherited model parameters to adapt. It compares actual learning with matched adaptation probes and investigates how prediction candidates and knowledge retention affect the interpretation of new task performance.
Critical strengths
The experimental controls are rigorous. The primary comparisons utilize ten sets of paired random seeds, matched adaptation protocols, and pre-specified statistical tests. The conclusion regarding inconsistent rankings is strongly supported across the tested settings.
Critical weaknesses
1. I would like to emphasize that the paper's novelty and scientific significance appear limited. The distinction between adaptation ability and observed task accuracy has already been investigated in [1], while classifier bias and competition between old and new classes are well established in [2]. The proposed decomposition (A=H+C) is an algebraic identity and does not constitute a new theoretical result. The masking reversal in Table 2 is a useful controlled diagnostic, but its underlying mechanism is closely related to the established effect of providing task identity and removing cross-task competition. The main probe advantage is only 0.971 percentage points, with no demonstrated practical improvement over actual replay under a positive retention constraint. While the combined probe and masking analysis provides additional empirical evidence, it remains unclear what sufficiently novel or consequential insight it establishes beyond these known phenomena.
2. The core findings have limited demonstrated generalizability, making their practical relevance to more complex continual learning settings uncertain. The primary result relies on a compact CNN trained for four epochs per task on CIFAR-10. Its magnitude is sensitive to the replay objective. Normalized compensation reduces the mean actual accuracy deficit against SGD from 11.151 to 1.881 points (Appendix L, Table 24). At sixteen epochs of continuation from the original four-epoch histories, the four-task mean actual deficit falls to 2.618 points, while the probe advantage is 0.595 points (Appendix N, Table 30). Most importantly, the full-budget CIFAR-100 experiment favors ER on actual, probe, and masked accuracy in all five seeds (Section 7, Table 4). Although this experiment changes multiple factors simultaneously, it substantially limits the evidence that the proposed ranking disagreement generalizes to longer, more realistic training regimes. The additional architecture and method comparisons provide some supporting evidence, but do not establish consistent behavior across settings. The paper does not sufficiently isolate the conditions under which the disagreement arises or demonstrate its practical implications for method selection.
References
[1] Re-Evaluating Continual Learning with Few-Shot Adaptation
[2] Large Scale Incremental Learning
Questions for authors
- What concrete conclusion about plasticity or continual learning method selection does the combined probe and masking analysis establish beyond [1] and [2]? In particular, which conclusion specifically depends on combining these diagnostics and cannot be inferred from the established findings considered separately?
- Can the authors further isolate the respective contributions of replay loss weighting and classifier-head training or competition to the observed ranking disagreement? The substantial reduction under normalized compensation and the opposing full-budget CIFAR-100 results raise concerns about the conditions under which the disagreement persists.