acceptodds
Under review as a conference paper at ICLR 2027

What Does New-Task Accuracy Measure in Continual Learning?

Abstract

Does lower new-task accuracy indicate a reduced ability to learn? In continual learning, the answer depends on the update procedure and prediction problem. We examine this inference by pairing actual learning with current-only adaptation from the same inherited parameters, then changing prediction candidates at fixed weights. Across ten paired CIFAR-10 seeds, experience replay trails SGD by 11.151 percentage points in actual new-task accuracy yet leads by 0.971 points after matched adaptation. Restricting actual predictions to current classes reverses the mean ranking without changing the probe scores. The apparent deficit thus coexists with better within-task discrimination and better matched adaptation. Retention gives the comparison its practical meaning: both standard probes have zero old-task accuracy, while a 20% retention requirement favors actual replay over the tested validation-selected probes on CIFAR-10.1/.2. Fixed-history continuations and a full-budget CIFAR-100 ResNet experiment show where the rankings change; in the latter, ER leads on both actual and probe accuracy in all five seeds. Together, these results distinguish an observed new-task deficit from weaker adaptation of inherited parameters and show why current and retained performance must be evaluated jointly.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, plasticity, experience replay, evaluation, stability-plasticity trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_PM1sOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether new task accuracy in continual learning reliably reflects the ability of inherited model parameters to adapt. It compares actual learning with matched adaptation probes and investigates how prediction candidates and knowledge retention affect the interpretation of new task performance.

Critical strengths

The experimental controls are rigorous. The primary comparisons utilize ten sets of paired random seeds, matched adaptation protocols, and pre-specified statistical tests. The conclusion regarding inconsistent rankings is strongly supported across the tested settings.

Critical weaknesses

1. I would like to emphasize that the paper's novelty and scientific significance appear limited. The distinction between adaptation ability and observed task accuracy has already been investigated in [1], while classifier bias and competition between old and new classes are well established in [2]. The proposed decomposition (A=H+C) is an algebraic identity and does not constitute a new theoretical result. The masking reversal in Table 2 is a useful controlled diagnostic, but its underlying mechanism is closely related to the established effect of providing task identity and removing cross-task competition. The main probe advantage is only 0.971 percentage points, with no demonstrated practical improvement over actual replay under a positive retention constraint. While the combined probe and masking analysis provides additional empirical evidence, it remains unclear what sufficiently novel or consequential insight it establishes beyond these known phenomena.

2. The core findings have limited demonstrated generalizability, making their practical relevance to more complex continual learning settings uncertain. The primary result relies on a compact CNN trained for four epochs per task on CIFAR-10. Its magnitude is sensitive to the replay objective. Normalized compensation reduces the mean actual accuracy deficit against SGD from 11.151 to 1.881 points (Appendix L, Table 24). At sixteen epochs of continuation from the original four-epoch histories, the four-task mean actual deficit falls to 2.618 points, while the probe advantage is 0.595 points (Appendix N, Table 30). Most importantly, the full-budget CIFAR-100 experiment favors ER on actual, probe, and masked accuracy in all five seeds (Section 7, Table 4). Although this experiment changes multiple factors simultaneously, it substantially limits the evidence that the proposed ranking disagreement generalizes to longer, more realistic training regimes. The additional architecture and method comparisons provide some supporting evidence, but do not establish consistent behavior across settings. The paper does not sufficiently isolate the conditions under which the disagreement arises or demonstrate its practical implications for method selection.

References

[1] Re-Evaluating Continual Learning with Few-Shot Adaptation

[2] Large Scale Incremental Learning

Questions for authors
  1. What concrete conclusion about plasticity or continual learning method selection does the combined probe and masking analysis establish beyond [1] and [2]? In particular, which conclusion specifically depends on combining these diagnostics and cannot be inferred from the established findings considered separately?
  2. Can the authors further isolate the respective contributions of replay loss weighting and classifier-head training or competition to the observed ranking disagreement? The substantial reduction under normalized compensation and the opposing full-budget CIFAR-100 results raise concerns about the conditions under which the disagreement persists.
Reviewer_DgJaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Authors ask whether lower new-task accuracy in class-incremental learning means a reduced ability to learn. At each task boundary, it compares three branches that share the same current-task minibatches:

  • actual: the learner’s own algorithm,
  • probe: current-only SGD starting from the learner’s inherited parameters,
  • fresh: current-only SGD starting from the original initialization.

These give an accounting identity A = H + C: the gap to the fresh reference (A) splits into a gap due to the inherited parameters (H) and a gap due to the update procedure (C).

Critical strengths

Solid protocol. The follow-up was frozen before running, uses new seeds, pairs seeds explicitly, prespecifies a single primary contrast with Bonferroni-adjusted bootstrap intervals, and reports every seed. Failures and test violations are kept rather than filtered out.

Thorough reproducibility. It includes hashed manifests, integer-count evidence, independent reimplementations that reproduce outputs bit-for-bit, scripts that recompute the statistics with only NumPy, and training replays. This is well above the norm.

Clean diagnostic. The three-branch setup and test-time candidate masking separate the update rule, the inherited state, and competition between old- and new-class outputs. The identity A = H + C is a simple, reusable tool.

Critical weaknesses

The main insight is largely already known, and key related work is missing.

  • That replay lowers new-task accuracy in class-IL because of competition between old and new class outputs, and that task masking removes this, is well established. Missing references include: (i) classifier-bias corrections: WA (Zhao et al., 2020), SS-IL (Ahn et al., 2021), ER-ACE (Caccia et al., 2022). The paper cites only BiC. (ii) intransigence: Chaudhry et al. (2018) defined intransigence as the gap to a reference model, which is conceptually very close to A and H.

  • That SGD’s high new-task accuracy in Class-IL simply reflects predicting only current classes after forgetting everything is also standard. The framework adds a careful measurement of known effects rather than a new understanding.

Small probe effect.

The probe advantage (ΔH = −0.97 pp) is small. It weakens with longer training (9 → 7 → 8 seeds positive at 4, 8 and 16 epochs). And it is fragile beyond the main setup. It splits 5/5 at the final task, and is negative in only 3/5 seeds for the ResNet, whose interval includes zero. It disappears in the properly trained CIFAR-100 setting. The primary configuration (tiny CNN, no augmentation, plain SGD with lr 0.01, 4 epochs, fresh accuracy around 77% on 2-way tasks) is also looking clearly undertrained. The result may describe an under-optimized regime rather than continual learning in general.

The plasticity framing is not supported.

The introduction motivates the work with plasticity loss (Dohare et al.), but 5-task streams with 4 epochs each cannot show plasticity loss, which typically appears over long streams. The CBP control replaces only 8 units and is effectively SGD here, so it says little about plasticity interventions.

Presentation seriously hurts readability.

I found it hard to follow and felt it reads like an audit log. The main text is packed with protocol details: SHA prefixes, wall-clock times, watchdog limits, RNG seed formulas, run labels, and notes about rounding artifacts from an earlier analysis version. It also has an overwhelming size. At 51 pages with 38 tables, the core message is buried. Much of the main-text material belongs in the appendix or a reproducibility report.

Chaudhry et al., 2018: A. Chaudhry, P. K. Dokania, T. Ajanthan, P. H. S. Torr. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. ECCV 2018.

WA (Zhao et al., 2020): B. Zhao, X. Xiao, G. Gan, B. Zhang, S.-T. Xia. Maintaining Discrimination and Fairness in Class Incremental Learning. CVPR 2020.

SS-IL (Ahn et al., 2021): H. Ahn, J. Kwak, S. Lim, H. Bang, H. Kim, T. Moon. SS-IL: Separated Softmax for Incremental Learning. ICCV 2021.

ER-ACE (Caccia et al., 2022): L. Caccia, R. Aljundi, N. Asadi, T. Tuytelaars, J. Pineau, E. Belilovsky. New Insights on Reducing Abrupt Representation Change in Online Continual Learning. ICLR 2022.

Questions for authors

Please see the weaknesses.

Reviewer_3neSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper raises an important question in continual learning, "Does lower new-task accuracy indicate a reduced ability to learn?". The paper sets up an empirical study to answer this question using CIFAR 10/100 datasets and ResNet-32 with experience replay (ER) and in some experiments DER++. The paper defines metrics Fresh F, Probe p, and Actual a which measure the follow-up accuracy relative to baseline SGD.

Critical strengths
  • The research question this paper raises "Does lower new-task accuracy indicate a reduced ability to learn?" is interesting and important to how we define stability/plasticity trade-offs in continual learning.
Critical weaknesses
  • The paper requires a substantial rewrite. It was difficult to follow in its current form.
  • For an empirical study, the experimental setup is limited using only CIFAR dataset and ResNet models and two continual learning techniques. It is unclear if the stated observations generalise to more challenging datasets and models and continual learning techniques.
  • No clear takeaway from the experiments or paper.
Questions for authors

How do the trends and observations generalise to different datasets/models/techniques?

Reviewer_JijcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether lower new-task accuracy in class-incremental learning necessarily indicates weaker learning or adaptation ability. The authors compare actual continual-learning updates with current-task-only adaptation probes initialized from historical checkpoints, as well as fresh-model references. They further investigate how prediction candidate sets, classifier calibration, retention constraints, and training budgets affect the comparison.The paper aims to demonstrate that new-task accuracy alone is insufficient to assess adaptation in continual learning and that current-task performance should be considered jointly with retained performance.

Critical strengths

The paper examines a issue in continual-learning evaluation: whether observed new-task accuracy can reliably reflect a model's ability to adapt to new data.

Critical weaknesses
  • The paper does not reveal any novel phenomena or draw meaningful new conclusions. The presented results and analyses read more like an experimental report than a research paper, and the work lacks contributions that would be of significant interest to researchers in the field.
  • The paper is poorly organized and lacks clear definitions and a coherent structure.
Questions for authors

No question to ask.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.