acceptodds
Withdrawn from ICLR 2027

Stability Where It Matters: Decision-Sensitive Representation in Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Continual learning with pre-trained models commonly mitigates forgetting by constraining parameters, gradients, or entire feature representations. While effective, these strategies do not explicitly distinguish representation changes that alter historical decisions from changes that are functionally irrelevant to them. We introduce Decision-Sensitive Drift Regularization (DSDR), a training-time principle that uses the decision geometry of previously learned classifiers to separate temporal representation drift into decision-sensitive and decision-null components. DSDR selectively suppresses only the former, directly linking representation stability to preservation of historical predictions while leaving complementary directions available for plasticity. The method requires no additional inference-time module and can be integrated into parameter-efficient continual adaptation without altering the inference procedure. Experiments under strict class-incremental learning on ImageNet-R across 5, 10, and 20-task settings, together with CIFAR-100 and DomainNet, demonstrate robust performance across task horizons with reduced forgetting and competitive final accuracy. These results support decision-sensitive drift as a simple and general principle for balancing stability and plasticity in continual learning. Code will be made available upon final acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6YGWOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DSDR to alleviate the stability–plasticity conflict in class-incremental learning. Its core idea is to exploit the feature-space geometry induced by the historical classifier to decompose temporal representation drift into decision-sensitive and decision-null components, and to regularize only the former, thereby reducing changes that may affect historical decisions while preserving greater representational flexibility for learning new knowledge.

Critical strengths
  1. The research question is clearly motivated, shifting the focus from the magnitude of representation drift to identifying which drift components actually affect historical classifiers. This perspective provides an insightful way to revisit stability in continual learning.
  2. The proposed method is conceptually simple and directly exploits the geometry induced by the historical classifier to decompose temporal representation drift, without introducing additional trainable modules.
  3. DSDR selectively regularizes classifier-sensitive drift while leaving complementary directions relatively unconstrained, providing a clear connection to the stability–plasticity trade-off in continual learning.
  4. As a training-time regularizer, DSDR can be readily integrated into existing PEFT-based continual learners without introducing additional inference-time parameters or computational overhead.
Critical weaknesses
  1. DSDR is largely built upon the established geometric properties of the row space and null space of the historical classifier. Its novelty therefore lies mainly in applying this geometry to representation drift regularization in CIL, while the core methodological contribution remains relatively limited.
  2. The definitions of “decision-sensitive” and “decision-null” are not sufficiently rigorous. The current derivation only guarantees that certain drift components do not change the historical logits, but this does not ensure that the final prediction over all observed classes remains unchanged under strict CIL. In particular, such drift may still affect new-class logits and old–new class competition.
Questions for authors
  1. DSDR estimates and constrains representation drift using only current-task samples, while its objective is to preserve historical-task decisions. How can drift measured on the current-task distribution reliably reflect and control representation changes on previous-task distributions?
  2. The random-subspace baseline already yields a substantial performance gain, while DSDR provides only a relatively small additional improvement. Moreover, this key ablation is conducted using a single seed. How robust is the advantage of the classifier-induced geometry across multiple random subspaces and random seeds?
  3. The correlation between decision-sensitive drift and prediction flips is relatively weak. What additional mechanistic evidence supports the claimed relationship between this drift component and changes in historical predictions?
  4. The weaker performance on CIFAR-100 is attributed to low image resolution, which is argued to produce more compressed representations and less informative classifier geometry. What direct evidence, such as feature rank or projected drift energy, supports this explanation?
  5. The performance advantage of DSDR over SplitLoRA decreases as the task sequence becomes longer, while the complementarity of SplitLoRA + DSDR is currently evaluated only on the 5-task ImageNet-R setting. How robust and generalizable is this complementarity across longer task sequences and additional datasets?
  6. Figure 1 uses different input images for the current and frozen models, whereas the temporal drift should be computed from the same input processed by both models. In addition, the figure does not clearly indicate that only P_D Δz is regularized. Why does the current figure differ from the formulation described in the method, and how should it be revised to accurately reflect the proposed procedure?
Reviewer_mXjGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Decision-Sensitive Drift Regularization (DSDR), which uses the historical classifier's row space to selectively constrain representation changes during class-incremental learning. The method is evaluated on ImageNet-R, CIFAR-100, and DomainNet, with ablations examining subspace selection, regularization strength, prediction changes, and compatibility with SplitLoRA.

Critical strengths

DSDR introduces no additional inference-time parameters or computation, as confirmed by Table 6. This makes it practical to integrate into existing parameter-efficient continual learning frameworks. The improvements on five-task ImageNet-R and DomainNet in Tables 1 and 2 suggest that selective representation regularization can be useful. The comparisons against full-drift and random-subspace constraints also provide preliminary evidence for the proposed design.

Critical weaknesses
  1. The manuscript does not comply with the required submission formatting, particularly regarding page margins and text width. This raises serious concerns about whether the main text would remain within the nine-page limit under the official template.

  2. The central theoretical claim is not fully established, and the methodological novelty requires clearer justification. Section 4.2, Eqs. (4)–(9), correctly establishes that drift in the null space of the historical classifier leaves its logits unchanged. However, this does not guarantee preservation of historical predictions under strict CIL. Prediction stability depends on classification margins, and newly introduced classes compete with historical classes under the global classifier. Historical predictions can therefore change even when all historical logits remain unchanged. Although the proposed loss constrains old-logit-sensitive drift, it does not explicitly address old-versus-new class margins or establish guarantees on prediction preservation. Furthermore, classifier row-space geometry has already been studied in [1], while subspace-based feature distillation has been explored in [2]. Although DSDR differs in its subspace construction and learning setting, the paper does not sufficiently establish the specific contribution of its classifier-derived regularization beyond these existing ideas.

  3. The experimental evidence is insufficient to fully support the claimed effectiveness and underlying mechanism. In Table 1, DSDR improves 20-task ImageNet-R FAA over SplitLoRA by only 0.06 percentage points (78.88 ± 0.57 versus 78.82 ± 0.28), a difference small relative to the reported variability. In Table 2, it underperforms SplitLoRA by 1.49 points on CIFAR-100. More importantly, the core ablation in Table 3 shows only a 0.75-point advantage over a rank-matched random subspace, based on a single seed and without uncertainty estimates or repeated random-subspace comparisons. Table 5 reports modest correlations of 0.132 and 0.090 between the two drift components and prediction flips in the unregularized setting, without uncertainty estimates or analysis of classification margins. While these findings provide preliminary support, they do not convincingly establish that the reported improvements are specifically attributable to classifier-derived subspace selection beyond the benefits of generic subspace regularization.

References

[1] The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation

[2] Subspace distillation for continual learning

Questions for authors
  1. How does the proposed objective protect historical predictions against newly introduced classes when it only constrains representation drift visible to the historical classifier? Can the authors establish a more explicit connection between the regularized quantity, classification margins (particularly old-versus-new class margins), and actual prediction flips?

  2. Can the authors provide repeated-seed comparisons against rank-matched random-subspace regularization and conventional old-logit distillation, with comparable hyperparameter tuning? For the random-subspace baseline, evaluating multiple independently sampled subspaces would help determine whether the classifier-derived geometry provides a reliable advantage. Please also clarify why SplitLoRA's five-task ImageNet-R FAA differs between Table 1 (81.92 ± 0.29) and Table 7 (80.87), and whether this reflects differences in experimental settings or implementation.

Reviewer_VmUTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the stability–plasticity trade-off in class-incremental learning with pre-trained models, arguing that constraining all representation changes can hinder new learning and that it is therefore necessary to identify which changes affect predictions on previously learned tasks. The authors propose Decision-Sensitive Drift Regularization (DSDR), which uses a frozen copy of the previous-task model as a reference and decomposes feature drift into components in the row and null spaces of the historical classifier, penalizing only the component that can alter old-class logits while leaving the remaining directions available for new learning, without adding parameters or computation at inference.

Critical strengths

The method is well motivated and easy to implement. It uses the historical classifier to identify which feature changes should be constrained, and the resulting regularization term can be added to existing training objectives without changing inference. The paper is well organized and clearly written. The derivation connects the classifier-based decomposition of feature drift to the training objective, making the method and its rationale easy to follow.

Critical weaknesses

The experiments are conducted on three benchmark datasets, but all use the same pre-trained ViT with LoRA. The method needs to be tested with more classes, longer task sequences, and different pre-training settings. Since DSDR leaves the historical classifier’s null space available for adaptation, an important question is whether its benefits persist as the classifier’s rank grows and that space shrinks. The current experiments do not adequately address this.

The gains over SplitLoRA are inconsistent. DSDR performs worse on CIFAR-100 and offers little improvement on longer ImageNet-R sequences. More runs with different seeds and task orders are needed to determine whether the gains are reliable and clarify when the method is useful. The authors attribute the weaker CIFAR-100 results to representation compression caused by low image resolution, but provide no controlled experiments or representation analysis to support this explanation.

DSDR can be viewed as feature distillation in a subspace defined by the historical classifier, relaxing the constraints imposed by full-feature matching. Its relationship to existing distillation objectives needs closer examination. Matching old logits through the same frozen classifier also leaves its null space unconstrained, so the paper should explain what DSDR’s projection and weighting contribute beyond output distillation. The full-feature and random-subspace ablations provide some evidence, but using the same regularization coefficient does not ensure comparable constraint strengths, and these tests cover only one task setting and one seed. Direct comparisons with output distillation, with regularization strengths tuned separately for each method, would make the assessment more convincing. Separate results for new-task learning and old-task retention are also needed to verify the claimed plasticity advantage.

Questions for authors

please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.