acceptodds
Withdrawn from ICLR 2027

Residual-Guided Adaptive Function-Space Projection for Replay-Free Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Replay-free continual learning projects new-task gradients away from stored old-function directions, but a fixed-memory sketch must allocate protection without suppressing the signal needed for the new task. We develop residual-guided adaptive function-space projection: a first-order bound motivates selecting protected rank from the streaming Jacobian residual, while a plasticity gate relaxes protection when too much new-task gradient would be removed. On Split CIFAR-100, NULLAUDIT reaches 71.8% average accuracy, 8.9 points of forgetting, and 75.2% new-task accuracy using 9.6 MB; the nearest sketched-Jacobian baseline reaches 71.0%, 9.8, and 74.1% using 12.4 MB. Fixed-rank and no-gate ablations distinguish adaptive rank selection and plasticity control from raw memory allocation alone. The reported evidence establishes an improved memory–stability–plasticity trade-off for the evaluated task sequence without treating the local residual bound as an empirically calibrated drift predictor or a lifelong preservation guarantee.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, lifelong learning, replay-free learning, catastrophic forgetting, gradient projection, function-space projection, stability-plasticity trade-off, adaptive rank selection

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_C6LjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes NullAudit, a replay-free continual learning method that uses a Jacobian sketch to protect previous-task outputs. The method selects the protected rank using a residual-based rule and relaxes protection through a gradient-retention gate. Results on Split CIFAR-100 suggest an improved accuracy–forgetting–memory trade-off over the compared methods.

Critical strengths
  1. The paper addresses a relevant problem: balancing stability and plasticity under limited memory without retaining previous-task examples.

  2. The authors explicitly acknowledge the limitations of the first-order analysis and distinguish the reported empirical improvements from stronger preservation guarantees.

Critical weaknesses
  1. The core method is insufficiently specified. The “pre-specified residual rule” is not explicitly defined. It is unclear how the residual is estimated, how the Jacobian is constructed and sampled, and how the gate threshold and rank adjustments are determined. These details constitute the proposed algorithm itself, and their omission makes the method difficult to understand and reproduce.

  2. The availability of the proposed control signal is unclear. Section 3 defines (J) as the old-output Jacobian evaluated at the current parameters, “before discarding data.” However, after previous-task data are discarded, it is unclear how the learner estimates the residual of the current (J), or accounts for the mismatch between the historical sketch and the current Jacobian as parameters change. This connection is essential to justify the residual-based controller in the replay-free setting.

  3. The distinction from existing work is not sufficiently established. Prior methods already use SVD and energy thresholds to determine the size of protected subspaces. Selecting protection dimensions using spectral information therefore does not, by itself, establish novelty. The authors should clarify the specific algorithmic advance introduced by their residual rule and plasticity gate, and substantiate its added value.

  4. The experimental evidence is limited and insufficiently documented. Evaluation is restricted to Split CIFAR-100, while essential details—including the architecture, task protocol, hyperparameters, and statistical variability—are missing. The reported ablations provide only partial metrics, making it difficult to assess the joint effect on accuracy, forgetting, and memory. Controlled comparisons with tuned fixed-rank alternatives, together with analyses of rank selection and gradient retention, are needed to substantiate the proposed mechanism.

Questions for authors

Please refer to the weaknesses above.

Reviewer_7ACGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a replay-free continual-learning controller that adjusts how many directions in a stored Jacobian sketch are protected. A residual signal guides rank selection, while a plasticity gate relaxes projection when it removes too much of the new-task gradient. The approach is motivated by a first-order bound on changes to old outputs. Experiments on Split CIFAR-100 compare accuracy, forgetting, and memory use with existing methods and include fixed-rank and no-gate ablations.

Critical strengths

The paper focuses on a real limitation of projection methods: protecting old tasks can remove useful directions for learning new ones. Considering both the sketch residual and the retained gradient is a plausible approach to balancing preservation and plasticity under a memory budget.

Critical weaknesses
  1. Section 4 refers to “a pre-specified residual rule” without giving the rule, leaving me unable to reconstruct how the method actually chooses and adjusts its rank. Please write out one complete update, including how the residual is obtained after old examples are discarded, how it selects the rank, and when the gate changes that choice.

  2. With accuracy going from 71.0% to 71.8% in Table 2 and the fixed-rank and no-gate ablations reporting only parts of the metric set, I can't tell whether residual guidance improves the overall trade-off. Putting the adaptive method and both controls in one comparison at the same memory budget, with accuracy, forgetting, new-task accuracy and variation across runs reported for each, would make that claim much easier to assess.

Questions for authors

Can you give pseudocode for the residual estimator, rank rule and gate, including what is stored after each task?

Does the adaptive controller beat a tuned fixed-rank baseline under the same memory budget across repeated runs and all reported metrics?

Reviewer_1v6COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript reads like an unfinished LLM-generated draft and is not ready for substantive peer review.

Critical strengths

I could not identify any clear strengths in the current submission.

Critical weaknesses

The submission is an LLM-generated draft, with insufficient methodological detail and experimental support to warrant further review in its current form.

Questions for authors

no

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.