- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes NullAudit, a replay-free continual learning method that uses a Jacobian sketch to protect previous-task outputs. The method selects the protected rank using a residual-based rule and relaxes protection through a gradient-retention gate. Results on Split CIFAR-100 suggest an improved accuracy–forgetting–memory trade-off over the compared methods.
Critical strengths
-
The paper addresses a relevant problem: balancing stability and plasticity under limited memory without retaining previous-task examples.
-
The authors explicitly acknowledge the limitations of the first-order analysis and distinguish the reported empirical improvements from stronger preservation guarantees.
Critical weaknesses
-
The core method is insufficiently specified. The “pre-specified residual rule” is not explicitly defined. It is unclear how the residual is estimated, how the Jacobian is constructed and sampled, and how the gate threshold and rank adjustments are determined. These details constitute the proposed algorithm itself, and their omission makes the method difficult to understand and reproduce.
-
The availability of the proposed control signal is unclear. Section 3 defines (J) as the old-output Jacobian evaluated at the current parameters, “before discarding data.” However, after previous-task data are discarded, it is unclear how the learner estimates the residual of the current (J), or accounts for the mismatch between the historical sketch and the current Jacobian as parameters change. This connection is essential to justify the residual-based controller in the replay-free setting.
-
The distinction from existing work is not sufficiently established. Prior methods already use SVD and energy thresholds to determine the size of protected subspaces. Selecting protection dimensions using spectral information therefore does not, by itself, establish novelty. The authors should clarify the specific algorithmic advance introduced by their residual rule and plasticity gate, and substantiate its added value.
-
The experimental evidence is limited and insufficiently documented. Evaluation is restricted to Split CIFAR-100, while essential details—including the architecture, task protocol, hyperparameters, and statistical variability—are missing. The reported ablations provide only partial metrics, making it difficult to assess the joint effect on accuracy, forgetting, and memory. Controlled comparisons with tuned fixed-rank alternatives, together with analyses of rank selection and gradient retention, are needed to substantiate the proposed mechanism.
Questions for authors
Please refer to the weaknesses above.