- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends the sufficient-statistics perspective of analytic continual learning to networks satisfying an input-anchoring property. It shows that their MSE loss and gradients can be determined by fixed moment tensors, enabling equivalence to matched full-replay training under specified conditions. The authors introduce an Anchored MLP (AMLP) and propose Loss Anchoring Points (LAP), which optimize synthetic features and labels to approximate these otherwise prohibitively large statistics. Experiments evaluate AMLP heads on frozen pretrained features across standard class-incremental benchmarks, with an additional Places365 experiment assessing LAP under a constrained storage budget.
Critical strengths
-
The connection between network structure and sufficient statistics is conceptually useful. The input-anchoring formulation explains how certain multilayer trainable architectures admit a representation of their entire MSE objective through fixed data statistics.
-
The experiments include useful controls and a relevant storage comparison. The reproduced VILA baseline, increased random-feature width, and gradient-trained linear head help examine the source of AMLP’s improvements. The paper also recognizes that full feature replay can require less storage than a high-dimensional analytic cache on standard benchmarks.
Critical weaknesses
-
The practical significance of the multilayer head needs clearer justification. In the experiments, the main architectural change is from an analytic linear classifier on fixed features to a trainable input-anchored multilayer head. The backbone remains frozen after the initial task. This extension is theoretically interesting and produces accuracy improvements. However, it remains unclear when its additional adaptability is sufficiently valuable to justify the extra parameters, iterative optimization, and memory-compression procedure. The advantage may depend on the quality of the upstream representation and its alignment with downstream tasks. Controlled comparisons across different feature conditions, with resource usage accounted for, would help establish when the multilayer structure offers a substantial practical advantage over a simpler analytic classifier.
-
LAP’s novelty relative to dataset condensation and distillation requires further clarification. LAP optimizes synthetic features and labels to match statistics of the original data, connecting directly to dataset condensation and distillation. The authors should clarify LAP’s distinctive contribution in its matching objective, theoretical guarantees, or optimization procedure, and explain what advantages specifically follow from input anchoring. Since resource constraints are central to continual learning, this distinction may also be evaluated in terms of synthesis cost, subsequent training cost, and persistent storage, rather than predictive accuracy alone.
-
The quantitative connection between exact-statistics theory and LAP under a finite budget is insufficiently established. Theorem 1 provides exact equivalence when the complete statistics are available. Theorem 2 assumes the existence of synthetic data that exactly reproduce those statistics and the attainment of corresponding global minimizers. These conditions do not establish the approximation quality achievable with the budgets used in practice. The paper does not sufficiently connect the number of synthetic points, the achieved moment-matching error, and deviations in loss, gradients, or subsequent learning performance. Reporting achieved matching errors and examining their relationship to downstream behavior would strengthen the connection between the theoretical motivation and the implemented method.
-
The experiments do not yet adequately establish LAP’s practical value across resource budgets. The main experiments on eight standard benchmarks use full historical feature replay. This choice has a reasonable storage motivation, but these results primarily validate the AMLP head rather than LAP compression. The substantive compression evaluation is concentrated on Places365 at one storage budget, with uniform subsampling as the main compression control. Evaluation across several budgets, especially tighter constraints, and comparisons with representative condensation or stronger subset-selection methods would better establish the method’s usefulness. The paper should also specify which persistent states are included in the budget and report synthesis time and subsequent training overhead.
Questions for authors
See Weaknesses.
4 comments
On Time, Memory, and Practicality
We focus the paper on the tradeoff between memory and accuracy, treating CL as a game of who can retain the most accuracy while requiring the least storage. As a result, we never tuned LAP’s fit steps, setting it at a sufficiently large number (48k) such that we weren’t worried about fit quality as a confound and caching the result. While we acknowledge that time/compute efficiency are important, we intentionally leave this as an open question for future work in settings which have more specific restrictions (e.g. robotics settings where models must update in real time).
In weakness 1, the reviewer asks when the “additional adaptability is sufficiently valuable to justify the extra parameters, iterative optimization, and memory-compression procedure”. To our understanding of the continual learning objective, the latter two costs, as measured in computational resources and time, are justifiable in the pursuit of higher accuracy at matched storage as they push forward the frontier. We are not alone in accepting this tradeoff. For example, ConLoRACUO, as mentioned by Reviewer K2Vy, makes a similar trade off when it trains LoRA adapters iteratively for each task.
This brings us to parameter counting. In the paper, we don’t count model parameters when matching storage, opting instead to match the width and depth of the heads. We describe this both in the first paragraph of section 4.1 and in the “Honest Storage Accounting” subsection where we provide explicit formulae for how we get our numbers. There are three reasons why we do this:
-
To our understanding, there is no pre-established standard in the subfield. Prior ACIL papers tend to count the state package as essentially free (see our honest storage accounting subsection), so we are forced to create our own standard.
-
Because of differences in class counts, matching parameters would lead to slight differences in model shape for each new benchmark. Let be the output shape of VILA’s backbone (1280), be the number of random features (16384), and be the number of classes. Then, an AMLP head has parameters while the VILA analytic head has . Therefore, across different datasets, true parameter matching would require adjusting the number of random features in the VILA head for each benchmark.
-
Not all parameters are created equally. Both the analytic head and the AMLP head rely on a set of frozen random features. It’s unclear how these features should be treated in terms of storage accounting. In a restrictive storage regime, an implementer could simply store a random seed and re-generate these parameters on demand without needing more than a few bytes of permanent storage.
Because of this complexity, we decided the cleanest comparison was a simple head-to-head: our 16384 features vs their 16384 features. The accompanying state difference is small: . For converting numbers to GB we multiply by a factor of to represent the number in fp32.
For our paper’s first experiment, the difference is null, as VILA’s 1.07GB statistics cache is larger than the largest dataset and the parameter difference combined. Not only that, but we explicitly compare to a VILA model with double the width to remove any doubt about the strength of our method.
For our paper’s second experiment, the difference is minor, if we count any parameters beyond VILA’s default model with head width 16384 in our state package, then the AMLP has a total storage of: GB and and a VILA analytic head with adjusted width has total state at least: . For , this is GB. We say ‘at least’ and use 365 in the formula because places 365 has 365 classes, so is at least 365 in our paper’s second experiment (iNaturalist and ImageNet-1k each have more classes).
Ultimately, we are confident (due to Experiment 1’s results) that the 3.6% increase in feature count will not change the ordering of our results, and we think that the existing matched feature count comparison is cleaner for the paper. However, we’re happy to run the tests and report them in this thread along with the rest of the paper’s updated Experiment 2.
In this case, the added parameters the reviewer identified are accounted for and justified directly in the storage budget. Our method is therefore practical under the lens of CL because it achieves a higher accuracy with matched storage.
On Dataset Distillation
We are excited that you brought up dataset distillation! We are currently planning a follow-up paper which directly explores a cross-subfield comparison between ACIL methods and Dataset Distillation methods for continual learning (expanding upon our Honest Storage Accounting argument from the paper).
That said, while we think the comparison to dataset distillation is both interesting and valuable, we think it belongs in a separate future work for the following reasons:
-
Novelty; LAP, while dataset shaped, is a novel approximation method in its own right. Dataset distillation primarily relies upon heuristics, often through bi-level optimization, such as matching performance, parameters, or gradient trajectories. Even our perhaps closest neighbor in the space, Kernel Inducing Points (KIP), which also uses a closed form single-level optimization approach, constructs its dataset such that kernel ridge regression achieves strong performance on the orginal dataset, and uses this as a proxy for the target neural network. LAP, in contrast, has an exact objective of the Frobenius distance between the LAP set’s sufficient statistics and the true dataset’s sufficient statistics. In the ideal case, typical dataset distillation methods reproduce the result of the initial dataset. In our ideal case, (when J=0), the LAP set is the dataset for the purposes of generating MSE gradients. The distinction is critical precisely in the continual learning setting because LAP admits the stacking of its approximations (Lemma 2 of our paper). Therefore, our method is far more than a simple application of an existing dataset distillation method to an ACIL model, but rather a natural consequence of the core question of the second half of our paper: “Can we effectively approximate the sufficient statistics in polynomial time?”.
-
Space; From a structural perspective, we view the purpose of this paper as establishing the input-anchoring theory and demonstrating its empirical value in the ACIL setting. With a tight 10-page limit, we see no way to give the cross-subfield comparison the full attention it deserves, and we don’t see it as a necessary inclusion to accomplish the goals of the paper (those being theoretical rigor and empirical feasibility in our subfield, as opposed to empirical dominance across CL).
With that said, we don’t intend to omit dataset distillation from the paper, and are happy to explicitly name the work and suggest this future comparison in both our literature review and in our future work section. Would this fairly address the concern in your eyes?
I am very grateful to the authors for their detailed comments. After carefully reviewing them, I believe that most of my concerns have been addressed to some extent. Therefore, I have decided to raise my rating. Good luck.
Summary
We note the following trends among the reviewer’s highlighted weaknesses:
A request for a demonstration of practical value of the method through further experimentation across different budgets and settings, including: “different feature conditions” (Weakness 1), different matching errors (Weakness 3), different synthetic point budgets (Weakness 3), and different datasets (Weakness 4)
A request for clarification on resource usage including: parameter counts (Weakness 1 and 4), and fitting/training time (Weakness 4)
A request for clarification of novelty of LAP relative to dataset distillation, and an empirical comparison (Weakness 2 and 4)
If we are missing something, please don’t hesitate to let us know.
We hope that for the most part, our proposed experimental slate addresses the requests for further experimentation. We are unsure what “different feature conditions” the reviewer had in mind, but we offer that changes in the underlying backbone manifest in changes to the data distribution which LAP compresses, so our expansion of our paper’s Experiment 2 to covering three datasets already partially covers this axis. If there are further changes the reviewer thinks are critical to see, we would be happy to continue the discussion.
To address the request for experiments at different matching errors and the request for clarification on fitting/training time, we propose a follow up experiment (if time allows) which ablates the number of fit steps for algorithm 1 on Places 365. This will show the relationship between moment matching error, accuracy, and time.
We have two follow up comments which we will post shortly, one tackling the practicality concern, time, and parameter efficiency, and one tackling the comparison to dataset distillation.