acceptodds
Under review as a conference paper at ICLR 2027

Input Anchoring Makes Your Model a Deep Analytic Continual Learner

Abstract

In class-incremental learning, models attempt to learn from a stream of tasks while having limited storage capacity. The strongest guarantee a continual learner can make is `absolute memorization' which, beyond simply avoiding catastrophic forgetting, promises to produce a model identical to its full replay counterpart without needing the original dataset. Recent work in Analytic Class Incremental Learning (ACIL) has demonstrated that linear layers with input dimension can be made into absolute memorizers with storage complexity . A variety of approaches leverage this by applying a wide linear layer atop large frozen pre-trained models. While ACIL methods achieve absolute memorization in linear regression models, no work has yet achieved absolute memorization in deep neural networks. In this paper, we define Input Anchoring and prove that degree- deep neural networks with this property achieve absolute memorization with storage complexity . We demonstrate that -nets and GaLU networks have the input-anchoring property with degree scaling linearly in their depth, and introduce our own simple and performant input-anchored architecture: the Anchored Multi-Layer Perceptron (AMLP). We show that the benefits of absolute memorization can be achieved with polynomial storage and time complexity via our Loss-Anchoring Points (LAP) approximation method, and leverage this finding to achieve state-of-the-art performance over the latest ACIL methods at matched storage and parameter counts on pre-trained model class-incremental benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class-incremental learning, lifelong learning, analytic continual learning, absolute memorization, pre-trained models, polynomial neural networks, gated linear units, sufficient statistics, tensor decomposition, coresets, dataset distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YX1ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the sufficient-statistics perspective of analytic continual learning to networks satisfying an input-anchoring property. It shows that their MSE loss and gradients can be determined by fixed moment tensors, enabling equivalence to matched full-replay training under specified conditions. The authors introduce an Anchored MLP (AMLP) and propose Loss Anchoring Points (LAP), which optimize synthetic features and labels to approximate these otherwise prohibitively large statistics. Experiments evaluate AMLP heads on frozen pretrained features across standard class-incremental benchmarks, with an additional Places365 experiment assessing LAP under a constrained storage budget.

Critical strengths
  1. The connection between network structure and sufficient statistics is conceptually useful. The input-anchoring formulation explains how certain multilayer trainable architectures admit a representation of their entire MSE objective through fixed data statistics.

  2. The experiments include useful controls and a relevant storage comparison. The reproduced VILA baseline, increased random-feature width, and gradient-trained linear head help examine the source of AMLP’s improvements. The paper also recognizes that full feature replay can require less storage than a high-dimensional analytic cache on standard benchmarks.

Critical weaknesses
  1. The practical significance of the multilayer head needs clearer justification. In the experiments, the main architectural change is from an analytic linear classifier on fixed features to a trainable input-anchored multilayer head. The backbone remains frozen after the initial task. This extension is theoretically interesting and produces accuracy improvements. However, it remains unclear when its additional adaptability is sufficiently valuable to justify the extra parameters, iterative optimization, and memory-compression procedure. The advantage may depend on the quality of the upstream representation and its alignment with downstream tasks. Controlled comparisons across different feature conditions, with resource usage accounted for, would help establish when the multilayer structure offers a substantial practical advantage over a simpler analytic classifier.

  2. LAP’s novelty relative to dataset condensation and distillation requires further clarification. LAP optimizes synthetic features and labels to match statistics of the original data, connecting directly to dataset condensation and distillation. The authors should clarify LAP’s distinctive contribution in its matching objective, theoretical guarantees, or optimization procedure, and explain what advantages specifically follow from input anchoring. Since resource constraints are central to continual learning, this distinction may also be evaluated in terms of synthesis cost, subsequent training cost, and persistent storage, rather than predictive accuracy alone.

  3. The quantitative connection between exact-statistics theory and LAP under a finite budget is insufficiently established. Theorem 1 provides exact equivalence when the complete statistics are available. Theorem 2 assumes the existence of synthetic data that exactly reproduce those statistics and the attainment of corresponding global minimizers. These conditions do not establish the approximation quality achievable with the budgets used in practice. The paper does not sufficiently connect the number of synthetic points, the achieved moment-matching error, and deviations in loss, gradients, or subsequent learning performance. Reporting achieved matching errors and examining their relationship to downstream behavior would strengthen the connection between the theoretical motivation and the implemented method.

  4. The experiments do not yet adequately establish LAP’s practical value across resource budgets. The main experiments on eight standard benchmarks use full historical feature replay. This choice has a reasonable storage motivation, but these results primarily validate the AMLP head rather than LAP compression. The substantive compression evaluation is concentrated on Places365 at one storage budget, with uniform subsampling as the main compression control. Evaluation across several budgets, especially tighter constraints, and comparisons with representative condensation or stronger subset-selection methods would better establish the method’s usefulness. The paper should also specify which persistent states are included in the budget and report synthesis time and subsequent training overhead.

Questions for authors

See Weaknesses.

4 comments

We note the following trends among the reviewer’s highlighted weaknesses:

  1. A request for a demonstration of practical value of the method through further experimentation across different budgets and settings, including: “different feature conditions” (Weakness 1), different matching errors (Weakness 3), different synthetic point budgets (Weakness 3), and different datasets (Weakness 4)

  2. A request for clarification on resource usage including: parameter counts (Weakness 1 and 4), and fitting/training time (Weakness 4)

  3. A request for clarification of novelty of LAP relative to dataset distillation, and an empirical comparison (Weakness 2 and 4)

If we are missing something, please don’t hesitate to let us know.

We hope that for the most part, our proposed experimental slate addresses the requests for further experimentation. We are unsure what “different feature conditions” the reviewer had in mind, but we offer that changes in the underlying backbone manifest in changes to the data distribution which LAP compresses, so our expansion of our paper’s Experiment 2 to covering three datasets already partially covers this axis. If there are further changes the reviewer thinks are critical to see, we would be happy to continue the discussion.

To address the request for experiments at different matching errors and the request for clarification on fitting/training time, we propose a follow up experiment (if time allows) which ablates the number of fit steps for algorithm 1 on Places 365. This will show the relationship between moment matching error, accuracy, and time.

We have two follow up comments which we will post shortly, one tackling the practicality concern, time, and parameter efficiency, and one tackling the comparison to dataset distillation.

AuthorsOpenReview ↗

On Time, Memory, and Practicality

We focus the paper on the tradeoff between memory and accuracy, treating CL as a game of who can retain the most accuracy while requiring the least storage. As a result, we never tuned LAP’s fit steps, setting it at a sufficiently large number (48k) such that we weren’t worried about fit quality as a confound and caching the result. While we acknowledge that time/compute efficiency are important, we intentionally leave this as an open question for future work in settings which have more specific restrictions (e.g. robotics settings where models must update in real time).

In weakness 1, the reviewer asks when the “additional adaptability is sufficiently valuable to justify the extra parameters, iterative optimization, and memory-compression procedure”. To our understanding of the continual learning objective, the latter two costs, as measured in computational resources and time, are justifiable in the pursuit of higher accuracy at matched storage as they push forward the frontier. We are not alone in accepting this tradeoff. For example, ConLoRACUO, as mentioned by Reviewer K2Vy, makes a similar trade off when it trains LoRA adapters iteratively for each task.

This brings us to parameter counting. In the paper, we don’t count model parameters when matching storage, opting instead to match the width and depth of the heads. We describe this both in the first paragraph of section 4.1 and in the “Honest Storage Accounting” subsection where we provide explicit formulae for how we get our numbers. There are three reasons why we do this:

  1. To our understanding, there is no pre-established standard in the subfield. Prior ACIL papers tend to count the O(d2)O(d^2) state package as essentially free (see our honest storage accounting subsection), so we are forced to create our own standard.

  2. Because of differences in class counts, matching parameters would lead to slight differences in model shape for each new benchmark. Let dod_o be the output shape of VILA’s backbone (1280), ww be the number of random features (16384), and CC be the number of classes. Then, an AMLP head has 2⋅do⋅w+w⋅C2 \cdot d_o \cdot w + w \cdot C parameters while the VILA analytic head has do⋅w+w⋅Cd_o \cdot w + w \cdot C. Therefore, across different datasets, true parameter matching would require adjusting the number of random features in the VILA head for each benchmark.

  3. Not all parameters are created equally. Both the analytic head and the AMLP head rely on a set of frozen random features. It’s unclear how these features should be treated in terms of storage accounting. In a restrictive storage regime, an implementer could simply store a random seed and re-generate these parameters on demand without needing more than a few bytes of permanent storage.

Because of this complexity, we decided the cleanest comparison was a simple head-to-head: our 16384 features vs their 16384 features. The accompanying state difference is small: (do⋅w=16384∗1280→0.084GB)(d_o \cdot w = 16384 * 1280 \rightarrow 0.084GB). For converting numbers to GB we multiply by a factor of 4/1094/10^9 to represent the number in fp32.

For our paper’s first experiment, the difference is null, as VILA’s 1.07GB statistics cache is larger than the largest dataset and the parameter difference combined. Not only that, but we explicitly compare to a VILA model with double the width to remove any doubt about the strength of our method.

For our paper’s second experiment, the difference is minor, if we count any parameters beyond VILA’s default model with head width 16384 in our state package, then the AMLP has a total storage of: m(do+Cfit)+w⋅do=1.157m(d_o+C^{\text{fit}}) + w \cdot d_o = 1.157GB and and a VILA analytic head with adjusted width w′w^\prime has total state at least: w′⋅w′+(w′−16384)∗1280+(w′−16384)∗365w^\prime \cdot w^\prime + (w^\prime - 16384) * 1280 + (w^\prime - 16384) * 365. For w′=16982w^\prime = 16982, this is 1.1571.157 GB. We say ‘at least’ and use 365 in the formula because places 365 has 365 classes, so CC is at least 365 in our paper’s second experiment (iNaturalist and ImageNet-1k each have more classes).

Ultimately, we are confident (due to Experiment 1’s results) that the 3.6% increase in feature count will not change the ordering of our results, and we think that the existing matched feature count comparison is cleaner for the paper. However, we’re happy to run the tests and report them in this thread along with the rest of the paper’s updated Experiment 2.

In this case, the added parameters the reviewer identified are accounted for and justified directly in the storage budget. Our method is therefore practical under the lens of CL because it achieves a higher accuracy with matched storage.

AuthorsOpenReview ↗

On Dataset Distillation

We are excited that you brought up dataset distillation! We are currently planning a follow-up paper which directly explores a cross-subfield comparison between ACIL methods and Dataset Distillation methods for continual learning (expanding upon our Honest Storage Accounting argument from the paper).

That said, while we think the comparison to dataset distillation is both interesting and valuable, we think it belongs in a separate future work for the following reasons:

  1. Novelty; LAP, while dataset shaped, is a novel approximation method in its own right. Dataset distillation primarily relies upon heuristics, often through bi-level optimization, such as matching performance, parameters, or gradient trajectories. Even our perhaps closest neighbor in the space, Kernel Inducing Points (KIP), which also uses a closed form single-level optimization approach, constructs its dataset such that kernel ridge regression achieves strong performance on the orginal dataset, and uses this as a proxy for the target neural network. LAP, in contrast, has an exact objective of the Frobenius distance between the LAP set’s sufficient statistics and the true dataset’s sufficient statistics. In the ideal case, typical dataset distillation methods reproduce the result of the initial dataset. In our ideal case, (when J=0), the LAP set is the dataset for the purposes of generating MSE gradients. The distinction is critical precisely in the continual learning setting because LAP admits the stacking of its approximations (Lemma 2 of our paper). Therefore, our method is far more than a simple application of an existing dataset distillation method to an ACIL model, but rather a natural consequence of the core question of the second half of our paper: “Can we effectively approximate the sufficient statistics in polynomial time?”.

  2. Space; From a structural perspective, we view the purpose of this paper as establishing the input-anchoring theory and demonstrating its empirical value in the ACIL setting. With a tight 10-page limit, we see no way to give the cross-subfield comparison the full attention it deserves, and we don’t see it as a necessary inclusion to accomplish the goals of the paper (those being theoretical rigor and empirical feasibility in our subfield, as opposed to empirical dominance across CL).

With that said, we don’t intend to omit dataset distillation from the paper, and are happy to explicitly name the work and suggest this future comparison in both our literature review and in our future work section. Would this fairly address the concern in your eyes?

I am very grateful to the authors for their detailed comments. After carefully reviewing them, I believe that most of my concerns have been addressed to some extent. Therefore, I have decided to raise my rating. Good luck.

Reviewer_K2VyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the problem of analytic continual learning in the context of class-incremental learning. The authors claim that the existing methods cannot achieve absolute memorization in deep neural networks and present a new technique to adapt the ACIL to deep networks. The theorectical section provides the process of how input anchoring and corresponding archietecture is constructed. Then this paper proposes the Anchored Multi-layer Percepton (AMLP) which utilizes input anchoring (in the form of GaLU in this paper) to store the statistics ad a loss-anchoring points (LAP) method is used to reduced the complexity of storage. TExperiments upon VILA demonstrates the performance of proposed AMLP. However, the empirical study is not sufficient under the setting adpoted in this paper and the implementation of the algorithm during CIL process is not well presented in the manuscript.

Critical strengths
  1. The motivation and research problem in this paper is clear and well formed.
  2. The technical framework of this paper is complete and sound.
  3. Introducing the input anchoring to construct a dataset for absolute memory is novel under the context of CIL.
Critical weaknesses
  1. Claim of "absolute memorization" lacks sufficient discussion. In ACIL, absolute memorization is formulated by transforming the CIL to recursive learning with a frozen backbone that yields identical results. In this paper, AMLP seems to constructs a new dataset for learning a deeper network. As such, what is the direct relation between this dataset to absolute memorization? More detailed analysis should be further provided.

  2. The proposed method still only focuses on the classification head. Although the classifier is deeper, the learning process cannot achieve feature adaptation in the internal layers.

  3. In sufficient empirical study. The main experiment is bulit upon VILA, a method utilizes extra CLIP model to provide features, which is not a widely adpoted setting of CIL. Since the CLIP is not necessary in the proposed method, a better way to achieve fair comparison is to use a single ViT. Also, the empirical study lacks abalation study between different input anchoring architure and the depth of AMLP LL.

  4. The first method to achieve absolute memorization in deep network is overstated. There are several analytical methods based on recursive least-square solution can achieve absolute memorization. For example, the DeepAFL [R1], CPNet [R2] and ACnnL [R3]. Although these methods do not focuses CIL itself, the formulation of recursive least square can achieve a theorectical guarantee of absulte memorization when applied to CIL. Also, it is interesting to have a comparative study with these techniques.

  5. Lack of Comparison of most up-to-date analytic learning-based technique, i.e., ConLoRACUO [R4]. ConLoRACUO utilizes analytic learning to merge lora of different tasks and the classifier, which can be a good baseline in the context of CIL with pre-trained ViT.

[R1] Jianheng Tang, et al., DeepAFL: Deep Analytic Federated Learning, ICLR 2026.

[R2] Huiping Zhuang, et al., Correlation projection for analytic learning of a classification network, Neural Process. Lett. 53 (2021) 3893–3914.

[R3] Huiping Zhuang, et al., An analytic formulation of convolutional neural network learning for pattern recognition, Information Sciences Volume 686, 2025, 121317.

[R4] Yue Lu. et al., Continual Low-Rank Adaptation Via Cumulative Unified Optimization, in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2026.3724818.

Questions for authors

My questions are mostly related to the experiments and the illustration of the "absolute memory", please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Weakness 4

Weakness 4's claim deals directly with the core novelty of the paper, so we take it seriously and address it first. We analyze each source separately and explain why each cannot be used to achieve deep absolute memorization.

In summary, our Input Anchored models provably (Proposition 3 + Proposition 1) are factorized weight linear regressions over a set of exponentially many features. This means they are absolute memorizers in the same sense that linear regressions are: by definition. Given the set of sufficient statistics collected over a stream, they can be trained from scratch to produce exactly the same result as full-batch gradient descent with MSE loss. The provided papers, on the other hand, use analytic techniques to train a deep model, but, due to the non-linearities within the target models, can’t (to our knowledge) achieve non-trivial absolute memorization.

To give a representative analogy, you can consider a LL-layer MLP trained with the following procedure: Take data matrix XX and weight matrix WiW_i. Use an analytic solve to compute the WiW_i which minimizes ∣∣XWi−y∣∣2||XW_i - y||^2. Set X=σ(XWi)X = \sigma(XW_i) and repeat to identify Wi+1W_{i+1} until all WW are determined.

While the technique involves an analytic solve, and the output is a deep model, the model itself remains highly non-linear, and thus can’t be updated recursively with new data in the same manner ACIL methods do. If, upon receiving a new data point, you update WiW_i, then the statistic set for Wi+1W_{i+1} becomes invalid (as it was computed with respect to WiW_i). If you leave WiW_i the same, then the model is frozen except for the last layer and it reduces to the L=1L=1 case ACIL methods already use. Therefore, the only absolute memorization available to this approach is to store the entire dataset, which is the trivial solution. We now show how the listed papers fit this analogy.

Paper-by-Paper analysis

R1 - DeepAFL: In section 3.1, they describe their core algorithm. In the last paragraph, they describe the learning sequence of W0→Ω1→W1→…ΩT→WTW_0 \rightarrow \Omega_1 \rightarrow W_1 \rightarrow \dots \Omega_T \rightarrow W_T. Just in our analogy above, they train each layer of their model sequentially. Ωt\Omega_t has knowledge of the final values of Ω1,…t−1\Omega_{1, \dots t-1} and is never updated again after it is trained. This non-update is critical because each Ωi\Omega_i is trained with respect to σ(Φi−1Bi−1)\sigma(\Phi_{i-1}B_{i-1}) where Φi−1\Phi_{i-1} is itself a function of Ωi−1\Omega_{i-1}. In their case, σ\sigma is GELU, so each learned Ωi\Omega_i for i>1i > 1 has a non-linear dependency on Ωi−1\Omega_{i-1}. This means that adding an additional datapoint requires retraining the entire stack, which has not been shown to be achievable without storage of the full dataset. They support this claim themselves in their conclusion, saying: “Applying our DeepAFL to [continual learning] remains non-trivial. This is because the features learned from a previous phase may become invalid in a subsequent one if updating the weights layer by layer. As a result, we would need a new approach to incorporate the phase-wise recursions as well”.

R2 - CP Net: This paper builds a similar formulation (See section 4.1.2) where each layer takes as input the previous layers output and builds relu(XXTP)\text{relu}(XX^TP) for a random projection PP. They then fit a weight matrix to match an objective, and the output of the layer is relu(relu(XXTP)W)\text{relu}(\text{relu}(XX^TP)W). Each layer is again trained sequentially, with the WW of one layer never being updated after its initial fit. Like DeepAFL, there is a non-linear dependence between the WW of one layer and the WW of a previous layer. This means that adding a datapoint to the dataset requires retraining the full stack over the full dataset.

R3 - ACnnL: This paper applies the same philosophy of the previous two to CNNs. It flattens the images to enable the linear regression treatment and builds each new weight matrix sequentially (see their Algorithm 1 at the end of section 4). They apply avg pool and leaky relu between layers, enforcing the same non-linear dependency of later weight matrices upon earlier weight matrices as the other papers. Therefore, ACnnL as well is not directly capable of absolute memorization.

Overall, a primary motivation of these methods is applying the speed of analytic solutions to train deep neural networks faster (e.g. single epoch), which they are generally successful in achieving. However, they do not enable absolute memorization in the style of input anchoring, and their sequential training methods mean that earlier layers can not adapt to the representational needs of later layers in the same way that an input anchored model trained with gradient descent can. Therefore, we dispute the claim that prior art has already covered the contributions of our paper. If there are further works which the reviewer would like us to analyze, please do not hesitate to ask.

AuthorsOpenReview ↗

Response to Weakness 1

The LAP set, the new dataset we construct, is the mechanism we use to practically approach absolute memorization without paying exponential storage costs, but is not itself the mechanism for absolute memorization. The paper is built in two halves, the first of which establishes the theory behind input-anchoring (the real absolute memorization mechanism) and the second of which establishes LAP as a practical mechanism for using the properties of Input Anchored models (An approximation method).

Here’s a breakdown of how absolute memorization appears in the paper, and where we discuss our connection to it, and how it relates to LAP:

  1. We define absolute memorization initially in the abstract, and expand it in the second paragraph of the introduction

  2. The core of the paper, the connection between Input Anchoring and absolute memorization can be found in Theorem 1 (which shows that the MSE loss landscape of an Input-Anchored model is determined fully by a finite set of sufficient statistics).

  3. In the sentence following Theorem 1’s proof, we clarify how this sufficient statistic argument enables absolute memorization: “Therefore LMSEL_{MSE} and ∇θLMSE\nabla_{\theta}L_{MSE} at every θ\theta are computable from (M,V,E∣∣y∣∣2)(M,V,E||y||^2) without storing any data points: full-batch gradient descent on the statistics produces the same iterates as full-batch gradient descent on the data, for any initialization and hyperparameters.” We’re happy to add an additional sentence here clarifying that this ability to reproduce gradients and loss exactly is what enables absolute memorization, as taking identical gradient steps along a training trajectory from the same initialization is guaranteed to produce the same output.

  4. We then go on to explain how this generalizes the algorithm and absolute memorization used by ACIL.

The motivation for LAP can be seen in Proposition 2: while AMLPs are naturally absolute memorizers, the storage complexity of the sufficient statistics is exponential: O((1+d)2L)O((1+d)^{2L}). This is constant in the number of datapoints NN, so for long enough datastreams this will eventually be more efficient than storing the dataset, however, the cost is steep. This is what motivates the second half of our paper: approximating the sufficient statistics to avoid the need to spend exponential time and storage. Because LAP is an approximation, it does not in practice truly achieve absolute memorization: the weights of the final model are different than they would have been if trained on the full dataset. However, we show in Experiment 2 (and even more precisely in the proposed update from our summary comment) that they do sufficiently capture enough information to meet / exceed the generalization performance of the full dataset. Detailed discussion of the motivation, technique, and experiments surrounding LAP can be found in section 3.3 and section 4.3.

If there are specific parts of the paper where the reviewer finds description confusing, lacking, or ambiguous, we’d be happy to expand upon them, just let us know where areas these are.

AuthorsOpenReview ↗

Response to Weaknesses 2, 3, and 5

We address the remaining weaknesses together, as we believe they stem from related concerns. The reviewer brings up ConLoRACUO [R4], a recent ACIL style CL attempt which updates LoRA adapters throughout the ViT, and asks why we do not compare to it. They suggest that VILA’s structure for their pre-trained model hasn’t been widely adopted by the field yet, and suggest that building off of it is unrepresentative of the broader CIL landscape. Finally, they question our study’s focus on the classifier head and request deeper ablations (this last point we address in our summary, addendum experiment 3) .

We begin by discussing ConLoRACUO [R4]. While the reviewer didn’t explicitly suggest that ConLoRACUO’s contribution conflicted with our core novelty, we hope that clarifying why it isn’t a direct competitor to our work will aid the discussion. Like [R1-R3] ConLoRACUO’s method involves Analytic steps but can’t be used for true absolute memorization. Unlike [R1-R3], though, it is applied to the continual learning setting. The reason it can’t be used for absolute memorization is because they apply updates to intermediate weight layers. Because the layers are interdependent via non-linearities this allows the representations to drift with per-layer fitting error rising to ~5–6% with depth (Fig. 5), and 7–13% cosine distance from the oracle class token (Fig. 6), increasing with task count. VILA describes why this is a problem in “The recursive dilemma” subsection of section 3.1 in their paper. By updating the underlying representations, they create “Gram Matrix Inconsistency”, removing any theoretical guarantees of absolute memorization. This isn’t necessarily a weakness of ConLoRACUO, but it shows that what they aim for is different from our work. They essentially use ACIL as a form of heuristic / regularizer to merge LoRA weights, and run updates analytically.

With that clarified, we now address the remaining concerns:

Why focus on the classifier head?‾\underline{\text{Why focus on the classifier head?}}

  1. As we suggest in our protocol (Section 4.1) and future work paragraph (Section 5), comparing a matched width and depth classifier over the same base allows us to show our improvement in a controlled setting where the only difference is the structure of our 16384 features (learned instead of frozen). The idea is to create a direct head-to-head setting with as few confounds (e.g. model shape) as possible.

  2. Extending the Input Anchoring method to the deep architecture of the ViT requires follow up work, since conventional ViTs are not input-anchored (they involve complex operations such as softmax and layer norm). In this work, we focus our experiments on the AMLP. We plan for future work to individually tackle each progressive step of complexity until we can build full Input Anchored transformer architectures. This pattern isn’t unique to us. ACnnL [R3] extends to CNNs the discoveries of CPNet [R2].

Why build off of VILA instead of ConLoRACUO?‾\underline{\text{Why build off of VILA instead of ConLoRACUO?}}

There’s nothing stopping us from building our method on top of ConLoRACUO instead of VILA. With ConLoRACUO using a generalized analytic classifier at its peak, there’s nothing in theory preventing us from extending their head to a multi-layer head to enable improvements through depth. At the same time, ConLoRACUO's structure makes true absolute memorization impossible, while VILA's does not, making VILA a cleaner base to compare to in our estimation.

Perhaps most importantly, to our understanding, ConLoRACUO was not published until August 18th 2026 (https://ieeexplore.ieee.org/abstract/document/11657961), meaning that by the time it was released our experiments built on VILA were already well underway. Under ICLR’s guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines) anything published after July 17th 2026 is considered contemporaneous work and we are not required to compare to it.

You ask why we use VILA’s base when it incorporates a dual-branch structure which the rest of the field hasn’t adapted yet. We do this in anticipation of the request you make in weakness 5: a request for comparison to the latest ACIL literature. With VILA only being published in May of this year (ICML 2026), it makes sense that their dual branch architecture hasn’t been widely adopted yet. For the purposes of our study, we see no reason why comparing to one of the latest winners as our base would invalidate our results. Our method applies to ACIL heads regardless of the structure of the underlying model.

Reviewer_5FijOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces input anchoring, allowing a model's output to be expressed as an inner product between a parameter tensor and a data tensor. For such models, the MSE loss is determined entirely by two moment tensors of the data, which can be accumulated across tasks in place of old data, extending the absolute memorization of analytic class-incremental learning to deeper models. The authors propose the Anchored MLP and Loss Anchoring Points (LAP), which approximates the moment tensors using a small set of synthetic points. Replacing VILA's analytic head with an AMLP improves accuracy on eight benchmarks, and LAP outperforms VILA and a uniform coreset at matched storage on Places365.

Critical strengths
  1. Clear theoretical formulation. The paper gives a clear formal definition of input anchoring. It shows that, for input-anchored networks, the full MSE loss landscape is determined by two moment tensors of the data (Theorem 1). This extends the absolute-memorization property of analytic CIL beyond linear heads. The authors also introduce the Anchored MLP (AMLP) as a concrete input-anchored architecture, which shows how the theory can be applied to deep models.

  2. Practical implementation via LAP. Storing the exact sufficient statistics grows exponentially with depth. The authors therefore propose Loss Anchoring Points, a low-rank approximation method that makes the approach usable in practice without materializing the full tensors.

Critical weaknesses
  1. The approximation quality of LAP is not directly evaluated. While exact moment preservation guarantees equivalence to full replay, LAP only approximates these moments. The paper does not directly quantify the resulting approximation errors in the loss landscape or gradients, or examine how these errors correlate with accuracy.

  2. The sensitivity to the memory size m is not studied. The LAP experiments use a single memory budget, so it is unclear how approximation quality and CIL performance change as m decreases.

Questions for authors
  1. How well does LAP preserve the full-replay moments, loss landscape, gradients, and training trajectories, and how do approximation errors correlate with accuracy?

  2. How do LAP's approximation quality and CIL performance change as m decreases?

  3. The title and framing emphasize deep networks, but all experiments appear to use a degree-2 AMLP. How do the method and LAP behave for larger L?

2 comments
AuthorsOpenReview ↗

Review Period Coming to a Close

We thank you again for your time. As the review period comes to a close, we wanted to reach out with a gentle nudge on the discussion. In case it got lost, our primary responses to your review are in our top level comments in this forum. The experiments you asked for are in the revised PDF: approximation quality in §4.3 and Appendix C.2, memory size in Table 3 (top), and larger L in Table 3 (bottom left). Our “Revised Draft Uploaded” comment maps each of your points to its location. Please let us know if anything remains open.

I thank the authors for providing a detailed rebuttal and experiments, most of which address my concerns.

I still have one remaining concern, which is the scalability of LAP with increasing network depth (L).

I think it would be interesting to investigate the relationship between L and m (e.g., whether increasing m can compensate for the performance degradation at large depths).

This might also help clarify the practical applicability of input anchoring to deeper architectures.

Overall, I will maintain my original rating of Weak Acceptance

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Initial Reviews and Planned Experiments

We thank the reviewers for their thoughtful feedback and constructive criticism! We are pleased to find that each of the reviewers noted the clarity, usefulness, or soundness of our theoretical / technical formulation as a strength of the paper.

We have reviewed the identified weaknesses and propose the following additional experiments to help alleviate them. We list these roughly in order of impact:

Experiment 1‾\underline{\text{Experiment 1}}

Summary of Weaknesses:

Both Reviewer 5Fij (Weakness 1, Question 1) and Reviewer YX1Z (Weakness 3) requested a more detailed analysis of LAP's performance with respect to absolute memorization, specifically along the lines of a loss landscape or gradient analysis. Secondarily Reviewer YX1Z (Weakness 4) points out that our primary LAP result is on a single dataset, Places 365.

We agree that these findings are critical, moreover, we go a step further: we think that the existing Experiment 2 demonstrates a weaker win for LAP than is strictly necessary. Batch noise, epoch count (as we point out in the paper), and batch size all pose possible confounds to LAP's increased final accuracy. To remedy we propose the following updated version of Experiment 2:

Experiment:

To eliminate confounds and better analyze LAP's proximity to absolute memorization, we are re-running Experiment 2 with full-batch gradient descent instead of mini-batch gradient descent. While this is less representative of what a real implementer might run, it allows us to directly compare the statistical information contained in the LAP set to that contained in the full dataset. Moreover, we are planning to extend the experiment to cover both the iNaturalist 2021 dataset and the ImageNet 1K dataset to demonstrate that the improvement is not a transient property of the Places-365 dataset, but a consistent one.

If successful, we believe this will strengthen the core claim of the paper that LAP does indeed effectively recover the relevant statistical information of the full dataset.

Moreover, every 100 steps in these new experiments, we measure: a) At the weights determined by the LAP arm's trajectory, the cosine similarity between the LAP set's gradient and the true data-set's gradient b) The loss computed by the full-dataset vs. the loss computed by the LAP set. c) Both the train and test accuracy of both arms.

Given that, as we expect, the LAP arm's final test accuracy and loss do not match the full-data's, we already know that absolute memorization is not being achieved; we expect the above measurements to help shed light on what exactly LAP is doing in this process such that we may address the concerns of Reviewer 5Fij and Reviewer YX1Z.

We also plan to report the moment matching loss as requested by Reviewer YX1Z.

Limitations: Because full-batch gradient descent on large datasets is orders of magnitude slower than mini-batch gradient descent, we limit the comparison to the final task of the CIL protocol. Notably, LAP is still fit without knowledge of future datapoints, so this experiment is strictly at a disadvantage relative to the full continual learning experiment (which gets to carry forward information from previous tasks in the form of weight initializations).

Experiment 2‾\underline{\text{Experiment 2}}

Both Reviewer 5Fij (Weakness 2, Question 2) and Reviewer YX1Z (Weakness 3 and 4) request further analysis of how LAP performs at different m budgets.

We propose running an experiment 2 ablation on the Places 365 dataset where we measure the effect at different m budgets. To match VILA's 1.07 GB, we used 574 LAP points per class. We are running the following bracket of points per class: [1, 5, 10, 50, 100, 200, 300, 400, 500, 574 (Experiment 2's default)]

Experiment 3‾\underline{\text{Experiment 3}}

Both Reviewer 5Fij (Question 3), and Reviewer K2Vy (Weakness 3) requested experiments which show how AMLPs and LAP perform at depths greater than 2. We selected depth 2 to keep the original experiments tightly contained by matching VILA's head's shape precisely, but are happy to show results with deeper models.

For consistency with the main results, we propose running an ablation on Places 365 with VILA's backbone. However, to avoid needing to build and train massive 16,384 x 16,384 dimensional weight matrices, we'll compress down the head dimension from 16,384 to 1,280 (VILA's output dimension before their random feature head). We will train models with matched width at each layer for L=2,3,4, and 5, and analyze LAP sets for each layer count.

Timeline‾\underline{\text{Timeline}}:

We expect to have results for the experiments within the next 3-4 days. If you feel one or more of these does not adequately address your concerns, please don't hesitate to let us know how and why.

If a concern of yours is not addressed at all in the above experiments (e.g. Reviewer K2Vy's novelty question and Reviewer YX1Z's practicality and synthesis time concerns), we plan on addressing them directly with a follow up comment soon.

AuthorsOpenReview ↗

Preliminary Results for Addendum Experiments

Experimental details:

We found through some preliminary experimentation that at the larger scales, dropout monotonically hurt the performance of the full-data AMLP while occasionally helping the LAP arms. For fairness, and to get you these results as fast as possible, we remove dropout entirely as an axis from these experiments and run everything at dropout 0. Notably, the ImageNet-1k benchmark is used in the training of VILA's backbone. We include it here since it's a staple large-scale dataset, but its performance should be taken with a grain of salt relative to methods whose backbones are not pre-trained on it. We ran a bracket of learning rates [1e-4, 3e-4, 1e-3, 3e-3, 1e-2] on ImageNet-1k for each of our arms, and all of them selected 1e-3 by best train accuracy. The rest of the experiments use 1e-3 as their learning rate without further tuning. This does mean that AMLP arms have a slight tuning advantage on ImageNet-1k. Additionally, for each dataset, we use a class-balanced subset leaving 732 (ImageNet-1k), 3068 (Places365), and 152 (iNat21) points per class.

Addendum Experiment 1:

MethodImageNet-1k (↑\uparrow)Places365 (↑\uparrow)iNat21 (↑\uparrow)
VILA analytic (H=16384H=16384)83.1154.9858.66
VILA analytic (H=16982H=16982)83.1555.0358.90
Uniform coreset83.6755.3752.36
Full data83.9156.6957.54
LAP84.1057.0860.32

Addendum Experiment 2:

Points per class151050100200300400500574
Coreset24.9737.7542.2847.8649.8252.4853.8554.6555.2255.37
LAP43.3041.1244.6654.1155.9056.9057.1657.1357.1257.08
Δ\Delta (LAP −- coreset)+18.33+3.37+2.38+6.26+6.08+4.42+3.31+2.48+1.90+1.71

Addendum Experiment 3 (574 points per class):

Depth LFull dataLAPUniform coresetΔ\Delta (LAP −- coreset)
253.8554.4253.52+0.90
355.9355.8954.08+1.81
456.4355.1752.72+2.45
556.1154.3751.51+2.86

For reference, VILA analytic at the same width (H=1280H=1280) achieves 50.47.

Analysis:

As you can see from the first table, the wins of LAP hold not only under the stricter full-batch gradient descent protocol but are maintained across three datasets. Interestingly, AMLP Full data does worse than VILA analytic on iNat21, but the gain from LAP is enough to overcome the difference, leaving LAP + AMLP still on top. We hypothesize that the small training window of 4k steps is a confound, as the AMLP with full data uses an iterative protocol while VILA Analytic uses a closed-form solve.

In our second addendum, it's clear that LAP's wins are achievable earlier than the matched storage point. LAP's accuracy plateaus at only 300 points per class, almost half of what the primary experiments use. It beats the full data arm from 200 points per class onward, beats VILA analytic from 100 points per class onward, and beats the uniform coreset at every budget.

Finally, for our third addendum, the AMLP's performance increases with depth, but LAP's performance relative to full data decreases. We suspect that a root cause of this is that the sufficient statistic set for deeper AMLPs grows exponentially in 2L2L, so LAP's approximation target becomes exponentially harder to realize. At the same time, LAP's lead over the uniform coreset grows with depth: past L=3L=3, both compressed arms decline, but the coreset declines much faster. Additionally, the width decrease seems to hurt VILA analytic more than it hurts the AMLP: going from width 16384 to 1280 costs VILA analytic 4.51 points, but the L=2L=2 AMLP only 2.84 (full data) and 2.66 (LAP). This matches what we would expect from the NTK / lazy training literature: a linear head over frozen random features is limited by how well its finite width approximates the underlying kernel, while the AMLP can adapt its features to the data.

Graph analysis:

OpenReview doesn't allow images in comments, so the full set of plots will be in our revised draft, which we expect to post by Sunday (November 15) along with the rest of this data. In brief: train accuracy on the real rows is higher for LAP (and matched on iNat21). The cosine similarity between LAP's gradient and the full-data gradient is high early in training but decays rapidly, matching our hypothesis that LAP converges to a different optimum in the loss landscape. However, the true loss of the point LAP achieves is generally close to the one the true gradients lead to, demonstrating that the loss landscapes aren't wildly different.

The fit-step ablation we proposed for Reviewer YX1Z (relating Algorithm 1's number of fit steps to moment-matching error, accuracy, and synthesis time) is currently running, and we expect to post its results soon.

AuthorsOpenReview ↗

Addendum Experiment 4

Addendum Experiment 4: LAP fit steps vs. moment-matching error, accuracy, and synthesis time

As promised to Reviewer YX1Z, we ablate the number of Algorithm 1 fit steps on Places365 under the same full-batch protocol as Addenda 1–3 (L=2L=2, 574 points per class). We report final test accuracy, the moment-matching loss J=JM+JVJ = J_M + J_V achieved by the fit (Lemma 1), and the wall-clock time to fit the LAP atoms on a single A40.

LAP fit stepsFinal test acc (%)JJJMJ_MJVJ_VA40 time per class (s)All 365 classes (GPU-h)
12556.681.06e-104.97e-115.67e-112.70.27
25056.777.09e-113.52e-113.57e-114.00.41
50056.834.85e-112.55e-112.30e-117.30.74
1k56.893.47e-111.94e-111.53e-1114.01.4
2k56.942.66e-111.57e-111.08e-1127.02.7
4k56.972.16e-111.35e-118.04e-1254.35.5
8k57.031.84e-111.21e-116.38e-12107.310.9
16k57.061.65e-111.11e-115.43e-12214.321.7
32k57.081.53e-111.04e-114.90e-12430.043.6
48k57.081.48e-111.01e-114.70e-12648.765.8

We see three main takeaways:

  1. Matching error and accuracy move together. As fit steps increase, JJ falls monotonically by roughly 7× (1.06e-10 → 1.48e-11), and test accuracy rises monotonically with it (56.68 → 57.08). This gives the direct link between achieved approximation quality and downstream performance requested by Reviewer YX1Z (W3) and Reviewer 5Fij (Q1).

  2. Most of the benefit arrives early. Synthesis cost grows linearly in fit steps (~13.5 ms per step per class), while accuracy gains diminish quickly. At 1k steps (14 s per class, 1.4 A40 GPU-hours for all 365 classes), LAP reaches 56.89, already above the full-data arm (56.69), VILA analytic (54.98), and the uniform coreset (55.37). Even at 125 steps (0.27 GPU-hours), LAP matches the full-data arm.

  3. The setting used in our paper was conservative. We originally chose 48k steps to rule out fit quality as a confound, not for efficiency. This ablation shows it buys only +0.19 points over 1k steps at 47× the cost. For practical settings where speed is in consideration, we recommend the 1k steps.

In the continual learning setting, Algorithm 1 runs independently per class, and classes arrive one per task in this benchmark. At 1k steps, the synthesis cost per incoming task is therefore ~14 s on a single A40, and fits for classes within a task can be run in parallel.

These results will be included in the revised draft.

AuthorsOpenReview ↗

Revised Draft Uploaded, Summary of Changes

We thank the reviewers again for their consideration. The revised PDF is now uploaded. The main text is 10 pages; all new material beyond Section 4.3 is in the appendix. Below we list what changed, where it is, and which concern it answers.

Where the addendum experiments are

In this threadIn the revised paper
Addendum 1 (full-batch, three datasets)Section 4.3 (Experiment 2), Figure 2 (right), Figure 3; full trajectories in Appendix C.2, Figure 5
Addendum 2 (memory budget)Appendix C, Table 3 (top), Figure 4 (left)
Addendum 3 (depth)Appendix C, Table 3 (bottom left)
Addendum 4 (fit steps, matching error, synthesis time)Appendix C, Table 3 (bottom right)
Original Experiment 2 (mini-batch, Places365)Unchanged, moved to Appendix C.3, Figure 6
Original Experiment 3 (head ablation)Unchanged, now titled "Head Ablation", Appendix D

Reviewer 5Fij

  • W1 / Q1 (approximation quality): Section 4.3 now compares LAP with full data under full-batch training on three datasets. Appendix C.2 reports gradient cosine similarity and true versus predicted loss along training; Table 3 relates the achieved matching error JJ to accuracy.
  • W2 / Q2 (memory size mm): Table 3 (top) and Figure 4 (left).
  • Q3 (larger LL): Table 3 (bottom left), with the relative error of the fitted tensors at each depth. LAP trails full replay at L≥4L \geq 4; Limitations notes that its advantage over full data decreases with depth.

Reviewer K2Vy

  • W1 (absolute memorization): a sentence after Theorem 1 states how identical losses and gradients give absolute memorization. Section 4.3 states plainly that LAP at our budgets does not reproduce the full-replay model.
  • W3 (depth ablation): Table 3 (bottom left).
  • Presentation of the CIL procedure: Appendix B now gives the full training and fitting procedure, and Algorithm 2 is corrected to show that first-task classes are compressed as well.
  • W2, W4, W5: See responses in thread.

Reviewer YX1Z

  • W1 / W4 (resource accounting): Section 4.2 now defines storage as non-model state carried across task boundaries. Appendix B.1 explains why we match features and not parameters, and reports VILA at the parameter-matched width H=16982H = 16982 (at most +0.24 points; no ordering changes).
  • W2 (dataset distillation): cited in Section 2.1 and named as future work in Section 5.
  • W3 (theory versus finite budget): Table 3 (bottom right) reports JJ against accuracy across fit steps; Table 3 (top) across budgets.
  • W4 (synthesis and training cost): Table 3 (bottom right) for fitting time. New since our last comment: Appendix C.1 and Figure 4 (right) vary the head's training batch size and report wall-clock time; at batch size 512 training takes 39.6 s against 46.6 s for a VILA Analytic task.

Corrections

  • Regularization. The original draft suggested LAP's lead came from regularization. With our latest results from the reviewers' requested ablations, our perspective has changed. Higher train accuracy does not fit that reading, so Section 4.3 withdraws it and leaves the mechanism open.
  • Algorithm 1 now normalizes JJ to a per-entry mean squared error, and Lemma 1 is restated to match. This deviates slightly from our code, which doesn't normalize based on class count and the first layer's width. We decided to adjust Algorithm 1's definition since it is a cleaner overall formulation and all reported JJ values are computed as in Algorithm 1. This leads to a minor weighting difference between JMJ_M and JVJ_V which we disclose in Appendix B; we did not have time to re-run the fits within the discussion period.
  • Removed a contradictory "22% fewer rows" statement in the original Experiment 2.
  • The Places365 subset has 3068 images per class (the original said 3063).
  • Proposition 2 now assumes C=O(d)C = O(d).
  • The abstract now reads "accuracy benefits of absolute memorization" (updated in the OpenReview abstract field as well).
  • Limitations and Future Work are separate paragraphs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.