acceptodds
Under review as a conference paper at ICLR 2027

QUILT: Continual Cross-entropy Learning from Moments and Centroids

Abstract

Exact moments support continual least squares, but do not determine the non- linear updates required by cross-entropy. We introduce QUILT, a readout learner that combines exact historical moments with a bounded, class-wise centroid mem- ory. The moments preserve the supervised label term, the centroids approximate the nonlinear prediction term, and a reusable matrix factorization supports de- scent on the compressed objective. An exact KL decomposition identifies the loss discarded by compression. An optional covariance term matches its second- order expansion. Under a Gaussian model, the converged corrected classifier is classification-consistent. Experiments cover seven datasets and four frozen encoders, with replay, functional-memory methods, and ACIL/GACL, DS-AL, and AIR under common retained-byte ceilings. The complete learner exceeds a matched RP-ridge head by 0.36 percentage points in a ten-seed replication, with a 95% paired interval of [0.29, 0.42]. At the larger budget, AIR and both QUILT variants have nearly equal mean accuracy. Their temperature-calibrated NLLs are 0.537 for AIR, 0.495 for QUILT-0, and 0.488 for QUILT-C across the 14 settings. The covariance correction itself has a small, conditional effect: changing only that term gives +0.39 points on native features and−0.04 on random features, with losses on individual datasets. Our evidence supports the combined memory design while limiting the accuracy claim for its covariance correction.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class incremental learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ijCuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes QUILT for continual classification with frozen features, combining exact historical moments with weighted class centroids to approximately optimize cross-entropy. Its main analysis expresses the loss gap induced by compression as a KL divergence between predictive distributions and uses this to construct an optional covariance correction. The experiments support the effectiveness of the full method in native-feature settings with small memory budgets, although the gains from the correction itself are modest and depend on the representation and hyperparameters.

Critical strengths
  1. I appreciate the paper’s clear separation of several issues that can easily be conflated: preserving label statistics, approximating the nonlinear prediction term, and ensuring descent of the compressed objective. Equation (5), Proposition 1, and Section 4.2 give these distinctions precise mathematical meaning. Using covariance in the update metric does not yield the same limiting solution as adding a covariance penalty to the objective. This analysis helps clarify the behavior of cross-entropy optimization under limited memory.
  2. The most convincing empirical evidence comes from the small-budget comparisons using native features. In Table 1, QUILT-C achieves an average accuracy of 88.21% on the vision tasks, compared with 87.24% for K-priors+. The comparison considers stored bytes, historical class weights, and whether the baselines have been sufficiently optimized. The authors also use ten additional seeds to examine the gains over matched RP-ridge and report the effects of the full method separately from those of the correction. Section 5.4 reports negative gains from the correction on random features, which lends credibility to the paper’s conclusions.
Critical weaknesses
  1. The existing ablations do not fully answer whether the exact matrix justifies its memory cost. The centroid-metric ablation in Appendix A.3 still retains the exact G for ridge initialization, while Appendix F.8 holds the centroid summary fixed and changes only the update metric. These experiments clarify the role of G within the current algorithm, but leave open whether a simpler memory allocation could suffice: omitting the full G, allocating the saved bytes to more labeled centroids, and directly optimizing their weighted cross-entropy. This bears directly on the necessity of the combined memory design, rather than merely adding another external baseline.
  2. The practical benefits need to be discussed in terms of accuracy, predictive loss, and training cost. In the large-budget results in Table 3, AIR, QUILT-0, and QUILT-C all achieve approximately 86.78% average accuracy across the 14 settings. QUILT-C has an advantage over AIR in calibrated NLL, but Table 22 reports a better value for K-priors (0.467) than for QUILT-C (0.488). These results therefore reflect trade-offs across metrics. Appendix A.4 states that timing was recorded, and Appendix B.2 explains the auditing overhead included in the timings, but the paper lacks directly comparable training times for the selected models. Such timings would help assess whether the gains justify the additional computation in practice.
  3. The empirical evaluation of continual compression remains limited to a fixed stream structure. Appendix A.2 uses ten disjoint class arrivals. Appendix F.9 already reports continual accuracy and forgetting metrics, but these results do not test the accumulation of compression error as the number of arrivals increases. Section 7 also acknowledges that repeated compression may accumulate error. Varying arrival granularity would be more informative about reliability over longer streams than adding further similar datasets. Classification consistency under the Gaussian model does not directly answer this question.
Questions for authors
  1. Could the authors evaluate the alternative described in Weakness 1 under the same byte budget, with regularization and optimization settings selected independently for that alternative?
  2. Could the authors report the time spent constructing the summaries and solving for the classification heads for QUILT, AIR, and K-priors, explicitly excluding validation search and auditing work?
  3. With the samples and memory budget held fixed, how do accuracy and historical loss change as the granularity of arrivals varies?
Reviewer_MyWuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces QUILT, a continual learning framework that combines exact historical moments with a bounded centroid memory to optimize cross-entropy loss without storing all previous examples. The main contributions include a memory-efficient learning approach, a covariance correction for information lost during compression, and theoretical analysis of the approximation error and convergence properties. Experiments across seven datasets and four frozen encoders show improvements over several replay and functional-memory baselines. However, performance is comparable to stronger analytic methods such as AIR under larger memory budgets.

Critical strengths

1/ The theoretical support is pretty strong. Combining exact moments with centroid memory provides an interesting way to approximate historical cross-entropy objectives. The theoretical analysis of compression error and covariance correction strengthens the methodological contribution. 2/ The paper also did comprehensive empirical evaluation. The paper evaluates multiple datasets, encoders, and memory budgets, including comparisons against strong continual learning baselines. The additional replication studies provide useful evidence for the reported improvements.

Critical weaknesses

The paper only shows limited improvement over strong analytic baselines. At larger memory budgets, QUILT achieves nearly identical average accuracy to AIR. The covariance correction also provides relatively small and inconsistent improvements, raising questions about its practical contribution. Also, it did not show computational efficiency and scalability. QUILT requires maintaining a quadratic-size moment matrix and performing matrix factorization and iterative optimization at each arrival. Although memory usage is carefully controlled, the additional computational overhead is not sufficiently compared against existing methods.

Questions for authors

Could the authors provide a comparison of training and update latency against analytic baselines such as AIR and ACIL under matched memory budgets? This would help establish whether the accuracy improvements justify the additional computation. Given that the covariance correction provides inconsistent improvements across feature representations and datasets, can the authors identify the conditions under which it is beneficial in practice? Some experiments with larger memory budgets effectively retain the entire training feature set. How much of QUILT's advantage remains when the centroid memory is strictly smaller than the accumulated training data across all evaluated settings?

Reviewer_9mSJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

QUILT learns a continual classifier on frozen features by combining exact historical moments with a bounded, class-wise centroid memory. The paper characterizes the loss discarded by compression, provides convergent updates for the compressed objective, and introduces an optional covariance correction with guarantees under a binary Gaussian model. Experiments span seven datasets and four encoders, including memory-matched comparisons and a ten-seed replication. I recommend weak acceptance: the memory design and analysis are useful contributions, although the accuracy gains are modest and the practical trade-offs need clearer evaluation.

Critical strengths
  1. The analysis gives a clear account of what centroid compression preserves and loses. I particularly appreciate the distinction between using covariance to guide optimization and including it in the objective: these can lead to different classifiers at convergence. The exact KL decomposition and Gaussian example make this distinction concrete.

  2. The experimental evidence is carefully qualified. Comparisons account for retained bytes, the gain over matched RP-ridge is replicated across ten additional seeds, and the correction is tested separately at fixed hyperparameters. Reporting its negative results, alongside AIR’s matching average accuracy and QUILT’s better calibrated NLL, makes the conclusions credible.

Critical weaknesses
  1. The experiments do not fully establish whether exact moments are the best use of the available memory. The centroid-metric ablation still uses the original Gram matrix for ridge initialization and keeps the centroid budget unchanged. A centroid-only CE learner that reallocates the Gram-matrix storage to additional centroids, with independently tuned optimization, would more directly test the benefit of the combined memory design.

  2. The practical advantage over the strongest analytic baseline remains unclear. AIR essentially matches QUILT’s average accuracy at the larger budget, while QUILT requires clustering and repeated optimization. The lower calibrated NLL is valuable, but a clean comparison of selected-model training time—including summary construction and factorization—would help establish when the additional work is worthwhile.

Questions for authors
  1. How does QUILT compare with a centroid-only CE learner that uses the same total byte budget, allocating the storage saved by omitting exact moments to more centroids?

  2. What are the per-arrival training times of the selected QUILT and AIR configurations, including all learner operations but excluding validation searches and evaluator-only checks?

Reviewer_v3pzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents QUILT, a method to fine-tune a classifier by retaining weighted class centroids and feature statistics that approximate the cross-entropy loss on earlier examples. The authors demonstrate this approach through experiments on several datasets and encoders.

Critical strengths

The authors demonstrate that QUILT-C outperforms K-priors+ in most cases (Table 1), which is strong support of their claims. The theoretical methodology is also supportive of their claims.

Critical weaknesses
  1. I find the paper difficult to understand in several parts. The introduction discusses several theoretical results before making the basic algorithm clear. A worked example of one QUILT-0 update (e.g., in a figure) would make this easier to understand, perhaps. In general, focusing on clarity rather than equations (which perhaps in part could be moved to the appendix?) may make it easier to parse.
  2. The feature matrix costs memory that could otherwise hold additional centroids. A centroid-only classifier with the same budget would clarify whether storing the matrix is worthwhile. I believe this is necessary to fully understand the benefits and gains.
  3. The paper feels very AI generated (whether or not it is). It lacks a clear description of the motivation for things, and reads as simply description of what was done, as facts without any connecting sentences or motivation. It seems to be written in a way that obfuscates the meaning. Even the first sentence "Pretrained encoders make continual classification feasible with a small trainable readout" is ambiguous and barely understandable to the wider audience. A much clearer sentence would have been: "In a frozen pretrained encoder, continual classification reduces to updating a linear classifier on fixed features." The whole paper reads like this.
  4. Although the paper may be technically sound, and the work is good, the writing of the paper is low enough quality that, to me, it warrants a clear reject. It is simply not up to the standards of ICLR. I recommend a thorough editing of the paper.
Questions for authors

How does QUILT compare with a classifier trained by cross-entropy on as many weighted centroids as the same memory budget permits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.