QUILT: Continual Cross-entropy Learning from Moments and Centroids

Exact moments support continual least squares, but do not determine the non- linear updates required by cross-entropy. We introduce QUILT, a readout learner that combines exact historical moments with a bounded, class-wise centroid mem- ory. The moments preserve the supervised label term, the centroids approximate the nonlinear prediction term, and a reusable matrix factorization supports de- scent on the compressed objective. An exact KL decomposition identifies the loss discarded by compression. An optional covariance term matches its second- order expansion. Under a Gaussian model, the converged corrected classifier is classification-consistent. Experiments cover seven datasets and four frozen encoders, with replay, functional-memory methods, and ACIL/GACL, DS-AL, and AIR under common retained-byte ceilings. The complete learner exceeds a matched RP-ridge head by 0.36 percentage points in a ten-seed replication, with a 95% paired interval of [0.29, 0.42]. At the larger budget, AIR and both QUILT variants have nearly equal mean accuracy. Their temperature-calibrated NLLs are 0.537 for AIR, 0.495 for QUILT-0, and 0.488 for QUILT-C across the 14 settings. The covariance correction itself has a small, conditional effect: changing only that term gives +0.39 points on native features and−0.04 on random features, with losses on individual datasets. Our evidence supports the combined memory design while limiting the accuracy claim for its covariance correction.