Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel in Continual Learning

Data-free class-incremental learning (DFCIL) relies on model inversion to synthesize replay samples from a frozen classifier, using stored feature statistics to mitigate catastrophic forgetting. Existing methods typically model these statistics with a diagonal covariance, discarding dependencies between feature dimensions, not because such correlations are unimportant, but because dense covariance modeling is computationally prohibitive. We show that this restriction can be lifted. We introduce REMIX, a structured covariance framework that parameterizes feature dependencies with a Laplace kernel over channel coordinates. As the Laplace kernel admits a linear-Gaussian state-space representation, REMIX enables exact likelihood and log-determinant computation in linear memory and near-linear time, without approximations or explicitly forming dense covariance matrices. Modeling feature correlations improves the fidelity of model inversion, yielding higher feature log-likelihoods and more coherent synthetic samples. Across CIFAR-100, Tiny-ImageNet, and CUB-200, REMIX consistently improves retention of earlier tasks across ResNet, ViT, and CLIP backbones. These results demonstrate that structured modeling of feature dependencies provides a practical and scalable alternative to the diagonal covariance assumption in data-free continual learning.