acceptodds
Under review as a conference paper at ICLR 2027

Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel in Continual Learning

Abstract

Data-free class-incremental learning (DFCIL) relies on model inversion to synthesize replay samples from a frozen classifier, using stored feature statistics to mitigate catastrophic forgetting. Existing methods typically model these statistics with a diagonal covariance, discarding dependencies between feature dimensions, not because such correlations are unimportant, but because dense covariance modeling is computationally prohibitive. We show that this restriction can be lifted. We introduce REMIX, a structured covariance framework that parameterizes feature dependencies with a Laplace kernel over channel coordinates. As the Laplace kernel admits a linear-Gaussian state-space representation, REMIX enables exact likelihood and log-determinant computation in linear memory and near-linear time, without approximations or explicitly forming dense covariance matrices. Modeling feature correlations improves the fidelity of model inversion, yielding higher feature log-likelihoods and more coherent synthetic samples. Across CIFAR-100, Tiny-ImageNet, and CUB-200, REMIX consistently improves retention of earlier tasks across ResNet, ViT, and CLIP backbones. These results demonstrate that structured modeling of feature dependencies provides a practical and scalable alternative to the diagonal covariance assumption in data-free continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Data-Free Continual Learning, Model Inversion, Laplace Kernel, Class-incremental Learning, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_HBPnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REMIX, a structured covariance modeling framework for model inversion in data-free class-incremental learning. Unlike diagonal covariance models, REMIX captures correlations between feature dimensions while avoiding the high cost of full covariance estimation. It assigns learnable coordinates to feature dimensions and uses a Laplace kernel to model their dependencies. By exploiting the Markov property of the corresponding Gaussian process, the method enables efficient likelihood computation through state-space recursion. The authors further introduce covariance fitting and cross-task fusion to preserve historical feature statistics without accessing past data. These statistics guide layer-wise model inversion to generate more representative pseudo-samples. Experiments across datasets and architectures show improvements in feature distribution fitting, sample quality, continual learning performance, and computational efficiency.

Critical strengths
  1. The paper identifies the limitations of diagonal covariance modeling in data-free continual learning and addresses an important trade-off between computational cost and knowledge retention.

  2. By exploiting the Markov property of the Laplace kernel, REMIX enables exact likelihood computation through efficient state-space recursions without dense covariance matrices. The covariance fitting and fusion mechanisms further support the proposed framework.

  3. Experiments across multiple datasets and architectures show consistent improvements over PMI, particularly on ResNet, in terms of accuracy, sample quality, and computational efficiency.

Critical weaknesses
  1. The choice of the Laplace Kernel is driven by computational efficiency rather than the actual structure of neural network features, lacking a deep theoretical connection to the research problem.
  2. Mapping features to a 1D space assumes complex high-dimensional dependencies can be captured by simple 1D distances, which fails to account for the multi-scale, non-local feature interactions in deep networks.
  3. REMIX maintains a shared covariance per layer rather than class-conditional covariances, which risks masking class-specific statistics crucial for generating distinct historical pseudo-samples.
  4. Fusing covariances via weighted averaging ignores mean differences across tasks and induces projection errors that accumulate over time, threatening long-term retention.
  5. Model parameters shift continuously in continual learning, meaning saved statistics across tasks do not share a unified feature space. The lack of cross-task alignment weakens the framework's reliability.
Questions for authors
  1. The kernel appears primarily motivated by computational efficiency. Why is a 1D Laplace kernel suitable for neural feature correlations? Do the gains come from its specific structure or simply from modeling off-diagonal dependencies?

  2. Representing high-dimensional correlations through 1D distances imposes strong structural constraints. How well does REMIX handle multi-scale or non-local dependencies that may not fit this assumption?

  3. REMIX uses a shared covariance across classes, potentially overlooking class-specific statistics. How does this design preserve class-level information, and why should better overall distribution fitting improve class-conditional inversion?

  4. Weighted covariance averaging may overlook differences in task means, while repeated projections may accumulate errors. How are the statistics centered, and what prevents these errors from affecting long-term retention?

  5. Backbone updates change feature representations across tasks, making historical statistics potentially incompatible with current ones. How does REMIX address this drift and justify direct covariance fusion over long task sequences?

3 comments

We thank the reviewer for the constructive questions and careful assessment. We address each point below and report additional experiments to clarify the design choices of REMIX.

Q1. Suitability of the Laplace kernel and contribution of covariance structure

The Laplace kernel is computationally efficient. Its Markov property (Doob's theorem) enables exact Gaussian likelihood evaluation in near-linear time. For D=2×105D=2\times10^5, LCM requires only 3.06 MB, compared with approximately 150 GB for a dense covariance. Learned coordinates adapt this structure to neural features (see HbXh, Q1).

Covariance-model comparison. Both the Gaussian likelihood objective and off-diagonal correlations contribute to the gains over PMI. REMIX achieves the highest accuracy (45.38%) among the evaluated covariance models, although differences between parameterizations are not necessarily statistically significant (see Prgq, Q1).

  • Revision: We expanded the Laplace-kernel motivation in Section 3 and added covariance-model and coordinate ablations to Appendix K.

Q2. Multi-scale and non-local dependencies

A single LCM chain imposes a 1D Markov structure. We therefore investigated a multi-chain extension:

ΣN=diag⁡(d)+∑k=1Ndiag⁡(w(k))K(a(k))diag⁡(w(k)).\Sigma_N = \operatorname{diag}(d) + \sum_{k=1}^{N} \operatorname{diag}(w^{(k)}) K(a^{(k)}) \operatorname{diag}(w^{(k)}).

We considered shared and independent orderings.

Multi-chain ablation. We evaluated up to four chains using held-out likelihood on ResNet-32/CIFAR-100 features. We measure the fraction of the dense-model likelihood improvement recovered:

Dense gain recovered=LModel−LDiagonalLDense−LDiagonal×100%.\text{Dense gain recovered} = \frac{\mathcal{L} _{\text{Model}} - \mathcal{L} _{\text{Diagonal}}}{\mathcal{L} _{\text{Dense}} - \mathcal{L} _{\text{Diagonal}}} \times 100\%.

ModelParameters (D=64D=64)Dense gain recovered
One chain (REMIX)19222%
2 chains, shared ordering25733%
4 chains, shared ordering38752%
2 chains, independent orderings32040%
4 chains, independent orderings57663%

Four independent chains recover 63% of the dense gain, compared with 22% for one chain. For flattened feature maps (D=4,096D=4{,}096--16,38416{,}384), four chains recover 40--43%, leaving some dependencies uncaptured.

  • Revision: We added the multi-chain formulation and results to Appendix M and referenced them in the main text.

Q3. Shared covariance and class-level information

Class-level information is preserved through the cross-entropy objective and class-conditional penultimate-layer targets used by CFS (Appendix Sections E and F). Shared layer statistics capture general feature dependencies and provide a common prior for inversion, following DeepInversion and PMI.

Class-conditional ablation. We compared shared statistics with class-specific diagonal and LCM statistics, fitted separately without merging (ResNet-32/CIFAR-100, 10 tasks, αstat=0.5\alpha_{\mathrm{stat}}=0.5).

StatisticsFinal accuracy (%)
REMIX45.38 ± 0.13
Class-conditional diagonal44.15 ± 0.25
Class-conditional LCM44.04 ± 0.58

Class-conditional statistics did not improve accuracy, possibly due to noisier estimates.

  • Revision: We added this ablation to Appendix K.

Q4. Task-mean differences and accumulated approximation errors

We analyzed five sources of error in sequential correlation aggregation on ResNet-32/CIFAR-100 (10 tasks), averaging over 31 BatchNorm layers and five seeds: omitting the between-task mean, averaging correlations instead of covariances, representation drift, a single LCM projection, and repeated LCM projections.

Appendix K.6 defines and details the computation of these errors. The table shows how each error contribution changes at task 10 relative to its average over tasks 2--10:

Source of errorChange (pp)
Between-task mean term+0.01
Averaging correlations+0.05
Representation drift+3.69
LCM projection, once−3.44
Repeated LCM projections+0.83

The between-task mean and correlation-averaging errors remain negligible. Repeated LCM projections add only 0.83 pp, whereas representation drift increases the error by 3.69 pp. The contribution of a single LCM projection decreases by 3.44 pp, reflecting a structural approximation rather than accumulated error.

  • Revision: We added the error decomposition and experimental results to Appendix K.6.

Q5. Representation drift and covariance compatibility

The marginal means and variances from BatchNorm remain consistent with the current teacher's feature representation. Historical correlations are invariant to per-channel shifts and rescaling, although they may become less representative as the backbone evolves. Nevertheless, REMIX's empirical results suggest that direct covariance fusion works well despite this potential drift. Explicitly addressing representation drift remains an interesting direction for future work.

Reviewer_HBPnOpenReview ↗

Official Comment

Thanks for the detailed rebuttal and extra experiments. While Q3 is largely resolved, several core issues still hold me back. First, the small accuracy gaps among covariance models do not convincingly justify choosing the Laplace kernel specifically (Q1). Second, the multi chain test actually highlights the limitation of the 1D design, as one chain captures just 22% of the dense likelihood gain compared to 63% for four chains, and downstream accuracy is not reported to justify this trade off (Q2). For Q4, reporting relative error changes in Appendix K.6 rather than absolute trajectories does not prove accumulated errors are small, and the mismatch between covariance space theory and correlation space aggregation in Algorithm 4 remains unclear (also, please check Table 7 where the +0.31 mean falls outside its [-0.13, +0.16] CI). Most importantly, representation drift (Q5) remains my biggest concern: channel wise invariance does not handle cross channel or nonlinear backbone shifts, and your own analysis shows drift driving a major error increase (+3.69 pp) without any alignment mechanism to fix it.

We thank the reviewer for the follow-up and address the remaining concerns below.

Representation drift.

Although REMIX does not update the stored cross-channel correlations of earlier tasks to account for representation drift, its PMI-based inversion replay pipeline (Tong et al., 2025) adapts the other components of replay to the evolving representation:

  1. Dynamic target re-estimation. Following PMI, after each task, the dreamed images of previous classes are passed through the newly trained teacher to recompute the class-specific feature targets (means and marginal standard deviations of backbone features) and the contrastive models used for Contrastive Feature Selection (CFS). The targets thus reflect how the current teacher represents the dreamed images, rather than an earlier feature space.

  2. Current-state generation. All dreamed images are regenerated after each task by inverting the current teacher with its own BatchNorm statistics.

  3. Progressive distillation. The student is distilled on these images through hard knowledge distillation (HKD), an L1 loss on the teacher's old-class logits (Gao et al., 2022), whose weight grows threefold over the 10 tasks (R-DFCIL's adaptive weighting, Appendix F).

Only the stored correlation does not follow the current teacher, so we measured its drift (Appendix K.6). The +3.69 pp is the change in the drift term between its mean over tasks 2–10 and task 10, not its magnitude. In absolute terms, drift lowers the information retained by the dense merge from 99.6% to 77.6% after 10 tasks, yet an LCM fitted to this merge is only 3.0 pp below one fitted to the true correlation (20.9% vs. 23.9%): much of the drift lies outside what one chain can represent.

Accumulated error and Algorithm 4.

Absolute trajectories. We added absolute correlation-information retention across all 10 tasks to Appendix K.6 (0% = diagonal, 100% = true correlation; 3 seeds, 31 layers):

TaskLCM fitted to true correlationREMIXDifference
123.6%23.6%0.0 pp
223.1%22.0%1.1 pp
422.6%20.9%1.7 pp
623.1%20.6%2.5 pp
823.3%20.1%3.2 pp
1023.9%19.5%4.4 pp

The first column remains stable at 22.6–23.9%. Of REMIX's 4.4 pp gap after 10 tasks, 1.4 pp comes from sequential refitting (the LCM fitted to the dense merge retains 20.9%), while the remaining 3.0 pp arises from merging, primarily due to representation drift.

Algorithm 4. By Proposition 1, every unit-diagonal LCM correlation matrix is itself an LCM covariance: R(ϑ)=Σ(1−ρ,ρ,a).R(\vartheta)=\Sigma(\mathbf 1-\rho,\sqrt{\rho},a). Therefore, the covariance-space results, including the inner products and projection, apply directly to correlations with (d,w)=(1−ρ,ρ)(d,w)=(\mathbf 1-\rho,\sqrt{\rho}). The diagonal residual vanishes identically, making the correlation-space aggregation in Algorithm 4 consistent with Proposition 1.

Table 7.

We thank the reviewer for their sharp eye in catching this error. We corrected the CUB-200 statistics in Table 7.

DatasetMetricPMI + MoE-AdapterREMIX + MoE-AdapterREMIX − PMI [95% CI]p-value
CIFAR-100Avg87.80 ± 0.9387.69 ± 1.02−0.11 [−0.24, +0.02]0.087
CIFAR-100Last81.26 ± 0.8181.32 ± 0.50+0.06 [−0.36, +0.47]0.771
CUB-200Avg78.79 ± 0.9179.10 ± 0.69+0.31 [−0.72, +1.34]0.518
CUB-200Last67.30 ± 1.2867.12 ± 1.01−0.18 [−1.66, +1.30]0.791

Choice of kernel and downstream accuracy.

We do not claim an accuracy advantage for the Laplace kernel; diagonal and low-rank alternatives achieve comparable results. We choose it for its computational efficiency: the Markov structure enables exact likelihood evaluation with O(D)\mathcal O(D) memory and O(Dlog⁡D)\mathcal O(D\log D) time. On flattened feature maps, it recovers 34% of the dense likelihood gain with 3 parameters per dimension, compared with 42% for blocks of size 16 using 7.5 parameters per dimension. It also scales to D≈5×105D\approx5\times10^5, where dense covariance is prohibitively expensive.

For per-channel statistics, four chains recover 52% of the dense likelihood gain with a shared ordering and 63% with independent orderings, compared with 22% for one chain. Multi-chain variants have not been evaluated in continual learning, so their downstream accuracy remains unknown. We view the single-chain design as an efficiency–expressiveness trade-off and can provide downstream accuracy results for multi-chain variants if requested by the reviewer.

Final remarks. We have updated the manuscript to include the corrected Table 7 statistics and the absolute representation-drift measurements. We hope these revisions address the remaining concerns. If any issues remain, please let us know, and we will be happy to address them.

Reviewer_PrgqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REMIX, a model inversion framework for data-free class-incremental learning. Its central component is a structured covariance model combining a Laplace kernel over learned one-dimensional feature coordinates with diagonal residual noise. The associated state-space representation permits exact Gaussian likelihood evaluation without constructing dense covariance matrices. The paper also introduces covariance fitting and continual aggregation procedures. Experiments show consistent improvements over PMI on ResNet-based benchmarks, while improvements in the CLIP-based setting are substantially smaller.

Critical strengths

A useful connection between structured covariance modeling and model inversion. Learning latent feature coordinates allows the method to exploit the computational structure of the exponential kernel even when feature dimensions have no predefined ordering. Exact likelihood evaluation under this structured model provides a practical alternative to diagonal Gaussian statistics.

Consistent improvements in the ResNet setting. Table 4 reports results over five runs across two datasets and three task configurations. Improvements over PMI range from 0.51 to 1.48 percentage points on CIFAR-100 and from 0.74 to 1.44 points on Tiny-ImageNet. The reported computational overhead is moderate in the evaluated settings, supporting the practical relevance of the approach.

Critical weaknesses

The experiments do not sufficiently isolate the benefit of the proposed covariance structure. REMIX introduces an exact Gaussian NLL statistical objective and retunes its weight. A diagonal Gaussian NLL baseline using the same inversion pipeline and a comparable tuning budget is needed to distinguish the benefit of cross-feature correlations from that of the objective and its weighting. Comparisons with alternative compact covariance models, such as diagonal-plus-low-rank models, would also clarify why the learned one-dimensional Laplace structure is preferable. This matters particularly for CLIP: Table 2 shows only +0.12 points in CIFAR-100 average accuracy and −0.02 points in CUB-200 average accuracy, without uncertainty estimates. The available evidence therefore supports the broader claims less strongly than the manuscript suggests.

The continual aggregation procedure has unresolved mathematical and procedural issues. Equation 5 averages task covariances, while Algorithm 5 separately averages their means. If the result is intended to approximate the pooled feature distribution, its covariance should also include the between-task mean term. Otherwise, the intended statistical target needs to be explicitly distinguished from pooled covariance. Furthermore, the claimed exact preservation of marginal variances is conditional: Algorithm 4 uses (d_i=\max(T_{ii}-w_i^2,\varepsilon)), which yields (\Sigma_{ii}=T_{ii}) only when (w_i^2\leq T_{ii}-\varepsilon). The algorithm does not explicitly enforce this constraint. These issues affect the interpretation and guarantees of a central contribution.

Questions for authors
  1. Can you provide a diagonal Gaussian NLL control with identical inversion settings and comparable hyperparameter tuning? How much of the improvement remains, and how does REMIX compare with a compact low-rank covariance model?
  2. What distribution is the merged covariance intended to represent? If it approximates pooled task features, how is the between-task mean contribution handled? How is exact marginal preservation enforced when optimizing (w)?
  3. Algorithm 5 fits and merges statistics before updating the feature extractor, then returns those statistics with the updated model. How are stored statistics kept compatible with the evolving feature representation in the ResNet setting?
  4. Can you report multiple-run uncertainty for Table 2 and clarify the validation protocol used to select (\alpha_{\mathrm{stat}})? Are the likelihood results in Figures 4 and 10 evaluated on held-out features?
3 comments

We thank the reviewer for the detailed feedback, which helped us clarify REMIX’s covariance modeling, statistical aggregation, and training procedure, and motivated additional experiments.

Q1. Diagonal control and low-rank model

Controlled comparison. We compare REMIX against a diagonal Gaussian likelihood on ResNet-32/CIFAR-100 (10 tasks), keeping the training pipeline unchanged. The diagonal control uses R=IR=I with BatchNorm variances and the same αstat\alpha_{\text{stat}} grid as REMIX (selected value: 0.5).

MethodFinal acc. (%)
PMI44.42 ± 0.57
Diagonal Gaussian NLL44.95 ± 0.31
REMIX45.38 ± 0.13

The gains over PMI come from both incorporating the Gaussian likelihood objective and modeling off-diagonal correlations.

  • Revision: We added the diagonal control, αstat\alpha_{\text{stat}} sweep, and covariance-model comparisons to Appendix K. In Section 4, we clarified the sources of the gains and referenced the corresponding results in Appendix K.

Q2. Merge target, between-task means, and marginals

Merge target. Means and variances come from the BatchNorm statistics, Σ=diag⁡(σ)R(ϑ)diag⁡(σ)+10−5I\Sigma=\operatorname{diag}(\sigma)R(\vartheta)\operatorname{diag}(\sigma)+10^{-5}I; merging acts only on correlations. After task tt, we compute the correlation R^t\hat R_t of the activations under the trained model, centered at the task mean, and refit the LCM to

\alpha_t=\frac{|\mathcal Y_{1:t-1}|}{|\mathcal Y_{1:t}|},\quad \beta_t=1-\alpha_t.$$ Thus, the target is a class-weighted average of within-task correlations, $\sum_k\pi_k\hat R_k$, rather than the pooled covariance. **Between-task means.** We do not explicitly model the between-task term $\Sigma_B=\sum_k\pi_k(\mu_k-\bar\mu)(\mu_k-\bar\mu)^\top$, treating its omission as an approximation. We measured its effect on ResNet-32/CIFAR-100 and found it negligible: omitting $\Sigma_B$ reduces retained correlation information by only 0.01 pp after 10 tasks. **Marginals.** Our implementation uses the normalized covariance from Prop. 1, $R(\vartheta)=S^{-1/2}\Sigma(\vartheta)S^{-1/2}$, where $S=\operatorname{diag}(d+w^2)$. This ensures $\operatorname{diag}R(\vartheta)=1$ for all parameter values. Consequently, optimizing $w$ changes the correlations without affecting the marginal variances, which are determined solely by $\sigma$. * *Revision:* We clarified the correlation-space merge in Eq. 5 and updated Algorithm 4. Appendix L reports the effect of omitting the between-task mean contribution. **Q3. Algorithm 5 and evolving representations** We thank the reviewer for catching the ordering discrepancy. In the implementation, for each task $t$: (1) we invert the teacher $f_{1:t-1}$ using $\vartheta_{1:t-1}$; (2) train $f_{1:t}$ on current data and replay, using the R-DFCIL distillation strategy also adopted by PMI; and (3) estimate $\hat R_t$ using the updated model $f_{1:t}$ and merge it into $\vartheta_{1:t}$. The resulting statistics are stored with $f_{1:t}$ for the next task. Thus, the statistics used to invert the teacher at the next task are those estimated using that teacher. All training losses are described in Appendix F. The marginal means and variances are obtained from the current teacher's BatchNorm layers and therefore remain consistent with its feature representation. Stored correlations are invariant to per-channel shifts and rescaling, although correlations estimated for earlier tasks may become less representative as the backbone evolves. * *Revision:* We corrected Algorithm 5 to reflect the actual order of model and statistics updates. **Q4. Uncertainty, $\alpha_{\text{stat}}$, and held-out likelihood** - *Table 2:* We report results over 10 ViT seeds, including standard deviations, paired 95% confidence intervals, and $t$-tests (see our reply to HbXh, Q2). - *$\alpha_{\text{stat}}$:* As specified in Section 4 (Experimental Setup), we followed the exact PMI protocol and hyperparameter settings, tuning only $\alpha_{\text{stat}}$. We selected the best-performing configuration and report its final accuracy averaged over five seeds. - *Held-out likelihood:* Figures 4 and 10 show in-sample log-likelihoods for LCM and diagonal baselines on ResNet-34 and ViT-B/16 ImageNet features (no validation/test split; only LCM is fitted). On held-out ResNet-32/CIFAR-100 features (20k train, 5k validation, 10k test), LCM improves over the diagonal Gaussian by +0.076 nats/dim for per-channel features and +0.369 for flattened maps, averaged over dimensions and layers. * *Revision:* We clarified in the captions of Figures 4 and 10 that the reported log-likelihoods are in-sample.

Thank you for the clarifications and additional results. Concerns about validation and stored correlations under evolving representations remain, so I maintain my score.

Thank you for taking the time to consider our response. We would like to clarify the two remaining concerns.

1. Validation of αstat\alpha_{\text{stat}}.

We followed exactly the same hyperparameter-selection protocol as PMI: αstat\alpha_{\text{stat}} was selected based on the final test accuracy of a single run, without a separate validation split, and the selected value was subsequently evaluated over five seeds. The diagonal control in Q1 followed the same protocol, ensuring a matched comparison.

2. Stored correlations under evolving representations.

Representation drift naturally affects historical correlations as the feature extractor evolves. In REMIX, correlations are computed at the end of each task using the corresponding trained backbone and then stored for subsequent tasks without being artificially refreshed, deliberately avoiding the compounding errors of synthetic re-estimation. To rigorously quantify this effect, we conducted a targeted evaluation on ResNet-32/CIFAR-100 (10 tasks, 3 seeds; Appendix K.6), comparing the incrementally merged LCM correlations with an exact oracle correlation computed from all previously seen real data under the current backbone.

Because REMIX obtains marginal means and variances directly from the current teacher's BatchNorm layers, the stored cross-channel correlations are structurally isolated and automatically scale to the current marginal variances when reconstructing the covariance.

Rather than undermining the method, these results support the effectiveness of REMIX's design: the incremental covariance fusion achieves 45.38% final accuracy while maintaining a compact, sequential representation of feature dependencies. Although explicitly addressing representation drift, particularly for high-dimensional spatial features and longer task sequences, remains an interesting direction for future work, our results demonstrate that direct sequential fusion is effective in practice.

Please see our discussion with Reviewer HBPn for further details.

We have addressed both concerns with additional explanations and experimental evidence. If you nevertheless consider either concern unresolved, we would appreciate a specific explanation of the remaining methodological issue or the evidence you believe is still missing. This would help us better understand your perspective and address any remaining concerns.

Comments on the paper

1 comment

We thank the reviewer for the constructive feedback. We conducted additional experiments on covariance assumptions, ViT results, and computational scalability, reported in Appendix K and referenced in the main text.

W2/Q1. Motivation for the Laplace kernel and comparison with alternative covariance models.

  • Motivation: The Laplace kernel, k(a,a′)=exp⁡(−∣a−a′∣)k(a,a')=\exp(-|a-a'|), defines the Ornstein–Uhlenbeck covariance. Its Markov property enables exact Gaussian likelihood evaluation with O(D)O(D) memory and O(Dlog⁡D)O(D\log D) time, making it practical for high-dimensional features (D≈2×105D\approx2\times10^5), where dense covariance is prohibitively expensive.

  • Learned coordinates: Since neural channels have no natural ordering, REMIX jointly learns their coordinates and channel-specific loadings. We evaluated the importance of this ordering on ResNet-32 using CIFAR-100 in a 10-task class-incremental setting (10 classes per task), averaged over five seeds. We compared learned coordinates against equally spaced frozen coordinates and randomly permuted learned coordinates.

    CoordinatesPer-channel (D=16D=16–6464)Flattened maps (D=4,096D=4{,}096–16,38416{,}384)
    Frozen+0.039 ± 0.001+0.361 ± 0.023
    Learned, permuted+0.002 ± 0.006−1.081 ± 0.132
    Learned+0.077 ± 0.006+0.378 ± 0.024

    Results show that the learned channel arrangement is essential for the likelihood improvement.

  • Covariance alternatives: Under the same five-seed setup (ResNet-32, CIFAR-100, 10 tasks), we compared REMIX with diagonal, low-rank, block-diagonal, sparse precision, and dense covariance models with shrinkage, varying only the correlation model.

    Correlation modelParameters per layerFinal accuracy (%)
    Diagonal044.95 ± 0.31
    Low-rank + diagonal19244.76 ± 0.64
    Block-diagonal22445.00 ± 0.45
    Sparse precision≈19244.34 ± 0.48
    Dense + shrinkage2,01644.78 ± 0.10
    REMIX19245.38 ± 0.13

    REMIX achieves the best final accuracy while requiring fewer parameters to store its covariance model than the alternative approaches.

  • Revision: We expanded the kernel motivation in Section 3 and added both ablations to Appendix K.

W1/Q2. Statistical significance of ViT-based results.

  • Concern: The original ViT improvements were small and lacked statistical significance tests.

  • New evidence (App. K): We repeated the CLIP ViT-B/16 experiments with ten seeds and report standard deviations, paired 95% confidence intervals, and two-sided paired tt-tests.

    DatasetMetricPMI + MoE-AdapterREMIX + MoE-AdapterREMIX − PMI [95% CI]p-value
    CIFAR-100Avg87.80 ± 0.9387.69 ± 1.02−0.11 [−0.24, +0.02]0.087
    CIFAR-100Last81.26 ± 0.8181.32 ± 0.50+0.06 [−0.36, +0.47]0.771
    CUB-200Avg78.79 ± 0.9179.10 ± 0.69+0.31 [−0.13, +0.16]0.780
    CUB-200Last67.30 ± 1.2867.12 ± 1.01−0.18 [−1.63, +1.12]0.633
  • Findings: None of the differences is statistically significant (p≥0.087p\geq0.087). Importantly, there is little room for further improvement in this setting: joint training, which provides an upper bound, exceeds PMI in final average accuracy by only 1.0 pp on CIFAR-100 and 1.7 pp on CUB-200. This limited headroom helps explain the small differences between REMIX and PMI.

  • Revision: We added the ten-seed results and statistical tests to Appendix K and revised the corresponding claims, Table 2, and abstract.

Q3. Computational overhead and scalability.

  • Feature dimensionality: LCM requires O(D)O(D) rather than O(D2)O(D^2) memory. Across all four ResNet-34 stages (D=376,320D=376{,}320), memory drops from 199.30 GB to 5.74 MB (35,541× reduction).

    DDLCM memoryDense covariance
    25,0880.38 MB2.34 GB
    50,1760.77 MB9.38 GB
    100,3521.53 MB37.52 GB
    200,7043.06 MB150.06 GB
    376,3205.74 MB199.30 GB
  • Number of tasks: After each task, REMIX fits the correlation models using 200 Adam steps and merges them into a single LCM. The method adds 3D3D parameters per layer, regardless of the number of tasks.

  • Backbone architecture: A100 runtimes and GPU memory usage show modest overhead across CNN and ViT backbones.

    SettingTasksPMIREMIXOverhead
    ResNet-32, CIFAR-10051 h 27 min1 h 37 min+11.5%
    ResNet-32, CIFAR-100102.35 h2.62 h+11.5%
    ViT-B/16, CIFAR-100103 h 33 min3 h 42 min+4.2%
    ViT-B/16, CUB-2001010.32 h10.60 h+2.7%
  • Revision: Runtime, GPU memory, and covariance-memory comparisons were already reported in Appendix K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.