acceptodds
Under review as a conference paper at ICLR 2027

Learning to Generate Continually in Latent Space via Bures-Wasserstein Optimization

Abstract

Continual learning for image generation aims to enable models to learn and synthesize images from new domains while preserving their ability to generate samples from previously learned ones, thereby avoiding the substantial computational cost of retraining modern generative models from scratch. The key challenge is to learn from new data without forgetting previously acquired knowledge. Existing approaches typically combine generative replay or distillation with regularization techniques, such as elastic weight consolidation (EWC), using generated samples to approximate previously learned distributions and constraining parameter updates to mitigate forgetting. However, while effective for pixel-space diffusion, we find that the assumptions underlying these regularization approaches do not generalize well to latent generative processes, such as flow matching. In this work, we propose **La**tent **B**ures-**W**asserstein (LaBW), a general framework for continual learning in latent image generative models with theoretical support. Instead of constraining model parameters, LaBW operates in the autoencoder latent space of generative models in a way that indirectly optimizes the generated image distribution in the Inception feature space. Specifically, our theoretical analysis establishes a local connection between perturbations of the latent endpoint distribution and changes in FID, revealing increased sensitivity to equal absolute covariance changes along low-variance latent directions. We further evaluate our approach across diffusion and flow matching models with both U-Net and DiT backbones on multiple benchmarks (e.g., CIFAR-10, LSUN Churches, and CelebA-HQ), demonstrating a substantial reduction in forgetting while maintaining or improving FID on both original and new domains compared to recent state-of-the-art approaches.

Primary area:
generative models
Keywords:
continual learning, latent generative models, Bures-Wasserstein distance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t5RtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual generation for latent diffusion and latent flow-matching models. LaBW constructs shared noisy latents from teacher replay samples, converts teacher and student outputs into one-step endpoint predictions, and matches their covariances with a Bures-Wasserstein loss in a low-variance subspace of the teacher latent covariance. The paper also analyzes the local relationship between small latent perturbations and Inception-feature FID.

Critical strengths
  1. Consistent results across generative formulations and architectures. LaBW reduces forgetting in four ID-to-OOD configurations and five-task CIFAR-10, covering LDMs/LFMs, U-Nets/DiTs, and 32/256 resolutions.
Critical weaknesses
  1. The theory does not directly justify the implemented LaBW loss. Theorem 1 asks how a small change in the final clean-latent distribution affects image FID after the decoder and Inception network. LaBW instead matches projected covariances of one-step z^0\hat z_0 predictions on teacher states. This leaves two gaps: (i) the paper does not show that latent-covariance BW approximates the theoretical feature-space FID sensitivity Q(V)Q(V); and (ii) it does not show that matching one-step predictions on teacher states controls the final distribution produced by the student’s full multi-step sampling process from noise. The method therefore regularizes projected second-order statistics of one-step predictions rather than demonstrably preserving the full endpoint distribution.
  2. The theory does not show that low-variance directions are always the most sensitive. The derived result is qivar=[4λi]−1Q(ηiui)q_i^{\mathrm{var}}=[4\lambda_i]^{-1}Q(\eta_i u_i). Although 1/λi1/\lambda_i favors low-variance directions, Q(ηiui)Q(\eta_i u_i) also varies with direction and can offset or reverse this ordering. The result also compares equal absolute variance changes, which impose larger relative changes on low-variance directions. The experiments likewise do not show universal superiority: the low-variance subspace is clearly better only on CIFAR-10, the full space is better on CelebA-HQ→Flowers102, and the variants are close on LSUN LDM. The low-variance choice should therefore be presented as an empirical heuristic rather than a theoretical conclusion.
Questions for authors

Could the authors provide a clearer explanation addressing the concerns I raised in the critical weaknesses section?

4 comments

Thank you for your thoughtful comments and for acknowledging the consistent improvements across generative models and experimental settings. We appreciate the opportunity to further clarify and strengthen the theoretical justification of our approach.

W1: The theory does not directly justify the implemented LaBW loss. Theorem 1 asks how a small change in the final clean-latent distribution affects image FID after the decoder and Inception network. LaBW instead matches projected covariances of one-step z^0\hat{z}_0 predictions on teacher states. This leaves two gaps: (i) the paper does not show that latent-covariance BW approximates the theoretical feature-space FID sensitivity Q(V)Q(V); and (ii) it does not show that matching one-step predictions on teacher states controls the final distribution produced by the student’s full multi-step sampling process from noise. The method therefore regularizes projected second-order statistics of one-step predictions rather than demonstrably preserving the full endpoint distribution.

A1: Thank you for the thoughtful insights. Our theoretical analysis characterizes the local sensitivity of feature-space FID to controlled perturbations of the latent distribution and motivates the design of the covariance regularisation. The effectiveness of the resulting objective in preserving the generated distribution during continual adaptation is evaluated empirically. Below, we clarify the connection between the analysis and the BW objective, and provide additional evidence relating one-step covariance preservation to that of fully sampled endpoints.

(i) Relating the latent BW loss to Q(V)Q(V) requires accounting for how the decoder-Inception map transforms latent changes. BW measures a finite covariance distance, while Q(V)Q(V) describes the local FID response along a given perturbation. A covariance change of a given magnitude can have different effects in feature space depending on this map, so a bound between the two quantities must account for that amplification.

We can make the relationship more precise using the same optimal affine transport as in Proposition 2. Given positive-definite teacher and student covariances ΣT\Sigma_T and ΣS\Sigma_S, it defines a displacement VBWV_{\mathrm{BW}} whose squared magnitude satisfies E[∣VBW∣2]=dBW2(ΣT,ΣS)\mathbb{E} [\\|V_{\mathrm{BW}}\\|^2] = d_{\mathrm{BW}}^2 (\Sigma_T, \Sigma_S). Under the regularity assumptions of Theorem 1 and assuming that the decoder-Inception map is LL-Lipschitz, we can derive from the Gelbrich inequality:

Q(VBW)≤L2dBW2(ΣT,ΣS)Q(V_{\mathrm{BW}}) \le L^2 d_{\mathrm{BW}}^2 (\Sigma_T, \Sigma_S)

The affine transport does not necessarily represent the student’s distributional change, so this does not establish a general bound for arbitrary teacher-student distributions. It does, however, extend the construction in Proposition 2 to arbitrary positive-definite target covariances and connect their finite BW distance to the local FID sensitivity along the corresponding displacement. We have added the full statement, construction, and proof in Appendix C.4.

(ii) Matching the covariances of one-step endpoint predictions does not, by itself, guarantee the preservation of the full endpoint distribution, but it is empirically intuitive and our additional experiments support its usefulness for preserving the covariance after multi-step sampling. Please refer to our answer A2 to Reviewer Th7z for further discussion and experimental evidence.

W2: The theory does not show that low-variance directions are always the most sensitive. The derived result is qivar=[4λi]−1Q(ηiui)q_i^{\mathrm{var}} = [4\lambda_i]^{-1} Q(\eta_i u_i). Although 1/λi1/\lambda_i favors low-variance directions, Q(ηiui)Q(\eta_i u_i) also varies with direction and can offset or reverse this ordering. The result also compares equal absolute variance changes, which impose larger relative changes on low-variance directions. The experiments likewise do not show universal superiority: the low-variance subspace is clearly better only on CIFAR-10, the full space is better on CelebA-HQ→Flowers102, and the variants are close on LSUN LDM. The low-variance choice should therefore be presented as an empirical heuristic rather than a theoretical conclusion.

A2: Thank you for these observations. We would like to clarify that we have always considered the low-variance regularization as a design choice that is theoretically motivated and empirically supported, rather than a theoretical guarantee of reduced forgetting. Below, we clarify how the sensitivity analysis, the choice of perturbation scale, and the subspace ablations inform this interpretation.

Regarding the sensitivity ordering, the inverse-eigenvalue factor alone does not determine which directions are most sensitive. As discussed in Sec. 3.2, the remaining term Q(ηiui)Q(\eta_i u_i) depends on the latent distribution, the Jacobian of the decoder-Inception map, and the covariance of the Inception features. We therefore evaluate the combined effect of both factors empirically, finding increased sensitivity in low-variance directions (Figure 2a). This finding, together with the theoretical analysis, motivates prioritising these directions. We have revised the abstract (Lines 025-032), contribution statement (Lines 115-119), and Sec. 3.2 (Lines 274-277) of the manuscript to clarify this distinction.

Concerning the choice of perturbation scale, equal relative perturbations would also be a valid and informative diagnostic. However, there is no general reason to expect the variance changes induced by adaptation to be proportional to the teacher’s eigenvalues. We therefore used a common absolute perturbation scale to study which directions are more sensitive to a given variance change.

Finally, in the subspace ablations, low-variance matching improves both retention and adaptation on CIFAR-10, while the other two settings have comparable performance across variants. On CelebA-HQ →\rightarrow Flowers102, full-space matching achieves a lower ID FID than low-variance matching (7.81 versus 7.89), but has a higher OOD FID (23.64 versus 23.44). We therefore interpret this as a trade-off rather than an overall advantage of full-space matching. We further examine these results in Appendix D.2, where Table 13 additionally shows that the variants achieve similar low-variance covariance preservation in these latter two settings. Since regularisation updates shared model parameters, its effects can extend beyond the selected subspace. Comparable forgetting is therefore consistent with comparable preservation of low-variance directions, even when the loss is applied elsewhere.

Thank you for the detailed response and experiments. These address several of my earlier questions. However, the theoretical connection to the implemented loss remains indirect and relies on restrictive assumptions, so the analysis is better viewed as motivation rather than a rigorous guarantee. Besides, both the theoretical discussion and evaluation focus heavily on FID. While this makes the results consistent, it gives a narrow view of distribution matching and generation quality, as FID can be easily hacked (https://arxiv.org/abs/2604.28190). I believe both the theory and evaluation could be strengthened, so I tend to maintain my original score.

AuthorsOpenReview ↗

Further responses to Reviewer t5Rt

Thank you for following up and for your feedback on our response. We are happy to further clarify the two remaining points from the reviewer.


1. Role of the theory, and its connection to the loss design.

  • Our overall positioning: We agree with your interpretation where the analysis provides the rationale for the loss design, while our empirical results support its effectiveness in reducing forgetting. This was also our intent in the original manuscript, where we noted after Proposition 2 that “the remaining term Q(ηiui)Q(\eta_i u_i) depends on the latent distribution, the decoder-Inception map, and the resulting FID geometry.” The sensitivity therefore depends on both the data and the model. We have made this distinction clearer in the revision.

  • Why the theory still matter: In the meanwhile, we believe that the theoretical analysis indeed matter because it tells us what to examine: changes of the same magnitude in different latent directions can have different effects after decoding. Our analysis quantifies this sensitivity, and our measurements show that FID is more sensitive to covariance changes in low-variance directions (Fig. 2). This finding guided our choice of which directions to preserve.

  • Connection to practical loss and its empirical effectiveness: The additional assumptions in Appendix C.4 are needed to establish a formal link between the latent loss and local FID sensitivity. However, the loss itself can be applied without assuming that the student’s distribution changes in this way during training. We tested it across four domain-transfer settings and class-incremental CIFAR-10, spanning latent diffusion and flow-matching models with U-Net and DiT backbones. LaBW consistently reduced forgetting relative to both baselines.


2. Evidence beyond FID.

Thank you for raising this, and we agree that FID alone gives an incomplete picture of generation quality. Although FID informed our analysis, LaBW does not directly optimize it as the training loss matches covariances in the autoencoder’s latent space.

Additional evaluation with more independent metrics: Reviewer 2RVa raised a similar question, and in response, we added precision, recall, density, and coverage [a,b] as four additional metrics across all settings (Appendix D.3, Tables 16–17). Across all four domain transfers, LaBW achieves higher recall and coverage on the original domain than both baselines, with comparable or higher precision and density. It also achieves the highest values on all four metrics on class-incremental CIFAR-10. These results provide evidence beyond FID that LaBW better preserves coverage of the original domain after adaptation.


[a] Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020

[b] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019

Reviewer_Th7zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LaBW to reduce forgetting when latent generative models (which previously trained for different images) learn for new tasks. It has improved the loss function by adding covariance matching loss between teacher and student clean latent predictions, focusing on low variance directions. The theoretical analysis motivated their choice, but its effectiveness is demonstrated empirically, showing reduced forgetting while maintaining comparable performance on new tasks.

Critical strengths
  1. Mathematical motivation is useful. The mathematical derivation connecting small latent perturbations to feature space FID sensitivity is useful and it has been used as motivation for the proposed regularization preserving covariance.
  2. Main experiments have shown a consistent reduction in forgetting. LaBW shows its effectiveness by covering the experiments over several datasets and training seeds, improving its performance against existing methods.
Critical weaknesses
  1. This study shows mixed support for prioritizing low variance directions. While the theoretical analysis addresses small perturbations, it does not directly prove that preserving low variance directions during training will reduce forgetting.
Questions for authors
  1. By matching the covariance of single step predictions, does it guarantee the preservation of the latent covariance generated through the entire sampling trajectory?
1 comment

Thank you for your thoughtful comments and for highlighting the mathematical motivation and consistent reduction in forgetting demonstrated by our approach. We appreciate the opportunity to further clarify the motivation and scope of our methodology.

W1: This study shows mixed support for prioritizing low variance directions. While the theoretical analysis addresses small perturbations, it does not directly prove that preserving low variance directions during training will reduce forgetting.

A1: Thank you for the insightful comment. In fact, we want to clarify that we always position the low-variance regularization as an empirical design motivated by theoretical analysis, rather than a theoretical guarantee of reduced forgetting. Specifically, qivar=14λiQ(ηiui)q_i^{\mathrm{var}}=\frac{1}{4\lambda_i}Q(\eta_i u_i) (Line 267) identifies an inverse-eigenvalue factor in the sensitivity to equal absolute variance changes. As explicitly discussed in lines 269-271, the remaining term depends on the latent distribution (e.g., dataset dependent), the decoder-Inception map, and the resulting FID geometry. Our analysis thus motivates prioritizing low-variance directions without asserting that this choice universally minimizes forgetting.

The ablations support this positioning. Low-variance matching improves both retention and adaptation on CIFAR-10 (Table 4), while the variants perform similarly on CelebA-HQ →\rightarrow Flowers102 and LSUN Churches →\rightarrow AFHQ with an LDM (Tables 4 and 12). Notably, Table 13 shows that the variants also achieve similar low-variance covariance preservation in these latter settings. Since regularization updates shared model parameters, its effects can extend beyond the selected subspace. Comparable forgetting is therefore consistent with comparable preservation of low-variance directions, even when the loss is applied elsewhere.

We have further clarified this positioning in the revised manuscript in the abstract (Lines 025-032), contribution statement (Lines 115-119), and Sec. 3.2 (Lines 274-277), emphasizing that the theoretical analysis motivates the design, while its effectiveness in reducing forgetting is rather established empirically.


Q1: By matching the covariance of single step predictions, does it guarantee the preservation of the latent covariance generated through the entire sampling trajectory?

A2: Thank you for this question. Matching the covariances of one-step endpoint predictions does not, by itself, guarantee the preservation of the covariance obtained by the full sampling process from a theoretical perspective but is empirically intuitive. This is because the model prediction at a noisy latent state determines both a clean endpoint estimate and the subsequent sampling update. For example, in our flow matching formulation, an Euler step can be written as

zt−Δt=zt−Δtvθ(zt,t)=(1−Δtt)zt+Δttz^0,θ(zt,t).z_{t- \Delta t} = z_t - \Delta t v_\theta (z_t, t) = \left(1 - \frac{\Delta t}{t} \right) z_t + \frac{\Delta t}{t} \hat{z}_{0,\theta}(z_t, t).

Regularising the covariance of these estimates therefore indirectly constrains the predictions driving the iterative sampling process across timesteps. As acknowledged in Sec. 6, we use these predictions as tractable proxies that allow distribution-level regularisation without repeatedly running and differentiating through the full sampler during training, which can be very expensive and lead to unstable gradients.

To show this consistency in a more direct way, we further evaluated teacher-student covariance distances using both one-step predictions and endpoints generated through the full sampler. Both distances are measured in the same fixed low-variance subspace.

SettingMethodOne-step covariance distance ↓\downarrowFull-sampler covariance distance ↓\downarrowID FID ↓\downarrow
CIFAR-10 8-to-2, LDMGD0.0830.60716.853
EWC0.0740.56516.810
LaBW0.0500.41915.390
LSUN Churches$\rightarrow$AFHQ, LDMGD0.5066.4847.456
EWC0.4656.2737.213
LaBW0.3625.4256.820
CelebA-HQ$\rightarrow$Flowers102, LFMGD0.1871.6469.268
EWC0.1711.5239.154
LaBW0.1331.3068.943
LSUN Churches$\rightarrow$AFHQ, LFMGD0.3093.22410.479
EWC0.2102.5989.337
LaBW0.1391.9398.492

Across all settings, LaBW’s lower one-step covariance distance corresponds to a lower distance between fully sampled endpoints and a lower ID FID. These results support the effectiveness of the proxy for preserving the full-sampling covariance in the evaluated subspace, and are aligned with the ID FID results. We have added these results to the manuscript in Sec. 5.2 (Lines 516-522) and Table 6.

Reviewer_2RVaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LaBW, a continual learning framework for latent diffusion models. It first collects a set of synthetic samples generated by the original teacher model. Then it collects the clean latent estimates from the teacher and student model separately. Based on the covariances of these estimates, LaBW defines a Bures-Wassterstein distance between the two covariance matrices projected into low-variance subspace. Over multiple datasets, LaBW reduces forgetting while being able to adapt to new domains.

Critical strengths
  • The theoretical analysis provides interesting intuition about why matching latent covariance might help improve FID
  • The quantitative results show consistent FID improvement over various datasets.
Critical weaknesses
  • It's hard to tell the difference between the qualitative results of different methods shown in Figure 3. The claimed "better preservation of details" has limited perceptual evidence.
  • LaBW is motivated by FID sensitivity and all reported performance comparison are based on FID. This raises the question of whether LaBW is hacking the metric or truly preserving the general generative capability over the old domain.
Questions for authors
  • Are there stronger qualitative results that show LaBW has clear improvement over the baselines?
  • Could the authors provide at least one more quantitative evaluation such as precision and recall to check the sample fidelity and distribution coverage?
  • Where is the definition of point-wise generative distillation loss?
4 comments

Thank you for your thoughtful comments and for recognizing the insights provided by our theoretical analysis and the consistent improvements across multiple datasets. We appreciate your suggestions for further strengthening our experimental validation and presentation.

Q1: Are there stronger qualitative results that show LaBW has clear improvement over the baselines?

A1: Thank you for the suggestion. We have updated Figure 3 with clearer examples from the ID distributions across all three settings, along with zoomed-in views of selected regions to highlight fine-grained differences between the methods, which make LaBW’s improved preservation of details in teacher-generated samples more apparent compared to the baselines. We also provide additional ID examples in Appendix D.4 (Figures 4-6).

Q2: Could the authors provide at least one more quantitative evaluation such as precision and recall to check the sample fidelity and distribution coverage?

A2: Thank you for the valuable comment. Intuitively, LaBW does not directly optimize FID but uses latent covariance matching as a tractable proxy for distribution preservation. Nevertheless, we understand that this is motivated by the theoretical analysis of local FID sensitivity, and agree that complementary metrics are surely helpful to assess whether the benefits extend beyond FID.

As suggested, we have additionally evaluated all continual learning methods across all experimental settings using Precision, Recall, Density, and Coverage (PRDC) [a,b], which assess sample fidelity and distribution coverage. We report the results below. Across all ID/OOD settings, LaBW achieves higher ID recall and coverage than both baselines, with comparable or higher precision and density. It also achieves the highest values on all four metrics in class-incremental CIFAR-10. The OOD results show generally small differences, with LaBW scoring slightly below the baselines on some metrics and a more apparent trade-off across metrics in LSUN Churches →\rightarrow AFHQ with an LFM, where LaBW lies between GD and Rank-1 Fisher EWC on all four. These results are consistent with the ID and OOD FID scores and provide complementary evidence of improved preservation on the original domains. We have added the PRDC metrics to Sec. 5.1 (Lines 375-377) and included the full results in Appendix D.3.

ID/OOD Settings

SettingDomainMethodPrecisionRecallDensityCoverage
CIFAR-10 (ID/OOD)IDGD0.741 ±\pm 0.0020.631 ±\pm 0.0020.829 ±\pm 0.0090.742 ±\pm 0.002
IDEWC0.747 ±\pm 0.0040.633 ±\pm 0.0010.835 ±\pm 0.0120.752 ±\pm 0.003
IDLaBW0.747 ±\pm 0.0030.642 ±\pm 0.0040.846 ±\pm 0.0070.777 ±\pm 0.005
OODGD0.742 ±\pm 0.0050.648 ±\pm 0.0070.910 ±\pm 0.0150.871 ±\pm 0.001
OODEWC0.737 ±\pm 0.0040.650 ±\pm 0.0020.896 ±\pm 0.0150.863 ±\pm 0.002
OODLaBW0.738 ±\pm 0.0040.644 ±\pm 0.0030.915 ±\pm 0.0070.870 ±\pm 0.001
LSUN-Churches →\rightarrow AFHQ (LDM)IDGD0.769 ±\pm 0.0010.551 ±\pm 0.0041.042 ±\pm 0.0060.849 ±\pm 0.004
IDEWC0.770 ±\pm 0.0010.557 ±\pm 0.0041.043 ±\pm 0.0100.853 ±\pm 0.001
IDLaBW0.770 ±\pm 0.0020.566 ±\pm 0.0021.044 ±\pm 0.0100.865 ±\pm 0.002
OODGD0.460 ±\pm 0.0020.466 ±\pm 0.0050.233 ±\pm 0.0020.330 ±\pm 0.004
OODEWC0.461 ±\pm 0.0050.470 ±\pm 0.0100.231 ±\pm 0.0040.329 ±\pm 0.005
OODLaBW0.455 ±\pm 0.0020.467 ±\pm 0.0050.228 ±\pm 0.0040.330 ±\pm 0.004
CelebA-HQ →\rightarrow Flowers102IDGD0.811 ±\pm 0.0020.549 ±\pm 0.0041.207 ±\pm 0.0110.905 ±\pm 0.002
IDEWC0.809 ±\pm 0.0020.551 ±\pm 0.0041.205 ±\pm 0.0080.905 ±\pm 0.002
IDLaBW0.811 ±\pm 0.0010.555 ±\pm 0.0041.205 ±\pm 0.0150.907 ±\pm 0.001
OODGD0.642 ±\pm 0.0060.496 ±\pm 0.0040.595 ±\pm 0.0040.578 ±\pm 0.009
OODEWC0.640 ±\pm 0.0110.498 ±\pm 0.0050.590 ±\pm 0.0090.561 ±\pm 0.004
OODLaBW0.646 ±\pm 0.0060.494 ±\pm 0.0080.591 ±\pm 0.0080.572 ±\pm 0.004
LSUN-Churches →\rightarrow AFHQ (LFM)IDGD0.675 ±\pm 0.0020.585 ±\pm 0.0020.699 ±\pm 0.0070.751 ±\pm 0.004
IDEWC0.680 ±\pm 0.0000.608 ±\pm 0.0010.716 ±\pm 0.0020.773 ±\pm 0.002
IDLaBW0.688 ±\pm 0.0020.619 ±\pm 0.0020.733 ±\pm 0.0050.789 ±\pm 0.004
OODGD0.616 ±\pm 0.0060.693 ±\pm 0.0060.459 ±\pm 0.0110.600 ±\pm 0.006
OODEWC0.573 ±\pm 0.0050.755 ±\pm 0.0080.399 ±\pm 0.0040.560 ±\pm 0.001
OODLaBW0.597 ±\pm 0.0020.724 ±\pm 0.0050.434 ±\pm 0.0040.584 ±\pm 0.007

Class-Incremental CIFAR-10 Setting

MethodPrecisionRecallDensityCoverage
GD0.723 ±\pm 0.0000.580 ±\pm 0.0040.773 ±\pm 0.0040.697 ±\pm 0.002
EWC0.727 ±\pm 0.0010.582 ±\pm 0.0030.780 ±\pm 0.0050.694 ±\pm 0.003
LaBW0.733 ±\pm 0.0000.591 ±\pm 0.0010.809 ±\pm 0.0000.720 ±\pm 0.004

\rule{2cm}{0.4pt}

[a] Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020

[b] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019

Q3: Where is the definition of point-wise generative distillation loss?

A3: Thank you for pointing this out. From a high-level perspective, the point-wise GD loss aims to preserve the teacher’s reverse process by encouraging the student to match the teacher’s predictions at individual intermediate latent points.

Specifically, the loss is defined as:

\mathcal L_{\mathrm{GD}} = \mathbb E \left[ \left\\| f_\theta(z_t, t, y) - f_T(z_t, t, y) \right \\|_2^2 \right], \qquad z_t = a_t z_0 + \sigma_t \epsilon,

where the expectation is over replay samples (z0,y)∼BT(z_0, y) \sim \mathcal{B}_T, uniformly sampled timesteps tt, and independent noise ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0,I). fθf_\theta and fTf_T denote the student and frozen teacher predictions, respectively, in the model’s parameterization (velocity for flow matching and noise, score, or clean sample for diffusion).

In contrast, our proposed LaBW objective operates at the distribution level, matching the covariances of the teacher and student one-step endpoint predictions, which provides a complementary form of regularisation.

We have updated the manuscript to include the definition at the end of Sec. 4.2 (Lines 347-351).

Thanks for the the new metrics and revisions. They address my questions regarding metric hacking and completeness. I will maintain my score of 3.

Comments on the paper

2 comments

Dear Reviewers and Area Chair,

Thank you again for your time and thoughtful comments. We wanted to follow up on our responses and revised manuscript. With the discussion deadline approaching, we would appreciate your feedback on whether we have addressed your concerns. We would be happy to discuss any remaining questions and provide further clarifications.

Best regards,

The Authors

AuthorsOpenReview ↗

Overall summary of rebuttal

We thank the Area Chair and all three reviewers for their time and valuable feedback.

We are encouraged that the reviewers highlighted the theoretical analysis is “useful” (R-Th7z) and provides an “interesting intuition” (R-2RVa), and for highlighting the consistent improvements in forgetting across datasets and training seeds (R-2RVa, R-Th7z), as well as across generative processes, backbones and resolutions (R-t5Rt).


We incorporated the additional experiments into the revised manuscript and summarize the main discussion points below.

1. Theoretical

  • Low-variance subspace motivation (R-Th7z, R-t5Rt): We clarified that the low-variance projection is motivated by our theoretical analysis, with its effectiveness in reducing forgetting empirically established and validated [Abstract (Lines 025-032), contribution statement (Lines 115-119), and Sec. 3.2 (Lines 274-277)]. Overall, the theoretical analysis aim to provide insights and guidance on our methodology designs and enhance our understanding of "where forgetting is more likely to happen" in the latent space, supported by experimental results. The theory alone, however, is not a strict guarantee of versatile practical scenarios, as many other ML research in the field in general.

2. Empirical

  • Additional evaluation metrics (R-2RVa, R-t5Rt): We further evaluated the continual learning methods on precision, recall, density, and coverage (PRDC), with the results being consistent with the FID scores [Sec 5.1 (Lines 375-377) and Appendix D.3].

  • One-step and full-sampling covariance distance (R-Th7z, R-t5Rt): We added a diagnostic that empirically shows the connection between one-step and full-sampling covariance distances between teacher and student, supporting the effectiveness of one-step predictions as proxies [Sec. 5.2 (Lines 516-522) and Table 6].


Particularly, as we are approaching the end of the rebuttal, we want to note that the remaining concerns raised by R-t5Rt are (1) the positioning of theoretical analysis and (2) additional evaluation metrics. For the first, we hope we have clarified the role of theoretical analysis in the summary and in our detailed responses; for the second, this is the same question raised by R-2RVa, which we addressed with additional evaluations using PRDC metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.