- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies continual generation for latent diffusion and latent flow-matching models. LaBW constructs shared noisy latents from teacher replay samples, converts teacher and student outputs into one-step endpoint predictions, and matches their covariances with a Bures-Wasserstein loss in a low-variance subspace of the teacher latent covariance. The paper also analyzes the local relationship between small latent perturbations and Inception-feature FID.
Critical strengths
- Consistent results across generative formulations and architectures. LaBW reduces forgetting in four ID-to-OOD configurations and five-task CIFAR-10, covering LDMs/LFMs, U-Nets/DiTs, and 32/256 resolutions.
Critical weaknesses
- The theory does not directly justify the implemented LaBW loss. Theorem 1 asks how a small change in the final clean-latent distribution affects image FID after the decoder and Inception network. LaBW instead matches projected covariances of one-step predictions on teacher states. This leaves two gaps: (i) the paper does not show that latent-covariance BW approximates the theoretical feature-space FID sensitivity ; and (ii) it does not show that matching one-step predictions on teacher states controls the final distribution produced by the student’s full multi-step sampling process from noise. The method therefore regularizes projected second-order statistics of one-step predictions rather than demonstrably preserving the full endpoint distribution.
- The theory does not show that low-variance directions are always the most sensitive. The derived result is . Although favors low-variance directions, also varies with direction and can offset or reverse this ordering. The result also compares equal absolute variance changes, which impose larger relative changes on low-variance directions. The experiments likewise do not show universal superiority: the low-variance subspace is clearly better only on CIFAR-10, the full space is better on CelebA-HQ→Flowers102, and the variants are close on LSUN LDM. The low-variance choice should therefore be presented as an empirical heuristic rather than a theoretical conclusion.
Questions for authors
Could the authors provide a clearer explanation addressing the concerns I raised in the critical weaknesses section?
4 comments
W2: The theory does not show that low-variance directions are always the most sensitive. The derived result is . Although favors low-variance directions, also varies with direction and can offset or reverse this ordering. The result also compares equal absolute variance changes, which impose larger relative changes on low-variance directions. The experiments likewise do not show universal superiority: the low-variance subspace is clearly better only on CIFAR-10, the full space is better on CelebA-HQ→Flowers102, and the variants are close on LSUN LDM. The low-variance choice should therefore be presented as an empirical heuristic rather than a theoretical conclusion.
A2: Thank you for these observations. We would like to clarify that we have always considered the low-variance regularization as a design choice that is theoretically motivated and empirically supported, rather than a theoretical guarantee of reduced forgetting. Below, we clarify how the sensitivity analysis, the choice of perturbation scale, and the subspace ablations inform this interpretation.
Regarding the sensitivity ordering, the inverse-eigenvalue factor alone does not determine which directions are most sensitive. As discussed in Sec. 3.2, the remaining term depends on the latent distribution, the Jacobian of the decoder-Inception map, and the covariance of the Inception features. We therefore evaluate the combined effect of both factors empirically, finding increased sensitivity in low-variance directions (Figure 2a). This finding, together with the theoretical analysis, motivates prioritising these directions. We have revised the abstract (Lines 025-032), contribution statement (Lines 115-119), and Sec. 3.2 (Lines 274-277) of the manuscript to clarify this distinction.
Concerning the choice of perturbation scale, equal relative perturbations would also be a valid and informative diagnostic. However, there is no general reason to expect the variance changes induced by adaptation to be proportional to the teacher’s eigenvalues. We therefore used a common absolute perturbation scale to study which directions are more sensitive to a given variance change.
Finally, in the subspace ablations, low-variance matching improves both retention and adaptation on CIFAR-10, while the other two settings have comparable performance across variants. On CelebA-HQ Flowers102, full-space matching achieves a lower ID FID than low-variance matching (7.81 versus 7.89), but has a higher OOD FID (23.64 versus 23.44). We therefore interpret this as a trade-off rather than an overall advantage of full-space matching. We further examine these results in Appendix D.2, where Table 13 additionally shows that the variants achieve similar low-variance covariance preservation in these latter two settings. Since regularisation updates shared model parameters, its effects can extend beyond the selected subspace. Comparable forgetting is therefore consistent with comparable preservation of low-variance directions, even when the loss is applied elsewhere.
Thank you for the detailed response and experiments. These address several of my earlier questions. However, the theoretical connection to the implemented loss remains indirect and relies on restrictive assumptions, so the analysis is better viewed as motivation rather than a rigorous guarantee. Besides, both the theoretical discussion and evaluation focus heavily on FID. While this makes the results consistent, it gives a narrow view of distribution matching and generation quality, as FID can be easily hacked (https://arxiv.org/abs/2604.28190). I believe both the theory and evaluation could be strengthened, so I tend to maintain my original score.
Further responses to Reviewer t5Rt
Thank you for following up and for your feedback on our response. We are happy to further clarify the two remaining points from the reviewer.
1. Role of the theory, and its connection to the loss design.
-
Our overall positioning: We agree with your interpretation where the analysis provides the rationale for the loss design, while our empirical results support its effectiveness in reducing forgetting. This was also our intent in the original manuscript, where we noted after Proposition 2 that “the remaining term depends on the latent distribution, the decoder-Inception map, and the resulting FID geometry.” The sensitivity therefore depends on both the data and the model. We have made this distinction clearer in the revision.
-
Why the theory still matter: In the meanwhile, we believe that the theoretical analysis indeed matter because it tells us what to examine: changes of the same magnitude in different latent directions can have different effects after decoding. Our analysis quantifies this sensitivity, and our measurements show that FID is more sensitive to covariance changes in low-variance directions (Fig. 2). This finding guided our choice of which directions to preserve.
-
Connection to practical loss and its empirical effectiveness: The additional assumptions in Appendix C.4 are needed to establish a formal link between the latent loss and local FID sensitivity. However, the loss itself can be applied without assuming that the student’s distribution changes in this way during training. We tested it across four domain-transfer settings and class-incremental CIFAR-10, spanning latent diffusion and flow-matching models with U-Net and DiT backbones. LaBW consistently reduced forgetting relative to both baselines.
2. Evidence beyond FID.
Thank you for raising this, and we agree that FID alone gives an incomplete picture of generation quality. Although FID informed our analysis, LaBW does not directly optimize it as the training loss matches covariances in the autoencoder’s latent space.
Additional evaluation with more independent metrics: Reviewer 2RVa raised a similar question, and in response, we added precision, recall, density, and coverage [a,b] as four additional metrics across all settings (Appendix D.3, Tables 16–17). Across all four domain transfers, LaBW achieves higher recall and coverage on the original domain than both baselines, with comparable or higher precision and density. It also achieves the highest values on all four metrics on class-incremental CIFAR-10. These results provide evidence beyond FID that LaBW better preserves coverage of the original domain after adaptation.
[a] Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020
[b] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019
Thank you for your thoughtful comments and for acknowledging the consistent improvements across generative models and experimental settings. We appreciate the opportunity to further clarify and strengthen the theoretical justification of our approach.
A1: Thank you for the thoughtful insights. Our theoretical analysis characterizes the local sensitivity of feature-space FID to controlled perturbations of the latent distribution and motivates the design of the covariance regularisation. The effectiveness of the resulting objective in preserving the generated distribution during continual adaptation is evaluated empirically. Below, we clarify the connection between the analysis and the BW objective, and provide additional evidence relating one-step covariance preservation to that of fully sampled endpoints.
(i) Relating the latent BW loss to Q(V) requires accounting for how the decoder-Inception map transforms latent changes. BW measures a finite covariance distance, while Q(V) describes the local FID response along a given perturbation. A covariance change of a given magnitude can have different effects in feature space depending on this map, so a bound between the two quantities must account for that amplification.
We can make the relationship more precise using the same optimal affine transport as in Proposition 2. Given positive-definite teacher and student covariances ΣT and ΣS, it defines a displacement VBW whose squared magnitude satisfies E[∣VBW∣2]=dBW2(ΣT,ΣS). Under the regularity assumptions of Theorem 1 and assuming that the decoder-Inception map is L-Lipschitz, we can derive from the Gelbrich inequality:
Q(VBW)≤L2dBW2(ΣT,ΣS)
The affine transport does not necessarily represent the student’s distributional change, so this does not establish a general bound for arbitrary teacher-student distributions. It does, however, extend the construction in Proposition 2 to arbitrary positive-definite target covariances and connect their finite BW distance to the local FID sensitivity along the corresponding displacement. We have added the full statement, construction, and proof in Appendix C.4.
(ii) Matching the covariances of one-step endpoint predictions does not, by itself, guarantee the preservation of the full endpoint distribution, but it is empirically intuitive and our additional experiments support its usefulness for preserving the covariance after multi-step sampling. Please refer to our answer A2 to Reviewer Th7z for further discussion and experimental evidence.