acceptodds
Under review as a conference paper at ICLR 2027

Tail-Preserving Diffusion via Geometric Regularization

Abstract

Class-conditional diffusion models trained on long-tailed data often generate minority-class samples that drift toward head-class semantics. We study this failure through the distinction between conditional geometry preservation and marginal coverage control. Our analysis shows that positive class-only reweighting preserves the population-optimal conditional denoising target, although it can alter shared-parameter optimization. Sampling-prior adjustment changes the marginal mixture while leaving each conditional distribution fixed. Under a local manifold model, we further bound deterministic reverse-trajectory departure in terms of initial distance and normal velocity, motivating geometric control of semantic drift. Guided by this analysis, we propose Geometry-aware Tail-preserving Score Regularization (GTSR). GTSR retains the standard denoising objective and adds a tail-specific regularizer on denoised estimates in a frozen semantic representation. A real-sample memory bank and rank-adaptive kNN-PCA define a local class reference, and the regularizer penalizes the normal-displacement energy fraction as a practical geometric proxy. Low-noise activation selects estimates that support reliable comparison with this reference. Evaluation on CIFAR10-LT, CIFAR100-LT, and Tiny-ImageNet-LT shows improved tail conditional accuracy, conditional fidelity, and semantic isolation, with consistent gains under DDPM and EDM parameterizations. These findings support geometry-preserving training complemented by sampling-time coverage control.

Primary area:
generative models
Keywords:
Diffusion models, Long-tailed learning, Class-conditional generation, Geometric regularization, Semantic preservation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KmmPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes GTSR for training diffusion models on long-tailed datasets. GTSR aims to preserve tail-class semantics and prevent semantic drift toward other classes by penalizing the normal displacement of generated samples from the target tail-class subspace. The paper also presents theoretical analyses of rebalancing strategies and provides geometric motivation for the proposed regularization. Experiments are conducted on CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Critical strengths

The proposed regularization method appears intuitively effective in improving the quality of tail-class samples, although several issues with the evaluation need to be addressed. Overall, the paper is reasonably clear and understandable.

Critical weaknesses
  1. My major concern is the reported FID scores. I noticed unusually low FID scores on CIFAR-10, including tail-class, head-class, and overall FID. In particular, the overall FID of the vanilla DDPM reported in this paper is 1.44, which is highly unexpected, considering that the original DDPM achieves an FID of around 3.0 on balanced CIFAR-10. More importantly, the models in this paper are trained on severely long-tailed datasets. The reported FID is even substantially lower than that achieved by the original EDM trained on balanced CIFAR-10. Another related observation is that the vanilla DDPM results reported in this paper on CIFAR-10 and CIFAR-100 (1.44 / 3.78) are significantly better than those reported in the CORAL paper (6.17 / 7.70). How exactly are the FID scores computed in this work? I find these results particularly concerning, as they substantially outperform established baselines trained on balanced datasets. I believe the authors must carefully investigate and explain these discrepancies, especially the FID evaluation protocol.

  2. Even setting aside the concerns about FID computation, I find the cFID results reported in the main text somewhat misleading. According to the description in the paper, the reported metric should be cFIDtail\text{cFID}_\text{tail}, which should be explicitly indicated in the tables. Moreover, I do not think comparing only cFIDtail\text{cFID}_\text{tail} against competing methods such as CORAL and CCUA provides a sufficiently comprehensive evaluation. A method could potentially improve tail-class generation quality at the expense of head-class performance. Therefore, I believe overall FID should also be reported for all competing methods, as is commonly done in prior work.

  3. I am not sure why the authors maintain the memory banks and class prototypes through dynamic updates. It seems that these quantities could be precomputed before training. In particular, since the proposed regularization is applied only to tail classes, which contain relatively few training samples, such precomputation should be straightforward. Both the clean training samples and the feature extractor are fixed, so I do not see a clear necessity for dynamically updating these quantities. Could the authors clarify the motivation and benefits of this design choice?

  4. The theoretical analyses in Section 3 also appear rather elementary. Sections 3.1 and 3.2 do not seem to provide much new insight. In particular, regarding Section 3.2, I question whether adjusting the class-label sampling distribution at inference time to increase the frequency of tail-class outputs is a practically meaningful or commonly adopted strategy. Section 3.3, on the other hand, essentially establishes that if the initial point is sufficiently close to the data manifold and the normal component of the trajectory velocity is controlled, then the trajectory remains close to the manifold over a finite time interval. While the authors use this result to motivate the proposed regularization, the actual regularizer is applied to the denoised sample rather than the normal component of the trajectory velocity. The connection between these two quantities is unclear, and the theoretical result does not seem to directly justify the proposed regularization.

Questions for authors

I hope the authors can carefully address the concerns regarding the experimental evaluation. In particular, the unusually low FID scores require a thorough explanation. A more comprehensive comparison with existing methods would also be appreciated. Although I also have concerns about the theoretical analysis, I do not consider them the primary issues that need to be addressed at this stage. My main concern is the validity and reliability of the experimental results.

Reviewer_pNwVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses class-conditional diffusion models trained on long-tailed data, where samples requested for tail classes drift toward head-class semantics. It argues that positive class-only loss reweighting leaves the population-optimal conditional denoiser unchanged but alters finite-sample optimization through shared parameters, that sampling-time prior shift changes only the marginal mixture, and (Prop. 3.1) that bounding the normal velocity of a reverse trajectory keeps it near the class manifold. Guided by this, GTSR adds to the standard DDPM loss a tail-only, low-noise regularizer: the one-step denoised estimate is embedded with a frozen encoder, its displacement to the class prototype is decomposed against a rank-adaptive kNN-PCA subspace built from real same-class features in a memory bank, and the fraction of displacement energy in the normal subspace is penalized. On CIFAR10-LT, CIFAR100-LT and Tiny-ImageNet-LT (IR = 100), under DDPM (5 seeds) and EDM (3 seeds), GTSR reports the best tail conditional accuracy, tail cFID and Tail→Head leakage against Vanilla, Reweighting, Balanced Sampling, Prior Shift, CFG, CORAL and CCUA, with extensive diagnostics (leakage over training, normal-energy ratios, independent-evaluator leakage, encoder swaps, diversity, VLM-judged perceptual quality) and released code.

Critical strengths
  1. The framing that separates conditional geometry (to be protected at training time) from marginal coverage (to be adjusted at sampling time) is clean and well illustrated: Fig. 1b–c shows prior shift raising marginal tail coverage while leaving leakage unchanged, and Table 3 shows that combining GTSR with reweighting hurts, consistent with the paper's own argument.
  2. The evaluation apparatus is unusually thorough for the mechanism it tests: leakage is re-measured under independent evaluators (a separately trained classifier, a self-supervised nearest-prototype assignment, DINOv2 and CLIP; Tables 10–11) and the geometry encoder is swapped while the evaluator is held fixed (Table 24), with gains persisting; within-class diversity does not collapse (Table 9); seeds, compute and hyperparameters are documented.
Critical weaknesses
  1. The supplementary code does not implement the method as described, so the reported results cannot be verified against it. In gtsr/diffusion.py (l. 260–261) the denoised estimate is embedded under torch.no_grad(), and phi in gtsr/classifier.py (l. 160) is itself decorated @torch.no_grad(). I executed the authors' modules: x0_hat.requires_grad is True, phi_fn(x0_hat).requires_grad is False, geoN.requires_grad is False, and the gradient of ddpm_loss + λ_N·geoN with respect to the noise prediction is identical to that of ddpm_loss (max abs difference 0.0). As released, L_N is logged but contributes no gradient, contradicting p. 4 ("gradients through φ(x̂₀) to x̂₀ and the denoiser are retained"); a GTSR run updates the model exactly as a Vanilla run. Further discrepancies: the README states the notebook used the opposite gate (t ≥ t₀) and that low-noise gating was added "to match the paper description"; the rank-adaptive PCA of Eq. 10–12 is not implemented (classes with fewer than max(k, r+2) bank features are skipped, no reliability weight, no unique-sample tracking); neighbours are taken around the denoised query rather than the prototype; the "Reweighting" baseline weights only tail samples at t ≤ t₀ rather than by class; all methods default to a class-balanced sampler; and the default hyperparameters (λ_N = 0.01, k = 16, r = 4, 20 epochs, base width 64) differ from Tables 25–26. Only CIFAR10-LT/DDPM is covered. The README describes the code as a refactor of a notebook, so the results may come from different code, but as submitted the central mechanism is not reproducible.
  2. The headline metrics are measured in the feature space the regularizer optimizes, on generators that appear far from converged. All FID and cFID values are computed from the frozen ResNet-18 evaluation classifier's 512-d features (metrics.py, feature_stats_from_loader(…, phi_fn, …), with phi_fn from the same classifier); Inception features are never used and the main text does not state this, so numbers such as "overall FID 1.44" are not comparable to the literature and the cFID/TAcc gains partly measure the regularizer's own objective (the independent-evaluator controls cover leakage only, not FID or TAcc). Training budgets are 100–150 epochs at batch 128/64 on ≈11–12k images (Table 26), i.e. on the order of 10⁴ optimizer steps, far below standard class-conditional DDPM practice; consistently, tail conditional accuracy is only 0.15–0.17 on CIFAR10-LT while the evaluator reaches 91.6% on real tail images (Table 5), and Reweighting's leakage is still rising at the end of training (Fig. 1a). Against the strongest baseline, CCUA, the TAcc gains are within one standard deviation (0.1710 ± 0.0020 vs 0.1691 ± 0.0025; 0.1160 ± 0.0019 vs 0.1134 ± 0.0023), while the effect-size appendix (Tables 21–22) compares against Vanilla and CORAL instead.
Questions for authors

Clarifications that could change my assessment:

  1. Which code produced the reported numbers? Please release it, or correct the supplementary code so that L_N propagates gradients and matches the paper (gating direction, prototype-centred kNN, rank adaptation and reliability weights, class-only reweighting baseline, hyperparameters), and confirm that the reported results are reproduced by it.
  2. Please report FID and tail cFID with standard Inception features (and/or DINOv2 features), and TAcc under an evaluator independent of φ, and state in the main text which feature space each metric uses.
  3. Do the GTSR gains persist when the generators are trained to a standard budget (e.g., ≥ 200k steps)? Please also report effect sizes against CCUA rather than Vanilla.
  4. "Balanced Sampling (Qin et al., 2023)" appears to refer to CBDM, whose published method is a distribution-adjustment regularizer rather than balanced minibatches; is CBDM proper compared anywhere?

Suggestion (not affecting the score): Prop. 3.1 assumes the bound on the normal velocity that its conclusion requires, and the regularizer penalizes a static displacement ratio rather than a velocity; the paper would read better presenting it as motivation rather than theoretical support.

Reviewer_XziNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses image generation with conditioned diffusion models trained on imbalanced, long-tail datasets, where limited training samples make it hard to generate accurate, realistic images for minority classes. Traditional approaches, such as loss reweighting, address class imbalance but don’t effectively mitigate leakage toward the majority classes.
The paper proposes a regularizer that proxies off-manifold drift using a low-noise, tail-only feature space. This penalizes the fraction of displaced energy orthogonal to the estimated local subspace. The regularization is restricted to tail-class examples at low-noise time steps, where denoised estimates reliably measure class relevance. The paper has a clear gating mechanism based on activation time and the presence of a tail-class scenario. The experiments report tail-conditional accuracy, conditional FID, and leakage on several standard datasets.

Critical strengths

The problem formulation is well defined, and the performance evaluation metrics align well with the research problem being tested. Measures include tail conditional accuracy and tail-to-head leakage. The paper proposes a geometrically motivated regularization approach to reduce semantic drift and preserve tail-class/minority-class semantics. The theoretical formulation seems aligned with the stated purpose. The experimental results are promising, as the proposed GTSR method reduces semantic leakage compared with baseline methods.

Critical weaknesses

The ablation studies do not establish the distinct contribution of the proposed geometric constraint. Comparisons with other feature-based regularization methods are warranted to establish the regularizer’s clear contribution. More explanation is required on how the regularizer explicitly contributes to semantic drift mitigation. Given the nature of the claim, the authors need to provide more customized synthetic datasets with tailored long-tail distributions and compare GTSR's impact against the SOTA. The performance improvement on the real datasets used compared to competitive methods such as CORAL and CCUA is marginal. The aforementioned scrutiny should explicitly highlight the intended behavior. Furthermore, a statistical test with confidence intervals may be needed to ensure the improvement is meaningful. The leakage mitigation toward other proximal minority classes should be examined. A detailed report of the full class confusion matrix is required to ensure the objectives are met.

Questions for authors

Can synthetic datasets with more exaggerated long-tail distributions and multiple minority classes in close proximity more clearly show that the regularizer does its job? This has to be compared with competitive methods. Can the authors report confidence levels to ensure insignificant results aren't counted as performance improvements? Can the authors provide the full-class confusion distribution for the standard datasets used and the proposed synthetic datasets? Can the authors clarify how minimizing normalized feature displacement actually controls semantic drift? How does the actual energy realignment guarantee the intended purpose?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.