acceptodds
Under review as a conference paper at ICLR 2027

Anchored Manifold Alignment Stabilizes Bilevel Refinement for Long-tailed Diffusion

Abstract

Diffusion models often suffer from degraded tail category generation when pro cessing long-tail distribution data due to extreme sample imbalance. Although Bilevel Optimization has recently been introduced for downstream task align ment in diffusion models, existing frameworks are predominantly designed for training from scratch. This approach readily induces calibration drift when fine tuning pre-calibrated models, undermining established tail category semantics. This paper proposes a phased decoupled fine-tuning framework specifically de signed for long-tail diffusion models. The core innovation lies in introducing the Static Pool Resampling (SPR) mechanism: during upper-layer objective evalua tion, resampling from a fixed anchor sample pool generated in the pre-calibration stage (Stage 1) transforms unconstrained sampling optimization into manifold constrained distribution feature alignment. Theoretical and experimental evidence demonstrates that this "resampled proxy evaluation" maintains statistical consis tency with hyper-gradient properties even under extreme long-tail distributions, ensuring both task-specific alignment and distributional integrity. This discovery provides crucial technical support for diffusion model fine-tuning. Results across multiple benchmark long-tail datasets demonstrate that our method enhances task performance while effectively mitigating calibration bias, validating its robustness and superiority under complex long-tail distributions. The code is provided in the supplementary material.

Primary area:
generative models
Keywords:
Diffusion Models, Long-tailed Image Synthesis, Generative Bilevel Optimization, Calibration Drift, Static Pool Resampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8uPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Decoupled Generative Refinement (DGR), a two-stage training and fine-tuning framework for long-tailed diffusion models. Stage 1 trains a model using Oriented Calibration (OC-LT) to improve tail-class generation through head-to-tail knowledge transfer and establish a calibrated initialization. Stage 2 fine-tunes this model using Static Pool Resampling (SPR), which reuses fixed, class-balanced seed–label pairs for upper-level evaluation and combines zeroth-order objective-gradient estimates with lower-level denoising gradients. The intended goal is to improve generation quality while preserving the tail-class semantics established in Stage 1. Validation includes FID/IS comparisons on CIFAR-10-LT, CIFAR-100-LT, and TinyImageNet200-LT, alongside class-wise diagnostics and unseen-seed evaluations.

Critical strengths

1.The experiment shows improvements over the same Stage-1 initialization across different datasets prove the proposed refinement is useful.

2.Evaluations on unseen seeds, class-wise and oracle-based diagnostics, also go beyond aggregate FID and attempt to assess whether improvements generalize beyond the fixed evaluation pool.

Critical weaknesses

1.Table 1 does not show consistent state-of-the-art performance: CORAL (2025) outperforms DGR on four of the six metrics. The more recent relevant CCUA [1] is absent from the comparisons.

2.The evaluation is also limited to CIFAR and TinyImageNet with a U-Net backbone. However, CCUA includes ImageNet-LT at 256×256 resolution with DiT. Evaluation at this scale and with a Transformer-based backbone would be important to show whether DGR’s benefits are limit in low-resolution U-Net settings.

3.The interpretation of Figure 5 overstates the tail-specific benefit: the accompanying text on page 9 claims that oracle accuracy improves most clearly for the Tail group, but the values printed on the bars show gains of 1.2 percentage points for Head (0.927→0.939), 1.6 for Medium (0.883→0.899), and only 0.9 for Tail (0.872→0.881).

Reference

[1] Chen, F., Villa, A., Liang, G., Fuxin, L., Lu, X., Tang, M. (2026). Contrastive Conditional–Unconditional Alignment for Long-Tailed Diffusion Model. In: Favaro, P., Kukelova, Z., Maki, A., Rohrbach, A., Schindler, K., Tombari, F. (eds) Computer Vision – ECCV 2026. ECCV 2026. Lecture Notes in Computer Science, vol 17022. Springer, Cham. https://doi.org/10.1007/978-3-032-37520-9_23

Questions for authors

see weaknesses

2 comments

Response to Reviewer 8uPk

Thank you for recognizing the value of comparing refinements from the same calibrated model and for checking the values in Fig. 5.

1. Extreme imbalance and comparisons with CORAL and CCUA

On CIFAR-10-LT at IF=1000, both refinement variants start from the same T2H checkpoint, trained at IF=100 and then adapted to IF=1000 (FID 7.88). Table 1 reports 10.05 for online refinement. The evolved SPR image pool scores 6.86, 12.9% below the Stage-1 value. That percentage describes the scored pool. Figure 4(b) separately tests the final generators on uniformly sampled labels and the same unseen seeds. Most classes favor SPR under extreme imbalance, with exceptions.

We agree that CORAL leads in four of the six Table 1 entries. At IF=1000 our evolved pool records 6.86 FID and 9.72 IS, compared with CORAL's published 11.03 and 9.13. We will identify the source of each score, the warm start and the evolved-pool evaluation in the table. This preserves the evidence for refinement under extreme imbalance while making the differences in evaluation protocol visible.

We will also add CCUA's published CIFAR results. Its Table 7 reports DDIM-100 FIDs of 5.56 on CIFAR-10-LT and 6.24 on CIFAR-100-LT. These U-Net results are directly relevant related work. We will specify each paper's training and scoring protocol alongside the scores so readers can distinguish cross-paper benchmarks from our shared-checkpoint comparison. Chen et al., CCUA, Table 7

2. Does the benefit transfer to a diffusion Transformer?

CCUA demonstrates its method on ImageNet-LT at 256 × 256 with SiT. Our evidence for DGR comes from U-Nets on CIFAR-LT and TinyImageNet200-LT. A direct comparison with CCUA's ImageNet-LT numbers changes the architecture, dataset, resolution and training history together. The experiment that would answer your question is to take one calibrated Transformer checkpoint and compare SPR with online refinement on ImageNet-LT using the same budget and final sampling protocol. We will state that DGR's benefit in this setting remains to be tested. Chen et al., CCUA, Table 6

3. Corrected interpretation of Fig. 5

You are right about the ordering. We will replace the sentence on page 9 with the following text. “Compared with unconstrained refinement, SPR increases oracle accuracy in all three groups. Medium has the largest gain, from 88.3% to 89.9% (+1.6 percentage points), followed by Head, from 92.7% to 93.9% (+1.2 points). Tail improves from 87.2% to 88.1% (+0.9 points).” The tail result matters because these classes have the fewest training examples. The figure supports improved tail semantic fidelity in this comparison, with the largest measured improvement in Medium.

Thank the authors't response. My concern about Figure 5 has been resolved. However, my concerns about the experiment remain:

  1. Adding CCUA’s published CIFAR results broadens the baseline comparison, but does not resolve the concern that DGR does not demonstrate consistent state-of-the-art performance.

  2. Concern about DGR's benefit is limited to a U-Net backbone remains. The authors provide no experiments on Transformer-based diffusion models and acknowledge that DGR’s benefits in this setting remain to be tested.

Therefore I will maintain my score.

Reviewer_kEFeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies task-driven refinement of pre-calibrated long-tailed diffusion models. It proposes Decoupled Generative Refinement (DGR), where Stage 1 uses OC-LT/T2H for long-tail calibration and Stage 2 evaluates an upper-level objective on a fixed class-balanced pool of 50K seed-label pairs. A two-sided zeroth-order estimate on this pool is fused with the score-matching gradient. The paper argues that Static Pool Resampling (SPR) anchors the refinement trajectory to the Stage-1 manifold and alleviates calibration drift. Experiments on CIFAR-10-LT, CIFAR-100-LT, and TinyImageNet200-LT show improvements over the authors’ T2H initialization and over an unanchored variant.

Critical strengths
  • The problem is relevant: downstream alignment can plausibly damage minority-class semantics of a calibrated long-tailed generator.

  • The two-stage design is intuitive and easy to integrate with an existing long-tailed diffusion model.

Critical weaknesses
  • The claimed manifold anchoring is not present in the stated objective. Eq. (4) defines Stage-1 anchor images xk0x_k^0, but Eqs. (6) and (10) only reuse (zk,yk)(z_k,y_k) and regenerate samples using the current model. There is no distance, feature matching, KL, score distillation, or trajectory-consistency term involving xk0x_k^0. Fixed inputs reduce Monte Carlo noise, but do not by themselves constrain the current generator to remain close to the Stage-1 manifold. The paper should describe its contribution more accurately as "fixed-seed / sample-average upper-level optimization," or alternatively, explicitly incorporate and validate an anchoring regularization term centered on xk0x_k^0.

  • All primary experiments utilize low-resolution, small-scale U-Net architectures. The TinyImageNet evaluation is limited to a single small table, four baselines, one experimental setting, and lacks results for CORAL, variance analysis, or fine-grained performance. The current evidence is insufficient to support claims regarding "scalability" or applicability to larger, general-purpose diffusion models; indeed, the authors themselves acknowledge in the "Limitations" section that the approach has not yet been validated for latent diffusion models or DiTs (Diffusion Transformers).

  • The theory does not match the implemented method. The main experiments set wk=1w_k=1, whereas Eq. (8) and Theorems B.1–B.2 require the density ratio pbal/qp_{bal}/q. A class-balanced label marginal does not imply equality between the generated conditional distribution and the desired balanced data distribution. Hence the claimed unbiasedness and consistency do not directly apply to the reported algorithm.

  • The method fixes the SPR pool size to KK=50K across all experiments, but provides no ablation on how performance, gradient stability, or computational cost varies with KK. This is particularly important because the theoretical claims explicitly depend on KK, while the required pool coverage may scale with the number of classes and the complexity of the data distribution. Without comparisons across smaller and larger pools, it remains unclear whether 50K is necessary, whether the reported gains are robust to this choice, and what accuracy–compute–memory trade-off SPR introduces.

  • The empirical comparison lacks statistical and fairness controls. Results mix literature values with the authors’ runs, and the IF=1000 T2H baseline is obtained by continued training from an IF=100 checkpoint. There are no independent-run statistics, standard deviations, confidence intervals, or significance tests. Several IS improvements are only 0.02–0.06.

Questions for authors
  • Real-world applications of generative models often involve high-resolution images and videos. Would caching 50K samples introduce substantial storage and efficiency overhead?

  • What is the exact upper-level objective, batch size, number of ZO directions, and total computational cost?

  • How does the update in Eqs. (10)–(11) correspond to the bilevel problem in Eq. (5)?

  • Can the authors provide CRN-only, refreshed-pool, pool-size, pool-distribution, and explicit anchoring baselines with multiple independent runs?

  • Please clarify the decreasing IS in Fig. 9(a) and correct the caption/figure-number inconsistencies.

2 comments

Response to Reviewer kEFe

Thank you for asking what stays fixed in SPR. Each pool entry keeps its initial noise and label. When sampled, its cached image is replaced by the current generator's output.

1. Storage and scale. For 50,000 entries at 32 × 32, the float32 RGB images and noise tensors plus int64 labels occupy 1.23 GB before serialization. The pool stays in host memory. Only selected inputs go to the GPU. At 256 × 256 the same pixel-space format would require 78.64 GB, increasing linearly with frame count for video. On TinyImageNet200-LT, FID fell from 17.81 to 16.90 in one U-Net setting. We lack a matched CORAL run, repeated runs and group-wise results there. High-resolution and Transformer settings remain untested.

2. Objective and cost. We minimize minibatch FID against fixed real-feature statistics. The CIFAR code computes these from the official test split during refinement. Final scoring uses a separate dataset-specific reference file. The documented CIFAR-10-LT IF=100 run used an upper-level batch of 128 and 36 images for previews. One U-Net update and one scheduler update each use one Gaussian direction with positive and negative perturbations (scale 0.001). Together they require four perturbed batch-FID evaluations and two unperturbed batch generations with 100-step DDIM. The logged training loop took 2.40 hours, excluding setup, pool construction and the final save. Initial construction adds about two hours per dataset. Subsequent refinement takes about as long as the online control.

3. Bilevel update and theory. Let L denote denoising loss, F the upper objective and V the minimum of L over the denoiser for a given scheduler. The penalty F/γ + L − V motivates Eqs. (10)–(11). The U-Net update adds the denoising gradient to the two-sided upper direction divided by γ. The scheduler also uses the difference between lower-loss partial derivatives at the two inner solutions. The code uses finite inner steps and normalized two-sided estimates on EMA parameters. The U-Net branch re-noises an already noised input, and the scheduler draws its target noise separately. These updates approximate Eq. (5). The documented run used γ = 5.

B.1 gives unbiasedness for the weighted per-sample surrogate at fixed parameters, in expectation over pool construction. Equality with the original gradient at Stage 1 requires E[(w − 1)h] = 0, where h is the per-seed reward gradient. Equal class counts do not establish this. We will revise §3.5 and B.1–B.3 to distinguish surrogate bias, finite-pool error and minibatch variability. The implementation uses unit weights and batch FID. In an earlier IF=100 sweep, scalar settings 0, 1, 1.5 and 2 yielded FIDs of 7.76, 6.18, 7.53 and 7.68. The records do not identify where the scalar entered relative to normalization, so this sweep cannot validate the density ratio in Eq. (8).

4. Controls and anchoring. Both variants share the checkpoint, refinement budget and noise–label inputs within each perturbation pair (CRN). Online draws fresh long-tailed inputs. SPR draws without replacement from a shuffled, class-balanced pool until a traversal ends. At IF=100, Stage-1 FID was 7.62. Online scored 9.67 on images regenerated from its final checkpoint. SPR scored 6.18 on the evolved pool, where unvisited entries retain Stage-1 images. Fig. 4 compares both final generators on the same unseen seeds and uniform labels. The class centroids and recall in Fig. 7 support our semantic interpretation of manifold anchoring. Tail oracle accuracy rises from 87.2% (online) to 88.1% (SPR) in Fig. 5. Fixed inputs alone give no distance bound.

We used K = 50,000 and one training run per setting. Balanced online sampling, periodic pool refresh, fixed long-tailed pools, other pool sizes and explicit anchor penalties remain untested separately. IS gains of 0.02–0.06 have unknown between-run variation. Evolved-pool FIDs were 18.9%, 12.9% and 16.0% lower than Stage 1 across the CIFAR settings. Both IF=1000 variants share a T2H checkpoint trained at IF=100 and adapted to IF=1000. Table 1 will identify this initialization, image sources and published baselines.

5. Fig. 9(a) and captions. Fig. 9(a) tracks additional refinement relative to the Table 1 baseline of 6.18. Across the plotted steps, full-pool FID falls from about 6.46 to 6.00 and IS from 9.72 to 9.49. Only FID enters the upper objective. The red band shows the standard deviation of IS. We will remove the claim of reduced FID variance. In Fig. 7(b), the horizontal axis is class index and the curves denote training steps. We will correct the caption and §A.3, use panel labels in Figs. 9–10, and correct §B.3 to cite Fig. 2(a) for gradient-direction similarity and Fig. 3 for the loss landscapes.

Thank you for the clarifications and candid acknowledgment of the limitations. However, the lack of key ablations—particularly on pool size and balanced online sampling—and repeated runs, together with the differing evaluation protocols, leaves the main claims insufficiently supported. I therefore maintain my Weak Reject recommendation.

Reviewer_ZHFaOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Decoupled Generative Refinement (DGR) for fine-tuning diffusion models trained on long-tailed data. The authors first use oriented calibration to obtain a pre-calibrated model. They then introduce Static Pool Resampling (SPR) during bilevel refinement. The main idea is to reuse a fixed pool of samples and seed-label pairs from the first stage to stabilize upper-level optimization. The paper also provides theoretical results on gradient estimation and stability. Numerical results show improvements in generation quality and several measures of tail-class performance.

Critical strengths

The problem considered in this paper is important. Fine-tuning a diffusion model on imbalanced data may improve generation quality while damaging the representation of minority classes. The authors propose a two-stage procedure to reduce this problem. I find the use of a fixed reference pool during refinement interesting. The method is also relatively easy to understand.

The numerical results show some improvements. Table 1 (p. 6) shows that DGR improves the FID of the DDPM+T2H baseline. The improvements are especially noticeable for CIFAR-10-LT with an imbalance ratio of 0.001. The comparison with the method without SPR is also useful. The authors evaluate generation with unseen random seeds in Subsection 4.4 (pp. 8-9). The class-wise results and semantic fidelity analysis provide additional evidence about minority-class generation.

Critical weaknesses

I have two main concerns.

My first concern is with the theory. In Subsection 3.4 (p. 4), SPR reuses fixed seed-label pairs during optimization. However, the generated samples still depend on the updated model parameters. I am not sure why fixing the seeds should keep the refined model close to the Stage-1 manifold. The paper does not impose an explicit constraint on the distance between the current samples and the original anchor samples. I also have a concern about Theorem B.1 (pp. 18-19). The proof changes the expectation from the Stage-1 distribution to the desired balanced distribution. However, it then identifies the resulting gradient with the gradient of the original alignment objective. This equality does not seem to hold without additional assumptions. I am not convinced that the claimed gradient consistency follows from the proof.

My second concern is with the numerical experiments. Table 1 (p. 6) shows that SPR performs better than the unanchored method. However, the comparison does not clearly separate the effects of fixed sampling, class balancing, and the size of the anchor pool. The main experiments use 50,000 samples with balanced seed-label pairs. I am not sure whether similar improvements could be obtained using class-balanced online sampling or fixed random seeds without the proposed manifold interpretation. The main tables also do not report results from multiple independent training runs. Without these results, I am not sure how stable the improvements are. The computational cost of the zeroth-order updates is also unclear.

Questions for authors

My questions are based on the comments for the critical weaknesses.

(1) In Subsection 3.4 (p. 4), SPR uses fixed seed-label pairs, but the generated samples change as the model is updated. How does this ensure that the refined model remains close to the original calibrated manifold? Can the authors provide a direct measure of the distance between the Stage-1 and Stage-2 distributions during training?

(2) Can the authors clarify Theorem B.1 (pp. 18-19)? The proof uses importance weights to change the expectation from the Stage-1 distribution to the desired balanced distribution. However, it is unclear why the resulting gradient should equal the gradient of the original alignment objective. I would also like to know how this theoretical result applies to the main experiments, where the importance weights are set to one.

(3) Can the authors compare SPR with class-balanced online sampling and fixed-seed sampling without class balancing? I would also like to see results for different anchor-pool sizes and several independent runs. Finally, how many zeroth-order evaluations are required per update, and what is the additional training time compared with the unanchored method?

1 comment

Response to Reviewer ZHFa

Thank you for recognizing the value of the class-wise and unseen-seed results. Your observation about the change of measure in B.1 identifies a missing condition in our proof.

1. What anchors refinement, and have we measured the distance to Stage 1?

Stage 1 supplies the calibrated generator and a pool of 50,000 noise–label pairs with equal counts per class. Each entry initially holds its Stage-1 image. Stage 2 shuffles the entries and takes successive minibatches without replacement until the iterator is exhausted. An update uses the selected noise and labels and writes the newly generated images back to those indices. Individual minibatches may have different class counts. The pool membership stays fixed, while the images evolve. The objective includes no distance penalty to the initial image.

We use manifold anchoring to describe the empirical preservation of calibrated class semantics during refinement. Figure 7(a) shows the displacement of class centroids in feature space from Stage 1 to Stage 2, and Fig. 7(b) tracks recall during refinement. In the complementary comparison in Fig. 5, tail oracle accuracy is 87.2% with online refinement and 88.1% with SPR. We did not measure a distance between the full Stage-1 and Stage-2 output distributions throughout training. We will distinguish the observed semantic behavior from a geometric guarantee, which fixed seeds alone cannot provide.

2. Corrected statement of Theorem B.1 and its relation to unit weights

Fix the Stage-1 parameters and freeze the corresponding importance weight w(ε) for each seed–label pair ε. Write r(φ, ε) for its pathwise reward and define the surrogate F(φ) = E[w(ε) · r(φ, ε)]. For any fixed φ chosen independently of the pool, the average of the weighted pathwise gradients over a randomly constructed pool is unbiased for the gradient of F, provided the pool represents the stated seed–label sampling law in expectation and differentiation can pass under the expectation. This follows by taking the expectation of the sample average with the weights held fixed.

For the original generator objective J(φ) = E[r(φ, ε)], let h(ε) be the pathwise reward gradient at the Stage-1 parameters. At that point the surrogate gradient minus the original gradient is E[(w(ε) − 1) · h(ε)]. The change of measure alone does not show that this term vanishes. We will qualify B.1 and the corresponding claim in Section 3.5. In B.2, a bound relative to the original gradient must account for both finite-pool sampling error and the difference between the population gradients. In B.3, fixing the pool fixes its empirical objective. Changing the minibatch still introduces variability into the updates.

The implementation uses unit weights and evaluates FID jointly over a batch. A balanced label marginal leaves the generated class-conditional distributions free to differ from their targets. B.1 therefore gives no unbiasedness guarantee for our normalized batch-FID update. In an earlier CIFAR-10-LT (IF=100) sweep, scalar settings of 0, 1, 1.5 and 2 gave FIDs of 7.76, 6.18, 7.53 and 7.68. The historical record does not establish where this scalar entered relative to gradient normalization. We report the sweep as a descriptive comparison. It does not test the per-sample density ratio in Eq. (8).

3. Sampling controls, pool size, independent runs, and cost

Sampling controls. On CIFAR-10-LT at IF=100, Stage-1 FID was 7.62. Online refinement with fresh noise and long-tailed labels scored 9.67 on images regenerated from its final checkpoint. SPR with a fixed, class-balanced pool scored 6.18 on the evolved pool. Both variants used the same starting checkpoint and refinement budget and shared noise and labels within each perturbation pair. We did not run separate controls for class-balanced online sampling or for fixed-seed sampling with long-tailed labels.

Pool size and runs. We used one pool size (K = 50,000) and one training run per setting. We therefore have no pool-size sweep or between-run standard deviation to report.

Cost. Each U-Net or scheduler update requires two perturbed batch-FID evaluations along one Gaussian direction. A cycle that updates both blocks requires four perturbed evaluations and two additional unperturbed batch generations. In the logged IF=100 run, the training loop took 2.40 hours at batch size 128. This timing excludes setup, initial pool construction and the final pool save. Relative to online refinement, SPR adds about two hours per dataset for initial pool construction. Subsequent refinement time is comparable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.