acceptodds
Under review as a conference paper at ICLR 2027

Couple to Control: Joint Initial Noise Design in Diffusion Models

Abstract

Diffusion models typically generate batches from independent Gaussian initial noises. We argue that independence is only one choice within a broader class of valid joint noise designs. One can instead specify a coupling: each noise remains marginally standard Gaussian, while the dependence across samples is chosen by design. This reframes initial-noise control from selecting individual seeds to designing a multi-sample gallery's joint distribution. The framework covers familiar constructions and supports direct sampling, structured initialization for noise optimization, and amortized learning of couplings. Repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, SD3, Qwen-Image, and FLUX.1-schnell with negligible initialization overhead. On its own, it matches or outperforms several recent test-time approaches for diverse generation on several diversity metrics at lower computational cost. Because it modifies only the initial noise, it can also be layered on top of contrastive noise optimization, particle guidance, and condition-annealed sampling to further improve pixel and perceptual diversity. We further extend the same design principle to video generation, where whole-latent coupling also yields significant gains in cross-video visual diversity. Code is available at https://anonymous.4open.science/r/diffusion-coupling-CE03/.

Primary area:
generative models
Keywords:
diffusion model, initial noise, coupling, diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zDcTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies joint initial-noise design for diffusion models. It proposes a low-cost repulsive coupling that introduces negative correlations among samples while preserving each noise sample’s standard Gaussian marginal distribution. The authors evaluate its effect on gallery-level diversity across image and video generation, examine its combination with existing diversity methods, and conduct ablations on gallery size and coupling strength.

Critical strengths

Strengths

  • The direct repulsive-coupling method requires neither model training nor test-time noise optimization and adds negligible computational overhead.
  • By preserving each noise sample’s Gaussian marginal distribution, the method can be applied to pretrained diffusion models without modifying their sampling procedures.
  • Experiments across image and video generation show improvements in gallery-level pixel and perceptual diversity while largely maintaining output quality and prompt alignment.
  • Coupled initialization is complementary to existing diversity methods and yields further gains when combined with them.
Critical weaknesses

Weakness

  • Although the paper evaluates pixel- and feature-space diversity, it remains unclear whether these gains translate into meaningful diversity for users. The exploratory multimodal-judge study shows only a modest preference for coupled galleries; a larger human evaluation would strengthen this claim.

  • The gains diminish substantially as gallery size KK increases. At K=32K=32, the pixel-level improvement is small, while the perceptual and feature-space gains are inconclusive, limiting the method’s utility for larger candidate sets.

  • The theoretical results establish maximal repulsion in noise space and increased separation for linear features, but do not guarantee greater perceptual or semantic diversity after a nonlinear generator. A clearer characterization of when noise-space repulsion transfers to output diversity would strengthen the theoretical motivation.

Questions for authors

See the weakness

3 comments

Thank you for the careful review and for recognizing the method's simplicity and empirical gains!

Summary of our changes:

  1. We agree that a user study is important for assessing whether the gains are meaningful to people. We therefore completed a blinded study of perceived diversity and best-image preference.

  2. We clarified how gallery size shapes the diversity gains. We strengthened the connection between input and output diversity with a sufficient condition and supporting experiments on neural features.

1. Human judgments of diversity and best-image choice

Thanks for the suggestion! We conducted a blinded SD3 study with 30 participants, each comparing coupled and IID galleries for ten prompts at K=2,3. They separately judged diversity and best-image choice, with ties allowed, yielding 300 judgments per question.

QuestionCoupled share of non-tied choicesAdvantage over IID
Diversity70.2%40.3 percentage points
Best image56.1%12.1 percentage points

These results show a strong diversity preference, with the same direction at both gallery sizes, and a modest best-image preference. We will do our best to recruit more participants and extend this evaluation.

Please see Section 5.2 and Appendix D.12 (blue text), including Table 23, for details and complete counts.

2. Larger galleries

We agree that the gains diminish as K increases. This is consistent with maximal repulsion's pairwise correlation of −1/(K−1), whose magnitude decreases with K. Our experiments at K=16,32 confirm this trend. Nevertheless, we believe small K is important in practice. For example, image-generation software often presents users with a small set of candidate images to compare and choose from. Meanwhile, prior work likewise evaluates small galleries: CNO uses K=3 or 5, and Particle Guidance uses K=4 for image generation.

3. How input diversity transfers to output diversity

Good point! We have added a theoretical guarantee and supporting experiments to explain when diversity in the initial noise carries through to generated features.

The intuition is as follows. Assuming the generated features are linear in the initial noise, then repulsion increases their expected squared distance by a known factor. For nonlinear features, diversity gain can be similarly explained by “how linear the feature is”.

Our existing analysis in Appendix A.7 (Proposition A.4) gives an exact expression for how noise correlation changes squared feature distance. The result separates linear and higher-order effects. We now derive a sufficient condition that can be directly estimated from data.

New sufficient condition. We use a linear predictor from noise to generated features and compare its prediction error with the total variance. If the error is small enough, Corollary A.5 guarantees an improvement in diversity under repulsion. The generator and feature extractor can both be nonlinear.

Supporting experiment. We fit the predictor offline, then evaluate the condition and output distances using 256 independent noise pairs (K=2) on a frozen CelebA-HQ model. The table reports ALADIN at c=−0.1 and −1, and DINOv2 at c=−1, relative to IID (c=0).

Squared-distance gain over IIDALADIN (c=−0.1)ALADIN (c=−1)DINOv2 (c=−1)
Predicted lower bound (estimated)+0.95%−64.56%−94.07%
Measured gain+2.60%+14.38%+9.65%

Interpretation.

  1. In all three reported settings, measured gains exceed the estimated lower bounds, consistent with the theory.
  2. The c=−0.1 case is interesting. The input noises are only weakly negatively correlated, yet the condition still gives a positive estimated lower bound for ALADIN (+0.95%). The observed gain (+2.60%) confirms the predicted improvement. This shows how the condition can identify increased output diversity even close to independent sampling.

Please see Appendix A.7 (blue text), including Proposition A.4, Corollary A.5, and Figure 4, for the analysis and experiments.

We hope these additional results clarify when and how diversity in the initial noise translates into diversity in the generated outputs.

Dear Reviewer zDcT,

Thank you again for your thoughtful review. We hope our response, including the new blinded human study, has addressed your questions about perceived diversity, why the gains diminish with larger galleries, and when repulsion in noise space increases output diversity. We would greatly appreciate any further thoughts you may have.

Best, Authors

Dear Reviewer zDcT,

We would like to kindly follow up, as the discussion period will close in less than a few hours. We hope our responses have addressed your concerns, and we would be happy to clarify any remaining questions in the time left. If you find our responses satisfactory, we would greatly appreciate it if you could take them into account in your final evaluation.

Thank you!

Authors

Reviewer_43HGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes designing the joint distribution (coupling) of initial noises for a batch of diffusion samples. Each noise stays marginally standard Gaussian, so single-sample quality is unchanged. The main instance is a repulsive Gaussian coupling with pairwise correlation. The authors prove that it is minimax-optimal for pairwise correlation and optimal for linear-feature diversity. Experiments show improved gallery diversity, additional gains when combined with cno, particle guidance, and cads, and an application to fixed-object background generation

Critical strengths
  • Nearly free and complementary to existing methods. The method only applies a linear transform to the initial noises. It needs no change to the model, sampler, or guidance. It also consistently improves CNO, PG, and CADS when used as their initialization (Table 2). This gives it clear practical value.
  • Theoretical guarantee on quality, with careful evaluation. The marginals are preserved, so expected single-image quality does not change (Prop. 3.2). The paired confidence intervals and BH correction support this claim well.
Critical weaknesses
  • Limited novelty over Jia et al. (2026a). Algorithm 1 already appears in Jia et al. (2026a, Sec. 4), as the authors acknowledge. That paper also covers diversity gains, partial noise negation, stepwise negation for stochastic samplers, RQMC, and integration with noise optimization as future work. This paper mainly extends the application and evaluation of an existing construction, and the related-work section does not make this difference clear.
  • The “general framework” is not shown to be useful. A coupling learned for diversity converges to the existing repulsive construction (B.1.5). The only other example is a simple brightness split (B.2). The paper should show a useful coupling that the repulsive construction cannot achieve.
Questions for authors
  • What is the key new contribution beyond Jia et al. (2026a)? In particular, how does this work relate to the partial negation (B.4.5), stepwise negation for stochastic samplers (B.4.1), and RQMC designs in that paper?
  • Is there a useful gallery objective that a learned coupling achieves but the repulsive construction does not?
  • How does the method compare with RQMC (e.g., scrambled Sobol) or Latin hypercube sampling in terms of diversity?
  • Can the authors perform a user study of diversity and human preference.
  • Is it possible to have couplings that control semantic diversity (e.g., objects or styles).
4 comments

Thank you for the detailed review and for recognizing the method's efficiency and complementarity!

Summary of changes: We clarified our contributions beyond Jia et al. in terms of the general coupling framework, theory, complementarity and image/video evaluation, and additional applications. We added structured and learned coupling experiments and a direct comparison with scrambled-Sobol RQMC.

To directly evaluate perceived diversity and best-image preference, we conducted a blinded human study with 30 participants.

All these changes are highlighted in blue in the revised paper, and we specify their locations in the detailed response below.

1. Contribution beyond Jia et al.

Thank you for raising this important point! We have clarified our contributions beyond Jia et al. in the following aspects:

  1. Framework and coupling designs. We formulate gallery control through joint initial-noise distributions with fixed Gaussian marginals. Jia et al.'s antithetic and K-sample repulsive constructions are special cases. The framework supports additional subspace-dependent and learned couplings, including learned shared subspaces, as shown in the experiments described below.
  2. Theory. We establish feasibility and marginal preservation, and optimality for worst-pair correlation and linear-feature diversity. We further characterize effects on smooth gallery objectives and nonlinear feature distances.
  3. Complementarity and image/video evaluation. We systematically evaluate multi-sample diversity across image and video models and show additional gains when combined with CNO, Particle Guidance, and CADS. Jia et al. place greater emphasis on negative output correlations and uncertainty quantification.
  4. Additional applications. In the revised version, we demonstrate applications such as semantic grouping, palette consistency, coordinated product collections, and consistent photographic appearance with varied content.
  5. Partial/stepwise negation, RQMC: Partial negation is a fixed subspace coupling that belongs to our family of noise couplings, stepwise negation couples the noises injected during stochastic sampling, and RQMC is an alternative noise design that we compare directly in Appendix D.7.1 (Table 19).

We agree that our contributions beyond Jia et al. should be stated more clearly. We have therefore rewritten the contribution paragraph in the Introduction. The responses below describe the new experiments and results.

2. Useful learned and structured couplings

We added experiments on two learned couplings for different gallery objectives.

In the first, the goal is four images forming two classes, with two distinct instances per class. Such groups are useful for comparing variants within a category and alternatives across categories. We learn noise correlations using a frozen ResNet classifier for grouping and LPIPS for within-class variation.

InitializationSuccessful CIFAR10 galleries
IID0/128
Repulsive1/128
Learned coupling25/128

In the second, the goal is a coherent portrait series with varied people and compositions. We learn shared noise directions using ALADIN for appearance agreement and DINO for content variation.

ModelALADIN distance reduction vs IIDDINO distance retained vs IID
SD1.514.4%95.2%
SD35.1%97.3%

These examples demonstrate useful gallery control: organizing alternatives into groups and giving a series a consistent appearance. Both couplings are fitted offline and reused during generation, with no additional diffusion calls.

There are other objectives for which structured couplings achieve significantly better results than repulsive coupling. For example, we added an experiment on creating collections with consistent color palettes and varied image content. On SD1.5, spectral coupling reduces color spread by 50.5% relative to repulsion. We added another experiment on creating two product collections with matching colors within each and different palettes across them. In this SDXL experiment, we generate four images forming two tea sets, each containing a teapot and a teacup with matching glazes, while the two sets use different palettes.

Please see Sections 3.3–3.4 for the formulation and Section 5.7 and Appendices E.1–E.4 (Tables 24–27) for the experiments. 1/2

3. Direct RQMC comparison

We added experiments comparing repulsive coupling with scrambled-Sobol RQMC under matched SD1.5 settings.

Gallery size KPixel L2 gain over RQMCLPIPS comparison with RQMC
2+6.9%Less than 0.5% difference
3+4.6%+1.7%
4+1.6%Less than 0.5% difference

Repulsion consistently improves pixel L2, while the LPIPS difference depends on gallery size.

Please see Section 5.1 and Appendix D.7.1 (Table 19) for details.

4. Human evaluation

Thanks for the suggestion! We conducted a blinded SD3 study with 30 participants, each comparing coupled and IID galleries for ten prompts at K=2,3. They separately judged diversity and best-image choice, with ties allowed, yielding 300 judgments per question.

QuestionCoupled share of non-tied choicesAdvantage over IID
Diversity70.2%40.3 percentage points
Best image56.1%12.1 percentage points

These results show a strong diversity preference, with the same direction at both gallery sizes, and a modest best-image preference. We will do our best to recruit more participants and extend this evaluation.

Please see Section 5.2 and Appendix D.12 (blue text), including Table 23, for details and complete counts.

We appreciate all your helpful suggestions. We hope our revisions clarify our contributions and that the human evaluation provides further evidence of the perceptual benefits of coupling. We’d love to hear your thoughts! 2/2

Dear Reviewer 43HG,

Thank you again for your thoughtful review. In the revision, we added experiments showing that learned couplings can group images by category or maintain a consistent style while varying their content. We also added a direct comparison with scrambled-Sobol RQMC and a blinded human study of diversity and image preference.

We hope these additions clarify what our main contribution and address the questions you raised. We would greatly appreciate any further thoughts.

Best,
Authors

Dear Reviewer 43HG,

We would like to kindly follow up, as the discussion period will close in less than a few hours. We hope our responses have addressed your concerns, and we would be happy to clarify any remaining questions in the time left. If you find our responses satisfactory, we would greatly appreciate it if you could take them into account in your final evaluation.

Thank you!

Authors

Reviewer_JmgeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a framework for controlling diversity in diffusion models by designing joint distributions of initial Gaussian noise samples while preserving their individual Gaussian marginals. The authors propose a simple repulsive coupling strategy that encourages diversity across generated samples without additional training or inference overhead. The approach is supported by theoretical analysis and experiments across several image and video diffusion models, demonstrating improved diversity and compatibility with existing sampling techniques.

Critical strengths
  • The paper presents a simple and elegant central idea: designing the joint distribution of initial Gaussian noises to improve diversity across generated samples, while preserving their individual Gaussian marginals.
  • The method is extremely efficient, requiring only a simple transformation of the initial noise, without additional training or inference overhead.
  • The paper presents strong experimental results across several diffusion models, consistently improving diversity while maintaining image quality and prompt alignment.
  • I liked the general formulation of noise coupling, supported by theoretical analysis, and its potential to enable more structured control over generated samples.
Critical weaknesses
  • The main repulsive Gaussian coupling construction builds on prior work (Jia et al., 2026a), which somewhat limits the novelty of the central algorithm. The broader coupling framework is interesting, but its practical contributions could be better established.
  • While the proposed method demonstrates improvements in pixel and perceptual diversity, the gains in semantic diversity are less convincing. A more extensive evaluation with more visual results or perhaps a small user study would help establish whether the generated results exhibit meaningful differences beyond low-level visual variations.
  • The proposed general coupling framework is not thoroughly explored experimentally across different objectives. Most results rely on the predefined repulsive construction, while learned couplings and structured control are demonstrated through relatively simple objectives, such as brightness clustering.
  • The discussion of related work could be more comprehensive, particularly regarding noise optimization and Gaussianity preservation. For instance, Chang et al. (CVPR 2026) investigate the statistical properties of optimized noise and propose a Gaussianity-preserving regularizer, which is relevant to the discussion in Section 3.4, where the marginal-preservation guarantees no longer necessarily hold after optimization.
Questions for authors
  • The general coupling framework seems promising, but its learned variants are demonstrated mainly through relatively simple objectives. Could the authors discuss its potential for controlling more complex semantic attributes? Could the method successfully optimize for aesthetic score or other neural-based metrics?
3 comments

Thank you for the thoughtful review and for recognizing the simplicity and efficiency of our approach!

Summary of changes: We added fixed structured couplings for palette consistency and product collections, and learned couplings using classification and appearance features. We clarified the role of Gaussianity regularization in seed refinement.

To directly evaluate perceived diversity and best-image preference, we conducted a blinded human study with 30 participants.

All these changes are highlighted in blue in the revised paper, and we specify their locations in the detailed response below.

1. Structured couplings beyond repulsion

We added two experiments on structured gallery control.

In the first, the goal is to match color palettes while retaining variation in image content. We encourage agreement in selected noise frequency bands while allowing the others to vary. On SD1.5, this reduces color spread by 45.1% while retaining 96.3% of IID DINO content distance.

In the second, the goal is to organize four images into two product collections of two images each, with matching colors within each collection and different palettes across collections. For example, a designer may want two tea sets, each containing a teapot and a teacup with matching glazes. We share low-frequency noise within each set and apply repulsion across sets. On SDXL, we report color distance between collections minus color distance within them (higher is better).

InitializationColor-distance gap ↑
IID−1.168
Repulsive−0.970
Grouped coupling1.451

Figure 3 illustrates the two tea sets.

These results show that simple structured couplings can support gallery objectives requiring both attribute consistency and variation.

Please refer to Section 5.7 and Appendices E.2–E.3, Tables 25–26, for details.

2. Learned couplings for semantic and appearance control

We added two experiments on learned couplings for different gallery objectives.

In the first, the goal is to generate four images in two classes, with two distinct instances per class. We learn noise correlations using a frozen ResNet classifier for grouping and LPIPS for within-class variation.

InitializationSuccessful CIFAR10 galleries
IID0/128
Repulsive1/128
Learned coupling25/128

In the second, the goal is consistent photographic appearance with varied people and compositions, as in a portrait series. We learn shared noise directions using ALADIN for appearance agreement and DINO for content variation.

ModelALADIN distance reduction vs IIDDINO distance retained vs IID
SD1.514.4%95.2%
SD35.1%97.3%

It is worth mentioning that the coupling is fixed once learned, and therefore generation does not introduce additional cost comparing with i.i.d. sampling.

Our current results show that learned couplings indeed have potential to control semantic, appearance, and neural-based metrics. A more comprehensive experimental study of these couplings’ applications and limitations is an important direction for future work.

Please see Sections 3.3–3.4 for the formulation and Section 5.7 and Appendices E.1 and E.4 (Tables 24 and 27) for the experiments.

Thank you for this helpful suggestion!

3. Human judgments of diversity and best-image choice

Thanks for the suggestion! We conducted a blinded SD3 study with 30 participants, each comparing coupled and IID galleries for ten prompts at K=2,3. They separately judged diversity and best-image choice, with ties allowed, yielding 300 judgments per question.

QuestionCoupled share of non-tied choicesAdvantage over IID
Diversity70.2%40.3 percentage points
Best image56.1%12.1 percentage points

These results show a strong diversity preference, with the same direction at both gallery sizes, and a modest best-image preference. We will do our best to recruit more participants and extend this evaluation.

Please see Section 5.2 and Appendix D.12 (blue text), including Table 23, for details and complete counts.

4. Chang et al. and Gaussianity

Thanks for pointing out this paper! Chang et al. propose a regularizer that encourages optimized latents to match Gaussian statistics. It can be used when refining the seeds of a coupled gallery. Our framework also offers another route: learn the coupling parameters offline, with Gaussian marginals preserved by construction, then reuse the learned coupling on new draws or prompts.

We added their work in Section 2 and explain both optimization routes in Section 3.4.

Thank you again for the helpful suggestions. We hope these additions clarify the framework's practical value. We welcome your feedback.

I'm satisfied with the proposed changes, therefore I'm keeping my acceptance score.

Thank you for your positive assessment and constructive feedback! We appreciate your helpful suggestions to improve our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.