acceptodds
Under review as a conference paper at ICLR 2027

MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-conditions Image Synthesis

Abstract

Recent advances in text-to-image (T2I) generation have enabled controllable image synthesis by incorporating conditions beyond text. However, most existing diffusion-based methods are limited to a single type of control condition (e.g., bounding boxes or keypoints), which restricts their flexibility. To address this limitation, we propose MixDiffusion, a training-free diffusion framework for multi-condition T2I generation. MixDiffusion theoretically supports an arbitrary number of control conditions—including bounding boxes, keypoints, sketches, depth maps, reference images, and text—by collaboratively integrating multiple pre-trained uni-condition diffusion models. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-condition image generation model from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. Owing to its training-free nature, MixDiffusion is easy to deploy and readily extensible to new control modalities.

Primary area:
generative models
Keywords:
Image Synthesis, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XhDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method that combines multiple conditional generation models within the generation process of a single base model to achieve multimodal control. Multimodal fusion is done by summing the residuals from each control model. Qualitative and quantitative experiments are used to demonstrate its effectiveness.

Critical strengths

A training-free method for controllable text-to-image generation is proposed. This method adopted a base-corrected integration formula a decreasing integration intensity to balance structural guidance and image refinement. The experiment results seems to be acceptable.

Critical weaknesses
  1. Only structural control is covered. Almost all of the conditions the paper combines are essentially structural controls. Control over other visual features, such as complex texture or color, is not considered, which greatly limits the contribution of the paper.

  2. Correlated conditions may only increase guidance strength. Many of the conditions provide the same information: Canny edges, depth, and pose all reinforce structural information. These highly correlated conditions do not independently control different, often coupled attributes such as structure and color. With the paper’s summation, combining them is roughly equivalent to increasing the control strength. This raises doubts about the contribution:

Does the improvement simply come from a larger total guidance? Is multi-condition control actually better than single-condition control?

A comparison with a single condition at a matched (scaled-up) control weight would help answer this.

Questions for authors
  1. Diffusion models have an inherent trade-off between generation quality and condition control. Does adding more conditions lower quality or make generation unstable?

  2. Control information in diffusion models has a fairly clear division over timesteps; for example, early steps mainly determine structure [1]. Would adding different conditions at different timesteps work better than injecting all conditions at every timestep? Different layers also favor different attributes, and this should be considered as well. [1] Zhang, Yuxin, et al. “ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models.” ACM Transactions on Graphics (TOG) 42.6 (2023): 1–14.

2 comments
AuthorsOpenReview ↗

MixDiffusion composes both structural and appearance controls; its gains are not generally reducible to a larger scalar guidance weight. New results with text rendering, color, and style preserve control accuracy while improving semantic and perceptual quality.

We thank the reviewer for recognizing the value of training-free composition and the acceptable empirical results. The review raises useful questions about control coverage, guidance magnitude, generation quality, and temporal routing. We provide new quantitative evidence for non-structural controls and clarify what the current formulation establishes.

1. The framework is not limited to structural control.

Figure 1 and Appendix A.4 already demonstrate color palettes, style and appearance references, face appearance, and visual-text rendering, in addition to pose, depth, boxes, and edges. We further evaluated a text prompt (TT) with mutually distinct text-rendering (VtVt), color (CoCo), and style (SySy) controls:

SettingCLIP-Score ↑IR rating ↑Artifact ↓ACC.T ↑[email protected] ↑
TVtTVt35.673.873.2376.9095.93
TVtCoTVtCo35.984.093.1876.5595.58
TVtCoSyTVtCoSy36.034.083.1976.5395.48

Adding color and style controls to text rendering. ACC.T is rendered-text accuracy; [email protected] is visual-text location accuracy.

From TVtTVt to TVtCoSyTVtCoSy, CLIP-Score rises from 35.6735.67 to 36.0336.03, ImageReward from 3.873.87 to 4.084.08, and artifacts decrease from 3.233.23 to 3.193.19, while the two OCR accuracies change by only 0.370.37 and 0.450.45 percentage points. Thus, appearance controls improve semantic/perceptual quality without materially weakening the original text and location constraints. We will add these results and foreground the existing appearance-control gallery.

2. Multi-condition composition is not generally equivalent to stronger single-condition guidance.

Let ri,t=ϵi−ϵ0r_{i,t}=\epsilon_i-\epsilon_0. Scaling one controller changes only the direction rj,tr_{j,t}, whereas MixDiffusion uses ∑iri,t\sum_i r_{i,t}. They coincide only in the degenerate case where the combined residual is collinear with the single residual. Appendix A.4 shows the opposite case: pose controls a foreground person, while a Canny map without that pose controls the background. Text rendering, color, and style likewise constrain different attributes, so no scalar applied to one controller can reproduce their joint effect.

To separate magnitude from direction for correlated structural controls, we will add the suggested matched-norm baseline:

λt=∥∑iri,t∥2∥rj,t∥2.\lambda_t=\frac{\left\lVert\sum_i r_{i,t}\right\rVert_2}{\left\lVert r_{j,t}\right\rVert_2}.

We will compare ϵ0+wtλtrj,t\epsilon_0+w_t\lambda_t r_{j,t} with the scheduled multi-condition sum. This directly tests whether the gain persists after matching total guidance strength.

Q1: Quality and stability.

Adding valid, complementary conditions does not inherently reduce quality: the new table improves CLIP/ImageReward and slightly reduces artifacts, while Table 3 also reports increasing aesthetic scores. Its 0.570.57-point drop at the strict pose threshold after adding Canny shows that complementarity, not control count alone, governs the result. The decreasing schedule addresses accumulated control strength: Table 2 shows that retaining the full residual sum throughout denoising is consistently worse than scheduled integration.

Mutually contradictory controls are different: no image can satisfy incompatible constraints simultaneously, so they fall outside the valid input domain and may reduce both adherence and aesthetics. The current method does not detect such conflicts; we will add explicit failure cases and state this boundary.

Q2: Timestep- and layer-specific control.

Our current rule is already time-aware: wt=(t/T)2w_t=(t/T)^2 emphasizes control during early structural formation and returns later refinement to the base model. A natural extension is condition-specific routing:

ϵ^t=ϵ0+∑iwi(t)(ϵi−ϵ0).\hat{\epsilon}_t=\epsilon_0+\sum_i w_i(t)(\epsilon_i-\epsilon_0).

This could assign different active windows to structure, color, style, or text. ProSpect supports this motivation within a single network. MixDiffusion deliberately composes heterogeneous pretrained models at their outputs; layer-specific injection would require aligned internal architectures and sacrifice black-box compatibility. We will discuss per-condition schedules and layer routing as clear extensions, while retaining output-level composition as the training-free, architecture-agnostic scope of this work.

AuthorsOpenReview ↗

Re-posted results and revision plan

Thank you again for the thoughtful review. We noticed that the table in our previous comment did not render correctly, so we repost it below:

SettingCLIP ↑ImageReward ↑Artifact ↓ACC.T ↑[email protected] ↑
T + Vt35.673.873.2376.9095.93
T + Vt + Co35.984.093.1876.5595.58
T + Vt + Co + Sy36.034.083.1976.5395.48

Vt, Co, and Sy denote visual-text, color, and style controls; ACC.T and [email protected] measure text accuracy and localization.

Appendix A.4 contains extensive results using independently specified color palettes, style/appearance references with texture cues, face appearance, visual text, and spatial conditions. To make this evidence easier to assess, we will move representative examples into the main paper and organize them as controlled comparisons.

We will also clarify that “independent control” refers to branches that can be varied or replaced separately at inference, rather than statistical independence among all input maps. This distinction will be stated explicitly in the revision.

Reviewer_kB6nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Controllable T2I generation relies on auxiliary conditions (boxes, pose, depth, sketch, edges, appearance) beyond text. Pretrained uni-condition generators cover many such controls, but combining them for new condition sets usually needs dedicated fusion modules or joint retraining, which does not scale as combinations grow.

This paper proposes MixDiffusion, a training-free framework that composes multiple pretrained compatible uni-condition diffusion models during sampling. All branches share the latent trajectory, text encoder, VAE, and noise schedule; each predicts noise from its own auxiliary input, and predictions are fused before a shared reverse update. The core is a base-corrected integration rule derived from a product-of-experts view: add each branch's residual relative to the base model instead of averaging predictions (which shrinks residuals by 1/N). A decreasing integration intensity lets controllers dominate structure early and the base model refine later; the authors relate this to a tempered expert product and sketch a flow-matching extension.

Contributions. (1) A training-free framework for combining uni-condition T2I models, allowing new compatible controllers without per-combination retraining. (2) A base-corrected fusion formula with a decreasing schedule, giving a probabilistic interpretation under stated assumptions and a practical prediction-level interface. (3) Experiments showing flexibility and competitive control adherence, with ablations isolating the fusion rule and schedule.

Validation. Models include TextDiffuser V2, T2I-Adapter, GLIGEN, IP-Adapter, DreamShaper, and SD v1.5, with ControlNet baselines, evaluated on COCO2017 person images using box IoU/AP, pose OKS/AP, depth and Canny RMSE, and an aesthetic predictor. MixDiffusion achieves the lowest Canny error and highest pose [email protected], ranks second on depth, and is competitive overall. Ablations show base-corrected integration clearly beats prediction averaging and unattenuated integration on box AP, and removing the decreasing schedule lowers box AP in every group. Adding complementary conditions improves box AP and aesthetic score; pose [email protected] rises, though the stricter [email protected] drops slightly after adding Canny, a tradeoff the authors report. Qualitative comparisons against Uni-ControlNet and DiffBlender are provided.

Critical strengths
  1. The paper identifies a real and growing bottleneck and offers a genuinely plug-and-play answer.

The motivating problem is not contrived: users increasingly want several controls at once (e.g., pose plus appearance plus layout), and the existing options each impose a cost — either a new integration module per modality set (Uni-ControlNet, DiffBlender) or joint retraining when conditions change. MixDiffusion's contribution is consequential because it reframes the problem from "learn a fusion network" to "fuse predictions at sampling time," which means any compatible pretrained controller can be added or removed without touching a single parameter. This is the kind of modularity that matters in practice, where controllers are heterogeneous, released independently, and constantly extended. It also gives the method a clear operational advantage over learned-integration baselines: no per-combination training data, no modality-specific heads, and graceful degradation to the base model by setting the schedule to zero. For acceptance, this is the strongest argument — the paper addresses a concrete, scalable need with a mechanism that is trivially adoptable by anyone already running a diffusion pipeline.

  1. The base-corrected residual-sum rule is a small, well-motivated correction with outsized empirical effect.

The distinction between averaging conditional predictions and summing their residuals relative to the base is the paper's sharpest technical point. Averaging shrinks each residual by 1/N, so adding controllers can paradoxically weaken guidance; the product-of-experts derivation instead yields a residual sum, and the schedule then tempers it over time. The empirical gap is not marginal — on TBDSC, box [email protected] goes from 63.02% (averaging) and 65.42% (unattenuated) to 88.51%, and the same ordering holds across all three condition sets and both thresholds. This matters for my assessment because it shows the framework is not just a recombination convenience: the fusion rule itself carries real weight, the ablation isolates it cleanly, and the gain is large enough that practitioners would notice. Combined with the decreasing schedule, which is shown to matter in every condition group, the paper delivers a defensible "why this rule and not the obvious one" story rather than an unexamined engineering choice.

Critical weaknesses
  1. The theoretical derivation relies on assumptions that do not hold in the target setting, so the "mathematically derived" claim exceeds what is actually established.

The product-of-experts derivation rests on three assumptions: that the control conditions are conditionally independent, that the experts have equal variance, and that the posterior mode can stand in for a sample. None of these holds in practice. Depth, pose, sketch, and Canny are often extracted from the same reference image, so the conditions are correlated by construction; equal variance is a modeling convenience; and using the mode means the per-step update is not a true draw from the composed distribution, with no analysis of accumulated error. The authors acknowledge these limitations, which is commendable, but acknowledging them does not resolve them: the derivation provides a motivation for the rule rather than a justification of it. This matters because the derivation is precisely what distinguishes the method from an ordinary heuristic. If it only holds under assumptions the target setting violates, the "mathematically derived" framing overstates the contribution, and the core result is better described as a well-motivated heuristic. Supporting the theoretical claim as stated would require at least an empirical test of the assumptions — for example, examining how performance changes as the controls become more correlated.

  1. The evaluation conflates the number of controls with the method, so the headline gains cannot be cleanly attributed to MixDiffusion.

Table 3 shows that adding depth, sketch, and Canny raises box AP from 84.55% to 88.51% and the aesthetic score from 4.46 to 5.15, and the abstract and introduction lean on these numbers. However, the comparison pits single-control ControlNet baselines (TB, TP) against multi-control MixDiffusion rows (TBD, TBDS, TBDSC), changing two variables at once: the amount of conditioning and the integration method. The authors point readers to Table 2 for the controlled fusion comparison, but Table 2 only compares fusion variants on box AP; it does not separate how much of the Table 3 gain comes from supplying more information versus from MixDiffusion's fusion and scheduling. A reader therefore cannot tell whether a comparable multi-control baseline would have captured most of the improvement. This bears directly on the paper's practical claim — that complementary conditions yield better control and aesthetics through this framework. If the gains are largely explained by additional conditioning rather than by the framework's design, the contribution is narrower than advertised. Attributing the improvement to MixDiffusion would require a controlled comparison, such as a learned multi-control baseline trained on the same condition sets, or a naive concatenation of controls into a single conditioning network.

Questions for authors

Q1: In Table 3, TB/TP are single-control ControlNet baselines, whereas TBD/TBDS/TBDSC are multi-control MixDiffusion rows. Is there a baseline with a comparable number of controls that is not MixDiffusion — for example, concatenating multiple controls into a single ControlNet, or a multi-control model trained on the same condition sets? If such a baseline exists, please report its numbers. If not, how do the authors determine that the Table 3 gains are attributable mainly to MixDiffusion's fusion and scheduling rather than to the additional conditioning?

Q2: The derivation relies on conditional independence and equal-variance assumptions. Have the authors conducted at least one experiment probing the boundary of these assumptions — for instance, deliberately increasing correlation between controls (e.g., extracting depth and Canny from the same image) and measuring how performance changes as correlation grows? If quantitative results are available, please provide them; if not, to what extent do the authors think the "mathematically derived" framing should be softened?

2 comments
AuthorsOpenReview ↗

MixDiffusion is theoretically defined beyond equal-variance experts and is empirically effective with correlated controls. Table 2 isolates fusion and scheduling, whereas Table 3 measures the effect of adding complementary information; we will make this distinction and the boundary of conflicting inputs explicit.

We thank the reviewer for recognizing MixDiffusion's practical modularity, the importance of base-corrected fusion, and the substantial controlled gains in Table 2. The questions identify three points that we can state more precisely: the scope of the expert model, the distinct purposes of Tables 2 and 3, and the evidence for correlated versus different-source controls.

\textbf{1. Independence, unequal variances, and the role of the mode.} The assumption is conditional rather than marginal: Ci⊥Cj∣(e,zt,t,y)C_i\perp C_j\mid(e,z_t,t,y). Consequently, controls extracted from one image being marginally correlated does not by itself disprove the assumption, although exact conditional independence need not hold. The fusion rule is exact under the stated expert model and remains a well-defined prediction-level composition outside it. Importantly, correlation and contradiction are different: correlated, compatible controls may reinforce shared structure; mutually inconsistent controls specify no coherent target and are outside the method's admissible input domain.

The controls also need not share a source. Figure 1 and the Appendix A.4 gallery show cases in which pose specifies the foreground person while a Canny map without that pose specifies the background; both simultaneously determine the generated scene. Depth, pose, sketch, and Canny can likewise be drawn from different sources and assigned complementary foreground/background roles.

Each expert already has its own mean/mode ϵi\epsilon_i; only the variance was shared to present Eq.(7) compactly. For expert-specific variances, the corresponding mode is \begin{equation} \epsilon_{\mathrm{joint}}= \frac{\displaystyle\sum_{i=1}^{N}\frac{\epsilon_i}{\sigma_i^2}-(N-1)\frac{\epsilon_0}{\sigma_0^2}} {\displaystyle\sum_{i=1}^{N}\frac{1}{\sigma_i^2}-(N-1)\frac{1}{\sigma_0^2}}, \end{equation} provided the combined precision is positive; Eq.(7) is its equal-variance special case. We will add this statement to the diffusion derivation.

The Gaussian experts are auxiliary distributions in prediction space. A denoiser outputs a point noise prediction, not a sample that MixDiffusion replaces with a MAP draw. We compose these predictions and then apply the standard DDIM reverse update; any transition stochasticity remains in that sampler. We will state explicitly that the probabilistic result is an exact mode derivation under the expert assumptions, not a claim of exact ancestral sampling from the unknown joint data distribution. We will also add conflicting-condition failure cases and analyze the resulting residual cancellation or artifacts.

\textbf{2. Tables 2 and 3 answer different questions.} Table 2 is the controlled horizontal comparison: within each fixed condition set (TBD, TBDS, or TBDSC), it changes only the fusion strategy among averaging, unattenuated integration, and scheduled MixDiffusion. It therefore isolates fusion and scheduling; on TBDSC, [email protected] is 63.02%63.02\%, 65.42%65.42\%, and 88.51%88.51\%, respectively. Table 3 is a vertical condition-addition analysis: it asks how performance changes as complementary inputs are added. Its conclusion is that additional conditions improve adherence and aesthetics in the evaluated setting, not that all of this gain is caused by fusion. We will revise the abstract and discussion to preserve this attribution precisely.

\textbf{Q1.} As mentioned in the previous response.

\textbf{Q2.} All main quantitative experiments combine conditions extracted from the same COCO image, so they already evaluate the correlated-input regime highlighted in the question. Different-source combinations are currently demonstrated qualitatively in Figure 1 and Appendix A.4, not as a quantitative correlation sweep. We will make this scope explicit and add the aforementioned conflicting-control failures and applicability analysis.

Reviewer_kB6nOpenReview ↗

response of author rebuttal

Thank you to the authors for the rebuttal. The rebuttal has partially addressed my concerns, and I will consider raising my score.

Reviewer_xkzuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MixDiffusion, a training-free framework for combining multiple pretrained single-condition diffusion models for multi-condition image generation.

The method combines noise predictions from different conditional models by summing their residuals relative to a shared text-conditioned base model. A quadratic decay schedule is further applied to reduce the contribution of conditional guidance during denoising.

The authors provide a product-of-experts interpretation of the fusion rule and evaluate the approach on COCO2017 with multiple spatial control conditions.

Critical strengths
  1. Practical motivation. Combining existing pretrained conditional diffusion models without additional training is a useful capability for controllable image generation.

  2. Promising experimental results. The method demonstrates improvements in several spatial control metrics, particularly bounding-box alignment and pose accuracy. The presented results suggest that combining multiple pretrained controllers can be practically useful.

Critical weaknesses

Limited novelty: the proposed method is essentially multi-condition classifier-free guidance (CFG).

The core formulation of MixDiffusion is

[ \hat{\epsilon}_t = \epsilon_0 + w_t \sum_i (\epsilon_i-\epsilon_0), ]

which is mathematically equivalent to applying multiple conditional guidance directions relative to a shared base prediction, with a time-dependent guidance scale.

This is a straightforward generalization of classifier-free guidance to multiple conditional denoisers. The proposed quadratic schedule, (w_t=(t/T)^2), is also a relatively simple heuristic rather than a fundamentally new sampling mechanism.

Although the authors provide a product-of-experts derivation, similar formulations have already been explored in compositional diffusion and multi-condition guidance. I do not find the current presentation sufficiently distinguishes MixDiffusion from these existing approaches or establishes a substantial methodological contribution beyond multi-condition CFG.

  1. No practical reasoning for inference-time model mixing.

ControlNet default supports multiple encoders and multiple conditional inputs. And nowadays unified model can do more than this. The paper topic has also no practical reason to use...

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

MixDiffusion composes independently trained conditional generators, rather than multiple conditions inside one model. Its theory extends composition to unequal-variance Flow Matching, and its practical role is to reuse specialist models when joint training data are unavailable.

We thank the reviewer for recognizing the practical motivation, clear presentation, and strong improvements in bounding-box alignment and pose accuracy. The two concerns allow us to clarify the central contribution: MixDiffusion addresses model composition at inference time; it is not a relabeling of multi-condition CFG, nor does its novelty rest on the quadratic schedule alone.

  1. MixDiffusion is model-level composition, not multi-condition CFG. The identical residual notation does not imply an identical modeling setting. Standard multi-condition CFG combines guidance directions obtained by evaluating multiple conditions within one jointly parameterized denoiser. MixDiffusion instead composes distinct, independently trained uni-condition generators, ϵi=ϵθi(zt,t,y,Ci)\epsilon_i=\epsilon_{\theta_i}(z_t,t,y,C_i), each with its own parameters and condition processor, through a common text-conditioned base ϵ0\epsilon_0. No component has been trained on the joint condition tuple, and no fusion parameters are learned. Relative to generic prediction composition, our rule explicitly factors out the shared text-conditioned base so that it is not counted NN times while preserving each model's own processor.

Under the stated common-base and conditional-independence assumptions, every conditional expert already contains the base distribution, so its repeated contribution must be removed: \begin{equation} q(e\mid C_{1:N},z_t,t,y)\propto q_0(e)\prod_{i=1}^{N}\left(\frac{q_i(e)}{q_0(e)}\right)^{w_t}. \end{equation} The mode is \begin{equation} \hat{\epsilon}t=\epsilon_0+w_t\sum{i=1}^{N}(\epsilon_i-\epsilon_0). \end{equation} Thus, CFG supplies a familiar algebraic starting point, while MixDiffusion establishes when distributions represented by separately trained models can be composed. This also explains why prediction averaging is not the product-of-experts solution: it shrinks every controller residual by 1/N1/N. Our ablations isolate averaging, unattenuated integration, and scheduled integration; for TBDSC, scheduling raises box [email protected]/0.75 from 65.42%/55.28%65.42\%/55.28\% to 88.51%/73.36%88.51\%/73.36\%. The quadratic wt=(t/T)2w_t=(t/T)^2 is an intentionally parameter-free tempering rule: early steps establish joint structure and later steps restore the base model's role in refinement. The contribution is the composition framework and its derivation, not the polynomial in isolation.

The theory further covers Flow Matching velocity fields whose prediction-space Gaussian experts have distinct variances. Eq.~(10) gives \begin{equation} v_{\mathrm{joint}}= \frac{\sum_{i=1}^{N}v_i/\tau_i^2-(N-1)v_0/\tau_0^2} {\sum_{i=1}^{N}\tau_i^{-2}-(N-1)\tau_0^{-2}}, \end{equation} when the combined precision is positive. Equal variances recover the residual sum. This variance-dependent composition is not implied by applying a shared CFG scale and states the corresponding model-composition rule for pretrained generative flows.

  1. Inference-time mixing remains practically important. MixDiffusion complements, rather than replaces, unified multimodal generators. Unified models are compelling when all desired modalities, joint training data, and retraining resources are available. Multi-ControlNet likewise supports multiple compatible control branches around a shared backbone. Our target regime is different: specialist capabilities already exist in separately trained models; users request a new combination after training; or the original datasets and pipelines are inaccessible. MixDiffusion then reuses compatible frozen components at prediction level, without reconstructing a joint dataset or optimizing a new fusion module.

This setting is common for community and proprietary LoRAs, which are often released only as weights while their training data remain private. When such models share a compatible backbone, latent space, timestep convention, and prediction parameterization, MixDiffusion can compose their capabilities directly and approximate the joint behavior that union-data training would otherwise target. The same interface lets users add or replace a specialist as needs evolve. We will sharpen these distinctions from same-model multi-condition CFG and multi-ControlNet, foreground Eq.~(10), and make the inaccessible-data/private-model use case explicit in the revision.

AuthorsOpenReview ↗

Brief clarification of our contribution

Thank you again for the thoughtful review. We would like to add one concise clarification regarding our contribution. The connection to CFG is deliberate: CFG is the single-expert, equal-variance case of our tempered expert formulation, rather than its complete form. The general expert rule is

ϵjoint=∑i=1Nϵi/σi2−(N−1)ϵ0/σ02∑i=1N1/σi2−(N−1)/σ02,∑i=1Nσi−2−(N−1)σ0−2>0.\epsilon_{\mathrm{joint}}=\frac{\sum_{i=1}^{N}\epsilon_i/\sigma_i^2-(N-1)\epsilon_0/\sigma_0^2}{\sum_{i=1}^{N}1/\sigma_i^2-(N-1)/\sigma_0^2},\qquad \sum_{i=1}^{N}\sigma_i^{-2}-(N-1)\sigma_0^{-2}>0.

Each model may have its own variance. Equal variances recover the residual-sum expression; setting N=1N=1 with tempering weight ss yields

ϵCFG=ϵ0+s(ϵ1−ϵ0).\epsilon_{\mathrm{CFG}}=\epsilon_0+s(\epsilon_1-\epsilon_0). The Flow Matching extension retains the same unequal-variance structure after replacing (ϵi,σi)(\epsilon_i,\sigma_i) with (vi,τi)(v_i,\tau_i). Thus, our theoretical contribution is a base-corrected composition principle for both diffusion and flows, not the quadratic schedule alone.

We further instantiate and evaluate this principle as training-free cooperation among multiple independently trained, frozen models. This is particularly useful for evolving community or private-model ecosystems: compatible specialists created by different parties can augment a fixed foundation model even when their training data cannot be pooled. The general rule also suggests confidence-aware composition, where input- or timestep-dependent variances reduce an expert's influence precisely when it is unreliable.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We thank the reviewers for their thoughtful feedback. The discussion helped us clarify the distinctive model-composition setting of MixDiffusion, its theoretical foundation, and the experimental evidence supporting its design.

Q1. What is the distinct contribution of MixDiffusion?

A. MixDiffusion enables training-free composition of independently trained, frozen uni-condition generators. Each generator retains its own parameters and condition processor, while all branches operate on a shared latent trajectory. This differs from applying multiple conditions within one jointly parameterized denoiser: no model is trained on the joint condition tuple, and no additional fusion module is optimized.

The integrated prediction is ϵ^t=ϵ0+wt∑i=1N(ϵi−ϵ0)\hat{\epsilon} _t=\epsilon _0+w _t\sum _{i=1}^{N}(\epsilon _i-\epsilon _0) with wt=(t/T)2w _t=(t/T)^2.

The base-corrected residual formulation prevents the shared text-conditioned base from being counted repeatedly. The decreasing schedule emphasizes control during early structure formation and restores the base model’s refinement ability during later steps.

This design is particularly useful when specialist models are developed independently or their original training data are unavailable. Compatible community or proprietary models can be combined, replaced, or extended without reconstructing a joint dataset or retraining the system.

Q2. What theoretical foundation supports the integration rule?

A. We derive the rule from a product-of-experts formulation. Under a shared base distribution and conditional independence given the prediction, latent, timestep, and text, each conditional expert contains the common base contribution. Dividing out its repeated contribution yields the base-corrected residual sum used by MixDiffusion.

The formulation also supports expert-specific variances, allowing each model’s contribution to reflect its uncertainty. Equal variances recover the main residual-sum rule. The same prediction-space construction naturally extends from diffusion noise predictions to flow-matching velocity fields, providing a unified composition principle for compatible pretrained generative models.

Importantly, the main COCO experiments already combine correlated conditions extracted from the same images, demonstrating that the resulting integration rule remains effective in realistic multi-control settings.

Q3. What evidence isolates the benefit of the proposed fusion strategy?

A. Table 2 compares fusion strategies while keeping each condition set fixed. For TBDSC, prediction averaging achieves 63.02%/50.90% box [email protected]/0.75, unattenuated integration achieves 65.42%/55.28%, and the complete MixDiffusion method reaches 88.51%/73.36%. The same ordering holds across all evaluated condition sets and both thresholds.

These controlled comparisons demonstrate two complementary effects:

  1. Base-corrected residual fusion preserves each controller’s contribution more effectively than prediction averaging.
  2. The decreasing integration schedule substantially improves over applying the full residual sum throughout denoising.

Table 3 further shows the benefit of incorporating complementary information. From TB to TBDSC, box [email protected] improves from 84.55% to 88.51%, box [email protected] from 69.86% to 73.36%, and the aesthetic score from 4.46 to 5.15.

Against prior multi-control methods, MixDiffusion obtains the best Canny RMSE (90.32) and pose [email protected] (86.03%), together with competitive depth performance (31.21 RMSE).

Q4. Does MixDiffusion extend beyond structural controls?

A. Yes. Figure 1 and Appendix A.4 combine spatial controls with color palettes, styles, appearance references, faces, and visual text. We additionally evaluated visual-text (Vt), color (Co), and style (Sy) controls quantitatively.

From T+Vt to T+Vt+Co+Sy:

  • CLIP-Score improves from 35.67 to 36.03;
  • ImageReward improves from 3.87 to 4.08;
  • artifact score decreases from 3.23 to 3.19;
  • text accuracy remains stable at 76.90→76.53;
  • localization accuracy remains stable at 95.93→95.48.

These results show that appearance controls improve semantic and perceptual quality while preserving the original text-rendering and localization constraints.

The benefit also comes from combining different residual directions rather than simply increasing a scalar guidance weight. With ri,t=ϵi−ϵ0r _{i,t}=\epsilon _i-\epsilon _0, MixDiffusion integrates ∑iri,t\sum _i r _{i,t}, allowing different experts to control complementary attributes such as foreground pose, background edges, color, style, and rendered text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.