- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free method that combines multiple conditional generation models within the generation process of a single base model to achieve multimodal control. Multimodal fusion is done by summing the residuals from each control model. Qualitative and quantitative experiments are used to demonstrate its effectiveness.
Critical strengths
A training-free method for controllable text-to-image generation is proposed. This method adopted a base-corrected integration formula a decreasing integration intensity to balance structural guidance and image refinement. The experiment results seems to be acceptable.
Critical weaknesses
-
Only structural control is covered. Almost all of the conditions the paper combines are essentially structural controls. Control over other visual features, such as complex texture or color, is not considered, which greatly limits the contribution of the paper.
-
Correlated conditions may only increase guidance strength. Many of the conditions provide the same information: Canny edges, depth, and pose all reinforce structural information. These highly correlated conditions do not independently control different, often coupled attributes such as structure and color. With the paper’s summation, combining them is roughly equivalent to increasing the control strength. This raises doubts about the contribution:
Does the improvement simply come from a larger total guidance? Is multi-condition control actually better than single-condition control?
A comparison with a single condition at a matched (scaled-up) control weight would help answer this.
Questions for authors
-
Diffusion models have an inherent trade-off between generation quality and condition control. Does adding more conditions lower quality or make generation unstable?
-
Control information in diffusion models has a fairly clear division over timesteps; for example, early steps mainly determine structure [1]. Would adding different conditions at different timesteps work better than injecting all conditions at every timestep? Different layers also favor different attributes, and this should be considered as well. [1] Zhang, Yuxin, et al. “ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models.” ACM Transactions on Graphics (TOG) 42.6 (2023): 1–14.
2 comments
Re-posted results and revision plan
Thank you again for the thoughtful review. We noticed that the table in our previous comment did not render correctly, so we repost it below:
| Setting | CLIP ↑ | ImageReward ↑ | Artifact ↓ | ACC.T ↑ | [email protected] ↑ |
|---|---|---|---|---|---|
| T + Vt | 35.67 | 3.87 | 3.23 | 76.90 | 95.93 |
| T + Vt + Co | 35.98 | 4.09 | 3.18 | 76.55 | 95.58 |
| T + Vt + Co + Sy | 36.03 | 4.08 | 3.19 | 76.53 | 95.48 |
Vt, Co, and Sy denote visual-text, color, and style controls; ACC.T and [email protected] measure text accuracy and localization.
Appendix A.4 contains extensive results using independently specified color palettes, style/appearance references with texture cues, face appearance, visual text, and spatial conditions. To make this evidence easier to assess, we will move representative examples into the main paper and organize them as controlled comparisons.
We will also clarify that “independent control” refers to branches that can be varied or replaced separately at inference, rather than statistical independence among all input maps. This distinction will be stated explicitly in the revision.
MixDiffusion composes both structural and appearance controls; its gains are not generally reducible to a larger scalar guidance weight. New results with text rendering, color, and style preserve control accuracy while improving semantic and perceptual quality.
We thank the reviewer for recognizing the value of training-free composition and the acceptable empirical results. The review raises useful questions about control coverage, guidance magnitude, generation quality, and temporal routing. We provide new quantitative evidence for non-structural controls and clarify what the current formulation establishes.
1. The framework is not limited to structural control.
Figure 1 and Appendix A.4 already demonstrate color palettes, style and appearance references, face appearance, and visual-text rendering, in addition to pose, depth, boxes, and edges. We further evaluated a text prompt (T) with mutually distinct text-rendering (Vt), color (Co), and style (Sy) controls:
Adding color and style controls to text rendering. ACC.T is rendered-text accuracy; [email protected] is visual-text location accuracy.
From TVt to TVtCoSy, CLIP-Score rises from 35.67 to 36.03, ImageReward from 3.87 to 4.08, and artifacts decrease from 3.23 to 3.19, while the two OCR accuracies change by only 0.37 and 0.45 percentage points. Thus, appearance controls improve semantic/perceptual quality without materially weakening the original text and location constraints. We will add these results and foreground the existing appearance-control gallery.
2. Multi-condition composition is not generally equivalent to stronger single-condition guidance.
Let ri,t=ϵi−ϵ0. Scaling one controller changes only the direction rj,t, whereas MixDiffusion uses ∑iri,t. They coincide only in the degenerate case where the combined residual is collinear with the single residual. Appendix A.4 shows the opposite case: pose controls a foreground person, while a Canny map without that pose controls the background. Text rendering, color, and style likewise constrain different attributes, so no scalar applied to one controller can reproduce their joint effect.
To separate magnitude from direction for correlated structural controls, we will add the suggested matched-norm baseline:
λt=∥rj,t∥2∥∑iri,t∥2.
We will compare ϵ0+wtλtrj,t with the scheduled multi-condition sum. This directly tests whether the gain persists after matching total guidance strength.
Q1: Quality and stability.
Adding valid, complementary conditions does not inherently reduce quality: the new table improves CLIP/ImageReward and slightly reduces artifacts, while Table 3 also reports increasing aesthetic scores. Its 0.57-point drop at the strict pose threshold after adding Canny shows that complementarity, not control count alone, governs the result. The decreasing schedule addresses accumulated control strength: Table 2 shows that retaining the full residual sum throughout denoising is consistently worse than scheduled integration.
Mutually contradictory controls are different: no image can satisfy incompatible constraints simultaneously, so they fall outside the valid input domain and may reduce both adherence and aesthetics. The current method does not detect such conflicts; we will add explicit failure cases and state this boundary.
Q2: Timestep- and layer-specific control.
Our current rule is already time-aware: wt=(t/T)2 emphasizes control during early structural formation and returns later refinement to the base model. A natural extension is condition-specific routing:
ϵ^t=ϵ0+∑iwi(t)(ϵi−ϵ0).
This could assign different active windows to structure, color, style, or text. ProSpect supports this motivation within a single network. MixDiffusion deliberately composes heterogeneous pretrained models at their outputs; layer-specific injection would require aligned internal architectures and sacrifice black-box compatibility. We will discuss per-condition schedules and layer routing as clear extensions, while retaining output-level composition as the training-free, architecture-agnostic scope of this work.