acceptodds
Under review as a conference paper at ICLR 2027

When Removing More Guidance Hurts Less: Text and Spatial Control in Diffusion

Abstract

Controlled diffusion models combine text prompts with spatial inputs such as edges or depth maps. We show that retaining text during an early interruption of spatial control can worsen final spatial adherence: temporarily removing both inputs preserves adherence better than removing spatial control alone, even though both are restored afterward. Across four configurations combining two control architectures with edge and depth inputs, this ordering holds in 76–81% of scenes and also yields lower mean perceptual distance to the fully conditioned output. An SDXL experiment reveals a conditional sign reversal: text helps adherence when spatial control is present but harms it during the interruption. Adding the effects of separate removals consequently predicts the wrong ordering of the two schedules. The effects depend on timing and on the outcome measured; a small adherence change can accompany a substantial image change. Full conditioning still gives better mean adherence than joint removal, and the SDXL perceptual distance estimate does not reproduce the advantage. These results show when separate ablations give the wrong conditional choice: deciding which input to retain requires measuring its effect in the presence or absence of the others, at the intended denoising stage.

Primary area:
generative models
Keywords:
diffusion models, multimodal conditioning, guidance, causal interventions, counterfactual rollouts, temporal attribution, controllable generation, sensitivity analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KtmoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies interactions between text and spatial conditioning through matched four-condition interventions during different denoising windows. A smaller SDXL experiment shows that removing text harms adherence when spatial control is present but improves it when control is absent, reversing the schedule ranking predicted by adding separate ablation effects. The paper also examines temporal dependence and distinguishes spatial adherence from perceptual distance to the fully conditioned output.

Critical strengths
  1. The counterintuitive observation was supported across multiple control configurations.
  2. The factorial comparison connects interactions to an actual decision error. The paper clearly distinguishes the direct joint-removal advantage from the factorial interaction and from a conditional sign reversal.
  3. The comparisons match scenes and initial latents, and the bootstrap resamples scenes while retaining their repeated generations together.
Critical weaknesses
  1. Emptying text makes the conditional and unconditional branches coincide, eliminating classifier-free guidance extrapolation as well as prompt information. Section 6 acknowledges this issue, but the key control of retaining the original prompt at guidance scale 1 is not reported. This matters because prior work already finds early guidance detrimental in other diffusion settings (Kynkaanniemi et al., 2024, https://arxiv.org/abs/2404.07724). The observed ordering is credible under the stated intervention, but its explanation and significance as text-spatial competition remain uncertain. Comparing empty text against retained text at scale 1, with spatial control both present and absent, would help establish what drives the conditional reversal.

  2. The four main configurations share an SD1.5 host, one 30-step DDIM protocol, and fixed guidance and control scales. The clearest sign reversal is demonstrated on only 50 development scenes in SDXL. That comparison also changes the control checkpoint, output resolution, and empty-conditioning convention together (Section 4). Moreover, the interaction interpretation was developed after inspecting outputs, as disclosed in Section 3.3. These facts do not invalidate the reported paired differences, but leave uncertainty about how reliably the central finding extends beyond the explored settings. A held-out SDXL replication and sensitivity to interruption length or sampler choice would substantially strengthen the evidence.

Questions for authors
  1. Regarding Weakness 1, could you compare three early-window settings: the original prompt at guidance scale 7.5, the original prompt at scale 1, and empty text, each with spatial control present and absent? Please keep the continuation unchanged. Does removing prompt information still improve adherence once extrapolation is disabled? This targeted control could materially change the interpretation of the main result.

  2. Regarding Weakness 2, please clarify when the 50 SDXL development scenes and the sign-reversal analysis were selected relative to inspecting the outputs. Is there a replication on held-out scenes under a fixed protocol? If available, reporting the two conditional effects and reversal frequencies on that set would materially strengthen confidence. Testing additional denoiser families would be useful future work, but is not essential to answer this question.

  3. Most of the figures in the article are probably AI-generated, which I don't find very intuitive. Could you please create them manually to make them more visually appealing?

Reviewer_EsLXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how temporarily removing text and spatial conditioning affects existing diffusion models. It reports that removing both early can preserve spatial adherence better than removing spatial conditioning alone.

Critical strengths

None.

Critical weaknesses
  1. I am not convinced that the paper establishes a sufficiently novel or significant contribution. It evaluates conditioning interventions on pretrained models, but provides neither a new method nor an explanation of the observed behavior. The analysis does not establish how this observation leads to a useful improvement in generation.
  2. The presentation makes the contribution difficult to understand, and the qualitative evidence is very limited. The comparisons are within-model ablations rather than comparisons to alternative methods. More fundamentally, the reported metrics do not make the practical significance of the observation convincing.
Questions for authors

What new understanding or practical consequence does the paper establish beyond the reported conditioning observation?

Reviewer_VMdFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how text and spatial conditioning interact across the denoising process of the diffusion models. By the 2×2 factorial design, the authors selectively remove text, spatial control, or both during early (near the initial noise), middle, or late denoising windows. Removing both conditions on early denoising steps preserves spatial adherence better than removing spatial control only. But this advantage largely disappears in later stages. An SDXL experiment moreover reveals that text conditioning effects can reverse depending on whether spatial control is present, demonstrating that independent ablations can be misleading. Overall, this paper highlights that conditioning effects should not treated as independent contributions, but must be evaluated jointly, at specific denoising stages, and with respect to the target metric.

Critical strengths
  1. The experimental protocols are well defined. Variable controls are presented clearly.
Critical weaknesses
  1. The main comparison doesn't give a use case. A sampling schedule is only useful when making good quality images. However, the core comparison in this paper is between two degraded scheduling process. With this one, where can we apply it?

  2. "Text removal" is being confused with the removal of classifier-free guidance. Although the authors acknowledged this as a formula (App A.4 Eqs(4)), they did not conduct experiments to test it.

  3. Generalizability among models was not established. And they were not tested with the latest models.

Questions for authors
  1. Please report the image quality metrics, such as FID and prompt coherence (CLIPScore).

  2. Please also conduct an experiment based on the guidance scale.

  3. It would strengthen the paper to validate the main findings on current rectified-flow transformers such as SD3.5 and FLUX

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.