When Removing More Guidance Hurts Less: Text and Spatial Control in Diffusion

Controlled diffusion models combine text prompts with spatial inputs such as edges or depth maps. We show that retaining text during an early interruption of spatial control can worsen final spatial adherence: temporarily removing both inputs preserves adherence better than removing spatial control alone, even though both are restored afterward. Across four configurations combining two control architectures with edge and depth inputs, this ordering holds in 76–81% of scenes and also yields lower mean perceptual distance to the fully conditioned output. An SDXL experiment reveals a conditional sign reversal: text helps adherence when spatial control is present but harms it during the interruption. Adding the effects of separate removals consequently predicts the wrong ordering of the two schedules. The effects depend on timing and on the outcome measured; a small adherence change can accompany a substantial image change. Full conditioning still gives better mean adherence than joint removal, and the SDXL perceptual distance estimate does not reproduce the advantage. These results show when separate ablations give the wrong conditional choice: deciding which input to retain requires measuring its effect in the presence or absence of the others, at the intended denoising stage.