DisControl: Local Control Strength Modulation Via Distortion of the Conditioning Signal

ControlNet has become the standard for spatial conditioning of diffusion models. At inference, a strength parameter modulates how strongly the condition is applied, trading adherence against diversity and image quality, and setting how much of the control map’s detail is imposed, so that a coarse map can be used at a lower strength; the same knob can be set per region, so that different parts of an image follow the condition to different degrees. This is implemented by locally scaling the adapter’s activations, an operation the network was never trained to receive, and we identify three limitations of it. First, under local modulation the scaled activations take the network out of distribution and image quality degrades severely. Second, a per-pixel strength map cannot be imposed on activations: it has to be resampled onto a representation coarser than the control map, as coarse as 8 × 8, and the backbone then mixes information between neighbouring positions, so a region boundary blurs either way and precise modulation on a specific object is out of reach. Third, the adherence actually realized at a given strength varies widely across images, as a result of the model picking up or not picking up on the control signal more randomly rather than smoothly. No method is explicitly designed for this modulation, and we propose a fundamental shift: from modulating internal activations at inference time to degrading the input at training time. In DisControl, the control signal is degraded by injecting noise at full pixel resolution and the adapter is trained on this family of degraded inputs, so that every strength is in distribution, region boundaries are sharp by construction, and modulation is part of the learned task rather than an intervention on top of it. Across backbones and control modalities, DisControl suppresses the dissimilarity artifacts that local modulation introduces, separates regions at pixel resolution, and reduces the variance of realized adherence. We show how our adapter can be trained from scratch or efficiently fine-tuned from existing ControlNets.