- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DisControl modulates local control strength by mixing spatially varying noise into control maps and training a ControlNet adapter with an explicit strength-map input. It aims to improve regional coherence, spatial separation, and the reliability of control-strength modulation.The paper addresses a practical limitation of local ControlNet modulation. The experiments demonstrate improved separation between controlled and released regions, alongside image-quality gains on SD1.5 and SDXL.
Critical strengths
The paper addresses a practical limitation of local ControlNet modulation. The experiments demonstrate improved separation between controlled and released regions, alongside image-quality gains on SD1.5 and SDXL.
Critical weaknesses
-
Limited novelty. The contribution primarily consists of noise-based degradation of conditioning inputs and explicit strength-map conditioning. While useful, the methodological advance appears modest.
-
Sensitivity and early saturation of the strength parameter. Figures 6 and 7 suggest that much of the response is concentrated in a narrow range of small values, followed by early saturation. Section 5.4 evaluates cross-image variability at matched mean adherence, which does not establish that the strength parameter is easy to tune. The effective range for intermediate control therefore needs clearer characterization.
-
Insufficient explanation of the SD3.5 results. Section 5.2 attributes the HPSv3++ decrease to possible adapter-induced quality costs and limitations of the metric. These hypotheses are plausible but unverified. Since both considerations also apply to other backbones, further evidence is needed to explain the contrasting quality trends across SD3.5, SD1.5, and SDXL.
Questions for authors
See the weakness part.
3 comments
Part 2/2
The SD3.5 quality result. We ran global-strength cells that bear on the first hypothesis of Section 5.2 (depth, both arms, same ids and per-image seeds, HPSv3++, n=200). At global s = 0 the vanilla ControlNet reduces to the base model, which scores 12.16 on SD3.5. At global s = 1 both adapters land about 1.3 below it, at 10.89 (vanilla) and 10.81 (DisControl), with the same adherence, so on SD3.5 the adaptation itself costs quality at full control, whichever arm is trained, possibly because of the larger capacity of the SD3.5 backbone or its sensitivity to fine-tuning. On SD1.5 and SDXL the opposite holds: both adapters at s = 1 score above the base model (SDXL 9.84 and 9.76 against 8.34). This is the backbone difference behind the contrasting trends. It also shows up in the released region: the DisControl adapter driven with a fully noised map (s = 0) is 0.66 below the base model on SD3.5 (11.50 against 12.16, p=0.0006), whereas on SDXL it matches the base model (8.45 against 8.34, p=0.25) and on SD1.5 it is above it. Activation scaling removes the adapter from the released region, noise modulation keeps it active, and on SD3.5 that region pays the adapter's cost. We will state this measurement in Section 5.2 in place of the hypothesis.
To address this, we trained the SD3.5 depth adapter as a LoRA instead of a ControlNet (rank 64, 84M trainable parameters against the ControlNet's 1.26B), a lower-capacity adapter that is more data- and parameter-efficient to fine-tune, driven by the same noise modulation. Under split-half modulation (left s = 1, right s = 0) it keeps the quality where both ControlNets lose it (HPSv3++, same 200 ids and seeds):
| SD3.5 depth, split-half (n=200) | HPSv3++ | vs vanilla ControlNet |
|---|---|---|
| vanilla ControlNet, scale | 10.19 | |
| DisControl ControlNet, noise | 10.10 | -0.8% |
| DisControl LoRA, noise | 11.75 | +15.3% |
So DisControl works with adapters other than the ControlNet, and those do not necessarily show the quality metric drop: on SD3.5 the LoRA adapter under the same mechanism improves HPSv3++ over both ControlNet counterparts, +15.3% over the vanilla ControlNet baseline. The drop in Table 1 is a property of the SD3.5 ControlNet adapter, not of the mechanism.
Since HPSv3++ is an automatic judge, we commit to providing a human preference study across backbones and modalities in the final version, or before the discussion period ends if time permits, asking whether judges prefer our method over the vanilla baselines under local modulation, with external judges recruited through Prolific. As a preliminary result, we ran a side-randomised pilot judged by one of the authors on 50 SD3.5 depth split-half pairs: DisControl was preferred on 35 pairs, activation scaling on 10, with 5 ties (78% of decided pairs, two-sided sign test p=0.0002). We acknowledge this is not a rigorous procedure and present it only as an indication or initial signal prior to the test on external judges. We also point again to Figures 1 and 4 in the main text, which show the kind of artifact that judges and the DreamSim metric catch but HPSv3++ does not.
Following up on our response, we report the results of the human preference study with external raters, which we were able to run during the discussion period.
Human preference study (external raters, Prolific). SD3.5 depth, split-half (left s = 1, right s = 0), the same ids and seeds as the new cells; 100 pairs per comparison against the Table 1 baseline (vanilla ControlNet, activation scaling). Raters were told each image was generated in two halves and asked in which image the halves fit together better as one scene (no seam, consistent lighting, objects continuing across the middle), the property DreamSim measures; the author pilot above asked for overall preference. Each pair was rated by two different raters, one per order, with attention-check pairs and a pre-set exclusion rule (one of 10 raters excluded).
| comparison | DisControl | baseline | tie | share of decided | sign test p |
|---|---|---|---|---|---|
| LoRA (noise) vs ControlNet (scale) | 89 | 49 | 42 | 64% | 0.0008 |
| ControlNet (noise) vs ControlNet (scale) | 78 | 56 | 46 | 58% | 0.07 |
Per pair (majority vote): 41 / 21 / 38 for the LoRA (p = 0.015) and 44 / 27 / 29 for the ControlNet (p = 0.057). No position bias (first-shown image picked 63 and 62 times of 180).
External raters prefer DisControl on the consistency between the two regions, significantly for the LoRA adapter and with a consistent lean for the ControlNet, in line with Table 1's DreamSim results.
Part 1/2
Thank you for the detailed review. These are fair points. We ran additional experiments that help answer your questions, and we are happy to provide further explanation if anything remains unclear. Unless noted otherwise, the new cells use the paper's protocol: the same test ids and per-image seeds.
Novelty. We agree that the mechanism is simple to state: degrading the control map is a natural way to modulate control. We think the contribution is larger than the mechanism alone. The mechanism that dominates practice, scaling the adapter's activations, is accepted as the standard strength knob used in practice, and the paper shows that it fails under local modulation. Practitioners share the intuition that neighbouring regions should not be given very different strengths, but to our knowledge nobody had looked at why, or measured it. Against that background, the finding that modulating the information content of the input is a valid solution for local strength is not obvious. Part of the paper's value is in questioning the widely used mechanism and proposing to replace it with a clean and fundamentally different one that removes many of its limitations. The analysis is also new in itself: we are the first to measure boundary resolution under local modulation and the per-image variance of adherence, and we find both results interesting on their own.
More generally, the idea behind the paper is the benefit of moving from an inference-time manipulation of activations, which comes with no guarantee of working in general, to a mechanism that is in distribution by construction, since the model is trained on the inputs it sees at inference, modulates at the resolution of the input, has an interpretable effect (the strength sets how much information of the control map is retained), and in principle applies to any conditional generative model, not only diffusion models.
Tuning the strength parameter. It is true that, because of the different mathematical nature of our parameter, it is not calibrated with respect to the activation-scaling strength and the useful range lies in the lower range of values. Section 5.4 does, however, address one aspect of tuneability indirectly: at a given mean adherence the per-image spread is lower under DisControl, so the relation between the strength value and the adherence it produces is less noisy than under activation scaling. The same value gives closer to the same result from one image to the next, and a value chosen on one image carries over to the others better than under activation scaling. In this sense the strength parameter is, if anything, easier to tune under DisControl. To address tuneability directly, we add an experiment that proposes a simple way to tune the strength in a more intuitive and controllable manner: a predictor that maps the desired adherence to the strength value that achieves it.
The user specifies a target adherence and the predictor returns the strength value to use. We fit it separately for each method, backbone and modality, in two passes. First pass: generate 1000 images at strengths drawn uniformly in [0, 1] and measure the adherence of each. With these samples we fit a simple four-parameter model, s(a)=w0a+w1exp(w2a)+w3, as the inverse mapping from adherence (a) to input strength (s). Second pass: generate 1000 new images at the strengths the predictor returns for 1000 equispaced target adherences, measure the adherece of the generate images to the control image, and report the mean absolute error (MAE) between the target and the obtained adherence. A lower MAE means that asking for a given adherence delivers it more reliably. The MAE is lower under DisControl on every backbone (depth, ControlNet adapters; adherence is the linear-MSE between the re-extracted depth and the control map, targets equispaced in [0, 0.05]). That suggests that the strength parameter under DisControl is more reliable as a strenght modulation knob with this easy fix to solve the issue of having the useful range of value on the lower range.
We are happy to provide additional details or experiments on this topic.