acceptodds
Under review as a conference paper at ICLR 2027

DisControl: Local Control Strength Modulation Via Distortion of the Conditioning Signal

Abstract

ControlNet has become the standard for spatial conditioning of diffusion models. At inference, a strength parameter modulates how strongly the condition is applied, trading adherence against diversity and image quality, and setting how much of the control map’s detail is imposed, so that a coarse map can be used at a lower strength; the same knob can be set per region, so that different parts of an image follow the condition to different degrees. This is implemented by locally scaling the adapter’s activations, an operation the network was never trained to receive, and we identify three limitations of it. First, under local modulation the scaled activations take the network out of distribution and image quality degrades severely. Second, a per-pixel strength map cannot be imposed on activations: it has to be resampled onto a representation coarser than the control map, as coarse as 8 × 8, and the backbone then mixes information between neighbouring positions, so a region boundary blurs either way and precise modulation on a specific object is out of reach. Third, the adherence actually realized at a given strength varies widely across images, as a result of the model picking up or not picking up on the control signal more randomly rather than smoothly. No method is explicitly designed for this modulation, and we propose a fundamental shift: from modulating internal activations at inference time to degrading the input at training time. In DisControl, the control signal is degraded by injecting noise at full pixel resolution and the adapter is trained on this family of degraded inputs, so that every strength is in distribution, region boundaries are sharp by construction, and modulation is part of the learned task rather than an intervention on top of it. Across backbones and control modalities, DisControl suppresses the dissimilarity artifacts that local modulation introduces, separates regions at pixel resolution, and reduces the variance of realized adherence. We show how our adapter can be trained from scratch or efficiently fine-tuned from existing ControlNets.

Primary area:
generative models
Keywords:
diffusion models, controllable generation, ControlNet, control strength modulation, local control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5AKqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DisControl modulates local control strength by mixing spatially varying noise into control maps and training a ControlNet adapter with an explicit strength-map input. It aims to improve regional coherence, spatial separation, and the reliability of control-strength modulation.The paper addresses a practical limitation of local ControlNet modulation. The experiments demonstrate improved separation between controlled and released regions, alongside image-quality gains on SD1.5 and SDXL.

Critical strengths

The paper addresses a practical limitation of local ControlNet modulation. The experiments demonstrate improved separation between controlled and released regions, alongside image-quality gains on SD1.5 and SDXL.

Critical weaknesses
  • Limited novelty. The contribution primarily consists of noise-based degradation of conditioning inputs and explicit strength-map conditioning. While useful, the methodological advance appears modest.

  • Sensitivity and early saturation of the strength parameter. Figures 6 and 7 suggest that much of the response is concentrated in a narrow range of small ss values, followed by early saturation. Section 5.4 evaluates cross-image variability at matched mean adherence, which does not establish that the strength parameter is easy to tune. The effective range for intermediate control therefore needs clearer characterization.

  • Insufficient explanation of the SD3.5 results. Section 5.2 attributes the HPSv3++ decrease to possible adapter-induced quality costs and limitations of the metric. These hypotheses are plausible but unverified. Since both considerations also apply to other backbones, further evidence is needed to explain the contrasting quality trends across SD3.5, SD1.5, and SDXL.

Questions for authors

See the weakness part.

3 comments

Thank you for the detailed review. These are fair points. We ran additional experiments that help answer your questions, and we are happy to provide further explanation if anything remains unclear. Unless noted otherwise, the new cells use the paper's protocol: the same test ids and per-image seeds.

Novelty. We agree that the mechanism is simple to state: degrading the control map is a natural way to modulate control. We think the contribution is larger than the mechanism alone. The mechanism that dominates practice, scaling the adapter's activations, is accepted as the standard strength knob used in practice, and the paper shows that it fails under local modulation. Practitioners share the intuition that neighbouring regions should not be given very different strengths, but to our knowledge nobody had looked at why, or measured it. Against that background, the finding that modulating the information content of the input is a valid solution for local strength is not obvious. Part of the paper's value is in questioning the widely used mechanism and proposing to replace it with a clean and fundamentally different one that removes many of its limitations. The analysis is also new in itself: we are the first to measure boundary resolution under local modulation and the per-image variance of adherence, and we find both results interesting on their own.

More generally, the idea behind the paper is the benefit of moving from an inference-time manipulation of activations, which comes with no guarantee of working in general, to a mechanism that is in distribution by construction, since the model is trained on the inputs it sees at inference, modulates at the resolution of the input, has an interpretable effect (the strength sets how much information of the control map is retained), and in principle applies to any conditional generative model, not only diffusion models.

Tuning the strength parameter. It is true that, because of the different mathematical nature of our parameter, it is not calibrated with respect to the activation-scaling strength and the useful range lies in the lower range of values. Section 5.4 does, however, address one aspect of tuneability indirectly: at a given mean adherence the per-image spread is lower under DisControl, so the relation between the strength value and the adherence it produces is less noisy than under activation scaling. The same value gives closer to the same result from one image to the next, and a value chosen on one image carries over to the others better than under activation scaling. In this sense the strength parameter is, if anything, easier to tune under DisControl. To address tuneability directly, we add an experiment that proposes a simple way to tune the strength in a more intuitive and controllable manner: a predictor that maps the desired adherence to the strength value that achieves it.

The user specifies a target adherence and the predictor returns the strength value to use. We fit it separately for each method, backbone and modality, in two passes. First pass: generate 1000 images at strengths drawn uniformly in [0, 1] and measure the adherence of each. With these samples we fit a simple four-parameter model, s(a)=w0 a+w1exp⁡(w2 a)+w3s(a) = w_0\, a + w_1 \exp(w_2\, a) + w_3, as the inverse mapping from adherence (aa) to input strength (ss). Second pass: generate 1000 new images at the strengths the predictor returns for 1000 equispaced target adherences, measure the adherece of the generate images to the control image, and report the mean absolute error (MAE) between the target and the obtained adherence. A lower MAE means that asking for a given adherence delivers it more reliably. The MAE is lower under DisControl on every backbone (depth, ControlNet adapters; adherence is the linear-MSE between the re-extracted depth and the control map, targets equispaced in [0, 0.05]). That suggests that the strength parameter under DisControl is more reliable as a strenght modulation knob with this easy fix to solve the issue of having the useful range of value on the lower range.

backbone, modalityMAE, activation scalingMAE, DisControlMAE reduction
SD1.5 depth0.01080.007926.9%
SD3.5 depth0.01320.009528.0%
SDXL depth0.01020.007922.5%

We are happy to provide additional details or experiments on this topic.

The SD3.5 quality result. We ran global-strength cells that bear on the first hypothesis of Section 5.2 (depth, both arms, same ids and per-image seeds, HPSv3++, n=200). At global s = 0 the vanilla ControlNet reduces to the base model, which scores 12.16 on SD3.5. At global s = 1 both adapters land about 1.3 below it, at 10.89 (vanilla) and 10.81 (DisControl), with the same adherence, so on SD3.5 the adaptation itself costs quality at full control, whichever arm is trained, possibly because of the larger capacity of the SD3.5 backbone or its sensitivity to fine-tuning. On SD1.5 and SDXL the opposite holds: both adapters at s = 1 score above the base model (SDXL 9.84 and 9.76 against 8.34). This is the backbone difference behind the contrasting trends. It also shows up in the released region: the DisControl adapter driven with a fully noised map (s = 0) is 0.66 below the base model on SD3.5 (11.50 against 12.16, p=0.0006), whereas on SDXL it matches the base model (8.45 against 8.34, p=0.25) and on SD1.5 it is above it. Activation scaling removes the adapter from the released region, noise modulation keeps it active, and on SD3.5 that region pays the adapter's cost. We will state this measurement in Section 5.2 in place of the hypothesis.

To address this, we trained the SD3.5 depth adapter as a LoRA instead of a ControlNet (rank 64, 84M trainable parameters against the ControlNet's 1.26B), a lower-capacity adapter that is more data- and parameter-efficient to fine-tune, driven by the same noise modulation. Under split-half modulation (left s = 1, right s = 0) it keeps the quality where both ControlNets lose it (HPSv3++, same 200 ids and seeds):

SD3.5 depth, split-half (n=200)HPSv3++vs vanilla ControlNet
vanilla ControlNet, scale10.19
DisControl ControlNet, noise10.10-0.8%
DisControl LoRA, noise11.75+15.3%

So DisControl works with adapters other than the ControlNet, and those do not necessarily show the quality metric drop: on SD3.5 the LoRA adapter under the same mechanism improves HPSv3++ over both ControlNet counterparts, +15.3% over the vanilla ControlNet baseline. The drop in Table 1 is a property of the SD3.5 ControlNet adapter, not of the mechanism.

Since HPSv3++ is an automatic judge, we commit to providing a human preference study across backbones and modalities in the final version, or before the discussion period ends if time permits, asking whether judges prefer our method over the vanilla baselines under local modulation, with external judges recruited through Prolific. As a preliminary result, we ran a side-randomised pilot judged by one of the authors on 50 SD3.5 depth split-half pairs: DisControl was preferred on 35 pairs, activation scaling on 10, with 5 ties (78% of decided pairs, two-sided sign test p=0.0002). We acknowledge this is not a rigorous procedure and present it only as an indication or initial signal prior to the test on external judges. We also point again to Figures 1 and 4 in the main text, which show the kind of artifact that judges and the DreamSim metric catch but HPSv3++ does not.

Following up on our response, we report the results of the human preference study with external raters, which we were able to run during the discussion period.

Human preference study (external raters, Prolific). SD3.5 depth, split-half (left s = 1, right s = 0), the same ids and seeds as the new cells; 100 pairs per comparison against the Table 1 baseline (vanilla ControlNet, activation scaling). Raters were told each image was generated in two halves and asked in which image the halves fit together better as one scene (no seam, consistent lighting, objects continuing across the middle), the property DreamSim measures; the author pilot above asked for overall preference. Each pair was rated by two different raters, one per order, with attention-check pairs and a pre-set exclusion rule (one of 10 raters excluded).

comparisonDisControlbaselinetieshare of decidedsign test p
LoRA (noise) vs ControlNet (scale)89494264%0.0008
ControlNet (noise) vs ControlNet (scale)78564658%0.07

Per pair (majority vote): 41 / 21 / 38 for the LoRA (p = 0.015) and 44 / 27 / 29 for the ControlNet (p = 0.057). No position bias (first-shown image picked 63 and 62 times of 180).

External raters prefer DisControl on the consistency between the two regions, significantly for the LoRA adapter and with a consistent lean for the ControlNet, in line with Table 1's DreamSim results.

Reviewer_pe9WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new way to modulate the control strength of ControlNet called DisControl. Instead of scaling the internal activations during inference, the authors propose to directly add noise to the input control map according to a user-defined strength map. The ControlNet is then trained with such degraded control maps, so it can learn how to respond to different control strengths. The authors claim that this could improve local control precision and avoid the artifacts caused by traditional activation scaling. Experiments on SD1.5, SDXL, and SD3.5 with depth, canny, and normal maps show improvements in local control consistency and adherence.

Critical strengths
  • I believe this paper studies an interesting and practical problem. Local control strength is quite useful for controllable image generation, and the authors point out some clear limitations of the commonly used activation scaling method. The motivation is reasonable and the proposed method directly addresses these problems.
  • The method itself is simple and intuitive. Instead of modifying the internal features, it directly changes the amount of information in the input control map. I also appreciate that the authors show how to adapt an existing ControlNet to DisControl without training everything from scratch.
  • The experiments are quite extensive. The authors evaluate on three different backbones and three control modalities. The comparisons are also relatively fair, with the same training data, steps, and inference seeds. The visual results on local control look promising, and the quantitative results on boundary separation are also encouraging.
Critical weaknesses
  • One concern is that I'm not sure how much of the improvement actually comes from the proposed noise degradation mechanism. In Section 4, DisControl not only degrades the original control map, but also explicitly feeds the strength map into ControlNet as an additional input. So the improvement could also come from directly conditioning the model on the desired strength, rather than the noise degradation itself. I feel like a simple baseline that feeds the original clean control map together with the strength map, without any noise degradation, would be helpful. This could better show whether degrading the control signal is really necessary, or if the model can already learn similar local modulation from the explicit strength map.
  • Another weakness is about the evaluation of continuous local strength modulation. One of the main selling points of DisControl is that it allows smoothly adjusting the control strength at different regions. However, most quantitative experiments on local controlare based on binary strength maps, where one region has strength 1 and the other has strength 0. I am curious to see, for example, what if two regions are given strengths 0.3 and 0.7? Would both regions follow the expected level of adherence? And would changing the strength of one region affect the other? I think some quantitative experiments on intermediate local strengths would be needed to fully support the claim.
Questions for authors
  • In Table 1, the HPSv3++ scores on SD3.5 actually drop for all three modalities, especially normals (-14.1%). Although DreamSim improves, I'm not sure if this necessarily means the overall generation quality is better. Could the authors provide more discussion on this trade-off?
  • Regarding the coherence evaluation, I am not sure if DreamSim between the two halves is a good measurement of boundary coherence. Two halves of a coherent image could naturally contain very different objects and textures, while two visually similar halves do not necessarily mean there is no visible seam. Some additional human evaluation on local coherence would be appreciated.
4 comments

Thank you for the detailed review. These are fair points. We ran additional experiments that help answer your questions, and we are happy to provide further explanation if anything remains unclear. Unless noted otherwise, the new cells use the paper's protocol: the same test ids and per-image seeds.

Does the gain come from the strength channel rather than from the degradation? We believe the strength channel alone, with a clean control map, would not produce the desired effect. The model is trained to reconstruct the real image (or its noise or velocity, depending on the parametrisation) from the control map and the strength map. If we trained it on the clean map with an additional strength channel, it would learn to rely on the clean map, which contains the information that minimises the diffusion objective, and adhere to it everywhere; it would have no reason to use the strength map, which carries no additional information useful for the reconstruction. Lowering the strength at inference would then change nothing.

The paper already contains the baseline closest to this one: Table 4, "Trained with intermediate scale". It is the setting described, clean control map plus the same per-pixel strength maps DisControl is trained on as an input channel, with one addition: the strength map also scales the adapter's residuals at train time, which in principle could give the strength a meaning. In practice it does not, for the same reason as above. Scaling the activations down does not remove the information from the map like degrading with noise does. The target still follows the clean map everywhere, so rather than learning to follow the map less where the strength is low, the model learns to be insensitive to the scale of the residuals, amplifying them and relying on the map in full, and it applies full control for any strength above 0.1 to 0.2 (Figure 5 in the supplementary material, Section 5.3): it separates a binary mask but does not interpolate between strengths. Degrading the map is the only one of these settings in which the objective itself rewards following the map less where the strength is low, which is why we attribute the gain to the degradation rather than to conditioning on the strength.

Please let us know if we understood your baseline suggestion correctly and whether this has addressed your concern.

Intermediate strengths. Split-half on SD1.5 depth (n=200, the first 200 of the 1000 ids), left half at s = 1, right half swept from 0 to 1; we report the right-half adherence, DreamSim between the halves and HPSv3++. Adherence is monotone in s for both methods. DreamSim varies much less for DisControl (0.35 to 0.29 over the sweep) than for scale modulation (0.48 to 0.29), where it grows as the two halves get more different strengths (right 0, left 1) and shrinks as they get more similar (both 1). This is in line with what we expected: local modulation with activation scaling trades quality for adherence, and DisControl largely removes that trade-off, so it can be used at intermediate strengths.

right-half s0.00.30.50.71.0
DisControl: adherence right0.02440.00580.00490.00400.0030
DisControl: DreamSim0.3560.3170.3100.2990.289
DisControl: HPSv3++7.608.188.238.197.97
vanilla: adherence right0.02920.01040.00520.00350.0031
vanilla: DreamSim0.4810.3920.3340.2980.287
vanilla: HPSv3++4.796.417.106.696.92

On the 0.3 / 0.7 case specifically (same cells, SD1.5 depth, n=200; adherence left / right): with (0.3, 0.7) DisControl gives 0.0063 / 0.0054 and with (0.7, 0.3) it gives 0.0043 / 0.0061, so each half follows its own level in both orders (vanilla: 0.0104 / 0.0046 and 0.0050 / 0.0105). On whether one region affects the other: DisControl's right half at s = 0.3 reads 0.0058, 0.0061 and 0.0066 as the left half goes from 1 to 0.7 to 0.3, a 14% change, where under activation scaling it reads 0.0104, 0.0105 and 0.0179, a 72% change. The regions are not independent under either method, but the cross-talk is much smaller under DisControl.

The SD3.5 HPSv3++ drop and the quality trade-off. The HPSv3++ drop on SD3.5 is a cost of the SD3.5 ControlNet adapter itself, not of the modulation mechanism, and with a different adapter there is no trade-off. We support this with two new results, detailed below: (1) global-strength cells showing that on SD3.5 any adapter costs quality against the base model, whichever arm is trained, and that DisControl pays this cost in the released region where activation scaling removes the adapter; (2) a LoRA adapter driven by the same noise modulation, on which both DreamSim and HPSv3++ improve over the vanilla ControlNet. We close with our commitment to a human preference study.

(1) Global-strength cells (depth, both arms, same ids and per-image seeds, HPSv3++, n=200). On SD3.5 the base model (the vanilla ControlNet at global s = 0) scores 12.16; at global s = 1 both adapters land about 1.3 below it, 10.89 (vanilla) and 10.81 (DisControl), with the same adherence, so on SD3.5 the adaptation itself costs quality at full control, whichever arm is trained. On SD1.5 and SDXL both adapters at s = 1 score above the base model. The cost also appears in the released region: the DisControl adapter driven with a fully noised map (s = 0) is 0.66 below the base model on SD3.5 (11.50 against 12.16, p=0.0006), whereas on SDXL it matches the base model and on SD1.5 it is above it. Activation scaling removes the adapter from the released region, noise modulation keeps it active, and on SD3.5 that region pays the adapter's cost. This is what the SD3.5 rows of Table 1 measure, and we will state it in Section 5.2 in place of the hypothesis.

(2) A LoRA adapter. We trained the SD3.5 depth adapter as a LoRA instead of a ControlNet (rank 64, 84M trainable parameters against the ControlNet's 1.26B), driven by the same noise modulation. Under split-half modulation it keeps the quality where both ControlNets lose it (HPSv3++, same 200 ids and seeds):

SD3.5 depth, split-half (n=200)HPSv3++vs vanilla ControlNet
vanilla ControlNet, scale10.19
DisControl ControlNet, noise10.10-0.8%
DisControl LoRA, noise11.75+15.3%

So DisControl works with other adapters, and with the LoRA the trade-off disappears on SD3.5: better DreamSim and better HPSv3++ than the vanilla ControlNet.

Human evaluation. Since HPSv3++ is an automatic judge, we commit to providing a human preference study across backbones and modalities in the final version, or before the discussion period ends if time permits, asking whether judges prefer our method over the vanilla baselines under local modulation, with external judges recruited through Prolific. As a preliminary result, we ran a side-randomised pilot judged by one of the authors on 50 SD3.5 depth split-half pairs: DisControl was preferred on 35 pairs, activation scaling on 10, with 5 ties (78% of decided pairs, two-sided sign test p=0.0002). We acknowledge this is not a rigorous procedure and present it only as an initial signal prior to the test on external judges. We would also ask the reviewer to look at the qualitative examples in Figures 1 and 4 of the main text for an intuition of the kind of artifact at stake: a seam and a change of content across the strength boundary that a human judge and DreamSim catch but HPSv3++ does not.

Is DreamSim between the halves a coherence measure? We agree that the two halves of a coherent image naturally differ, so the absolute DreamSim value is not what carries the information; the level relative to the natural one is. That natural level can be measured: a split-half generation with s = 1 on both halves is mathematically the same as a global generation at s = 1, and the DreamSim between its halves is the in-distribution dissimilarity of a coherent image given by a regular ControlNet with a clean input. On SD1.5 depth (n=200) it is 0.287 for activation scaling and 0.289 for DisControl. The two methods behave differently when the halves are given different strengths: at (1, 0) activation scaling rises to 0.481, 68% above its own same-strength level, while DisControl rises to 0.356, 23% above.

(1) Thank the authors for their clarification on this point. I had some misunderstanding regarding the training process. Given the clean map with additional control strength channel may indeed eliminate the effect of the control strength channel under current training setting. My intuition was that since your goal is to achieve different levels of control adherence, and you are giving the model explicit control strength as an additional parameter, the gain therefore cannot be fully credited to the noise degradation mechanism. Current experiments do not fully seperate the contribution of these two components. One way is maybe to train a model with only a noise-degraded map and no control strength channel. Otherwise, the authors could make the contribution more precise, to acknowledge that noise degradation and the explicit control strength parameter may work together to improve the performance, rather than attributing the improvement solely to noise degradation.

(2) Additional experiments on intermediate strength, especially the [0.3/0.7] vs. [0.7/0.3] cases, are appreciated. This directly addressed my concern about continuous local strength modulation.

(3) Thank the authors for providing additional experimental evidence of their hypothesis of performance drop on SD3.5. That indeed helps explaining the problem. The human evaluation is also encouraged to showcase the actual quality comparison between base and DisControl. One thing I am still curious about is why normals in particular drops so much from the base, compared to the rather modest drop on depth and canny? A brief discussion is helpful.

(4) Thank you for the clarification on DreamSim usage. The comparison against [1/1] is a more meaningful reference and my concern is largely addressed.

Thank you for the follow-up.

(1) Noise degradation and the explicit strength input. Our earlier experiments pointed to the input degradation as the main driver of the improvement over the baseline: a model trained on the degraded map alone, without the strength channel, already gave the quality improvement and the dissimilarity reduction, which is expected, since it is the degradation that makes the degraded inputs in-distribution. The strength channel added a small gain in the variance property of Section 5.4 (a slightly lower spread of adherence across images at a given strength) and none in quality. Those runs predate the current corpus, protocol and training recipe, so they are not directly comparable to the paper's models and we cannot show a clean isolation at the moment. We believe the explicit strength input is not the main driver of the quality or adherence improvements. The ablation needs a full model to be trained, which did not fit in the discussion period; the final version will include it: degraded map alone against degraded map plus strength input, same data, recipe and steps, evaluated with the split-half protocol.

(3) Why normals drops more on SD3.5. We do not know for certain, but we have a hypothesis. On qualitative inspection the normals extractor is noisier than the depth and canny extractors: its maps give less accurate geometry and edges. The diffusion model may therefore struggle to learn the image distribution conditioned on these maps, and activation scaling disproportionately favours the baseline here because it removes the adapter's influence on part of the image, whereas DisControl keeps the adapter active everywhere. This may also affect the training dynamics or the time to reach good quality, but we have not verified it.

Human preference study (external raters, Prolific). SD3.5 depth, split-half (left s = 1, right s = 0), the same ids and seeds as the new cells; 100 pairs per comparison against the Table 1 baseline (vanilla ControlNet, activation scaling). Raters were told each image was generated in two halves and asked in which image the halves fit together better as one scene (no seam, consistent lighting, objects continuing across the middle), the property DreamSim measures; the author pilot above asked for overall preference. Each pair was rated by two different raters, one per order, with attention-check pairs and a pre-set exclusion rule (one of 10 raters excluded).

comparisonDisControlbaselinetieshare of decidedsign test p
LoRA (noise) vs ControlNet (scale)89494264%0.0008
ControlNet (noise) vs ControlNet (scale)78564658%0.07

Per pair (majority vote): 41 / 21 / 38 for the LoRA (p = 0.015) and 44 / 27 / 29 for the ControlNet (p = 0.057). No position bias (first-shown image picked 63 and 62 times of 180).

External raters prefer DisControl on the consistency between the two regions, significantly for the LoRA adapter and with a consistent lean for the ControlNet, in line with Table 1's DreamSim results.

Reviewer_PRJGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that ControlNet’s strength knob, which scales the adapter’s activations, is too coarse for per-region control. It proposes DisControl, which instead mixes noise into the control map per pixel and trains the adapter on these noisy inputs with the goal of getting better image quality, cleaner region boundaries and better adherence.

Critical strengths

The proposed method addresses limitations of ControlNet and allows for finer edits. It does this by adding one input channel at no additional cost at inference.

Critical weaknesses

The quality improvement is not clearly attributable to the proposed mechanism. The paper does not compare the two methods at global s = 1 or s = 0, or reports adherence to the control map inside the controlled region. The results could simply mean that DisControl produces better-looking images in general, or follows the control map less closely, rather than handling local modulation better. This is made harder to judge by the baselines, which are the authors’ own ControlNets trained with batch size 4 for under half an epoch, rather than the public checkpoints the paper already uses for its conversion experiment.

Two of the three headline claims are contradicted by the paper’s own results. The claim that region boundaries are “sharp by construction” is not supported by Table 2. As the measurement zooms in on the boundary, the separation between the two regions shrinks for both methods (DisControl 2.24 → 1.71, baseline 1.98 → 1.44), with a roughly constant gap between them. DisControl separates the regions somewhat better, but its boundary is not sharp either. Similarly, the abstract and Section 5.4 state that the spread in adherence is reduced across modalities, but Table 5 shows it is mixed for Canny and larger for normals at every level.

The qualitative results mostly show a region either following the control map or drifting slightly from it, and the quantitative results cover only on/off splits. The paper does not show use case beyond existing practice (masking the ControlNet output or the control map). It is therefore unclear what DisControl enables in practice beyond a somewhat more coherent on/off split, and even that is not fully established without absolute adherence numbers for each region (see first point above).

Questions for authors

Clarifications:

  • Are the main-table DisControl results from the model trained from scratch or the converted public ControlNet? Which setting does Figure 3 use?
  • In Table 1, SD3.5 depth and normals both report a baseline HPS of 10.187. Is that a typo?

Additional results that could change my assessment:

  • What are image quality and adherence for both methods at global s = 1 and s = 0? This is the main control needed to attribute the gains to the mechanism.
  • For the split-half and object-mask experiments, what is the absolute adherence error inside and outside the region for each method, not just the ratio?
  • How does DisControl compare with the public ControlNet checkpoints, and with blanking the control map outside the region?
  • Can you report at least one quantitative experiment with intermediate strengths in both regions or a smooth strength map? This is the application mentioned in the intro.
3 comments

Thank you for the detailed review. These are fair points. We ran additional experiments that help answer your questions, and we are happy to provide further explanation if anything remains unclear. Unless noted otherwise, the new cells use the paper's protocol: the same 1000 test ids and per-image seeds.

Image quality and adherence at global s = 1 and s = 0. Depth, whole image at one strength, both methods, same ids and seeds. Quality is HPSv3++ (higher is better); adherence is the linear-MSE between the depth re-extracted from the generated image and the control map (lower is better); the delta is paired per image. At s = 0 the vanilla ControlNet reduces to the base model; DisControl receives a fully noised map.

backbonesvanilla ControlNet HPSv3++DisControl HPSv3++delta (win rate)adherence vanilla / DisControl
SD1.5 (n=1000)16.927.54+0.62 (64%), p<0.00010.0034 / 0.0030
SD1.5 (n=200)06.237.34+1.11 (64%), p<0.00010.040 / 0.033
SDXL (n=200)19.849.76-0.09 (52%), p=0.670.0015 / 0.0022
SDXL (n=200)08.348.45+0.11 (56%), p=0.250.033 / 0.031
SD3.5 (n=200)110.8910.81-0.07 (45%), p=0.340.0017 / 0.0019
SD3.5 (n=200)012.1611.50-0.66 (41%), p=0.00060.030 / 0.029

Global quality does not account for the split-half gap. On SDXL and SD3.5 the two models are indistinguishable at global s = 1 and s = 0, and on SD1.5 the global difference (0.6 to 1.1) is well below the 2.7 split-half gap of Table 1. DreamSim is only defined under local modulation, where DisControl leads on all three backbones.

Absolute adherence inside / outside the region (vanilla / DisControl, n=1000): We report the absolute values of adherence that were used for the ratios.

backbonesplit-half insidesplit-half outsideSAM insideSAM outside
SD1.50.0082 / 0.00570.0278 / 0.02410.0091 / 0.00620.0287 / 0.0251
SDXL0.0054 / 0.00460.0264 / 0.02200.0066 / 0.00630.0274 / 0.0253
SD3.50.0055 / 0.00490.0253 / 0.02050.0061 / 0.00510.0249 / 0.0230

Public checkpoints in scale mode, same split-half protocol, maps from the estimator each model card specifies (adherence omitted because of the different estimator; the SDXL row uses the first 200 of the 1000 ids):

backbonearmHPSv3++DreamSimdelta vs public
SD1.5 depth (n=1000)control_v11f1p_sd15_depth, scale5.430.378
our vanilla ControlNet, scale4.640.479-0.79
DisControl, noise7.340.352+1.91 (74%), p<0.0001
SDXL depth (n=200)controlnet-depth-sdxl-1.0, scale5.800.348
our vanilla ControlNet, scale8.210.386+2.41
DisControl, noise9.310.307+3.51 (90%), p<0.0001

On SD1.5 the public checkpoint is the stronger baseline and we will add it to the table. On the training budget: batch size 4 follows the reference ControlNet training recipe in the diffusers examples (https://github.com/huggingface/diffusers/tree/main/examples/controlnet), both arms get the identical budget, and 100k steps is a budget at which the properties we measure (adherence, separation, coherence) are empirically stable across our checkpoint sweeps; larger-scale training could still improve absolute image quality, which we have not tested.

Blanking the map outside the region. With the vanilla ControlNet and the map zeroed on the s = 0 half, the zeroed-out half comes out as a flat solid or noisy texture instead of content. We will include more qualitative examples in the final version. HPSv3++ / adherence inside / adherence outside / DreamSim, adherence against the real map. The n=200 rows use the first 200 of the 1000 ids, so the split-half columns differ slightly from the full-set values in Table 1 main text:

backboneblankingvanilla split-halfDisControl split-half
SD1.5 (n=1000)1.37 / 0.028 / 0.057 / 0.6574.64 / 0.008 / 0.028 / 0.4797.34 / 0.006 / 0.024 / 0.352
SDXL (n=200)4.04 / 0.018 / 0.044 / 0.4888.21 / 0.005 / 0.026 / 0.3869.31 / 0.004 / 0.020 / 0.307
SD3.5 (n=200)4.64 / 0.024 / 0.049 / 0.63310.19 / 0.006 / 0.026 / 0.42110.10 / 0.005 / 0.021 / 0.307

Intermediate strengths. Split-half on SD1.5 depth (n=200, the first 200 of the 1000 ids), left half at s = 1, right half swept from 0 to 1; we report the right-half adherence, DreamSim between the halves and HPSv3++. Adherence is monotone in s for both methods. DreamSim varies much less for DisControl (0.35 to 0.29 over the sweep) than for scale modulation (0.48 to 0.29), where it grows as the two halves get more different strengths (right 0, left 1) and shrinks as they get more similar (both 1). This is in line with what we expected: local modulation with activation scaling trades quality for adherence, and DisControl largely removes that trade-off, so it can be used at intermediate strengths.

right-half s0.00.30.50.71.0
DisControl: adherence right0.02440.00580.00490.00400.0030
DisControl: DreamSim0.3560.3170.3100.2990.289
DisControl: HPSv3++7.608.188.238.197.97
vanilla: adherence right0.02920.01040.00520.00350.0031
vanilla: DreamSim0.4810.3920.3340.2980.287
vanilla: HPSv3++4.796.417.106.696.92

"Sharp by construction" and the spread claim. We agree the wording overstates Table 2 and will soften it. The convergence of the two regions at finer grids is expected for any method: depth is strongly correlated locally, so once the control information is removed on one side of the boundary, the other side still conditions the depth near it, and the measured separation must shrink with the grid size. Empirically, the baseline's separation is smaller at every grid size and the relative gap to ours is largest at the finest resolution (+13.5% at N = 16 against +18.5% at N = 128 on average). There are several reason to expect finer local modulation from our method: activation scaling acts on feature maps at the adapter's resolution grid, and there is no guarantee that a feature at a grid location is spatially aligned with the pixels it is meant to control, whereas degrading the input at the pixel level needs no such assumption, since the network is trained to estimate the conditional distribution given the degraded map directly. On the spread claim we will restrict the statement to the modalities where it holds.

Clarifications. The main-table and split-half results are the model trained from scratch on MONET. Figure 3 is a convergence study at global s = 1 (from scratch vs folded from the public checkpoint) that predates the MONET runs: both arms were trained on an earlier in-house Flux-generated corpus and evaluated on its own held-out split (different control maps and captions), which is why its absolute HPSv3++ level differs from Table 1. We will state this in the caption and regenerate the figure on MONET for the revision. The two 10.187 values are not a typo; the measurements coincide at three decimals, but differ in the individual values.

Since our response we have run a human preference study with external raters (Prolific) on SD3.5 depth under split-half modulation, 100 pairs per comparison against the Table 1 baseline (vanilla ControlNet, activation scaling), each pair rated by two different raters in both orders, with attention checks. Raters were told each image was generated in two halves and asked in which image the halves fit together better as one scene, the property DreamSim measures. They preferred DisControl on the LoRA adapter in 64% of decided ratings (89 to 49, 42 ties, sign test p = 0.0008) and on the ControlNet in 58% (78 to 56, 46 ties, p = 0.07). The full protocol and per-pair results are in our reply to Reviewer pe9W.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.