acceptodds
Withdrawn from ICLR 2027

DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To enhance the controllability of text-to-image diffusion models, current ControlNet-like models, have explored various control signals to dictate image attributes. However, existing methods either handle conditions inefficiently or use a fixed number of conditions, which does not fully address the complexity of multiple conditions and their potential conflicts. This underscores the need for innovative approaches to manage multiple conditions effectively for more reliable and detailed image synthesis. To address this issue, we propose a novel framework, DynamicControl, which supports dynamic combinations of diverse control signals, allowing adaptive selection of different numbers and types of conditions. Our approach begins with a double-cycle controller that generates initial real score sorting for all input conditions by leveraging pre-trained conditional generation models and discriminative models. This controller evaluates the similarity between extracted conditions and input conditions, as well as the pixel-level similarity with the source image. Then, we integrate a Multimodal Large Language Model (MLLM) to build an efficient condition evaluator. This evaluator optimizes the ordering of conditions based on the double-cycle controller's score ranking. Our method jointly optimizes MLLMs and diffusion models, utilizing MLLMs' reasoning capabilities to facilitate multi-condition text-to-image (T2I) tasks. The final sorted conditions are fed into a parallel multi-control adapter, which learns feature maps from various visual conditions and integrates them to modulate ControlNet, thereby enhancing control over generated images. Through both quantitative and qualitative comparisons, DynamicControl demonstrates its superiority over existing methods in terms of controllability, generation quality and composability under various conditional controls.

Primary area:
generative models
Keywords:
controllable image generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5bh5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DynamicControl addresses text-to-image generation with multiple visual conditions derived from the same subject. A double-cycle controller assigns training supervision using condition-reconstruction consistency and CLIP similarity to a source image. An MLLM evaluator learns condition rankings, a learned threshold selects a subset, and a multi-control adapter combines the selected features to guide Stable Diffusion. The evaluator is trained first and then frozen for diffusion-model training. Experiments cover multi-condition generation on MultiGen-20M and Subject-200K, single-condition tasks including segmentation on ADE20K and COCO-Stuff, ablations, computational cost, and a user study.

Critical strengths
  1. A useful distinction between accepting multiple conditions and deciding which ones to use. The proposed evaluator turns condition choice into a learned, example-dependent decision rather than simply aggregating every available signal. Distilling source-dependent consistency scores into an evaluator that does not require the source image at inference is a plausible approach. The specific contribution is learned subset selection: AnyControl already accommodates varying numbers and modalities through attention (Sun et al., 2024, Sec. 3.2; arXiv:2406.18958), so the fixed-number characterization in the introduction overstates that distinction. The proposed selection mechanism remains potentially useful, but its novelty should be framed at this narrower level.
  2. Substantial empirical coverage with improvements on several complementary measures. Table 1 reports better FID, CLIP score, SSIM, and MUSIQ on both multi-condition datasets, including comparisons with PixelPonder and OmniControl2. Table A1 also provides direct control-fidelity measures: ADE20K mIoU rises from 43.64 for ControlNet++ to 48.56, and depth RMSE falls from 28.32 to 23.21. Tables 3 and A1 average four generation groups. The 30-participant, 50-image user study (Appendix B.4; Fig. A1), additional qualitative results, and cost/model-size comparisons (Tables A2–A3) provide evidence beyond a single automatic metric. This is a substantial experimental effort, although it does not yet isolate the proposed selection mechanism.
Critical weaknesses
  1. The central selection mechanism is underspecified, and its conflict-resolution rationale is not established. Sections 3.1–3.4 do not specify how heterogeneous condition losses are normalized and combined into ranking targets, what the targets and probabilities in L_eval represent, or how the threshold is parameterized and optimized through discrete selection. The gradient path from L_condition and L_image to the evaluator is also unclear when the teacher generator receives each condition independently. Appendix A adds implementation settings but not this missing algorithm. Moreover, Eq. (2) measures CLIP embedding similarity, not the pixel-level similarity repeatedly claimed; the abstract describes joint optimization although Sec. 3.4 freezes the evaluator before diffusion training. The teacher scores each condition in isolation. A strong individual score does not establish that a condition adds information to a selected subset or avoids conflict with another condition. The evaluator can observe all inputs, but the paper does not show that its supervision captures these interactions, nor that the teacher rankings remain useful for the final multi-condition generator. Source-image resemblance also need not coincide with satisfying an edited prompt. The same-subject scope is reasonable, and cross-subject composition can remain future work; what is missing is targeted evidence about redundancy or conflict within that stated scope, sufficient to support the claim that selected conditions are mutually beneficial and conflict-free.
  2. The experiments support overall performance more strongly than adaptive-selection benefits, and key comparison details remain unclear. Table 4 supplies cumulative loss ablations, and Fig. 7 compares fixed counts with adaptive selection, but the latter shows one prompt and gives no clear sample count or variability. There is no reported dataset-level comparison against all-condition fusion, random or fixed-top-k selection, and a simpler learned scorer under the same adapter and training budget. Table A4 is particularly informative: removing MoE or criss-cross attention changes each metric by at most 0.01, leaving their claimed architectural importance unsupported at the reported precision. Ranking accuracy, selected-count distributions, and quality versus retained-condition count would help distinguish useful selection from benefits of additional consistency training or model capacity. The evaluation protocol also needs reconciliation. Table 1 defines “all” as using every condition, leaving unclear whether DynamicControl actually selects a subset. Its MultiGen-20M FID is 10.24, versus 12.01 in Tables 2/A3/A4, while CLIP scores use markedly different ranges across tables without a stated implementation or scaling. Appendix A describes both a merged validation/test split and validation-only evaluation. The recent Table 1 baselines lack detailed checkpoint and conditioning configurations. These are unresolved reporting differences, not evidence that the results are invalid, but they limit fair comparison and attribution. In particular, the Subject-200K FID difference from PixelPonder (10.28 versus 10.61) should be accompanied by variability rather than an unqualified superiority claim.
Questions for authors
  1. Can the authors provide executable pseudocode for teacher-score normalization, ranking supervision, evaluator outputs, and threshold learning, identifying the trainable modules and gradient paths in each stage? Please clarify whether “all” means all available or all retained conditions and demonstrate, on same-subject examples, that the selected subset improves joint control rather than merely collecting high-scoring individual conditions.
  2. Can the authors report a matched, dataset-level comparison of adaptive selection, all-condition fusion, random/fixed-top-k subsets, and a simpler scorer, including selected counts, seed variability, and runtime? Please reconcile the split and metric definitions across Tables 1–4/A3–A4 and document the recent baselines’ configurations; these clarifications would determine whether the reported improvements support the proposed selection mechanism.
Reviewer_9uWAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing controllable T2I diffusion models handle multiple visual conditions inefficiently: they either activate one condition at a time (costly) or use a fixed number of conditions, ignoring conflicts and varying importance. To address this, the authors propose DynamicControl, a framework that adaptively selects varying numbers and types of conditions. It comprises: (1) a Double-Cycle Controller (training-only, uses source images to compute ground-truth rankings); (2) an MLLM-based Condition Evaluator (fine-tuned via LoRA, predicts rankings at inference without source images); and (3) a Multi-Control Adapter (uses MoE and cross-attention to encode selected high-scoring conditions).

Main Contributions

New Insight & Dynamic Selection: Reveals that prior methods underutilize multiple conditions and proposes a dynamic selection scheme to avoid deviations from input conditions.

Efficient Condition Evaluator: Leverages an MLLM supervised by the double-cycle controller to rank conditions efficiently at inference without requiring source images.

Flexible Multi-Control Adapter: Introduces alternating multi-control fusion and alignment blocks to adaptively capture features from a dynamic number of conditions.

Strong Results: Comprehensively outperforms existing methods across diverse conditional controls.

Validation

Evaluated on MultiGen-20M and Subject-200K across five conditions (canny, hed, segmentation, openpose, depth):

Quantitative: Outperforms baselines (T2I-Adapter, Uni-ControlNet, ControlNet++, etc.) on FID, CLIP Score, SSIM, and MUSIQ.

Qualitative: Gradually adding high-scoring conditions yields increasingly refined and text-aligned images.

Ablation: Confirms the effectiveness of the loss functions and adaptive selection scheme over fixed-number baselines.

Efficiency: Avoids exponential inference time growth of serial processing (~34s on 40G GPU).

Critical strengths
  1. A novel and well-motivated dynamic condition selection paradigm. The paper identifies a genuine bottleneck in controllable T2I generation: existing methods either serialize conditions (costly) or fix their number (inflexible), ignoring inter-condition conflicts and varying importance. The proposed DynamicControl directly addresses this by adaptively selecting conditions based on consistency scores. This is consequential because it shifts the problem from "how to fuse all conditions" to "which conditions to trust," which is a more principled and practical formulation. The strong quantitative gains (e.g., FID 10.24 vs. 11.85 for PixelPonder; CLIP 82.45 vs. 78.60) and user study results (>75% preference) substantiate that this paradigm shift delivers real improvements.

  2. An elegant solution to the inference-time source-image problem via MLLM distillation. The double-cycle controller provides reliable ground-truth rankings during training but cannot operate at inference (no source image, high generative cost). The authors' use of an MLLM, fine-tuned via LoRA under the controller's supervision, is a clever and technically sound bridge. This is consequential because it makes the dynamic selection scheme actually deployable—the evaluator runs efficiently without source images or repeated diffusion sampling. The ablation (Table 4) confirms that adding the LLM loss yields the best FID/CLIP scores, validating that this distillation is not merely an engineering convenience but a performance-critical component.

Critical weaknesses
  1. The method's effectiveness is highly dependent on the MLLM evaluator's judgment, with no guarantee against incorrect rankings. The entire dynamic selection mechanism hinges on the MLLM's ability to rank conditions accurately. While the double-cycle controller supervises training, at inference the MLLM operates independently. If the evaluator misranks a condition (e.g., assigns a high score to a conflicting or low-information condition), the Multi-Control Adapter will blindly encode it, leading to degraded generation. The paper provides no analysis of the evaluator's failure cases or robustness—e.g., what happens when all conditions receive low scores, or when the MLLM is presented with ambiguous prompts. This is consequential because the core contribution's reliability rests entirely on an unexamined component; a single misranking could cascade into a poor image, and the paper does not quantify how often this occurs.

  2. The training pipeline is complex and resource-intensive, with no cost analysis. DynamicControl requires a two-stage training process: first, generating images for all conditions via a pre-trained model to compute double-cycle rankings; second, training the MLLM evaluator with LoRA; third, training the Multi-Control Adapter. This is significantly more complex than end-to-end training of ControlNet variants. The paper reports inference efficiency (Table A2) but entirely omits the computational cost of the training pipeline—GPU hours, memory requirements for the double-cycle generation phase, or the cost of fine-tuning the MLLM. This is consequential because reproducibility and practical adoption are hindered; a reader cannot assess whether the gains justify the training overhead, and the paper's claim of efficiency is therefore incomplete.

Questions for authors

Clarification Requests

Evaluator failure cases: What happens when the MLLM evaluator assigns low scores to all input conditions, or when the top-ranked condition is still conflicting with the text prompt? Does the threshold learning mechanism guarantee at least one condition is selected? Please provide an analysis or example of such failure modes.

Threshold optimization details: The paper states the selection threshold is a learnable parameter. Through which loss term is it optimized, and how does it behave across different datasets (e.g., does it converge to a similar value for MultiGen-20M vs. Subject-200K)? A brief clarification would help assess the method's robustness.

Training cost disclosure: Can you provide the training time, GPU memory, and number of GPU hours required for (a) the double-cycle ranking generation phase and (b) the MLLM evaluator fine-tuning? This is needed to evaluate the method's practical efficiency claims.

Suggestions for Future Work

Robustness to misranking: A systematic study on how the Multi-Control Adapter behaves when the evaluator is intentionally perturbed (e.g., random shuffling of rankings) would quantify the sensitivity of the pipeline to evaluator errors and strengthen confidence in the approach.

Mismatched conditions: The paper acknowledges limitations with completely mismatched conditions (Fig. 8). Exploring whether the evaluator can explicitly detect and reject such conflicts—rather than forcing a combination—would be a valuable extension.

Reviewer_Q2b9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DynamicControl for text-to-image generation with multiple visual conditions. It learns to select useful conditions and adapt their number for each input. A double-cycle controller provides ranking supervision based on condition consistency and image similarity. An MLLM learns to predict the ranking. The selected conditions are then fused to guide image generation. Experiments on multiple datasets show gains over the compared methods.

Critical strengths
  1. Adaptive condition selection is a useful idea. It allows the model to choose both the conditions and their number for each input, instead of relying on a fixed combination.
  2. The method improves overall generation results across multiple datasets. The reported results support its practical value for multi-condition image generation.
Critical weaknesses
  1. Important details of the ranking supervision are missing. L_cond can take different forms for different condition types. However, the paper does not explain how these losses are made comparable. For example, depth errors and edge errors may have different scales. Figure 3 shows that L_cond and L_img are added to produce ranking scores. However, it remains unclear how these scores are converted into target condition-token sequences or the targets c_i used in L_eval. These details are needed to reproduce the training process.
  2. The central motivation needs stronger evidence. The paper argues that multiple conditions can interfere with each other. However, it lacks a broad set of paired examples showing that adding a condition harms generation. Figure 7 provides limited visual evidence, and the sample size behind its scores is unclear. More examples should identify which conditions conflict and show the results before and after adding them. Statistics on how often this happens would also help establish the importance of the problem.
  3. The need for some adapter modules is not supported by the ablation results. In Table A4, the full model has an FID of 12.01. Removing MoE gives 12.02, while removing criss-cross attention gives 12.00. These results show almost no benefit from either module. The paper should explain why the added complexity is needed. The MoE experts and routing mechanism also need a clearer description.
Questions for authors

Please address the concerns raised in Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.