acceptodds
Withdrawn from ICLR 2027

MoLaGen: Multimodal Conditional Layout Control for Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In the layout-to-image generation task, instances are typically guided by local prompts and spatial coordinates. Since text-based conditions alone are often insufficient for precise instance-level specification, incorporating a reference image for each instance enables more accurate and fine-grained conditional description. To achieve such multimodal conditional layout control, it is essential to encode and fuse visual, textual and spatial modalities, thereby enabling effective instance-level conditioning. Most existing approaches rely on separate encoders and fusion modules for multimodal conditioning, which may lead to representational discrepancies with the base generative model. Instead, we leverage the MM-DiT model itself to encode local prompts and reference images, and concatenate the resulting tokens as unified generation conditions. For the spatial modality, we propose a spatially-aware CondLoRA fine-tuning network that introduces spatial-specific mapping weights and employs token-dependent routing to activate different CondLoRA branches for spatial control. Finally, we propose the **M**ultim**o**dal Conditional **La**yout Controllable **Gen**erative Network (MoLaGen). In addition, we introduce a new dataset construction pipeline for synthetic data generation, instance-level annotation, and data filtering. Qualitative and quantitative experiments demonstrate the effectiveness of our method.

Primary area:
generative models
Keywords:
generative model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mhvXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduce a multi-modal conditional control for layout-to-image generation. The conditions include a global prompt, per-instance images, local descriptions, and bounding boxes. The multimodal conditions are tokenized by MM-DiT backbone, avoiding the modality misalignment in the feature space. It also introduces a network for spatial control, which incorporates spatial embeddings into low-rank adaptation. To train the model, the authors generate 400K synthetic training images. And the method is evaluated on COCO-MIG and MS-Bench.

Critical strengths

The proposed method is a great unification of textual, spatial, and visual control. Integrating these conditions within an existing multimodal backbone is effective and practical.

Critical weaknesses
  • Missing failure cases and failure-pattern analysis. The paper presents successful examples without systematically examining limitations. Please include representative failures and quantify performance across challenging conditions, such as overlapping boxes, small objects, and conflicts between reference appearance and local descriptions.

  • Missing comparisons with current SOTA VLM baselines. Current Claude and GPT models can also take the specialized layouts for image generation. Please also compare with the these VLMs with same conditions and prompts. if a VLM such as Claude needs external adapter to achieve the same generation quality as your method, it deserves to be called out.

  • Unclear ability to follow long, detailed instance descriptions. The evaluation does not establish whether MoLaGen preserves fine-grained local instructions as description length and complexity increase, particularly when reference images already convey much of the requested appearance. Previous work Rethinking the Training and Evaluation of Rich-Context Layout-to-Image Generation (Cheng et al., NeurIPS 2024) already addressed the long description challenge and offered evaluation method. Please have a discussion or evaluation on this.

Questions for authors
  • How much does each conditional modality contribute to the final performance? The manuscript lacks the ablation study on the component effectiveness including spatial CondLoRA, modality-specific branches and instance isolation.

  • Can the model follow local text when it requests an attribute or pose change relative to the reference image?

Reviewer_5UtyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoLaGen, a framework for multimodal layout-controlled image generation that combines instance-level reference images, local text descriptions, and bounding boxes. Built on FLUX.1 Kontext, the method introduces CondLoRA to incorporate spatial information into low-rank adaptation, with separate visual and textual branches activated through modality-specific routing. An instance-isolated attention mask restricts interactions between conditions belonging to different objects to reduce cross-instance interference. The authors also develop a three-stage pipeline for constructing approximately 400,000 synthetic training images, covering image synthesis, instance-level annotation, and quality filtering. Experiments on COCO-MIG and MS-Bench assess spatial alignment, semantic consistency, and reference-image similarity, with reported improvements on several layout-control metrics over the evaluated baselines.

Critical strengths
  1. The paper addresses a relevant problem: jointly controlling instance appearance, semantics, and spatial layout using reference images, local prompts, and bounding boxes.

  2. The method is clearly presented, with diagrams explaining CondLoRA, modality-specific routing, and the attention mask.

  3. The MS-Bench results show improved spatial alignment over the evaluated baselines, supported by qualitative examples of multi-instance generation.

Critical weaknesses
  1. The technical novelty is insufficiently established. InstanceAssemble[1] already combines bbox control, textual/visual conditioning, and lightweight LoRA adaptation, yet it is neither discussed nor compared against. CondLoRA’s formulation amounts to a LoRA update plus a low-rank spatial conditioning term; its substantive advantage over existing designs remains unclear.

  2. No ablations establish the contribution of the proposed components. Spatial conditioning, modality-specific routing, and the instance-isolated attention mask are introduced without controlled comparisons. The experiments therefore cannot distinguish architectural gains from the effects of the Kontext backbone and newly constructed training data.

  3. The evaluation does not convincingly establish layout-control superiority. COCO-MIG mixes methods with different input conditions and leaves the evaluation reference-image construction unspecified. The marginal SR/I-SR gains over GLIGEN are not corroborated by OverLayBench[2] or LayoutSAM-Eval[3], while relevant comparisons with Laytrol[4], InstanceAssemble[1], and Ideogram 4[5] are absent.

  4. The claimed efficiency advantage is unsupported. The paper reports neither added parameter counts nor inference latency, despite using rank-512 LoRA, an additional spatial rank of 128, and separate visual/text branches. Without quantitative comparisons, parameter efficiency and low computational overhead remain assertions rather than demonstrated contributions.

[1] InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention, NeurIPS 2025.

[2] OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps, NeurIPS 2025.

[3] CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, ICCV 2025.

[4] Laytrol: Preserving Pretrained Knowledge in Layout Control for Multimodal Diffusion Transformers, AAAI 2026.

[5] Ideogram 4.0 Technical Details: Open model at the forefront of design.

Questions for authors
  1. Novelty and ablations: What distinguishes CondLoRA from existing LoRA-based layout-conditioning approaches such as InstanceAssemble? Can you provide controlled ablations isolating spatial conditioning, modality-specific routing, and the attention mask while keeping the backbone and training data fixed?

  2. COCO-MIG protocol: How are the evaluation reference images generated, and do all image-guided methods receive identical references? Does reference construction use original COCO images or only the benchmark’s text descriptions?

  3. Layout-control evidence: Can you substantiate the claimed layout-control advantage through comparisons with Laytrol and InstanceAssemble, and evaluation on complex layouts using OverLayBench or LayoutSAM-Eval? Ideogram 4 would also be a relevant comparison for the text-and-box setting.

  4. Efficiency: What are the total added/trainable parameter counts and inference latency under matched hardware, resolution, sampling steps, and instance counts? What evidence motivates rank 512, and how does reducing the rank affect performance and computational cost?

Reviewer_C8taOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-instance image generation conditioned on a global prompt and instance-level tuples of reference images, local prompts, and bounding boxes. Built on FLUX.1 Kontext, MoLaGen encodes bounding boxes using Fourier features and an MLP, then injects these spatial features into reference-image and local-text transformations through CondLoRA. The method uses separate visual and textual adapters and an attention mask restricting interactions between instance conditions. The authors also construct a synthetic training dataset of approximately 400,000 images and evaluate layout accuracy and reference consistency on COCO-MIG and MS-Bench.

Critical strengths
  1. Clear layout improvements with competitive reference similarity. On MS-Bench, MoLaGen improves mIoU from 60.21 to 66.09 and AP from 32.99 to 46.33 over MS-Diffusion. DINO and CLIP-I also improve from 0.425 and 0.698 to 0.455 and 0.705, respectively. Although similarity remains below ContextGen, these results support a favorable combination of layout accuracy and reference consistency. The comparisons include relevant methods such as ContextGen, LAMIC, and MS-Diffusion.

  2. A clear conditioning interface and potentially valuable dataset. Explicitly associating each reference image and local description with a target box provides a practical control interface. The dataset covers 645 fine-grained categories and aims to vary backgrounds, poses, and viewpoints while preserving identity.

Critical weaknesses
  1. The necessity of CondLoRA and modality-specific branches is insufficiently established. The experiments evaluate the complete system but lack minimal controls against simpler conditioning. A priority comparison would use ordinary LoRA with corresponding bbox embeddings injected only at the input, while retaining the same backbone, data, training budget, conditioning inputs, routing scope, and attention mask. A shared-versus-separate adapter comparison at compatible locations, particularly single-stream blocks, would further test modality specialization. Efficiency claims also require matched inference-time or memory measurements rather than relying solely on the low-rank formulation.

  2. The dataset’s distinction from prior work and its identity–diversity quality need stronger evidence. Appendix A.1 incompletely characterizes prior construction strategies: ContextGen already includes semantic editing of reference images for pose and viewpoint changes. The distinction should therefore be stated more precisely. Moreover, generation prompts and CLIP/DINO filtering do not establish that references simultaneously preserve identity and exhibit meaningful foreground variation. A random manual audit should report the joint rate of preserved identity and actual pose/viewpoint changes, alongside background-only changes and near-duplicates. Human identity consistency should be examined separately within the person subset.

Questions for authors
  1. Discuss related work using bbox as direct input. For example, ConsistCompose (CVPR'26) supports multi-reference layout control by expressing coordinates in text. The central question is whether dedicated layer-wise spatial injection is necessary.

  2. Coverage of closely related baselines. MUSE uses a global prompt together with per-subject reference images, bounding boxes, and class text. It is cited but not experimentally compared. Could the authors provide a comparison or explain why it is unsuitable?

  3. Necessity of the attention mask. What motivates allowing instance conditions to attend to noisy-image tokens while blocking direct access to other instances and the global prompt? A controlled training ablation that enables only cross-instance condition attention would help establish whether hard isolation improves identity confusion and layout metrics over learned attention.

  4. RoPE assignments for multiple references. Section 3.1 describes Kontext’s positional convention, but the assignments for multiple reference images and local prompts remain unclear. Do references share positional indices, use distinct image indices, or receive spatial offsets? Do bounding boxes affect RoPE coordinates, or are they incorporated exclusively through CondLoRA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.