acceptodds
Under review as a conference paper at ICLR 2027

SpaDiff: Spatial Conditioning Adapter for Open-Vocabulary Segmentation-to-Image

Abstract

When creating visual designs, a highly desired feature is precise spatial control over image generation. Segmentation-mask-to-image (S2I) generation provides such control by binding a textual description, or regional prompt, to each specific image region present in the segmentation mask during generation. While open-vocabulary S2I methods allow arbitrary prompts, they fine-tune the entire base model, so other extensions trained on the original weights no longer combine with them reliably. Adapters, lightweight modules trained on top of a frozen base model, preserve this composability, but existing S2I adapters are restricted to closed vocabularies and built on older base models, transferring poorly to modern Multi-Modal Diffusion Transformers (MM-DiTs). We present , an open-vocabulary S2I adapter for the MM-DiT FLUX that keeps the base model frozen. SpaDiff combines two mechanisms: (1) an adaptive attention masking strategy that binds regional prompts to their corresponding image region by suppressing image and text tokens interactions softly with a learned per-timestep, per-layer, and per-head strength to avoid disrupting the frozen base model; and (2) a mechanism for adhering to exact mask boundaries, explored in two variants, along with a lightweight feature gating mechanism. SpaDiff achieves state-of-the-art performance among adapter-based methods on both the closed-vocabulary COCO-Stuff and open-vocabulary SACap-Eval benchmarks, while approaching the performance of methods that fine-tune the full base model.

Primary area:
generative models
Keywords:
Diffusion Models, Spatial Conditioning, Segmentation-mask-to-image, Computer Vision, Architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S7YsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The work addresses a method for segmentation-to-image generation, where segmentation maps are given along with the prompt to generate an image that should respect all conditioning. They add a set of trainable parameters to process the additional segmentation conditioning, and restrict the cross attention between the text and image / segmentation to improve the binding of target objects to target regions.

They evaluate how well the images match the layout conditioning, as well as image quality. They additionally ablate several key design decisions, including architecturally how the conditioning is integrated and whether masking is hard or soft.

Critical strengths

S1. Coverage of metrics. The work does a good job of measuring a variety of aspects (quality, respect to the segmentation) and ablations--if my concerns about the results themselves are resolved (see W2), then the work would have good experimental coverage.

Critical weaknesses

W1. Missing regions of related work that significantly impact novelty. This work does not mention or compare to very similar work in layout-guided generation, which is the same task but with larger boxes rather than tight segmentation. In fact, several works have already applied this to MM-DiT [1,2,3], using very similar mechanisms to the work presented here (and also to previous methods in training-free layout conditioning, like [4,5]). Soft vs hard binding has been explored as well [1]. This greatly impacts the novelty of the work presented here, which effectively takes similar mechanisms to [1,2,3] to bind text to regions, and applies IP-Adapters to add the additional conditioning. At the very least, the novelty of the work would need to be reframed. However, I am still concerned that the novelty is very limited, essentially combining two existing methods while providing little in terms of new insights.

Similarly, the justification of the problems statement feels somewhat weak to me. FLUX is no longer SOTA, and some newer models (e.g. FLUX-Kontext) just take additional conditioning as an input--adding conditioning after pre-training is quite a bit less important if SOTA models already apply it during pre-training.

W2. Poor experimental results and missing key comparisons. Some of the experimental results are very concerning (Table 1 and 2). Specifically, the proposed methods and other FLUX-based methods do not really out-perform SD1.5-based methods on the image quality metrics (FID, MAN-IQA). Given that FLUX is a (much) stronger model, these methods would be expected to perform much better, from the base model alone. The fact that they do not likely implies that likely either the metric is not able to accurately measure what is intended (which is problematic towards the soundness of the numerical results), or the methods are generating extremely poor-quality images (which is problematic towards the value of the method). Also, the work does not consistently beat comparative methods.

In addition, some key comparisons are missing to understand how the model truly compares to other work. First of all, ControlNet (which is a key baseline) is also available for FLUX, but is only compared with on SD1.5 in this work, which underestimates the accuracy it could be providing. Also, to differentiate between the layout-guided methods [1,2,3], ablations with their forms of binding to regions, soft/hard, etc would be important towards understanding how this work differentiates (which would also go towards justifying novelty, see W1).

References:

[1] Region-aware text-to-image generation via hard binding and soft refinement. Chen et al. ICCV'25.

[2] Stitch: Training-free Position Control in Multi-modal Diffusion Transformers. Bader et al. arXiv'25.

[3] Training-free regional prompting for diffusion transformers. Chen et al. arXiv'24.

[4] Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. Lian et al. TMLR'24.

[5] Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. Yang et al. ICML'25.

Questions for authors

Q1. Do you have any intuitions on why the FLUX-based models do not perform better than those based on SD1.5 on FID and MAN-IQA?

4 comments

We thank the reviewer for their time and for recognizing the coverage of our metrics and ablations. Below, we address the concerns:

Relation to [1-5] and novelty (Weakness 1)

We appreciate the given references, but we respectfully disagree that they address the same task with a similar approach:

Task

[1-5] are training-free methods for compositional text-to-image generation. Their regions are rectangular and usually derived from the prompt by an LLM; none is designed for, or evaluated on, a segmentation mask. Segmentation-mask-to-image (S2I) is a different task: the mask is the user's specification and the image must follow every contour (beginning of Section 3 contains a clear problem statement).

Method

[1-5] first turn the prompt into a coarse layout of rectangular regions, each with its own sub-prompt (produced by an LLM, or by the user in [1,3]). None of them learns any parameters. They differ in how the image is then generated:

  • [1] RAG: in the first timesteps, each sub-prompt is generated separately and pasted into its region ("hard binding"); in the remaining timesteps, each region is refined with a more detailed description and mixed into the image ("soft refinement").
  • [2] Stitch: in the first timesteps, each sub-prompt is generated inside its bounding box via Region Binding (three fixed hard attention-masking rules). The objects are then cut out, combined with a separately generated background, and generation continues without constraints.
  • [3] Regional Prompting: in the first timesteps, each regional prompt is bound to its region with fixed hard attention-masking rules, and the result is blended with the base-prompt latent using a manually chosen ratio. The remaining timesteps run without constraints.
  • [4] LMD (U-Net): a latent is generated for each box, and the object is kept inside by guiding the latent with an energy on the cross-attention maps. The object latents are then composed into a single latent that guides the final generation.
  • [5] RPG (U-Net): at every timestep, a latent is generated for each sub-prompt, resized to its region, concatenated with the others, and blended with the base-prompt latent.

Relation to SpaDiff

  • [1,4,5] use no attention masking and have no mechanism in common with SpaDiff.
  • [2,3] use fixed hard attention masking in the first timesteps. This is the kind of masking we include as the "hard" baseline in our ablation study of adaptive masking (Section 4.2.2). Such rules are also used by EliGen and Seg2Any, which we cite and compare against; we adopt them from Seg2Any and do not claim them as a contribution.
  • None has a mechanism for following the shape of a region, which SpaDiff provides through contour tokens.

Soft vs. hard in [1]

Only the wording coincides.

  • In [1], they name two consecutive stages that are always used together: 1) Hard binding (first timesteps): each object is generated on its own from a short description and pasted into its region of the image, replacing what was there; 2) Soft refinement (remaining timesteps): each region is refined using a more detailed description, and the result is mixed into the image with a hand-set weight instead of replacing it. The two stages thus differ in whether image content is replaced or mixed. No attention mask is involved, and nothing is learned.
  • In SpaDiff, "hard" and "adaptive" (soft) are two alternatives for the same operation: an attention mask with identical rules. Token pairs that should not interact receive either −∞ (hard) or a finite penalty that is learned per timestep, layer and head (adaptive). The two are compared against each other (Section 4.2.2), not combined.

The question we study, whether an attention mask on a frozen MM-DiT should block interactions or only suppress them, and at which timesteps, layers and heads, is therefore not explored in [1], which uses no attention mask.

IP-Adapter

SpaDiff does not use IP-Adapter anywhere. Contour tokens enter joint attention as a third stream, following OminiControl. IP-Adapter is mentioned only as the inspiration for our decoupled variant, an ablated alternative (described in Section 3.1 and evaluated in Section 4.2.1).

Problem statement (Weakness 1.2)

We agree that newer models merit consideration, but FLUX.1 Kontext conditions on a reference image and an edit instruction rather than on a segmentation mask with free-form regional descriptions, which is our task. More generally, any base model ships with a finite set of built-in conditions, and adapters remain the way to add further controls without retraining, which is our motivation. We use FLUX.1-dev to compare against existing S2I baselines and assess composition with OminiControl on a common backbone. Our contribution concerns adding regional control to a pretrained model; evaluating its transfer to newer backbones remains future work.

Image-quality metrics and comparison to baselines (Weakness 2, Question 1)

SD1.5-based methods are ahead only on FID in Table 1 and on MAN-IQA in Table 2, and each case has a specific reason. FID measures how close the generated images are to the images of the benchmark dataset, and FreestyleNet and PLACE fine-tune the full SD1.5 backbone on that dataset, whereas FLUX-based methods retain FLUX's own image distribution, entirely so in our case since the backbone is frozen; Seg2Any reports the same ordering in its paper. MAN-IQA depends on the resolution at which images are generated, and in Table 2 the SD1.5-based methods generate at a lower resolution than EliGen and Seg2Any; the same effect is visible when comparing SpaDiff at its two resolutions in that table. In the other table, each metric favours FLUX-based methods: FID in Table 2 and MAN-IQA in Table 1. Both metrics are therefore sound when read with this in mind, and both are widely used in diffusion literature. The methods are also not generating poor-quality images, as the qualitative results in Figures 3 and 7 show. Finally, SpaDiff does achieve the best results on several metrics (Table 2), but outperforming fine-tuned methods was never our goal: we aim to match their quality while keeping the backbone frozen, which gives SpaDiff the composability of an adapter (Table 6).

Missing key comparisons (Weakness 2.2)

To our knowledge, no public ControlNet for FLUX supports segmentation conditioning; the available ones cover conditions such as canny, depth and pose. Such a model would also be closed-vocabulary and could not be evaluated on SACap-Eval. [1, 2, 3] address prompt-driven compositional generation and not segmentation-mask-to-image. The only form of binding they share with our setting, the fixed hard attention masking of [2,3], is already covered by the hard-masking baseline in our ablation of adaptive masking (Section 4.2.2).

We hope this clarifies how SpaDiff differs from [1-5], and would be grateful to hear whether this addresses the reviewer's concerns.

W1. Referring to my original review, the characterization was 'but with larger boxes rather than tight segmentation', covering the same task distinctions that the authors refer to in the response. This is exactly why it is important to compare with these methods--because while the preciseness of the changes differs, the underlying mechanisms controlling generation are the same. It is quite likely that a method operating at the pixel-level would outperform one operating only on blocks of patches, but it is important to validate this, as it would be a significant problem if it is not better. It is also important to discuss methodological differences between the two related fields, to keep the wider field united and understand where differences in findings diverge (e.g. if it is because the specifics of the task really cause other solutions to fail, or if different sub-fields just investigated different things rather than relying on each other).

Regarding the difference from IP-Adapters, there seems to be no ablation covering the specifics of adapter construction. In my original review, I assumed that the details of the adapter were not an intended source of novelty, and applied the weakness therefore to novelty (as they were stated to have relations to IP-Adapters and OmniControl's methodology). However, if they are intended to be a source of novelty, then these design decisions would need to be systematically evaluated.

W1.2. However, segmentation maps could be another form of input to edit. For example, the model could be given the segmentation map as an input and asked to turn it into an image, thereby providing the same function. It would need to be shown that these models fail at this task. And while evaluating on FLUX.1 for comparison to other works is valid, it also remains a distinct limitation of the work that it is not clear how it adapts to newer models.

W2. Even if these metrics have been used before, does not mean they are appropriate for measuring image quality here. We know there to be a significant increase in image quality from SD1.5 to FLUX.1. If the presented metrics cannot measure that, then they are not sufficient for showing that the images are of good quality. Showing results on qualitative examples is not sufficient evidence of overall quality either, as the ability to present a small subset of good images is not the same as measuring systematic quality increase. Hence, it remains that there is no metric presented that can accurately capture the quality of the images, making their overall quality unclear.

Even with the goal of providing training-free alternatives, it is very problematic if that quality comes from upgrading to a significantly better backbone (as is the gap from SD1.5 to FLUX.1) and then providing modifications that degrade performance back to the level of the much older backbone. As it stands, there is no quantitative results giving evidence that this is not the case. This is why it is important to measure this, with a metric that can accurately capture quality differences.

W2.2. Thank you for the clarification.

We thank the reviewer for the follow-up.

Comparison with layout-guided binding (W1)

“while the preciseness of the changes differs, the underlying mechanisms controlling generation are the same.”

The underlying mechanisms of [1-5] are not all the same, as detailed in our first reply.

“It is quite likely that a method operating at the pixel-level would outperform one operating only on blocks of patches, but it is important to validate this, as it would be a significant problem if it is not better.”

Where the mechanisms are similar, our paper already contains the comparison: the hard-masking baseline of our adaptive masking ablation (Section 4.2.2) applies such fixed masking rules to the frozen FLUX backbone and performs poorly compared to adaptive masking (15.08 vs. 23.37 MIoU, 37.77 vs. 29.89 FID). More generally, Table 5 as a whole validates the reviewer's concern: none of its variants uses conditional integration, so all of them bind text to regions through attention masking alone, i.e. at the level of blocks of patches. The best of them reaches 23.37 MIoU, whereas adding contour tokens raises this to 43.39 in the same setting (Table 6), confirming that pixel-level control is needed for S2I.

Our comparison also includes the baseline closest to layout-guided generation. EliGen receives the layout directly instead of inferring it from the prompt as [1-5] do, but it is trained on rectangular regions and does not integrate the contours of the segmentation mask in any way, so it also operates on blocks of patches. As noted in Section 4.1, it therefore supports only loose positioning and reaches 52.86 MIoU on SACap-Eval, compared to 94.02 for SpaDiff (Table 2).

Adapter construction (W1)

“Regarding the difference from IP-Adapters, there seems to be no ablation covering the specifics of adapter construction”

We understand "adapter construction" as the way the contour conditional tokens are integrated into FLUX (Section 3.1), since this is where our design relates to IP-Adapter and OminiControl. This conditional integration is ablated systematically: we compare unified integration, where the condition tokens form a third stream in joint attention as in OminiControl, with decoupled integration, where they are processed by a parallel attention operation inspired by IP-Adapter's decoupled attention, each combined with hard and adaptive masking (Table 4). For the decoupled variant, we further ablate how its output is gated by ZeroScale (Section 4.2.1, Table 3).

“if they are intended to be a source of novelty, then these design decisions would need to be systematically evaluated.”

We do not claim the two integration schemes as novel, as they follow OminiControl and IP-Adapter; our contributions relate to ZeroScale and adaptive masking, whose design is ablated in Table 5.

Editing models and newer backbones (W1.2)

“segmentation maps could be another form of input to edit [...] It would need to be shown that these models fail at this task.”

With an editing model, the input would be a colour-coded segmentation map, and the prompt would have to state which description belongs to each colour. We expect this to work poorly as the number of regions grows, since every binding between a colour and its description has to be resolved from the prompt alone. As this setup is far from S2I, where each region is bound to its description directly, we do not consider this comparison relevant for our paper.

“while evaluating on FLUX.1 for comparison to other works is valid, it also remains a distinct limitation of the work that it is not clear how it adapts to newer models.”

We agree that transfer to newer MM-DiTs has not been tested, and we acknowledge this as a limitation; since SpaDiff relies only on joint attention over separate token streams, it does not depend on properties specific to FLUX.1.

Image-quality metrics (W2)

“it remains that there is no metric presented that can accurately capture the quality of the images, making their overall quality unclear.”

We agree that FID and MAN-IQA do not always capture the gap between the backbones. Table 2, however, also reports two metrics trained on human preferences, PickScore and ImageReward, and these do capture it: all FLUX-based methods score above the SD1.5-based ones (ImageReward 0.40 to 0.77 vs. -1.01 and -0.84; PickScore 21.66 to 22.30 vs. 19.94 and 20.09), and SpaDiff lies within the FLUX range on both (0.56 and 21.84). These scores are computed over the full evaluation set, not a selection of examples, and provide the requested quantitative evidence that SpaDiff does not degrade FLUX to the level of SD1.5.

We hope these clarifications address the remaining concerns.

Reviewer_esiMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds an open-vocabulary segmentation-to-image adapter on FLUX while keeping the base frozen, so that it can be stacked with other adapters; it follows Seg2Any for mask-to-contour encoding and OminiControl for parameter reuse. Three pieces are contributed: a unified versus decoupled conditional stream, a ZeroScale per-feature gate of 175K parameters in place of ControlNet's 537M matrix, trained in two phases, and an adaptive attention mask that replaces the hard −∞-\infty on disallowed text–image pairs with a learned penalty per (timestep, layer, head). Evaluation is on COCO-Stuff and SACap-Eval with MIoU, FID, MAN-IQA and VQA-based regional scores, plus ablations of the gate, the integration type and the three mask-conditioning factors, and qualitative multi-conditioning with OminiControl.

Critical strengths
  • The gap is real and the design is parsimonious: a third MM-DiT stream with a LoRA active only on the conditional path, and a gate that replaces a d×dd\times d matrix with a per-feature vector. The two-phase schedule is justified by an observed failure (the gate stays near zero when trained jointly), not asserted.
  • The adaptive mask is a very low-dimensional intervention with a large payoff (MIoU 15.08→23.37 without conditional integration), and the factorial ablation over tt, ℓ\ell, hh isolates which factors matter and turns the learned parameters into a claim about where layout is decided in FLUX.1.
Critical weaknesses
  • Composability, the paper's entire reason for existing, is never measured. The multi-conditioning evidence is three qualitative figures; there is no test that SpaDiff's layout fidelity survives with OminiControl loaded, that OminiControl's adherence survives with SpaDiff loaded, or that either degrades less than a baked-in method would.
  • The main comparison is confounded. SpaDiff is trained at 5122512^2 and evaluated at 102421024^2, and MIoU is the metric most sensitive to resolution, yet the gap to Seg2Any (85.41 vs 95.03) is attributed to resolution without a control. Meanwhile four of seven baselines run on SD1.5, so beating them reflects the backbone, and the only comparable opponent wins on MIoU on both benchmarks — "SOTA among adapters" holds only because no other open-vocabulary S2I adapter exists on FLUX.1.
Questions for authors
  1. Please report Seg2Any at 5122512^2 (or SpaDiff at 102421024^2). How much of the 10-point SACap MIoU gap is resolution?
  2. Report MIoU with and without OminiControl loaded, and edge-F1 or depth-RMSE with and without SpaDiff loaded, against a baked-in method under the same protocol.
  3. Add the missing control for Table 5: a single global learnable scalar aa with no (t,ℓ,h)(t,\ell,h) dependence. How much of the gain comes from softening rather than from the learned dependence?
2 comments

We thank the reviewer for a thorough review, for recognizing the gap that SpaDiff addresses, and recognizing the contributions we made as well as the useful take-aways the ablation study brings. The suggested experiments have strengthened the paper, and we address the concerns below.

Resolution and the MIoU gap to Seg2Any (Weakness 2.1, Question 1)

We thank the reviewer for the observation that the gap in MIoU mentioned in the paper must be supported by an experiment. We would first like to clarify that the evaluation protocol was not confounded: it has followed the same native-resolution rule since the submitted version. We distinguish two resolutions:

  • Generation resolution: each method generates at its native training resolution (PLACE and FreestyleNet at 512×512, EliGen and Seg2Any at 1024×1024).
  • Evaluation resolution: following the official SACap-Eval protocol of Seg2Any, all generated images are resized to the benchmark's 1024×1024 resolution, and MIoU is computed against the same ground-truth masks for every method.

The sole exception was SpaDiff, trained at 512×512 but evaluated using 1024×1024 generations. This placed SpaDiff at a disadvantage rather than favoring it. We agree that attributing the gap to resolution required a control and we have now tested it. Generating at SpaDiff’s native resolution raises MIoU from 85.41 to 94.02, closing 89.5% of the gap to Seg2Any (95.03; updated Table 2). SpaDiff also achieves the best FID (15.47 vs. 17.88 for Seg2Any), CLIP, and Color/Shape/Texture scores. We did not evaluate Seg2Any at 512x512, as that would move it off its own training resolution.

Changes in the revision:

  • The native-resolution SpaDiff result is now reported as the main result (Table 2), with the 1024 result kept in grey as a generalization check.
  • The evaluation protocol is now stated explicitly in Section 4, and the untested attribution in Section 4.1 is replaced by these numbers.

Backbone influence and “SOTA among adapters” claim (Weakness 2.2)

We agree that comparisons with SD1.5-based methods cannot isolate architectural improvements from differences in backbone capabilities. These methods are included to provide broader context, following the evaluation protocol of Seg2Any, rather than to establish backbone-independent superiority. Since these approaches were designed for their respective backbones, directly transferring them to FLUX would require substantial methodological changes and would no longer constitute a direct comparison with the original methods. We also agree that “SOTA among adapters” overstates what the current comparisons establish.

Changes in the revision

  • We revise the SOTA claim to more accurately reflect “the first open-vocabulary segmentation-to-image adapter for MM-DiTs” in abstract, introduction and conclusion.

Global scalar control for Table 5 (Question 3)

A single suppression value aa shared across all timesteps, layers and heads, trained as in Table 5, converges to a≈0.2a \approx 0.2, effectively disabling the mask and reverting to base FLUX:

MethodMIoU↑FID↓CLIP↑
Hard masking15.0837.7727.4
Full (t,ℓ,ht,\ell,h)23.3729.8929.7
Timestep only12.4436.0930.4
Layer only11.8336.9330.3
No per-head (t,ℓt,\ell)17.2232.6329.7
Scalar (new)10.7137.7030.5

The scalar has the best prompt alignment but the lowest MIoU: it no longer binds regions. MIoU rises with each learned factor (10.71 → 23.37), so the gain comes from learning when, where and how strongly to suppress, not from softening alone.

Changes in the revision:

  • New "Scalar" row in Table 5 and qualitative examples in App. C
  • Discussion of the result in Section 4.2.2

Composability with OminiControl (canny).

Each S2I method is evaluated alone and combined with OminiControl; values in parentheses show the change caused by loading the second adapter. Bold = best, italic = second best.

COCO-Stuff

MethodMIoU ↑Edge F1 ↑Edge Prec. ↑FID ↓MAN-IQA ↑
SpaDiff43.3933.0839.5619.5254.00
Seg2Any37.7934.0136.9920.0456.81
OminiControl33.3568.5372.5418.3952.08
SpaDiff + OminiControl40.72 (−2.67)69.00 (+0.47)67.84 (−4.70)15.4653.75 (−0.25)
Seg2Any + OminiControl38.29 (+0.50)68.51 (−0.02)63.73 (−8.81)15.2954.23 (−2.58)

ADE20K

MethodMIoU ↑Edge F1 ↑Edge Prec. ↑FID ↓MAN-IQA ↑
SpaDiff52.3532.0535.3931.3555.01
Seg2Any45.6932.1632.8335.1658.56
OminiControl40.2367.6567.5524.4053.62
SpaDiff + OminiControl52.05 (−0.30)66.42 (−1.23)62.86 (−4.69)23.1654.39 (−0.62)
Seg2Any + OminiControl46.65 (+0.96)65.80 (−1.85)58.52 (−9.03)24.1455.24 (−3.32)

Composability gap (Weakness 1, Question 2)

We agree that composability was supported only qualitatively, and ran the requested experiment (at the end of the above comment).

  • Protocol: We combine OminiControl (Canny) with SpaDiff and Seg2Any on COCO-Stuff (in-distribution) and ADE20K (out-of-distribution), all conditioning strengths at their default of 1 with no tuning and generate with a single seed the full validation set of each dataset. Global captions are used so OminiControl receives the text prompt it was trained with; this is why standalone MIoU differs from Table 1 (Seg2Any's COCO-Stuff checkpoint was trained without global captions). We therefore compare each method with and without the second adapter rather than absolute values. For Seg2Any we used the most favourable integration: each token group is routed to its own LoRA, so canny tokens run on base FLUX plus the OminiControl LoRA, while image and text tokens use Seg2Any's backbone LoRA.
  • SpaDiff's layout fidelity survives. MIoU changes by -0.30 on ADE20K and -2.67 on COCO-Stuff with OminiControl loaded, and stays well above OminiControl alone (+11.8 and +7.4). We attribute the COCO-Stuff drop to the two sets of condition tokens competing in joint attention at untuned strength, consistent with the limitation stated in Section 5.
  • OminiControl's adherence survives. Edge F1 changes by -1.23 (ADE20K) and +0.47 (COCO-Stuff) with SpaDiff loaded, and FID improves over either adapter alone on both datasets.
  • Comparison with the baked-in method. On this pair Seg2Any also composes: its MIoU does not drop and OminiControl's edge F1 is within 0.62 of the SpaDiff combination. We do not claim otherwise. The one consistent difference is edge precision, which drops about twice as much with Seg2Any (-9.03 / -8.81) as with SpaDiff (-4.69 / -4.70), offset by higher recall.
  • Scope of the Seg2Any result. We caution against generalising Seg2Any's result beyond this pair, for two reasons. First, canny is the most favourable case: the edge map is extracted from the same image as the segmentation and already encodes the layout (OminiControl alone reaches 33.4 / 40.2 MIoU), so the two controls agree and reinforce each other, which can mask interference. Controls not redundant with the layout, such as subject, style or fill conditioning, offer no such support. Second, the evaluation stays close to Seg2Any's training setting. Its LoRA modifies the image and text pathways for every token, so each further shift (another adapter, more adapters, other resolutions or real-world inputs) moves co-loaded adapters further from the weights they were trained on. Even in this favourable case we observe a larger loss of edge precision and MAN-IQA and occasional colour artifacts. We did not have time to test these settings in the rebuttal, so we state this as an expectation that follows from the design, not as a measured result. For SpaDiff, the same pathways are unchanged in all these settings; Fig. 7 shows it combined with fill and depth on out-of-distribution inputs.

This result led us to state the claim more precisely. SpaDiff's advantage is structural: its LoRA is active only on its own condition tokens, so the image and text pathways that every other FLUX adapter was trained against are unchanged for any adapter, any number of adapters and any resolution. Seg2Any places a LoRA on those pathways, so compatibility is an empirical property that has to be re-validated for each combination; it holds for canny here, under an integration we built to favour it.

Changes in the revision

  • We add the table to the paper (Table 6).
  • We show qualitative results of the composability experiment in Appendix D.
  • We state the claim more precisely in the abstract and introduction: baked-in methods give no guarantee of compatibility, whereas SpaDiff leaves the pathways other adapters were trained against unchanged.

We hope this addresses the critical weaknesses identified and the questions of the reviewer, and would be glad to hear whether the reviewer finds these revisions sufficient.

Reviewer_fnPuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpaDiff is an open-vocabulary segmentation-to-image adapter built on FLUX.1-dev. It binds regional text descriptions to segmentation regions through soft attention masking, with suppression strengths learned as functions of timestep, layer, and attention head. Contour tokens provide finer boundary control through either unified or decoupled conditional integration; the decoupled variant additionally uses a ZeroScale feature gate. The final configuration combines unified integration with adaptive masking. Experiments on COCO-Stuff and SACap-Eval compare segmentation adherence and image quality, while qualitative architectural examples demonstrate composition with OminiControl adapters.

Critical strengths

This paper addresses a useful combination of open-vocabulary regional control and compatibility with existing FLUX adapters. Adaptive masking provides a concrete mechanism for adding regional constraints without relying on full backbone fine-tuning.

Critical weaknesses
  1. Limited novelty. SpaDiff largely follows Seg2Any, mainly restricting adaptation to the condition branch and introducing adaptive soft masking. This makes the overall innovation limited.
  2. The comparison methodology is somewhat outdated. Would it be possible to include more recent models, such as DreamRender, Flux 2, etc.?
  3. The newly introduced components provide limited performance gains.
Questions for authors

Is it possible to compare newer models, such as DreamRender, Flux 2, etc.?

1 comment

We thank the reviewer for their time and for recognizing the usefulness of SpaDiff. Below, we address the concerns:

Novelty beyond Seg2Any (Weakness 1)

SpaDiff indeed builds on Seg2Any: we adopt its contour encoding of the segmentation mask and its attention-masking rules. Our goal, however, is to provide open-vocabulary S2I as an adapter on a frozen FLUX, so that it composes reliably with existing FLUX adapters, which is not guaranteed by baked-in models such as Seg2Any. To our knowledge, SpaDiff is the first that achieves this goal for MM-DiTs. We make this possible by introducing the following mechanisms, together forming our contribution:

  1. Adaptive masking. Seg2Any’s hard masking method does not produce coherent images when applied to a frozen backbone (Figure 5). SpaDiff’s adaptive masking learns the suppression strength per timestep, layer, and head, which raises MIoU from 15.08 to 23.37 and decreases FID from 37.77 to 29.89 (Table 5). This indicates that letting the model learn when, where and how strongly to suppress is important for generating spatially-coherent outputs. The learned mask values in Figure 7 show that layout is decided at early timesteps, in specific middle and final layers, and by a subset of attention heads. This extends prior layer-level analyses of FLUX to timesteps and heads and provides useful guidance for further research on the impact of these factors.
  2. ZeroScale. Conditional integration is necessary to incorporate precise contours present in the segmentation mask. We follow OminiControl’s approach (unified; also present in Seg2Any) and ablate it with a decoupled version (Section 4.2.1). We introduce ZeroScale, a per-feature gate with ≈175K parameters, instead of ≈537M for a ControlNet-style zero-convolution, which removes the graininess of decoupled integration (MAN-IQA 50.4 → 54.2; Table 3). This provides a computationally inexpensive solution applicable beyond S2I to other adapters that add conditional features to a frozen model.

Performance gains (Weakness 3)

We agree that SpaDiff, w.r.t. a fully-finetuned Seg2Any, has comparable performance or marginal-at-best gains: SpaDiff reaches comparable layout adherence at its native resolution within 1.0 MIoU on SACap-Eval (94.02 vs. 95.03; updated Table 2) and 1.5 MIoU on COCO-Stuff (44.13 vs. 45.67). However, we did not set out to merely compete with Seg2Any and instead, our aim is to close the gap between an adapter-style framework and a baked-in one. Moreover, there are a few performance gains worth mentioning: compared with the frozen-backbone alternative (hard masking), adaptive masking improves MIoU by 55% (15.08 → 23.37; Table 5) and, in the full model, FID by 3.4 points (23.46 → 20.02; Table 4). SpaDiff also achieves the best FID (15.47 vs. 17.88), CLIP score and Color/Shape/Texture scores on SACap-Eval.

Newer models such as DreamRenderer and FLUX.2 (Weakness 2, Question)

We agree that comparing against recent methods is important. DreamRenderer, however, targets a different setting: it is a training-free controller for depth- or canny-conditioned generation, where the depth or canny map defines the scene structure and boxes or masks only assign text to instances. In S2I, the segmentation mask is the only structural input, so DreamRenderer is not a direct baseline; its results on S2I when given depth maps are reported in the Seg2Any paper.
From a practical viewpoint, building SpaDiff on a newer base model like FLUX.2 would likely improve image quality. However, from a scientific viewpoint, it would not change the question we study, namely how to obtain open-vocabulary S2I control on a frozen MM-DiT, nor the mechanisms we propose: adaptive masking and conditional integration rely only on joint attention over separate token streams, which is the defining property of MM-DiTs rather than a property of a specific FLUX version. Moreover, our baselines and the adapters SpaDiff is designed to compose with (e.g. OminiControl) are built on FLUX.1, so changing the backbone would confound the comparison rather than isolate the effect of our contributions.

Changes in the revision:

  • Clarified contributions relative to Seg2Any in Section 1.
  • Provided native-resolution SpaDiff results in Table 2 which more accurately reflect the performance relative to Seg2Any.
  • Noted in Section 5 that SpaDiff can be transferred to newer MM-DiTs, which is a direct route to higher image quality in practical applications

We would be grateful to hear whether this addresses the reviewer's concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.