- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work addresses a method for segmentation-to-image generation, where segmentation maps are given along with the prompt to generate an image that should respect all conditioning. They add a set of trainable parameters to process the additional segmentation conditioning, and restrict the cross attention between the text and image / segmentation to improve the binding of target objects to target regions.
They evaluate how well the images match the layout conditioning, as well as image quality. They additionally ablate several key design decisions, including architecturally how the conditioning is integrated and whether masking is hard or soft.
Critical strengths
S1. Coverage of metrics. The work does a good job of measuring a variety of aspects (quality, respect to the segmentation) and ablations--if my concerns about the results themselves are resolved (see W2), then the work would have good experimental coverage.
Critical weaknesses
W1. Missing regions of related work that significantly impact novelty. This work does not mention or compare to very similar work in layout-guided generation, which is the same task but with larger boxes rather than tight segmentation. In fact, several works have already applied this to MM-DiT [1,2,3], using very similar mechanisms to the work presented here (and also to previous methods in training-free layout conditioning, like [4,5]). Soft vs hard binding has been explored as well [1]. This greatly impacts the novelty of the work presented here, which effectively takes similar mechanisms to [1,2,3] to bind text to regions, and applies IP-Adapters to add the additional conditioning. At the very least, the novelty of the work would need to be reframed. However, I am still concerned that the novelty is very limited, essentially combining two existing methods while providing little in terms of new insights.
Similarly, the justification of the problems statement feels somewhat weak to me. FLUX is no longer SOTA, and some newer models (e.g. FLUX-Kontext) just take additional conditioning as an input--adding conditioning after pre-training is quite a bit less important if SOTA models already apply it during pre-training.
W2. Poor experimental results and missing key comparisons. Some of the experimental results are very concerning (Table 1 and 2). Specifically, the proposed methods and other FLUX-based methods do not really out-perform SD1.5-based methods on the image quality metrics (FID, MAN-IQA). Given that FLUX is a (much) stronger model, these methods would be expected to perform much better, from the base model alone. The fact that they do not likely implies that likely either the metric is not able to accurately measure what is intended (which is problematic towards the soundness of the numerical results), or the methods are generating extremely poor-quality images (which is problematic towards the value of the method). Also, the work does not consistently beat comparative methods.
In addition, some key comparisons are missing to understand how the model truly compares to other work. First of all, ControlNet (which is a key baseline) is also available for FLUX, but is only compared with on SD1.5 in this work, which underestimates the accuracy it could be providing. Also, to differentiate between the layout-guided methods [1,2,3], ablations with their forms of binding to regions, soft/hard, etc would be important towards understanding how this work differentiates (which would also go towards justifying novelty, see W1).
References:
[1] Region-aware text-to-image generation via hard binding and soft refinement. Chen et al. ICCV'25.
[2] Stitch: Training-free Position Control in Multi-modal Diffusion Transformers. Bader et al. arXiv'25.
[3] Training-free regional prompting for diffusion transformers. Chen et al. arXiv'24.
[4] Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. Lian et al. TMLR'24.
[5] Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. Yang et al. ICML'25.
Questions for authors
Q1. Do you have any intuitions on why the FLUX-based models do not perform better than those based on SD1.5 on FID and MAN-IQA?
4 comments
Problem statement (Weakness 1.2)
We agree that newer models merit consideration, but FLUX.1 Kontext conditions on a reference image and an edit instruction rather than on a segmentation mask with free-form regional descriptions, which is our task. More generally, any base model ships with a finite set of built-in conditions, and adapters remain the way to add further controls without retraining, which is our motivation. We use FLUX.1-dev to compare against existing S2I baselines and assess composition with OminiControl on a common backbone. Our contribution concerns adding regional control to a pretrained model; evaluating its transfer to newer backbones remains future work.
Image-quality metrics and comparison to baselines (Weakness 2, Question 1)
SD1.5-based methods are ahead only on FID in Table 1 and on MAN-IQA in Table 2, and each case has a specific reason. FID measures how close the generated images are to the images of the benchmark dataset, and FreestyleNet and PLACE fine-tune the full SD1.5 backbone on that dataset, whereas FLUX-based methods retain FLUX's own image distribution, entirely so in our case since the backbone is frozen; Seg2Any reports the same ordering in its paper. MAN-IQA depends on the resolution at which images are generated, and in Table 2 the SD1.5-based methods generate at a lower resolution than EliGen and Seg2Any; the same effect is visible when comparing SpaDiff at its two resolutions in that table. In the other table, each metric favours FLUX-based methods: FID in Table 2 and MAN-IQA in Table 1. Both metrics are therefore sound when read with this in mind, and both are widely used in diffusion literature. The methods are also not generating poor-quality images, as the qualitative results in Figures 3 and 7 show. Finally, SpaDiff does achieve the best results on several metrics (Table 2), but outperforming fine-tuned methods was never our goal: we aim to match their quality while keeping the backbone frozen, which gives SpaDiff the composability of an adapter (Table 6).
Missing key comparisons (Weakness 2.2)
To our knowledge, no public ControlNet for FLUX supports segmentation conditioning; the available ones cover conditions such as canny, depth and pose. Such a model would also be closed-vocabulary and could not be evaluated on SACap-Eval. [1, 2, 3] address prompt-driven compositional generation and not segmentation-mask-to-image. The only form of binding they share with our setting, the fixed hard attention masking of [2,3], is already covered by the hard-masking baseline in our ablation of adaptive masking (Section 4.2.2).
We hope this clarifies how SpaDiff differs from [1-5], and would be grateful to hear whether this addresses the reviewer's concerns.
W1. Referring to my original review, the characterization was 'but with larger boxes rather than tight segmentation', covering the same task distinctions that the authors refer to in the response. This is exactly why it is important to compare with these methods--because while the preciseness of the changes differs, the underlying mechanisms controlling generation are the same. It is quite likely that a method operating at the pixel-level would outperform one operating only on blocks of patches, but it is important to validate this, as it would be a significant problem if it is not better. It is also important to discuss methodological differences between the two related fields, to keep the wider field united and understand where differences in findings diverge (e.g. if it is because the specifics of the task really cause other solutions to fail, or if different sub-fields just investigated different things rather than relying on each other).
Regarding the difference from IP-Adapters, there seems to be no ablation covering the specifics of adapter construction. In my original review, I assumed that the details of the adapter were not an intended source of novelty, and applied the weakness therefore to novelty (as they were stated to have relations to IP-Adapters and OmniControl's methodology). However, if they are intended to be a source of novelty, then these design decisions would need to be systematically evaluated.
W1.2. However, segmentation maps could be another form of input to edit. For example, the model could be given the segmentation map as an input and asked to turn it into an image, thereby providing the same function. It would need to be shown that these models fail at this task. And while evaluating on FLUX.1 for comparison to other works is valid, it also remains a distinct limitation of the work that it is not clear how it adapts to newer models.
W2. Even if these metrics have been used before, does not mean they are appropriate for measuring image quality here. We know there to be a significant increase in image quality from SD1.5 to FLUX.1. If the presented metrics cannot measure that, then they are not sufficient for showing that the images are of good quality. Showing results on qualitative examples is not sufficient evidence of overall quality either, as the ability to present a small subset of good images is not the same as measuring systematic quality increase. Hence, it remains that there is no metric presented that can accurately capture the quality of the images, making their overall quality unclear.
Even with the goal of providing training-free alternatives, it is very problematic if that quality comes from upgrading to a significantly better backbone (as is the gap from SD1.5 to FLUX.1) and then providing modifications that degrade performance back to the level of the much older backbone. As it stands, there is no quantitative results giving evidence that this is not the case. This is why it is important to measure this, with a metric that can accurately capture quality differences.
W2.2. Thank you for the clarification.
We thank the reviewer for the follow-up.
Comparison with layout-guided binding (W1)
“while the preciseness of the changes differs, the underlying mechanisms controlling generation are the same.”
The underlying mechanisms of [1-5] are not all the same, as detailed in our first reply.
“It is quite likely that a method operating at the pixel-level would outperform one operating only on blocks of patches, but it is important to validate this, as it would be a significant problem if it is not better.”
Where the mechanisms are similar, our paper already contains the comparison: the hard-masking baseline of our adaptive masking ablation (Section 4.2.2) applies such fixed masking rules to the frozen FLUX backbone and performs poorly compared to adaptive masking (15.08 vs. 23.37 MIoU, 37.77 vs. 29.89 FID). More generally, Table 5 as a whole validates the reviewer's concern: none of its variants uses conditional integration, so all of them bind text to regions through attention masking alone, i.e. at the level of blocks of patches. The best of them reaches 23.37 MIoU, whereas adding contour tokens raises this to 43.39 in the same setting (Table 6), confirming that pixel-level control is needed for S2I.
Our comparison also includes the baseline closest to layout-guided generation. EliGen receives the layout directly instead of inferring it from the prompt as [1-5] do, but it is trained on rectangular regions and does not integrate the contours of the segmentation mask in any way, so it also operates on blocks of patches. As noted in Section 4.1, it therefore supports only loose positioning and reaches 52.86 MIoU on SACap-Eval, compared to 94.02 for SpaDiff (Table 2).
Adapter construction (W1)
“Regarding the difference from IP-Adapters, there seems to be no ablation covering the specifics of adapter construction”
We understand "adapter construction" as the way the contour conditional tokens are integrated into FLUX (Section 3.1), since this is where our design relates to IP-Adapter and OminiControl. This conditional integration is ablated systematically: we compare unified integration, where the condition tokens form a third stream in joint attention as in OminiControl, with decoupled integration, where they are processed by a parallel attention operation inspired by IP-Adapter's decoupled attention, each combined with hard and adaptive masking (Table 4). For the decoupled variant, we further ablate how its output is gated by ZeroScale (Section 4.2.1, Table 3).
“if they are intended to be a source of novelty, then these design decisions would need to be systematically evaluated.”
We do not claim the two integration schemes as novel, as they follow OminiControl and IP-Adapter; our contributions relate to ZeroScale and adaptive masking, whose design is ablated in Table 5.
Editing models and newer backbones (W1.2)
“segmentation maps could be another form of input to edit [...] It would need to be shown that these models fail at this task.”
With an editing model, the input would be a colour-coded segmentation map, and the prompt would have to state which description belongs to each colour. We expect this to work poorly as the number of regions grows, since every binding between a colour and its description has to be resolved from the prompt alone. As this setup is far from S2I, where each region is bound to its description directly, we do not consider this comparison relevant for our paper.
“while evaluating on FLUX.1 for comparison to other works is valid, it also remains a distinct limitation of the work that it is not clear how it adapts to newer models.”
We agree that transfer to newer MM-DiTs has not been tested, and we acknowledge this as a limitation; since SpaDiff relies only on joint attention over separate token streams, it does not depend on properties specific to FLUX.1.
Image-quality metrics (W2)
“it remains that there is no metric presented that can accurately capture the quality of the images, making their overall quality unclear.”
We agree that FID and MAN-IQA do not always capture the gap between the backbones. Table 2, however, also reports two metrics trained on human preferences, PickScore and ImageReward, and these do capture it: all FLUX-based methods score above the SD1.5-based ones (ImageReward 0.40 to 0.77 vs. -1.01 and -0.84; PickScore 21.66 to 22.30 vs. 19.94 and 20.09), and SpaDiff lies within the FLUX range on both (0.56 and 21.84). These scores are computed over the full evaluation set, not a selection of examples, and provide the requested quantitative evidence that SpaDiff does not degrade FLUX to the level of SD1.5.
We hope these clarifications address the remaining concerns.
We thank the reviewer for their time and for recognizing the coverage of our metrics and ablations. Below, we address the concerns:
Relation to [1-5] and novelty (Weakness 1)
We appreciate the given references, but we respectfully disagree that they address the same task with a similar approach:
Task
[1-5] are training-free methods for compositional text-to-image generation. Their regions are rectangular and usually derived from the prompt by an LLM; none is designed for, or evaluated on, a segmentation mask. Segmentation-mask-to-image (S2I) is a different task: the mask is the user's specification and the image must follow every contour (beginning of Section 3 contains a clear problem statement).
Method
[1-5] first turn the prompt into a coarse layout of rectangular regions, each with its own sub-prompt (produced by an LLM, or by the user in [1,3]). None of them learns any parameters. They differ in how the image is then generated:
Relation to SpaDiff
Soft vs. hard in [1]
Only the wording coincides.
The question we study, whether an attention mask on a frozen MM-DiT should block interactions or only suppress them, and at which timesteps, layers and heads, is therefore not explored in [1], which uses no attention mask.
IP-Adapter
SpaDiff does not use IP-Adapter anywhere. Contour tokens enter joint attention as a third stream, following OminiControl. IP-Adapter is mentioned only as the inspiration for our decoupled variant, an ablated alternative (described in Section 3.1 and evaluated in Section 4.2.1).