- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes StructFlow, which is a flow-matching framework that replaces i.i.d. Gaussian noise with spatially structured noise, in which pixels within local regions share a common noise component to introduce an inductive bias aligned with image locality.
To stabilise training, it introduces spatial anchoring, progressive coherence annealing, and mixed-coherence training strategies. Experiments across unconditional, class-conditional, and text-conditioned generation demonstrate competitive image quality, improved local editing and structure preservation, and the potential to adapt pre trained generative models through lightweight post-training.
Critical strengths
The motivation is intuitive, and the concept is simple. The implementation is also straightforward, essentially extending pixel-space flow matching to the superpixel space.
Critical weaknesses
-
The practical motivation of the paper is not clear. In the classic pixel-space diffusion image generation model, the spatial correlation of natural images is already captured by the inductive bias of the convolutional U-Net used as the denoising network, rather than encouraging spatial-correlation in the initial noise. Similarly, in latent diffusion, the spatial correlation is also encouraged by the VAE latent token computed from spatially local image patches. Also, since flow matching does not fundamentally require an i.i.d. Gaussian source, in fact any source distribution can be used to train the velocity field, the paper needs to establish why structured spatial-correlated noise offers a meaningful advantage over existing approaches.
-
Excessive dependence on superpixel segmentation maps. The method relies on superpixel maps, but their availability and generation at inference time, particularly for generic text-to-image generation, are unclear. The demonstrated applications, in fig-4 and fig-5 mainly on structure-preserving fine-grained local editing task, and global structure-preserving image colour/style modification, are limited. If segmentation maps are available, the authors should explain why using them as auxiliary conditioning in a standard U-Net or DiT is less effective than modifying the source noise, and provide direct comparisons to justify the added complexity. For a generic image generation task, using a pre-defined super pixel map as a weak guidance can be restrictive.
Moreover, if a superpixel segmentation map is known a priori, why cannot you simply use it as an auxiliary condition (via attention or concatenation) added to the standard denoising pipeline (either UNet or DIT) ? A substantial ablation study or comparison on this is necessary to demonstrate the value of the method.
Questions for authors
-
Address the above mentioned weaknesses, especially the suggested ablation study of using a super pixel-map as a condition in the denoising /reverse flow match process, and compare the results.
-
Can you method be applied to latent space where image patch tokens instead of raw pixels are used ? In this case, how to use the super pixel segmentation map ?
5 comments
W2 and Q1. Why not use the superpixel map as an auxiliary condition?
We thank the reviewer for this suggestion and additional ablation. We agree that it is the most important comparison to make, and we ran it directly.
We would also like to note the broader perspective. Conditioning branches such as ControlNet, channel concatenation or extra cross-attention are the established way to add control. StructFlow explores a different route: it places the spatial inductive bias directly in the source noise, with no conditioning branch at all. As Reviewer 6GHi also noted, this "could offer a new paradigm for improving controllability in generative models". We therefore see the two as complementary directions rather than competing ones. Still, we fully agree that the comparison is informative, and we report it below.
There are two conceptual differences:
- No architectural change. StructFlow only changes the source distribution. Channel concatenation changes the input shape, and a cross-attention route adds new modules and parameters.
- Editing arises naturally. A condition can tell the network where the regions are. With i.i.d. Gaussian noise, however, redrawing the noise inside one region has no reason to stay inside that region. In StructFlow, the region's noise is tied to the region by construction.
Setup. Starting from the same SANA-1600M checkpoint and the same training data and hyperparameters, we post-trained:
- (i) Channel concat: standard Gaussian flow matching, with the SLIC map concatenated to the DiT input (zero-initialised projection).
- (ii) Cross-attention: standard Gaussian flow matching, with the SLIC map injected through an extra trained cross-attention layer in every block (zero-initialised output).
- (iii) StructFlow (SLIC), post-trained with a comparable short budget, for a fair comparison.
In terms of cost, StructFlow uses exactly the same network as standard flow matching, with no extra parameters or FLOPs. The cross-attention variant adds 201M parameters (+12.5%) and 17% more FLOPs per forward pass. Channel concatenation is nearly free (+0.07M parameters), but it changes the input layer of the model.
Due to the limited rebuttal period, all of them were post-trained for 3k steps.
We then evaluate two tasks, using the same protocol for every model:
- Structure preservation (PartiPrompts, same setup as in the paper): each model generates from the SLIC map of a reference image, with all noise redrawn per sample. We measure how well the layout of the reference is kept (DINO self-similarity, depth RMSE, Canny-edge F1), sample diversity, and quality.
- Local editing (PIE-Bench++, same setup as in the paper): each model generates a base image, CLIPSeg selects the region to edit, and only the noise inside that region is redrawn. There is no blending and no masking during denoising. We measure change outside the region (lower is better) and diversity inside it (higher is better).
Structure preservation
| Method | Self-sim ↓ | Depth RMSE ↓ | Canny F1 ↑ | CLIP ↑ | CLIP-IQA ↑ |
|---|---|---|---|---|---|
| (i) Channel concat | 0.242 | 0.217 | 0.516 | 28.27 | 0.751 |
| (ii) Cross-attention | 0.241 | 0.217 | 0.500 | 27.98 | 0.742 |
| (iii) StructFlow (SLIC) | 0.221 | 0.196 | 0.539 | 28.01 | 0.746 |
Local editing
| Method | Outside LPIPS ↓ | Outside RMSE ↓ | Inside LPIPS ↑ | CLIP-IQA ↑ |
|---|---|---|---|---|
| (i) Channel concat | 0.226 | 0.211 | 0.182 | 0.736 |
| (ii) Cross-attention | 0.228 | 0.212 | 0.177 | 0.738 |
| (iii) StructFlow (SLIC) | 0.191 | 0.172 | 0.179 | 0.735 |
All three models can follow the reference layout to some extent, and under the same budget StructFlow already preserves the structure slightly better. The main gap is in local editing. With the auxiliary-condition models, redrawing the noise inside the selected region also noticeably changes the rest of the image. StructFlow keeps the outside of the region clearly better preserved (lower Outside LPIPS and RMSE) while keeping the same diversity inside it (similar Inside LPIPS).
Q2. Does the method apply in latent space, and how is the superpixel map used there?
Yes. In fact, all of our text-to-image results already operate in latent space: SANA is a latent DiT on a DC-AE (f32) latent. VAE latents keep the spatial layout of the image, since each latent position corresponds to a fixed image patch. This is also why cross-attention maps in latent diffusion models line up so well with objects in the image [1].
Using the superpixel map in latent space is therefore straightforward:
- Compute SLIC on the full-resolution image.
- Sample the structured noise at image resolution with that mask.
- Bicubically downsample the noise to the latent grid and rescale it to unit variance.
This keeps segment boundaries smooth even at a high downsampling factor. We will make this procedure more explicit in the revised paper.
We are grateful to the reviewer for these comments and questions which substantially strengthened the paper. We hope the new experiments and clarifications address the concerns, and we would be happy to discuss any further questions.
[1] Hertz, Amir, et al. "Prompt-to-prompt image editing with cross attention control."
We sincerely thank the reviewer for the careful reading and the thoughtful questions. We were glad to hear that the motivation felt intuitive and the method simple and clear. Below, we address each of the reviewer's concerns (W: Weakness, Q: Question).
W1. Practical motivation: why structure the source noise?
We agree that the denoiser already carries a locality bias (convolutions in a U-Net, patch tokens in a latent DiT), and that flow matching does not require an i.i.d. Gaussian source. In fact, this flexibility is exactly what StructFlow builds on.
Our motivation comes from the analysis in Fig. 1 and the Finding in Sec. 3.1. Images, and their VAE latents, are strongly local, yet the source noise has no spatial structure at all: every local patch of noise is independent. When we measure within- vs. across-superpixel similarity along the generation trajectory, the noisy state xt starts with essentially no structure. The predicted velocity, however, already shows a large locality gap at t=0. In other words, the network's own locality bias is spent manufacturing structure that the source does not provide.
StructFlow asks a simple question: what happens if the source already carries this locality? Doing this naively is unstable, so we introduce the three stabilisation strategies of Sec. 3.3. With them, StructFlow matches standard flow matching in image quality. On top of that, local editing and structure-preserving generation appear out of the box:
A model trained with standard Gaussian flow matching does not have these abilities. It typically needs extra modules or task-specific data to gain them. That, in short, is the practical value of the idea.
W2. Availability of superpixel maps at training and inference
Training. Superpixels come essentially for free: SLIC runs in milliseconds on the training images we already have. We see this as similar in spirit to using a strong VLM to produce dense captions as a cheap but rich training signal.
Inference. We agree that an exact superpixel map is not available when generating from scratch. Sec. 4.5 and Appendix E study several cheap options:
The model readily accepts these masks and generates high-quality images with them. For structure preservation and local editing, the mask is simply the SLIC map of the image being edited or followed, so it is always available.
Two of our other responses give further evidence on this point. In our response to Reviewer 6GHi (Q2), we quantify how far each inference mask type is from the training SLIC masks and how this affects generation quality. In our response to Reviewer Pqhj (W1), we study what happens when the model is trained on masks that are independent of the image.
W2. Global structure-preserving colour/style modification
Our method does support global structure-preserving modification. Keeping the same structured noise and changing only the prompt changes the colour and style of the whole image while its layout is preserved. This needs no inversion, attention injection or additional control.
Following the reviewer's concern, we conducted additional experiments to support this claim. The new qualitative results are available here: https://anonymous.4open.science/api/repo/StructFlow-ICLR-2027-Rebuttal/file/rebuttal_global_style.png (also Fig. 11 in the revised paper).