acceptodds
Under review as a conference paper at ICLR 2027

Semantic Self-Flow: Relation Learning in Self-Supervised Flow Matching

Abstract

Diffusion and flow models still struggle to capture image semantics, such as spatial relations between objects. Recently, Self-Flow showed that self-distillation across different noise levels for the student and teacher views can reduce reliance on local evidence for reconstruction and improve semantic representations. However, a random student mask does not target any particular semantic dependency; for example, it leaves most of the subject at the lower noise level, revealing its location, and hence the spatial relation, without requiring the relation condition. We introduce Semantic Self-Flow (SSF), which places stronger noise where recovery depends on the semantic constraint of interest. For spatial relations, the subject is noised together with three alternative locations around the reference object, which remains at the lower noise level, making the relation condition informative for reconstruction. On ImageNet, when the relation is flipped under the same noise, both SSF outputs follow their respective relation conditions for 32% of test prompts, compared with 2% for Self-Flow, while class-conditional FID remains nearly unchanged.

Primary area:
generative models
Keywords:
flow matching, self-supervised learning, spatial relations, diffusion models, text-to-image generation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_yJWTOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work focuses on conditional image generation and text-to-image generation. It proposes Semantic Self-Flow (SSF), which investigates whether deliberately introducing information asymmetry, rather than relying on random masking, can make selected semantic information a necessary condition for the recovery process. This is a well-motivated angle and offers useful insights for existing self-supervised representation learning.

Critical strengths

The explicit design of alternative-relation decoys, which forces the model to rely on relation words and reference locations to infer the subject position, is a concrete and feasible solution. The leakage analysis of spatial uncertainty also provides solid support for the experimental motivation.

Critical weaknesses

However, the claim that “the distribution is approximately independent of r” requires more rigorous measurement. In addition, only about 8.2% of ImageNet samples can be automatically annotated with spatial relations, which may significantly limit scalability to larger datasets.

Questions for authors

1 In Table 1, the sensitivity of the proposed method under SiT-XL/2 is only 10.0%. Such a low sensitivity suggests substantial limitations of the method, and the paper should discuss this more thoroughly.

2 The method requires relation annotations. How are these annotations obtained? What is their scale? Can this approach scale to large-scale databases?

3 As shown in Figure 4, Self-Flow does not exhibit flipping behavior. Why does Self-Flow ignore the relation condition, whereas the proposed method appears to utilize it? If Dq consists of object-region tokens cropped from other training images, how are these tokens sampled? Does the sampling depend on the relation word? How sensitive are the results to this design?

4 Figure 1 describes a mixed sampling strategy. How are the 8.2% relation-aware images selected? The paper should provide quantitative results on how this proportion affects image generation and parameter sensitivity.

5 The statement that “masked regions follow approximately the same distribution regardless of the true relation” needs clarification. How should “approximately” be understood or guaranteed? Could the proportion of relation-aware images or the sampling strategy affect this approximation?

6 The “look better” evaluation in Figure 3 appears subjective and lacks an objective metric. Moreover, why does Self-Flow achieve a higher “look better” preference rate?

2 comments

We thank the reviewer for finding our angle well motivated and the decoys a concrete and feasible solution.

Approximate independence of the mask from the relation (W1, Q5)

We measure the approximation with a classifier that predicts the relation from the reference box and a mask, without the relation word, on 1,830 held-out pairs of COCO images.

Classifier inputRelation accuracy (%)
Reference box62.6
Reference box and subject mask93.2
Reference box and mask of SSF60.2

The subject mask reveals the relation (93.2%), whereas the mask of SSF adds nothing over the box alone (60.2% vs. 62.6%). The proportion of relation-aware images changes how often this mask is used, not what it reveals, whereas random decoy locations break the construction and lower Both accuracy (33.7% vs. 42.1%).

Relation annotations and scalability (W2, Q2)

The annotations are automatic (Section 3.3.1 and Appendix A.2), and annotating all 1.28M ImageNet images takes about 33 hours on four B200 GPUs. For text-to-image, we use the SCOP pairs of COCO (Appendix A.5). The approach scales, since SSF needs annotations for only part of the images, the pipeline is linear in the number of images, and newer models or existing object annotations can replace its stages, e.g., Gemma 4 and Qwen3.8 confirm 88% and 86% of 1,000 sampled labels.

Why the sensitivity is only 10% (Q1)

The sensitivity (Appendix B.1) measures how much the model's prediction for the same noisy image changes when we flip only the relation in the prompt, e.g., from left of to right of. A value near 100% is not expected, since only the location of the subject should change, whereas its appearance, the reference, and the background should remain. Further, we find that even large changes yield small values, e.g., swapping the subject category, from bird above branch to cup above branch, changes the prediction by only 14.9% (Self-Flow 11.2%). Flipping the relation changes it by 10.0% for SSF vs. 1.1% for Self-Flow.

Why Self-Flow ignores the relation, and how the decoys are sampled (Q3)

With a random mask, most of the subject stays at the lower noise level and reveals its location, and hence the relation, so the denoiser reaches a low loss without the relation word. SSF hides the subject and the three alternative locations while keeping the reference clean, so the relation word is needed to locate the subject. The decoys come from a bank of subject regions of other training images, indexed by their relation and the cell of their reference. For each alternative relation, we draw 16 regions from the cell of the current reference and use the first one that does not overlap the reference, at its original position. Hence, the sampling depends on the relation word only through the choice of the alternatives. The design matters, since random decoys reach 33.7% Both accuracy and no decoys 27.6%, compared to 42.1% for SSF, and with a second training seed, SSF again outperforms both (43.6% vs. at most 40.9%).

Selection of the 8.2% and the effect of the proportion (Q4)

The 8.2% are all images retained by the annotation pipeline, without further selection. The table reports other proportions with SiT-B/2 at 200k steps.

Relation-aware share (200k steps)Both, SSF (%)Sensitivity, SSF (%)FID-50k, SSF
4.1% (half of the labelled images)12.63.612.22
8.2% (default)28.06.311.80
50% (labelled images oversampled)54.512.417.68
100% (labelled images only)31.428.170.73

Up to 50%, a larger proportion raises both the generation accuracy and the sensitivity, but oversampling to 50% increases the FID-50k, whereas at 8.2%, SSF keeps the FID of Self-Flow (11.84). With the same oversampling, Self-Flow reaches 48.2%, and SSF remains 6.3 points ahead, significant with 95% confidence. Training on the labelled images only (100%) reduces the data to 105k images, which lowers the image quality of both methods (FID-50k of 70.73 vs. 86.68 for Self-Flow) and their generation accuracy compared to 50%. SSF remains ahead (31.4% vs. 25.6%), significant with 95% confidence.

The look evaluation and why raters prefer Self-Flow (Q6)

In a new single-image study, 34 raters rate the look of each image from 1 to 5 and, as an objective criterion, state whether it shows its caption. They find a small look gap (2.53 vs. 2.66), whereas the images of SSF show the caption more often (61% vs. 46%). A head-to-head comparison can show a large preference gap even for a small difference in quality, and the preference for Self-Flow often arises when SSF draws an unusual layout that the prompt requests, e.g., keyboard above monitor.

We are happy to discuss any further questions or suggestions.

Thank you for the positive and constructive review. We will add the analyses from our response to the final version.

Reviewer_wW9QOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Semantic Self-Flow" (SSF), which builds upon the existing "Self-Flow" framework by incorporating a relation-aware noise allocation strategy. This strategy, used in conjunction with standard random perturbations, enhances the generative model's utilization of spatial relationship conditions. The approach is evaluated on ImageNet, text-to-image generation tasks, and synthetic datasets. Through experiments involving relation flipping, diagnostic tests for information leakage via masks, and relation recognition, the authors demonstrate the improvements offered by SSF over the original Self-Flow. The primary contribution lies in the training perturbation strategy and the analysis of its underlying mechanisms, rather than in the proposal of a new network architecture or learning objective.

Critical strengths
  1. The research problem is clearly defined, and the strategy is purposefully designed. The authors note that random perturbations do not necessarily compel the model to leverage relational conditions; conversely, masking only the target object may inadvertently reveal the answer through the mask's location. Protecting the reference object while introducing "distractor" regions is a well-conceived solution to this issue.
  2. There is solid, controlled evidence supporting the overall effectiveness of the strategy. By employing identical noise-sampling methods for both the baseline and the proposed approach (using SiT-B/2), the authors effectively isolate the impact of the mask-positioning strategy. As shown in Table 7, the proportion of cases where the model correctly generates both the original and the flipped relationships increases from 3% to 37%.
  3. The mechanistic analysis goes beyond merely reporting performance gains. The study identifies that target masks can provide unintended relational cues, and that the inclusion of distractor regions helps mitigate this leakage. Appendix B further distinguishes between the model's responses to changes in relationships versus changes in object categories. Additionally, the experiments on mixing ratios (Appendix F) demonstrate that more relation-aware perturbation is not necessarily better. These analyses provide valuable insights into how the strategy functions.
Critical weaknesses
  1. There remains a gap in the evidence regarding the specific contribution of individual components versus the strategy as a whole. The proposed method simultaneously introduces noise injection in target regions, protection for reference objects, the construction of distractor regions, and a mixed-sampling scheme. While the experiments confirm the efficacy of the combined design, they do not fully disentangle the independent contribution of each component to the final generative performance. Specifically, Table 4 evaluates whether a classifier can infer relationships from the masks themselves, rather than assessing how the removal of distractor regions—while holding other conditions constant—affects the model's ability to adhere to specified spatial relationships. Similarly, the study lacks a comparison of generative performance in scenarios where reference-object protection is omitted, or where relation-aware distractor regions are replaced by randomly placed distractors (using an equivalent "noise budget"). For research primarily focused on perturbation strategies, these gaps limit the explanatory power of the proposed contributions.
  2. Insufficient task-level baselines; contributions are largely confined to improvements over Self-Flow. While Self-Flow serves as a reasonable and necessary direct baseline, the quantitative comparisons focus almost exclusively on this single method. Other highly relevant works—such as MDT, which utilizes masking to facilitate the learning of contextual relationships, and CoMPaSS, which is more directly oriented toward spatial relationship generation—are discussed in the paper (with the latter’s SCOP dataset even being employed), yet no comparable quantitative results against these representative methods are provided. The intention here is not to demand direct comparisons between models trained on vastly different scales, but rather to clarify the value of the proposed strategies relative to existing spatial relationship learning schemes, ideally by using consistent data, supervision, and computational budgets.
  3. Understanding of why and when these strategies are effective remains incomplete. Experiments on synthetic data support the importance of mixed sampling; however, they are insufficient to establish the scope of its applicability to real-world data, nor do they explain why applying relationship-aware perturbations to 100% of instances leads to performance degradation. Furthermore, the current results do not clearly distinguish whether performance gains stem from reduced answer leakage, the preservation of valid reference information, or alterations to training difficulty. Consequently, while the paper offers valuable insights into the underlying mechanisms, claims regarding the necessity and generalizability of specific design choices should be made with caution.
Questions for authors
  1. Beyond the leakage classification experiment in Appendix A.6, are there any controlled results for spatial relationship generation tasks that compare conditions with/without perturbed regions or with/without the protection of reference objects? If so, please describe the experimental setup and findings; if not, please clarify which component-level conclusions are supported by the existing evidence.
  2. Have you conducted any control experiments where relationship-aware perturbation locations are replaced by random locations—while keeping the number of perturbed regions, their total area, and the noise budget constant? Such an analysis would help distinguish the value of "specific counterfactual locations" from the general benefit of simply increasing the number of perturbed regions.
  3. Have you performed any experiments that allow for a direct comparison with existing spatial relationship generation methods, such as CoMPaSS? If not, please explain the rationale behind focusing the primary comparisons solely on "Self-Flow," and clarify whether the paper aims to demonstrate an "improvement in relational learning for Self-Flow" or a broader advantage of the proposed method across various tasks. Regarding MDT, please further elaborate on the key differences between the two approaches in terms of training information and the objectives of relational learning.
  4. Regarding hybrid sampling: Is the proportion of real-world data used determined by data availability, or is it selected based on an independent validation set? Has there been any analysis of the ratio of real-world data, and is there direct evidence supporting the claim that using 100% relation-aware perturbations leads to failure? Please distinguish between experimental observations and explanations that have yet to be verified.
1 comment

We thank the reviewer for finding the problem well defined, the decoys well conceived, and the analysis insightful.

Generation results without decoys or without reference protection (W1, Q1)

We added these controls (first table of the general response, SiT-B/2 at 400k steps). Removing the decoys or reference protection lowers Both accuracy from 42.1% to 27.6% and 32.2%, respectively, and a random mask with reference protection (SSF without subject noise and decoys) reaches 23.2%, compared to 0.9% for Self-Flow. Hence, both reference protection and the decoys contribute to the gain (W3), whereas more relation-labelled data does not explain it, since Self-Flow with the same oversampling remains below SSF (Q4). With a second training seed, SSF again outperforms the variants we retrained (43.6% vs. at most 40.9%), and the final version will report three seeds.

Random decoy locations with the same number, area, and budget (W1, Q2)

With the three decoys at random locations instead of the locations of the alternative relations, and the same budget and noise levels, Both accuracy drops from 42.1% to 33.7% (SSF with random decoys, second table of the general response). Since a decoy must not overlap the reference, the decoys of SSF, which lie next to it, are smaller than the random ones (46 vs. 57 tokens in total). We therefore also match the total area of the random decoys to that of SSF (45 vs. 46 tokens), which gives 35.1% (SSF with area-matched random decoys), 7.0 points below SSF, significant with 95% confidence. Hence, the locations of the decoys matter beyond their number and area.

Comparison with MDT (W2, Q3)

MDT removes a random 30–50% of the tokens in a second pass, fills them with a side-interpolator, and infers the missing parts from their context, without annotations. We extend MDTv2-B/2 by choosing its removed tokens by the placement of SSF, with the same number of tokens and the objective of MDT. Random removal improves FID but not relation learning, whereas our placement raises the Both accuracy of MDT from 1.6% to 27.2% at nearly the same FID-50k, and both its gain over MDT and its gap to SSF are significant with 95% confidence. The final version will repeat this comparison with SiT-XL/2.

Model (400k steps, seed 1)Original (%)Flipped (%)Both (%)Sensitivity (%)FID-50k
Self-Flow35.613.10.90.47.25
MDTv242.316.81.60.55.31
MDTv2 with the placement of SSF57.937.827.25.85.39
SSF67.853.142.17.17.28

Comparison with CoMPaSS (W2, Q3)

CoMPaSS fine-tunes text-to-image models on curated spatial pairs (SCOP) with an encoding of the token order (TENOR). Since SSF changes pretraining, the two are not directly comparable. Instead, we apply the CoMPaSS recipe, i.e., 10k SCOP-only steps with TENOR, to both pretrained text-to-image models, Self-Flow and SSF, and evaluate Both accuracy on the 1,186 COCO test captions with the Qwen3-VL judge.

Both accuracy (%), text-to-imageBefore CoMPaSSAfter CoMPaSS
Self-Flow10.29.4
SSF16.319.7
Gap (points)6.110.4

The gap grows by 4.3 points, significant with 95% confidence, and an OWLv2 judge also finds SSF ahead (34.2% vs. 17.9%). CoMPaSS fine-tuning raises the FID-5k of both models, and that of SSF remains lower (30.02 vs. 32.67, compared to 25.35 vs. 26.22 before). Hence, SSF pretraining and CoMPaSS fine-tuning are complementary, and the final version will repeat this comparison at full scale.

Mixing ratio (W3, Q4)

The 8.2% is determined by data availability, i.e., all 104,956 images retained by the annotation pipeline, and was not selected on a validation set. The table reports other ratios with SiT-B/2 at 200k steps. Self-Flow has no relation-aware masks, so its 4.1% model is its 8.2% model.

Relation-aware share (200k steps)Both, SSF (%)Both, Self-Flow (%)FID-50k, SSFFID-50k, Self-Flow
4.1% (half of the labelled images)12.6(same as 8.2%)12.22(same as 8.2%)
8.2% (default)28.01.411.8011.84
50% (labelled images oversampled)54.548.217.6816.57
100% (labelled images only)31.425.670.7386.68

(i) Oversampling (50%) raises Both accuracy for both methods, and SSF remains 6.3 points ahead, significant with 95% confidence, at the cost of a higher FID-50k, whereas at 8.2%, SSF keeps the FID of Self-Flow.

(ii) Fewer relation masks (4.1%) make SSF learn about half as fast, since its 12.6% at 200k steps is close to its 10.0% at 100k steps with 8.2%.

(iii) Labelled images only (100%) reduces the data to 105k images, which lowers the image quality of both methods. SSF remains ahead, significant with 95% confidence, and also has the lower FID-50k.

We are happy to discuss any further questions or suggestions.

Reviewer_rVQ8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Semantic self flow builds on the recent work self-flow and improves on it. The core idea is to capture relational semantic better which self-flow misses due to its random masking strategy. in SSF the authors add a notion of object semantics while deciding masking along with adding counterfactual decoys to avoid leakage and shortcut learning during training. They use pseudlalbed data from off the shelf VLMs and object detection models to create a training dataset. Experiments cover two ImageNet model sizes, COCO text-to-image training from scratch, and synthetic relations and counting.

Critical strengths
  1. The approach has been clearly presented and is easy to follow. Also the authors motivate the reasoning on why proposed method should work well with clear examples and visual examples in the method figure.
  2. For different model sizes, SSF shows clear improvements in following relation commands with little to no effect on general class condition performance. With SiT-B/2 the improvement is from 3->37% (success on original and flipped prompts) and 2->32% for SiT-XL/2.
  3. Visually the SSF approach also shows better relational prompt following than SF.
Critical weaknesses
  1. Under the current setup it is still a bit unclear if the decoy is the biggest contributor to improvement for relation based generation. While a model is trained without decoys for counting (Table 13), this is not done for relations. This makes it hard to understand if the improvement is from heavy noise on subject, keeping the reference clean or decoys. The decoys are also smaller than subject on average so that might also lead to the network being able to identify the subject more easily (Appendix A.3).
  2. Given humans prefer SF images more than SSF (46% vs 26%, Figure 3). Can we do a larger scale FID computation of images generated using spatial prompt for which we have real images too. It is unclear if overall in relation based cases is the model producing better images or not.
  3. A bit skeptical of qwen3 vl judge for the text to image results given the judge can make mistakes and also works very poorly on above and below (Appendix C). Maybe human evals here can help.
Questions for authors

I would appreciate if the authors can answer my questions and clarify around contributions, possible size discrepancy between subject and decoys, FID computation and vlm based evals.

4 comments

We thank the reviewer for finding our method easy to follow and its gains in relation following clear.

Contribution of each component (W1)

We train SiT-B/2 for 400k steps and remove one component at a time, giving its share of the 64 high-noise tokens to random tokens outside the reference (548 test prompts with nine initial noises each).

Variant (SiT-B/2, 400k steps)Subject noiseReference protectionDecoysOriginal (%)Flipped (%)Both (%)Sensitivity (%)FIDsFID
Self-Flow✗✗✗35.613.10.90.435.8745.86
SSF without subject noise and decoys✗✓✗53.534.423.25.235.2044.21
SSF without reference protection✓✗✓62.343.032.25.834.6346.16
SSF without decoys✓✓✗58.939.427.65.834.6245.40
SSF with random decoys✓✓random63.745.133.76.134.5046.16
SSF✓✓✓67.853.142.17.134.6045.38

Reference protection alone raises Both accuracy from 0.9% to 23.2% (SSF without subject noise and decoys), and removing the decoys or reference protection, or randomizing the decoy locations, lowers SSF from 42.1% to 27.6%, 32.2%, and 33.7%, respectively. Thus, all components contribute. With a second training seed, SSF again outperforms the variants we retrained (43.6% vs. at most 40.9%). The final version will report three seeds.

Size of the decoys (W1)

We agree that a decoy is on average smaller than the subject (about 15 vs. 43 tokens). Equalizing the sizes is not trivial, and we see three options:

  • (i) Moving noise from the subject to the decoys, i.e., decoys of the size of the subject with the same budget. The decoys then overlap the subject (35% of its tokens vs. 6.5% in SSF), and less of the subject is noised (66% vs. 73% of its tokens). We find that Both accuracy at 200k steps drops from 26.1% to 11.9%.
  • (ii) Less noise on the subject, i.e., high noise on only a decoy-sized part of the subject.
  • (iii) More noise on the decoys, i.e., decoys of the size of the subject with a larger budget (128 instead of 64 tokens), which departs from the budget of Self-Flow.

We are training (ii) and (iii), and will report their results during the discussion.

Image quality (W2)

On relation prompts, the images of SSF have a similar quality to those of Self-Flow and show the requested relation more often.

(i) FID. We generate the 548 test prompts with nine initial noises and both relations (9,864 images per model) and compare them with ImageNet, the source of the prompts. SSF has a lower FID with both model sizes, and its text-to-image FID-5k against COCO is also lower (25.35 vs. 26.22).

Model (9,864 images each)FIDsFIDPrecisionRecallInception Score
Self-Flow, SiT-XL/2, 1.3M steps34.734.60.760.10107
SSF, SiT-XL/2, 1.3M steps32.630.00.710.1483
Self-Flow, SiT-B/2, 400k steps35.041.30.840.07103
SSF, SiT-B/2, 400k steps30.428.20.790.1386

(ii) Human study. We also add a single-image study, since a head-to-head comparison can exaggerate a small difference in quality. We asked 34 raters to rate single images generated by SiT-XL/2 from 1 to 5. The images of SSF look only slightly worse (2.53 vs. 2.66), and they show the caption more often (61% vs. 46%) and (42% vs. 24%) for the flipped ones.

(iii) Additional metrics. We also add sFID, where SSF has the largest gap (28.2 vs. 41.3 with SiT-B/2), precision and recall [1], i.e., the fraction of generated images within the feature manifold of real images (fidelity) and vice versa (diversity), and the Inception Score [2], which rewards confidently classified images. Self-Flow has a higher precision and Inception Score, and SSF a higher recall, consistent with SSF drawing the requested layouts, including unusual ones, whereas Self-Flow draws the typical layout for both relations.

Reliability of the Qwen3-VL judge (W3)

To validate the judge, we re-evaluated all 2,372 text-to-image test images of each model with two recent and larger VLMs, using the same prompt and rule, and with OWLv2, applying the position rule of our ImageNet evaluation to its boxes.

Accuracy (%), Self-Flow / SSFPer imageBoth
Qwen3-VL-8B (paper)29.8 / 36.810.2 / 16.3
Qwen3.8 (27B)34.5 / 43.110.0 / 18.5
Gemma 4 (31B)31.7 / 40.98.1 / 16.9
OWLv247.3 / 58.817.6 / 31.3

Every judge finds SSF ahead, significant with 95% confidence.

We are happy to discuss any further questions or suggestions.

[1] Kynkäänniemi et al. Improved precision and recall metric for assessing generative models. NeurIPS, 2019.

[2] Salimans et al. Improved techniques for training GANs. NeurIPS, 2016.

AuthorsOpenReview ↗

Update on the size of the decoys

Size of the decoys (W1), all three options at 400k steps

Our response reported (i) at 200k steps, while (ii) and (iii) were still training. To equalize the sizes of the decoys and the subject, (i) moves noise from the subject to decoys of its size within the same budget of 64 tokens, (ii) applies high noise to only a decoy-sized part of the subject, and (iii) uses decoys of the size of the subject with a larger budget of 128 tokens. The table reports all three options at 400k steps, and the final version will report three seeds for all variants.

Variant (SiT-B/2, 400k steps, seed 1)Original (%)Flipped (%)Both (%)Sensitivity (%)FIDsFID
SSF without decoys58.939.427.65.834.6245.40
(i) Moving noise from the subject to the decoys58.839.628.46.134.6144.43
(ii) Less noise on the subject61.741.531.66.334.9345.59
SSF67.853.142.17.134.6045.38
(iii) More noise on the decoys, 128 tokens60.342.530.97.235.0745.35
SSF, 128 tokens67.351.339.77.934.7145.82

Equalizing the sizes lowers Both accuracy in all three options, significant with 95% confidence, and (i) is on par with SSF without decoys (28.4% vs. 27.6%). In (i) and (iii), the decoys overlap the subject (35% of its tokens vs. 6.5% in SSF), and in (ii), the visible half of the subject reveals its position without the relation word. This indicates that the smaller decoys let SSF noise the whole subject with little overlap. Moreover, the sizes do not let SSF ignore the relation (item 4 of the general response).

I thank the authors for coming back with the experimental results to answer the weaknesses and concerns. The new ablations that were run answer my main question around the contribution of each component. Also the remaining results give satisfactory answers to other weaknesses. I have raised my rating based on the rebuttal to 4.

Thank you for the constructive review and for raising your rating. We will add the component ablations and the analysis of the decoy size to the final version.

Comments on the paper

1 comment

We thank the reviewers for their constructive comments. We summarize the main additions below and will add them to the final version.

  1. Ablations. We train SiT-B/2 variants for 400k steps that keep only reference protection, remove reference protection, remove the decoys, or place the decoys at random locations, also with the same area as the decoys of SSF. SSF outperforms every variant in each training seed (tables below).
  2. Methods beyond Self-Flow. Choosing the removed tokens of MDT by the placement of SSF raises its Both accuracy from 1.6% to 27.2%. Moreover, CoMPaSS fine-tuning raises Both accuracy from 16.3% to 19.7% for SSF, whereas Self-Flow changes from 10.2% to 9.4%, i.e., SSF pretraining complements it.
  3. Mixing ratio. When both methods oversample the labelled images to 50% of every batch, SSF remains 6.3 points ahead (54.5% vs. 48.2%), significant with 95% confidence.
  4. Why the decoys help. When the noise pattern alone shows where the subject is, flipping the relation word still changes the prediction of SSF 98–99% as much as with a uniform noise level, compared to 23–27% for the model trained without decoys.
  5. A single-image human study. 34 raters rate single images from 1 to 5 and state whether each image shows its caption. The images of SSF show the caption more often (61% vs. 46%) and look only slightly worse (2.53 vs. 2.66).

In the ablations, we remove one component at a time, and its share of the 64 high-noise tokens goes to random tokens outside the reference. We evaluate 548 test prompts with nine initial noises each, i.e., 4,932 pairs. Sensitivity is the relative change of the predicted velocity when only the relation is flipped, and FID and sFID use the 4,932 images of the original relation against ImageNet. The final version will report three training seeds.

Variant (SiT-B/2, 400k steps)Subject noiseReference protectionDecoysOriginal (%)Flipped (%)Both (%)Sensitivity (%)FIDsFID
Self-Flow✗✗✗35.613.10.90.435.8745.86
SSF without subject noise and decoys✗✓✗53.534.423.25.235.2044.21
SSF without reference protection✓✗✓62.343.032.25.834.6346.16
SSF without decoys✓✓✗58.939.427.65.834.6245.40
SSF✓✓✓67.853.142.17.134.6045.38

We also replace the decoys of SSF by random decoys, with and without the same total area as the decoys of SSF.

Variant (SiT-B/2, 400k steps)Original (%)Flipped (%)Both (%)Sensitivity (%)FIDsFID
SSF with random decoys63.745.133.76.134.5046.16
SSF with area-matched random decoys65.647.135.15.834.9246.42
SSF67.853.142.17.134.6045.38

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.