- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study extrapolative data generation (generating data that is outside the support of the training data) from a structural perspective.
They first consider different orders of conditional data generation with graph structures, and find that the traditional method, which starts from the specifications (conditions) and generates the features (outputs) based on them, may leave the conditional probability undefined for some novel specifications. The authors then advocate adopting a new order/structure, where one instead starts from the features and compares and corrects them against the given specifications. They further provide statistical analysis on this new structure and provide identifiability guarantees.
Following the proposed structure, the authors further propose new weighting and guidance methods during sampling, called SEDGE, where they seem to first generate based on the first specification, and then resample/guide the sampling direction via the second specification. They verify their method on synthetic data, molecule generation, and also text-to-image generation.
Critical strengths
- The authors reconsider the traditional generation structure, and propose a novel framework for generating extrapolative data with better guarantees.
- The authors conduct extensive theoretical and empirical investigations based on their framework.
Critical weaknesses
-
I am unconvinced of the practicality of SEDGE, which seems to require lots of additional concepts and quantities and is not effective or flexible enough. As in the paper, the traditional method has the problem of giving an undefined probability for novel composed conditions, but only when there are no proper assumptions. However, in image generation, current diffusion models largely handle compositional generation[1, 2], and a common belief is that this is due to the generalizability of neural networks and text embeddings. The benefit of the new method is identifiability, which is less well-defined and important in image generation.
-
SEDGE only marginally improves over SANA on most metrics. SANA itself is an old backbone, so this may not compensate for the extra complexity SEDGE brings. Also, the images produced by SEDGE seem oversaturated.
-
To me, the writing is not clear enough. Are we always considering a fixed set of features and two specifications ? How can we formalize those under a general text-to-image generation setting?
[1] P. Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024.
[2] C. Saharia et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. NeurIPS 2022.
Questions for authors
Please refer to the weakness. I am not an expert in this area and my rating is preliminary.
7 comments
Official Rebuttal (Part 2/3)
W2: Given the choice of an older backbone, do the gains over SANA justify SEDGE’s added complexity? Some samples also appear oversaturated.
AW2: Thank you for raising these practical points. Below, we explain our choice of backbone and baselines, how to interpret the quantitative results, what SEDGE changes and whether it pays off, and the saturation of some samples.
Backbone and direct baseline choice. Aligner (ConceptAligner) is our primary baseline because it is the closest methodological comparison to SEDGE. It models structured dependencies between latent text and image concepts and studies when individual concepts are identifiable. In other words, Aligner follows the structure in Fig. 1(a), with links from specifications to features, whereas ours goes from features to specifications (Fig. 1(b)). Since Aligner builds on SANA, we also include the original SANA as a baseline. SEDGE’s formulation is independent of the choice of image backbone, and using the same SANA-based generator as Aligner lets us assess what our method contributes to extrapolative generation at the same backbone scale. We also compare qualitatively with newer and larger models, including SD3.5-Large, QwenImage, Z-Image, and GPT5.2, in Fig. 4. Following the focus of your question, we compare with SANA below.
Performance improvements. We appreciate your careful inspection of the results. The gains over SANA on the CLIP score and the GPT rating may look small, and we explain below how we read them.
CLIP score. The gain is smallest here (29.72 vs. 29.48). CLIP mainly measures general image–prompt alignment and, likely because of how it is trained, is less sensitive to how the objects are connected. This connection is important for judging whether an extrapolation is valid, and it often occupies only a small part of the image. For example, in Figure 4, the prompt asks for “a suitcase with bird wings.” SEDGE depicts wings attached to the suitcase itself, while SANA also depicts an unrequested bird, yet CLIP assigns the SANA image a higher score (30.86 vs. 28.49). We therefore regard the CLIP score as a reference for general prompt–image alignment rather than an indicator of extrapolation.
GPT rating. Across our 75 prompts, the mean GPT-rated prompt alignment score is 8.058 for SEDGE, 7.696 for SANA, and 7.691 for Aligner on a 0–10 scale. The mean gain is 0.362 points over SANA (95% CI [0.020, 0.787]) and 0.367 points over Aligner (95% CI [0.082, 0.711]), with both intervals computed by paired bootstrap resampling over prompts (Table 10). Both confidence intervals exclude zero, supporting a statistically significant improvement in GPT-rated prompt alignment over each baseline on this benchmark.
Overall, we believe the improvement over SANA in image generation is real, although modest on these two metrics. In text-to-image generation, extrapolation may be harder to test because of the rich pretrained features. We therefore encourage readers to also examine the generated images (Fig. 4 and Fig. 8).
What SEDGE changes and whether it pays off. Compared with Aligner, SEDGE keeps the same concept space and the same SANA-based generator. The main change is the direction of the concept-level model: instead of modeling image concepts given text concepts, it models text concepts given image concepts, and samples image concepts by guiding a diffusion prior with this likelihood (Algorithm 4). The benefit of this change is clearest where the novel combination is guaranteed to be unseen. In our synthetic experiments (Table 1), SEDGE-LWR improves validity over direct conditional diffusion in all five settings: Weighted Discrete 13.55% → 77.55%, Linear 83.22% → 91.98%, Cubic 78.12% → 91.26%, Smoothstep 85.55% → 91.55%, and 2D Bridge 80.65% → 87.39%. The gap widens at 33 dimensions (1–5% → 17–60%; see our response to Reviewer JvHd). In molecular generation, the target success rate improves from 25.20% (STGG+) to 37.49% (Table 2). In image generation, pretrained components may already have seen similar combinations, so the measured gains are smaller.
Saturation. We inspected all images in our manuscript and found that some displayed samples might appear saturated in Figure 9 in Appendix F.3.2. To quantify the difference across methods, we measured the mean HSV saturation of images re-rendered for the 75 prompts. At guidance strength , SEDGE has higher mean saturation than Aligner (0.371 vs. 0.294) but lower than the original SANA (0.453). We will report these measurements alongside the examples in Figure 9.
Official Rebuttal (Part 3/3)
W3: Does the formulation always assume a fixed feature set and two specifications, and how does it apply to general text-to-image generation?
AW3: No, the formulation does not assume a fixed feature set or two specifications. The example with three features and two specifications in Fig. 2(b) appears often because it is the simplest case that shows the whole logic of the structural view. Below, we explain (1) what this example is used for, (2) how it extends to more features and specifications, (3) how the features, specifications, and their structure can be learned, and (4) how all of this applies to text-to-image generation.
(1) The two-specification example. This is our running example: Section 2 introduces it as motivation (Fig. 1), and Section 4 reuses it for clarity. Section 3.2 uses two specifications, and , to show the two cases that matter for extrapolation: specifications that depend on separate features (Fig. 2(a)) and specifications that share features (Fig. 2(b)). The key questions already appear in this case: when the novel combination is identifiable, when valid samples can be constructed, and when the distribution can be recovered approximately.
(2) More features and specifications. These three key questions are answered in the general case. The formulation in Section 3.1 has features and specifications , and Section 3.3 and Appendix C.5–C.8 give the corresponding results under the matching structural and support conditions: identifiability when specifications do not share features (Theorem 1), construction of valid samples when they do (Theorem 2), and approximate recovery of the distribution (Theorem 3). The formulation is therefore not limited to two specifications.
(3) Learning the features, specifications, and structure. They do not have to be given. When the structure is unknown, we learn which features each specification depends on by fitting conditional likelihoods with sparse input masks (Section 4.2). When the features or specifications are not directly observed, we learn a representation of them, with identifiability conditions in Appendix D. Structure estimation is evaluated in Section 5.1 and Appendix F.1.6, and the case where features or specifications are not given in Appendix F.1.7.
(4) Text-to-image generation. This task combines (2) and (3). We use the concept representations from Aligner: contains 64 image concepts, each a 64-dimensional vector, and contains 64 text concepts, each a 768-dimensional vector. For our running example, “a peacock eating ice cream,” we encode the entire prompt without manually splitting it into two specifications, and the resulting 64 text-concept vectors jointly define the target specifications. The encoder fixes the number of concepts, while their values vary with the prompt.
The structure is learned as a soft mask whose rows correspond to text concepts and whose columns correspond to visual concepts. Each row indicates how strongly that text specification depends on each visual concept. When two or more rows assign substantial weight to the same column, that visual concept serves as a shared feature for those specifications, as does in Fig. 2(b). Algorithm 4 combines all 64 text-concept log-likelihoods to guide visual-concept sampling. We will add this mapping between the theoretical notation and the image implementation to Section 5.3.
We hope these clarifications and additional measurements address your concerns about practicality and scope. Thank you again for your feedback.
Official Rebuttal (Part 1/3)
Dear Reviewer wBLL,
We thank the reviewer for the feedback and address each concern below.
AW1: We respond in four parts: what compositional generation currently achieves, the key points of SEDGE, the role of identifiability, and the practical evidence.
Compositional generation for text-to-image. Thank you for the suggested citations. Modern text-to-image models, such as those in [1, 2], show that learned representations and inductive biases can support compositional generation. At the same time, compositional generation is not yet solved by scale. For the largest models in [1], GenEval accuracy is 0.98–0.99 for single objects but 0.33–0.40 for Position and 0.43–0.60 for Color Attribution. In our Fig. 4, recent large models, such as SD3.5-Large and QwenImage, produce concept leakage or spurious components on novel combinations. Therefore, the generalization of current models does not yet guarantee reliable generation for novel combinations.
The key points of SEDGE. We agree with the reviewer that extrapolation requires proper assumptions; SEDGE makes them explicit. It provides a structural view to analyze the extrapolation problem in a more general way: it does not assume that extrapolation is a compositional generation problem (Fig. 2(a) in our paper); instead, it derives conditions on the data that give the boundary of when extrapolation can be done feasibly and reliably, and when it is impossible. Such conditions matter because they tell whether the generated content can be relied on before more effort is invested in evaluating it.
The structural view provides a clear boundary. When the specifications do not share features (Fig. 2(a)), the novel distribution is identifiable (Proposition 1). When they share features (Fig. 2(b)), it is generally not identifiable (Proposition 2), but valid data points can still be constructed (Proposition 3), and the distribution can be recovered approximately when the excluded data are rare (Proposition 4). We found that the essential structural assumption is the conditional independence of the specifications given the features. The other conditions are not criteria for our method to work, but conditions on the data distribution. The algorithm is practical and only uses the structural information given by the data.
About identifiability. There are two identifiability results in our work. To avoid misunderstanding, we explain both. The first concerns the feature distribution under a novel combination of specifications (Section 3). It sets the boundary of the problem: if this distribution is not identifiable, models that fit the training data equally well can generate differently for the novel combination, and the data cannot tell which one is right. The second concerns the case where the features and specifications are not given (Appendix D); it is not the main contribution of our work. To answer your question more directly: we agree that identifying the whole distribution is less critical in image generation, where a valid image for the prompt is what matters. For that goal, the relevant result is the construction of valid data points (Proposition 3), which does not require identifiability.
From analysis to practical generation. Guided by the theoretical results, we developed practical algorithms that use the structural information for extrapolative generation. In synthetic experiments with controlled data-generating processes, SEDGE-LWR improves validity over conditional diffusion in all five settings (e.g., from 13.55% to 77.55% in the Weighted Discrete setting; Table 1). In real-world applications, we adapt the structural modeling to molecules and images.
Molecular design and image generation. We exclude the entire joint high-LogP, high-TPSA region from training while retaining coverage of the individual property targets. Across 520 held-out targets, SEDGE achieves 37.49% target success, compared with 25.20% for STGG+ (Table 2), where success requires satisfying both target tolerances. On the same A100 at batch size 32, SEDGE takes 26.4% less time per request at matched FP32 precision. For image generation, SEDGE retains the existing image generator and adds only a concept prior and a likelihood model; users provide an ordinary text prompt. On our 75-prompt benchmark, SEDGE improves GPT-rated prompt alignment over Aligner by 0.367 points on a 0–10 scale (paired-bootstrap 95% CI [0.082, 0.711]). These applications provide evidence for the practical value of the structural design.
Finally, we highlight that we do not introduce complex conditions or assumptions to make SEDGE work. We study the conditions on the data under which extrapolative generation can be done feasibly or reliably, and we provide practical samplers for this purpose from a structural perspective. Compared with existing work on extrapolation, we do not impose restrictions on the functional form of the generating process, so the framework applies broadly.