acceptodds
Under review as a conference paper at ICLR 2027

Structural Extrapolative Data Generation: Hardness, Feasibility, and Formal Conditions

Abstract

Achieving reliable extrapolation, specifically, extrapolative data generation beyond the support of the training distribution, is a fundamental challenge and inherently requires strong assumptions. This paper aims to address this problem and proposes a framework for Structural Extrapolative Data GEneration (SEDGE) based on suitable assumptions on the underlying data-generating process. We provide conditions under which data satisfying novel specifications can be generated reliably, together with the approximate identifiability of the distribution of such data under certain “conservative” assumptions, as well as the inherent non-identifiability of this distribution without such assumptions. On the algorithmic side, we develop two methods for extrapolative data generation: structure-informed likelihood-weighted resampling (LWR) and likelihood-guided diffusion (LGD). We verify the extrapolation performance on synthetic data and also consider extrapolative property-guided molecule design and extrapolative text-guided image generation as real-world scenarios to illustrate the validity of the proposed framework.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Extrapolative generation, graphical models, probabolistic models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wBLLOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study extrapolative data generation (generating data that is outside the support of the training data) from a structural perspective.

They first consider different orders of conditional data generation with graph structures, and find that the traditional method, which starts from the specifications (conditions) and generates the features (outputs) based on them, may leave the conditional probability undefined for some novel specifications. The authors then advocate adopting a new order/structure, where one instead starts from the features and compares and corrects them against the given specifications. They further provide statistical analysis on this new structure and provide identifiability guarantees.

Following the proposed structure, the authors further propose new weighting and guidance methods during sampling, called SEDGE, where they seem to first generate based on the first specification, and then resample/guide the sampling direction via the second specification. They verify their method on synthetic data, molecule generation, and also text-to-image generation.

Critical strengths
  1. The authors reconsider the traditional generation structure, and propose a novel framework for generating extrapolative data with better guarantees.
  2. The authors conduct extensive theoretical and empirical investigations based on their framework.
Critical weaknesses
  1. I am unconvinced of the practicality of SEDGE, which seems to require lots of additional concepts and quantities and is not effective or flexible enough. As in the paper, the traditional method has the problem of giving an undefined probability for novel composed conditions, but only when there are no proper assumptions. However, in image generation, current diffusion models largely handle compositional generation[1, 2], and a common belief is that this is due to the generalizability of neural networks and text embeddings. The benefit of the new method is identifiability, which is less well-defined and important in image generation.

  2. SEDGE only marginally improves over SANA on most metrics. SANA itself is an old backbone, so this may not compensate for the extra complexity SEDGE brings. Also, the images produced by SEDGE seem oversaturated.

  3. To me, the writing is not clear enough. Are we always considering a fixed set of features X=(X1,X2,…,Xn)X=(X_1, X_2, \dots, X_n) and two specifications (Z1,Z2)(Z_1, Z_2)? How can we formalize those under a general text-to-image generation setting?

[1] P. Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024.

[2] C. Saharia et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. NeurIPS 2022.

Questions for authors

Please refer to the weakness. I am not an expert in this area and my rating is preliminary.

7 comments
AuthorsOpenReview ↗

Official Rebuttal (Part 1/3)

Dear Reviewer wBLL,

We thank the reviewer for the feedback and address each concern below.

AW1: We respond in four parts: what compositional generation currently achieves, the key points of SEDGE, the role of identifiability, and the practical evidence.

Compositional generation for text-to-image. Thank you for the suggested citations. Modern text-to-image models, such as those in [1, 2], show that learned representations and inductive biases can support compositional generation. At the same time, compositional generation is not yet solved by scale. For the largest models in [1], GenEval accuracy is 0.98–0.99 for single objects but 0.33–0.40 for Position and 0.43–0.60 for Color Attribution. In our Fig. 4, recent large models, such as SD3.5-Large and QwenImage, produce concept leakage or spurious components on novel combinations. Therefore, the generalization of current models does not yet guarantee reliable generation for novel combinations.

The key points of SEDGE. We agree with the reviewer that extrapolation requires proper assumptions; SEDGE makes them explicit. It provides a structural view to analyze the extrapolation problem in a more general way: it does not assume that extrapolation is a compositional generation problem (Fig. 2(a) in our paper); instead, it derives conditions on the data that give the boundary of when extrapolation can be done feasibly and reliably, and when it is impossible. Such conditions matter because they tell whether the generated content can be relied on before more effort is invested in evaluating it.

The structural view provides a clear boundary. When the specifications do not share features (Fig. 2(a)), the novel distribution is identifiable (Proposition 1). When they share features (Fig. 2(b)), it is generally not identifiable (Proposition 2), but valid data points can still be constructed (Proposition 3), and the distribution can be recovered approximately when the excluded data are rare (Proposition 4). We found that the essential structural assumption is the conditional independence of the specifications given the features. The other conditions are not criteria for our method to work, but conditions on the data distribution. The algorithm is practical and only uses the structural information given by the data.

About identifiability. There are two identifiability results in our work. To avoid misunderstanding, we explain both. The first concerns the feature distribution under a novel combination of specifications (Section 3). It sets the boundary of the problem: if this distribution is not identifiable, models that fit the training data equally well can generate differently for the novel combination, and the data cannot tell which one is right. The second concerns the case where the features and specifications are not given (Appendix D); it is not the main contribution of our work. To answer your question more directly: we agree that identifying the whole distribution is less critical in image generation, where a valid image for the prompt is what matters. For that goal, the relevant result is the construction of valid data points (Proposition 3), which does not require identifiability.

From analysis to practical generation. Guided by the theoretical results, we developed practical algorithms that use the structural information for extrapolative generation. In synthetic experiments with controlled data-generating processes, SEDGE-LWR improves validity over conditional diffusion in all five settings (e.g., from 13.55% to 77.55% in the Weighted Discrete setting; Table 1). In real-world applications, we adapt the structural modeling to molecules and images.

Molecular design and image generation. We exclude the entire joint high-LogP, high-TPSA region from training while retaining coverage of the individual property targets. Across 520 held-out targets, SEDGE achieves 37.49% target success, compared with 25.20% for STGG+ (Table 2), where success requires satisfying both target tolerances. On the same A100 at batch size 32, SEDGE takes 26.4% less time per request at matched FP32 precision. For image generation, SEDGE retains the existing image generator and adds only a concept prior and a likelihood model; users provide an ordinary text prompt. On our 75-prompt benchmark, SEDGE improves GPT-rated prompt alignment over Aligner by 0.367 points on a 0–10 scale (paired-bootstrap 95% CI [0.082, 0.711]). These applications provide evidence for the practical value of the structural design.

Finally, we highlight that we do not introduce complex conditions or assumptions to make SEDGE work. We study the conditions on the data under which extrapolative generation can be done feasibly or reliably, and we provide practical samplers for this purpose from a structural perspective. Compared with existing work on extrapolation, we do not impose restrictions on the functional form of the generating process, so the framework applies broadly.

AuthorsOpenReview ↗

Official Rebuttal (Part 2/3)

W2: Given the choice of an older backbone, do the gains over SANA justify SEDGE’s added complexity? Some samples also appear oversaturated.

AW2: Thank you for raising these practical points. Below, we explain our choice of backbone and baselines, how to interpret the quantitative results, what SEDGE changes and whether it pays off, and the saturation of some samples.

Backbone and direct baseline choice. Aligner (ConceptAligner) is our primary baseline because it is the closest methodological comparison to SEDGE. It models structured dependencies between latent text and image concepts and studies when individual concepts are identifiable. In other words, Aligner follows the structure in Fig. 1(a), with links from specifications to features, whereas ours goes from features to specifications (Fig. 1(b)). Since Aligner builds on SANA, we also include the original SANA as a baseline. SEDGE’s formulation is independent of the choice of image backbone, and using the same SANA-based generator as Aligner lets us assess what our method contributes to extrapolative generation at the same backbone scale. We also compare qualitatively with newer and larger models, including SD3.5-Large, QwenImage, Z-Image, and GPT5.2, in Fig. 4. Following the focus of your question, we compare with SANA below.

Performance improvements. We appreciate your careful inspection of the results. The gains over SANA on the CLIP score and the GPT rating may look small, and we explain below how we read them.

CLIP score. The gain is smallest here (29.72 vs. 29.48). CLIP mainly measures general image–prompt alignment and, likely because of how it is trained, is less sensitive to how the objects are connected. This connection is important for judging whether an extrapolation is valid, and it often occupies only a small part of the image. For example, in Figure 4, the prompt asks for “a suitcase with bird wings.” SEDGE depicts wings attached to the suitcase itself, while SANA also depicts an unrequested bird, yet CLIP assigns the SANA image a higher score (30.86 vs. 28.49). We therefore regard the CLIP score as a reference for general prompt–image alignment rather than an indicator of extrapolation.

GPT rating. Across our 75 prompts, the mean GPT-rated prompt alignment score is 8.058 for SEDGE, 7.696 for SANA, and 7.691 for Aligner on a 0–10 scale. The mean gain is 0.362 points over SANA (95% CI [0.020, 0.787]) and 0.367 points over Aligner (95% CI [0.082, 0.711]), with both intervals computed by paired bootstrap resampling over prompts (Table 10). Both confidence intervals exclude zero, supporting a statistically significant improvement in GPT-rated prompt alignment over each baseline on this benchmark.

Overall, we believe the improvement over SANA in image generation is real, although modest on these two metrics. In text-to-image generation, extrapolation may be harder to test because of the rich pretrained features. We therefore encourage readers to also examine the generated images (Fig. 4 and Fig. 8).

What SEDGE changes and whether it pays off. Compared with Aligner, SEDGE keeps the same concept space and the same SANA-based generator. The main change is the direction of the concept-level model: instead of modeling image concepts given text concepts, it models text concepts given image concepts, and samples image concepts by guiding a diffusion prior with this likelihood (Algorithm 4). The benefit of this change is clearest where the novel combination is guaranteed to be unseen. In our synthetic experiments (Table 1), SEDGE-LWR improves validity over direct conditional diffusion in all five settings: Weighted Discrete 13.55% → 77.55%, Linear 83.22% → 91.98%, Cubic 78.12% → 91.26%, Smoothstep 85.55% → 91.55%, and 2D Bridge 80.65% → 87.39%. The gap widens at 33 dimensions (1–5% → 17–60%; see our response to Reviewer JvHd). In molecular generation, the target success rate improves from 25.20% (STGG+) to 37.49% (Table 2). In image generation, pretrained components may already have seen similar combinations, so the measured gains are smaller.

Saturation. We inspected all images in our manuscript and found that some displayed samples might appear saturated in Figure 9 in Appendix F.3.2. To quantify the difference across methods, we measured the mean HSV saturation of images re-rendered for the 75 prompts. At guidance strength η=1\eta=1, SEDGE has higher mean saturation than Aligner (0.371 vs. 0.294) but lower than the original SANA (0.453). We will report these measurements alongside the examples in Figure 9.

AuthorsOpenReview ↗

Official Rebuttal (Part 3/3)

W3: Does the formulation always assume a fixed feature set and two specifications, and how does it apply to general text-to-image generation?

AW3: No, the formulation does not assume a fixed feature set or two specifications. The example with three features and two specifications in Fig. 2(b) appears often because it is the simplest case that shows the whole logic of the structural view. Below, we explain (1) what this example is used for, (2) how it extends to more features and specifications, (3) how the features, specifications, and their structure can be learned, and (4) how all of this applies to text-to-image generation.

(1) The two-specification example. This is our running example: Section 2 introduces it as motivation (Fig. 1), and Section 4 reuses it for clarity. Section 3.2 uses two specifications, Z1Z_1 and Z2Z_2, to show the two cases that matter for extrapolation: specifications that depend on separate features (Fig. 2(a)) and specifications that share features (Fig. 2(b)). The key questions already appear in this case: when the novel combination is identifiable, when valid samples can be constructed, and when the distribution can be recovered approximately.

(2) More features and specifications. These three key questions are answered in the general case. The formulation in Section 3.1 has features X=(X1,…,Xd)\mathbf{X}=(X_1,\ldots,X_d) and specifications Z=(Z1,…,Zn)\mathbf{Z}=(Z_1,\ldots,Z_n), and Section 3.3 and Appendix C.5–C.8 give the corresponding results under the matching structural and support conditions: identifiability when specifications do not share features (Theorem 1), construction of valid samples when they do (Theorem 2), and approximate recovery of the distribution (Theorem 3). The formulation is therefore not limited to two specifications.

(3) Learning the features, specifications, and structure. They do not have to be given. When the structure is unknown, we learn which features each specification depends on by fitting conditional likelihoods with sparse input masks (Section 4.2). When the features or specifications are not directly observed, we learn a representation of them, with identifiability conditions in Appendix D. Structure estimation is evaluated in Section 5.1 and Appendix F.1.6, and the case where features or specifications are not given in Appendix F.1.7.

(4) Text-to-image generation. This task combines (2) and (3). We use the concept representations from Aligner: X\mathbf{X} contains 64 image concepts, each a 64-dimensional vector, and Z\mathbf{Z} contains 64 text concepts, each a 768-dimensional vector. For our running example, “a peacock eating ice cream,” we encode the entire prompt without manually splitting it into two specifications, and the resulting 64 text-concept vectors jointly define the target specifications. The encoder fixes the number of concepts, while their values vary with the prompt.

The structure is learned as a soft mask whose rows correspond to text concepts and whose columns correspond to visual concepts. Each row indicates how strongly that text specification depends on each visual concept. When two or more rows assign substantial weight to the same column, that visual concept serves as a shared feature for those specifications, as X2X_2 does in Fig. 2(b). Algorithm 4 combines all 64 text-concept log-likelihoods to guide visual-concept sampling. We will add this mapping between the theoretical notation and the image implementation to Section 5.3.

We hope these clarifications and additional measurements address your concerns about practicality and scope. Thank you again for your feedback.

Reviewer_55EGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structural Extrapolative Data Generation (SEDGE), a graphical-model framework for generating data under unseen combinations of individually observed specifications while explicitly modeling the selection process that excludes these combinations from training. It distinguishes conditions for constructing feasible samples from those for identifying their conditional distribution, and develops likelihood-weighted resampling and likelihood-guided diffusion methods evaluated on synthetic data, molecular design, and text-to-image generation.

Critical strengths
  • The paper offers a useful structural perspective on extrapolative generation. In particular, the distinction between specifications with and without shared features helps explain why unseen combinations can differ in difficulty. Separating the existence of feasible samples from identification of the full target distribution is also conceptually valuable.
  • The framework is supported by constructive algorithms and complementary experiments. The synthetic studies assess validity, distributional fidelity, and coverage, while the molecular and image experiments illustrate potential applications of the structural perspective beyond controlled distributions.
Critical weaknesses
  • The scope of extrapolation and the interpretation of the guarantees would benefit from a more unified explanation. Although the binary unseen-combination setting is explicitly defined, the transitions between individual specifications, specification groups, and continuous specification spaces are harder to follow. A worked example distinguishing population support, selected training support, and finite-sample absence would improve accessibility.
  • The connection between the theoretical assumptions and the real-world applications remains insufficiently established. The synthetic selection audits and recovery experiments are useful, but it is less clear whether the learned molecular and image representations satisfy the conditional-independence, shared-support, and selection assumptions required by the relevant results. The paper acknowledges that blockwise grouping and DPS guidance are approximate extensions that need not preserve all theoretical conditions. More application-specific diagnostics, or a clearer statement of which guarantees remain applicable, would help distinguish evidence for the theoretical framework from improvements obtained through structure-inspired modeling.
Questions for authors
  • Could the authors provide a unified discrete/continuous example clarifying the population, selection mechanism, and target support, and specify an error measure or bound for the approximate-identifiability result under Assumption 3?
  • Which theoretical assumptions are expected to hold—and which are explicitly relaxed—in the molecular and text-to-image implementations? Could the authors provide diagnostics of these assumptions and clarify how the image evaluation establishes unseen specification combinations relative to the training distribution?
8 comments
AuthorsOpenReview ↗

Official Rebuttal (Part 1/3)

Dear Reviewer 55EG,

We sincerely thank the reviewer for the careful reading and constructive suggestions. We address each concern below.

W1&Q1 (part 1): Can a unified example clarify individual, grouped, and continuous specifications, while distinguishing population support, selected training support, and finite-sample absence?

AW1&Q1 (part 1): Thank you for suggesting a unified example. We will add a worked example based on the Linear setting in Appendix F.1.1 to clarify these distinctions.

The example. Consider a population with independent features X1,X2,X3∼Unif[0,1]X_1,X_2,X_3\sim\mathrm{Unif}[0,1] and two specifications that share the feature X2X_2 (denoted CC in Appendix F.1.1):

Z1=0.8X1+0.6X2,Z2=0.8X3+0.6X2.Z_1=0.8X_1+0.6X_2,\qquad Z_2=0.8X_3+0.6X_2.

A sample is retained in training (S=1S=1) unless Z1∈I1=[0.3135,0.3165]Z_1\in I_1=[0.3135,0.3165] and Z2∈I2=[1.0835,1.0865]Z_2\in I_2=[1.0835,1.0865]. Thus, selection depends only on the specifications, and the entire combination I1×I2I_1\times I_2 is excluded from training. Each individual band remains represented with other values of the other specification.

The target is P(X∣Z1=0.315,Z2=1.085)P(\mathbf{X}\mid Z_1=0.315,Z_2=1.085). Its support is the line segment

x_1=\frac{0.315-0.6x_2}{0.8},\qquad x_3=\frac{1.085-0.6x_2}{0.8}.$$ **Three cases.** The following three specification pairs illustrate the distinction between feasibility, extrapolation, and finite-sample absence. Since the specifications are continuous, presence here refers to support and density, rather than positive probability of an exact value. | Specification pair $(z_1,z_2)$ | Population distribution | Selected training distribution | Interpretation | |---|---|---|---| | $(0.10,1.30)$ | Infeasible: $Z_1=0.10$ requires $X_2\le 1/6$, which implies $Z_2\le 0.90$ | Absent, although each individual value has positive marginal density | No valid sample exists; this is outside the feasibility conditions of our setup. | | $(0.315,1.085)$ | Positive density | Excluded by selection, although each individual value has positive marginal density | A novel combination and the extrapolation target of the paper. | | $(0.70,0.70)$ | Positive density | Positive density, even if no finite training sample has exactly these values | Finite-sample absence within the observed support. | In this example, the population feature support is the unit cube. Selection removes the region whose specifications lie in $I_1\times I_2$, and the target line segment lies inside that excluded region. However, the target-compatible pairs $(x_1,x_2)$ remain in the observed marginal support with other values of $x_3$, and the pairs $(x_2,x_3)$ remain with other values of $x_1$. The structural construction combines these observed components through their common shared-feature values. Thus, novelty in our setup concerns a combination excluded from the training distribution, rather than one missing only from a finite sample. At an observed specification value such as $(0.70,0.70)$, selection based only on the specifications gives $P(\mathbf{X}\mid\mathbf{Z}=\mathbf{z})=P^{\mathcal{D}}(\mathbf{X}\mid\mathbf{Z}=\mathbf{z})$. The conditional distribution is identified there, although it still needs to be estimated from finite data. **Binary, grouped, and continuous specifications.** The same example also connects the different specification types. For binary specifications, define $B_i=\mathbf{1}\\{Z_i\in I_i\\}$. Training contains $(B_1,B_2)\in\\{(0,0),(1,0),(0,1)\\}$ but never $(1,1)$, recovering the binary unseen-combination setting of Section 3.2. For specification groups, the condition is that each entire group occurs jointly in training, while the target combination across groups is absent. Observing every member separately is not sufficient to establish coverage of the group. For example, a bundle of attributes describing a peacock must be represented jointly before it can be treated as an observed group to combine with another specification group. For continuous specifications, positive-probability target bands provide a direct connection to the event-based formulation. Exact target values, such as $(0.315,1.085)$, are handled through conditional densities. In each case, the central distinction is the same: the requested combination must be feasible in the population, its individual specifications or groups must have observed coverage, and their joint target must be excluded from the training distribution. We will make these distinctions explicit in the revised presentation.
AuthorsOpenReview ↗

Official Rebuttal (Part 2/3)

W2&Q2: Which theoretical assumptions hold or are approximated in the molecular and image implementations? Can application-specific diagnostics clarify their scope and the novelty of the image combinations?

AW2&Q2: Thank you for highlighting the need to distinguish the theoretical conditions from their practical approximations. The synthetic experiments test the framework under known data-generating and selection mechanisms. In the molecular and image experiments, the underlying feature–specification structure is unknown, so we assess some assumptions indirectly through diagnostics on the learned representations. We report these diagnostics below.

Molecular generation. Following Section 3.2, let Z1=1Z_1=1 denote high LogP (LogP≥5\mathrm{LogP}\geq 5), Z2=1Z_2=1 denote high TPSA (TPSA≥100A˚2\mathrm{TPSA}\geq 100 \text{Å}^2), and S=0S=0 denote exclusion by the joint hold-out rule. By construction, S=0S=0 exactly when Z1=Z2=1Z_1=Z_2=1.

Assumption 1 (selection and marginal coverage) holds in the constructed dataset. The assumption requires P(S=1)>0P(S=1)>0, P(Z1=1,Z2=1)>0P(Z_1=1,Z_2=1)>0, and P(Zi=1∣S=1)>0P(Z_i=1\mid S=1)>0 for each ii. The joint region contains 1,042 molecules, all excluded by the rule. Among the 98,958 retained molecules, 9,766 have high LogP only and 11,561 have high TPSA only. Thus, the target combination exists in the original dataset, and each individual condition remains represented after selection. At the continuous property values used for evaluation, all 520 test targets also have training neighbors within each property's tolerance.

Assumption 2 (shared-feature overlap) is assessed empirically. It requires the shared feature X2X_2 to have positive density in both groups at a common value x~2\tilde{x}_2, i.e., pD(x~2∣Z1=1)>0p^D(\tilde{x}_2\mid Z_1=1)>0 and pD(x~2∣Z2=1)>0p^D(\tilde{x}_2\mid Z_2=1)>0, where pDp^D is the density after selection. We compared the high-LogP-only and high-TPSA-only training groups using the learned shared representation CC. At a fixed prefix length of 64 tokens, cross-group neighbors were about 18% farther away than within-group neighbors for high-LogP queries and 17% farther away for high-TPSA queries.

Assumption 3 (near-zero exclusion) is relaxed. It requires P(S=0)≈0P(S=0)\approx 0 and P(S=0∣Zi=1)≈0P(S=0\mid Z_i=1)\approx 0 for i=1,2i=1,2. Let PnP_n denote the empirical distribution of the n=100,000n=100{,}000 molecules. Overall, Pn(S=0)=1,042/100,000P_n(S=0)=1{,}042/100{,}000 (1.04%). Within the high-LogP group, Pn(S=0∣Z1=1)=1,042/10,808P_n(S=0\mid Z_1=1)=1{,}042/10{,}808 (9.64%). Within the high-TPSA group, Pn(S=0∣Z2=1)=1,042/12,603P_n(S=0\mid Z_2=1)=1{,}042/12{,}603 (8.27%).

Conditional independence in the molecular representation. First, does the other private block improve prediction once the shared block and the property's own block are available? On 9,953 in-distribution validation molecules at the 50% prefix, adding the other block increased standardized MSE by 0.00017 for LogP and reduced it by 0.00030 for TPSA. Both confidence intervals included zero. The additional block thus gave little benefit to this predictor.

Second, given all three blocks, does the other measured property improve prediction? In a separate analysis at the same prefix, we combined in-distribution and held-out validation molecules and weighted them to reflect the original region proportions. Adding measured TPSA reduced standardized MSE for LogP by 0.184, and adding measured LogP reduced it for TPSA by 0.194. These diagnostics characterize the information accessible to the fitted probes and leave exact conditional independence unresolved.

Text-to-image generation. The image data lack a controlled selection rule, and the true dependencies between visual and text concepts are unknown. At a post hoc mask threshold of 0.1, 124 of 4,096 possible connections remain. Of these, 69% connect to image concepts used by multiple text concepts. In the learned representation, text-concept residual norms remained correlated (mean pairwise correlation 0.28), so we treat the factorized likelihood as an approximation.

Unseen image combinations. We manually wrote seed prompts, expanded them with GPT, and manually selected 75 unusual compositions. Across 5,890,279 SEDGE training captions, one prompt matched verbatim after lowercasing and removing the leading article. We found two more matches by checking whether captions containing all prompt content words within 12 tokens described the same elements in the same relation. In total, 3 of 75 compositions were found, with no match detected for the other 72. Overlap with SANA and ConceptAligner pretraining data remains unknown.

AuthorsOpenReview ↗

Official Rebuttal (Part 3/3)

Q1 (part 2): Can the approximate-identifiability result under Assumption 3 be stated with an error measure or quantitative bound?

AQ1 (part 2): Thank you for this suggestion. We define the conservative solution and approximate identifiability below, to provide a base for the quantitative discussion. We then introduce a strengthened Assumption 3* yielding a quantitative Proposition 4*.

Let PD=P(X∣S=1)P^{\mathcal{D}}=P(\mathbf{X}\mid S=1) denote the observed distribution and P⋆=P(X∣Z1=1,Z2=1)P^\star=P(\mathbf{X}\mid Z_1=1,Z_2=1) the target distribution. Let pp denote the corresponding density. Same as the main setting of our manuscript shown in Fig. 2(b), S=1S=1 excludes the target case when Z1=1Z_1=1 and Z2=1Z_2=1. We first formally define the conservative solution, reusing the decomposition in Eq. (16) in the proof of Proposition 4 for p(X∣Z1=1,Z2=1)p(\mathbf{X} \mid Z_1=1, Z_2=1).

Conservative solution. We define

=\frac{1}{A} \frac{p^{\mathcal{D}}(X_2\mid Z_1=1)\ p^{\mathcal{D}}(X_2\mid Z_2=1)} {p^{\mathcal{D}}(X_2)} \ p^{\mathcal{D}}(X_1\mid X_2,Z_1=1) \ p^{\mathcal{D}}(X_3\mid X_2,Z_2=1),$$ where $A$ normalizes the distribution, and the ratio is set to zero where $p^{\mathcal{D}}(X_2)=0$. This construction replaces the population factors in the decomposition of $p^\star$ with their observed-data counterparts, so $\widehat P$ is a functional of $P^{\mathcal{D}}$ alone. Assumption 3 is what justifies this replacement. *Approximate identifiability.* Over a specified population model class $\mathcal{P}$, a target $\theta(P)$ is $\epsilon$-approximately identifiable in total variation (TV) if there exists a functional $T$ of the observed distribution such that $$\sup_{P\in\mathcal{P}} \operatorname{TV}\left(\theta(P),T(P^{\mathcal{D}})\right)\le\epsilon.$$ When $\epsilon=0$, exact identifiability is achieved. In our case, we have $\theta(P)=P^\star$, $T(P^{\mathcal{D}})=\widehat P$. Therefore, we need to give Assumption 3 quantitative bounds in order to derive approximate identifiability precisely. *Strengthened Assumption 3*.* For fixed constants $b_0,b_1,b_2\in[0,1)$, assume that, almost everywhere on the support of the target shared-feature $X_2$, $$P(S=0\mid X_2=x_2)\le b_0,\qquad P(S=0\mid X_2=x_2,Z_i=1)\le b_i,\quad i=1,2.$$ This condition provides a more precise and local characterization of the exclusion by the selection, compared to the averaged form of the original Assumption 3. *Quantitative Proposition 4\*.* Under Assumption 1, the structure in Fig. 2(b), and the strengthened Assumption 3*, the conservative distribution is well-defined and satisfies $$\operatorname{TV}(P^\star,\widehat P) \le \epsilon, \qquad \epsilon:=\tanh(B/4)\le B/4, \qquad B:=\sum_{i=0}^{2}\log\frac{1}{1-b_i}.$$ Thus, the target is $\epsilon$-approximately identifiable over the model class satisfying these fixed bounds and structural assumptions. The proof, which we will include in the revised manuscript, uses Bayes’ rule to express the target as a normalized reweighting of the conservative distribution. Then, the local exclusion bounds control the variation of the log-weight and hence bound the TV error. For Theorem 3, the same argument applies to $k$ specification groups under an explicit joint conditional factorization of the groups and their private feature blocks given the shared features. With the corresponding group coverage conditions and local exclusion bounds, the result uses $$B=(k-1)\log\frac{1}{1-b_0} +\sum_{i=1}^{k}\log\frac{1}{1-b_i}.$$ We will include the full proof and the extensions for multiple specifications in the revised appendix, after the existing equations and theorems, to keep the equation numbering consistent. We hope the worked example, application diagnostics, and quantitative bound address your concerns. Thank you again for your thoughtful suggestions.
Reviewer_JvHdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper takes a structural view on the extrapolation capabilities of generative models beyond their training data. It departs from the usual, restrictive approaches that constrain the generative process (e.g., linear, additive or compositional) where, rather than modeling specifications as parents of features, they model specifications as children of the features. A selection variable is introduced to explain why novel combinations might be missing from the training data, and which parts of the novel conditional distribution can survive. This is answered by a separation principle where, when specifications share no features, the novel distribution is identifiable; whereas when they share features, it is not. Though, in the unidentifiable scenario, it is argued that valid samples can still be constructed, and a “conservative” assumption that the unobserved regime is small restores an approximate identifiability result. A likelihood-weighted resampling and likelihood-guided diffusion algorithm is proposed to implement the resulting factorization, and are evaluated on synthetic data, property-guided molecule design, and text-to-image generation.

Critical strengths

The structural framing is a genuine contribution and the "peacock eating icecream" example makes the intuition land quickly. Introducing a selection variable and flipping the graph structure from "features-to-specifications" to "specifications-to-features" answers why a specification combination is absent, and separates the question of generating a valid sample from the question of recovering its distribution. Propositions 2 and 3 together say something non-obvious, that one can construct points satisfying a specification combination whose distribution is provably not identifiable, and this distinction seems worth more emphasis than it currently receives. The appendices are sufficiently thorough and the paper is candid about where its practical algorithms fall short of its theory as in blockwise grouping and DPS guidance failing to establish the required conditional independences.

Critical weaknesses
  1. It is unclear to me whether the theoretical claims are supported by sufficient experimental settings. The molecular experiment uses a blockwise approximation and the image experiment uses DPS guidance, which the authors acknowledge. Synthetic settings directed at the theory, are also relatively low-dimensional settings.

  2. The central theorety reads less formal and more like heuristics at times. Specifically, "approximately identifiable" and "conservative solution" are never formally defined. Assumption 3 and Theorem 3 would benefit from a metric or bound quantifying the approximation error and its relationship with the unobserved regime. It is also not clear how to make use of the shared-feature case, given what seems to be some tension between the setup requiring the novel region to be positive and the assumption requiring it to be near-null.

  3. The transition from theory to algorithms is asserted and the method section does not provide sufficient explanation as to why the anchor and completion models are conditional diffusions while the specification likelihood is fit by maximum likelihood, why two samplers are proposed rather than one, or what regime each is meant for. Some discussion of when each of the two samplers should be preferred, rather than only which scored better, would make the contribution more useful.

  4. The presentation needs a careful pass. There is a duplicated sentence, frequent typos, notational inconsistencies between the main body and the appendices, and unnecessarily long sentences with multiple subordinate clauses back to back.

Questions for authors
  1. Could the authors give a formal definition of approximate identifiability, conservative solution, and any error bounds? Relatedly, as I brought up in W2, is it plausible to have near-nullity and positivity of the novel region simultaneously?

  2. The paper repeatedly argues for avoiding strong assumptions, but isn't conditional independece of the specifications given the features a srong assumption?

  3. What is the practical difference between LWR and LGD proposals and what do the authors attribute LWR generally outperforming LGD? Importance resampling is generally expected to degrade as dimension grows while guidance is not, so I would expect the ordering to reverse in higher-dimensional settings than those tested. Also, in case I missed it, clarify which sampler is used in the molecule and image experiments reported.

8 comments
AuthorsOpenReview ↗

Official Rebuttal (Part 1/4)

Dear Reviewer JvHd,

We sincerely thank the reviewer for the careful reading and constructive suggestions, which will help us improve the rigor and clarity of the manuscript. We address each concern below.

W1&Q2: How do the experimental settings support the theoretical results? The synthetic experiments are relatively low-dimensional. And is conditional independence (CI) a strong assumption?

AW1&Q2: We first explain the theoretical claims we make, where CI is the essential assumption. Then, we explain how the experiments support the theoretical results. Finally, we provide additional results from higher-dimensional synthetic experiments.

Theoretical claims and CI assumption. Overall, our theory provides a structural perspective that gives conditions for when extrapolative generation is applicable and when it is not. Importantly, the conditions are not prerequisites for applying our methods; rather, they are conditions on the data that describe when extrapolative generation can be done reliably.

The conditions can be grouped into those regarding feasibility and those regarding difficulty. CI is the essential structural assumption for making extrapolative generation feasible. With CI, the following decomposition holds, i.e., given the features X\mathbf{X}, the specifications are mutually independent. Using the example of two binary specifications, we have

p(Z1,Z2∣X)=p(Z1∣X)p(Z2∣X).p(Z_1,Z_2\mid\mathbf{X})=p(Z_1\mid\mathbf{X})p(Z_2\mid\mathbf{X}).

This is the key to approaching p(X∣Z1=1,Z2=1)p(\mathbf{X}\mid Z_1=1,Z_2=1) from data that excludes the case where Z1=1Z_1=1 and Z2=1Z_2=1. However, without CI, any decomposition of p(X∣Z1=1,Z2=1)p(\mathbf{X}\mid Z_1=1,Z_2=1) would generally involve a joint-specification term such as p(Z2=1∣X,Z1=1)p(Z_2=1\mid\mathbf{X},Z_1=1), which cannot be identified from training data missing the target combination without additional assumptions.

Building on this, Assumption 2 is a mild condition under which one can obtain valid data points that satisfy the novel combination, and Assumption 3 is a condition under which the distribution can be recovered approximately, while sparsity only makes the search procedure for those data points easier.

CI is a weaker condition than those in existing work. Existing work on extrapolative generation usually imposes functional restrictions, such as additive decoders, compositional mechanisms, or linear latent relations (Lachapelle et al., 2023; Wiedemer et al., 2023; Saengkyongam et al., 2023). Compared to them, our CI condition allows nonlinear dependence on parent features, including interactions among them.

Synthetic and real-world experiments support the theoretical claims. The synthetic experiments directly test the theoretical claims under controlled data-generating processes, while the molecular and image experiments illustrate the usefulness of the structural approach in practice. Compared to the baselines, SEDGE-LGD/LWR make use of CI explicitly and decompose p(X∣Z1=1,Z2=1)p(\mathbf{X}\mid Z_1=1,Z_2=1) following the structure, and they improve over the baselines across the five settings on most metrics for evaluating extrapolative generation results (Table 1).

The real-world experiments adapt the structural factorization to practical settings, implemented by block-wise extension or DPS. For molecule design, our proposal shows an obvious improvement over STGG+ in the strictly extrapolative setting. For image generation, pretrained visual and language components may already have encountered the evaluated combinations in some way. Still, the structural method, which encourages sparsity, makes the extrapolative generation results better than those of the compared methods, especially in the image examples.

Additional synthetic experiments with higher dimensions. Following your suggestion, we extend all five settings from their original dimensions to 11, 22, and 33 total feature dimensions, by enlarging the private feature blocks, while keeping the shared-feature and specification dimensions the same. Across three seeds, LWR has higher validity than direct conditional diffusion (CD) in all 25 setting–dimension combinations, and LGD improves over CD in all five settings at both 22D and 33D. For instance, the 33D results are shown below (validity %, mean ± standard deviation across seeds).

Setting at 33DCDSEDGE-LGDSEDGE-LWR
Weighted Discrete2.9 ± 0.513.6 ± 3.116.7 ± 3.7
Linear4.4 ± 0.847.8 ± 2.959.7 ± 2.3
Cubic4.7 ± 1.340.1 ± 0.755.9 ± 2.6
Smoothstep3.5 ± 2.248.7 ± 4.359.8 ± 1.8
2D Bridge0.7 ± 0.618.9 ± 5.328.0 ± 8.9

The results show that the advantage of capturing the structural properties for extrapolative generation is consistent as the private feature dimension increases.

AuthorsOpenReview ↗

Official Rebuttal (Part 2/4)

W2&Q1: Can approximate identifiability and the conservative solution be formally defined with quantitative error bounds? Is near-nullity compatible with positive probability of the novel combination and the shared-feature overlap in Assumption 2?

AW2&Q1: Thank you for your suggestion. In the following, we address your two questions in turn.

Rigorous definitions with quantitative error bounds. We define the conservative solution and approximate identifiability below, to provide a base for the quantitative discussion. We then introduce a strengthened Assumption 3* yielding a quantitative Proposition 4*.

Let PD=P(X∣S=1)P^{\mathcal{D}}=P(\mathbf{X}\mid S=1) denote the observed distribution and P⋆=P(X∣Z1=1,Z2=1)P^\star=P(\mathbf{X}\mid Z_1=1,Z_2=1) the target distribution. Let pp denote the corresponding density. Same as the main setting of our manuscript shown in Fig. 2(b), S=1S=1 excludes the target case when Z1=1Z_1=1 and Z2=1Z_2=1. We first formally define the conservative solution, reusing the decomposition in Eq. (16) in the proof of Proposition 4 for p(X∣Z1=1,Z2=1)p(\mathbf{X} \mid Z_1=1, Z_2=1).

Conservative solution. We define

=\frac{1}{A} \frac{p^{\mathcal{D}}(X_2\mid Z_1=1)\ p^{\mathcal{D}}(X_2\mid Z_2=1)} {p^{\mathcal{D}}(X_2)}\ p^{\mathcal{D}}(X_1\mid X_2,Z_1=1)\ p^{\mathcal{D}}(X_3\mid X_2,Z_2=1),$$ where $A$ normalizes the distribution, and the ratio is set to zero where $p^{\mathcal{D}}(X_2)=0$. This construction replaces the population factors in the decomposition of $p^\star$ with their observed-data counterparts, so $\widehat P$ is a functional of $P^{\mathcal{D}}$ alone. Assumption 3 is what justifies this replacement. *Approximate identifiability.* Over a specified population model class $\mathcal{P}$, a target $\theta(P)$ is $\epsilon$-approximately identifiable in total variation (TV) if there exists a functional $T$ of the observed distribution such that $$\sup_{P\in\mathcal{P}} \operatorname{TV}\left(\theta(P),T(P^{\mathcal{D}})\right)\le\epsilon.$$ When $\epsilon=0$, exact identifiability is achieved. In our case, we have $\theta(P)=P^\star$, $T(P^{\mathcal{D}})=\widehat P$. Therefore, we need to give Assumption 3 quantitative bounds in order to derive approximate identifiability precisely. *Strengthened Assumption 3*.* For fixed constants $b_0,b_1,b_2\in[0,1)$, assume that, almost everywhere on the support of the target shared-feature $X_2$, $$P(S=0\mid X_2=x_2)\le b_0,\qquad P(S=0\mid X_2=x_2,Z_i=1)\le b_i,\quad i=1,2.$$ This condition provides a more precise and local characterization of the exclusion by the selection, compared to the averaged form of the original Assumption 3. *Quantitative Proposition 4\*.* Under Assumption 1, the structure in Fig. 2(b), and the strengthened Assumption 3*, the conservative distribution is well-defined and satisfies $$\operatorname{TV}(P^\star,\widehat P) \le \epsilon, \qquad \epsilon:=\tanh(B/4)\le B/4, \qquad B:=\sum_{i=0}^{2}\log\frac{1}{1-b_i}.$$ Thus, the target is $\epsilon$-approximately identifiable over the model class satisfying these fixed bounds and structural assumptions. The proof, which we will include in the revised manuscript, uses Bayes’ rule to express the target as a normalized reweighting of the conservative distribution. Then, the local exclusion bounds control the variation of the log-weight and hence bound the TV error. For Theorem 3, the same argument applies to $k$ specification groups under an explicit joint conditional factorization of the groups and their private feature blocks given the shared features. With the corresponding group coverage conditions and local exclusion bounds, the result uses $$B=(k-1)\log\frac{1}{1-b_0} +\sum_{i=1}^{k}\log\frac{1}{1-b_i}.$$ **Positivity and near-nullity of the novel region.** We interpret the novel region to be the unobserved combination $Z_1=1$, $Z_2=1$. Positivity is Assumption 1: $P(Z_1=1,Z_2=1) \gt 0$. Near-nullity is Assumption 3: $P(S=0)\approx 0$ and $P(S=0\mid Z_i=1)\approx 0$ for $i=1,2$. They are compatible because each individual specification can be well covered by the selected data, while an unselected combination can be rare but still possible. In our manuscript's example, we may have observed almost all scenarios of "peacock" or "eating ice cream" but never "peacock eating ice cream", although it is possible in the whole population. We further give a numerical example. Suppose that, at every value of the shared feature $X_2$, each specification holds with probability $r=0.01$, independently of the other, and that selection excludes exactly the novel combination. Then $$P(Z_1=1,Z_2=1)=P(S=0)=r^2=10^{-4} \gt 0, \qquad P(S=0\mid Z_i=1)=r=0.01.$$ So the novel region has positive probability (Assumption 1), while only 0.01% of all samples and 1% of the samples of each individual specification are excluded (Assumptions 3 and 3\*). Positivity and near-nullity therefore hold simultaneously.
AuthorsOpenReview ↗

Official Rebuttal (Part 3/4)

W3: How does the theory lead to the model choices? Why are two samplers proposed? What regime is each meant for?

AW3: We answer the three questions in order.

From theory to model choices. The three model components correspond to the three factors of the conservative solution in Eq. (17) in the proof of Proposition 4:

\underbrace{p^{\mathcal{D}}(X_1,X_2\mid Z_1=1)}_{\text{anchor}} \underbrace{P^{\mathcal{D}}(Z_2=1\mid X_2)}_{\text{specification likelihood}} \underbrace{p^{\mathcal{D}}(X_3\mid X_2,Z_2=1)}_{\text{remaining-feature conditional}}.$$ The construction determines the role of each component, not the model family. The anchor and the remaining-feature conditional are distributions over features from which we need to generate samples, so any conditional generative model can be used, e.g., diffusion models, normalizing flows, or autoregressive models. The specification likelihood serves a different purpose: it is evaluated at the requested specification $Z_2=1$ and used to weight or guide samples according to their shared features $X_2$, so any model with a tractable conditional likelihood can be used. Among these alternatives, we use conditional diffusion models for the anchor and the remaining-feature conditional, as a flexible and widely used class of conditional generators. We fit the specification likelihood by conditional maximum likelihood, using a conditional spline flow, because it can then be evaluated directly and efficiently. In Appendix F.1.3, we have compared alternative likelihood models. **Why two samplers.** Both samplers target the same $\widehat p$ but incorporate the specification likelihood in different ways. LWR generates candidates from the anchor and resamples them according to the likelihood weights. LGD incorporates the second specification during denoising through a learned noise-aware likelihood gradient. Both then use the remaining-feature conditional to generate the remaining private features. We propose both because they trade off differently: LWR needs no additional model but relies on a finite candidate pool, while LGD avoids the pool but requires an additional noise-aware likelihood model and gradient evaluations during denoising. **When each sampler is preferred.** LWR is preferred when the anchor produces enough compatible candidates within the available sampling budget. Its effective sample size (ESS), together with the fraction of candidate pools in which all weights are zero, provides a practical diagnostic of this. Increasing the candidate budget can reduce Monte Carlo error, but does not remove errors in the learned anchor, specification likelihood, or remaining-feature conditional. LGD is intended for the regime where compatible candidates are too rare for the available budget, and its quality depends on the accuracy of the learned guidance gradients; the same consideration motivates the guidance used in our image experiment. Following your suggestion, we additionally test this in an extension of the Linear setting with $m$ shared coordinates in $X_2$ and $p$ private coordinates in each of $X_1$ and $X_3$. LWR uses a candidate count of $K=1,024$ per generated sample. Results are over three seeds, each with 32 targets × 32 draws. Entries are validity (%), mean ± standard deviation / mean joint MMD. For LGD, the noise-aware likelihood is trained three times per seed and the results are averaged. | $m$ | $p$ | Total dim. | SEDGE-LWR | SEDGE-LGD | LWR ESS | All-zero pools | |---:|---:|---:|---:|---:|---:|---:| | 1 | 1 | 3 | 91.1 ± 0.6 / 0.0024 | 73.5 ± 7.0 / 0.0151 | 95.6 | 0% | | 1 | 4 | 9 | 84.8 ± 2.6 / 0.0213 | 69.5 ± 2.2 / 0.0280 | 100.2 | 0% | | 1 | 16 | 33 | 59.4 ± 1.5 / 0.0277 | 46.4 ± 2.8 / 0.0296 | 88.6 | 0% | | 2 | 2 | 6 | 77.9 ± 2.2 / 0.0039 | 47.6 ± 6.1 / 0.0147 | 11.2 | 0% | | 3 | 3 | 9 | 41.2 ± 0.9 / 0.1843† | 41.4 ± 1.8 / 0.0161 | 1.5 | 47.2% | † To still report numbers in this row, pools in which all weights are zero are sampled uniformly. Algorithm 1 itself would report insufficient candidate coverage for such pools. ESS is computed over the remaining pools. LWR has higher validity and lower MMD than LGD in the four settings where ESS stays well above 1. With three shared coordinates, ESS drops to 1.5 and nearly half of the candidate pools have all-zero weights. In that case, LGD has comparable validity and avoids the degradation in MMD. In our response to your Q3, we provide further details. We will add these explanations to Section 4 and provide more details in the appendix. **W4:** The presentation needs revision, including duplicated sentences, typos, notational inconsistencies, and long sentences. **AW4:** We thank the reviewer for the careful reading. We will revise the manuscript to remove duplicated sentences, correct typos, standardize notation between the main text and appendices, and shorten long sentences. We will mark the revisions in blue in the updated PDF.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for the time and care you devoted to reviewing our paper. We appreciate all your constructive suggestions. We have taken them seriously and revised our manuscript accordingly. We have uploaded the revised version and summarize the main changes below.

  • Approximate identifiability (Appendix C.9). We now define approximate identifiability in total variation and prove an explicit bound on the recovery error under local exclusion conditions. This makes the guarantee more precise by showing how the recovery error depends on the amount of local exclusion. [JvHd W2/Q1, 55EG Q1]
  • Choice of sampler (Appendices E.3 and F.1.9). We clarify when to prefer LWR or LGD. New experiments show that increasing the private feature dimension alone had little effect on LWR’s effective sample size, while increasing the candidate pool improved validity when importance weights were concentrated. [JvHd W3/Q3]
  • Real-world experiments (Sections 5.2–5.3 and Appendices F.2–F.3). We clarify which assumptions hold by construction and which are approximated, and report new diagnostics for the learned molecular and image representations. For the image benchmark, we also checked for compositional overlap with the training captions and measured saturation across methods. [JvHd W1/Q2, 55EG W2/Q2, wBLL W2/W3]

We also corrected typos throughout the manuscript. Revised text is highlighted in blue.

As generative models become increasingly capable, we believe it is important to understand when they can reliably generate samples for specification combinations unobserved during training. Our work takes a step toward this understanding by providing a general structural framework that clarifies the possibilities and limits of extrapolative generation without prescribing a particular parametric form for the underlying mechanisms.

Thank you again for your thoughtful feedback. We welcome further questions and look forward to continuing the discussion.

I would like to thank their authors for their rebuttal. Dear Reviewers, as we are approaching the end of the author-reviewers discussion period, I would appreciate it if you could take a moment to acknowledge the rebuttal and see if it addresses your concerns or no. Thank you for your time and service.

AuthorsOpenReview ↗

Thank you for a productive discussion

Dear Area Chair,

Thank you for handling our submission and for encouraging the discussion. As the discussion period closes, we would like to briefly summarize a productive rebuttal, which has improved the rigor and clarity of the manuscript both theoretically and empirically. None of this would have been achieved without the thoughtful suggestions and feedback of the reviewers.

During the discussion, Reviewers 55EG and wBLL raised their ratings to 3, and Reviewer JvHd stated that they lean towards raising their score to a weak accept, noting that the structural framework and the formal results are now in good shape.

Most points are already reflected in the revised PDF. In their latest comments, Reviewers JvHd and 55EG asked how far the real-data experiments support the framework. We replied in the discussion and summarize our answer here. These experiments implement the central structural view of the paper: conditional independence and sparsity are imposed by the model, which is learned from raw data under this structure, and generation is carried out in the learned representation. The molecular ablation indicates that this structure contributes to the gain (target success of 25.20% for STGG+, 28.96% for SEDGE (blocks only), and 37.49% for SEDGE; Table 2). Assumptions 2 and 3 are conditions of a different kind: they concern the data itself and cannot be enforced by any method. We therefore report diagnostics on how closely the real data follow them (Appendices F.2.3, F.3.4 and F.4).

We have updated the revision once more to include this discussion in Appendix F.4. Furthermore, we will make the contributions of conditional independence and sparsity clearer empirically.

To summarize our paper: on the theory side, it characterizes when generating data for unseen combinations of specifications is hard, when it becomes feasible, and under which conditions. On the practical side, it shows how this structure can be built into existing molecular and image generators. We believe this view is useful as generative models become more capable and widely adopted, and are increasingly asked to produce such combinations across a wide range of domains.

We thank the reviewers again for a constructive discussion.

Best regards,

Authors of Submission 22981

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.