- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies geometric compositional generation in diffusion models. The authors decompose the total generation error into two sources: inference-time approximation error and distributional estimation error. From this perspective, they analyze the sensitivity of geometric compositions to source-distribution estimation errors and show that even small estimation errors can be significantly amplified when the target distribution is poorly covered by the source distributions. They further demonstrate that improving inference-time approximation through correction methods does not necessarily improve compositional generation and may even worsen performance when distributional estimation errors are present.
Critical strengths
The paper provides an interesting theoretical analysis showing that even small errors in estimating source distributions can lead to large errors in their geometric composition. Another interesting empirical evidence that despite reducing inference-time approximation error in principle, can worsen compositional generation when source distributions are imperfectly estimated.
Critical weaknesses
Concerns with the paper
Interpretation of the two error sources: Equation (3) only provides an upper bound through the triangle inequality. The two error terms do not form an exact additive decomposition of the total error. Furthermore, the underlying deviations may counteract one another, meaning that reducing either error term does not necessarily reduce the total generation error. The authors should clarify the practical significance of separately analyzing these two errors and how their interaction affects compositional generation.
Consider three one-dimensional Gaussian distributions with identical variance:
$ \hat P^w=\mathcal N(2,1) $
$ \tilde P=\mathcal N(0,1). $
Suppose we use the Wasserstein-2 distance. For Gaussians with identical variance, this is simply the distance between the means.
undefinedQuestions for authors
Addressing all the weakness will answer my questions
7 comments
Response (part 2)
W4: Conditional independence and inference-time approximation
It is true that conditional independence need not be preserved during noising. Note that the Factorized Conditionals assumption is slightly stronger than conditional independence, and is preserved during noising (Lemma 2 in G.5). We can attribute increased failures in non-Factorized settings (compared to their Factorized counterpart) to inference-time approximation error precisely because of this. Of course, this can be muddied by introducing estimation error, which is also what our experiments are designed to analyze.
W5: Insufficient experimental separation of errors
We disagree. Our synthetic experiments explicitly include the analytic score functions, where there is no distributional estimation error, and the Factorized Conditional distributions are designed to incur no inference-time approximation error. This allows us to directly measure the impacts of each source of error and their interplay.
The two error terms are not directly measured, limiting the conclusions about their relative importance.
For our non-synthetic settings, the terms cannot be directly measured, hence our setup. We carefully design experiments so that they vary whether the target is ID (low estimation error) or OOD (high estimation error), and vary inference-time approximation error by varying whether they are approximately factorized (low approximation error) or non-factorized (high approximation error). The experimental results follow the same patterns as those in the Gaussian settings, as well as the intuition from Section 3.2, giving us high confidence that the experimental settings are effective at both disentangling the two errors and studying their combination (depending on quadrant).
W6: Limited practical contribution
We agree that the framework does not directly establish how to improve compositional generation. Our goal is complementary: we seek to provide a deeper understanding of the problem of compositional generation, identifying two highly relevant sources of error and analyzing their interplay. We believe scientific understanding of a challenging problem is by itself valuable, whether or not it immediately produces a solution.
However, it can also be demonstrated how a straightforward application of our analysis can immediately yield better results by selecting the inference procedure. As a simple instantiation of this, the rESS sensitivity metric can be combined with the factorization heuristic in Appendix E.2 to determine whether the target composition is likely factorized or not and if it is likely OOD or not. This can then be used to select between FKC (K=16) and naive composition, which here would yield up to +20.1%/+10.4% success rate (rooms/faces) or 16x cheaper inference over using only naive composition or using only FKC, depending on the setting.
Additionally, the sensitivity metric may be used by practitioners to assess when compositions may be unreliable, without access to ground-truth data from the composition. Lastly, as a straightforward proof-of-concept, we demonstrate with an additional experiment that the sensitivity metric can be used to decide how to allocate data to each source when such data is expensive to collect (see our response to Reviewer Kkz4, Q4 for further details).
So, while the goal of our work is not to propose methods to improve compositional generation, the analysis it provides can be used to do so, and we are hopeful that the increased scientific understanding of the problem will be used to build further progress. We will expand our current discussion of these implications in the Discussion section.
References
[1] Block et al. (2022), Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
[2] Chen et al. (2023), Sampling is as Easy as Learning the Score
[3] Chen et al. (2023), Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions
[4] Benton et al. (2024), Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization
[5] Chen et al. (2023), Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data
[6] Oko et al. (2023), Diffusion Models are Minimax Optimal Distribution Estimators
W1 Interpretation of the two error sources
I understand that Equation (3) provides a mathematically valid decomposition and that the two terms correspond to distinct error mechanisms. However, I am still unclear about the extent to which this separation is informative for understanding the total compositional error.
The triangle inequality permits introducing any intermediate distribution, so its validity and worst-case tightness do not themselves justify the explanatory value of this particular separation.
More importantly, both terms in Equation (3) depend on the estimated source distributions. Consequently, the inference-time approximation error is not independent of distributional estimation error, and the two errors may reinforce or compensate for one another.
While the authors provide specific examples demonstrating such behavior, I do not see a general mathematical justification for analyzing these errors separately when drawing conclusions about overall compositional performance.
Could the authors clarify whether their decomposition is intended primarily as an operational distinction between two error mechanisms, or whether they claim that the separate error terms provide a meaningful characterization of the total error beyond a worst-case upper bound? If the latter, what theoretical result supports this interpretation?
We thank the reviewer for their thorough feedback. Below, we address the weaknesses raised and what we believe may be some conceptual misunderstandings about the content of the paper.
Weaknesses
W1: Interpretation of the two error sources
We agree that the underlying deviations can cancel one another out. However, this worst-case upper bound is very standard in statistical analysis, see for instance [1, 2, 3, 4]. We reiterate that, without further assumptions, the triangle inequality in Equation (3) can be tight, and the bound in Theorem 1 is tight (distributions exist whose composition error matches the upper bound up to constant factors), though we agree that exactly characterizing what further assumptions are both reasonable and useful for analysis of the interactions is a promising (but very challenging) line of future work.
A large portion of our analysis is dedicated to how the two error sources interact in practice, and we characterize four separate regimes, the quadrants of Figure 2, Figure 3, and Table 1, where these interactions (or lack thereof) play out differently. We additionally provide several theoretical examples highlighting a form of 'bias-variance' tradeoff that can occur between naive composition and exact composition under estimation errors (Section 3.2, and Corollaries 1.1, 1.2 in Appendix G.4). Consequently, we feel this separation is already clearly justified. If the reviewer has specific concerns or questions about these points, we would be happy to provide further clarification.
W2: Limited scope of compositional generation
We acknowledge that the form of composition we study is not all-encompassing, and there are some notions of composition which do not fit under this umbrella. We focus on weighted composition because of its rigorous motivation through identifiability theory (Section 2) and causal representation learning (Appendix B), recent methodological advances like FKC that reduce inference-time approximation error, and because it covers a wide range of practical settings (Section 1), including applications in biology, classifier-free guidance, Bayes composition, and popular forms of score-level arithmetic, for instance.
Indeed, CoInD by Gaudi et al. (2025) explicitly uses the notion of geometric composition expressed at the score level (their Section 2, Eqs 1-3), which falls under our Definition 1. In fact, the CoInD training objective explicitly penalizes differences between the joint score and the score corresponding to the compositions of the marginal scores, though as discussed in our Section 3.2 the score-addition identity need not hold at intermediate timesteps. In fact, the CoInD objective can be viewed as a regularization encouraging agreement between joint conditioning and naive score composition, and thus it may exhibit a kind of bias-variance tradeoff as discussed for naive composition in our work; our analysis may be directly useful for understanding these contributions.
Okawa et al. (2023) define classes as attribute tuples, and define a concept graph describing relationships between classes, but do not provide a formal definition of the target distribution in terms of the source distributions. Such settings fall within our framework if the underlying data-generating process admits the causal representation and modular structure described in Appendix B, which yields our weighted composition formula. For instance, their (shape, size, color) experiment could be represented this way. We would be happy to provide further details if the reviewer would find it helpful, but for now we omit formal notation for the sake of brevity.
We will add these as cited examples to the paper and discuss the connection between weighted composition and logical score operations, as in CoInD, and the Okawa et al. concept graph framework.
W3: Insufficient explanation of distributional estimation error
Nearly all statistical learning techniques produce estimation errors. The fact that estimation errors arise is unavoidable and taken as a given. For further details on properties of estimation error in diffusion models, we recommend [5, 6], but this is not our focus.
We disagree that the paper "does not adequately explain why models can accurately generate individual conditions while failing at their composition". This is exactly the point of the paper: estimation error on individual conditions can be low, so that models can accurately generate individual conditions, but this does not imply that they will accurately generate from the composition. This is directly established in Proposition 1 and Theorem 1, and further discussed in Section 3.2 where inference-time approximation error adds another potential failure mode.