acceptodds
Under review as a conference paper at ICLR 2027

Breakdown of Local Denoising as Semantic Speciation

Abstract

The dynamics of generative models exhibit two apparently distinct temporal windows: a speciation window, in which a sample commits to a semantic class, and a nonlocality window, in which local context windows become insufficient for generation. Motivated by evidence of their near-concurrence in a variety of frontier models, we investigate their relationship through the spatial distribution of semantic information. Under a "common cause" hypothesis, we prove that the nonlocality window must lie in the speciation window. This hypothesis postulates that semantic labels explain a fraction of the correlations between distant tokens, a condition that is natural for many real datasets. We further give conditions under which both windows shrink to a single limiting time as system size grows, defining a “phase transition”, and verify this behavior analytically in Gaussian mixtures. Together, these results identify conditions under which semantic information explains the concurrence of speciation and nonlocality, connecting two complementary perspectives on the emergence of semantic structure in generative modeling.

Primary area:
generative models
Keywords:
generative models, diffusion model, critical window, semantic speciation, local denoising, information theory, phase transition, context window

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DYP4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Under the "common-cause" assumption, this work establishes that the non-locality window—the period during which non-local features influence generation—must lie within the speciation window, during which class-label information influences generation.

Critical strengths

This paper makes an interesting connection between non-local feature influence and label influence. The authors provide substantial theory under certain assumptions, and the derivation is theoretically correct. The authors also perform an empirical study to validate their theory.

Critical weaknesses

Questioning assumptions made by the authors

1. Common cause Assumption (Misleading naming convention. This implies completely different things in the causality literature.)

Even when a semantic variable S is a genuine common cause of regions A and C, conditioning on the intermediate region B can preserve or increase their residual dependence, violating the proposed inequality. Therefore, under the "common-cause" assumption,

It(A;C∣B,S)≤(1−α)It(A;C∣B),0<α≤1(1)I_t(A;C\mid B,S) \leq(1-\alpha)I_t(A;C\mid B), \quad 0<\alpha\leq 1 \qquad(1)

does not always hold.

2. The empirical justification that Eq. (1) holds true for natural images is incorrect

The experiments on Stable Diffusion 3 measure reductions in the local–global velocity prediction gap rather than the CMI quantities appearing in the hypothesis. A reduction in the prediction gap does not necessarily imply a corresponding reduction in CMI. Therefore, the empirical evidence does not directly establish that Hypothesis 1 holds for natural images.

How does this assumption hold in practice? Here is what the authors claim:

Δloc(t)4≤K(t)It(A;C∣B)\Delta_{\mathrm{loc}}(t)^4 \leq K(t)I_t(A;C\mid B)

But observing

Δconditional<Δunconditional\Delta_{\mathrm{conditional}} < \Delta_{\mathrm{unconditional}}

does not imply

It(A;C∣B,S)<It(A;C∣B).I_t(A;C\mid B,S) < I_t(A;C\mid B).

The authors should provide more direct validation of the assumption or clarify the limitations of applying the containment theorem to realistic data distributions. Based on Hu et al., for natural image datasets (for example, CIFAR), relying on the mentioned results, I would request that the authors provide evidence for all time steps tt.

It(A;C∣B,S)≤(1−α)It(A;C∣B)I_t(A;C\mid B,S) \leq(1-\alpha)I_t(A;C\mid B)

3. Core gap suppression assumption also does not hold.

This assumption is not always true and does not hold for all annulus tripartitions.

Counterexample: A zebra with black and white stripes. Imagine a close-up photograph of a zebra. AA: A small patch of its body, potentially containing a black or white stripe. BB: The surrounding annulus, containing nearby stripes and texture. CC: A distant patch of the same zebra's body. SS: The semantic label describing a particular stripe pattern, such as "the two selected patches have opposite colors."

Minor Corrections

  1. L32: "Such features must therefore emerge during inference." How do the authors validate this claim? The "must" claim does not hold.

  2. "Generation of a pixel is determined by its neighborhood and not the image" (Kamb and Ganguly, 2024). The attribution is wrong. The cited work establishes that, under a locality constraint, the instantaneous score prediction at a pixel depends only on its local neighborhood. This does not imply that the final generated pixel is independent of global image information, since information can propagate through successive denoising steps. Furthermore, the analysis primarily concerns convolutional diffusion models and does not establish this property for diffusion models in general. The authors should clarify this distinction and qualify their claim accordingly.

  3. The authors push all the relevant experiments into the appendix, which is a slight abuse of the paper limit.

References

Hu et al., Local Diffusion Models and Phases of Data Distributions.

Questions for authors

Addressing all the weaknesses will answer all my questions. I would be willing to raise my score if the authors provide appropriate reasoning for the weaknesses.

2 comments
AuthorsOpenReview ↗

Rebuttal part 1

Thank you for the review, and for finding the connection between nonlocal feature influence and label influence interesting and the derivations correct. We address your concerns in four points.


1. "Common cause Assumption (Misleading naming convention. This implies completely different things in the causality literature.)" (Weakness 1)

We acknowledge that "common cause" has a different meaning in causal inference. We are happy to rename the assumption to avoid this confusion. We stress that we did not state or use that Eq. (1) is implied by causal-inference "common cause", and note that our claims and results are not invalidated by this point.

Revision. We will rename the assumption, and state the scope of the hypothesis explicitly.


2. "A reduction in the prediction gap does not necessarily imply a corresponding reduction in CMI." (Weakness 2)

We agree that a smaller prediction gap does not establish the CMI inequality, as stated in Appendices D and G (and we will stress this more in the main text). We did not claim that Eq. (3) turns a reduction in the score gap into the CMI reduction required by Hypothesis 1.

Why score gaps. A direct test would estimate It(A;C∣B,S)I_t(A;C\mid B,S) and It(A;C∣B)I_t(A;C\mid B) from samples of high-dimensional images, which is notoriously hard: without strong assumptions on the distribution, no estimator can reliably measure, or even certify, such quantities from a feasible number of samples (McAllester & Stratos, 2020; Poole et al., 2019; Shah & Peters, 2020). Score gaps are also the more relevant quantity: the locality gap is proportional to the extra denoising error on AA from replacing the global denoiser with the best local one (Appendix A.4, "Bounding Local Denoiser Error with CMI"). A smaller conditional gap therefore means that, once the label is supplied, a local denoiser loses less relative to the global one. In Stable Diffusion 3 Medium (Figure 2 in Section 3.2.3; setup and limitations in Appendix D), we measure the change in prediction when image–image attention is restricted to a 15×1515\times15 token neighborhood, along the same generation trajectories with and without the scene description (64 scenes, three nested description lengths, three seeds each).

Why we do not report direct estimates of α\alpha. Hu et al. (2025) estimate It(A;C∣B)I_t(A;C\mid B) with MINE for a single-pixel AA on MNIST and Fashion-MNIST, without labels, and find it small at both ends of the trajectory with a peak at intermediate times. We attempted the label-conditional extension, estimating 1−It(A;C∣B,S)/It(A;C∣B)1-I_t(A;C\mid B,S)/I_t(A;C\mid B) with the same estimator, but the estimates were not stable enough to bound α\alpha. This is expected. Each CMI is estimated as the difference of two variational lower bounds on mutual information, which is a bound in neither direction, and these lower bounds have high variance (Song & Ermon, 2020). The ratio is moreover ill-conditioned wherever It(A;C∣B)I_t(A;C\mid B) is small, which includes both ends of the trajectory. We therefore test the hypothesis through its operational counterpart, the score gap, and stress that Figure 2 is only consistent with Hypothesis 1, as stated in the caption of Figure 2.

Revision. We will re-emphasize the limitation of Figure 2 in Section 3.2.3 and list a direct estimate of α\alpha on natural images as an open problem, as already implied in the limitations (Appendix G).

AuthorsOpenReview ↗

Rebuttal part 2

3. "Score gap suppression assumption also does not hold." (Weakness 3)

The score-gap suppression inequality appears only in Section 3.1, inside a heuristic argument that motivates the information theoretic version (i.e., Hypothesis 1); no theorem uses it. Its only role is to give the intuition in operational terms, through the score function. We remark that Theorem 1 relies only on Hypothesis 1, and only for the tripartition to which it is applied: the theorem fixes one annulus tripartition and does not assume the hypothesis for all of them.

We also stress that we do not claim nor need the information theoretic assumption (Hypothesis 1) to hold for every distribution and label. Hypothesis 1 is a sufficient condition. It isolates one mechanism, the label accounting for a fixed fraction of the long-range conditional dependence, under which the nonlocality window must lie inside the speciation window. A sufficient condition is informative precisely because it is not universal. It tells us which distributions should show concurrence and which fall outside its scope: labels that are redundant across regions satisfy the hypothesis, while synergistic labels, such as parity, counts or relational labels, can violate it, and there concurrence is not guaranteed (Section 3.2.3).

Revision. We will state in Section 3.1 that the score-gap suppression inequality is a heuristic that fails for synergistic labels.


4. Minor points

(a) L32, "Such features must therefore emerge during inference." This sentence states an obvious logical consequence. Generation starts from noise, which carries no semantic structure. A sampler that reproduces a distribution with semantic structure must output samples that have this structure, so the structure must appear at some point during generation.

(b) Attribution to Kamb & Ganguli. We agree that "generation of a pixel" was ambiguous. We mean the single-step denoiser that updates the pixel during generation, not the distribution of the pixel at the end of generation, and this is the sense in which Kamb & Ganguli (2024) establish locality. Overlapping local denoisers can create long-range correlations over many steps, for example because each pixel's dependence region grows with every step.

(c) Experiments in the appendix. The central experiment supporting Hypothesis 1, Figure 2, is in the main text (Section 3.2.3). The appendices contain implementation details and complementary forward–backward experiments (Appendix B). As the paper is theoretical, we kept the main text focused on the theoretical contribution.

Revision. We will rephrase L32, replace "generation of a pixel" with "the denoiser update of a pixel".

Reviewer_u5uhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In the present work, the authors consider generative diffusion models and build a link between the so-called speciation phenomenon, a phase of the reverse diffusion where the trajectory commits to a data cluster with a defined label, and the emergence of non-local dependencies in the denoising score that drives the reverse process. Their construction is based on a reasonable but not general assumption, the "common cause" hypothesis, which requires the semantic label to explain away part of the dependence between a target region and distant regions of the image, conditioned on a buffer region around the target. In this scenario, the authors prove that, if it exists, the non-locality window is contained within the speciation window. Under additional assumptions on class separation and information thresholds, they show that both windows concentrate at the pure-noise endpoint as input size grows, using a time parametrization on the interval [0,1]. They illustrate these results through explicit calculations for a two-component Gaussian mixture.

Critical strengths
  • The paper addresses a relevant question: under what conditions can semantic speciation explain the emergence of non-locality along reverse trajectories in diffusion models? In this direction, it provides an explicit sufficient condition and an associated precise containment result, which appears convincing (I have not checked the details of the proof).

  • The authors provide a concrete interpretation of the instantaneous score gap, linking it to the excess mean squared error incurred by restricting optimal denoising to local observations.

  • The Gaussian-mixture example allows explicit calculation and comparison of the information quantities underlying the two windows (speciation and non-locality).

Critical weaknesses
  • The common-cause hypothesis seems to start from the assumption that semantic labels explain a fixed fraction of distant conditional dependence. The containment theorem formalizes an intuitive consequence of this assumption. But it remains unclear what additional insight it provides beyond the concurrence (already observed in previous work). It assumes a nonempty nonlocality window and does not fully establish matching peaks or comparable widths.

  • The common-cause inequality is required to hold throughout the noise trajectory, with a size-independent α\alpha for the scaling result. Its applicability to natural datasets, however, is not convincingly established (e.g., relational labels, such as whether distant objects have the same shape, can violate it).

  • The paper does not establish whether the bound (3) is tight or quantitatively informative. It guarantees that small CMI implies a small instantaneous score gap, but e.g. it does not establish that large CMI implies denoising failure, or that the two quantities identify the same transition window.

  • In the Gaussian mixture, conditioning on the label removes all spatial dependence, so the common-cause hypothesis holds with maximal α=1\alpha=1. This is a useful consistency check but provides limited evidence for settings with residual within-class correlations.

  • Concentration at t=1t=1 seems to partly reflect the compact time parametrization: the familiar logarithmic speciation time maps to this endpoint. Moreover, the Gaussian example’s peak CMI vanishes as N−2N^{-2}, and the nonlocality window survives only because the information tolerance δ\delta is also set to decrease with size.

  • In general, the presentation is not always clear. Many definitions come after a concept has been already mentioned, the notation for the partition A,B,C is not always clear, the figures are not well described and integrated in the discussion. The final sections 3.2.4–3.3 would benefit from clearer intuition and a sharper separation between assumptions, proven conclusions, and interpretation.

Questions for authors
  • What prediction or interpretation follows from the containment theorem that was not already suggested by previous empirical observations? Can the authors demonstrate a case where the theorem provides additional explanatory or practical value?

  • Can the authors provide a quantitative test of the common-cause inequality? How does Figure 2 constrain the required CMI reduction?

  • How tight is Eq. (3), even in the solvable Gaussian example?

  • Can the authors extend the Gaussian example to include residual within-class spatial correlations and a nontrivial 0<α<10<\alpha<1?

  • Can the authors express the scaling results in log-SNR or an appropriately rescaled time coordinate? Would the windows narrow? What remains of the nonlocality window at a fixed information tolerance?

  • Why is the speciation start defined using I0(S;AB)−It(S;B)I_0(S;AB)-I_t(S;B), rather than comparing the same observation region at both times? Doesn't this imply a nonzero value at t=0t=0?

  • Please specify the statistical meaning and choice of “true semantic labels,” clarify the partition geometry and buffer radius. Showing the threshold-defined windows and multiple system sizes in the Gaussian plots would also help.

4 comments
AuthorsOpenReview ↗

Rebuttal part 1

Thank you for the detailed review, and for recognizing the relevance of the question, the precise containment result, and the excess-error interpretation of the score gap. We address your concerns in seven points below. Several points overlap with questions from Reviewer RrDo, and we refer to that response for details.


1. "What prediction or interpretation follows from the containment theorem that was not already suggested by previous empirical observations?" (Weakness 1, Q1)

Previous work observed that the two windows roughly coincide in specific models, but did not explain why, or when this should be expected. Hypothesis 1 gives a sufficient condition for containment that does not depend on the architecture or the noising process. Its value lies in delineating scope: when the hypothesis holds, containment is guaranteed; when it fails, there is no guarantee, which identifies where non-containment can arise. As with any mechanism-level explanation, the natural follow-up question is which data distributions realize it, and with what α\alpha.

This is a familiar pattern in science: a phenomenon seen across many systems is explained by isolating a simple assumption that suffices to produce it, and the question becomes when that assumption is realized. A sufficient condition is informative precisely because it is not universal. It tells us which distributions should show containment and which fall outside its scope: labels that are redundant across regions satisfy the hypothesis, while synergistic labels, such as parity, counts or relational labels, can violate it, and there containment is not guaranteed.

Revision. We will state the scope of the hypothesis explicitly and separate the assumption, the proven conclusion and its interpretation.


2. "The common-cause inequality is required to hold throughout the noise trajectory, with a size-independent α\alpha for the scaling result. Its applicability to natural datasets, however, is not convincingly established (e.g., relational labels, such as whether distant objects have the same shape, can violate it)." (Weakness 2, Q2)

Applicability to natural datasets, e.g., presence of relational labels. Datasets indeed and admit relational/synergistic labels, which can violate Hypothesis 1. We discuss this class (parity, counts) in Section 3.2.3, and it is exactly the class outside the scope of Theorem 1. We view this as a feature of a sufficient condition rather than a defect: it tells us both when to expect concurrence and which kinds of labels fall outside the guarantee. To establish concurrence of a locality transition with a label that undergoes speciation, we need that specific label to satisfy common cause; we do not need Hypothesis 1 to hold for all possible labels. Figure 2 is evidence that such labels exist at the level of the score functions themselves.

The inequality is not needed throughout the trajectory. The proof of Theorem 1 uses Hypothesis 1 only at times when It(A;C∣B)>δI_t(A;C\mid B)>\delta, and there it only needs the label to explain an amount αδ\alpha\delta of the dependence:

\quad\text{whenever}\quad I_t(A;C\mid B)>\delta .$$ Theorem 1 holds under this weaker condition, which constrains the label only inside the nonlocality window, where long-range dependence is substantial. We will emphasize this in our revision of the paper. *How the hypothesis can be tested, and what Figure 2 shows.* A direct test would estimate $I_t(A;C\mid B,S)$ and $I_t(A;C\mid B)$ from samples of high-dimensional images, which is notoriously hard: without strong assumptions on the distribution, no estimator can reliably measure, or even certify, such quantities from a feasible number of samples ([McAllester & Stratos, 2020](https://proceedings.mlr.press/v108/mcallester20a.html); [Poole et al., 2019](https://arxiv.org/abs/1905.06922); [Shah & Peters, 2020](https://arxiv.org/abs/1804.07203)). We therefore use score gaps, which are also the more relevant quantity (point 3). In Stable Diffusion 3 Medium (Figure 2 in Section 3.2.3; setup and limitations in Appendix D), we measure the change in prediction when image–image attention is restricted to a $15\times15$ token neighborhood, along the same generation trajectories with and without the scene description (64 scenes, three nested description lengths, three seeds each). Conditioning reduces the gap at high noise, where the gap is large, and leaves it essentially unchanged at low noise, where it is small, so the reduction appears where the condition above is needed. Figure 2 does not, however, bound the required CMI reduction quantitatively: Eq. (3) bounds the gap by the CMI and not the reverse, so a smaller conditional gap is consistent with Hypothesis 1 rather than a measurement of $\alpha$. *Revision.* We will state the form of Theorem 1 with the weaker assumption, which strengthens the result.
AuthorsOpenReview ↗

Rebuttal part 2

3. "How tight is Eq. (3), even in the solvable Gaussian example?" (Weakness 3, Q3)

We agree that Eq. (3) is one-sided, but we do not rely on tightness or the reverse direction of the inequality. The bound is not meant to be tight; its role is to guarantee that a vanishing CMI makes local denoising accurate as well as justify why large score gap leads to large CMI, validating previous work's usage of score gap as a probe of phase transitions. It is useful in two cases: when CMI is small, it implies a small instantaneous score gap (away from transition); when score gap is large, it yields a large CMI (large score gap can detect transition). We also agree that the bound alone does not identify the two transition windows, but we never used the bound to make such a claim.

The score gap is the operationally relevant quantity in its own right: as you note, it equals, up to the factor t4/(1−t)2t^4/(1-t)^2, the excess mean-squared error of the optimal local denoiser over the global one (Appendix A.4). This is why our Stable Diffusion experiment measures the score gap rather than the CMI.

For Gaussian-mixture example, Eq. (3) is indeed not quantitatively tight. Using the Appendix E.3 parameters, at t≃0.877t\simeq0.877, exact Gaussian-mixture formulas evaluated numerically give It(A;C∣B)≃0.041994I_t(A;C\mid B)\simeq0.041994 and Δloc2≃0.096666\Delta_{\rm loc}^2\simeq0.096666. The right-hand side of Eq. (3), 4dA(dA+3)It(A;C∣B)/t4≃864d_A(d_A+3) I_t(A;C\mid B)/t^4\simeq86, thus exceeds Δloc4≃9.3×10−3\Delta_{\rm loc}^4\simeq9.3\times10^{-3} by about four orders of magnitude. The bound supplies a sufficient condition for small score error. We remark that this bound is never used to imply an equivalence of transition windows.

Revision. We will remark that Eq. 3 is not tight and stress that it is useful when the score gap is large or when CMI is small.


4. "Can the authors extend the Gaussian example to include residual within-class spatial correlations and a nontrivial 0<α<10<\alpha<1?" (Weakness 4, Q4)

The simplest extension is to introduce a second source of long-range dependence that SS does not determine. Let U=±1U=\pm1 be a second fair binary variable, independent of SS, and give each pixel two channels, X0=(X0(1),X0(2))∈RN2×RN2X_0=(X_0^{(1)},X_0^{(2)})\in\mathbb R^{N^2}\times\mathbb R^{N^2}, with

\qquad X_0^{(2)}\mid U=u\ \sim\ \mathcal N(u\mu\vec 1,\ \sigma^2 I_{N^2}),$$ independent given $(S,U)$, and $X_t=(1-t)X_0+tZ$ with $Z\sim\mathcal N(0,I_{2N^2})$. The first channel is the Gaussian mixture of Appendix E with label $S$. The second is an identical copy whose component $U$ is not part of the label. Each region contains both channels of its pixels, $X_R=(X_R^{(1)},X_R^{(2)})$. Given $S$, the second channel is still a two-component mixture, so any two of its pixels have covariance $(1-t)^2\mu^2$. The within-class correlations are therefore long-range. *This construction gives $\alpha=1/2$.* The pair $(S,X^{(1)})$ is independent of $(U,X^{(2)})$, so every density factorizes over the two channels, and each CMI splits into one term per channel: $$I_t(A;C\mid B)=I_t(A^{(1)};C^{(1)}\mid B^{(1)})+I_t(A^{(2)};C^{(2)}\mid B^{(2)}),$$ $$I_t(A;C\mid B,S)=\underbrace{I_t(A^{(1)};C^{(1)}\mid B^{(1)},S)}_{= 0}+I_t(A^{(2)};C^{(2)}\mid B^{(2)}).$$ In the second line, the first term vanishes because, given $S$, the first channel factorizes over pixels (Appendix E). The second term is the same as in the first line, because $S$ carries no information about the second channel. The two channels have the same distribution, so the two terms in the first line are equal. Hence $$I_t(A;C\mid B,S)=\tfrac12 I_t(A;C\mid B)$$ at every $t$ and system size, and Hypothesis 1 holds with $\alpha=1/2$ exactly. *General $\alpha$.* With $k+1$ such channels, each with its own binary component, and a label consisting of $j$ of these components, the same argument gives $\alpha=j/(k+1)$. Thus $\alpha$ grows with the informativeness of the label, as anticipated in Section 3.2.3. *Revision.* We will add this example to Appendix E.
AuthorsOpenReview ↗

Rebuttal part 3

5. "Can the authors express the scaling results in log-SNR or an appropriately rescaled time coordinate? Would the windows narrow? What remains of the nonlocality window at a fixed information tolerance?" (Weakness 5, Q5)

We agree that the closure of the windows is related to the compact time parametrization, while noting that this parametrization is common in the literature we build on (e.g. Hu et al., 2025; Zhang et al., 2026). Let us use the other time parametrization used by Biroli et al. for the Ornstein–Uhlenbeck (OU) noising process and stick to the unit-mean, unit-variance Gaussian mixture in Appendix E.3. Our interpolation maps to OU time through

\qquad \widehat{\tau}(t)=\frac{\tau(t)}{\log N},$$ which matches the speciation time they find, $\tau_S=\log N$. With patch size $|A|$, buffer size $|B|=\beta N^2$ with fixed $0<\beta<1$, nonlocality threshold $\delta_N=\kappa/N^2$, and speciation threshold $\alpha\delta_N$, the speciation endpoints (in our definitions) in [Biroli et al.](https://arxiv.org/pdf/2402.18491)’s rescaled time are $$\begin{aligned} \widehat{\tau}\left(t_{\mathrm{spec}}^{\mathrm{start},\alpha\delta_N}\right) &=1-\frac{\log\log N+\log(4/\beta)}{2\log N} +o\left(\frac{1}{\log N}\right), \widehat{\tau}\left(t_{\mathrm{spec}}^{\mathrm{end},\alpha\delta_N}\right) &=2-\frac{\log(2\alpha\kappa)}{2\log N} +o\left(\frac{1}{\log N}\right). \end{aligned}$$ Its rescaled width thus tends to 1. For nonlocality, let $x_-<x_+$ be the outer crossings of the limiting rescaled CMI profile at level $\kappa$: $$|A|x_\pm\left[F'(\sqrt{x_\pm})-F'(\sqrt{\beta x_\pm})\right]=\kappa.$$ Here $F$ is the integral in Eq. (63), and primes denote differentiation with respect to the squared argument, as in Eq. (70). The crossings depend on $|A|$, $\beta$, and $\kappa$, but not on $N$. The nonlocality endpoints are $$\begin{aligned} \widehat{\tau}\left(t_{\mathrm{nonloc}}^{\mathrm{start},\delta_N}\right) &=1-\frac{\log x_+}{2\log N} +o\left(\frac{1}{\log N}\right), \widehat{\tau}\left(t_{\mathrm{nonloc}}^{\mathrm{end},\delta_N}\right) &=1-\frac{\log x_-}{2\log N} +o\left(\frac{1}{\log N}\right). \end{aligned}$$ Its rescaled width is vanishing, since \$\frac{\log(x_+/x_-)}{2\log N} +o\left(\frac{1}{\log N}\right) \longrightarrow0.\$ In the rescaled time of Biroli et al., the nonlocality window therefore sharpens onto $\widehat{\tau}=1$, their speciation time, which is also the limit of the speciation start. Only the speciation end lags, at $\widehat{\tau}\to2$, because the vanishing threshold $\alpha\delta_N\propto N^{-2}$ probes the far tail of $I_t(S;ABC)$. Our claims therefore stand: containment compares sets of times and holds in any time coordinate, and the nonlocality window concentrates at the speciation time in the coordinate in which Biroli et al. find speciation to sharpen. As for a fixed tolerance: for a fixed-size patch the peak CMI scales as $N^{-2}$, so nothing remains at large $N$. If instead $A$ and $B$ are fixed fractions of the image, every information quantity depends on $t$ only through $N(1-t)$. A fixed $\delta$ then suffices, and both windows close onto $\widehat{\tau}=1$, with widths of order $1/\log N$. *Revision.* We will add the rescaled-time scalings above to Section 3.3, with their derivation in Appendix E, and plot both windows in $\widehat{\tau}$ for several system sizes. We will state that the convergence of both windows to $t=1$ in Section 3.3 and Theorem 2 refers to compact time, and add the fixed-fraction case at a fixed tolerance.
AuthorsOpenReview ↗

Rebuttal part 4

6. "Why is the speciation start defined using I0(S;AB)−It(S;B)I_0(S;AB)-I_t(S;B), rather than comparing the same observation region at both times? Doesn't this imply a nonzero value at t=0t=0?" (Q6)

Yes. This question was also raised by Reviewer RrDo; our response there gives the full details. The start quantity decomposes as

=I_0(S;A\mid B) +\big[I_0(S;B)-I_t(S;B)\big],$$ where $\big[I_0(S;B)-I_t(S;B)\big]$ is the loss from noising in region B and $I_0(S;A\mid B)$ is an offset at $t=0$ The definition uses $AB$ at $t=0$ because the proof bounds $I_t(S;AB)$ by its clean value. Comparing the same region at both times is equally possible: Theorem 1 then holds with the start threshold lowered from $\alpha\delta$ to $\alpha\delta-I_0(S;A\mid B)$. The offset must therefore be small compared with $\alpha\delta$. It is not small for every distribution and we can easily construct adversarial examples: if the label is a fair bit stored entirely in $A$, with $B$ and $C$ independent noise, the offset is $\log 2$, although there is no long-range dependence. However, we have good reason to believe that in real images, this offset is small. For a small patch of a clean image, the exact decomposition $$I_0(S;A\mid B) =I_0(S;A\mid BC)+I_0(A;C\mid B)-I_0(A;C\mid B,S) \le I_0(S;A\mid BC)+I_0(A;C\mid B)$$ shows that the offset is small when the patch adds little label information once the rest of the image is observed, and the clean data are approximately local across the buffer. In a new experiment on 1,000 ImageNet validation images, filling a $4\times4$, $8\times8$ or $16\times16$ hole by harmonic interpolation from its boundary changes the posterior of a frozen ConvNeXt-V2 Large classifier by $5.9\times10^{-6}$, $7.7\times10^{-5}$ and $5.7\times10^{-4}$ nats on average. This establishes an estimated upper bound on $I_0(S;A\mid BC)$ to the extent that the classifier approximates the true posterior. If instead the clean data are already nonlocal, $I_0(A;C\mid B)>\delta$, Hypothesis 1 gives $I_0(S;A\mid B)\ge\alpha I_0(A;C\mid B)>\alpha\delta$, so both windows start at $t=0$ and containment still holds. *Revision.* We will add the decomposition and the experiment, and state that the offset shifts the start threshold and must be small relative to $\alpha\delta$. --- **7. Presentation, labels and geometry (Weakness 6, Q7)** - *Statistical meaning of "true semantic labels."* $S$ is a random variable defined jointly with the clean image, $p(x_0,s)$. "True" means the label attached to each sample by the data distribution (the ImageNet class, or the mixture component in the Gaussian example), as opposed to a classifier's prediction; all posteriors are taken under this joint law. - *Partition geometry and buffer radius.* $A$ is a pixel or small patch, $B$ the square annulus of radius $r$ around it, and $C$ the rest (Section 2.1). In the identity-covariance Gaussian mixture only the region sizes matter; the numerics use $|A|=16$, $|B|=80$ and $|C|=304$ pixels of a $20\times20$ image. We also sketch the tripartition in Figure 1 on which the figure is shown. - *Gaussian plots.* Showing the threshold-defined windows and several system sizes is not hard to do. We will include those in future revisions.
Reviewer_RrDoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the behavior of sampling for diffusion models. They consider two time windows: (i) a speciation window which is the time during sampling when a model commits itself to a particular semantic class and (ii) a nonlocality window, which is a window where the model relies on non-local information. Their main assumption, which is stated in Hypothesis 1, postulates that a semantic label SS explains at least an α\alpha fraction of the information between a local and a non-local path. More precisely if AA is a patch, BB are the local neighbors around AA and CC is everything else then they postulate It(A;C∣B,S)<=(1−α)It(A;C∣B)I_{t}(A ; C | B, S) <= (1- \alpha) I_{t}(A; C | B) which is saying that adding a semantic label has explained away an α\alpha fraction of the information that CC contains about AA given BB (i.e., reducing reliance of AA on nonlocal information).

Their main theorem is that under Hypothesis 1

It(A;C∣B)≤min⁡(I0(S;AB)−It(S;B),It(S;ABC))I_{t}(A; C | B) \leq \min(I_0(S; AB) - I_t(S;B), I_{t}(S; ABC))

from this inequality their theorem follows immediately. Let

J(t)=It(A;C∣B)J(t) = I_{t}(A;C | B) G(t)=I0(S;AB)−It(S;B)G(t) =I_0(S; AB) - I_t(S;B) L(t)=It(S;ABC)L(t) = I_{t}(S; ABC)

So αJ(t)≤min⁡(G(t),L(t))\alpha J(t) \leq \min(G(t), L(t))

Let tstartnlt_{\text{start}}^{nl} and tendnlt_{\text{end}}^{nl} be the first and last times J(t)J(t) (i.e., the mutual information between AA and CC) exceeds a prescribed threshold δ\delta. Then if J(t)>δJ(t) > \delta then it follows for $\min(G(t), L(t)) > \alpha \delta $ with J(t)=min⁡(G(t),L(t))J(t) = \min(G(t), L(t)) the speciation window is defined as the first and last times that J(t)J(t) exceeds a prescribed threshold δα\delta \alpha, call them tstartspect_{start}^{spec} and tendspect_{end}^{spec}. Under these assumptions then $[t_{start}^{spec}, t_{end}^{spec}] \subseteq [t_{\text{start}}^{spec}, t_{\text{end}}^{spec}] $. Their definitions on the starting and endtimes of the speciation windows use G(t)G(t) and L(t)L(t) in lieu of the minima but under the minima I described above this gives essentially the same window.

The proof follows from monotonicity of MI plus the MI chain rule and the main argument is given below

Assume It(A;C∣B,S)≤(1−α)It(A;C∣B)I_{t}(A ; C | B, S) \leq (1- \alpha) I_{t}(A; C | B)

Rearrange αIt(A;C∣B)≤It(A;C∣B)−It(A;C∣B,S)\alpha I_{t}(A ; C | B) \leq I_{t}(A; C | B) - I_{t}(A; C |B, S)

Use the identity It(A;C∣B)−It(A;C∣B,S)=It(A;S∣B)−It(A;S∣C,B)I_{t}(A; C | B) - I_{t}(A; C |B, S) = I_{t}(A; S | B) - I_{t}(A; S |C, B)

Bound the RHS above twice. First use monotonicity of MI and chain rule to note It(A;S∣B)=It(S;AB)−It(S;B)<I0(S;AB)−It(S;B)I_{t}(A; S| B) = I_{t}(S; AB) - I_{t}(S; B) < I_{0}(S; AB) - I_{t}(S; B)

And a second time with $I_{t}(A; S | B) = I_{t}(S ; AB) - I_{t}(A ;S) \leq I_{t}(S; AB) \leq I_{t}(S ; ABC) $.

From here the theorem is proved.

Critical strengths

The paper does a good job of motivating an interesting question about the inference time behavior of diffusion models and I think strikes a good balance of empirical motivation and theoretical discussion. I think the examples used to motivate the theory are also well explained and easy to understand.

Critical weaknesses

My main problem with the paper is that I think the explanation and interpretation are doing a lot more work than the mathematics. Moreover, while the question itself is interesting, I don't think the experiments or the theory do a satisfactory job of explaining the phenomenon that the authors describe (though, to be fair, the authors explicitly acknowledge some of these limitations in their Stable Diffusion experiments).

I frequently found myself having a difficult time relating the mathematical definitions to the authors' discussion. In particular, it almost feels like the definitions of the speciation and nonlocality windows were chosen so that the speciation window would contain the nonlocality window. The start and end of the speciation window are defined using precisely the quantities that arise as upper bounds in the proof, and I'm not convinced that these definitions capture the phenomenon observed in forward-backward experiments.

I think much of the discussion helps illustrate the intuition, but the authors ultimately rely on that intuition a little too heavily, since I don't think the experiments sufficiently support the paper's main thesis. I also think the main mathematical argument is relatively straightforward once the common-cause hypothesis is assumed (see above). I found it surprisingly difficult to follow this relatively simple, but nontrivial, argument with the lengthy discussions of seemingly related concepts that the authors end up not using (i.e., the posterior label agreement).

I think there is an interesting question and a good idea here, but I don't think the paper is ready for publication based on the evidence presented and the exposition. I therefore recommend rejection.

Questions for authors

I don't think I am sold on hypothesis 1, why should conditioning on a semantic label explain a uniformly positive fraction of the long-range dependence in realistic image distributions? What experiment could the authors run to show this and how does this experiment give evidence of hypothesis 1?

If we decompose I0(S;AB)−It(S;B)I_{0}(S ; AB) - I_{t}(S; B)

The chain-rule identity used in the proof is exact It(A;C∣B)−It(A;C∣B,S)=It(A;S∣B)−It(A;S∣B,C)I_{t}(A; C | B) - I_{t}(A; C | B, S) = I_{t}(A; S | B) - I_{t}(A; S | B, C), there's nothing mathematically question about it. But then the authors bound this by

It(A;C∣B)≤min⁡(I0(S;AB)−It(S;B),It(S;ABC))I_{t}(A; C | B) \leq \min(I_0(S; AB) - I_t(S;B), I_{t}(S; ABC))

and they use precisely these two upper bounds to define the endpoints of their speciation window.

Are these quantities independently justified measures of the onset and end of semantic speciation, or are they chosen primarily because they make the containment theorem follow?

Especially the first quantity I0(S;AB)−It(S;B)I_0(S; AB) - I_t(S;B). It combines additional information from observing AA with information lost by noising BB. It isn't simply the loss of semantic information due to noise.

I'm not sure to what extent Theorem 1 explains the observed overlap between speciation and nonlocality. The speciation window is defined using the same quantities that appear in the upper bound on nonlocality, so the containment seems to follow almost by construction once the common-cause assumption is made. How do we know that this definition captures the speciation window observed in forward–backward experiments?

3 comments
AuthorsOpenReview ↗

Rebuttal part 1

Thank you for the careful review, and for finding the question and its empirical motivation interesting. We agree that several connections were not made explicit enough. We address your concerns in four points.


1. "How do we know that this definition captures the speciation window observed in forward–backward experiments?" and "containment seems to follow almost by construction once the common-cause assumption is made"

Lemma 1 provides this link, independently of Hypothesis 1 and Theorem 1; we should have stated its role explicitly. In an ideal FB experiment, the original and reconstructed labels are independent draws from the same posterior given the noisy observation, so

PFB(R,t)=E∑spt(s∣XR,t)2.P_{\mathrm{FB}}(R,t)=\mathbb E\sum_s p_t(s\mid X_{R,t})^2 .

Since the average posterior equals the prior, both parts of Lemma 1 can be written in terms of this observable (the minimum is over labels with positive prior probability) as

\le I_t(S;R) \le \frac{P_{\mathrm{FB}}(R,t)-\sum_s p(s)^2}{\min_s p(s)},$$ which is a simple rewrite of Eq. (9) in the main text, and $$-\log P_{\mathrm{FB}}(R,t) \le H(S)-I_t(S;R) \le h_2\big(1-P_{\mathrm{FB}}(R,t)\big)+\big(1-P_{\mathrm{FB}}(R,t)\big)\log(L-1).$$ So FB agreement near the prior-sampling baseline $\sum_s p(s)^2$ (equal to $1/L$ for $L$ balanced classes) is equivalent to nearly zero label information, and near-perfect agreement is equivalent to nearly complete label information. Applied to $R=B$ and $R=ABC$, this motivates a window running from the loss of reliable local recovery to the loss of global recovery. Because the containment proof only uses information identities, Lemma 1 appeared unused. Combined with that proof, however, it gives containment directly in FB terms. The proof yields $\alpha I_t(A;C\mid B)\le I_t(S;A\mid B)$, and $I_t(S;A\mid B)$ is at most both $H(S\mid X_{B,t})$ and $I_t(S;ABC)$. Lemma 1 then gives, at every time with $I_t(A;C\mid B)>\delta$, $$h_2\big(1-P_{\mathrm{FB}}(B,t)\big)+\big(1-P_{\mathrm{FB}}(B,t)\big)\log(L-1)>\alpha\delta \quad\text{and}\quad P_{\mathrm{FB}}(ABC,t)-\sum_s p(s)^2>\alpha\delta\min_s p(s).$$ The first inequality requires the local FB failure rate $1-P_{\mathrm{FB}}(B,t)$ to exceed $\alpha\delta$ up to a logarithmic factor. Under Hypothesis 1, nonlocality therefore occurs only while FB reconstruction from the local region fails and FB reconstruction from the whole image is still above chance. This statement involves only FB success rates, not our definitions of the speciation endpoints. Its local part is informative when the clean local region identifies the label, as in our Gaussian mixture model and Theorem 2. *Revision.* We will state Lemma 1 in this form, explain its role, and add the FB form of Theorem 1 as a corollary. We will also replace the phrase "the same window" with a precise statement: the bounds relate the endpoint regimes and convert between tolerances, but they do not give identical crossing times at finite thresholds, and they do not identify our empirical classifier-logit cosine statistic with exact FB agreement.
AuthorsOpenReview ↗

Rebuttal part 2

2. "The start quantity combines additional information from observing AA with information lost by noising BB."

This is correct:

=I_0(S;A\mid B) +\big[I_0(S;B)-I_t(S;B)\big],$$ where $\big[I_0(S;B)-I_t(S;B)\big]$ is the loss from noising in region B and $I_0(S;A\mid B)$ is an offset at $t=0$ The offset shifts the threshold of the start definition by exactly $I_0(S;A\mid B)$. It is not small for every distribution: if the label is a fair bit stored entirely in $A$, with $B$ and $C$ independent noise, the offset is $\log 2$, although there is no long-range dependence. A small offset is therefore a property of the data and the tripartition, which must be justified. *Why it is small for a small patch of a clean image.* The exact decomposition $$I_0(S;A\mid B) =I_0(S;A\mid BC)+I_0(A;C\mid B)-I_0(A;C\mid B,S) \le I_0(S;A\mid BC)+I_0(A;C\mid B)$$ does not use Hypothesis 1. The first term is the label information the patch adds once the rest of the image is observed; the second is clean-data nonlocality across the buffer. The offset is small when both are. *Evidence that $I_0(S;A\mid BC)$ is small.* Removing much more than a small patch barely affects recognition. With random holes covering 20% of Places2 images filled by DF-Net, top-5 accuracy is 82.1% (vs. 82.1% clean) for ResNet-18 and 84.3% (vs. 84.8%) for DenseNet-161 ([Black et al., WACV 2020](https://openaccess.thecvf.com/content_WACV_2020/papers/Black_Evaluation_of_Image_Inpainting_for_Classification_and_Retrieval_WACV_2020_paper.pdf), Table 2). Accuracy does not measure information, so we ran a new experiment on the full classifier posterior. *New experiment.* We use 1,000 ImageNet validation images (one random image per class, no filtering by correctness) at $256\times256$, with three random locations per image and square holes of side 4, 8 and 16 pixels. Each hole is filled by harmonic interpolation from its observed boundary only, and a frozen ConvNeXt-V2 Large classifier (`convnextv2_large.fcmae_ft_in22k_in1k_384`) evaluates the clean and filled full images, so the observed context is $BC$. $I_0(S;A\mid BC)$ is the expected KL divergence between the label posteriors with and without $A$, $$I_0(S;A\mid BC)=\mathbb E\ D_{\mathrm{KL}}\big(p_0(S\mid ABC)\ \Vert\ p_0(S\mid BC)\big).$$ We measure its classifier counterpart $\mathbb E\ D_{\mathrm{KL}}\big(r(S\mid ABC)\ \Vert\ r(S\mid BC)\big)$, where $r(S\mid ABC)$ is the classifier's posterior on the clean image and $r(S\mid BC)$ its posterior after interpolation. This approximates $I_0(S;A\mid BC)$ to the extent that the classifier, applied to the clean and filled images, approximates the true posteriors given $ABC$ and $BC$. | Hole | Mean posterior KL (nats) | 95% bootstrap interval | Top-1 accuracy after filling | | --- | ---: | ---: | ---: | | $4\times4$ | $5.9\times10^{-6}$ | $[4.4, 7.6]\times10^{-6}$ | 87.20% | | $8\times8$ | $7.7\times10^{-5}$ | $[6.1, 9.4]\times10^{-5}$ | 87.20% | | $16\times16$ | $5.7\times10^{-4}$ | $[4.4, 7.3]\times10^{-4}$ | 87.17% | Clean top-1 accuracy is 87.20%. Intervals use 5,000 image-level bootstrap resamples (keeping each image's three locations together) and do not include classifier error. On average, a small patch adds almost no label information; this does not mean every patch is uninformative, and it does not by itself establish clean-data locality. Together with small $I_0(A;C\mid B)$, it justifies a small offset. *If the clean data are already nonlocal.* If $I_0(A;C\mid B)>\delta$, Hypothesis 1 gives $$I_0(S;A\mid B)\ge I_0(A;C\mid B)-I_0(A;C\mid B,S)\ge\alpha I_0(A;C\mid B)>\alpha\delta,$$ so both windows start at $t=0$ and containment still holds. The theorem does not require an initially local regime. *Revision.* We will add the decomposition and the experiment, and state that the offset shifts the start threshold and must be small relative to $\alpha\delta$, rather than identify the start quantity with noise-induced loss unconditionally.
AuthorsOpenReview ↗

Rebuttal part 3

3. "Why should conditioning on a semantic label explain a uniformly positive fraction of the long-range dependence?"

We do not claim the hypothesis holds for all labels. Hypothesis 1 is a sufficient condition: it isolates one mechanism---the label accouting for fixed fraction of the long-range conditional dependence---under which the nonlocality window must lie inside the speciation window. The contribution is identifying this mechanism, not the difficulty of the derivation.

This is a familiar pattern in science: a phenomenon seen across many systems is explained by isolating a simple assumption that suffices to produce it, and the question becomes when that assumption is realized. A sufficient condition is informative precisely because it is not universal. It tells us which distributions should show concurrence and which fall outside its scope: labels that are redundant across regions satisfy the hypothesis, while synergistic labels, such as parity, counts or relational labels, can violate it, and there concurrence is not guaranteed.

A uniform fraction is not needed. The proof uses Hypothesis 1 only at times when It(A;C∣B)>δI_t(A;C\mid B)>\delta, and there it only needs the label to explain an amount αδ\alpha\delta of the dependence:

\quad\text{whenever}\quad I_t(A;C\mid B)>\delta .$$ Theorem 1 holds under this weaker condition, which constrains the label only inside the nonlocality window, where long-range dependence is substantial. *How the hypothesis can be tested.* A direct test would estimate $I_t(A;C\mid B,S)$ and $I_t(A;C\mid B)$ from samples of high-dimensional images, which is notoriously hard: without strong assumptions on the distribution, no estimator can reliably measure, or even certify, such quantities from a feasible number of samples ([McAllester & Stratos, 2020](https://proceedings.mlr.press/v108/mcallester20a.html); [Poole et al., 2019](https://arxiv.org/abs/1905.06922); [Shah & Peters, 2020](https://arxiv.org/abs/1804.07203)). We therefore use score gaps, which are also the more relevant quantity: the locality gap is proportional to the extra denoising error on $A$ from replacing the global denoiser with the best local one (Appendix A.4), whereas Eq. (3) relates it to the CMI only through a bound that is not tight. In Stable Diffusion 3 Medium (Figure 2 in Section 3.2.3; setup and limitations in Appendix D), we measure the change in prediction when image–image attention is restricted to a $15\times15$ token neighborhood, along the same generation trajectories with and without the scene description (64 scenes, three nested description lengths, three seeds each). Conditioning reduces the gap at high noise, where the gap is large, and leaves it essentially unchanged at low noise, where it is small, so the reduction appears where the condition above is needed. Since Eq. (3) bounds the gap by the CMI and not the reverse, this is consistent with Hypothesis 1 rather than a proof of it. *Revision.* We will state the weaker form of the assumption in Theorem 1, state the scope of the hypothesis explicitly, and separate the assumption, the proven conclusion and its interpretation. --- **4. What the observed concurrence suggests about natural images** Concurrence of the two windows is observed empirically across frontier models. Since the common-cause hypothesis suffices for concurrence, this observation is consistent with much of the semantic structure of natural images being redundant across space, with each region carrying partial, overlapping evidence about the label, rather than relational, with the label fixed only by distant regions jointly. This is not a proof, because the hypothesis is not necessary. It does, however, give a clear picture of how semantic information is arranged in real images and turns that picture into a property that can be tested. We are currently working on strengthening this connection to find necessary conditions for concurrence, as well as conditions for the existence of the locality and semantic speciation windows themselves.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.