- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether semantic properties of images generated by deterministic diffusion samplers can be directly predicted from the initial noise and intermediate states. The authors conduct three sets of experiments to demonstrate that deterministic sampling leads to greater semantic predictability and to examine how this predictability relates to that of the final clean images.
Critical strengths
The paper is generally well-written and clearly organized. The experimental design is also fairly comprehensive.
Critical weaknesses
My main concern is that I do not feel I have gained many new insights from this paper. While the experiments are comprehensive, most of the observations and conclusions seem relatively intuitive to me.
For a deterministic sampler, we can view the generation process as learning a transport map from the noise distribution to the data distribution. We might reasonably expect the learned transport to exhibit some degree of geometric regularity. Therefore, if a semantic property exhibits a relatively simple structure in the data space (e.g., clustering or some degree of linear separability), it seems natural that its preimage in the noise space could also exhibit some degree of predictability. From this perspective, I do not find the observation that semantic properties can be predicted from the initial noise particularly surprising. What I would find more interesting is understanding what properties of the learned transport make such predictability possible, and how the geometry of the transport affects the accessibility of semantic information. However, the paper does not seem to investigate these questions in much depth.
Relatedly, the relationship between semantic predictability and the confidence of the final clean image, examined in the second set of experiments, also seems relatively intuitive. Low-confidence samples are often closer to the decision boundaries of the classifier, and their preimages under a deterministic transport may likewise lie near more ambiguous regions in the noise space. Therefore, the observation that such samples are harder to predict from the initial noise does not seem particularly unexpected. Similarly, the near-chance prediction accuracy under DDPM is understandable given the additional randomness introduced during stochastic sampling. More generally, stochastic and deterministic diffusion processes can share the same time-dependent marginal distributions while inducing different couplings between the initial noise and the final samples. This distinction already provides some intuition for why deterministic sampling may allow stronger seed-level predictability.
Overall, I feel that the paper mainly provides empirical confirmation of phenomena that are already broadly consistent with our understanding of diffusion models, rather than offering substantially new insights. I would be more interested in understanding what properties characterize the ODE transport induced by diffusion models, how these properties relate to semantic predictability in the noise space, and how different training methods and sampling configurations influence the learned transport. Such an investigation would, in my view, provide a deeper understanding of the observed phenomena. As it stands, I do not find the current contributions sufficiently compelling to justify acceptance.
Questions for authors
I have two additional questions about the experiments.
-
I'm a bit concerned about the confidence-stratified experiments. For low-confidence samples, the pseudo-labels given by the classifier may be less reliable, and even small perturbations could change the predicted labels. This also means that learning to predict these labels could be inherently more difficult. Since the paper always uses classifier-predicted labels as ground truth, I'm wondering how much of the observed difference in seed-level predictability actually comes from the transport, rather than the difficulty of predicting these labels in the first place.
-
I'm also curious about the LDA results in Table 8. Why is the LDA accuracy so much higher than that of the CNN probes in Table 7?
2 comments
Thank you for the rebuttal. My main concern remains. While the results provide empirical measurements, I still feel that a deeper investigation into the underlying mechanisms is needed to offer sufficiently meaningful new insights. I am inclined to maintain my original assessment.
Response to Q1 - Q2 / W1
We thank the reviewer for the thoughtful assessment. Our study quantifies practical semantic recoverability from raw generation states. We group the contribution and confidence concerns below, while distinguishing the measured patterns from the open question of their geometric explanation.
W1. Contribution and transport geometry
We agree that regularity of the learned transport provides a plausible intuition for seed-level predictability. However, this intuition does not establish how accurately a finite-data probe can predict endpoint semantics or how accuracy changes during generation. Our contribution is a systematic measurement of this recoverability along realized reverse trajectories.
Although every DDIM state determines the endpoint label, probe accuracy varies substantially across sampling times and confidence strata. The new CIFAR-10 DDIM–DDPM comparison shows that the difference between samplers is largest early in sampling and nearly closes after 80 reverse updates. The confidence controls below test the robustness of the variation across strata.
We regard these results as quantitative evidence about practical semantic recoverability. Characterizing which geometric properties produce these patterns remains an important open direction. We will make this scope and the empirical contribution explicit in the revision.
Q1 / W1. Confidence hierarchy and label ambiguity
We agree that the difficulty of reproducing the assigned labels can contribute to the confidence hierarchy. We examined this using comparisons where final-image label recovery is nearly complete, normalization by endpoint accuracy, and an independent confidence classifier.
On MNIST, seed accuracy drops from 84.19% at level 1 to 56.29% at level 7, while final-image probe accuracy stays at or above 99.96% across these levels (Table 7). On Fashion-MNIST, seed accuracy drops from 81.40% to 56.21% across the same levels, while final-image probe accuracy stays at or above 98.71%. Thus, substantial seed-level differences occur even among strata where the final-image probe recovers almost all assigned labels.
For CIFAR-10, we normalize within each confidence level ℓ and paired replicate r using
Rℓ,r(t)=Aℓ,r(0)−cAℓ,r(t)−c,c=0.10.
Here t=1 is the seed and t=0 the final image. This measures the fraction of above-chance final-image accuracy recovered at time t. We compute the ratio within each replicate before averaging. At the seed, mean normalized accuracies for levels 1, 5, and 10 are 41.9%, 32.6%, and 25.8%, respectively. The hierarchy persists after this adjustment.
We also kept classifier A's final-image labels as targets and formed confidence deciles within each A-assigned class using an independently trained classifier B's top-two logit margin. Mean held-out seed accuracies at levels 1, 5, and 10 were 45.7%, 36.8%, and 19.8%, respectively. The ordering therefore persists when a separate classifier supplies confidence.
We will incorporate these controls and normalized results into the revision and discuss their implications for interpreting the confidence hierarchy.
Q2. LDA statistic versus CNN accuracy
We thank the reviewer for identifying this inconsistency. Table 8 reports an LDA discriminability statistic based on the ratio of between-class to within-class scatter. We incorrectly described it as classification accuracy in the table and accompanying text. Its numerical values measure class separation, so they do not establish higher predictive accuracy than the CNN results in Table 7. We will correct the terminology and provide the precise calculation and scaling used in Table 8.