acceptodds
Under review as a conference paper at ICLR 2027

Timestep Weighting in Frozen Diffusion Classifiers: Global Partitions and Pointwise Reachability

Abstract

Diffusion classifiers aggregate class-conditional denoising losses across timesteps, often using weights selected for accuracy. Yet evaluating particular weighting rules leaves open which decisions any shared weighting can express. We characterise this decision family for frozen scorers with nonnegative weights. Linear and mixed-integer programs compare a label-aware pointwise ceiling with the best shared weighting under the same strict-margin criterion, separating individually feasible decisions from jointly feasible ones. A positive gap occurs in all six audited loss tables, spanning two model families and two datasets. On Stable Diffusion 1.5, the gaps are exactly 11 and 10 images on the original Pet and CIFAR-10 tables, respectively, and 2–3 images on higher-accuracy CIFAR-10 noise blocks with Pillow (PIL) preprocessing; each table contains 100 images. Rank-preserving controls implicate cross-timestep class-order agreement in restricting reachability. A controlled timestep-support intervention changes pointwise reachability on Stable Diffusion. In an idealised Gaussian model with correctly specified means and class-shared covariances, we characterise Bayes-preserving metrics, prove spectral and cone obstructions, and derive an exact distance to the preserving set. The finite-table audit separates fitting shortfalls from shared-family incompatibility; the Gaussian results separately characterise population partition preservation under explicit assumptions.

Primary area:
generative models
Keywords:
diffusion models, diffusion classifiers, zero-shot classification, timestep weighting, pointwise reachability, linear programming, mixed-integer programming, exact certification, Bayes decision regions, Gaussian score models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LVMnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the timestep weighting problem in diffusion classifiers. The authors find that task-specific weighting can outperform uniform weighting, while sample-specific weighting can further outperform a single shared weighting scheme. The authors also attempt to explain these observations through theoretical analysis.

Critical strengths

Both the experimental and theoretical analyses appear to be reasonably solid. I also find the research question meaningful for gaining a better understanding of diffusion classifiers.

Critical weaknesses
  1. My biggest concern with this paper is that it still requires substantial editing, rewriting, and restructuring. I am not convinced that the current manuscript has undergone sufficiently careful revision and reorganization of the initial AI-assisted drafts (this is further supported by the authors' AI use statement). At least for me, the paper is very difficult to read, with many expressions and formulations that are unnecessarily complicated, uncommon, or difficult to follow. I believe the authors need to put considerably more effort into rewriting the manuscript so that it can provide clear and meaningful insights to the broader machine learning community, rather than merely presenting a collection of technically detailed results.

  2. Another significant issue with the presentation is that nearly every paragraph in the experimental sections of the main text refers to results or discussions in the appendices. As a result, I do not find the main paper sufficiently self-contained. Moreover, the appendices span more than 50 pages and contain a large number of additional experiments, including failed or inconclusive exploratory studies, experiments only loosely related to the main research questions, and overly detailed implementation and verification records (e.g., the extensive descriptions of code versions, hashes, random seeds, execution environments, and verification procedures in Appendix Q). I do not think the manuscript in its current form is well suited to a conference with a nine-page main-text limit. The authors should carefully reconsider the organization and presentation of the entire paper, identify the most important contributions, and substantially streamline the remaining material.

  3. Regarding the technical content, one potential concern is that the paper does not establish a convincing connection between its theoretical and empirical analyses. Most of the attempts in the appendices to validate the Gaussian assumptions for real diffusion models or connect the Gaussian theory to the observed behavior were unsuccessful. Furthermore, I feel that the connection to diffusion models is not particularly strong in some parts of the paper. The empirical analysis can largely be viewed as a general classifier aggregation problem, while many of the theoretical results are derived under assumptions or relaxations that are not specific to diffusion models. However, I would like to emphasize that this is not my primary concern with the paper. Even though there is a gap between the theory and actual diffusion models, I still believe that this type of analysis is meaningful and can potentially provide useful insights into the behavior and limitations of diffusion classifiers.

Finally, I would like to reserve my judgment on the novelty and significance of the paper's contributions to some extent. As discussed above, the presentation makes it difficult for me to fully identify and appreciate the key insights of this work. I do not rule out the possibility that the paper contains more substantial contributions than the current writing is able to convey.

Questions for authors

I do not have any further questions for the authors. My only request is that the manuscript, at least from my perspective, would benefit greatly from more substantial human revision and editing to improve its readability and clarity.

Reviewer_qgtMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of timestep weighting in frozen diffusion classifiers. The authors use LP and MILP to compare the best accuracy achievable with shared weights against a pointwise upper bound. Experiments on Stable Diffusion 1.5 and PixArt-α show that shared weighting cannot achieve this upper bound. The paper also provides theoretical analysis under a Gaussian model to explain the limitations of timestep weighting.

Critical strengths
  1. Interesting research topic. The paper studies the fundamental limitations of timestep weighting rather than proposing another weighting method. This provides a useful perspective on diffusion classifiers.
  2. Rigorous experiments. The authors use LP and MILP to verify the performance limits and conduct experiments across different models, timestep settings, and preprocessing methods.
Critical weaknesses
  1. Limited connection between theory and experiments. The theoretical analysis relies on Gaussian assumptions, but the experiments in Section 5.3 do not provide sufficient evidence that these assumptions hold for real diffusion models. More importantly, the theory investigates whether timestep weighting can recover Bayes-optimal decision boundaries, while the experiments demonstrate that different images may require incompatible timestep weights. The paper does not clearly establish whether the theoretical limitations can explain the empirical sharing gaps. Therefore, the connection between the theoretical analysis and the experimental findings remains insufficiently explored.
  2. Insufficient analysis of the sharing gap and its practical implications. Although the paper demonstrates the existence of sharing gaps, their magnitude varies significantly across different evaluation settings. The authors could provide more analysis of why these differences occur. Furthermore, since the pointwise ceiling requires ground-truth labels, it remains unclear how these findings could help improve practical diffusion classifiers.
Questions for authors
  1. What factors determine the magnitude of the sharing gap? The sharing gap decreases from 10–11 images in the original experiments to 2–3 images under some higher-accuracy PIL settings, while the additional 500-image experiment still shows a noticeable gap. Can the authors further explain these differences? In particular, how do preprocessing, timestep selection, and cross-timestep class-order agreement influence the sharing gap?
  2. How are the Gaussian theoretical results related to the empirical sharing gaps? The theoretical analysis shows that timestep weighting cannot always recover Bayes-optimal decision boundaries, while the empirical results demonstrate incompatibility between different images under shared weighting. Given that Section 5.3 provides limited support for the Gaussian assumptions, can the authors clarify whether the theoretical limitations explain the observed sharing gaps in real diffusion classifiers?
  3. What practical insights can be drawn from the proposed analysis? Since the pointwise ceiling requires ground-truth labels and cannot be directly achieved during inference, how can the sharing gap help guide the development of better timestep weighting strategies? Furthermore, given the relatively small improvement on held-out data, can the authors discuss how the identified limitations relate to practical classification performance?
Reviewer_9m1NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies classification using frozen conditional diffusion models, where predictions are obtained by aggregating class-conditional denoising losses across different noise levels. It investigates which labels can be recovered by nonnegative timestep weighting.  The authors use linear and mixed-integer programming to characterize these possibilities on fixed loss tables. They also develop an analysis under an idealized Gaussian model, characterizing when timestep weighting preserves Bayes decision regions and identifying obstructions to doing so.

Critical strengths
  • The authors provide an extensive theoretical analysis of when nonnegative timestep weighting preserves Bayes decision regions in an idealized Gaussian model. The characterization of preserving metrics and the distinction between spectral and nonnegative-cone obstructions make the limitations of this model precise.
  • The comparison between the best shared weighting and the ceiling obtained by choosing weights separately for each example distinguishes limitations of the weighting family from shortcomings of a particular fitting procedure. The use of optimization certificates makes some of the claims about the audited loss tables precise.
Critical weaknesses
  • The motivation for the studied classification mechanism is insufficiently developed. The introduction identifies zero-shot reuse of pretrained diffusion models as a motivation, but I did not find a sufficiently compelling explanation of when this approach is preferable to standard methods (standard classifiers, VLMs). The experiments do not include such comparisons, leaving the practical significance of the studied decision family difficult to assess.
  • The presentation makes the central takeaway difficult to identify. As written, the extensive technical development makes it difficult to identify the principal insight that should change how a reader understands or uses diffusion classifiers.
  • The implications for actual diffusion classifiers remain unclear. I would like a clearer explanation of what useful conclusion follows from this paper for practical diffusion classifiers.
Questions for authors
  1. What concrete setting motivates using the studied diffusion classifier over a conventional classifier or a vision-language model?
  2. What is the intended connection between the finite-table audit and the Gaussian analysis? Given the findings in Section 5.3, which insights from the Gaussian theory should inform our understanding of the measured diffusion classifiers, and which should be viewed as results about the idealized model alone? 3, Can the authors provide a simple worked example showing how the audit would guide a meaningful choice of timestep support or weighting family? In particular, what decision does the audit help a practitioner make beyond observing that a more flexible weighting family can express additional predictions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.