acceptodds
Under review as a conference paper at ICLR 2027

Transferable Stability Prediction for Budgeted Diffusion Classification

Abstract

Diffusion models can classify images by comparing class-conditional denoising losses, but these comparisons are sensitive to sampled noise, making reliable prediction computationally expensive. Adaptive sampling could reduce this cost by allocating more draws to predictions that have not stabilized, yet sampling need changes with both the model and the observations seen so far. We ask whether a learned stability rule can remain useful across these changing conditions and transfer across tasks and models. We introduce CNSS, which represents class competition through relative losses and rankings across noise levels and repeated samples, without depending on label identities. A lightweight predictor is trained, without ground-truth labels, to predict agreement with a five-sample reference; once trained, its weights and normalization are frozen and used to allocate a shared sampling budget. Across eight datasets, transferring ImageNet-trained predictors between Stable Diffusion 1.5 and 2.0 improves accuracy averaged over the tested budget range by 1.71 and 1.95 percentage points over uniform sampling at matched compute. Using schedules selected on each receiving model’s source data, the transferred predictors also outperform strong native and transferred quadratic baselines. These results show that diffusion-classification losses contain reusable information for allocating further sampling.

Primary area:
generative models
Keywords:
Diffusion-based Classification, Test-time Compute Allocation, Adaptive Sampling, Cross-domain Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qWN5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a limited sampling budget for diffusion based zero shot classification. It introduces TAPS, a transferable policy that uses class permutation invariant features from denoising losses to predict which images need additional noise draws. The readouts and sampling schedule are trained on source data, then frozen and transferred across tasks and diffusion backbones while adapting priorities to observed losses. Experiments show that TAPS generally improves budgeted classification over uniform allocation and several adaptive baselines, although the transfer benefit depends on the receiving model and donor policy.

Critical strengths
  1. The paper addresses a practically important bottleneck in diffusion based classification and formulates transfer as a complete policy problem. Freezing the readouts, normalization, and schedule while allowing priorities to adapt to receiving backbone observations is a meaningful distinction from methods that require target side refitting.

  2. The experimental design includes several useful controls. Shared budgets, cached losses, paired bootstrap intervals, source subset refits, readout ablations, fixed schedule comparisons, and within image stability diagnostics make the claimed role of observation dependent allocation more credible than a comparison against uniform sampling alone.

Critical weaknesses
  1. The transfer claim is stronger than the evidence in some parts of the paper. The main transfer study covers only two closely related Stable Diffusion backbones and eight image classification tasks, while the SDXL experiment is small and shows that the advantage over stronger controls depends on the donor. More evidence is needed to distinguish a generally transferable signal from a signal that transfers mainly between related diffusion models and preprocessing pipelines.

  2. The policy depends on a source selected stage schedule that is frozen during transfer, and the schedule itself can contribute more than the readout. The paper reports that replacing the receiver selected schedule with a balanced schedule reduces AUC substantially, while the readout gain over a shared schedule is much smaller. This makes it difficult to determine whether the main improvement comes from transferable stability prediction or from source specific allocation of sampling depth. The relationship between finite reference disagreement and actual classification accuracy is also indirect, since the self supervision target is not semantic error.

Questions for authors
  1. Can the authors provide transfer results across substantially different diffusion families or conditioning pipelines, rather than primarily between SD 1.5 and SD 2.0? In particular, please report whether the same frozen policy transfers across different text encoders, image resolutions, noise schedules, or prediction parameterizations, and quantify performance degradation as the receiver becomes less similar to the donor.

  2. How much of the reported gain is attributable to the source selected schedule rather than the TAPS readout? Please include a fully controlled comparison in which transferred TAPS, transferred Quad, Margin, and uniform allocation all use the same schedule, as well as a comparison in which each policy uses a schedule selected independently on the receiver. Please report both macro AUC and per task results.

  3. The training target is disagreement with a five draw reference prediction rather than semantic correctness. How sensitive are the results to the reference draw count and to the choice of disagreement target? Please evaluate alternative reference counts, such as three and ten draws, and report correlations between predicted risk, disagreement, and eventual classification error or accuracy improvement from an additional draw.

  4. How does TAPS behave under different candidate set sizes and class imbalance? The feature dimension is designed to be independent of the number of classes for sufficiently large candidate sets, but the experiments appear to mix tasks with very different class counts. Please provide controlled results varying the number of candidate classes and clarify whether the same readout and schedule are used without task specific calibration.

  5. What is the practical overhead of computing the feature map, applying the readouts, and reranking active images at every stage? Since the paper emphasizes efficient inference, please report wall clock latency and memory for TAPS, uniform allocation, and the strongest adapted baselines under realistic batch sizes, including cache construction and routing overhead. It would also be useful to show whether the ranking overhead remains negligible when the number of images or candidate classes is much larger.

Reviewer_koaWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies budgeted inference for diffusion classifiers. A diffusion classifier predicts the class with the lowest class-conditional denoising loss, averaged over repeated noise draws. More draws give more stable predictions but cost more. The paper asks how a fixed total draw budget should be allocated across a batch of images, and whether one allocation policy can be reused across tasks and backbones without refitting.

The proposed method, TAPS (Transferable Allocation via Predicted Stability), builds class-permutation-invariant features from the losses already observed for each image. These features include normalized gaps between the leading aggregate scores, per-timestep local gaps and scales, the loss profiles and ranks of the two leading classes, agreement between timestep-wise and aggregate winners, and dispersion across draws. At each stage, an ℓ2-regularized logistic readout predicts whether the current prediction disagrees with a five-draw reference prediction; this target requires no ground-truth labels. Images are ranked by this predicted risk, and the top-ranked images receive one more draw. How many images continue at each stage is fixed by a schedule selected on source data with a training-free Margin heuristic. The readouts, feature normalization, and schedule are fitted on ImageNet with one backbone and then frozen. The receiving backbone supplies only the losses and the final mean-loss predictions, and computing the features requires no extra denoiser evaluations.

Experiments transfer the policy between Stable Diffusion 1.5 and 2.0 across eight target datasets (12,023 images), with nine timesteps, at most five draws, and average budgets from 1.5 to 4.5 draws per image. The transferred policy improves macro-averaged normalized accuracy–budget AUC over uniform allocation by 1.718 pp on SD1.5 and 1.945 pp on SD2.0. It also exceeds Margin, a quadratic logistic control (Quad), and adapted PETS-OKG and Blend-ASC baselines, and comes close to policies fitted natively on the receiving backbone.

Critical strengths

Strength

A rigorous, budget-exact, and well-controlled experimental protocol. All policies share the same cached losses, the same FP32 mean-loss decision rule, and exactly ⌊NB⌋ draws per batch, with the initial draws charged to the budget (Eq. 5). Comparisons are paired across policies and draw orders, and the rerouting bootstrap recomputes the allocation on every resampled batch (App. B.2). Shared-schedule controls separate which images continue (the readout) from how many continue (the schedule). The fixed-initial-risk and pair-excluded-reference controls isolate specific design choices. Why it matters to my assessment: this level of control is uncommon in the adaptive-inference literature, and it makes the reported contrasts interpretable. It is the main reason I consider the experiments technically sound and take the paper’s numbers at face value, including the numbers on which my main concerns rest.

Critical weaknesses

Weakness 1: The motivation does not establish a significant research question, and to the extent that the problem exists, existing methods already solve it.

The paper motivates TAPS with three premises:

diffusion classifiers are expensive to run; an allocation policy is hard to transfer without refitting; relative denoising-loss patterns might provide a transferable signal.

None of these premises is established.

First, the paper gives no scenario in which a budget-constrained diffusion classifier would be needed. In the literature, diffusion classifiers are valued mainly for properties specific to generative classifiers, such as robustness, or as probes of what diffusion models understand (e.g., Jaini et al., ICLR 2024; Chen et al., ICML 2024; Krojer et al., NeurIPS 2023), not for efficiency. If efficiency is the concern, CLIP also needs no training and requires one image-encoder pass per image, whereas the classifier here needs 90–918 U-Net passes per draw. SD1.5 and SD2.0 are themselves conditioned on the text encoders of CLIP ViT-L/14 and OpenCLIP ViT-H/14, and the complete CLIP models are public zero-shot classifiers. The paper reports neither this baseline nor any compute saved at matched accuracy, and Appendix A.8 states that the experiments establish neither end-to-end speedup nor amortization of the source-training cost.

Second, the claim that transfer without refitting is difficult is supported only by the single example in Fig. 1. That example shows that stability depends on the observation, not that a policy is hard to transfer. The paper’s own results contradict the claim: native and transferred TAPS reach almost identical macro AUCs (52.116 vs. 52.100; 57.515 vs. 57.485), and transferred Quad matches native Quad. Because native policies are also fitted on ImageNet, these comparisons cover only the backbone shift. Cross-task transfer is never compared with refitting on the target task.

Third, existing methods already address the problem. The top-2 Margin ranking rule has no fitted parameters, transfers across tasks and backbones by construction, and already delivers about 80% of TAPS’s gain over uniform allocation. The diffusion-classifier literature also offers training-free adaptive sampling. Clark & Jaini (2023) stop once a single candidate remains, and Li et al. (2023) prune classes adaptively. The paper labels these methods “complementary” but neither compares against them nor combines TAPS with them. Cross-sample allocation under a fixed budget (PETS, Blend-ASC, difficulty-adaptive self-consistency) and the transfer of learned stopping rules across tasks and models (Adaptive Thinking, LearnStop) have also been studied.

Why it matters to my assessment: the central research question is neither shown to matter nor left open by existing methods. This is the main reason for my recommendation, and careful execution cannot compensate for it.

Weakness 2: The proposed method adds little beyond existing components, its new elements are not shown to address the stated problem, and its gains are marginal and unvalidated.

Each component of TAPS has a direct precedent in the literature the paper cites:

the agreement-with-full-budget target corresponds to CATs and CALM; learned predictors that transfer across tasks and models correspond to Adaptive Thinking and LearnStop; budgeted cross-sample allocation corresponds to MSDNet and Blend-ASC; the schedule is a grid search driven by Margin.

What remains is a set of hand-designed 67- to 107-dimensional loss-statistic features with one logistic regression per stage.

The paper gives no evidence that these new elements are what solves the stated problem. It presents class-permutation invariance as the source of transferability, but the inputs to Margin and Quad are equally invariant (App. A.7). It shows no case in which naive transfer fails and TAPS fixes it. The schedule, which has the largest effect on results, comes from Margin; selecting the schedule with TAPS’s own scores is worse at low budgets on SD1.5 (−1.355 pp accuracy at B = 1.5, App. D.2). The training target is also misaligned with the evaluation. Appendix A.6 acknowledges that disagreement with the five-draw reference is a surrogate whose optimality holds only when selected images are sampled to the full budget at once, whereas TAPS adds one draw at a time and is evaluated by accuracy.

The resulting gains are marginal. TAPS exceeds Margin by only 0.291 / 0.390 pp AUC and native Quad by 0.140 / 0.279 pp (SD1.5 / SD2.0). About 80% of its 1.718 / 1.945 pp gain over uniform allocation comes from Margin and its schedule. The gains are also unstable at the task level. On SD2.0, TAPS is significantly below both Quad variants on Flowers-102 and below native TAPS on Aircraft, with intervals entirely below zero (Table 9).

The key properties the method claims or relies on are not validated:

no end-to-end speedup, wall-clock time, or amortization of the source cost is reported; transfer is tested only between sibling backbones, plus a single-task SDXL test with 320 images and no intervals; no architecturally different backbones (DiT, flow matching) are evaluated; the feature shift caused by the candidate count changing from 1,000 to 10–102 classes is not analyzed.

Why it matters to my assessment: even granting the problem, the method contributes about 0.3 pp over a training-free heuristic, and that gain is not shown to come from its claimed mechanism. The paper’s central claim, that TAPS learns relative class competition as a transferable signal, is therefore not supported.

References († = not cited in the submission)

† Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via a single diffusion model. In ICML, 2024.

† Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.

Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. In NeurIPS, 2023.

Zhe Dong, Fang Qin, and Manish Shah. When does learning to stop help? A cost-aware study of early exits in reasoning models. arXiv:2606.30852, 2026.

Austin Feng, Marius Alonso, Ambroise Odonnat, Vasilii Feofanov, and Ievgen Redko. Optimal self-consistency for efficient reasoning with large language models. In ICML, 2026.

Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018.

† Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. In ICLR, 2024.

† Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, and Siva Reddy. Are diffusion models vision-and-language reasoners? In NeurIPS, 2023.

Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In ICCV, 2023.

Pingzhi Li, Bairu Hou, Yun Zhu, Yihao Feng, Ke Ye, Tao Lei, Zhifeng Chen, Tianlong Chen, and Xianzhi Du. Adaptive thinking: Large language models know when to think in latent space. In ICLR, 2026.

† Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023.

Zhangyi Liu, Huaizhi Qu, Xiaowei Yin, He Sun, Yanjun Han, Tianlong Chen, and Zhun Deng. PETS: A principled framework towards optimal trajectory allocation for efficient test-time self-consistency. arXiv:2602.16745, 2026.

† William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.

† Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.

Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. In EMNLP, 2021.

Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In NeurIPS, 2022.

† Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, and Kan Li. Make every penny count: Difficulty-adaptive self-consistency for cost-efficient reasoning. In Findings of NAACL, 2025.

Questions for authors

Questions that could change my assessment (clarifications and analyses feasible with the existing caches)

  1. Use case and the CLIP baseline (W1). In what concrete scenario would a budget-constrained diffusion classifier with K ≤ 5 and J = 9 be preferred over the zero-shot CLIP ViT-L/14 and OpenCLIP ViT-H/14 models whose text encoders condition SD1.5 and SD2.0? Please report the zero-shot accuracy and per-image cost of these CLIP models on the eight target tasks. If the intended use case relies on properties specific to generative classifiers, such as robustness, does adaptive allocation preserve those properties?
  2. Compute actually saved (W1). At matched accuracy, how many draws, or how much wall-clock time, does TAPS save relative to uniform allocation and to Margin? Please include the cost of the five-draw ImageNet source cache, which by my count from App. B.1, Table 7, and Eq. 29 is about 45M logical denoiser evaluations. Over how many target images or tasks is this source cost amortized?
  3. Evidence that transfer without refitting is hard (W1). Is there any setting in which naive transfer fails, that is, in which a transferred policy falls below its native counterpart or below Margin and TAPS fixes this? For cross-task transfer, how does TAPS compare with a readout refitted on the target task’s reserved unlabeled images (App. B.1)? Why must refitting be avoided, given that it needs no labels?
  4. Comparison with existing training-free adaptive methods (W1). At matched budgets, how does TAPS compare with the statistical elimination and early stopping of Clark & Jaini (2023) and with the adaptive class pruning of Li et al. (2023)? Does TAPS still add a gain when combined with them? How does the paired t-statistic on the top-2 gap (Quad’s v₉) perform as a standalone training-free priority?
  5. The learned component versus Margin (W2). Can you report paired rerouting confidence intervals for transferred TAPS minus Margin, both for the complete policies and under a shared schedule? Can you also report an oracle allocation bound under the same schedule, for example by ranking images by their true disagreement with the five-draw reference? What fraction of the oracle–Margin gap does TAPS close?
  6. What drives the advantage (W2). Margin and Quad are also class-permutation invariant (App. A.7). What evidence attributes TAPS’s advantage to its new relative-competition features rather than to having more inputs? Can you also report the Eq. 8 interaction and the pair-excluded within-image ranking for Margin and Quad?
  7. Robustness of the small gains (W2). Do the 0.1–0.4 pp gains over Margin and Quad hold on an independent set of five noise draws, with the reference recomputed and the frozen policies unchanged?

Clarification requests

  1. Which Margin schedules q were selected on SD1.5 and SD2.0 at each budget, and at how many budgets do the two differ?
  2. Were the following choices fixed before any target-task results were seen: the Margin-selected schedule (rather than the readout’s own schedule), the feature set, and the PETS-OKG prior calibration?
  3. Can you report the per-budget accuracy of the main method, transferred TAPS? Table 15 omits it.
Reviewer_Zm4gOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce TAPS (Transferable Allocation via Predicted Stability), which uses class-permutation-invariant features of relative class competition to predict disagreement against a five-draw reference prediction and allocate fixed sampling budgets. The paper also compares with several adaptive allocation baselines, runs cross-backbone transfer experiments, ablates features, and analyzes stagewise stability updates. These controlled experiments help clarify the contributions of the proposed components.

Critical strengths
  1. The paper demonstrates that a lightweight stability predictor trained on one diffusion backbone can guide adaptive test-time computation on another backbone without retraining.
  2. The stability predictors are trained with finite-draw reference predictions, without requiring ground-truth classification labels.
Critical weaknesses
  1. Limited evidence supporting cross-backbone transferability. One of the central claims of this paper is that the proposed stability predictor can generalize across different diffusion backbones without receiver-side retraining. However, the main experiments primarily focus on SD1.5 and SD2.0, which share similar architectural designs. It remains unclear whether the proposed method can generalize to diffusion transformer (DiT)-based architectures, such as Hunyuan-DiT or SD3. Furthermore, the results in Table 14 raise additional concerns regarding the robustness of cross-backbone transfer. Specifically, the SD1.5-trained Full predictor underperforms the simpler Quad predictor when transferred to SDXL, whereas the SD2.0-trained Full predictor outperforms Quad. The authors should provide further analysis to explain this discrepancy and clarify which properties of the source backbone affect transferability.

  2. Relatively marginal improvements over strong baselines. Although the proposed method consistently improves upon uniform allocation in the main experiments, its improvements over stronger baselines are relatively modest. As shown in Table 2, transferred TAPS improves normalized accuracy–budget AUC by less than 0.5 percentage points compared with native Quad on both SD1.5 and SD2.0. Given the additional feature engineering and learned stability predictors introduced by TAPS, the practical significance of these improvements deserves further discussion.

  3. High computational cost of diffusion-based classification. The underlying diffusion classification framework requires evaluating class-conditional denoising losses across multiple diffusion timesteps for every candidate class. Consequently, even a single noise draw involves a substantial number of denoiser evaluations, and the overall computational cost increases with the number of candidate classes. Although TAPS reduces the number of noise draws allocated to individual images, it does not directly address the computational cost of evaluating all candidate classes at each draw. It would be helpful for the authors to provide an end-to-end computational analysis, including wall-clock inference time, GPU utilization, and scalability with respect to the number of candidate classes, to better demonstrate the practical efficiency of the proposed method.

  4. Limited investigation of larger reference sampling budgets. The proposed stability predictor is trained using disagreement with a five-draw reference prediction, and the experiments primarily consider a maximum of five noise draws. However, it remains unclear whether the same approach would remain effective with substantially larger reference budgets, such as 10 or 20 draws.

  5. Unclear robustness to variations in the number of candidate classes. The proposed feature representation is designed to be independent of class identities and to have a fixed dimensionality for candidate sets containing at least five classes. However, this property does not necessarily guarantee robustness when the number of candidate classes varies substantially.

  6. Sensitivity to batch composition. The proposed allocation policy is batch-relative, selecting the images with the highest predicted instability scores within each batch for additional sampling. Consequently, the number of noise draws allocated to an individual image may depend on the other images included in the same batch. This raises concerns about the robustness and consistency of the allocation decisions under different batch compositions and batch sizes. The authors should investigate how changes in batch composition affect per-image allocation, overall classification performance, and computational efficiency.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.