Transferable Stability Prediction for Budgeted Diffusion Classification

Diffusion models can classify images by comparing class-conditional denoising losses, but these comparisons are sensitive to sampled noise, making reliable prediction computationally expensive. Adaptive sampling could reduce this cost by allocating more draws to predictions that have not stabilized, yet sampling need changes with both the model and the observations seen so far. We ask whether a learned stability rule can remain useful across these changing conditions and transfer across tasks and models. We introduce CNSS, which represents class competition through relative losses and rankings across noise levels and repeated samples, without depending on label identities. A lightweight predictor is trained, without ground-truth labels, to predict agreement with a five-sample reference; once trained, its weights and normalization are frozen and used to allocate a shared sampling budget. Across eight datasets, transferring ImageNet-trained predictors between Stable Diffusion 1.5 and 2.0 improves accuracy averaged over the tested budget range by 1.71 and 1.95 percentage points over uniform sampling at matched compute. Using schedules selected on each receiving model’s source data, the transferred predictors also outperform strong native and transferred quadratic baselines. These results show that diffusion-classification losses contain reusable information for allocating further sampling.