acceptodds
Under review as a conference paper at ICLR 2027

Improved Distributional Diffusion Models

Abstract

Distributional Diffusion Models (DDMs) replace the standard mean-prediction denoiser with a distributional denoiser trained via a scoring rule objective, learning a stochastic approximation to rather than its conditional mean. However, scaling DDMs to modern image-generation settings faces two obstacles: (i) multi-particle training incurs overhead that scales with the number of particles, (ii) DDMs use globally fixed scoring rule hyperparameters, forcing a single trade-off across sampling budgets. We mitigate these limitations by deferring particle expansion to late transformer layers, and the hyperparameter trade-off by introducing time-dependent scoring rule schedules informed by the dynamical regimes of Biroli et al. (2024). Combined with a DiT-based latent setup, these changes make DDM training practical on class-conditional ImageNet-, achieving 4.48 FID at 4 steps and 2.38 at 50 steps with DiT-XL/2, from a single model trained from scratch in one stage, without a teacher, self-distillation or JVPs. The result is a stochastic few-step generator whose FID does not degrade as the sampling budget grows from 4 to 50 NFE, and the same recipe transfers to text-to-image generation. Code and pre-trained models will be released with the camera-ready.

Primary area:
generative models
Keywords:
diffusion models, distributional diffusion models, accelerated sampling of diffusion models, scoring rules, generative models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QdnQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses two practical limitations of Distributional Diffusion Models: the cost of processing multiple stochastic predictions for each noisy example, and the difficulty of using one fixed scoring-rule configuration across different sampling budgets. It proposes sharing early transformer computation across particles and injecting particle-specific noise only in late layers. It combines this architecture with adaptive noise conditioning, time-dependent generalized energy-score parameters λ(t)\lambda(t) and β(t)\beta(t), and a modified training-time distribution.

The resulting iDDM model is trained without a diffusion teacher or self-distillation. ImageNet experiments include architectural and objective ablations, independent component additions, compute-matched comparisons, additional quality and coverage metrics, and a conditional-diversity diagnostic. At XL scale, the paper reports FID values of 4.48, 3.04, 3.03, and 2.38 at 4, 8, 16, and 50 sampling steps, respectively. A text-to-image experiment also shows improved performance at small sampling budgets, with some important compositional limitations.

The contribution is principally a computational and empirical improvement to an existing distributional-training framework, supported by a limited theoretical rationale for the direction of its schedules.

Critical strengths
  1. A consequential computational modification, with evidence about its representational trade-off. Deferred expansion separates feature extraction shared by all particles from the computation that must depend on auxiliary noise. This changes the placement of stochastic computation while preserving shared parameters across particles; it is not simply a renamed loss or an independently parameterized multi-head model. For L=28L=28, ℓstart=24\ell_{\mathrm{start}}=24, and m=4m=4, the idealized block-count cost is (24+4⋅4)/(28⋅4)≈0.357(24+4\cdot4)/(28\cdot4)\approx0.357 of naïve DDM, or approximately 1.43 times single-particle FM. Table 12's measured XL training-step ratio is approximately 1.41 relative to FM, supporting the practical significance of the saving.

    Table 2(a) and Figure 4 also test the important boundary: too little particle-specific computation can hurt quality, and unguided generation prefers more distributional layers, whereas guidance can reverse that preference. Selecting a short distributional tail is therefore supported by comparisons, not merely asserted as an efficiency heuristic. This offers a useful direction for future stochastic generative architectures.

  2. A substantial, reasonably controlled body of validation supports a complete empirical contribution. The paper includes cumulative ablations in Table 1, conditioning and depth comparisons in Table 2, schedule and time-sampling comparisons in Table 3, independent additions in Table 8, and multiple quality and coverage metrics in Table 9. Most importantly, Table 10 addresses the extra training cost: at the larger matched budget, four-step FID improves from 26.36 for FM to 13.13 for iDDM, while fifty-step FID is 4.21 versus 4.61. Thus, the few-step benefit survives compute matching, while the many-step trade-off is visible.

    The final XL checkpoint also improves monotonically across the reported step grid, and Table 13 extends the method to text-to-image with improvements in four-step FID, GenEval, human-preference score, and most compositional categories. Together, these results establish more than a parameter-tuning anecdote. The authors also acknowledge stronger alternatives at very small budgets and do not need to solve posterior calibration or every application to make this a complete empirical study.

Critical weaknesses
  1. The full-posterior interpretation is stronger than the default objective and diagnostics justify. Strict propriety of the global energy score applies at λ=1\lambda=1 and 0<β<20<\beta<2. The default schedule deliberately uses λ(t)<1\lambda(t)<1 away from the noise endpoint, introducing a bias whose Gaussian-surrogate interpretation is discussed in Appendix D. In addition, the default local score in Appendix F identifies per-position marginals in the proper-score regime, not the full joint law across positions. For example, equal mixtures on {(−1,−1),(1,1)}\{(-1,-1),(1,1)\} and {(−1,1),(1,−1)}\{(-1,1),(1,-1)\} have identical coordinate marginals and identical expected local scores, despite opposite dependence structures.

    Appendix F explicitly acknowledges the loss of cross-spatial constraints, and Appendix B correctly states that Table 11 measures conditional diversity rather than posterior accuracy. These are appropriate caveats, but they should also qualify the abstract and the main-text characterization of the model as learning p(x1∣xt)p(x_1\mid x_t). The evidence establishes a useful stochastic, time-dependent, biased distributional predictor; it does not establish calibrated full-posterior sampling.

    The regime connection is similarly a rationale for schedule direction, not a demonstrated necessity of the estimated transition times. Appendix D's rearrangement argument supports assigning stronger shrinkage to higher SNR under its covariance criterion. The selected linear schedule does not require those thresholds, and Table 3 contains alternatives with comparable or better FID in some settings. This limits the conceptual claim, although it does not undermine the practical recipe.

  2. The strength and boundaries of the claimed performance frontier need tighter characterization. The large improvement over naïve DDM and the matched-compute FM result are convincing. However, the tables do not report uncertainty across independent training runs. Smaller choices, such as linear versus dynamical-regime schedules or isolated conditioning refinements, are therefore not established with the same confidence. Table 1 also shows that adding the schedules improves four-step FID from 15.74 to 14.33 while worsening fifty-step FID from 5.47 to 5.69; the final improvement at both budgets follows the subsequent change to time sampling. The evidence should be described as a tested combination with component interactions, rather than assigning every gain to the schedule alone.

    The transfer result has a meaningful exception. In Table 13, iDDM's spatial-relation score is below FM at every evaluated budget: 0.1178 versus 0.2901 at sixteen steps, and 0.0232 versus 0.0443 at four steps. Smaller degradation from an already lower reference score does not imply better absolute compositional performance. The text-to-image comparison also uses equal update counts rather than an established matched-compute budget.

    Finally, Tables 4 and 12 should be read with their stated architectural and training differences. The paper already separates controlled experiments from system-level context, which is appropriate. Its stronger total-training-cost comparisons nevertheless depend on the backbone and implementation used for benchmarking, and its monotone-FID result depends on separately selected CFG scales at the tested budgets. Clarifying these conditions would improve confidence in the claimed efficiency frontier.

Questions for authors
  1. How should the full-posterior language be interpreted given the default local score and λ(t)<1\lambda(t)<1? Please distinguish stochastic conditional prediction, identification of local marginals, and calibration of the joint posterior. Can you provide a tractable conditional-distribution check, or explicitly narrow the claim to what the objective and Table 11 establish? This could change my assessment of the paper's conceptual contribution; I am not requesting large-scale posterior-reference experiments.

  2. Does the selected time-dependent schedule outperform a jointly retuned constant (λ,β)(\lambda,\beta) configuration under the same late-expansion architecture, time sampler, compute budget, and per-budget guidance selection? Please identify the relevant comparison and explain which result depends on the regime anchors rather than on generic time-dependent weighting. Appendix D justifies direction under a surrogate, but Table 3 does not establish a uniquely best profile; this distinction affects the claimed novelty and mechanism.

  3. How many independent training runs and evaluation draws support the main results and the smaller ablation differences? Please give uncertainty for the decisive schedule comparisons and state whether CFG and EMA choices were selected on a separate tuning set or the reported evaluation samples. Also, does the final checkpoint retain its qualitative improvement with one fixed CFG across budgets, or only after selecting CFG separately? These answers would affect confidence in the reported frontier.

  4. Is the spatial-relation deficit in Table 13 reproducible across evaluation draws and reasonable guidance choices, and what evidence helps explain it? Is the FM/iDDM transfer comparison matched in total training compute, or only in training steps? Please distinguish the supported few-step improvements from compositional capabilities that worsen. This could change my assessment of the breadth of transfer; it is not a request for additional application domains.

  5. Which depth and implementation were used for each Table 12 benchmark, especially iMF, whose published Table 4 checkpoint has 48 layers? Do the total-cost ratios combine compatible measured step costs and training lengths? Please also state inference latency or FLOPs under matched guidance settings, and clarify whether the reported step/NFE counts include conditional and unconditional CFG computation. These details could materially affect the practical efficiency comparison.

4 comments

Dear reviewer QdnQ,

Thank you for highlighting the importance of our contributions as "a consequential computational modification", highlighting our novelty "it is not simply a renamed loss or an independently parameterized multi-head model" supported by "substantial, reasonably controlled body of validation supports a complete empirical contribution"

We address your questions below.

[W1] The full-posterior interpretation is stronger than the default objective and diagnostics justify.
[Q1] How should the full-posterior language be interpreted given the default local score and λ(t)<1\lambda(t)<1? Please distinguish stochastic conditional prediction, identification of local marginals, and calibration of the joint posterior.

We agree that stochastic conditional prediction and accurate joint-posterior recovery are distinct objectives. Importantly, none of our central empirical claims requires posterior calibration, as our contributions concern training efficiency, few-step generation quality, and conditional diversity.

We characterize the theoretical scope of these guarantees:

  • Section 2.2 establishes that the global energy score is strictly proper for the specific choice of λ=1\lambda = 1 and β∈�0�\beta \in �0�. Section 3.2 explicitly motivates the deliberate variance shrinkage introduced by λ(t)<1\lambda(t) < 1, further analyzed in Appendix D. We inherit it from the original DDM, which already uses λ(t)<1\lambda(t) < 1 and studies this trade-off with a constant λ\lambda. Importantly, time dependence does not inherently compromise strict propriety: our framework also permits λ=1\lambda = 1 and β∈�1�\beta \in �1�, preserving strict propriety at every timestep. Thus, the variance shrinkage introduced by our chosen λ(t)<1\lambda(t) < 1 is a deliberate design choice, not an inherent limitation of time-dependent scheduling.
  • Appendix F explains that local scoring identifies per-position conditional marginals in the strictly proper regime, without guaranteeing their joint dependencies. This does not imply spatial independence, and empirically it’s not an observed behavior since predictions are generated jointly by the network with image and ξ spatially mixed with attention. Under our chosen λ(t)<1\lambda(t) < 1, exact marginal recovery is not guaranteed either.
  • Appendix B.1 explicitly states that Table 12 diagnoses conditional diversity, not posterior accuracy.

To make these distinctions more prominent, we have further refined the introduction to distinguish conditional distribution learning from posterior fidelity and explain the deliberate trade-off with prediction concentration. We have also lightly edited Section 4.2 (positioning paragraph) to emphasize stochastic conditional prediction and diversity rather than suggesting accurate posterior recovery. All changes are reported below and available in the pdf and highlighted in blue.

These refinements make explicit that our empirical conclusions do not rely on calibrated joint-posterior sampling, while retaining the distributional motivation underlying iDDM.

lines 71-72 “DDMs thus offer a non-adversarial alternative to learn conditional distributions beyond their means, while potentially allowing the learned predictions to be more concentrated than the true posterior”

lines 88-90 “Removing both bottlenecks makes distributional denoising practical with large transformer backbones while retaining the goal of approximating p(x1∣xt)p(x_1\mid x_t) rather than only its conditional mean, and permitting deliberate variance shrinkage at high SNR”

lines 412-413 “(i)~It is stochastic: it learns a stochastic conditional predictor trained to sample from an approximate p(x1∣xt)p(x_1 \mid x_t), whereas MeanFlow, iMF, and IMM learn deterministic maps.”

AuthorsOpenReview ↗

Response 2

[Q2] Does the selected time-dependent schedule outperform a jointly retuned constant �0��0� configuration under the same late-expansion architecture, time sampler, compute budget, and per-budget guidance selection?

We thank the reviewer for this suggestion and have run the requested control. We trained a 5×5 grid of constant �1��1� configurations under the same late-expansion architecture, time sampler, compute budget, and per-budget guidance selection as Config D (which corresponds to the �2��2� cell). The resulting heatmap is included as Figure 5 in Appendix B.1.

The grid confirms that the preferred constant depends on the inference budget, consistent with the original DDM paper (De Bortoli et al., 2025, Appendix K.2): the best 4-step constant is �3��3� = (1, 1.95), with FID 14.93 at 4 steps and 5.27 at 50 steps, whereas the best 50-step constant is λ=0.01\lambda=0.01, β=1.95\beta=1.95 (which is very close to a standard diffusion model with λ=0,β=2\lambda=0, \beta=2)), with 4.97 at 50 steps but 25.71 at 4 steps. Time-dependent schedules improve on this reference in the same setting (Table 3b). A β\beta-only schedule with an early switch at t=0.1 reaches 13.06 at 4 steps and 5.10 at 50 steps, improving over the best 4-step constant by 1.87 FID while remaining within single-seed noise of the best 50-step constant (5.10 vs. 4.97; [1]). Adapting the scoring rule across diffusion time therefore outperforms every constant configuration we tested at 4 steps, without a measurable cost at 50 steps.

Our default linear schedule (Config E) yields more modest improvements: relative to the best 4-step constant, it improves 4-step FID from 14.93 to 14.33, at the cost of 50-step FID (5.27 to 5.69). We retain this simple linear parameterization for our main experiments, while the early-switch results demonstrate that alternative time-dependent schedules can achieve further improvements over jointly retuned constants.

[1] Dufour et al., 2026, The FID Lottery

[W1] The regime connection is similarly a rationale for schedule direction, not a demonstrated necessity of the estimated transition times. Appendix D's rearrangement argument supports assigning stronger shrinkage to higher SNR under its covariance criterion. The selected linear schedule does not require those thresholds, and Table 3 contains alternatives with comparable or better FID in some settings. [Q2] Please identify the relevant comparison and explain which result depends on the regime anchors rather than on generic time-dependent weighting [W2] Table 1 also shows that adding the schedules improves four-step FID from 15.74 to 14.33 while worsening fifty-step FID from 5.47 to 5.69; the final improvement at both budgets follows the subsequent change to time sampling. The evidence should be described as a tested combination with component interactions, rather than assigning every gain to the schedule alone.

The dynamical regimes motivate the direction and transition regions of the schedules, not uniquely optimal thresholds. As explicitly stated in Section 3.2, “The regime thresholds interpret the schedule direction but are not required in practice.” Appendix D further supports the orientation analytically. Unlike generic time-dependent weighting, scheduling λ(t)\lambda(t) and β(t)\beta(t) changes the scoring rule itself. Importantly, Table 1 (D →\rightarrow E) and Table 8 isolate its contribution: 4 step FID improves from 15.74 to 14.33, with only a minor change at 50 steps (5.47 to 5.69). The subsequent improvement from t-sampling is complementary and separately ablated.

[Q3] state whether CFG and EMA choices were selected on a separate tuning set or the reported evaluation samples. Does the final checkpoint retain its improvement with fixed CFG across budgets?

We use fixed EMA length of σ=0.5σ=0.5 [4], we don’t sweep this value. CFG was selected by minimizing FID under the standard 50k-sample evaluation protocol, consistently applied across all methods. Selecting CFG independently for each sampling budget is standard practice [1,2,3] and essential for a fair comparison, since optimal guidance generally varies with the number of steps [5].

FID typically follows a U-shaped curve with CFG, with fewer steps favoring stronger guidance. Fixing a low CFG yields monotonically improving FID, but sacrifices optimal quality at individual budgets. Even with CFG optimized separately at each budget (common practice [1, 2, 3]), some competing methods degrade as the step count increases, whereas iDDM consistently improves.

[1] Jonathan Ho et al., 2022, Classifier-Free Diffusion Guidance.
[2] Prafulla Dhariwal et al., 2021, Diffusion Models Beat GANs on Image Synthesis.
[3] Zhengyang Geng et al., 2026, Improved mean flows: On the challenges of fastforward generative models.
[4] Tero Karras et al., 2024, Analyzing and Improving the Training Dynamics of Diffusion Models.
[5] Dufour et al., 2026, The FID Lottery

AuthorsOpenReview ↗

Response 3

[W2] However, the tables do not report uncertainty across independent training runs. [Q3] How many independent training runs and evaluation draws support the main results and the smaller ablation differences?

All models are trained with a single seed per configuration, consistent with common practice [1] in computationally intensive generative modeling. Our main conclusions are supported by large differences: under controlled DiT-B settings, iDDM reduces 4 step FID from 43.68 to 13.13 and 50 step FID from 10.04 to 4.56 relative to naïve DDM, with large gains also at matched compute.

For perspective, The FID Lottery (Dufour et al., 2026) reports a training-seed FID coefficient of variation of approximately 1-2% on ImageNet-256 using SiT. Although this is not a direct variance estimate for iDDM, the magnitude of our improvements is far beyond the variability observed in that study. Moreover, our improvements persist across model scales, sampling budgets, and complementary evaluation metrics.

[1] Dufour et al., 2026, The FID Lottery

[W2] In Table 13, iDDM's spatial-relation score is below FM at every evaluated budget: 0.1178 versus 0.2901 at sixteen steps, and 0.0232 versus 0.0443 at four steps. Smaller degradation from an already lower reference score does not imply better absolute compositional performance. The text-to-image comparison also uses equal update counts rather than an established matched-compute budget. [Q4] Is the spatial-relation deficit in Table 13 reproducible across evaluation draws and reasonable guidance choices, and what evidence helps explain it? Is the FM/iDDM transfer comparison matched in total training compute, or only in training steps? Please distinguish the supported few-step improvements from compositional capabilities that worsen.

We agree that smaller degradation does not imply better absolute compositional performance. Our claim concerns degradation as the step count is reduced, and we now state this more explicitly. We have added a compute-matched comparison (Table 15), in which iDDM is trained for fewer iterations so that total training compute equals FM’s. There, iDDM still degrades less than FM from 16 to 4 steps on every metric, and at 4 steps it is better in absolute terms on all metrics except spatial relations. On spatial relations iDDM is below FM at every budget, which we now report in the text-to-image section (Appendix B.2).

[W2] The paper already separates controlled experiments from system-level context, which is appropriate. Its stronger total-training-cost comparisons nevertheless depend on the backbone and implementation used for benchmarking [Q5] Which depth and implementation were used for each Table 12 benchmark, especially iMF, whose published Table 4 checkpoint has 48 layers? Do the total-cost ratios combine compatible measured step costs and training lengths? Please also state inference latency or FLOPs under matched guidance settings, and clarify whether the reported step/NFE counts include conditional and unconditional CFG computation. These details could materially affect the practical efficiency comparison.

Table 13 benchmarks all methods within the official meanflow/iMF JAX codebase, using identical hardware and batch settings. FM, meanflow, and iDDM use the same 28-layer DiT backbone, while iMF uses its published 48-layer architecture. Total training-cost ratios combine measured per-step runtimes with official training lengths, yielding approximately 1.67x for meanflow and 17.6x for iMF relative to iDDM. These comparisons reflect the published training recipes, including their architectural differences.

Reported step counts refer to denoising steps, not CFG-adjusted NFEs. Explicit CFG requires two model evaluations per step (e.g., 4 steps correspond to 8 forward evaluations). We updated Table 8 and 9 headers and the abstract from “NFE” (previously incorrectly used, the values reported are already denoising steps) to “steps” to match the convention held throughout the paper.

Regarding inference cost, all iDDM evaluations sample only one particle per denoising step, therefore there is no multi-particle overhead at inference time. Combined with the chosen ξ conditioning mechanism (fixed-inner-width) that preserves the attention and FFN dimensions, the results in near-identical inference cost compared to FM under matched sampling steps and CFG settings (~0.96x FM throughput, measured on DiT-B/2) .

Thank you for the additional experiments and clarifications. The retuned constant-parameter comparisons and compute-matched transfer results strengthen the empirical contribution. I also appreciate the clearer distinction between stochastic conditional prediction and calibrated posterior recovery.

My remaining concerns mainly concern uncertainty around smaller ablation differences and the scope of the schedule and transfer claims, including the spatial-relation deficit. These limitations do not outweigh the demonstrated computational contribution and substantial few-step improvements.

Overall, I maintain my original rating of weak acceptance.

Reviewer_veFEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates improvements to the training of Distributional Diffusion Models (DDMs). DDMs employ stochastic denoisers that take auxiliary noise as an additional input to learn the conditional distribution p(x1∣xt)p(x_{1} | x_{t}), rather than only its conditional mean. The authors propose Deferred Population Expansion to reduce training costs and introduce time-dependent scoring rule schedules motivated by previously studied reverse diffusion dynamics. The proposed method achieves competitive performance in both 4-step and 50-step generation settings without requiring a pretrained teacher model or distillation.

Critical strengths
  • Improved training efficiency. The paper presents a simple yet effective approach to reducing the computational overhead of training conventional DDMs.

  • Extensive experimental validation. The paper provides comprehensive experimental evaluations and ablation studies that clearly illustrate the contributions of individual components.

Critical weaknesses
  • Lack of guarantees for accurate conditional distribution learning. It is unclear whether the proposed method can accurately recover the target conditional distribution. In particular, the use of time-dependent scoring rules and local scoring objectives raises questions about whether the learned distribution matches the true conditional distribution.

  • Lack of evaluation in the 1–2 step generation regime. The paper does not provide experimental results for 1-step or 2-step generation. Since these settings are important in many recent studies on fast generative modeling, evaluating the proposed method in these regimes would provide a more comprehensive assessment of its effectiveness.

  • Limited methodological novelty. The contributions of this paper appear somewhat incremental, as they primarily focus on architectural modifications and hyperparameter scheduling rather than introducing a fundamentally new learning framework or objective.

Questions for authors
  • What are the specific advantages of stochastic few-step generative models over deterministic alternatives? It would be helpful to clarify the benefits of stochasticity, particularly when deterministic methods achieve competitive or better generation quality.

  • Can the authors provide more general guidelines for hyperparameter selection? The proposed method involves several hyperparameters that appear to be determined primarily through empirical tuning. It would be valuable to provide a more systematic tuning strategy or practical guidelines that generalize across different datasets, model scales, and generation settings.

3 comments

Dear reviewer veFE,

Thank you for acknowledging our "simple yet effective approach to reducing the computational overhead of training conventional DDMs" supported by "extensive experimental validation".

Below we address your questions.

[W1] Lack of guarantees for accurate conditional distribution learning. It is unclear whether the proposed method can accurately recover the target conditional distribution. In particular, the use of time-dependent scoring rules and local scoring objectives raises questions about whether the learned distribution matches the true conditional distribution.

At the level of the objective, the property the reviewer asks for is available within our framework: with λ(t)\lambda(t) = 1 and β(t)∈�0�\beta(t) \in �0� the score is strictly proper at every noise level, i.e. its unique population optimum is the true conditional (for the per-token marginals under the local kernel, for the joint law under the global kernel). As for any diffusion objective, this is a statement about the optimum, not about the trained network. Our default departs from it deliberately, and both configurations are in the paper (Table 3b-ii).

Exact posterior recovery is however not our objective. If p(x1∣xt)p(x_1 \mid x_t) were learned exactly at every t, diffusion would be unnecessary: one could sample p(x1∣x0)p(x_1 \mid x_0) from Gaussian noise in a single step. We approximate the posterior only because it yields a better few-step approximation of the transition kernel.

p(xs∣xt)=∫p(xs∣xt,x1) p(x1∣xt) dx1p(x_s \mid x_t) = \int p(x_s \mid x_t, x_1)\, p(x_1 \mid x_t)\, \mathrm{d}x_1

than the Dirac at E[x1∣xt]\mathbb{E}[x_1 \mid x_t] used by standard diffusion. Our contribution improves this approximation, it does not claim to recover the posterior.

Regarding the two specific concerns:

(i) In principle, time dependence does not necessarily affect propriety. As mentioned above, for λ(t)\lambda(t) = 1 and β(t)∈�1�\beta(t) \in �1�, the score remains strictly proper at each timestep. The posterior variance bias introduced by λ(t)<1\lambda(t) < 1 is deliberate, as discussed in Section 3.2 and analytically motivated in Appendix D. This is not specific to iDDM: the original DDM already uses λ<1\lambda<1, so our scheduling can inherit this trade-off and does not introduce it.
(ii) Local scoring only guarantees per-position marginal recovery in the strictly proper regime. but does not imply spatial independence. With λ(t)<1\lambda(t)<1, accurate marginal recovery is not guaranteed.

More broadly, our approach is grounded in theory, here strictly proper scoring rules, but we depart from it deliberately where this improves generation in practice (see our response on novelty for precedents of this type of contribution). Our central claims are an empirically validated improvement in few-step generation through stochastic conditional modeling and do not rely on posterior calibration. We clarified this distinction in the introduction (lines 71-72, 88-90) and in Section 4.2 (lines 412-413).

[W2] Lack of evaluation in the 1–2 step generation regime. The paper does not provide experimental results for 1-step or 2-step generation. Since these settings are important in many recent studies on fast generative modeling, evaluating the proposed method in these regimes would provide a more comprehensive assessment of its effectiveness.

We agree that 1-2 step generation is also an important research direction. However, our work targets a complementary operating regime: achieving competitive few-step generation while retaining the ability to improve quality with additional sampling steps. This flexibility is not guaranteed by methods optimized for the 1-2 step regime (Table 4, MeanFlow and MFM all budgets, DDM (naïve) only at 50 steps), some of which degrade as the sampling budget increases. Our focus is on the 4-50-step range, where iDDM offers this flexibility from a single checkpoint trained from scratch, without distillation. We explicitly acknowledge the limitations at 1-2 steps in Section 4.2 and in the Limitations section and do not claim competitiveness in that regime. We report 1 step and 2 step performance in an additional table (Table 10) in the Appendix B.1 to provide a more complete characterization of its performance outside its intended operating regime. Future work will focus on understanding and improving the performance in the 1-2 step regime, but it’s beyond the scope of this work.

AuthorsOpenReview ↗

Response (2)

Limited methodological novelty. The contributions of this paper appear somewhat incremental, as they primarily focus on architectural modifications and hyperparameter scheduling rather than introducing a fundamentally new learning framework or objective.

Our contribution is to show that distributional diffusion is competitive with the current state of the art, by analyzing and addressing the limitations of the original method. This type of contribution has clear precedents: EDM [Karras et al., NeurIPS 2022, Outstanding Paper Award], EDM2 [Karras et al., CVPR 2024], improved consistency training [Song & Dhariwal, ICLR 2024] and Improved MeanFlow [Geng et al., CVPR 2026] all start from an existing formulation and show that revisiting the objective, architecture or training procedure substantially changes its effectiveness and scalability. iDDM is such a refinement of DDM, aimed at making distributional denoising practical at DiT scale. We believe that this type of contribution is of interest for the ML community as we explore the scalability of DDMs. In that respect, we believe our contributions should be interpreted in this light. We recall our two main contributions below and explain why we consider these are important improvements over the original DDM algorithm.

We motivate below the novelty of our contributions:

Deferred expansion is both an efficiency mechanism and a tool to investigate where and how stochasticity should enter the network to separate shared and particle-specific processing. Without it, transferring the original DDM recipe to a transformer backbone performs substantially worse (Table 4). It differs from conventional multi-head prediction as the particles are not produced by independently parameterized heads, but are samples from a single shared stochastic denoiser.

Scheduling λ(t)\lambda(t) and β(t)\beta(t) is a change of objective, not a hyperparameter. The original DDM trains with one fixed scoring rule at all noise levels, iDDM trains with a different generalized scoring rule at each diffusion time, so the trade-off between conditional spread and mean-like prediction varies along the trajectory. The shape of the schedule is motivated by how the conditional posterior p(x1∣xt)p(x_1 \mid x_t) evolves across the dynamical regimes of the reverse process (Sec. 3.2, App. D), which determines where a distributional prediction matters and where a mean-like one suffices.

[Q1] What are the specific advantages of stochastic few-step generative models over deterministic alternatives?

Unlike deterministic conditional-mean predictors, stochastic denoisers can represent multiple plausible outcomes for the same noisy state, enabling conditional uncertainty estimation and ensemble prediction without relying solely on variation in initial conditions. This is particularly valuable when multiple plausible outcomes can arise from the same observation. In biology, aggregated gene-expression measurements can correspond to multiple distinct single-cell profiles [1], while in weather forecasting, stochastic ensembles capture alternative atmospheric evolutions and their associated uncertainties [2, 3]. Importantly, these works already demonstrate successful applications of DDM-based methods beyond image generation, motivating further improvements to this modeling framework. Our goal is therefore not on demonstrating these downstream applications, but on making stochastic conditional modeling computationally efficient at scale.

[1] Fishman et al., 2026, Count bridges enable modeling and deconvolving transcriptomic data.
[2] Wiese et al., 2026, Just Weather Scoring: Efficient End-to-end Nowcasting with Distributional Diffusion.
[3] Park et al., 2026, Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecasting

[Q2] Can the authors provide more general guidelines for hyperparameter selection? The proposed method involves several hyperparameters that appear to be determined primarily through empirical tuning.

While it’s true that we extensively ablate the proposed schedules that in principle are data dependent (as Dynamical Regimes are data dependent, Biroli et al. 2024), we also demonstrate that the practical approach allows to simply transfer the linear schedule to a 1.6B text-to-image generation model (different dataset, different model scale, different generation setting) without re-tuning, making the recipe simple and practical. As a general rule of thumb we therefore recommend the linear schedule, ξ conditioning via concatenation, and a small sweep over depth on late layers if compute permits. We added this recommendation in the updated PDF (lines 388-389) and reported below:

“As a practical recommendation, we suggest using the linear schedule, ξ conditioning via concatenation, and only a few distributional layers, optionally sweeping the expansion depth if compute permits.”

Most of my concerns have been addressed, and I am cautiously raising my score to 3. I acknowledge the methodological refinements and empirical improvements. However, my concern about the limited theoretical novelty relative to the original DDM framework remains. The rebuttal clarifies the theoretical motivation and scope of the method, but I still view its principal contribution as methodological and empirical rather than a substantial theoretical advance.

Reviewer_cEVdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper improves the cost and performance of distributional diffusion models. It shares transformer computation across particles and introduces independent noise in later layers, then adjusts the scoring-rule parameters and time sampling used in training. The experiments show stronger few-step generation than the original DDM recipe, including comparisons at matched training compute. A single checkpoint supports several sampling budgets, with additional text-to-image results.

Critical strengths
  1. Sharing the early computation addresses a clear cost of particle-based training. The main ablations examine both expansion depth and noise conditioning, so the paper gives some explanation for why late expansion works rather than only reporting the final model.
  2. The few-step gains remain in the main paper's comparison at matched training compute. The larger model and text-to-image results also suggest that the recipe is useful beyond the smaller ablations. The paper reports the remaining disadvantage against flow matching at a longer sampling budget.

DDM already introduced distributional denoising with scoring rules. The contribution here is making that approach easier to train at scale and improving its generation results.

Critical weaknesses
  1. The posterior interpretation needs qualification. Section 3.2 explains that using lambda below one shrinks variance, and the proposed schedule uses this deliberately. The trained distribution should therefore not be described simply as recovering the true conditional posterior. This does not undermine the reported generation gains, but the introduction should distinguish learning a distribution from accurately estimating that posterior.
  2. The architecture needs a clearer comparison with DISCO Nets, which already introduced noise after deterministic feature extraction. The paper cites this work for its loss, but should also discuss the architectural connection. The evidence here supports the transformer implementation, shared particle computation, and training recipe; introducing stochasticity late in a network is an existing idea.

The introduction should also mention methods that improve a pretrained model by optimizing its inference configuration. Examples include LD3, Align Your Steps, Differentiable Diffusion Sampler Search, and S4S. These optimize timesteps or sampler parameters while keeping the denoiser fixed. Please cite representative work alongside the discussion of solvers, distillation, and few-step training.

Equation (5) includes both an integral over time and an expectation over the time-sampling distribution. These appear to be two ways of writing the same average. If gθ(t)g_\theta(t) denotes the expected score at a fixed time, the intended expression seems to be

=-\int_0^1 p(t)w(t)g_\theta(t)\,dt.$$ Please use one form and clarify the weighting when the time-sampling density is nonuniform. There is also an unfinished attribution on page 10, lines 508–509: “We build upon existing evaluation pipelines from and and detail evaluation protocols…” Please name and cite the pipelines actually used. This is a concrete text defect, not evidence of AI authorship or fabricated results. I lean toward acceptance. The main results support a useful improvement to DDM training, although the posterior interpretation and novelty claims need to be more precise.
Questions for authors
  1. Can the introduction distinguish the conditional posterior from the variance-shrunk distribution favored by the proposed loss? Which claims require posterior accuracy, rather than good generated samples?
  2. What architectural changes distinguish deferred expansion from the late-noise design in DISCO Nets? Also, since guidance is tuned for the sampling budgets in the main experiments, does the same checkpoint retain its improving quality curve with one fixed guidance value?
3 comments

Dear reviewer cEVd

Thank you for highlighting that our contributions are "making easier to train at scale and improving its generation results" with a "recipe useful beyond the smaller ablations"

Below we address your questions.

[W1] The posterior interpretation needs qualification. Section 3.2 explains that using lambda below one shrinks variance, and the proposed schedule uses this deliberately. The trained distribution should therefore not be described simply as recovering the true conditional posterior. This does not undermine the reported generation gains, but the introduction should distinguish learning a distribution from accurately estimating that posterior.
[Q1] Can the introduction distinguish the conditional posterior from the variance-shrunk distribution favored by the proposed loss? Which claims require posterior accuracy, rather than good generated samples?

We agree and see how the proposed changes would make the qualifications even more explicit. We updated the introduction to reflect the even more explicit qualifications:

lines 71-72 “DDMs thus offer a non-adversarial alternative to learn conditional distributions beyond their means, while potentially allowing the learned predictions to be more concentrated than the true posterior”

lines 88-90 “Removing both bottlenecks makes distributional denoising practical with large transformer backbones while retaining the goal of approximating p(x1∣xt)p(x_1\mid x_t) rather than only its conditional mean, and permitting deliberate variance shrinkage at high SNR”

Additionally, none of our central claims requires accurate or calibrated posterior recovery: our conclusions are mainly grounded in empiricism and concern training efficiency, generation quality across sampling budgets, and conditional diversity, rather than posterior calibration. To further prevent non-intended interpretation, we modified the positioning paragraph in Section 4.2:

lines 412-413 “(i)~It is stochastic: it learns a stochastic conditional predictor trained to sample from an approximate p(x1∣xt)p(x_1 \mid x_t), whereas MeanFlow, iMF, and IMM learn deterministic maps.”

Changes are available in the new PDF version and are highlighted in blue.

[W2] The architecture needs a clearer comparison with DISCO Nets, which already introduced noise after deterministic feature extraction. The paper cites this work for its loss, but should also discuss the architectural connection. The evidence here supports the transformer implementation, shared particle computation, and training recipe; introducing stochasticity late in a network is an existing idea.
[Q2] What architectural changes distinguish deferred expansion from the late-noise design in DISCO Nets?

The two approaches share the idea of sharing computation first before having separate per-particle stochastic prediction but differ in realization. DISCO Nets implement the idea via a heterogeneous architecture composed of a convolutional feature extraction and an MLP prediction head. We instead use a homogeneous transformer backbone that allows us to vary the boundary between shared processing and particle-specific prediction at arbitrary depth without modifying the underlying architecture or introducing a separate prediction head. We make use of this flexibility to (1) improve computational cost and (2) more importantly, show that both where particle-specific computation begins and how noise is injected matter a lot in practice. For the where in Figure 4 shows that the number of distributional layers strongly affects generation quality and optimal guidance. For the how, DISCO Nets and DDM use concatenation at the separate prediction head and network input, respectively, while FGN [2] injects in normalization layers. Although all these mechanisms (and more in Table 2b) are reasonable a priori, their effectiveness varies substantially in practice (Table 2b). These were previously unexplored degrees of freedom in DISCO Nets, DDM, and FGN.

We agree that the conceptual relationship with DISCO Nets should be highlighted more and made explicit so we have added a paragraph in the related works and a connection to DISCO Nets in the method section.

[1] Diane Bouchacourt et al., 2016, DISCO Nets: DISsimilarity COefficient Networks.
[2] Ferran Alet et al., 2025, Skillful joint probabilistic weather forecasting from marginals

AuthorsOpenReview ↗

Response (2)

[Q2] Also, since guidance is tuned for the sampling budgets in the main experiments, does the same checkpoint retain its improving quality curve with one fixed guidance value?

Not for every fixed value, but that’s not the property we are interested in. Across all models, FID generally follows a U-shaped curve as a function of CFG with optimal guidance strength, with fewer steps favoring stronger guidance and more steps favoring weaker guidance strength. So, similarly to the published methods we compare to, keeping CFG constant across sampling budgets does not produce the optimal results for that budget. While fixing CFG at the 50 steps optimum yields monotonically improving FID with increasing step counts (as it would be increasingly more fit for the step count), it would not reflect the best achievable quality at 4 steps. With guidance fixed at the 4 steps optimum, 50 steps FID is worse than 4 steps FID, because that strength over-guides once discretization error is small.

Our claim concerns the standard protocol [1, 2, 3] in which guidance is selected per sampling budget (which we apply to all methods in Table 4). Even under individually tuned CFG per sampling budget, some methods don’t yield gains when more compute is available, while iDDM does. We updated Section 4.2 (lines 418-419) to make the point even more explicit.

lines 418-419 “(iii)~A single checkpoint serves the full 4-50 step range without degrading: its FID is non-increasing in the number of steps (4.48 2.38), whereas MeanFlow (2.93 3.29) and MFM (1.97 5.74) are stronger at 4-8 steps but get worse with additional sampling compute, even with guidance optimized separately for each sampling budget, following established evaluation practice [1, 2, 3]”

[1] Jonathan Ho and Tim Salimans, 2022, Classifier-Free Diffusion Guidance.
[2] Prafulla Dhariwal and Alexander Quinn Nichol, 2021, Diffusion Models Beat GANs on Image Synthesis.
[3] Zhengyang Geng et al., 2026, Improved mean flows: On the challenges of fastforward generative models

The introduction should also mention methods that improve a pretrained model by optimizing its inference configuration

We agree that these methods should also be included in the presentation. We updated the introduction (lines 60-61) and the related work section (lines 448-449). Our contribution concerns training objectives and architectures, while inference optimization is applied on a frozen pretrained model. Inference optimization is orthogonal to our contributions and potentially applicable to iDDM, therefore we don’t compare these approaches directly.

Changes are available in the new PDF version and are highlighted in blue and reported below.

lines 60-61 “[...] improving the numerical solver (Karras et al., 2022; Lu et al., 2022; 2025; Zheng et al., 2023) or learning one (Tong et al., 2025; Frankel et al., 2025; Sabour et al., 2024; Watson et al., 2022), by distilling [...]”

lines 449-450 “improved numerical samplers [...] and learned samplers (Tong et al., 2025; Frankel et al., 2025; Sabour et al., 2024; Watson et al., 2022) reduce discretization error of a frozen model [...]”

Equation (5) and Please use one form and clarify the weighting when the time-sampling density is nonuniform.

We corrected Eq. (5) using the proposed formulation. In all experiments, we use uniform loss weighting w(t)=1w(t)=1 (Table 5 and Table 6), while varying the time-sampling distribution p(t)p(t) in the train t sampling ablation.

unfinished attribution on page 10

We updated the PDF to fill in the missing details. Below is the complete sentence for reference:

“We build upon existing evaluation pipelines from ADM [Dhariwal & Nichol, 2021] using torch-fidelity [Obukhov et al., 2020] for ImageNet experiments and detail evaluation protocols and metrics in Appendix B.1, while details for Text-to-Image experiments are available in Appendix B.2”

Reviewer_cEVdOpenReview ↗

Response to rebuttal

Thanks for the clarifications. The revised posterior interpretation and discussion of DISCO Nets address my main concerns. The explanation of guidance tuning across sampling budgets answers my question, and the reported corrections address my remaining comments. I maintain my weak acceptance recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.