acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Improving Diffusion Training via Cross-Timestep Task Affinity

Abstract

Non-uniform training schemes have been widely used to accelerate diffusion model training; however, most existing approaches rely on static strategies and often exhibit unstable performance. Empirical evidence suggests that prior static methods are built on the assumption that timestep importance remains constant throughout training, an assumption that is inherently limited because it fails to capture the dynamically evolving interdependencies among denoising tasks during optimization. To address this limitation, we introduce task affinity to quantify the optimization influences among denoising tasks at different timesteps. By further relating task dependencies to the training optimization state, an affinity-aware adaptive timestep sampling strategy is derived. This strategy dynamically adjusts timestep preferences and yields robust and consistent acceleration across a wide range of model architectures, model scales, and datasets.

Primary area:
generative models
Keywords:
Diffusion Models, Training Acceleration, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k8yNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a novel framework for non-uniform diffusion training, to accelerate convergence and the final performance. The paper proposes to understand and analyze the principle of non-uniform training from a multi-task learning perspective. It uses task affinity as a proxy to measure the importance of timesteps, which can achieve adaptive and dynamic timetep sampling based on the online task affinity. Experiments show it can improve the convergence and final performance on the reported models and benchmarks.

Critical strengths

Strengths

  1. The motivation is intriguing and reasonable. Non-uniform training for diffusion models is a long-standing direction. However, as stated in the paper, there is no clear consensus on which timesteps are truly important. This paper proposes to analyze and understanding it from a dynamic multi-task learning perspective.
  2. The evidence of task affinity supports the motivation, and helps us understand how non-uniform training schemes work and why they can be unstable.
  3. Finally, based on the task affinity, the paper proposes an effective dynamic timestep sampling strategy. The method can improve the convergence and final performance on the reported models and benchmarks.
Critical weaknesses

Weaknessses

  1. Lack of experiments on SoTA models and training paradigms. The training paradigms has shifted to flow-matching, while the paper only tests DDPM-like schedule. Besides, the involved models and datasets are relatively small, and the text-to-image models (at least on MSCOCO) is not used. One of the key contributions claimed in the paper is the generalization and stability of the proposed non-uniform method. Therefore, lack of these experiments may limit the practicability of the proposed method.

  2. The paper claims a positive relationship between task affinity and the training convergence. However, it is mainly supported by visual comparison rather than quantitative results. Including rigorous quantitative analysis would strengthen the paper.

  3. Although Figure 4 visualizes the task affinity changes during the different training iterations, the paper does not provide corresponding sampling probability curves to highlight the actual sampling difference between the proposed method and existing methods.

Questions for authors

Please see Weaknesses

Reviewer_izrQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the instability of existing non-uniform timestep sampling strategies for diffusion models. The authors introduce task affinity to quantify optimization interactions among denoising timestep groups and propose an adaptive sampling strategy that dynamically adjusts sampling probabilities based on these interactions.

Experiments across multiple datasets and U-Net and DiT architectures report improved FID and more consistent training acceleration compared to existing methods. Additional analyses examine robustness across model scales, noise schedules, and hyperparameter settings.

Critical strengths
  1. Insightful analysis of timestep interactions from a multi-task learning perspective.

The paper provides an interesting perspective on diffusion training by explicitly characterizing optimization interactions among denoising tasks at different timesteps. The identification of gain regions and their evolution throughout training offers a plausible explanation for the effectiveness and instability of existing timestep sampling strategies. This analysis provides a meaningful motivation for adaptive sampling rather than relying on fixed timestep preferences.

  1. Broad empirical evaluation across diverse settings.

The paper evaluates the proposed method across multiple datasets, diffusion architectures, model scales, and noise schedules. The inclusion of wall-clock training comparisons that account for affinity estimation overhead is particularly valuable for assessing practical training efficiency.

Critical weaknesses
  1. Insufficient ablation studies to validate the contribution of task affinity.

The ablations mainly focus on hyperparameters without isolating the contribution of task affinity. Without comparisons against simpler adaptive sampling strategies, it remains unclear whether the improvements arise from task affinity or adaptive sampling alone. This is a major concern, as it leaves the paper's central contribution insufficiently validated.

  1. Limited theoretical justification.

Eq. (9) does not directly follow from the objective in Eq. (8), while Proposition 3.5 bounds approximation error without guaranteeing affinity ranking preservation. These gaps weaken the theoretical justification for the proposed sampling strategy.

Questions for authors
  1. Can the authors provide comparisons with simpler adaptive sampling strategies, such as loss-based sampling or shuffled affinity scores, under matched computational budgets to isolate the contribution of task affinity?

  2. What justifies the softmax sampling rule in Eq. (9) given the objective in Eq. (8)? Additionally, how sensitive is the sampling strategy to potential ranking errors introduced by the affinity approximation in Proposition 3.5?

  3. Can the authors report results across multiple random seeds, including means and standard deviations, to assess the statistical reliability of the reported improvements? Some gains over the strongest baselines appear marginal, particularly on CIFAR-10 and MetFaces.

Reviewer_L8DqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates diffusion training from a multi-task learning perspective and introduces cross-timestep task affinity to measure how optimizing one timestep group affects the optimization of other groups. The authors further propose an affinity-aware adaptive timestep sampling strategy that estimates these interactions and adjusts the sampling distribution accordingly.

Critical strengths
  • The observation that cross-timestep optimization relationships evolve during training provides a useful perspective on why fixed timestep sampling or weighting strategies may not be consistently optimal.

  • The method is evaluated across multiple datasets and model architectures and generally demonstrates consistent improvements over the corresponding baseline.

Critical weaknesses
  • Limited novelty. Several key ingredients have been investigated by prior work: viewing diffusion timesteps as multi-task learning objectives, analyzing positive/negative transfer across timesteps, and dynamically adapting timestep sampling. The main novelty appears to be using the dynamic evolution of cross-timestep affinity as a unified explanation for static sampling instability and as the signal for adaptive sampling, but overall the methodological novelty seems incremental.

  • The proposed loss-based affinity may be closely related to gradient alignment. For a small one-step update, the change in the target-group loss induced by adding another group's gradient should be closely related to the gradient inner product gi⊤gjg_i^⊤ g_j. Therefore, it is unclear whether the proposed affinity captures substantially different information from a simpler and cheaper gradient-alignment signal. This is important because the proposed affinity estimation introduces non-trivial computational overhead.

  • The reliability and computational cost of affinity estimation. The paper reports approximately 13.12% overhead under its default setting and also notes that stable affinity estimation may require large-batch computation. However, it does not clearly specify the amount of data used for each virtual one-step update or the subsequent loss evaluation.

  • The O(K2)O(K^2) pairwise affinity computation may be unnecessary for adaptive sampling. The proposed method constructs pairwise inter-group affinities using Eq. (4), whose full computation scales quadratically with the number of timestep groups. However, the final sampler only requires an aggregated group-to-context score. According to Eq. (5), (Gi,Bi)(G_i,B_i) contains all groups and therefore can be shared across all ii, while only KK leave-one-group-out configurations are required for the denominator. It would be valuable to investigate whether directly estimating Eq. (5) can substantially reduce the computational cost while retaining similar performance.

  • Large-scale validation against strong baselines remains limited. The paper includes ImageNet-1K experiments, but the large-scale results do not provide comparisons against strong static timestep strategies such as Min-SNR and SpeeD at the same level of comprehensiveness as the smaller-scale experiments. Given the additional cost of affinity estimation, a compute-matched comparison with strong baselines on a larger model would considerably strengthen the practical claims.

Questions for authors
  • Comparison with gradient-based affinity. Since the proposed metric should be closely related to gi⊤gjg_i^\top g_j​ under a first-order approximation, could the authors compare against an adaptive sampler that directly uses gradient inner products or cosine similarity between timestep groups? This would clarify whether the proposed loss-based affinity provides additional information that justifies its computational cost.

  • Stability versus batch size. Could the authors report the stability of affinity estimation as a function of batch size? Please also clarify whether the same samples are used for the gradient update and loss estimation.

  • Direct estimation of Eq. (5). Have the authors considered directly estimating the group-to-context affinity in Eq. (5)? Since the all-group update can be shared, could such a formulation reduce the practical complexity toward O(K)O(K) group configurations? How would its performance and estimation variance compare with the current implementation?

  • Larger-scale evaluation. Could the authors provide experiments on ImageNet-1K using larger models, with compute-matched comparisons against strong baselines such as Min-SNR and SpeeD?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time in evaluating this paper. Since there is only one day left for this stage, could you please read the rebuttal and provide a response? This is important for the review process.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.