acceptodds
Withdrawn from ICLR 2027

BEYOND VARIANCE MATCHING: DATA-DEPENDENT LAYERWISE STARTS FOR DIFFUSION NETWORKS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion networks are almost always started from variance-preserving random draws such as Xavier or He initialization and then refined by an optimizer. These initializers stabilize activation and gradient scales, yet they ignore the training dis- tribution: the first gradient steps must discover task structure that a cheap linear algebra pass could already suggest. We ask whether a data-dependent start can place ε-prediction diffusion denoisers at better weights that reduce subsequent training cost. We introduce Layerwise Closed-Form initialization (LCF), which visits each affine map in depth order, fits a throwaway linear probe to form a residual-augmented target, takes a single exact line-search gradient step from the layer’s default random draw toward that target, restores the true nonlinear opera- tor, and proceeds to the next layer. Each hidden weight is thus fit by least squares on the post-activation features of the layers below it, rather than obtained by fac- torizing one linear map across depth. On 12 diffusion datasets × 5 denoisers, LCF improves step-0 validation error on 60/60 settings; on convolutional denoisers it also yields positive net optimizer savings after subtracting its own cost, on 12/12 large diffusion CNNs (mean +19.7 net steps) and 6/6 base CNNs on grayscale sets (mean +33.5). At this loose target a zero-initialized output layer matches these savings at no cost; at targets within 1.1–1.5× of the converged loss, LCF saves more net steps than it on 9/12 datasets. These results support a training-cost savings claim for diffusion CNNs: a cheap closed-form start can replace optimizer steps.

Primary area:
optimization
Keywords:
warm start, greedy layerwise fitting, closed-form initialization, data-dependent initialization, diffusion networks, training efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_y7DUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Layerwise Closed-Form initialization (LCF) for diffusion denoisers, using linear probes, residual-augmented targets, and layerwise least-squares updates. The authors evaluate initialization quality and training efficiency across multiple datasets and architectures. LCF consistently improves initial validation loss, while positive net compute savings are mainly observed for small convolutional denoisers.

Critical strengths

1.The paper addresses a relevant problem and evaluates initialization cost rather than relying solely on initial loss. The FLOP-adjusted steps-to-target metric is a useful aspect of the evaluation.

2.The authors include competitive initialization baselines, a zero-initialized output-head control, and longer convergence experiments. The negative results on other architectures help clarify the method's limitations.

Critical weaknesses

1.Firstly, the source of the improvement is not clear. In Table 4, when the residual strength α is set to zero, the earlier calculation results are exactly the same. This makes me question the necessity of the proposed residual detection mechanism. Moreover, the paper fails to clearly distinguish the contributions from the output head fitting, the update of the hidden layer, and the scaling of the weight.

2.Then, the claim about efficiency lacks sufficient support. LCF and the main baseline use different initial weight distributions, making the comparison difficult to explain. The positive computational results mainly focus on small-scale diffusion CNNs, while the results of U-Net are negative. The reported savings effect is based on the estimated number of floating-point operations (FLOPs), rather than the actual measured training time. It is still unclear whether this method has practical advantages under controlled comparison conditions.

Questions for authors

1.Can the authors report results using identical initial weight tensors for LCF and its baselines, together with separate ablations of the residual probe, hidden-layer updates, and output-head fitting?

2.Does LCF reduce actual end-to-end GPU training time, including initialization overhead? Can the authors provide this comparison for at least one standard diffusion U-Net?

Reviewer_NwZkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a weight initialization for diffusion generative models. They claim that by initializing weights in a data-informed way, instead of random initialization, the diffusion model can be trained in fewer iterations. To support that claim, the authors train models with and without the proposed initialization and measure how fast they reach a specified target metric. The results show limited success and fail to prove that the proposed initialization has meaningful differences against randomly chosen weights.

Critical strengths

None

Critical weaknesses
  • The main issue with the paper is how it evaluates the models. The authors look at the mean-squared denoising error on a validation set to judge whether a model has reached similar denoising performance in fewer steps using their initialization. However, MSE is rarely used as a metric to judge the performance of diffusion models. The more common choices are Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), which measure whether the distribution of samples produced by the model matches the real target distribution. Without such distributional metrics, it is hard to judge whether there are differences between the model with and without the proposed initialization. The authors need to rerun their experiments and evaluate at least one such metric to complement the MSE results and draw conclusions.

  • Even with this incomplete evaluation, the method is shown to not work on a practical diffusion architecture, U-Nets. The go-to approach for image generation is the U-Net, and the MSE results show that the method does not improve the convergence of U-Net-based diffusion models. The rest of the experiments in the paper focus on simple CNNs, which are rarely used in image generation tasks. The authors also did not test their initialization on more recent diffusion architectures (Diffusion Transformers). Thus, the claims made about speeding up training are specific to the CNN models the authors made it work on and do not seem to have the potential to transfer to practical settings.

  • The classification and language-modeling experiments in Section 5.3 are unnecessary. The authors discuss that the proposed method does not work on these two additional settings, for which they give a vague explanation in Remark 2. The inclusion of this result does not need to be in the main text.

  • The references seem to be using the wrong format. The authors should review the submission guidelines and use the correct format.

Questions for authors
  • Is the net metric comparable across models, given that it takes into account FLOPs? A more expected metric would be a speedup, measuring how many times faster training becomes to achieve the same result using the proposed initialization. Also, what is the unit of measure for net? Does a +25 in net just translate to 25 fewer optimization steps? What is a meaningful range where net can be considered to be significant?
Reviewer_a3L5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Layerwise Closed-Form initialization (LCF), a data-dependent initialization procedure for epsilon-prediction diffusion networks. Before ordinary training, LCF uses an initialization batch to construct residual-augmented local targets, updates each affine layer using a ridge solve or one line-search gradient step, and fits the output head to the noise targets. The paper evaluates whether the reduction in steps-to-target exceeds the estimated FLOP cost of initialization. LCF consistently improves step-0 MSE, but positive net gains are concentrated on small diffusion CNNs and are not observed for the tested U-Nets or most other architectures.

Critical strengths

The paper studies a useful question that is often overlooked in initialization work: whether a better initial loss actually reduces total training cost after accounting for the cost of the initializer.

The proposed layerwise procedure is conceptually interesting and is accompanied by elementary analyses of the ridge solution, the line-search update, and the residual target. The paper also reports several negative results and includes LSUV and Zero-Init as meaningful baselines.

The empirical study covers multiple datasets, architectures, seeds, ablations, and long-horizon convergence runs. In particular, the authors acknowledge that Zero-Init matches LCF at the loose 0.85 target and appropriately restrict their strongest positive claim to diffusion CNNs.

Critical weaknesses
  1. The experiments do not establish that LCF's specific layerwise construction drives the gains. LCF uses 512 of 1,024 training examples and their noise targets to update the network before “step 0,” but is not compared with compute-matched end-to-end warm-up or head-only regression. Moreover, on the three Table 4 datasets, alpha = 0, 0.25, and 0.5 give identical net steps at the 0.85 target, although alpha = 0 removes the residual augmentation. The improvement may therefore come from supervised optimization before training, particularly fitting the output head, rather than from the proposed residual-guided mechanism.

  2. The evidence does not support broad applicability to modern diffusion architectures. All tested denoisers omit normalization, and positive net gains at the common 0.85 target concentrate on simple CNNs: LCF obtains 0/12 wins on both U-Net families and only 1/12 on diffusion MLPs. The tighter-target experiments that favor LCF are then restricted to the small CNNs. Without evaluation on a modern residual U-Net or DiT with normalization and attention, the practical scope of the method remains unclear.

  3. The baseline protocols are not fully matched. LCF starts from PyTorch's default initialization, whereas Xavier and Zero-Init use different Xavier/Kaiming distributions, but no default-initialization-only control is reported. The common learning rate is selected using Xavier, LSUV, and LCF on five large-CNN datasets rather than tuned for every method, and LCF additionally receives per-matrix spectral-norm learning-rate scaling. The latter is ablated on only three datasets. Since the main metric is steps-to-target, these differences can directly affect the reported ranking.

  4. The training-cost claim relies on analytic FLOP accounting rather than measured system cost. LCF requires Gram matrices, matrix solves, im2col operations, power iteration, and layerwise feature storage, whose wall-clock and memory costs need not be proportional to convolutional-training FLOPs. Without end-to-end runtime and peak-memory measurements, it is unclear whether the reported net-step savings translate into practical savings.

  5. The evaluation does not yet establish faster improvement in actual generative quality. The original 0.85 MSE target is loose enough that Zero-Init matches LCF, while the stronger result relies on plateau-relative targets introduced after observing this outcome and evaluated only on the CNNs. These runs measure interpolated time-to-noise-MSE rather than FID or another sampling-quality metric. Consequently, the experiments show faster attainment of selected denoising-MSE thresholds, but not necessarily faster diffusion-model training in the practical generative sense.

Questions for authors
  1. Under the same initialization batch and FLOP budget, how does LCF compare with end-to-end full-batch GD/Adam warm-up, a line-search step on the actual diffusion loss, and head-only ridge regression?

  2. What are the results for a PyTorch-default-only baseline, and can all methods be compared under matched initialization and separately tuned learning-rate protocols?

  3. Does LCF provide net savings at plateau-relative targets on a modern residual U-Net or DiT with normalization and attention?

  4. What are the measured initialization time, total wall-clock time, and peak memory relative to ordinary training and the strongest baselines?

  5. Does the earlier MSE reduction translate into improved FID or another generation-quality metric at matched compute, and are the conclusions stable across pre-specified target thresholds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.