acceptodds
Under review as a conference paper at ICLR 2027

Depth as Time in One-Step Generative Models

Abstract

The recent wave of one-step generative models, which compress the multi-step trajectory of diffusion via either distillation or learned flow maps, has reached an inflection point where they can generate high-quality images. Here, we ask a natural question that follows from these advances: what happens to the denoising trajectory of multi-step diffusion when generation is compressed into a single forward pass? We offer an empirical observation we call depth as time: the denoising computation that multi-step diffusion performs across sampling steps appears to unfold across the depth of a single forward pass, and can be recovered by decoding intermediate layers with the model’s own output head. Most interestingly, we show that this depthwise computation depends on the transport task a flow map is trained to solve. The most surprising case is MeanFlow, where probing shorter transport intervals reveals both denoising and renoising within a single network evaluation. In contrast, In contrast, generators trained without a time-indexed transport task, such as drifting models, do not exhibit the same depthwise denoising. Consequently, we show that models that exhibit the depthwise denoising phenomenon are more compressible across the layerwise computation: a MeanFlow SiT-L/2 model can be compressed by 16.6× in parameters into a single time-conditioned block. Together, these results suggest that the temporal computation of diffusion is not eliminated by one-step generation, but reorganized across network depth.

Primary area:
generative models
Keywords:
One-step generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xYGeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an intriguing empirical observation that the denoising dynamics typically distributed across multiple sampling steps may instead emerge across network depth in one-step generative models. The authors introduce the concept of depth as time, showing that progressive denoising can be observed across network layers by decoding intermediate representations with the model's output head. They further demonstrate that this behavior depends on the transport objective, with MeanFlow exhibiting denoising followed by renoising. Motivated by these observations, they propose a time-conditioned weight-sharing architecture that achieves substantial parameter compression while maintaining competitive generation quality.

Critical strengths
  1. Novel and insightful empirical observation regarding the internal computation of one-step generative models. The central observation that denoising-like computation progressively emerges across network depth is interesting and potentially important.

  2. Empirical analysis across different generative objectives. The comparison among time-indexed flow-map models and Drifting Models is valuable because it suggests that depthwise denoising may be associated with the training objective rather than simply being an inherent property of deep neural networks.

  3. The work connects an interpretability finding to a concrete practical application. The reported 16.6× parameter compression of MeanFlow SiT-L/2, from approximately 459M to 27.6M parameters, is substantial.

Critical weaknesses
  1. The authors attribute the observed denoise-then-renoise phenomenon to the interval-conditioned transport task learned by MeanFlow. However, the paper does not establish whether a conventional multi-step flow-matching model exhibits similar non-monotonic depthwise behavior, which is an important baseline.

  2. The evidence does not conclusively establish that depthwise representations follow an actual denoising trajectory.

  3. The compression results do not establish a clear computational advantage over conventional architectures. The experimental evaluation should report wall-clock latency, peak memory usage, FLOPs, and quality under matched inference compute budgets.

Questions for authors

Q1: How can the authors rule out generic hierarchical refinement as an alternative explanation?

Q2: If the depthwise denoising pattern is deliberately disrupted or strengthened during training, how does this affect the compressibility of the resulting model?

Reviewer_mZrROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks a central question: when one-step generative models remove explicit multi-step denoising, does the denoising process disappear, or does it move inside the network? The authors’ empirical findings suggest that it may be reorganized along network depth, even showing a “denoise, then re-noise” pattern—a phenomenon they call “depth as time.” Using this structure, they reduce MeanFlow’s parameter count by a factor of approximately 16.6.

Critical strengths
  1. Interesting work with insightful findings. Moving the diffusion process into network depth may reveal a connection between looped models and diffusion models. It is also interesting that some one-step generators exhibit layer-wise refinement related to the transport task, which can be read out by the output head and used for parameter sharing.
  2. The paper demonstrates practical value. With a 16.6× reduction in parameters, additional post-training improves FID from 11.7 after compression to 4.9, close to the original model’s 4.0. This shows the practical potential of shared modules.
Critical weaknesses
  1. The causal evidence is not strong enough. The poorer compression results for drifting models do not establish that the cause is the lack of time-indexed tasks. The benefits of an internal clock could also reflect a general advantage of explicit stage encoding.
  2. Parameter efficiency and computational efficiency should be evaluated separately. A shared module still needs to run repeatedly, so a 16.6× parameter reduction does not imply a comparable speedup. Achieving quality close to the original model also requires additional post-training.
Questions for authors
  1. Does your observation suggest that distilling a flow model into a one-step model essentially turns it into a looped model, and that adding time conditioning and post-training with the FD loss turns it back into a flow model?
  2. The paper shows a non-monotonic “denoise, then re-noise” pattern, but later maps increasing layer indices to monotonically decreasing noise timesteps. Is there a mismatch in this design? How exactly are the two related?
  3. On CIFAR-10, removing the clock increases FID from 21.27 to 45.45, showing that explicit stage information is very helpful in this setting. But I’m curious: how does this clock relate to diffusion time? Are they completely unrelated, or could we connect the two? I will consider raising my score if all my concerns are addressed.
Reviewer_3QzuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about the phenomenon of “depth as time” which is an observation that one step generative models (distilled/fine-tuned from diffusion models) do not erase multi-step denoising process of diffusion models, instead they do it across their internal layers. This phenomenon seems to be tied to both the architecture and conditioning. For architecture, this probably works with residual transformer architectures such as DiT and SiT. In addition, time-conditioning is necessary. Motivated by this, the paper proposes a recurrent trasnformer architecture that uses the exact same network weights across the depth. Through this, the paper was able to compress Meanflow-SiT architecture by 16X. The original “depth as time” has been investigated on a toy Gaussian dataset, CIFAR10, ImageNet-256, and on larger architectures such as SANA and Flux. In addition, the parameter compression has been empirically verified on Meanflow SiT-B/2 and SiT-L/2 architectures.

Critical strengths
  1. The motivation is interesting as it investigates the internal workings of one-step generators that are fine-tuned or distilled from diffusion models.
  2. The paper shows that in diffusion transformer based architectures, we can use the network’s own output head to get predictions.
  3. The proposed weight-tied architecture (Meanflow-SiT) allows the depth to be treated as an inference time block.
Critical weaknesses
  1. The qualitative observation in 8-Gaussians (Figure 1) is probably a geometric consequence of small incremental updates in the MLP and not a proof that the student has internalized the denoising trajectory of the teacher. Any smooth residual network will transport a Gaussian distribution outward to 8 individual gaussians with “connections” similar to the figure in the paper.
  2. The paper uses L2 distance instead of more reliable distributional metrics to evaluate convergence. The standard pixel/latent-space L2L_2 distance only measures sample wise Euclidean proximity to the final prediction x~0L\tilde{x}_0^L. In image space, L2L_2 is dominated by low-frequency background color, and does not capture sharpness. In case of 8-Gaussians, it would have been more meaningful to compute KL divergence or Wasserstein distance.
  3. In appendix, it is mentioned that for fitting the scalars αt,σt\alpha_t, \sigma_t, the paper uses 179 training and 77 held out trajectories. This won’t capture all 1000 ImageNet classes, and is also small for open-vocabulary T2I models.
  4. The quality of images in the visualizations is poor. In Figure 7b, calling the blurry intermediate blob a "cleaner image structure" that gets "renoised" overstates its quality.
  5. In line 371 the paper claims that that FLUX-schnell's reversal in x~ri\tilde{x}_r^i is "consistent with the model returning toward the prescribed noisy endpoint" which is mathematically impossible, because FLUX-schnell isn't even conditioned on rr. Therefore, the network does not know what the "prescribed noisy endpoint" rr is.
  6. In Section 4, the paper considers the task of partial transport and observes “denoise-then-renoise” pattern in case of Meanflow models as shown in Figure 7. This pattern can be explained as a simple scaling artifact of the proposed readout formula x~ri=xt−(t−r)vi\tilde{x}_r^i = x_t - (t - r)v_i. $\tilde{x}_r^i $ appears cleaner at intermediate blocks when uiu_i has a magnitude that is t/(t−r)t/(t-r) larger than viv_i. As the magnitude of uiu_i decreases down closer to 1 at the later layers, (t−r)ui(t - r)u_i stops over-cancelling xtx_t and the noise component appears.
  7. There is a potential confounder in Section 3.2 from the type of network parameterization (velocity vs. clean image). In Equation 3, the layerwise endpoint prediction for velocity-parameterized models is defined as x~ri=xt−(t−r)vi\tilde{x}_r^i = x_t - (t - r)v_i. For full one-step generation from pure noise to a clean image (t=1,r=0t = 1, r = 0), this becomes x~0i=ϵ−vi\tilde{x}_0^i = \epsilon - v_i. This parameterization explicitly adds +1.0ϵ+1.0\epsilon outside the network at every block ii, and requires the output head WW to project out ϵ\epsilon at the final block LL (since vL=ϵ−x~0Lv_L = \epsilon - \tilde{x}_0^L). In the case of drifting models, the output head WW directly outputs the clean image x~0L\tilde{x}_0^L without subtracting from ϵ\epsilon. Thus, the output head in drifting models is trained to filter out ϵ\epsilon and read out only semantic image features. Consequently, when the output head is applied to intermediate states hih_i in drifting, its weights are orthogonal to ϵ\epsilon, which results in σ^i≈0\hat{\sigma}_i \approx 0 across all blocks. To prove that σ^i≠0\hat{\sigma}_i \neq 0 arises from task conditioning (τ\tau) rather than velocity parameterization, the paper should evaluate an x0x_0-parameterized flow/consistency model (or a velocity/displacement-parameterized drifting model).
Questions for authors

I hope the authors can address the concerns above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.