acceptodds
Under review as a conference paper at ICLR 2027

The Noise-Transmission Burden of Diffusion Network: Why Image Prediction Can Outperform Noise Prediction

Abstract

Clean-image prediction (-prediction) and noise prediction yield equivalent population-optimal denoisers, yet can perform very differently in finite diffusion networks. We relate this gap to a noise-transmission burden: forming a clean-image estimate from a noise predictor requires cancellation of corruption carried by the input. Under Gaussian corruption, a rank- linear encoder imposes a denoising-error floor for bare noise prediction even with unrestricted downstream computation, while image-prediction risk remains bounded. Analytic output preconditioning cancels this contribution and yields an image-prediction-equivalent learned branch; EDM suppresses it at high noise, with its learned target approaching a scaled clean image. Without input compression, and assuming exact affine recovery of the noisy patch, we characterize the minimum hidden width needed to approximate the population denoiser to a chosen accuracy. This width is determined by the patch dimension and the covariance spectrum of the denoiser's residual beyond its best affine approximation from that patch. Targeted JiT experiments on ImageNet examine these mechanisms. Learned output shortcuts substantially improve generation under the tested objective. Across two resolution/patch settings, widening improves generation quality and simultaneous affine access to noisy-input and clean-image information. Together, these results identify input compression and simultaneous input/denoiser accessibility as two concrete architectural mechanisms behind prediction-target differences.

Primary area:
generative models
Keywords:
Diffusion models, noise prediction, image prediction, output preconditioning, architectural bottleneck

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_spM5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies why image prediction can outperform noise prediction in diffusion models with rank bottlenecks. It derives an exact noise-recovery error for a linear input bottleneck and analyzes EDM-style preconditioning in this setting. It also studies the effect of hidden width on noise prediction. Experiments on JiT/ImageNet are used to test these predictions.

Critical strengths

S1. Sound: the paper appears technically sound.

S2. Analysis: compared to RAE, the input-rank analysis removes the low-rank output assumption (RAE, Thm. 1) and focuses on the information lost at the input.

Critical weaknesses

W1. Novelty and clarity: the paper makes a formal advance over RAE by removing its low-rank output assumption and analyzing information lost at the input bottleneck. However, I do not see a substantially new insight or practical consequence beyond the bottleneck limitation already identified in RAE. EDM and AsymFlow already use input/output shortcuts. The paper's "exact-preconditioning" case reduces to image prediction as in JiT. The new analysis therefore seems to refine existing explanations without establishing a new modeling recommendation. Explaining the contribution relative to these papers {RAE, JiT, AsymFlow, EDM} is thus essential and should be done in one place and very early in the paper.

W2. Width analysis: Thm. 5 assumes that the noisy patch can be exactly recovered from the hidden state by a linear readout. This assumption is not required by the noise prediction loss. Also, the narrow model doesn't seem to satisfy this assumption empirically: its input-recovery R^2 is 0.7785, although wider models approach 0.9965. The experiments therefore show that width helps, but do not establish the mechanism in Thm. 5 as the explanation of this improvement. This seems weaker than the abstract’s claim that the results “identify input compression and simultaneous input/denoiser accessibility as two concrete architectural mechanisms.”

Questions for authors

Q1. If a practitioner is already familiar with {RAE, JiT, EDM, AsymFlow}, what concrete modeling or design decision should change after reading this paper?

10 comments

We thank the reviewer for the careful assessment. Section, figure, table, and appendix references below are to the revised manuscript unless attributed to another paper. The recovery values quoted in W2 (0.7785 and 0.9965) come from the submitted probe protocol, which the revision refreshes (Figure 5; Appendix B.2.2).


“W1. Novelty and clarity: the paper makes a formal advance over RAE by removing its low-rank output assumption and analyzing information lost at the input bottleneck. However, I do not see a substantially new insight or practical consequence beyond the bottleneck limitation already identified in RAE. EDM and AsymFlow already use input/output shortcuts. The paper's "exact-preconditioning" case reduces to image prediction as in JiT. The new analysis therefore seems to refine existing explanations without establishing a new modeling recommendation. Explaining the contribution relative to these papers {RAE, JiT, AsymFlow, EDM} is thus essential and should be done in one place and very early in the paper.”

Prior work recognizes that predicting high-dimensional noise is hard. Our contribution separates the representation needed to transport the noisy input from the representation needed for denoising corrections, and this yields quantitative predictions about architectural failure, required width, and effective capacity allocation. The revised introduction lists our contributions and compares them with JiT, RAE, EDM, and AsymFlow immediately afterwards (Section 1).

1. JiT: why an identity term has a representation cost

JiT motivates image prediction through the manifold hypothesis. Its abstract states that “natural data should lie on a low-dimensional manifold,” whereas noise and velocity targets occupy the ambient space, and its introduction names an information-preservation requirement: “the network needs to preserve all information about the noise.” JiT then advocates clean-image prediction and shows that it can succeed with coarse patches and narrow input embeddings. Our results build on these observations.

The dimensionality of the target distribution does not by itself explain why the corresponding prediction function is hard to learn. In JiT’s convention zt=tx+(1−t)ϵz_t=t x+(1-t)\epsilon, let Dt(zt)=E[x∣zt]D_t(z_t)=\mathbb E[x\mid z_t]. The optimal noise and velocity predictors satisfy

\qquad v_t^{\ast}(z_t)=\frac{D_t(z_t)-z_t}{1-t}.$$ Relative to clean-image prediction, both need only known scalings and an identity contribution. Adding the identity term introduces no new nonlinear denoising computation, yet it can change the performance of a finite network by a large margin. JiT leaves this unexplained: the identity map can produce full-dimensional Gaussian outputs, but it should be trivial to compute, and an input-to-output shortcut can supply it exactly (as our exact-preconditioning control does; Figure 2). JiT does not quantify the irreducible error caused by input compression, or explain why a representation wide enough to preserve the input may still be insufficient for denoising. The second question arises in JiT’s own experiment: with 768-dimensional patches and width 768 the model fails, and JiT's Section 4.2 suggests that additional capacity may be needed, “e.g., to handle positional embeddings,” without characterizing the additional representation that denoising corrections require. Our results identify two costs. **An identity map is simple to compute, but it cannot recover information discarded at the input.** Theorem 1 and Corollary 2 establish that a rank-$r$ linear input bottleneck imposes a denoising-error floor $\sigma_t^2(d-r)$ on bare noise prediction, regardless of downstream width, depth, or expressivity, because the missing Gaussian components cannot be inferred from the retained input. Computation after the compression cannot remove this error, whereas a path carrying the intact input can bypass the restriction. With the $768\rightarrow128\rightarrow768$ embedding, the trained noise predictor's per-coordinate MSE is 300.836 against the predicted floor of 300.833 at $t=0.05$ and 0.83387 against 0.83333 at $t=0.5$ (Figure 2b), and our response to Reviewer jynx (W2/Q1) extends this to five encoder ranks and five noise levels. The cost of the identity term comes from an information constraint rather than from the difficulty of learning a high-dimensional function. **Enough width to transport the input need not be enough to support denoising corrections.** Theorem 5 identifies the constraint behind the width-768 failure: at width $d_p$, exact affine input recovery leaves room only for affine denoising readouts, so nonlinear or contextual corrections require additional coordinates or come at the cost of input recovery. The number of additional coordinates is set by the covariance spectrum of the residual beyond the best affine denoiser. These results give empirical predictions more specific than a general benefit from widening. In Figure 4a, noise prediction at width $d_p+c$ corresponds approximately to image prediction at width $c$, with the offset fixed by patch geometry, and much of the sharp widening benefit persists when attention, MLP, and conditioning dimensions remain fixed. Together these findings identify a representation overhead and show that accommodating it need not require proportionally enlarging every internal computation.

2. RAE: input information can be lost even when output expressivity is unrestricted

RAE’s Theorem 1 considers networks of the form g(xt,t)=Bf(Axt,t)g(x_t,t)=Bf(Ax_t,t), and its proof derives a loss floor by restricting predictions to the column space of the final linear map BB (RAE, Appendix B.1). Our result identifies a different obstruction: information discarded at the input cannot be recovered even when subsequent computation and outputs are unrestricted.

Consider a patch embedding of the form

768→128→768,768 \rightarrow 128 \rightarrow 768,

followed by full-width Transformer and a full-rank output head. The embedding compresses the input, but the nonlinear decoder after it need not produce outputs confined to a 128-dimensional linear subspace, so RAE’s output-subspace bound does not establish failure for this architecture. RAE’s Appendix B.1 notes that when d≥nd\ge n, its argument gives “a trivial lower bound in Eq 5.” Here, dd denotes model width and nn denotes input dimension. Our example has d=n=768d=n=768, so their spectral-tail bound is ∑i=d+1nλi=0,\sum_{i=d+1}^{n}\lambda_i=0, even though the effective input map has rank only 128.

Our bound depends on the input rank, and expanding the representation back to 768 dimensions does not restore the Gaussian information discarded by the initial projection. Theorem 1 and Corollary 2 imply that bare noise prediction incurs an induced clean-image error of at least

σt2(768−128)=640σt2\sigma_t^2(768-128)=640\sigma_t^2

per patch, regardless of the width, depth, or expressivity of the downstream decoder. Capacity added after the input bottleneck cannot remove this floor. The model must preserve the missing input information, supply it through another path, or change the prediction parameterization. Our result isolates an information-theoretic limitation that persists after the output-expressivity restriction underlying RAE’s bound has been removed.

3. AsymFlow: reducing transmitted noise versus quantifying its representation cost

AsymFlow recognizes the burden of transmitting noise through learned features. Its asymmetric target Pϵ−x0P\epsilon-x_0 reduces the number of noise directions the network must predict, and a analytic reconstruction supplies the remaining contribution. We address a question this construction leaves unresolved: when full noise prediction is retained, how much representation capacity does noise transmission consume, and how much additional capacity is needed for denoising?

Preserving all input dimensions does not settle this question. Consider a model whose width equals the patch dimension, dh=dpd_h=d_p, so there need be no input information loss. AsymFlow’s analysis does not establish whether this width suffices for nonlinear denoising, and it does not characterize the additional width required. Under affine input recovery and an affine output head, Theorem 5 shows that width dpd_p permits only affine denoising. At width dp+cd_p+c, the additional cc feature directions support corrections beyond the best affine denoiser, with attainable error determined by the covariance spectrum of those corrections.

AsymFlow’s noise-subspace rank and our additional feature width describe different quantities. The former controls how much noise enters the learned target, and the latter controls denoising capacity while retaining full noise prediction. The architectural prescriptions differ as well. AsymFlow reduces the transmission burden by changing the prediction target, whereas our results support accommodating that burden by widening the residual representation while keeping attention, MLP, and conditioning dimensions fixed. This gives practitioners who retain noise prediction a design option.

4. EDM and exact preconditioning: which shortcut removes the cost

EDM combines a noisy-input skip with a learned output using noise-dependent coefficients, and JiT relates its benefit to approaching image prediction at high noise. Theorem 3 and Corollary 4 make this quantitative for any linear combination of image and noise targets with a scalar shortcut: the discarded-noise contribution is (β−cskipσ)2(d−r)(\beta-c_{\mathrm{skip}}\sigma)^2(d-r), so what matters is the noise coefficient remaining in the learned target, not the presence of a shortcut. For EDM the contribution decays to zero at high noise but remains at finite noise: at t=0.5t=0.5 it is 0.03333 per coordinate, and EDM's measured MSE is 0.04747 against 0.01570 for image prediction (Figure 2b). Exact preconditioning, cskip=β/σc_{\mathrm{skip}}=\beta/\sigma, is the unique coefficient that removes the contribution at every noise level. It is image-prediction-equivalent by design and serves as our control (measured MSE 0.01571). The theory's predictive content lies in the non-equivalent cases: it quantifies EDM's remaining term, and when the shortcut coefficients are learned, training drives the effective skip to the cancellation value from bare-noise and additive initializations (∣c~skip∣<0.0045|\widetilde c_{\mathrm{skip}}|<0.0045 over 0.05≤t≤0.50.05\le t\le0.5; Figure 6).

Reviewer_jynxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates architectural sources of the performance gap between clean-image prediction and noise prediction, despite their equivalence at the unrestricted population optimum. Under Gaussian corruption, the authors derive an exact noise-recovery risk decomposition for predictors whose inputs pass through a rank-deficient linear encoder. This yields an unavoidable contribution of (\sigma_t^2(d-r)) to the induced clean-image estimation risk of bare noise prediction. The paper then characterizes how output preconditioning changes this contribution, including exact cancellation and the high-noise behavior of EDM. Separately, under an explicit assumption of exact affine recoverability of the noisy patch, the authors characterize the hidden width required to approximate the population denoiser through affine readouts. Experiments with JiT on ImageNet examine prediction targets, output shortcuts, width scaling, and affine accessibility of input and clean-image information.

Critical strengths
  1. The input-compression result provides a precise and useful distinction between population equivalence and architectural equivalence. The analysis allows arbitrary measurable downstream computation and requires only finite second moments for the clean data. Consequently, the discarded-noise term isolates an information-access limitation rather than a limitation of a particular decoder architecture or optimizer. This is a meaningful refinement of the broader observation that predicting high-dimensional corruption can be more demanding than predicting clean data.
  2. The output-preconditioning analysis connects the theoretical obstruction to a concrete architectural intervention. The exact-cancellation control recovers performance close to direct image prediction under the tested implementation, while learned shortcuts substantially improve over bare noise prediction. The manuscript also explicitly distinguishes its fixed-time risk results from sampling guarantees and acknowledges that the width theorem applies to a restricted representation class. These qualifications are important to interpreting the contribution correctly.
Critical weaknesses
  1. The practical explanatory value of the width theorem remains insufficiently established. Theorem 5 assumes that the noisy patch is exactly recoverable from the hidden representation through an affine map. The authors correctly acknowledge that this is an imposed representation strategy, rather than a necessary condition for accurate noise prediction. However, the experiments do not establish that this strategy governs the observed width dependence. The narrow models in Table 3 do not satisfy exact affine input recovery, and the width sweep simultaneously changes parameter count and attention configuration. Moreover, Appendix C states that the residual covariance spectrum appearing in the theorem is not estimated. The results therefore demonstrate an association between widening, improved affine accessibility, and better FID, but do not directly validate the theorem’s quantitative width–accuracy characterization or distinguish it from more general capacity and optimization effects.
  2. The experiments provide limited quantitative validation of the paper’s most distinctive theoretical predictions. The broader motivation concerning the burden of representing high-dimensional noise is already present in the cited JiT and AsymFlow literature; the clearest additional contribution here is the exact risk characterization under an explicit input restriction. Yet the principal bottleneck experiment uses a single compression setting, and the reported FID comparisons do not test how clean-image risk scales jointly with encoder rank and noise level. The exact-preconditioning result is a valuable consistency check, but its agreement with image prediction is also expected from their algebraic equivalence away from endpoint clamps. In addition, generation results use single training runs, and Appendix C documents nonuniform EMA choices across some width comparisons. These issues do not invalidate the theorems, but they limit how strongly the empirical results establish the proposed mechanisms.
Questions for authors
  1. Quantitative validation: Can the authors report clean-image estimation risk across several encoder ranks and noise levels, separating the predicted discarded-noise contribution from the remaining error? This would directly test the new risk characterization rather than relying primarily on generation quality.
  2. Relevance of the width assumption: Can the authors provide a controlled experiment, potentially using a tractable synthetic distribution, that enforces affine input recoverability and measures the residual covariance spectrum? Does the observed approximation error then follow the spectral-tail prediction of Theorem 5?
  3. Robustness of the width comparison: Do the main width trends persist under a common EMA selection rule? For configurations whose generation and probe evaluations use different saved parameter averages, do the conclusions remain unchanged when both evaluations use the same weights?
8 comments

We thank the reviewer for the careful assessment and concrete suggestions for quantitative validation. We now report controlled synthetic tests of both quantitative predictions, the rank–noise scaling of the bottleneck floor (Theorem 1) and the spectral tail of the width theorem (Theorem 5), together with broader input-recovery probes, controls separating representation width from internal computation, and residual-spectrum estimates. We also compare measured denoising errors directly with the predicted noise-dependent floors and confirm that the widening trends persist under a common EMA rule and matched checkpoints. These additions address the empirical relevance of the assumptions and the quantitative predictions highlighted in the review. Section, figure, table, and appendix references below are to the revised manuscript; quotations from the review keep the submitted numbering, in which Table 3 corresponds to the revised Figure 5 and Appendix B.2.2, and Appendix C to the revised Appendix B.2.


W1 / Q2

“... the experiments do not establish that this strategy governs the observed width dependence. The narrow models in Table 3 do not satisfy exact affine input recovery, and the width sweep simultaneously changes parameter count and attention configuration. Moreover, Appendix C states that the residual covariance spectrum appearing in the theorem is not estimated. ...”

“Relevance of the width assumption: Can the authors provide a controlled experiment ... that enforces affine input recoverability and measures the residual covariance spectrum? Does the observed approximation error then follow the spectral-tail prediction of Theorem 5?”

We have added evidence on each point the reviewer raises: the empirical relevance of affine input recovery, the architectural confounds in widening, the residual spectrum underlying the width prediction, and, as requested in Q2, a controlled test of the quantitative width–accuracy characterization.

Affine input recovery in trained models. Released ADM, DiT, U-ViT, and LDM noise predictors achieve 99.45%–99.85% affine input recovery at moderate noise (t=0.5t=0.5, effective noise approximately one), measured on held-out images with affine readouts shared across spatial positions in their respective pixel or latent spaces. In our matched epoch-120 EMA1 ImageNet256 sweep without a bottleneck embedding, recovery rises from 78.30% at width 768 to 95.16% at 896 and 99.40% at 1536 (Figure 5; Appendix B.2.2). Theorem 5 identifies a structural constraint at width 768: with dh=dpd_h=d_p, exact affine input recovery forces the representation to be an affine function of the noisy patch alone, so its best affine clean readout is the affine estimate ℓt\ell_t. Nonlinear or cross-patch denoising corrections at this width therefore require imperfect input recovery, and the incomplete recovery of the width-768 model is consistent with this constraint. Extra width allows exact recovery and these corrections to coexist.

Separating representation width from internal computation. Our new Fixed controls widen the residual stream while holding the attention projection, MLP intermediate, and AdaLN conditioning dimensions fixed. Parameter growth is then linear rather than quadratic in width at fixed depth. On ImageNet128, widening from 192 to 256 still reduces FID from 396.26 to 116.59, close to 114.38 under standard widening. Thus, the sharp benefit of wider representations persists without jointly widening every internal computation (Figure 4a).

The depth controls further test how capacity is allocated. At width 384, increasing the Fixed model's depth from 12 to 24 improves FID from 102.68 to 86.36, compared with 90.06 for the standard model at similar parameter counts (32.65M versus 33.02M). Denoising measurements show larger relative gains from further widening at low noise, consistent with the residual-spectrum interpretation of Theorem 5, and additional depth further improves low-noise denoising at fixed width (Figure 4c).

Measuring the residual spectrum. We now estimate the covariance spectrum of the correction beyond the best affine patch denoiser, using a pretrained ADM denoiser as a proxy for the population denoiser. On ImageNet128, from t=0.1t=0.1 to t=0.9t=0.9, residual magnitude decreases by a factor of 35.7 while the number of directions explaining 95% of its variance increases from 4 to 116. The low-noise correction is therefore smaller but more broadly distributed across directions, consistent with the larger relative denoising gains from further widening at low noise (Figure 4c–d; Appendix B.2.4).

Controlled validation of the spectral-tail prediction (Q2). We conducted controlled synthetic experiments with exact population denoisers and affine input recovery enforced by construction. The measured approximation errors closely follow the spectral-tail prediction of Theorem 5 in every setting reported below.

We use two patch distributions with closed-form posterior mean E[X1∣Yt]\mathbb{E}[X_1\mid Y_t], so ℓt\ell_t, RtR_t and the spectrum of Cov(Rt)\mathrm{Cov}(R_t) come from the exact denoiser rather than a learned proxy (Xt=tX1+(1−t)εX_t=tX_1+(1-t)\varepsilon, Yt=Xt/tY_t=X_t/t as in the paper).

(i) Prescribed spectra (dp=64d_p=64): X1=Uz+σ0nX_1=Uz+\sigma_0 n with orthonormal U∈R64×rU\in\mathbb{R}^{64\times r}, zj=±sjz_j=\pm s_j equiprobable and σ0=0.1\sigma_0=0.1. The posterior mean factorizes into one tanh term per column of UU and agrees with ℓt\ell_t on span(U)⊥\mathrm{span}(U)^\perp, so Cov(Rt)\mathrm{Cov}(R_t) has rank rr, is diagonal in the basis UU, and has eigenvalues given by 1-D integrals (computed by quadrature). The scales sjs_j therefore set the spectrum; we use r=4,8,16,32r=4,8,16,32 equal scales, which give rr equal eigenvalues.

(ii) ImageNet patches (dp=192d_p=192, the patch size of our ImageNet128 experiments). To obtain a natural-image spectrum while keeping the denoiser exact, we fit a 128-component full-covariance GMM by EM to 1.216M 8×88\times8 RGB patches from 38k ImageNet images (log-likelihood on 64k held-out patches from 2k further images matches training) and use it as the data distribution: clean patches are sampled from this GMM, so its posterior mean, the responsibility-weighted sum of per-component Wiener filters, is the exact population denoiser. Here Cov(Rt)\mathrm{Cov}(R_t) is full rank with a decaying spectrum, and we estimate it by Monte Carlo from 2M noisy samples.

Networks. For each tt and each extra width k=dh−dpk=d_h-d_p, we train h(Xt)=[Xt;g(Xt)]h(X_t)=[X_t; g(X_t)], with gg a 6-block residual MLP with kk outputs, and a linear noise readout. Since Xt=tYtX_t=tY_t, a fixed linear map recovers the noisy patch exactly, so the assumption of Theorem 5 holds by construction. We train on the exact posterior mean (same minimizer as the clean target) starting from the best affine readout, with AdamW for 30k steps, then freeze hh and refit the readout by least squares on fresh samples (approximating the infimum over the readout).

Results. Entries are measured / predicted: the error of the trained network, E∥X^1,m(ε)−E[X1,m∣Yt]∥2\mathbb{E}\lVert \widehat X_{1,m}^{(\varepsilon)}-\mathbb{E}[X_{1,m}\mid Y_t]\rVert ^2 on held-out samples, over the spectral tail ∑j>kλj(Cov(Rt))\sum_{j>k}\lambda_j(\mathrm{Cov}(R_t)), the minimum error predicted by Theorem 5; ≈0\approx0 denotes errors below 10−510^{-5}.

(i) Prescribed spectra at t=0.5t=0.5 (columns: rank rr of Cov(Rt)\mathrm{Cov}(R_t), equal eigenvalues):

krank 4rank 8rank 16rank 32
00.1955 / 0.19500.3912 / 0.39010.7792 / 0.78021.561 / 1.560
4≈0 / 00.1959 / 0.19500.5842 / 0.58511.367 / 1.365
8≈0 / 0≈0 / 00.3900 / 0.39011.173 / 1.170
16≈0 / 0≈0 / 0≈0 / 00.7824 / 0.7802
32≈0 / 0≈0 / 0≈0 / 0≈0 / 0

Each direction carries the same eigenvalue λ=0.0488\lambda=0.0488, so Theorem 5 predicts an error of (r−k)λ(r-k)\lambda for k<rk\lt r and zero for k≥rk\ge r. The measured error matches this prediction in every cell (within 0.5%) and vanishes once kk reaches the rank.

(ii) ImageNet patches, as % of E∥Rt∥2\mathbb{E}\lVert R_t\rVert ^2 remaining:

kt=0.3t=0.5t=0.7t=0.9
099.4 / 100100.1 / 100100.3 / 100100.3 / 100
85.23 / 5.0325.8 / 25.761.3 / 61.187.4 / 87.1
161.20 / 1.168.79 / 8.7535.7 / 35.675.6 / 75.4
320.24 / 0.212.06 / 2.0213.3 / 13.254.5 / 54.2
640.05 / 0.020.25 / 0.202.24 / 2.1322.3 / 22.1
1280.04 / <0.010.07 / <0.010.19 / 0.051.85 / 1.48
95% dirs92250103

The measured error matches the predicted fraction in every cell, to within 0.7 percentage points, including the k=96k=96 and k=192k=192 settings not shown. The table also shows the noise dependence that Theorem 5 ties to width: at high noise the residual concentrates in few directions (95% of it in 1, 4 and 9 directions at t=0.1,0.2,0.3t=0.1,0.2,0.3), so 8 extra coordinates remove about 95% of it at t=0.3t=0.3; at low noise it spreads over 103 directions, and 64 extra coordinates still leave 22% at t=0.9t=0.9. This is the trend measured with the ADM proxy in the revised Section 4.2.

Lower bound. The spectral tail is the minimum error over this representation class, so these trained networks closely approach the optimum of Theorem 5. Measured errors and the ImageNet-patch tails are Monte Carlo estimates, so a few entries fall below the tail by at most 0.62% of E∥Rt∥2\mathbb{E}\lVert R_t\rVert ^2.

We will add these results as a new appendix section.

W2 / Q1

“... the principal bottleneck experiment uses a single compression setting ... The exact-preconditioning result is a valuable consistency check, but its agreement with image prediction is also expected from their algebraic equivalence away from endpoint clamps.”

“Quantitative validation: Can the authors report clean-image estimation risk across several encoder ranks and noise levels, separating the predicted discarded-noise contribution from the remaining error?”

We now directly compare measured clean-image MSE with the predicted discarded-noise contribution across seven noise levels (Figure 2b; Appendix B.1.3). This tests the denoising-risk prediction underlying the generation comparisons.

Predicted contribution and measured error. For the 768→128768\rightarrow128 input embedding, we compute the per-coordinate discarded-noise lower bound from Corollary 4 using the rank upper bound of 128. The table shows two representative noise levels and separates this bound from the remaining measured error.

ParameterizationttPredicted boundMeasured MSEMSE − bound
Bare noise0.05300.83333300.836≈0.0027\approx0.0027
Image0.0500.110160.11016
Exact0.0500.110190.11019
EDM0.050.000144≈0.1103\approx0.1103≈0.1102\approx0.1102
Bare noise0.50.833330.83387≈0.00054\approx0.00054
Image0.500.015700.01570
Exact0.500.015710.01571
EDM0.50.03333≈0.04747\approx0.04747≈0.01414\approx0.01414

Bare noise prediction, without preconditioning, lies close to the predicted floor at both noise levels. EDM retains a visible contribution at moderate noise (t=0.5t=0.5), while at high noise (t=0.05t=0.05) its floor becomes negligible and its MSE approaches the image-prediction controls. The remainder includes clean-signal uncertainty and model approximation and optimization error. The full seven-time comparison is in Figure 2b and Appendix B.1.3.

Exact and learned preconditioning. The exact-cancellation model is a control by design: Theorem 3 identifies cskip=β/σc_{\mathrm{skip}}=\beta/\sigma as the unique coefficient that removes the discarded-noise term, and the control's error matches image prediction (0.01571 vs 0.01570 at t=0.5t=0.5). The theory goes beyond this equivalence. It quantifies the term that remains for shortcuts that are not equivalent to image prediction, such as EDM's 0.03333 at t=0.5t=0.5 in the table above. When (cskip,cout)(c_{\mathrm{skip}},c_{\mathrm{out}}) are learned as functions of tt, training drives the effective skip to this cancellation value on its own: Learned 2 and Learned 3, initialized from bare-noise (0,1)(0,1) and additive (t,1)(t,1) coefficients, reach ∣c~skip∣<0.0045|\widetilde c_{\mathrm{skip}}|<0.0045 over 0.05≤t≤0.50.05\le t\le0.5, the same level as Learned 1, which starts at exact preconditioning (Figure 6; Appendix B.1.2).

Encoder-rank sweep. We ran the requested sweep in a setting where every term of the risk decomposition is computable. Clean 8×88\times8 RGB patches (d=192d=192) are sampled from the ImageNet-patch GMM of Q2, so E[X∣Z]\mathbb E[X\mid Z] has a closed form for any linear encoder. Each model applies a learned rank-rr linear encoder Z=AYZ=AY followed by a nonlinear decoder (a 6-block residual MLP of width 512 plus a linear branch from ZZ). For r∈{16,32,64,128,192}r\in\lbrace 16,32,64,128,192\rbrace and t∈{0.1,0.3,0.5,0.7,0.9}t\in\lbrace 0.1,0.3,0.5,0.7,0.9\rbrace we train separate noise- and image-prediction models (50 runs, 30k steps each). The clean-image risk of each noise predictor is the discarded-noise term σ2(d−r)\sigma^2(d-r) plus a remaining error, which we measure on 262k held-out samples against the exact E[X∣Z]\mathbb E[X\mid Z] for its learned encoder. Values are per coordinate.

Reviewer_F1f4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides theoretical and empirical justifications for why image prediction can outperform noise prediction in diffusion models. The paper analyses the prediction errors of the two variants given a rank-r linear input bottleneck, how preconditioning can help bypass the noise prediction burden, and the required network width to approximate the denoiser while keeping the noisy patch affinely recoverable. Evaluations using JiT on ImageNet128/256 empirically support the superior performance of image prediction (though JiT has previously demonstrated this), quantify the noise recovery ceiling, and test the width and output preconditioning hypotheses.

Critical strengths
  • The paper provides a new theoretical analysis on parameterization choices that the community historically chose, or avoided, largely based on intuition or empirical findings. The proposed theory is sound, and most theoretical predictions have accompanying empirical evaluations.

  • The analyses cover architectural designs that have been adopted in practice, such as adding skip connections to forward the input to the network’s prediction, and place related techniques, such as EDM preconditioning, into a unified framework (e.g., Corollary 4).

Critical weaknesses
  • The key theory seems to reconfirm the intuition that the community already has (low-dimensional bottleneck destroys high-dimensional independent signal) and designs already adopted (e.g., networks that predict noise have skip connections, EDM preconditioning, or using image prediction in bottlenecked models like JiT), rather than providing a new intuition or an unexpected finding. So, the direct findings are unlikely to change practice.

  • Some of the analyses rely on scenarios or assumptions that may not reflect actual behaviors of a trained network. For example, in Section 3.3 L269, if the input recovery assumption is not true or far from what happens in practice, the theory, results, and implications may not be relevant or useful. The correlation between input recovery and network width in Table 3 does not show that this assumption is necessary, and good input recovery can be just a byproduct of representing both the noise and the clean image. While the authors agree with this themselves, it reinforces the limited impact.

Questions for authors
  • Why is input preservation a reasonable assumption or representation class that would lead to meaningful findings?

  • What are other implications of the work that could affect current practice?

4 comments

We thank the reviewer for the careful assessment and for highlighting the need to clarify the contribution and its practical relevance. We now report direct tests of the predicted denoising errors and width requirements, broader probes of public checkpoints, and controls that separate representation width from internal computation. Below, we explain how this evidence supports the analysis and informs concrete modeling decisions. All references below are to the revised manuscript.

W1

“The key theory seems to reconfirm the intuition that the community already has (low-dimensional bottleneck destroys high-dimensional independent signal) and designs already adopted (e.g., networks that predict noise have skip connections, EDM preconditioning, or using image prediction in bottlenecked models like JiT), rather than providing a new intuition or an unexpected finding. So, the direct findings are unlikely to change practice.”

Our contribution is to quantify how input compression affects denoising accuracy across prediction targets and preconditioning choices, and test these predictions against trained models. This turns the existing intuition about high-dimensional noise into a measurable account of the performance gap.

A bound that tracks actual denoising loss. For a rank-rr linear encoding of a dd-dimensional input, Theorem 1 and Corollary 2 give a denoising-error floor of σt2(d−r)\sigma_t^2(d-r) for bare noise prediction, while optimal image-prediction risk remains bounded as noise increases. The bound holds even with unrestricted downstream computation and output. With JiT's 768→128→768768\rightarrow128\rightarrow768 embedding, measured per-coordinate MSE is 300.836 against the predicted floor of 300.833 at t=0.05t=0.05, and 0.83387 against 0.83333 at t=0.5t=0.5 (Figure 2b). This identifies an error contribution that increasing downstream capacity cannot remove while the input restriction remains fixed.

Predicting the denoising effect of preconditioning. Theorem 3 and Corollary 4 quantify how the target and scalar shortcut coefficients change this contribution. For EDM it decays to zero at high noise, where measured denoising loss approaches image prediction; at t=0.5t=0.5, the predicted contribution is 0.03333 and measured MSE is approximately 0.04747, compared with 0.01570 for image prediction (Figures 2b and 7). Exact preconditioning cancels the contribution and is image-prediction-equivalent, serving as a consistency check. When the skip and output coefficients are learned, training drives the effective skip of all three variants to the cancellation value at moderate and high noise (∣c~skip∣<0.0045|\widetilde c_{\mathrm{skip}}|<0.0045 for 0.05≤t≤0.50.05\le t\le0.5), including the two initialized away from it (Figure 6).

The practical consequence is to distinguish when a model needs additional downstream capacity from when its input information or residual prediction task must change. Q2 below gives a concrete application to choosing patch size and representation width under a compute budget.

W2 / Q1

“Some of the analyses rely on scenarios or assumptions that may not reflect actual behaviors of a trained network. For example, in Section 3.3 L269, if the input recovery assumption is not true or far from what happens in practice, the theory, results, and implications may not be relevant or useful. The correlation between input recovery and network width in Table 3 does not show that this assumption is necessary, and good input recovery can be just a byproduct of representing both the noise and the clean image. While the authors agree with this themselves, it reinforces the limited impact.”

“Why is input preservation a reasonable assumption or representation class that would lead to meaningful findings?”

Why this representation class. At a fixed time, the optimal noise predictor satisfies fε,t,m∗(Yt)=(Yt,m−fX1,t,m∗(Yt))/σtf^\ast_{\varepsilon,t,m}(Y_t)=(Y_{t,m}-f^\ast_{X_1,t,m}(Y_t))/\sigma_t: the noisy patch minus the denoised patch, scaled. A representation in which the noisy patch is affinely accessible lets the final linear layer form this difference directly, and it already supplies every affine denoiser of the patch, including the best one, ℓt\ell_t, at no extra width. What remains is the residual Rt=fX1,t,m∗(Yt)−ℓt(Yt,m)R_t=f^\ast_{X_1,t,m}(Y_t)-\ell_t(Y_{t,m}), which carries the nonlinear and cross-patch information. Theorem 5 gives the exact cost of representing it: at width dh=dp+kd_h=d_p+k, the optimal approximation error is ∑j>kλj(Cov⁡Rt)\sum_{j>k}\lambda_j(\operatorname{Cov}R_t). The class thus turns the width question into a measurable property of the data, the residual spectrum, which we estimate on ImageNet (Figure 4d) and test directly in a controlled setting below. The input-bottleneck and preconditioning results in Theorems 1 and 3 do not require this assumption.

Well-trained noise predictors closely satisfy the assumption. Released ADM, DiT, U-ViT, and LDM checkpoints achieve 99.45%–99.85% affine noisy-input recovery at moderate noise in their respective pixel or latent spaces (Figure 5; Appendix B.2.2). In our matched ImageNet256 JiT sweep, recovery increases from 78.30% at width 768 to 95.16% at 896 and 99.40% at 1536, alongside improved denoising accuracy. These measurements use held-out images and affine readouts shared across spatial positions.

Controlled validation of the width prediction. We also test Theorem 5 with exact population denoisers and affine input recovery enforced by construction. On an ImageNet-patch Gaussian-mixture distribution, measured approximation errors agree with the predicted spectral tails within 0.7 percentage points of residual energy across all 32 tested width–noise settings. This directly validates the quantitative prediction within the assumed representation class (details in our response to Reviewer jynx, W1/Q2).

The byproduct interpretation leads to the same width cost. Theorem 5 applies to every representation from which the noisy patch is affinely recoverable, regardless of why training produces it. If recovery arises as a byproduct of representing noise and clean-image information, the representation is still in this class, and an affine readout from it approximates the population denoiser at width dh=dp+kd_h=d_p+k with error at least ∑j>kλj(Cov⁡Rt)\sum_{j>k}\lambda_j(\operatorname{Cov}R_t). The theorem therefore does not require recovery to be necessary; it characterizes the width cost of the representation that trained noise predictors closely approach.

Q2

“What are other implications of the work that could affect current practice?”

A concrete application is reducing token count through larger patches while retaining noise prediction. This makes each denoising step cheaper, but increases the input dimension that each token must represent. Our results inform how to choose representation width and allocate internal computation in this setting.

Where this tradeoff arises. DiT (Peebles and Xie, 2023, Sections 3.2 and 5) studies this compute–quality tradeoff: halving patch side quadruples the number of tokens and at least quadruples Transformer GFLOPs, while smaller patches improve FID at a fixed backbone size. FlexiDiT (Anagnostidis et al., 2025, Sections 3.3 and 4.1) uses larger patches during early denoising steps and smaller patches later. Its noise-predicting ImageNet model supports latent patch sizes 2 and 4; the larger patches reduce token count fourfold, enabling compute savings, with quality degradation as coarse-patch use is increased further. Thus, practitioners already face the question of how aggressively they can reduce token count without sacrificing denoising and generation quality. In latent space the patch dimension stays small (16 to 256 for DiT's 32×32×432\times32\times4 latents, against DiT-XL width 1152), but in pixel space it reaches the model width quickly: JiT/16 has patch dimension 768, equal to the JiT-B width, and JiT/32 has 3072. The checks below matter in this pixel-space regime.

What our results recommend checking. Doubling patch side also quadruples patch dimension. If the model width is held fixed, this can create a rank-deficient input embedding; Theorem 1 and Corollary 2 quantify the resulting discarded-noise contribution, which additional downstream computation cannot remove. Even after avoiding this strict bottleneck, Theorem 5 shows that, under affine input recovery, additional representation width supports corrections beyond linear denoising. The width required depends on the residual spectral tail and can reach 2dp2d_p for exact population-denoiser approximation when the residual has full rank. This motivates checking input rank and denoising accuracy as patch size grows, rather than selecting patch size from attention cost alone.

Wider representations need not require uniformly wider internal computation. Our new controls test whether useful denoising and generation gains can be recovered with linear parameter growth in representation width at fixed patch size and depth.

On ImageNet128, we increase the residual-stream/model width while holding the attention projection and AdaLN conditioning dimensions at 192 and the SwiGLU intermediate dimension at 512. This gives linear, rather than quadratic, parameter growth with model width at fixed depth. Widening from 192 to 256 still reduces FID from 396.26 to 116.59, close to 114.38 for unrestricted computation (Figure 4a). Thus, the large improvement does not require simultaneously widening every internal projection.

Additional depth can then close the remaining gap at approximately matched parameter counts. At width 384, the current results are:

ImageNet128 noise predictorBlocksParametersFID-10K
Fixed1216.65M102.68
Fixed + depth2432.65M86.36
Unrestricted internal computation1233.02M90.06

These are epoch-200 EMA1 evaluations (Figure 4b; Appendix B.2.1). They support testing a wide residual representation with narrower internal computations and additional depth, rather than assuming all component widths must grow together.

The denoising measurements give the same capacity-allocation question a noise-dependent interpretation. At t=0.1t=0.1, widening to 256 already yields about 79% MSE reduction relative to the width-192 noise model across the tested families. At t=0.9t=0.9, unrestricted widening from 256 to 384 increases that relative reduction from 12.2% to 19.3%; at representation width 384 with fixed internal dimensions, increasing depth from 12 to 24 improves it from 12.4% to 17.8% (Figure 4c). Width sets the best attainable approximation in Theorem 5, and depth supplies the computation to approach it.

The practical recommendation is to consider wider residual representations with fixed internal dimensions when larger patches are desirable for reducing token count, then allocate additional computation separately, including through depth. Our width controls demonstrate this flexibility at fixed patch size, and the theory and measurements motivate applying it to larger-patch designs. If input compression is retained, Theorem 3 and Corollary 4 additionally quantify how the target and shortcut can attenuate or cancel its discarded-noise contribution.

Thank you again for your constructive review, which helped us strengthen the paper. Additionally, to address your question about practical implications, we developed a training-free calibration for pretrained image predictors. It improves denoising and, under relaxed or unclamped sampling, FID on pretrained JiT. Please see [A3] in our follow-up to Reviewer spM5 for the derivation and results. We would be grateful for your updated assessment, and please let us know if any questions remain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and New Calibration That Improves Pretrained Image Generation

We thank the reviewers for their feedback, which has helped us substantially strengthen the paper. In our paper,

  • We quantify the noise-transmission burden as a denoising-risk lower bound, closely tracking the denoising losses of our JiT noise and velocity predictors and quantifying EDM’s residual burden (Figure 2).
  • We quantify how much model width noise prediction requires for a given denoising accuracy under stated assumptions, supported by synthetic tests (our follow-up to Reviewer jynx, [A2]), ImageNet experiments, and pretrained-model probes (Figures 4 and 5).

During this rebuttal, we further developed a calibration method for pretrained image predictors derived from our input-bottleneck theorem, requiring no backbone training or fine-tuning.

Most notably, on the released 2.18B-parameter Pyramid-JiT512 text-to-image model, calibration reduces COCO FID-30K from 17.75 to 16.87 (4.9%), under its original sampling settings. This demonstrates a concrete practical application of our input-bottleneck analysis. Our latest response to Reviewer spM5 provides the experimental protocol and detailed results, with implementation and reproducibility materials in the linked repository.

We would appreciate your consideration of these additions in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.