- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies why image prediction can outperform noise prediction in diffusion models with rank bottlenecks. It derives an exact noise-recovery error for a linear input bottleneck and analyzes EDM-style preconditioning in this setting. It also studies the effect of hidden width on noise prediction. Experiments on JiT/ImageNet are used to test these predictions.
Critical strengths
S1. Sound: the paper appears technically sound.
S2. Analysis: compared to RAE, the input-rank analysis removes the low-rank output assumption (RAE, Thm. 1) and focuses on the information lost at the input.
Critical weaknesses
W1. Novelty and clarity: the paper makes a formal advance over RAE by removing its low-rank output assumption and analyzing information lost at the input bottleneck. However, I do not see a substantially new insight or practical consequence beyond the bottleneck limitation already identified in RAE. EDM and AsymFlow already use input/output shortcuts. The paper's "exact-preconditioning" case reduces to image prediction as in JiT. The new analysis therefore seems to refine existing explanations without establishing a new modeling recommendation. Explaining the contribution relative to these papers {RAE, JiT, AsymFlow, EDM} is thus essential and should be done in one place and very early in the paper.
W2. Width analysis: Thm. 5 assumes that the noisy patch can be exactly recovered from the hidden state by a linear readout. This assumption is not required by the noise prediction loss. Also, the narrow model doesn't seem to satisfy this assumption empirically: its input-recovery R^2 is 0.7785, although wider models approach 0.9965. The experiments therefore show that width helps, but do not establish the mechanism in Thm. 5 as the explanation of this improvement. This seems weaker than the abstract’s claim that the results “identify input compression and simultaneous input/denoiser accessibility as two concrete architectural mechanisms.”
Questions for authors
Q1. If a practitioner is already familiar with {RAE, JiT, EDM, AsymFlow}, what concrete modeling or design decision should change after reading this paper?
10 comments
1. JiT: why an identity term has a representation cost
JiT motivates image prediction through the manifold hypothesis. Its abstract states that “natural data should lie on a low-dimensional manifold,” whereas noise and velocity targets occupy the ambient space, and its introduction names an information-preservation requirement: “the network needs to preserve all information about the noise.” JiT then advocates clean-image prediction and shows that it can succeed with coarse patches and narrow input embeddings. Our results build on these observations.
The dimensionality of the target distribution does not by itself explain why the corresponding prediction function is hard to learn. In JiT’s convention , let . The optimal noise and velocity predictors satisfy
\qquad v_t^{\ast}(z_t)=\frac{D_t(z_t)-z_t}{1-t}.$$ Relative to clean-image prediction, both need only known scalings and an identity contribution. Adding the identity term introduces no new nonlinear denoising computation, yet it can change the performance of a finite network by a large margin. JiT leaves this unexplained: the identity map can produce full-dimensional Gaussian outputs, but it should be trivial to compute, and an input-to-output shortcut can supply it exactly (as our exact-preconditioning control does; Figure 2). JiT does not quantify the irreducible error caused by input compression, or explain why a representation wide enough to preserve the input may still be insufficient for denoising. The second question arises in JiT’s own experiment: with 768-dimensional patches and width 768 the model fails, and JiT's Section 4.2 suggests that additional capacity may be needed, “e.g., to handle positional embeddings,” without characterizing the additional representation that denoising corrections require. Our results identify two costs. **An identity map is simple to compute, but it cannot recover information discarded at the input.** Theorem 1 and Corollary 2 establish that a rank-$r$ linear input bottleneck imposes a denoising-error floor $\sigma_t^2(d-r)$ on bare noise prediction, regardless of downstream width, depth, or expressivity, because the missing Gaussian components cannot be inferred from the retained input. Computation after the compression cannot remove this error, whereas a path carrying the intact input can bypass the restriction. With the $768\rightarrow128\rightarrow768$ embedding, the trained noise predictor's per-coordinate MSE is 300.836 against the predicted floor of 300.833 at $t=0.05$ and 0.83387 against 0.83333 at $t=0.5$ (Figure 2b), and our response to Reviewer jynx (W2/Q1) extends this to five encoder ranks and five noise levels. The cost of the identity term comes from an information constraint rather than from the difficulty of learning a high-dimensional function. **Enough width to transport the input need not be enough to support denoising corrections.** Theorem 5 identifies the constraint behind the width-768 failure: at width $d_p$, exact affine input recovery leaves room only for affine denoising readouts, so nonlinear or contextual corrections require additional coordinates or come at the cost of input recovery. The number of additional coordinates is set by the covariance spectrum of the residual beyond the best affine denoiser. These results give empirical predictions more specific than a general benefit from widening. In Figure 4a, noise prediction at width $d_p+c$ corresponds approximately to image prediction at width $c$, with the offset fixed by patch geometry, and much of the sharp widening benefit persists when attention, MLP, and conditioning dimensions remain fixed. Together these findings identify a representation overhead and show that accommodating it need not require proportionally enlarging every internal computation.2. RAE: input information can be lost even when output expressivity is unrestricted
RAE’s Theorem 1 considers networks of the form , and its proof derives a loss floor by restricting predictions to the column space of the final linear map (RAE, Appendix B.1). Our result identifies a different obstruction: information discarded at the input cannot be recovered even when subsequent computation and outputs are unrestricted.
Consider a patch embedding of the form
followed by full-width Transformer and a full-rank output head. The embedding compresses the input, but the nonlinear decoder after it need not produce outputs confined to a 128-dimensional linear subspace, so RAE’s output-subspace bound does not establish failure for this architecture. RAE’s Appendix B.1 notes that when , its argument gives “a trivial lower bound in Eq 5.” Here, denotes model width and denotes input dimension. Our example has , so their spectral-tail bound is even though the effective input map has rank only 128.
Our bound depends on the input rank, and expanding the representation back to 768 dimensions does not restore the Gaussian information discarded by the initial projection. Theorem 1 and Corollary 2 imply that bare noise prediction incurs an induced clean-image error of at least
per patch, regardless of the width, depth, or expressivity of the downstream decoder. Capacity added after the input bottleneck cannot remove this floor. The model must preserve the missing input information, supply it through another path, or change the prediction parameterization. Our result isolates an information-theoretic limitation that persists after the output-expressivity restriction underlying RAE’s bound has been removed.
3. AsymFlow: reducing transmitted noise versus quantifying its representation cost
AsymFlow recognizes the burden of transmitting noise through learned features. Its asymmetric target reduces the number of noise directions the network must predict, and a analytic reconstruction supplies the remaining contribution. We address a question this construction leaves unresolved: when full noise prediction is retained, how much representation capacity does noise transmission consume, and how much additional capacity is needed for denoising?
Preserving all input dimensions does not settle this question. Consider a model whose width equals the patch dimension, , so there need be no input information loss. AsymFlow’s analysis does not establish whether this width suffices for nonlinear denoising, and it does not characterize the additional width required. Under affine input recovery and an affine output head, Theorem 5 shows that width permits only affine denoising. At width , the additional feature directions support corrections beyond the best affine denoiser, with attainable error determined by the covariance spectrum of those corrections.
AsymFlow’s noise-subspace rank and our additional feature width describe different quantities. The former controls how much noise enters the learned target, and the latter controls denoising capacity while retaining full noise prediction. The architectural prescriptions differ as well. AsymFlow reduces the transmission burden by changing the prediction target, whereas our results support accommodating that burden by widening the residual representation while keeping attention, MLP, and conditioning dimensions fixed. This gives practitioners who retain noise prediction a design option.
4. EDM and exact preconditioning: which shortcut removes the cost
EDM combines a noisy-input skip with a learned output using noise-dependent coefficients, and JiT relates its benefit to approaching image prediction at high noise. Theorem 3 and Corollary 4 make this quantitative for any linear combination of image and noise targets with a scalar shortcut: the discarded-noise contribution is , so what matters is the noise coefficient remaining in the learned target, not the presence of a shortcut. For EDM the contribution decays to zero at high noise but remains at finite noise: at it is 0.03333 per coordinate, and EDM's measured MSE is 0.04747 against 0.01570 for image prediction (Figure 2b). Exact preconditioning, , is the unique coefficient that removes the contribution at every noise level. It is image-prediction-equivalent by design and serves as our control (measured MSE 0.01571). The theory's predictive content lies in the non-equivalent cases: it quantifies EDM's remaining term, and when the shortcut coefficients are learned, training drives the effective skip to the cancellation value from bare-noise and additive initializations ( over ; Figure 6).
We thank the reviewer for the careful assessment. Section, figure, table, and appendix references below are to the revised manuscript unless attributed to another paper. The recovery values quoted in W2 (0.7785 and 0.9965) come from the submitted probe protocol, which the revision refreshes (Figure 5; Appendix B.2.2).
Prior work recognizes that predicting high-dimensional noise is hard. Our contribution separates the representation needed to transport the noisy input from the representation needed for denoising corrections, and this yields quantitative predictions about architectural failure, required width, and effective capacity allocation. The revised introduction lists our contributions and compares them with JiT, RAE, EDM, and AsymFlow immediately afterwards (Section 1).