acceptodds
Under review as a conference paper at ICLR 2027

Polynomial Denoisers on the Image Manifold: Volterra Networks for Self-Contained Pixel Diffusion

Abstract

Natural images are often conjectured to lie on a low-dimensional manifold within the high-dimensional pixel space. Recent work leverages this to build self-contained pixel-space diffusion models without specialized machinery or pretrained components. A plain Transformer on raw image patches suffices, provided it predicts the clean image rather than the noise and compresses each patch through an aggressive bottleneck. The mixing operator, however, has remained softmax attention, inherited from sequence modeling rather than derived from denoising. For Gaussian noise, the optimal denoiser admits a canonical expansion in polynomial functionals of the observation, the Wiener–Volterra series. We prove that softmax attention, and any normalized mixing rule (including linear attention), cannot realize any polynomial map beyond an affine one. In contrast, Volterra layers realize this class directly and, on a data manifold, achieve approximation rates governed by the manifold dimension rather than the ambient pixel dimension. We therefore propose the architecture these results prescribe; raw image patches passed through a bottleneck embedding, followed by cascaded low-rank Volterra layers, which carry no activation function because each layer is nonlinear in its own right. To our knowledge this is the first pixel-space diffusion architecture whose mixing operators are Volterra layers throughout, with no attention, no convolution, no pretrained components, and no activation function on the residual path. On ImageNet and , the resulting model is competitive with self-contained pixel-space diffusion at matched parameter and FLOP budgets.

Primary area:
generative models
Keywords:
Polynomial Denoisers, Image Generation, Diffusion, Denoising

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2PJPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method that use the Hu's model as a backbone to estimate true image directly (instead of estimate noise) for diffusion model. It proposed to replace their transformer's token mixer and channel mixer with Volterra series. They also provide a dimension approximation rate for the optimal denoiser for the data on the manifold.

Critical strengths

It is new to use Volterra series in transformer to approximate true images for the diffusion model.

Critical weaknesses
  1. The paper uses flow-matching method, however, it substantially discusses the existence of approximation for the score ft(x)f_t(x), which is not needed for the flow-matching method. Also, the approximation rate proof is not novel enough. It follows from classical polynomial approximation/Stone-Weierstrass arguments and Tweedie’s formula has been substantially discussed.
  2. I feel the role of the manifold in the paper is quite limited. Especially when compared to this paper (not cited): Z. Zhang et al. Diffusion Model for Manifold Data: Score Decomposition, Curvature, and Statistical Complexity, https://arxiv.org/abs/2603.20645 Another relevant reference that is not cited is: M Chen, et. al. Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data, ICLR 2023, https://arxiv.org/abs/2302.07194
Questions for authors

For the diffusion model, the training step runs only for one randomly selected tt, the paper claims that their method is not an "analogy with sequence modeling." So, what is the different from the classical image denoising?

2 comments
AuthorsOpenReview ↗

Response to 2PJP

Thank you for the review and the references.

W1. "The paper uses flow-matching method, however, it substantially discusses the existence of approximation for the score, which is not needed for the flow-matching method."

We agree that a score approximation is not needed, and the paper does not use one. The analysis is about the flow-matching minimizer itself.

  • What the analysis covers.
    • Our network predicts the clean image and forms the velocity from it.
    • The flow-matching loss is minimized when that prediction equals the denoiser ft(z)=E[x∣zt=z]f_t(z)=\mathbb{E}[x\mid z_t=z].
    • Derivation. With zt=tx+(1−t)ϵz_t=tx+(1-t)\epsilon, the velocity target is v=x−ϵ=(x−zt)/(1−t)v=x-\epsilon=(x-z_t)/(1-t). Its mean given zt=zz_t=z is (ft(z)−z)/(1−t)(f_t(z)-z)/(1-t). The squared loss is minimized when vθv_\theta equals this mean, that is, when the predicted clean image equals ft(z)f_t(z).
    • The paper states this in the first paragraph of Section 3.1 and derives it in Appendix B.2.
  • What the results cover. Proposition 1 is about ftf_t. Propositions 2 and 3 are about which networks can represent its polynomial approximants.
  • Where the score appears. In the main text it appears in one sentence after Proposition 1, only to note that the same rate holds if the network is parametrized by noise, velocity or score instead. Appendix B.2 lists it as one of four equivalent targets.

We make this explicit at the start of Section 3.

W2. "Also, the approximation rate proof is not novel enough. It follows from classical polynomial approximation/Stone-Weierstrass arguments and Tweedie's formula has been substantially discussed."

Agreed, and the paper credits these tools. The new results are what is built on top of them.

  • What is classical, and credited.
    • Section 2.2 credits the polynomial rates on manifolds to Ragozin (1970).
    • Tweedie's formula (Robbins 1956; Efron 2011) is used only for the Jacobian identity.
  • What is new.
    1. Through the linear bottleneck that pixel diffusion models already use, the denoiser is approximated by a polynomial in a low-dimensional variable, at a rate set by the intrinsic dimension.
    2. Stacks of low-rank Volterra layers realize these polynomials exactly.
    3. A softmax attention layer, or any mixing layer whose weights are nonnegative and sum to one, cannot realize a polynomial beyond an affine one. In a Transformer, the polynomial content must therefore come from the feed-forward layers (Appendix B.6).
  • Results 2 and 3 turn the approximation result into an architecture.

W3. "I feel the role of the manifold in the paper is quite limited." (with reference to Zhang et al., 2026, and Chen et al., 2023)

Thank you, both papers are now cited. They study a complementary question, and the connection to our analysis is direct.

  • Different questions. Chen et al. (ICML 2023) and Zhang et al. (2026) ask how well a network can learn the score on a subspace or manifold, and from how many samples. We ask which architecture matches the structure of the denoiser.
  • The connection.
    • On a linear subspace VV, the setting of Chen et al., the denoiser depends only on the projection onto VV, so a bottleneck that projects onto VV loses nothing.
    • Reason. For x∈Vx\in V, ∣z−tx∣2=∣PVz−tx∣2+∣PV⊥z∣2\\|z-tx\\|^2=\\|P_Vz-tx\\|^2+\\|P_V^\perp z\\|^2. The second term does not depend on xx, so it cancels from the posterior.
    • Our Lemma 1 bounds what the bottleneck loses on a curved manifold.
    • Zhang et al. decompose the score separately at large and small noise, with a curvature term at small noise. This parallels the noise-level regimes of our Appendix B.3.
  • Where the manifold is used.
    • It sets the approximation rate through its dimension and reach (Proposition 1, Lemma 1).
    • It explains why clean-image prediction works through a bottleneck while noise and velocity prediction do not (Appendix B.2).
    • It is the controlled variable of the study in Section 5.1.

Q. "For the diffusion model, the training step runs only for one randomly selected [t]... So, what is the different from the classical image denoising?"

Each training step is a classical denoising problem. That is exactly why we derive the architecture from denoising theory rather than from sequence modeling.

  • Training at one random noise level per step is a stochastic estimate of the average over noise levels. Because the network is given the noise level, the optimum is the best denoiser at every level.
  • What differs from classical denoising is the regime.
    • One network must cover all noise levels, including near-pure noise, where it acts as a generative prior rather than a local smoother.
    • Its errors accumulate along the sampling trajectory.
    • It is class-conditional and used with guidance.
  • Appendix B.3 tracks our bounds across all noise levels.

Dear Reviewer,

We sincerely appreciate your time and valuable feedback. We have carefully addressed your concerns through detailed clarifications and additional information in our rebuttal.

As the discussion period is approaching its end, we would be grateful if you could take a moment to review our responses and let us know whether your concerns have been adequately addressed.

Thank you again for your time.

Reviewer_NRenOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a theoretical with the mixing operation in transformers, the softmax. They propose replacing components with a theoretically motivated Volterra layer instead. They report better FID than JiT across a couple model scales at comparable compute.

The main contribution is the architecture and the empirical results - they demonstrate that the new architecture, at minimum, doesn't degrade the performance, and it does seem to improve things.

Validation occurs through generative metrics.

Critical strengths

The main strengths are: consistent empirical gains, and a novel architecture.

The model demonstrates consistent improvements over scale (although I have mild concerns over sequence length). It beats the baseline with lower parameters and comparable FLOPs at all sizes.

The architecture is novel, and a simple enough. It replaces both attention and ffn with their Volterra mixers, which shows great promise.

Critical weaknesses

There is an important missing ablation, self attention + FFN. I would like to see the results of this at the same scale as the other ablations in that table.

Questions for authors

Table 2 is missing a cell - I would like to see that provided. That would be self-attention + FFN.

Training hyperparameters are not mentioned at all in the paper. I would like to see this reported.

It seems like the token mixer has quadratic scaling as a function of sequence length - can the authors confirm this, and discuss how they feel longer sequences should be approached?

2 comments
AuthorsOpenReview ↗

Response to NRen

Thank you for the positive assessment and the concrete requests.

Q1. "Table 2 is missing a cell... That would be self-attention + FFN."

All four cells of the 2×2 are now reported.

Self-attention + FFN was already reported in Table 2, and the self-attention + Volterra channel mixer cell was missing, and has been added now.

Token mixerChannel mixerFID-50K
Self-attentionFFN3.66
VolterraFFN3.54
Self-attentionVolterra3.49
VolterraVolterra3.41
  • All cells use the recipe of Q2-B.
  • Swapping in a Volterra mixer helps at either position: 3.54 for the token mixer and 3.49 for the channel mixer, against 3.66 for the self-attention + FFN baseline. The two gains are complementary, and with both swapped PoD-B/16 reaches the best FID, 3.41.
  • This table replaces Table 2 in the paper.

Q2. "Training hyperparameters are not mentioned at all in the paper."

While some details are provided in Sec. 5.2 and App. E, we share the full configuration below. This is now included in the paper.

  • Note: Empty sections in L,H,G indicate that the same config as B is utilized.
BLHG
depth / width12 / 76824 / 102432 / 128040 / 1664
patch size / tokens16 / 256 at 256 px; 32 / 256 at 512 px
bottleneck dim.128128256256
token rank; channel rank10; 1
quadratic-branch scale init0.1
in-context class tokensnone
optimizerAdamW, β = (0.9, 0.95), weight decay 0
learning ratebase 1.25e-5, scaled by batch/256 (5e-5 at batch 1024)
batch, warmup1024, 5 epochs
EMA decay0.9999
precisionbfloat16 autocast
epochs800
time samplinglogit-normal(−0.8, 0.8)
label dropout; velocity clip0.1; max(1 − t, 0.05)
samplerHeun, 50 steps, final step Euler, EMA weights
CFG2.9
evaluationFID and IS on 50K samples, ADM reference statistics

Q3. "It seems like the token mixer has quadratic scaling as a function of sequence length - can the authors confirm this, and discuss how they feel longer sequences should be approached?"

Yes. The token mixer is quadratic in sequence length, in both cost and parameters, so longer sequences cost more. Reducing this cost is left to future work. Currently, we handle larger resolutions using larger patches and keeping the same token length.

Dear Reviewer,

We sincerely appreciate your time and valuable feedback. We have carefully addressed your concerns through detailed clarifications and additional information in our rebuttal.

As the discussion period is approaching its end, we would be grateful if you could take a moment to review our responses and let us know whether your concerns have been adequately addressed.

Thank you again for your time.

Reviewer_wtDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Polynomial Denoisers (PoD), a self-contained pixel-space diffusion architecture that replaces Transformer self-attention and feed-forward layers with low-rank Volterra layers that model higher-order polynomial interactions. The authors theoretically motivate this design through polynomial approximation of optimal denoisers on low-dimensional data manifolds and show that normalized attention cannot represent nonlinear polynomial maps beyond affine ones under their stated conditions.

Critical strengths
  • This paper introduces Volterra-based polynomial mixers as a new alternative to attention and feed-forward networks for pixel-space diffusion.
  • Provides approximation-theoretic analysis connecting polynomial denoisers to the intrinsic dimensionality of the data manifold.
  • The proposed method requires no pretrained tokenizer or external pretrained components, simplifying the overall modeling pipeline.
Critical weaknesses
  • The theoretical scope is limited. The analysis focuses on individual patches and does not establish the same approximation guarantees for full-image denoising.
  • The polynomial expressivity guarantee is established for a normalization-free model, whereas the proposed architecture uses RMSNorm. It would be better to further discuss the theory-architecture gap.
  • In the experiments, practical justification of the proposed method is limited. The paper acknowledges that the theoretical representability threshold is impractically large, and it does not show that practical low-rank configurations achieve the claimed approximation guarantees.
Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to wtDq part 1/3

Thank you for the precise review.

W1. "The analysis focuses on individual patches and does not establish the same approximation guarantees for full-image denoising."

Short answer: Proposition 1's guarantee holds for the full image. The full-image denoiser has the same polynomial structure as the patch denoiser, at the same rate K−2sK^{-2s}, in a few linear features of all the patch codes (Proposition 4 below), and the architecture can represent it (Corollary 2). For any weights, including trained ones, the next comment splits the full-image error exactly into the per-patch error and a context error that only the token mixers can reduce (Proposition 5).

Proposition 4 (Full-image version of Proposition 1). Let images x=(x1,…,xN)\boldsymbol x=(x_1,\dots,x_N) lie on a compact C∞C^\infty submanifold Mimg⊂MN\mathcal M_{\rm img}\subset\mathcal M^N of dimension dimgd_{\rm img}, with law ρimg\rho_{\rm img}, and let z=tx+σϵ\boldsymbol z=t\boldsymbol x+\sigma\boldsymbol\epsilon with σ=1−t\sigma=1-t and ϵ∼N(0,I)\boldsymbol\epsilon\sim\mathcal N(0,I). Let Ft(z)=E[x∣z]F_t(\boldsymbol z)=\mathbb E[\boldsymbol x\mid\boldsymbol z] be the full-image denoiser and π(z)=(π(z1),…,π(zN))\boldsymbol\pi(\boldsymbol z)=(\pi(z_1),\dots,\pi(z_N)) the stacked patch codes, with π\pi as in Proposition 1. (a) FtF_t is real-analytic on RND\mathbb R^{ND}. (b) There is a linear map Ψ:RNd′→R2dimg+1\Psi:\mathbb R^{Nd'}\to\mathbb R^{2d_{\rm img}+1} whose composition with π\boldsymbol\pi restricts to a smooth embedding of tMimgt\mathcal M_{\rm img}, such that for every s>0s>0 and degree KK there is a polynomial QKQ_K with sup⁡z∈tMimg∥Ft(z)−QK(Ψπ(z))∥2 ≤ CimgK−2s,\sup_{\boldsymbol z\in t\mathcal M_{\rm img}}\big\|F_t(\boldsymbol z)-Q_K\big(\Psi\boldsymbol\pi(\boldsymbol z)\big)\big\|^2\ \le\ C_{\rm img}K^{-2s}, where CimgC_{\rm img} depends on ss, tt, the geometry of tMimgt\mathcal M_{\rm img} and of its embedding Ψ∘π\Psi\circ\boldsymbol\pi, and ∥Ft∥Cs\|F_t\|_{C^s}.

In brief.

  • Proposition 1, one level up. Proposition 1 says the patch denoiser is, up to CK−2sCK^{-2s}, a degree-KK polynomial in a few linear features of the patch. Proposition 4 says the same of the full-image denoiser, with linear features of all the patch codes. The exponent is unchanged. The constant and the number of features, 2dimg+12d_{\rm img}+1 rather than 2dM+12d_{\mathcal M}+1, are set by the image manifold.
  • Why it works. Proposition 1 needs only a linear map that embeds the data manifold. The bottleneck π\pi embeds every patch, so the stacked codes embed the whole image, and one more linear map Ψ\Psi reduces them to 2dimg+12d_{\rm img}+1 numbers.
  • Where the token mixer comes in. Ψ\Psi combines codes from different patches, which no per-patch computation can do. In our network, the token mixers are the only cross-patch step. Without them, the network stays a data-dependent distance from FtF_t, the value of context (Proposition 5).
  • Realizability (Corollary 2). For depth L≥4L\ge4 and width above a threshold Pimg\mathsf P_{\rm img}, the depth-LL networks on the whole image contain Q∘Ψ∘πQ\circ\Psi\circ\boldsymbol\pi for every polynomial QQ of degree KL−3K_{L-3}, so they reach FtF_t within 26sCimgKL−2s2^{6s}C_{\rm img}K_L^{-2s}, the exponent of Proposition 2(i). The token and channel mixers compute Ψπ\Psi\boldsymbol\pi, with the positional embedding supplying the patch-dependent weights, and the stacks of Proposition 2(i) compute QQ. As with P0\mathsf P_0, the threshold is a certificate, not a design (see W3). The argument in our answer to W2 carries it over to RMSNorm.

Proof of Proposition 4.

  • (a) As in the proof of Proposition 1(a), with ρimg\rho_{\rm img} in place of ρdata\rho_{\rm data}: the denoiser of a compactly supported prior under Gaussian noise is real-analytic.
  • Embedding. π\boldsymbol\pi is injective and immersive on tMimg⊂(tM)Nt\mathcal M_{\rm img}\subset(t\mathcal M)^N, because π\pi is on tMt\mathcal M (Proposition 1(b)). By compactness it is an embedding. Whitney's projection argument, as in Step 1 of the proof of Proposition 1, gives a linear Ψ\Psi into R2dimg+1\mathbb R^{2d_{\rm img}+1} for which Ψ∘π\Psi\circ\boldsymbol\pi is still an embedding.
  • (b) Steps 1 to 3 of the proof of Proposition 1(b) use only that the linear map embeds the manifold and that the target is smooth. Applied to FtF_t and Ψ∘π\Psi\circ\boldsymbol\pi, they give QKQ_K.

In the paper: Propositions 4 and 5 and Corollary 2, with the explicit construction, in the new Appendix C.8, summarized in Section 3.

AuthorsOpenReview ↗

Response to wtDq part 2/3

W1, continued: what holds for any weights. Proposition 4 is about the target denoiser, and Corollary 2 is an existence result. For a given network, trained or not, the following identity says exactly where its full-image error comes from.

Proposition 5 (Error split). Take the noise model of Proposition 4 and any image distribution whose patches have law ρdata\rho_{\rm data}. The denoiser of Proposition 1, ft(zi)=E[xi∣zi]f_t(z_i)=\mathbb E[x_i\mid z_i], sees patch ii alone. The full-image denoiser at patch ii, Fi=E[xi∣z]F_i=\mathbb E[x_i\mid\boldsymbol z], sees the whole image, and ft(zi)=E[Fi∣zi]f_t(z_i)=\mathbb E[F_i\mid z_i]. Let δFi=Fi−ft(zi)\delta F_i=F_i-f_t(z_i) and Δctx=E∣δFi∣2=mmsepatch−mmseimg.\Delta_{\rm ctx}=\mathbb E\\|\delta F_i\\|^2=\mathrm{mmse}_{\rm patch}-\mathrm{mmse}_{\rm img}. For a network with output GiG_i at token ii, let hi(zi)=E[Gi∣zi]h_i(z_i)=\mathbb E[G_i\mid z_i] and δGi=Gi−hi(zi)\delta G_i=G_i-h_i(z_i). Then, for any weights and every t∈(0,1)t\in(0,1):

(i) E∥Fi−Gi∥2=Epatch+Ectx\mathbb E\|F_i-G_i\|^2=\mathcal E_{\rm patch}+\mathcal E_{\rm ctx}, where Epatch=E∥ft(zi)−hi(zi)∥2\mathcal E_{\rm patch}=\mathbb E\|f_t(z_i)-h_i(z_i)\|^2 and Ectx=E∥δFi−δGi∥2\mathcal E_{\rm ctx}=\mathbb E\|\delta F_i-\delta G_i\|^2.

(ii) With the token mixers off, δGi=0\delta G_i=0 and Ectx=Δctx\mathcal E_{\rm ctx}=\Delta_{\rm ctx}.

(iii) In general Ectx=Δctx−Γ\mathcal E_{\rm ctx}=\Delta_{\rm ctx}-\Gamma with Γ=2a−b\Gamma=2a-b, a=E⟨δFi,δGi⟩a=\mathbb E\langle\delta F_i,\delta G_i\rangle, b=E∥δGi∥2b=\mathbb E\|\delta G_i\|^2. If b>0b>0, then Γ≤a2/b≤Δctx\Gamma\le a^2/b\le\Delta_{\rm ctx}, with equality in the first when a=ba=b.

(iv) Δctx→0\Delta_{\rm ctx}\to0 as t→1t\to1 and as t→0t\to0, and it is not a function of patch-level quantities.

In brief.

  • Δctx\Delta_{\rm ctx} is the value of context: how much better the best estimate of a patch gets when the rest of the image is seen. It is fixed by the data.
  • Epatch\mathcal E_{\rm patch} is the per-patch error that Propositions 1 and 2(i) address.
  • Token mixers off: the network pays all of Δctx\Delta_{\rm ctx}, exactly, with no assumption on how patches depend on each other.
  • Token mixers on: it pays Δctx−Γ\Delta_{\rm ctx}-\Gamma. Γ\Gamma is at most the share of Δctx\Delta_{\rm ctx} that the token mixers' contribution δGi\delta G_i explains. This is an identity, not a guarantee: for arbitrary weights Γ\Gamma can be negative. Training raises it, because Γ\Gamma enters the loss with a minus sign.
  • Patch-level facts alone cannot settle it: duplicated and independent patches have the same patch statistics, yet Δctx>0\Delta_{\rm ctx}>0 for the first and =0=0 for the second. That is why Proposition 4 needs an image-level assumption.

Proof. All expectations are conditional on the class label. If patch statistics vary with position, use the law of patch ii and its own denoiser.

  • Value of context. xi−ft(zi)=(xi−Fi)+δFix_i-f_t(z_i)=(x_i-F_i)+\delta F_i. The first term has mean zero given z\boldsymbol z, and the second is a function of z\boldsymbol z, so mmsepatch=mmseimg+E∣δFi∣2\mathrm{mmse}_{\rm patch}=\mathrm{mmse}_{\rm img}+\mathbb E\\|\delta F_i\\|^2.
  • (i). Fi−Gi=(ft(zi)−hi(zi))+(δFi−δGi)F_i-G_i=\big(f_t(z_i)-h_i(z_i)\big)+(\delta F_i-\delta G_i). By the tower rule E[δFi∣zi]=0\mathbb E[\delta F_i\mid z_i]=0, and E[δGi∣zi]=0\mathbb E[\delta G_i\mid z_i]=0 by definition of hih_i, so the two brackets are orthogonal.
  • (ii). With token gates at zero, GiG_i depends on ziz_i only, so Gi=hi(zi)G_i=h_i(z_i).
  • (iii). Expanding, Ectx=Δctx−2a+b\mathcal E_{\rm ctx}=\Delta_{\rm ctx}-2a+b, and a2/b−Γ=(a−b)2/b≥0a^2/b-\Gamma=(a-b)^2/b\ge0. Cauchy–Schwarz gives a2≤Δctxba^2\le\Delta_{\rm ctx}b. The loss is mmseimg+Epatch+Δctx−Γ\mathrm{mmse}_{\rm img}+\mathcal E_{\rm patch}+\Delta_{\rm ctx}-\Gamma.
  • (iv). Δctx≤mmsepatch≤Dσ2/t2→0\Delta_{\rm ctx}\le\mathrm{mmse}_{\rm patch}\le D\sigma^2/t^2\to0 as t→1t\to1. As t→0t\to0, z\boldsymbol z carries no information, so Δctx→0\Delta_{\rm ctx}\to0. If x1=x2x_1=x_2, the two observations average to one at noise σ/2\sigma/\sqrt2, so Δctx=mmse(σ)−mmse(σ/2)>0\Delta_{\rm ctx}=\mathrm{mmse}(\sigma)-\mathrm{mmse}(\sigma/\sqrt2)>0. Independent patches give Δctx=0\Delta_{\rm ctx}=0.
AuthorsOpenReview ↗

Response to wtDq part 3/3

W2. "The polynomial expressivity guarantee is established for a normalization-free model, whereas the proposed architecture uses RMSNorm."

The approximation guarantee carries over to the architecture with RMSNorm. Only exact equality of the function classes does not.

In brief.

  • Proposition 1 is about the target denoiser, not the network, so RMSNorm cannot affect it.
  • For Proposition 2(i), hold one channel at a large constant λ\lambda.
    • Every token's norm is then dominated by λ\lambda, so RMSNorm becomes almost a fixed rescaling, which the AdaLN scale undoes.
    • As λ\lambda grows, the RMSNorm network converges to the normalization-free one.
    • So whatever the normalization-free class can approximate, the RMSNorm architecture can too.
  • A numerical check on a random 4-block PoD stack confirms this. With outputs of size about 2.7, the maximum error is 0.068, 0.0061 and 0.00065 at λ=102,103,104\lambda=10^2,10^3,10^4, falling as 1/λ1/\lambda.
  • This answers Appendix B.5. Exact containment cannot hold: the final RMSNorm bounds the output of every network in the class, and a non-constant polynomial is unbounded. Approximation on compact sets, where the data lives, does hold.

Proof.

  • Setup. The embedding bias sets one reserved channel to λ\lambda, plus O(1)O(1) from the positional embedding. It affects nothing else and nothing changes it: the channel mixers' factors have zero columns for it, the token mixers act channel by channel, and the per-channel residual gates are zero on it.
  • One token. For a token u=(λ+O(1),v)u=(\lambda+O(1),v), RMSNorm followed by the AdaLN scale λ/d\lambda/\sqrt d gives, on the other channels, v(1+(∥v∥2+O(λ))/λ2)−1/2=v(1+O(1/λ)),v\big(1+(\|v\|^2+O(\lambda))/\lambda^2\big)^{-1/2}=v(1+O(1/\lambda)), uniformly for vv in a compact set. The same holds in the final layer.
  • Whole network. Each sublayer is polynomial, hence Lipschitz on compact sets. By induction, the normalized network converges uniformly to the normalization-free one as λ→∞\lambda\to\infty.
  • Conclusion. inf⁡gsup⁡tM∥ft−g∥2≤C KL−2s\inf_g\sup_{t\mathcal{M}}\|f_t-g\|^2\le C\,K_L^{-2s} also holds over the RMSNorm architecture, approached as λ→∞\lambda\to\infty and at the cost of one channel. Like Proposition 2(i), this is a statement about what the architecture can represent.

In the paper: Corollary 1, a corollary to Proposition 2(i) in the revised Appendix B.5, stating the distinction between approximation and exact equality.

W3. "The paper acknowledges that the theoretical representability threshold is impractically large, and it does not show that practical low-rank configurations achieve the claimed approximation guarantees."

Short answer: the threshold is a worst-case certificate, not the width actually needed.

  • Why the threshold overstates the need. It is the width needed to represent every polynomial of a given degree. The denoiser needs only one of them, so much smaller widths can suffice. The paper notes that it is not a practical width (Remark 8).
  • Existing evidence (Section 5.1). With a practical low-rank configuration, the Volterra block stays closest to the manifold at every ambient dimension. It is two to five times closer than MLP and attention as the dimension grows 256-fold.

Dear Reviewer,

We sincerely appreciate your time and valuable feedback. We have carefully addressed your concerns through detailed clarifications and additional proofs in our rebuttal.

As the discussion period is approaching its end, we would be grateful if you could take a moment to review our responses and let us know whether your concerns have been adequately addressed.

Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their time and feedback. They noted these strengths:

  • Volterra mixers as a new alternative to attention and feed-forward layers (wtDq, NRen, 2PJP);
  • a novel and simple architecture (NRen);
  • gains over JiT that are consistent across scales, with fewer parameters (NRen);
  • an approximation analysis that ties polynomial denoisers to the intrinsic dimension of the data (wtDq);
  • a pipeline free of tokenizers and pretrained components (wtDq).

Summary of our responses

  • Theory stated per patch (wtDq).
    • Proposition 1's guarantee holds for the full image. The full-image denoiser is, at the same rate K−2sK^{-2s}, a polynomial in a few linear features of all the patch codes (new Proposition 4).
    • The architecture can represent it, with the token mixers supplying the cross-patch step (new Corollary 2).
    • For any weights, the full-image error splits exactly into the per-patch error and a context error that only the token mixers reduce (new Proposition 5).
    • Proposition 2(ii) is already proved at image level.
  • RMSNorm (wtDq). The approximation guarantee holds for the RMSNorm architecture that we train (new Corollary 1). Only exact equality of the function classes is specific to the normalization-free model, because the final RMSNorm bounds the output.
  • Practical widths (wtDq). The width threshold is a worst-case certificate: it represents every polynomial of a given degree, while the denoiser needs only one. In Section 5.1, practical low-rank configurations already stay closest to the manifold.
  • Missing ablation cell (NRen). The 2×2 ablation at B/16 is now complete. A Volterra mixer helps at either position, and PoD, with both, is best (FID 3.41, against 3.66 for self-attention + FFN).
  • Hyperparameters (NRen). We give the full training and sampling configuration for B, L, H and G. It will be added to the paper as a table.
  • Sequence length (NRen). The token mixer is quadratic in sequence length. At 512 px we use larger patches to keep the same number of tokens. Reducing this cost is left to future work.
  • Flow matching and novelty (2PJP).
    • No result in the paper relies on approximating the score. The analysis concerns the minimizer of the flow-matching loss itself (Section 3.1, first paragraph). In the main text, the score appears in one sentence, as a remark after Proposition 1.
    • The classical tools are credited in Section 2.2. The new results are built on them: the polynomial approximation through the linear bottleneck at a rate set by the intrinsic dimension, its exact realization by low-rank Volterra stacks, and the limit on attention.
    • Chen et al. (ICML 2023) and Zhang et al. (2026) are now cited. They study score learning on subspaces and manifolds, a complementary question, and we state the connection.
  • Presentation (NRen, 2PJP). We add a notation summary to Section 3.

All new arguments, results and tables are added to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.