acceptodds
Under review as a conference paper at ICLR 2027

How Do DiTs Denoise? An Aggregator-Predictor View of Attention and MLPs

Abstract

How do diffusion transformers (DiTs) represent image content and learn rules for reusing it to generalize? We develop an aggregator-predictor view of DiTs as input-adaptive local denoisers, in which context aggregation and patch prediction provide functional roles for interpreting attention and MLPs. We first establish that, for Gaussian Markov random field (GMRF) priors, the global Bayes denoiser can be approximated using only a subset of context patches, with error decaying exponentially in context radius. We further show that patchwise linear denoisers, often invoked to explain diffusion generalization, recover the empirical local denoiser with sufficiently informative aggregation. Within this view, AdaLN-modulated attention naturally parameterizes an input- and noise-dependent aggregator, while the MLP admits a learned key-value representation for tokenwise patch prediction. Our experiments support this view. DiTs trained on real images exhibit noise-dependent denoising contexts resembling those in the analytic GMRF setting, while attention mass exhibits a qualitatively corresponding broadening pattern. In a controlled CelebA64 study, shared-memory patch predictors used in place of DiT MLPs achieve competitive denoising and generative performance. Finally, cross-dataset component transplantation reveals compatibility patterns consistent with the proposed roles of QK in spatial weighting, VO in feature transport, and MLPs in patch prediction. Together, these results provide a component-level account of DiT denoising and motivate architecture-specific fine-tuning and controlled intervention.

Primary area:
generative models
Keywords:
diffusion transformers, diffusion models, mechanistic interpretability, denoising, representation learning, model generalization, model adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_yAKCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Gaussian Markov random field (GMRF) based statistical framework to study local denoising. Then the paper presents an aggregator-predictor interpretation of the component of attention blocks in a DiT. Finally, experiments try to establish the applicability of the GMRF framework on DiTs, explores an alternative to replace the MLPs, and try to understand the role of the components empirically by transplanting components between DiT and the same DiT’s finetuned to a different dataset.

Critical strengths
  1. The GMRF mathematical formulation for denoising behavior is interesting and could be valuable if better connected to DiTs in practice.
  2. I especially like the transplantation experiments which can have interesting implication for understanding and manipulation of generated images.
Critical weaknesses
  1. The connection between theory and practice feels very limited. Firstly, the aggregator-predictor decomposition does not directly capture multilayer residual DiTs, as the authors acknowledge. Furthermore, the GMRF analysis assumes a sparse dependency graph whose relationship to natural image dependencies is not established. Section 5.1 only shows qualitative similarities between GMRF denoising and DiT sensitivity maps (locality, noise-dependent context increase, and disconnected contexts). These behaviors are already expected of many image denoisers, so their co-occurrence provides limited validation. A stronger connection would involve deriving specific non-trivial predictions from the theory and validating them experimentally on trained DiTs.
  2. The aggregator-predictor interpretation is sound but largely follows from the standard mathematical structure of transformer layers, making its novelty unclear.
  3. SMLP's competitive performance is interesting, but sharing memory across layers is not clearly motivated by the theoretical framework. It is also unclear what this establishes about ordinary DiT MLPs.
  4. The paper is difficult to follow at times. Some mathematical notation is insufficiently explained (e.g., sf(.)). Figure 6 should be included in the main paper, as it summarizes the proposed architectural interpretation.
Questions for authors
  1. Can the authors provide a specific non-trivial prediction derived from the GMRF framework that can be tested on DiTs? Apart from the qualitative similarity shown, are there any other evidence supporting the use of this framework for the denoising of real images?
  2. Given that the aggregator-predictor roles are not functionally separable in multilayer residual DiTs (Appendix D.5), what evidence supports this interpretation of the individual components in trained DiTs?
5 comments

We thank the reviewer for the feedback and interest in the transplantation experiments. The revision clarifies two connections: how restricted context can support accurate denoising, and how the single-layer architectural interpretation motivates hypotheses tested in full multilayer DiTs. We strengthen the evidence for both and make the connection between each experiment and its motivating question explicit.

W1 / Q1 — What quantitative prediction connects the GMRF result to denoising real images?

We now directly measure how much denoising accuracy is lost when context is restricted, extending the original sensitivity analysis with an intervention on the model’s available context.

The GMRF result motivates this accuracy–context comparison. Local denoising is the conditional mean of a clean patch given restricted noisy context. Under a GMRF prior, its excess risk relative to the global denoiser admits an exponentially decaying upper bound in context radius (Proposition 3). Corollary 4 also describes how a certified context radius grows with noise over its stated range.

We examine the corresponding question empirically with released ImageNet256 JiT-B/16 and JiT-L/16 (131.3M and 459.1M parameters). We freeze each model, retain the target patch, and progressively supply nearby context, excluding other patches throughout the network. We measure denoising MSE against the clean target and compare it with full-input denoising.

For JiT-L, retaining 32 of 256 patches gives:

Noise levelMSE with 32 patchesFull-input MSEAdditional MSE
σ=1\sigma=10.01610.01510.0010
σ=4\sigma=40.05320.04360.0096

Thus, one-eighth of the image context retains much of the full-input accuracy at σ=1\sigma=1, while the same restriction incurs a substantially larger accuracy cost at σ=4\sigma=4. Correspondingly, keeping excess MSE below 0.003 requires 13 patches at σ=1\sigma=1 and 74 at σ=4\sigma=4. Figure 2 shows the full accuracy–context comparison and the context needed to meet this fixed tolerance across noise levels.

These measurements quantify both the usefulness of restricted context and its dependence on noise in pretrained real-image models. They strengthen the empirical basis for local denoising by measuring the accuracy retained when context is actually removed.

W2 — What does the interpretation add beyond standard transformer algebra?

The value of our interpretation is that it connects familiar architectural operations to population quantities, motivating testable hypotheses about what DiT components learn and how they can be reused or replaced.

We first express local denoising as context aggregation followed by patch prediction. PLD provides an established analytic estimator of these operations: fixed linear context selection and softmax retrieval from training patches. Our framework interprets PLD and DiT as alternative aggregation–prediction estimators for population local denoising. A fixed context determines a population conditional mean, but that prediction can admit different aggregation and prediction representations. DiT attention provides input-adaptive aggregation, while MLP predicts through learned key–value pairs (Figure 3).

This population connection gives us a reason to ask questions that the architectural algebra alone leaves open:

  • Can context selection transfer across models? Models trained on the same distribution share a denoising problem. We ask whether context selected by one remains useful to another in the cross-model attention test described in Q2 below.

  • Can another predictor replace MLP? Viewing MLP as one possible estimator makes its memory organization a design choice, motivating alternatives such as a bank shared across layers. This motivates the predictor replacement in full multilayer models summarized in Q2 below, with detailed comparisons in W3.

  • Can context selection be reused across related domains? Similar spatial patch dependencies suggest that source context selection may remain useful even when target appearance changes. This motivates the transplantation experiment and subsequent MetFaces allocation test described in Q2 below.

Q2 — How does the single-layer interpretation extend to multilayer residual DiTs?

We use the single-layer algebra to motivate hypotheses, then test those hypotheses in complete multilayer DiTs. The algebra identifies context weighting by QK, feature transformation by VO, and patch prediction by MLP in the single-layer construction.

Appendix B.3 describes the corresponding operations in a residual stack: attention aggregates hidden features, and the MLP maps the contextualized features to a residual update. These operations interact across depth because each layer changes the features received by subsequent layers. We therefore use the single-layer interpretation to motivate hypotheses about context usefulness, predictor replacement, and parameter compatibility, and test these hypotheses in complete multilayer DiTs.

All the following findings come from multilayer models:

  • Attention selects useful context that transfers across models (Figure 4). Viewing attention as a context aggregator motivates testing whether its selected patches remain useful to another model solving the same denoising problem. We test this by selecting patches with one model's attention and measuring another model's denoising error using that context. The selection transfers between CelebA128 MLP/SMLP models and ImageNet256 JiT-B/JiT-L. Input-adaptive attention is especially useful at low noise. Thus, the aggregation interpretation motivates a prediction that is supported across complete networks with different sizes and predictor architectures.

  • MLP can be replaced with another patch predictor. Viewing MLP as one possible estimator motivates sharing learned key–value memories across layers. We test this replacement in full multilayer models, where it retains competitive generation quality. See W3 below for the comparisons and results.

  • Source QK remains partly compatible across related domains (Figure 1). Similar spatial patch dependencies suggest that source context selection may remain useful while target appearance changes. After jointly fine-tuning QK, VO, and MLP, we transplant selected adapted groups back into the source model. Across nine transfers using U-ViT64, DiT128, and LFM256, QK-only transplants remain closer to Source, while VO+MLP transplants are closer to the jointly adapted Target in both DINO and spatial DINO distance. Recognizable source-like and target-like generations persist despite the component interactions, supporting partial compatibility in these multilayer networks. Reusing QK parameters does not require unchanged attention maps: updates to earlier layers can change the features on which later attention operates.

  • The same interpretation motivates effective fine-tuning allocations (Tables 2–3). We then test the practical implication on MetFaces by independently fine-tuning different parameter groups with matched trainable counts. Across all three backbones, QK-only adaptation performs worse, while keeping QK frozen retains FID competitive with allocation across all three groups. This supports allocating adaptation capacity toward feature transformation and patch prediction for these related-domain transfers.

MetFaces generation at epoch 1,000 (Table 3).

AllocationU-ViT64 KID ↓U-ViT64 FID ↓DiT128 KID ↓DiT128 FID ↓LFM256 KID ↓LFM256 FID ↓
QK6.83 ± 0.0758.67 ± 0.0916.91 ± 0.0781.98 ± 0.2211.52 ± 0.0980.18 ± 0.09
VO5.60 ± 0.0354.91 ± 0.2412.21 ± 0.1474.72 ± 0.139.60 ± 0.1876.77 ± 0.24
MLP5.92 ± 0.1454.47 ± 0.0812.35 ± 0.1073.64 ± 0.279.67 ± 0.1975.45 ± 0.17
VO+MLP5.89 ± 0.1954.36 ± 0.1312.75 ± 0.4473.99 ± 0.169.41 ± 0.0875.20 ± 0.10
QK+VO+MLP5.62 ± 0.1754.50 ± 0.1612.60 ± 0.1773.94 ± 0.149.49 ± 0.0675.41 ± 0.15
Full25.69 ± 0.1866.37 ± 0.1229.34 ± 0.8084.30 ± 0.2751.19 ± 0.54103.68 ± 0.56

Values are mean ± sample SD across three training seeds; KID is multiplied by 1,000. Restricted allocations have matched trainable counts within each backbone; Full updates the entire denoiser.

These findings support the empirical usefulness of the aggregator–predictor view in multilayer DiTs. The single-layer algebra motivates the questions, while the full-network experiments establish the reported transferability, replaceability, and compatibility results.

W3 — Why share predictor memory across layers, and what does SMLP establish about MLPs?

The framework views MLP as one possible patch-predictor estimator, with a particular layerwise organization of keys and values. This motivates considering other predictor designs, including a shared memory bank.

PLD predicts by retrieving clean training patches at the target position. An ordinary MLP instead retrieves learned values using coefficients determined by learned keys and GELU. We use “key–value memory” in the established interpretation of feed-forward layers, following Geva et al. (2021). Ordinary MLPs share their keys and values across patch positions within a layer, while different layers maintain separate banks. SMLP explicitly shares a bank across layers. Its evaluation tests this alternative predictor design; it does not claim that ordinary DiT MLPs already implement a layer-shared bank.

We therefore ask which properties support generation. Must memory be patch-specific? Must retrieval use normalized softmax weights? And, because prediction is repeated across depth, must each layer maintain its own bank? Table 4 organizes learned PLD, shared cross-attention, SMLP, and ordinary MLP by these properties.

The smaller CelebA64 comparison now includes two training seeds. Eight-head SMLP achieves lower FID than eight-head shared cross-attention at all three parameter budgets in both seeds. Relative to MLP, SMLP improves FID at the small and middle budgets in both seeds; the largest-budget ordering varies (Figure 9). The results suggest that memory need not be patch-specific, softmax retrieval can be restrictive, and layers can share a bank while retaining competitive generation.

We further compare 134M-parameter MLP/SMLP pairs on CelebA128, ImageNet128, and ImageNet256. SMLP achieves lower FID in all three larger-model comparisons.

DatasetMLP FID-50k ↓SMLP FID-50k ↓
CelebA12814.51 ± 0.0711.13 ± 0.04
ImageNet12864.75 ± 0.2256.93 ± 0.21
ImageNet25678.34 ± 0.1075.50 ± 0.40

Values are mean ± sample SD over three sampling seeds, using one training seed per model.

A shared bank could also support future representation alignment or memory augmentation, building on memory-based designs such as persistent-memory augmentation (Sukhbaatar et al., 2019) and GMem (Tang et al., 2025).

The tested shared-memory predictor retains competitive generation and achieves lower FID in all three larger-model comparisons, extending the evidence to diverse natural images and higher resolutions. This supports treating MLP as one estimator within a broader class whose memory organization can be changed while preserving effective patch prediction.

W4 — Clarify notation, improve the flow, and move the architectural diagram into the main paper.

We have clarified the notation throughout the revision and moved the PLD/DiT architectural comparison into the main text as Figure 3. We have also reorganized the writing so that each experiment follows its motivation from the framework, making the connection between the interpretation and the findings easier to follow.

Reviewer_M3QNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To better understand how diffusion transformers (DiT) generalize, the authors frame the common motif of an attention module followed by an MLP module into an aggregator-predictor view. The authors start by presenting a novel error bound between a spatially global denoiser and local patch denoiser (Sec 3.1) and establish the statistical quantity the local patch denoiser targets (Sec 3.2). Then the authors present an explanation of attention modules within their aggregator-predictor view (Sec 4.1), followed by a connection between MLPs and shared-memory patch predictors (Sec 4.2). The authors experimentally validate their theoretical result on a toy example in Figure 2, and establish a qualitative connection to real-world examples in Figure 3 (Sec 5.1). The authors replace MLP modules with the proposed shared-memory patch predictors (Sec 5.2). The authors ablate different components of a DiT in groupings motivated by the proposed aggregator-predictor view (Sec 5.3).

Critical strengths
  1. The paper addresses an important problem related to interpreting deep networks.
  2. Theorem 1 seems to be a novel theoretical result, and the authors experimentally confirm this result in Figure 2.
Critical weaknesses
  1. There are some concerns with novelty.

    a. The authors state “it is not clear when PLD achieves low population denoising risk or how a neural architecture can implement it” (line 150). However, the PLD is a fairly common interpretation of attention and has been explicitly implemented before (see Neighborhood Attention at CVPR’23 or Neural Nearest Neighbors Networks at NeurIPS’18).

    b. The authors state they develop an aggregator-predictor view for the DiTs, but this interpretation is already established for patchwise linear denoisers (e.g. see non-local means and video non-local Bayes). As the authors do not apply their theory directly to the DiT, instead studying the network’s receptive field using gradient information in Sec 5.1, the aggregator-predictor view does not seem novel.

    c. Theorem 3 appears to restate known results about the non-local means denoiser.

    d. The results in Sec 5.1 are computed using standard gradient-based methods, similar to methods in GradCam. This seems to be disconnected to the author's major theoretical perspective.

  2. The motivation and significance of the experimental results, and their connection to the proposed theoretical view is unclear. Said another way, I find different sections of the paper disconnected from each other.

    a. While Theorem 1 is a novel result, it is not clear to me where this result is used in the remainder of the paper. In particular, I do not understand how this perspective motivates or explains the results in Sec 5.2 and 5.3.

    b. I do not understand how the aggregator-predictor view motivates the study of replacing MLPs with shared-memory predictors (Sec 4.2).

    c. Motivated by the author’s argument in Sec 4.2, in Sec 5.2 the authors explore replacing MLPs with shared-memory predictors. The authors report that the shared MLP (SMLP) module achieves a similar denoising quality and slightly better FID score compared to a standard MLP. However, it’s not clear to me why the “explicit, layer-shared memory bank” (line 420) supports the proposed aggregator-predictor view.

    d. In Sec 5.3, the authors replace different components of the network. However, the significance of each replaced component is not clear to me. For example, the authors state “VO (101) yields fragmented textures in the natural-image-to-face transfers” (line 475). However, I do not know how this connects to the proposed aggregator-predictor view or the decomposition of a spatially global denoiser into patchwise denoisers.

    e. The authors motivate this paper as an attempt to understand how DiTs generalize, but the authors do not explicitly connect any theoretical or experimental results back to network generalization.

Questions for authors
  1. How should we interpret the results in Figure 1? I do not understand how the modular replacement of network components connects to the paper’s aggregator-predictor view.

  2. Would the authors clarify the novelty of their aggregator-predictor perspective of a DiT and their results in Theorem 3? In particular, how is this contribution meaningfully different from the standard interpretation of non-local means denoisers and/or video non-local Bayes?

  3. Would the authors clarify how their aggregator-predictor view motivates replacing an MLP layer with a module that has a learned, shared memory? In other words, how does Sec 4.2 and the experiments in Sec 5.2 connect with the aggregator-predictor view?

  4. Can the authors distinguish how their results in Sec 5.1 are different from the results in papers like GradCam and GradCam++? In particular, how do these results validate their theoretical findings?

6 comments

We thank the reviewer for the feedback. The revision clarifies how our aggregator–predictor view connects established denoising ideas to DiT components and motivates experiments on component reuse and replacement.

W1(a–c) / Q2 — What does the perspective add beyond existing denoising and attention methods, and what is the role of original Theorem 3?

[A1] Our framework connects PLD and DiT as alternative aggregation–prediction estimators for population local denoising. This connection motivates experiments about what DiT components learn and which parts can be reused or replaced.

As the reviewer pointed out, selecting context and predicting a clean patch are established ideas. Non-local means and video non-local Bayes (Arias and Morel, 2018) use similar patches for denoising. Neighborhood Attention (Hassani et al., 2023) and Neural Nearest Neighbors (Plötz and Roth, 2018) provide concrete ways to restrict or learn context selection. We discuss these precedents in Section 2 and Appendix F.

A locality account is also established for diffusion generalization (Kamb and Ganguli, 2025; Niedoba et al., 2025; Lukoianov et al., 2025). We express such locality as local denoising, the conditional mean of a clean patch given restricted noisy context. We justify it as an estimation target in [A2] and propose a framework that decomposes local denoising into aggregation and prediction, providing population-level targets for interpreting DiT computations. A fixed context determines a population conditional mean, but that prediction can admit different aggregation and prediction representations. This motivates the hypotheses below.

We then interpret the analytic PLD model as an estimator of population aggregation and prediction (Figure 3). For each position and noise level, it applies the same fixed linear context-selection rule to every noisy image and retrieves clean training patches using softmax weights. Original Theorem 3, now Proposition 5 in Appendix A.5, states when this equals local denoising under the empirical training-image law. Its purpose is to identify the target estimated by PLD.

We then ask how DiT estimates these operations more flexibly. Its attention can select different contexts for different noisy inputs, and its MLP predicts through learned key–value pairs. Figure 3 places PLD and DiT side by side to make this connection visible. The architectural interpretation follows from their algebra, and its connection to population local denoising motivates the following tests:

  • Input-adaptive attention [A3]: attention can adapt context to each noisy input, motivating tests of when this flexibility improves denoising.
  • Cross-model attention transfer [A3]: models trained on the same distribution share a denoising problem, suggesting that context selected by one may help another.
  • Alternative patch predictors [A4]: viewing MLP as one possible estimator motivates testing different key–value memory designs, including sharing a bank across layers.
  • Related-domain QK compatibility [A5]: similar spatial patch dependencies suggest that source context selection may remain useful while patch reconstruction adapts. This motivates the transplantation and MetFaces allocation tests.

Our aggregator–predictor framework thus connects these experiments and explains what their findings tell us about modularity, replaceability, and transferability.

W2(a,e) — How does the GMRF result connect to the experiments and to diffusion generalization?

[A2] We examine an accuracy requirement underlying locality-based explanations of generalization: whether restricting context can preserve accurate denoising.

As mentioned in [A1], we use “generalization” because locality is an established explanation of diffusion models’ generalization. Our contribution connects this account to denoising risk: Proposition 1 isolates the locality error, and under a GMRF prior this error admits an exponentially decaying upper bound in context radius (Proposition 3). Corollary 4 also describes how a certified context radius grows with noise over its stated range. Our theoretical result concerns denoising risk; a finite-sample generalization guarantee would require additional analysis.

Our new experiment directly measures the accuracy cost of restricting context in pretrained real-image denoisers. We use released ImageNet256 JiT-B/16 and JiT-L/16 (131.3M and 459.1M parameters), freeze each model, and progressively supply nearby patches while excluding other patches throughout the network. We measure target-patch denoising MSE against the clean target and compare it with the same model receiving full input.

For JiT-L, retaining 32 of 256 patches gives:

Noise levelMSE with 32 patchesFull-input MSEAdditional MSE
σ=1\sigma=10.01610.01510.0010
σ=4\sigma=40.05320.04360.0096

Thus, one-eighth of the image context retains much of the full-input accuracy at σ=1\sigma=1, while the same restriction incurs a substantially larger accuracy cost at σ=4\sigma=4. Correspondingly, keeping excess MSE below 0.003 requires 13 patches at σ=1\sigma=1 and 74 at σ=4\sigma=4. Figure 2 shows the full accuracy–context comparison and the context needed to meet this fixed tolerance across noise levels.

The GMRF analysis therefore motivates the accuracy–context question, and the new experiment quantifies its practical counterpart in trained DiTs. Together, they support accurate local denoising as a basis for our aggregator–predictor interpretation.

W1(d) / Q4 — How does the attention evidence go beyond gradient visualization such as Grad-CAM?

[A3] We directly test whether attention-selected patches help denoising, by restricting the model's available context and measuring the resulting prediction error.

Our aggregator interpretation motivates testing context usefulness. We freeze a recipient DiT, retain the same nine nearest patches including the target, then add patches according to each rule. Context stays restricted throughout the network; we measure excess target-patch MSE relative to full-input denoising as context grows.

For attention, we average probabilities across heads and layers to obtain one ranking for each noisy input and target patch. For comparison, we use spatial Distance (Kamb and Ganguli), Gradient sensitivity (Niedoba et al.), and Covariance-based Wiener scores (Lukoianov et al.). We also compare attention computed from the current input with attention averaged over calibration images. This asks whether adapting context to the particular noisy image helps prediction.

Figure 4 shows that input-adaptive attention is especially useful at low noise, while spatial distance can match or outperform it at high noise, particularly on ImageNet. Thus, adapting context to visible image structure is most useful at low noise.

Pooling attention has precedent in transformer analysis: attention rollout averages heads before combining layers (Abnar and Zuidema, 2020), while DreamRelation averages maps across heads and layers when analyzing video DiTs (Wei et al., 2025). The network retains head- and layer-specific attention patterns that this simple summary omits.

Our population view also motivates testing context transfer across models: the predictive information available in a specified context depends on the image distribution and denoising task. Different networks may represent this information differently, yet donor-selected context may remain useful to a recipient. We test this between CelebA128 MLP/SMLP and ImageNet256 JiT-B/JiT-L. The selected context transfers across independently trained models with different sizes and predictor architectures (Figure 4), supporting reuse beyond the particular network that selected it.

W2(b,c) / Q3 — Why does the framework motivate shared-memory predictors, and what does their performance establish?

[A4] Our framework views MLP as one possible patch-predictor estimator among many. This motivates exploring which design choices support generation.

PLD retrieves clean training patches at the target position. An ordinary MLP instead retrieves learned values using coefficients determined by learned keys and GELU. We use “key–value memory” in the established interpretation of feed-forward layers, following Geva et al. (2021). Its keys and values are shared across patch positions within a layer, while different layers maintain separate banks.

We therefore ask: must memory be patch-specific? Must retrieval use normalized softmax weights? And, because prediction is repeated across depth, must every layer maintain a separate bank? Table 4 organizes learned PLD, shared cross-attention, SMLP, and ordinary MLP along these dimensions. SMLP tests whether an explicitly shared bank can support patch prediction; it does not claim that ordinary DiT MLPs already implement a layer-shared bank.

We first compare these designs on CelebA64 across three parameter budgets, now with two training seeds per configuration. Eight-head SMLP achieves lower FID than eight-head shared cross-attention at all three budgets in both seeds. Relative to MLP, SMLP improves FID at the small and middle budgets in both seeds; the largest-budget ordering varies (Figure 9). These results suggest that memory need not be patch-specific, softmax retrieval can be restrictive, and layers can share a bank while retaining competitive generation.

We further compare 134M-parameter MLP/SMLP pairs on CelebA128, ImageNet128, and ImageNet256. SMLP achieves lower FID in all three larger-model comparisons.

DatasetMLP FID-50k ↓SMLP FID-50k ↓
CelebA12814.51 ± 0.0711.13 ± 0.04
ImageNet12864.75 ± 0.2256.93 ± 0.21
ImageNet25678.34 ± 0.1075.50 ± 0.40

Values are mean ± sample SD over three sampling seeds, using one training seed per model.

These findings support sharing predictor memory across layers, treating memory organization as a design choice within a broader class of patch-predictor estimators.

Reviewer_WVCiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an aggregator–predictor view of DiT denoising: attention aggregates input- and noise-dependent context, while MLPs predict individual patches from the aggregated features. Theoretically, the authors show that local denoisers approximate the global Bayes denoiser with exponentially decaying error under GMRF priors, and that patchwise linear denoisers recover the empirical local denoiser given sufficiently informative aggregation. They further interpret QK as spatial weighting, VO as feature transport, and MLPs as learned key–value memories. Validation includes sensitivity and attention analyses on real-image DiTs, shared-memory replacements for MLPs on CelebA64, and cross-dataset component transplantation.

Critical strengths
  • The aggregator–predictor framework connects statistical denoising, diffusion generalization, and DiT architecture in a coherent and interpretable way.
  • The shared-memory predictors and component transplantation provide concrete architectural and intervention-based evidence beyond visualization alone.
Critical weaknesses
  • The theoretical and empirical parts are only loosely connected. Theorem 1 applies to GMRF priors, while Theorem 3 assumes that the aggregation already preserves the empirical local conditional mean. For trained DiTs on real images, the connection between attention and denoising context is supported only by a shared qualitative broadening trend, not by a direct test that attention selects the patches causally used by the final denoiser.

  • The shared-memory experiment demonstrates that a separately trained alternative architecture can achieve competitive performance, but it does not show that standard DiT MLPs actually implement a layer-shared memory. Moreover, the comparison is restricted to unconditional CelebA64 and one training seed per configuration, so the reported MLP-SMLP differences lack uncertainty estimates.

  • The component-transplant results do not uniquely identify the proposed QK, VO, and MLP roles. The transplanted components were adapted jointly, so their effects can reflect parameter magnitude, interface compatibility, and co-adaptation. In addition, the reported correlations use only 12 dependent transfer directions, while EA and SA measure similarity to the jointly adapted model rather than target-domain generation quality.

Questions for authors
  • Can the authors quantify the correspondence between attention mass and denoising sensitivity at the patch, image, layer, and noise-level scales? Comparisons with distance-only or uniform baselines, together with patch masking or attention interventions, would help determine whether high-attention patches causally contribute to denoising.

  • Is there evidence that the attention representations learned by the DiTs satisfy the sufficiency condition required by Theorem 3? More generally, which theoretical conclusions are expected to extend beyond the GMRF and empirical-prior settings?

  • Can the authors report multi-seed uncertainty for the main MLP versus SMLP comparisons in Figure 4? Can analyses of the ordinary MLP's learned keys, values, or activation patterns show that it behaves like the proposed memory mechanism, rather than merely being replaceable by one?

  • Can the authors report target-quality metrics, such as FID or denoising loss, for all transplant states and include matched-update or matched-perturbation controls? How stable are the reported correlations when accounting for shared source models, repeated datasets, and adaptation seeds?

7 comments

We thank the reviewer for the concrete suggestions. We add direct attention tests, broader predictor comparisons, and target-quality and norm-matched transplantation controls, clarifying how each experiment follows from our framework.

W1 / Q2 — How do the GMRF result and PLD sufficiency condition connect to trained DiTs?

The two results address different questions: the GMRF result bounds the accuracy cost of restricted context, while original Theorem 3 concerns exact recovery by PLD with fixed linear aggregation and Gaussian retrieval. The latter, now Proposition 5 in Appendix A.5, gives a sufficiency condition under which that analytic estimator recovers the local conditional mean under the empirical image distribution.

We start from the established locality account of diffusion generalization (Kamb and Ganguli, 2025; Niedoba et al., 2025; Lukoianov et al., 2025). We express local denoising as the conditional mean of a clean patch given restricted noisy context, then decompose it into context aggregation and patch prediction. A fixed context determines a population conditional mean, but that prediction can admit different aggregation and prediction representations. These provide statistical targets for interpreting PLD and DiT. This population formulation applies beyond the stated priors.

We then ask how much denoising accuracy is lost by restricting context. Under a GMRF prior, this excess risk admits an exponentially decaying upper bound in context radius (Proposition 3), and a certified context radius grows with noise over the range in Corollary 4. These results justify the established locality heuristic under the stated assumptions.

We now directly measure the accuracy cost of restricting context in pretrained ImageNet256 JiT-B/L (Figure 2). We freeze each model, exclude unselected patches throughout the network, and measure denoising MSE against the clean target. For JiT-L, retaining 32 of 256 patches gives MSE 0.0161 versus 0.0151 with full input at σ=1\sigma=1, and 0.0532 versus 0.0436 at σ=4\sigma=4. Thus, the same restriction adds about 0.0010 and 0.0096 MSE, respectively. Keeping excess MSE below 0.003 requires 13 patches at σ=1\sigma=1 and 74 at σ=4\sigma=4. These measurements quantify how much accuracy restricted context retains and how the required context changes with noise.

PLD connects this target to a concrete estimator (Figure 3): for each position and noise level, it uses fixed linear context selection and softmax retrieval from clean training patches. Proposition 5 gives its exact-recovery condition; corresponding conditions for learned nonlinear aggregation remain a direction for future work.

We then ask how DiT estimates these operations more flexibly. Its attention can select different contexts for different noisy inputs, and its MLP predicts through learned key–value pairs. Figure 3 places PLD and DiT side by side to make this connection visible. Through their architectural algebra, we interpret attention and MLP as more flexible estimators of aggregation and prediction, a view that also applies beyond GMRF priors. This motivates hypotheses that we test on trained DiTs:

  • Input-adaptive attention is especially useful at low noise. DiT’s aggregation can adapt to each noisy input, motivating tests of when this flexibility improves denoising. See Q1 below.
  • Attention selects useful context that transfers across models. Models trained on the same distribution share a denoising problem, motivating a test of whether context selected by one remains useful to another. See Q1 below.
  • MLP can be replaced with another patch predictor at larger scales. Viewing MLP as one possible estimator motivates different key–value memory organizations, including sharing a bank across layers. See W2/Q3 below.
  • Source QK remains partly compatible across related domains. Similar spatial patch dependencies suggest that source context selection may remain useful while patch reconstruction adapts. This motivates transplantation and the MetFaces allocation study. See W3/Q4 below.

The architectural interpretation motivates tests of modularity, replaceability, and transferability, connecting the local-denoising target to the experiments below.

Q1 — Do attention-selected patches actually help denoising, beyond a similar sensitivity map?

Yes: attention-selected context is especially useful at low noise because it adapts to the input, while simple spatial-distance selection can outperform it at high noise.

Our aggregator interpretation motivates testing whether selected patches help denoising. We restrict a frozen denoiser to progressively added context and measure how quickly error approaches that of full-input denoising. We compare attention with spatial Distance (Kamb and Ganguli), Gradient sensitivity (Niedoba et al.), and Covariance-based Wiener scores (Lukoianov et al.). Attention scores average probabilities across heads and layers for each noisy input and target patch.

Figure 4 shows that even this pooled summary selects useful context at low noise, while nearby patches provide strong information at high noise, particularly on ImageNet.

Pooling attention has precedent in transformer analysis: attention rollout averages heads before combining layers (Abnar and Zuidema, 2020), while DreamRelation averages maps across heads and layers when analyzing video DiTs (Wei et al., 2025). The network retains head- and layer-specific attention patterns that this simple summary omits.

The population connection also motivates testing context transfer. Models trained on the same distribution share a denoising problem, while their internal representations and aggregator–predictor decompositions can differ. Context selected by one may therefore remain useful to another. We find this transfer between CelebA128 MLP and SMLP, and between ImageNet256 JiT-B and JiT-L, supporting reuse across predictor architectures and model sizes (Figure 4).

W2 / Q3 — Does SMLP reveal shared memory in ordinary MLPs, and are the performance differences reproducible?

We use “key–value memory” in the established interpretation of feed-forward layers, following Geva et al. (2021): the first map supplies keys, the second supplies values, and the activation determines their coefficients. Ordinary MLPs maintain separate keys and values across layers, whereas SMLP explicitly shares a learned bank. The SMLP experiment tests whether this alternative organization supports generation; it does not claim that ordinary DiT MLPs already implement a layer-shared bank.

Our framework views MLP as one possible patch-predictor estimator, alongside alternatives such as PLD’s retrieval from training patches. This motivates asking which predictor properties support generation: must memory be specific to each patch position? Must retrieval use normalized softmax weights? And, because prediction is repeated across depth, must every layer maintain a separate bank? We compare learned PLD, shared cross-attention, SMLP, and ordinary MLP along these dimensions (Table 4).

For reproducibility, we added a second training seed for every configuration in the small CelebA64 comparison. Figure 9 shows mean FID and the range across the two seeds. SMLP improves on MLP at the small and middle parameter budgets in both seeds; the largest-budget ordering varies. Eight-head SMLP also improves on eight-head shared cross-attention at all three budgets in both seeds. These results support sharing memory across patches and layers and show the benefit of GELU retrieval in the tested shared-bank designs.

We further compare approximately 134M-parameter MLP/SMLP pairs on CelebA128, ImageNet128, and ImageNet256. SMLP achieves lower FID in all three larger-model comparisons.

DatasetMLP FID-50k ↓SMLP FID-50k ↓
CelebA12814.51 ± 0.0711.13 ± 0.04
ImageNet12864.75 ± 0.2256.93 ± 0.21
ImageNet25678.34 ± 0.1075.50 ± 0.40

Values are mean ± sample SD over three sampling seeds, using one training seed per model. Together, the small comparison provides replication across two training seeds, while the larger comparisons show that shared-memory prediction remains effective on more varied images and at higher resolutions.

W3 / Q4 — What does transplantation test, and how do quality metrics, update magnitude, and dependent transfer directions affect its interpretation?

The reviewer is correct that EA and SA measure similarity to generations from Source and Target DiTs. This is precisely their purpose in our transplantation experiment: to trace which jointly learned updates carry target-associated appearance and whether context-selection parameters remain compatible across related domains.

We distinguish the datasets from the models:

  • Source dataset: CelebA or CelebA-HQ, depending on the backbone.
  • Target dataset: AAHQ, AFHQ-v2, or FFHQ.
  • Source DiT: the model pretrained on the source dataset.
  • Target DiT: the Source DiT after jointly fine-tuning QK, VO, and MLP on the target dataset.

Our framework separates population local denoising into context aggregation and patch prediction. When two datasets have similar spatial patch dependencies but different appearances, the source context-selection rule may remain useful, while reconstructing target patches requires adapting the predictor. The one-layer DiT algebra connects this possibility to parameter groups: QK determines context weights for given features, whereas VO transforms the selected features and MLP predicts from them. This motivates testing whether Source DiT's QK remains compatible with Target DiT's VO+MLP, and whether VO+MLP updates carry more target-associated appearance than QK updates.

We jointly fine-tune QK, VO, and MLP, then copy selected adapted groups into Source while leaving the remaining groups at their source values. All transplants use the same joint run, with one fine-tuning seed per setting. Initial sampling noise is paired across Source, Target, and transplants, allowing us to compare the appearance changes carried by each group.

We find that Target DiT's QK with Source DiT's VO+MLP preserves more source-associated appearance, while Source DiT's QK with Target DiT's VO+MLP acquires more target-associated appearance (Figure 1). Because these components coadapt during fine-tuning, exchanging only some groups can disrupt generation. Recognizable images with this consistent appearance preference therefore provide evidence of partial QK compatibility across these related domains. Reusing QK parameters does not require unchanged attention maps: updates to earlier layers can change the features on which later attention operates.

Paired DINO and spatial DINO distances (previously EA and SA) quantify this appearance preference. DINO compares transplant generations with corresponding Source and Target generations; spatial DINO compares their patch-feature self-similarity patterns. Both support the same preference across all nine U-ViT64, DiT128, and LFM256 transfers (Figures 1 and 10).

Dependent directions. We remove the correlation analysis because directions share source models and datasets. We instead compare each transplant with its Source and Target within each setting.

Target denoising improves, while target FID has a mixed ordering. Tables 7–9 report FID and denoising loss against both real source and target datasets. VO+MLP transplantation improves held-out target denoising loss over Source in all nine settings, but achieves lower target FID than QK in four. For example, in DiT128 transfers, VO+MLP has lower FID on AFHQ (161.41 versus 231.79), whereas QK has lower FID on AAHQ (49.63 versus 58.10). Paired DINO distances directly address our question about resemblance to Source and Target generations; FID assesses agreement with the real target distribution. Partial transplantation can alter coherence and diversity alongside appearance, so greater Target resemblance need not yield better FID.

Update magnitude. We rescale MLP and combined VO+MLP effective-weight updates to the Frobenius norm of the QK update (Appendix D.6). QK stays closer to Source in all nine settings, while VO+MLP is closer to Target than Source in only two.

Magnitude dependence is compatible with our finding about jointly learned updates. Our question concerns their appearance effects, including learned directions and magnitudes. Equalizing norms weakens Target resemblance, but VO+MLP remains closer to Target than QK in all nine settings. Thus, update magnitude contributes to the original contrast while its relative ordering persists. The finding does not require greater appearance change per unit parameter displacement.

Practical implication. These findings suggest that, for related domains, retaining source QK and allocating more of a limited fine-tuning budget to VO and/or MLP may improve adaptation. We test this implication separately on 1,057 MetFaces training portraits, independently optimizing different groups with matched trainable counts and three training seeds per backbone (Tables 2–3).

Reviewer_SDtEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an aggregator–predictor view of how Diffusion Transformers (DiTs) denoise images. It describes attention as aggregating context for each image patch, with QK determining spatial weights and VO transforming the selected features, while MLPs predict patch content from the aggregated representation. The authors show that, under a Gaussian Markov random field prior, a local denoiser approximates the global Bayes denoiser with an error that decreases exponentially as the context radius grows. Experiments examine how denoising context changes with noise level, replace MLPs with shared-memory patch predictors, and transplant attention and MLP components across datasets.

Critical strengths
  1. Clear component-level interpretation. The aggregator–predictor framework connects attention, MLPs, and AdaLN to distinct operations in patch denoising.

  2. Statistical foundation for local denoising. The paper proves that, under a Gaussian Markov random field prior, local denoising approaches the global Bayes denoiser as the context radius increases.

  3. Complementary empirical evidence. Sensitivity analysis, shared-memory predictor replacements, and cross-dataset component transplants examine the framework from different angles.

Critical weaknesses
  1. Limited model scale and task settings. Small-scale, unconditional 64×6464\times64 experiments are insufficient to establish whether the conclusions generalize to large-scale conditional DiTs.

  2. Insufficient cross-dataset validation of shared-memory replacement. Evidence from face datasets alone is insufficient to determine whether a shared predictor is suitable for natural images with complex layouts and relationships among multiple objects.

  3. Component transplantation does not uniquely identify functional roles. Changes in the performance of hybrid models may arise from component incompatibility, rather than solely reflecting the functions performed by individual components.

  4. The related work section should be included in the main text. Related work is an important part of the paper and should not be confined to the appendix. The current discussion is also overly long; we recommend selecting the most relevant material and incorporating it into the main text.

  5. Unclear compatibility with representation alignment methods. Representation alignment methods such as REPA play an important role in modern generative models. How can the proposed method accommodate and integrate these features?

Questions for authors

Please refer to the Weaknesses section.

6 comments

We thank the reviewer for the feedback. We have substantially expanded the experimental scope and clarified how each experiment follows from our aggregator–predictor view. The revision tests attention transfer, predictor replacement, and component compatibility in full multilayer networks, including larger models, higher resolutions, and class-conditional generation.

W1 — Do the findings extend beyond small, unconditional 64×64 models?

[A1] We now test models from 44M to 459M parameters at 64×64, 128×128, and 256×256 resolutions, including released class-conditional ImageNet models. Across these settings, we find evidence for transferable attention, replaceable patch predictors, and component compatibility across related domains, all consistent with our framework.

Attention selects useful context that transfers across models. Our framework views attention as aggregating information for population denoising. Models trained on the same distribution share a denoising problem, motivating us to ask whether patches selected by one remain useful to another despite differences in their internal representations. We test released class-conditional ImageNet256 JiT-B/16 and JiT-L/16, alongside our CelebA128 MLP/SMLP models. Attention-selected context transfers across model sizes and predictor architectures (Figure 4). Moreover, input-adaptive attention is especially useful at low noise, while simple spatial-distance selection is competitive at high noise. These findings support the hypothesis that useful context selection can transfer across models solving the same denoising problem.

MLP can be replaced with another patch predictor at larger scales. Our framework views MLP as one possible patch-predictor estimator, motivating alternative organizations of its learned key–value pairs, including sharing a bank across layers. Please see [A2] below for the larger-model comparisons and results.

Source QK remains partly compatible across related domains. Our framework suggests that datasets with similar spatial patch dependencies may share useful context-selection rules even when patch reconstruction must adapt to different appearances. This motivates testing whether source QK remains compatible with adapted VO+MLP. We test transplantation using released U-ViT64, our DiT128, and released latent LFM256, transferred from CelebA or CelebA-HQ to AAHQ, AFHQ-v2, and FFHQ. Please see [A3] below for the findings and their interpretation across these nine settings.

This interpretation also motivates effective fine-tuning allocations. We independently fine-tune the same three backbones on 1,057 MetFaces training images, comparing parameter-group allocations with matched trainable counts. Across all three backbones, QK-only adaptation performs worse, while allocating the budget to VO and/or MLP with QK frozen retains FID competitive with allocation across all three groups (Table 3). This provides a practical test of the implication suggested by transplantation: for related domains, retaining source QK while adapting feature transformation and patch prediction can be an effective use of a limited fine-tuning budget.

MetFaces generation at epoch 1,000 (Table 3).

AllocationU-ViT64 KID ↓U-ViT64 FID ↓DiT128 KID ↓DiT128 FID ↓LFM256 KID ↓LFM256 FID ↓
QK6.83 ± 0.0758.67 ± 0.0916.91 ± 0.0781.98 ± 0.2211.52 ± 0.0980.18 ± 0.09
VO5.60 ± 0.0354.91 ± 0.2412.21 ± 0.1474.72 ± 0.139.60 ± 0.1876.77 ± 0.24
MLP5.92 ± 0.1454.47 ± 0.0812.35 ± 0.1073.64 ± 0.279.67 ± 0.1975.45 ± 0.17
VO+MLP5.89 ± 0.1954.36 ± 0.1312.75 ± 0.4473.99 ± 0.169.41 ± 0.0875.20 ± 0.10
QK+VO+MLP5.62 ± 0.1754.50 ± 0.1612.60 ± 0.1773.94 ± 0.149.49 ± 0.0675.41 ± 0.15
Full25.69 ± 0.1866.37 ± 0.1229.34 ± 0.8084.30 ± 0.2751.19 ± 0.54103.68 ± 0.56

Values are mean ± sample SD; KID is multiplied by 1,000. All three backbones use three training seeds, each evaluated with the same sampling seed. Restricted allocations have matched trainable counts within each backbone; Full updates the entire denoiser.

W2 — Does shared-memory prediction work on natural images with complex layouts?

[A2] Yes: the revised comparison includes ImageNet at both 128×128 and 256×256 resolutions.

The motivation from our framework is that MLP is one possible estimator of the patch predictor. PLD predicts by retrieving from training patches, whereas MLP predicts by retrieving from learned key–value pairs. These estimators belong to a broader class of patch predictors with different choices of key–value memory organization. Accordingly, we examine which properties support generation, including memory sharing across patches and layers and softmax versus GELU retrieval (Section 4.2).

In each pair, we replace ordinary layerwise MLPs with SMLP predictors and train the full model. SMLP uses a learned key–value bank shared across layers. Table 1 reports the following FID-50k results at epoch 400:

DatasetMLP FID-50k ↓SMLP FID-50k ↓
CelebA12814.51 ± 0.0711.13 ± 0.04
ImageNet12864.75 ± 0.2256.93 ± 0.21
ImageNet25678.34 ± 0.1075.50 ± 0.40

Values are mean ± sample SD over three sampling seeds, using one training seed per model. The smaller CelebA64 study now includes two training seeds, with the FID range shown in Figure 9.

SMLP achieves lower FID in all three larger-model comparisons. The ImageNet results show that the shared-memory alternative remains effective beyond aligned face images.

W3 — Could transplant effects reflect component incompatibility rather than functional roles?

[A3] Yes; transplant effects can reflect component incompatibility, and we use transplantation precisely to test compatibility and trace the effects of jointly learned updates.

Our framework motivates this test: datasets with similar geometry but different appearances may have similar patch dependencies, allowing source context selection to remain useful while patch reconstruction adapts. The one-layer algebra connects this possibility to QK, which determines context weights for given features (Section 4.1). We therefore ask whether source QK remains compatible with adapted VO+MLP, and which jointly learned updates carry target-associated appearance.

We first jointly adapt QK, VO, and MLP from a source face dataset to a target dataset. We then copy selected adapted groups into the source model and compare generations under the same initial noise. Source denotes the pretrained model, and Target denotes the jointly adapted model.

In this sense, component transplantation is a diagnostic tool for visually and quantitatively measuring the effects of parameter changes during fine-tuning between similar datasets. Poor generation is a natural possibility: parameters coadapt during fine-tuning, so transplanting only some components can disrupt their compatibility.

The recognizable generations are therefore especially revealing: adapted QK with source VO+MLP preserves more source-associated appearance; source QK with adapted VO+MLP acquires more target-associated appearance (Figure 1 and Appendix Figures 11–13). Paired DINO distances, which measure distances between generated images in DINO feature space, corroborate this visual assessment across all nine settings. Spatial DINO distances in Appendix D.3, which compare patch self-similarity matrices in DINO feature space, support the same observation (Figure 10).

Reusing QK parameters does not require unchanged attention maps: updates to earlier layers can change the features on which later attention operates. The observed appearance preference supports partial QK compatibility across these related domains, consistent with reusing context-selection parameters while adapting feature transformation and patch prediction.

W4 — Bring a shorter, focused related-work discussion into the main text. The current discussion is too long.

[A4] In this revision, we dedicate Section 2 to the work most directly related to our questions: locality in diffusion models, context selection and attention, MLPs as key–value memories, and component-wise adaptation. Additional literature appears in Appendix F.

We also place each experiment directly after its motivation, making the connection between the framework and the experiment explicit.

W5 — How could the framework accommodate representation alignment such as REPA?

[A5] Thank you for raising representation alignment.

  1. Representation alignment can be interpreted as denoising with a clean-feature target. Let umu_m be the unit-normalized projection of a hidden token computed from noisy input, and vmv_m the unit-normalized frozen encoder feature of the corresponding clean image. Then −⟨um,vm⟩=12∥um−vm∥2−1-\langle u_m,v_m\rangle=\tfrac12\|u_m-v_m\|^2-1. Thus, under this normalization, REPA's cosine loss is equivalent to squared-error prediction of normalized clean features from noisy input. This suggests the same aggregator–predictor interpretation: attention aggregates context relevant for prediction, while tokenwise transformations reconstruct the clean representation. The prediction target is now an encoder feature. Studying which local contexts suffice for this prediction is a natural extension of our framework.

  2. Another relevant finding is that the spatial structure of the alignment target strongly predicts REPA's generation performance (Singh et al., 2026). Our framework suggests a possible explanation: when feature prediction and image denoising rely on similar patch dependencies, they may benefit from similar context-aggregation rules. Attention trained with the alignment signal could therefore select context useful for denoising as well. This connects to the compatibility of context-selection rules suggested by our component-transplantation results.

  3. Lastly, SMLP offers a shared key–value bank that alignment supervision could shape through projected hidden states or memory readouts, making the learned information available across layers. Exploring whether this shared memory improves representation alignment and generation is a promising direction for future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Framework and main findings

We thank the reviewers for their feedback. The revision clarifies how our aggregator–predictor view motivates experiments about what DiT components learn and which parts can be reused or replaced.

We start from the established locality account of diffusion denoising and express local denoising as a population conditional mean for a specified context. Our framework decomposes this prediction into context aggregation and patch prediction. The single-layer architectural algebra then interprets attention and MLP as flexible estimators of these operations, connecting DiT to the established analytic PLD model (Figure 3). The GMRF analysis and new context-restriction experiments further examine when restricted context can preserve accurate denoising (Figure 2).

This interpretation motivates hypotheses that we test in complete multilayer networks. The expanded experiments span 64×64 to 256×256 images and models from 44M to 459M parameters, using our trained DiTs and publicly released U-ViT, JiT, and LFM models.

  • Attention selects useful context that transfers across models. Viewing attention as a context aggregator motivates testing whether its selected patches help denoising, including in another model trained on the same distribution. Attention-selected context is especially useful at low noise, whereas spatial-distance selection can be competitive or better at high noise. Selected context also transfers across model sizes and predictor architectures (Figure 4).

  • The predictor view motivates effective alternatives to layerwise MLPs. Treating MLP as one possible patch-predictor estimator motivates testing different key–value memory organizations. A learned bank shared across layers retains competitive generation and achieves lower FID in all three larger-model comparisons on CelebA128, ImageNet128, and ImageNet256 (Table 1).

  • Source QK remains partly compatible across related face domains. Similar spatial patch dependencies suggest that source context selection may remain useful while patch reconstruction adapts. After joint fine-tuning, transplanting adapted QK preserves more source-associated appearance, whereas transplanting adapted VO+MLP produces more target-associated appearance. Paired DINO and spatial DINO measurements support this pattern across nine transfer settings (Figure 1 and Appendix D). The norm-matched control identifies update magnitude as a contributor to this appearance contrast.

These findings motivate a separate practical test: can limited adaptation capacity be allocated to VO and/or MLP while retaining source QK? Our matched-budget MetFaces experiments, with three training seeds per backbone, find that QK-only adaptation performs worse, while keeping QK frozen remains competitive with adapting all three groups (Table 3). Together, the experiments clarify the transferability, replaceability, and partial compatibility of DiT components through questions motivated by the aggregator–predictor view.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.