- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Gaussian Markov random field (GMRF) based statistical framework to study local denoising. Then the paper presents an aggregator-predictor interpretation of the component of attention blocks in a DiT. Finally, experiments try to establish the applicability of the GMRF framework on DiTs, explores an alternative to replace the MLPs, and try to understand the role of the components empirically by transplanting components between DiT and the same DiT’s finetuned to a different dataset.
Critical strengths
- The GMRF mathematical formulation for denoising behavior is interesting and could be valuable if better connected to DiTs in practice.
- I especially like the transplantation experiments which can have interesting implication for understanding and manipulation of generated images.
Critical weaknesses
- The connection between theory and practice feels very limited. Firstly, the aggregator-predictor decomposition does not directly capture multilayer residual DiTs, as the authors acknowledge. Furthermore, the GMRF analysis assumes a sparse dependency graph whose relationship to natural image dependencies is not established. Section 5.1 only shows qualitative similarities between GMRF denoising and DiT sensitivity maps (locality, noise-dependent context increase, and disconnected contexts). These behaviors are already expected of many image denoisers, so their co-occurrence provides limited validation. A stronger connection would involve deriving specific non-trivial predictions from the theory and validating them experimentally on trained DiTs.
- The aggregator-predictor interpretation is sound but largely follows from the standard mathematical structure of transformer layers, making its novelty unclear.
- SMLP's competitive performance is interesting, but sharing memory across layers is not clearly motivated by the theoretical framework. It is also unclear what this establishes about ordinary DiT MLPs.
- The paper is difficult to follow at times. Some mathematical notation is insufficiently explained (e.g., sf(.)). Figure 6 should be included in the main paper, as it summarizes the proposed architectural interpretation.
Questions for authors
- Can the authors provide a specific non-trivial prediction derived from the GMRF framework that can be tested on DiTs? Apart from the qualitative similarity shown, are there any other evidence supporting the use of this framework for the denoising of real images?
- Given that the aggregator-predictor roles are not functionally separable in multilayer residual DiTs (Appendix D.5), what evidence supports this interpretation of the individual components in trained DiTs?
5 comments
Q2 — How does the single-layer interpretation extend to multilayer residual DiTs?
We use the single-layer algebra to motivate hypotheses, then test those hypotheses in complete multilayer DiTs. The algebra identifies context weighting by QK, feature transformation by VO, and patch prediction by MLP in the single-layer construction.
Appendix B.3 describes the corresponding operations in a residual stack: attention aggregates hidden features, and the MLP maps the contextualized features to a residual update. These operations interact across depth because each layer changes the features received by subsequent layers. We therefore use the single-layer interpretation to motivate hypotheses about context usefulness, predictor replacement, and parameter compatibility, and test these hypotheses in complete multilayer DiTs.
All the following findings come from multilayer models:
-
Attention selects useful context that transfers across models (Figure 4). Viewing attention as a context aggregator motivates testing whether its selected patches remain useful to another model solving the same denoising problem. We test this by selecting patches with one model's attention and measuring another model's denoising error using that context. The selection transfers between CelebA128 MLP/SMLP models and ImageNet256 JiT-B/JiT-L. Input-adaptive attention is especially useful at low noise. Thus, the aggregation interpretation motivates a prediction that is supported across complete networks with different sizes and predictor architectures.
-
MLP can be replaced with another patch predictor. Viewing MLP as one possible estimator motivates sharing learned key–value memories across layers. We test this replacement in full multilayer models, where it retains competitive generation quality. See W3 below for the comparisons and results.
-
Source QK remains partly compatible across related domains (Figure 1). Similar spatial patch dependencies suggest that source context selection may remain useful while target appearance changes. After jointly fine-tuning QK, VO, and MLP, we transplant selected adapted groups back into the source model. Across nine transfers using U-ViT64, DiT128, and LFM256, QK-only transplants remain closer to Source, while VO+MLP transplants are closer to the jointly adapted Target in both DINO and spatial DINO distance. Recognizable source-like and target-like generations persist despite the component interactions, supporting partial compatibility in these multilayer networks. Reusing QK parameters does not require unchanged attention maps: updates to earlier layers can change the features on which later attention operates.
-
The same interpretation motivates effective fine-tuning allocations (Tables 2–3). We then test the practical implication on MetFaces by independently fine-tuning different parameter groups with matched trainable counts. Across all three backbones, QK-only adaptation performs worse, while keeping QK frozen retains FID competitive with allocation across all three groups. This supports allocating adaptation capacity toward feature transformation and patch prediction for these related-domain transfers.
MetFaces generation at epoch 1,000 (Table 3).
| Allocation | U-ViT64 KID ↓ | U-ViT64 FID ↓ | DiT128 KID ↓ | DiT128 FID ↓ | LFM256 KID ↓ | LFM256 FID ↓ |
|---|---|---|---|---|---|---|
| QK | 6.83 ± 0.07 | 58.67 ± 0.09 | 16.91 ± 0.07 | 81.98 ± 0.22 | 11.52 ± 0.09 | 80.18 ± 0.09 |
| VO | 5.60 ± 0.03 | 54.91 ± 0.24 | 12.21 ± 0.14 | 74.72 ± 0.13 | 9.60 ± 0.18 | 76.77 ± 0.24 |
| MLP | 5.92 ± 0.14 | 54.47 ± 0.08 | 12.35 ± 0.10 | 73.64 ± 0.27 | 9.67 ± 0.19 | 75.45 ± 0.17 |
| VO+MLP | 5.89 ± 0.19 | 54.36 ± 0.13 | 12.75 ± 0.44 | 73.99 ± 0.16 | 9.41 ± 0.08 | 75.20 ± 0.10 |
| QK+VO+MLP | 5.62 ± 0.17 | 54.50 ± 0.16 | 12.60 ± 0.17 | 73.94 ± 0.14 | 9.49 ± 0.06 | 75.41 ± 0.15 |
| Full | 25.69 ± 0.18 | 66.37 ± 0.12 | 29.34 ± 0.80 | 84.30 ± 0.27 | 51.19 ± 0.54 | 103.68 ± 0.56 |
Values are mean ± sample SD across three training seeds; KID is multiplied by 1,000. Restricted allocations have matched trainable counts within each backbone; Full updates the entire denoiser.
These findings support the empirical usefulness of the aggregator–predictor view in multilayer DiTs. The single-layer algebra motivates the questions, while the full-network experiments establish the reported transferability, replaceability, and compatibility results.
W3 — Why share predictor memory across layers, and what does SMLP establish about MLPs?
The framework views MLP as one possible patch-predictor estimator, with a particular layerwise organization of keys and values. This motivates considering other predictor designs, including a shared memory bank.
PLD predicts by retrieving clean training patches at the target position. An ordinary MLP instead retrieves learned values using coefficients determined by learned keys and GELU. We use “key–value memory” in the established interpretation of feed-forward layers, following Geva et al. (2021). Ordinary MLPs share their keys and values across patch positions within a layer, while different layers maintain separate banks. SMLP explicitly shares a bank across layers. Its evaluation tests this alternative predictor design; it does not claim that ordinary DiT MLPs already implement a layer-shared bank.
We therefore ask which properties support generation. Must memory be patch-specific? Must retrieval use normalized softmax weights? And, because prediction is repeated across depth, must each layer maintain its own bank? Table 4 organizes learned PLD, shared cross-attention, SMLP, and ordinary MLP by these properties.
The smaller CelebA64 comparison now includes two training seeds. Eight-head SMLP achieves lower FID than eight-head shared cross-attention at all three parameter budgets in both seeds. Relative to MLP, SMLP improves FID at the small and middle budgets in both seeds; the largest-budget ordering varies (Figure 9). The results suggest that memory need not be patch-specific, softmax retrieval can be restrictive, and layers can share a bank while retaining competitive generation.
We further compare 134M-parameter MLP/SMLP pairs on CelebA128, ImageNet128, and ImageNet256. SMLP achieves lower FID in all three larger-model comparisons.
| Dataset | MLP FID-50k ↓ | SMLP FID-50k ↓ |
|---|---|---|
| CelebA128 | 14.51 ± 0.07 | 11.13 ± 0.04 |
| ImageNet128 | 64.75 ± 0.22 | 56.93 ± 0.21 |
| ImageNet256 | 78.34 ± 0.10 | 75.50 ± 0.40 |
Values are mean ± sample SD over three sampling seeds, using one training seed per model.
A shared bank could also support future representation alignment or memory augmentation, building on memory-based designs such as persistent-memory augmentation (Sukhbaatar et al., 2019) and GMem (Tang et al., 2025).
The tested shared-memory predictor retains competitive generation and achieves lower FID in all three larger-model comparisons, extending the evidence to diverse natural images and higher resolutions. This supports treating MLP as one estimator within a broader class whose memory organization can be changed while preserving effective patch prediction.
W4 — Clarify notation, improve the flow, and move the architectural diagram into the main paper.
We have clarified the notation throughout the revision and moved the PLD/DiT architectural comparison into the main text as Figure 3. We have also reorganized the writing so that each experiment follows its motivation from the framework, making the connection between the interpretation and the findings easier to follow.
We thank the reviewer for the feedback and interest in the transplantation experiments. The revision clarifies two connections: how restricted context can support accurate denoising, and how the single-layer architectural interpretation motivates hypotheses tested in full multilayer DiTs. We strengthen the evidence for both and make the connection between each experiment and its motivating question explicit.
W1 / Q1 — What quantitative prediction connects the GMRF result to denoising real images?
We now directly measure how much denoising accuracy is lost when context is restricted, extending the original sensitivity analysis with an intervention on the model’s available context.
The GMRF result motivates this accuracy–context comparison. Local denoising is the conditional mean of a clean patch given restricted noisy context. Under a GMRF prior, its excess risk relative to the global denoiser admits an exponentially decaying upper bound in context radius (Proposition 3). Corollary 4 also describes how a certified context radius grows with noise over its stated range.
We examine the corresponding question empirically with released ImageNet256 JiT-B/16 and JiT-L/16 (131.3M and 459.1M parameters). We freeze each model, retain the target patch, and progressively supply nearby context, excluding other patches throughout the network. We measure denoising MSE against the clean target and compare it with full-input denoising.
For JiT-L, retaining 32 of 256 patches gives:
Thus, one-eighth of the image context retains much of the full-input accuracy at σ=1, while the same restriction incurs a substantially larger accuracy cost at σ=4. Correspondingly, keeping excess MSE below 0.003 requires 13 patches at σ=1 and 74 at σ=4. Figure 2 shows the full accuracy–context comparison and the context needed to meet this fixed tolerance across noise levels.
These measurements quantify both the usefulness of restricted context and its dependence on noise in pretrained real-image models. They strengthen the empirical basis for local denoising by measuring the accuracy retained when context is actually removed.
W2 — What does the interpretation add beyond standard transformer algebra?
The value of our interpretation is that it connects familiar architectural operations to population quantities, motivating testable hypotheses about what DiT components learn and how they can be reused or replaced.
We first express local denoising as context aggregation followed by patch prediction. PLD provides an established analytic estimator of these operations: fixed linear context selection and softmax retrieval from training patches. Our framework interprets PLD and DiT as alternative aggregation–prediction estimators for population local denoising. A fixed context determines a population conditional mean, but that prediction can admit different aggregation and prediction representations. DiT attention provides input-adaptive aggregation, while MLP predicts through learned key–value pairs (Figure 3).
This population connection gives us a reason to ask questions that the architectural algebra alone leaves open:
Can context selection transfer across models? Models trained on the same distribution share a denoising problem. We ask whether context selected by one remains useful to another in the cross-model attention test described in Q2 below.
Can another predictor replace MLP? Viewing MLP as one possible estimator makes its memory organization a design choice, motivating alternatives such as a bank shared across layers. This motivates the predictor replacement in full multilayer models summarized in Q2 below, with detailed comparisons in W3.
Can context selection be reused across related domains? Similar spatial patch dependencies suggest that source context selection may remain useful even when target appearance changes. This motivates the transplantation experiment and subsequent MetFaces allocation test described in Q2 below.