acceptodds
Under review as a conference paper at ICLR 2027

Should We Skip Diffusion?

Abstract

Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity decoder in denoising. To enable effective denoising at all noise levels, these features must capture both high-level abstract structures and low-level details. However, skip/residual connections in the encoder allow shallow features to bypass successive transformations, which may limit progressive abstraction, or at least make it difficult to disentangle different levels of abstraction. We propose DDT-RFE, which removes the residual connections around the Self-Attention and MLP operations in each encoder block while maintaining stable training. To retain the information that abstraction discards but that the decoder still needs, we fuse the input patch embedding with intermediate and final encoder features to form the encoder output. The decoder thus has access to information from multiple encoder depths, while each encoder block is able to learn more abstract representations. DDT-RFE achieves overall improvements over DDT across visual understanding tasks, including image classification, semantic segmentation, object discovery, and semantic correspondence, while using fewer encoder blocks. It also achieves a lower FID for image generation on ImageNet.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
diffusion transformers, representation learning, residual connections

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wwiaOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies information flow in the condition encoder of a Decoupled Diffusion Transformer (DDT). DDT-RFE removes the attention and MLP residual connections inside encoder blocks, introduces orthogonal/mixed query-key initialization and orthogonal MLP initialization, and sums the patch embedding and all encoder-block outputs to condition the velocity decoder. Both DDT and DDT-RFE are trained without REPA, allowing the authors to study representations learned from the generative objective. Experiments use ImageNet-1k at 256 x 256 for 80 epochs at L and XL scales. The method improves several frozen-feature recognition tasks, particularly object discovery and VOC segmentation, and reduces unguided generation FID from 8.98 to 7.68 at L scale and from 7.27 to 6.81 at XL scale. Appendices A.3-A.4 provide residual-baseline and initialization/fusion ablations.

Critical strengths
  1. The benefit appears across several distinct representation tasks and two model scales. The evidence extends beyond qualitative PCA images: VOC2012 object-discovery CorLoc rises from 42.5 to 58.4 at L scale and 41.3 to 57.8 at XL scale (Table 2); SPair-71k PCK rises from 33.3 to 37.2 and 35.2 to 39.9 (Table 3). The large VOC segmentation improvements are also consistent across scales (Table 4). These results make the architectural recipe practically interesting even though some scene-segmentation gains are small or absent in the single-block setting.

  2. The paper checks the generative role of the representation and includes meaningful controls. The reduced best-block-to-encoder-output classification gap in Table 1 supports the usefulness of the actual decoder condition, while Table 5 shows better FID under a shared training schedule and sampler. The 12/12 residual baseline and Appendices A.3-A.4 strengthen the study: the method is not compared only against one unmatched encoder depth, and the dependence on initialization and fusion is explicitly examined. The reported recall reductions also make clear that generation quality does not improve uniformly on every metric.

Critical weaknesses
  1. The ablations support the combined recipe more strongly than they isolate the proposed abstraction mechanism. At L scale, the main method changes residual pathways, initialization, gating, feature fusion, and the encoder/decoder allocation relative to the primary 20/4 DDT. The 12/12 DDT control in Tables 5-7 addresses depth allocation, and the residual DDT with initialization plus fusion in Tables 6-7 is useful, but that latter control is only 20/4. It therefore does not provide the direct 12/12 comparison with identical initialization and fusion that would isolate the residual-pathway choice. Moreover, Table 8's severe no-fusion classification degradation is largely recovered when gates are also removed: top-1 accuracy is 60.7 versus 60.9 for the full method. Fusion still has meaningful benefits: without fusion/gates, FID is 8.34 rather than 7.68 and SPair PCK is 32.87 rather than 37.24. These results reveal an important interaction and task dependence rather than a clean demonstration that aggregation is necessary for semantic abstraction. A matched 12/12 residual-versus-residual-free comparison, together with a more restrained interpretation of the gate/fusion interaction, would materially strengthen the central explanation. This concern does not erase the measured performance of the complete method.

  2. The repeated best-block/noise-level reporting is insufficiently separated from final evaluation. Section 5.1.1 explicitly selects the best validation accuracy, while Tables 2-4 report the best over blocks and noise levels; Section 5.1.3 introduces SPair-71k test pairs but does not identify a separate selection set. The manuscript and appendix do not specify the selection/reporting split for each dense task, the full noise-level search, or how multiscale block combinations and probe settings were fixed. This is a protocol ambiguity, not evidence that test labels were actually used for selection. It matters because some improvements, such as 0.2 mIoU on XL ADE20K multiscale probing, are small and no repeated-run uncertainty is reported. Please document a held-out selection protocol and report fixed selections on the reporting split, with uncertainty for the small differences. That would distinguish transferable improvements from optimistic best-of-search summaries.

Questions for authors
  1. Clarification affecting causal attribution: Is there a 12/12 residual DDT with the same initialization and feature sum as DDT-RFE? If so, what happens to generation, object discovery, and segmentation? If not, how should the abstraction claim be limited given the existing 20/4 control and Table 8's gate/fusion interaction?
  2. Clarification affecting validity of the reported representation gains: For each task in Tables 1-4, which examples and labels selected the block, noise level, multiscale combination, and probe settings, and which independent split supplied the reported score? Please give the noise-level grid and explain how the SPair-71k test set was reserved.
  3. Clarification affecting the efficiency claim: The method reduces encoder depth but retains the total block count by deepening the decoder (12/12 versus 20/4 and 14/14 versus 22/6). What measured extraction cost supports the representation-efficiency statement, and what are the end-to-end training/sampling costs? The current evidence directly establishes a shallower encoder; the scope of the broader cost claim should be explicit.
6 comments

Dear reviewer wwia,

We sincerely appreciate your insightful and constructive feedback, which has been instrumental in improving our work. Please find our detailed point-by-point responses below, along with the corresponding revisions in the manuscript (highlighted in blue).

Weakness 1. At L scale, the main method changes residual pathways, initialization, gating, feature fusion, and the encoder/decoder allocation relative to the primary 20/4 DDT. The 12/12 DDT control in Tables 5-7 addresses depth allocation, and the residual DDT with initialization plus fusion in Tables 6-7 is useful, but that latter control is only 20/4. It therefore does not provide the direct 12/12 comparison with identical initialization and fusion that would isolate the residual-pathway choice.

Response:

We have now trained a residual DDT-L/2 with 12 encoder and 12 decoder blocks, matching the depth of DDT-RFE-L/2. Both models use the same feature fusion rule: they sum the input patch embedding and all 12 encoder-block outputs to condition the decoder. They also share the same 80-epoch training schedule, optimizer, learning rate, and batch size.

We evaluated the epoch-80 EMA checkpoints using the paper’s within-benchmark protocol. For classification and segmentation, we trained linear probes on the corresponding training splits. Object discovery and semantic correspondence required no downstream training. The table reports the best scores on each reporting split over the configurations evaluated for each model. For classification, Top-5 is reported at the configuration selected by Top-1 accuracy.

TaskDatasetMetricResidual DDT-L/2 + fusion + init (12/12, new)DDT-RFE-L/2 (12/12, ours)
Image generationImageNet-1kFID-50K ↓9.537.68
Image generationImageNet-1kInception Score ↑102.75118.37
Image classificationImageNet-1kBest single block, Top-1 / Top-5 (%) ↑47.48 / 71.6760.92 / 83.47
Image classificationImageNet-1kFull encoder output, Top-1 / Top-5 (%) ↑46.57 / 70.9359.67 / 82.40
Object discoveryVOC2012 (trainval)CorLoc (%) ↑38.7358.40
Object discoveryCOCO20k (COCO train2014 subset)CorLoc (%) ↑28.1441.35
Semantic segmentationVOC2012 (val)Single-block mIoU ↑28.9657.00
Semantic segmentationVOC2012 (val)Multi-block mIoU ↑34.7060.45
Semantic segmentationADE20K (validation)Single-block mIoU ↑16.7226.84
Semantic segmentationADE20K (validation)Multi-block mIoU ↑19.2229.70
Semantic segmentationCOCO-Stuff 10K (test)Single-block mIoU ↑16.4326.01
Semantic segmentationCOCO-Stuff 10K (test)Multi-block mIoU ↑18.6428.22
Semantic correspondenceSPair-71k (test)[email protected] (%) ↑18.7037.24

DDT-RFE achieves better generation scores and outperforms the residual control on all four downstream tasks.

...together with a more restrained interpretation of the gate/fusion interaction, would materially strengthen the central explanation. This concern does not erase the measured performance of the complete method.

Without feature fusion, removing the gates increases ImageNet linear-probe accuracy and improves FID. Our intuition is that channel-wise gating may suppress directions carrying patch-specific variation. During training, its interaction with attention and MLP transformations may progressively concentrate features into fewer dominant directions, leading to a low-effective-rank representation. Without fusion, earlier features reach the decoder only through the final encoder output, making spatial information lost along this chain difficult to recover. Removing the gates may help the encoder preserve this information.

In Table 8, with feature fusion disabled, removing the gates raises ImageNet linear-probe accuracy from 6.7% to 60.7% and lowers FID-50K from 16.81 to 8.34. One possible explanation is that channel-wise gating attenuates differences between patches. Let uiu_i be the feature of patch ii before a gate and hi=(1+g)⊙uih_i=(1+g)\odot u_i its gated feature. Since patches in the same image share the gate, the squared distance between two patch features becomes

∥hi−hj∥22=∑c(1+gc)2(ui,c−uj,c)2.\lVert h_i - h_j \rVert_2^2 = \sum_c (1 + g_c)^2 (u_{i,c} - u_{j,c})^2.

If the gate downweights channels that distinguish patches more than it downweights other channels, it suppresses patch-specific variation. Repeated across layers, this could concentrate features in fewer dominant directions and lower their effective rank. Without fusion, the decoder receives encoder features only from the final block, so it has no direct access to spatial detail preserved in earlier blocks. Removing the gates may help preserve that detail through depth.

Weakness 2. evaluation protocol and further experiments on selection on separate data

Response:

Thank you for raising this point. We kept the diffusion backbone frozen for all downstream tasks. Below, we describe how we selected the encoder block and noise level tt, which split we used to report each score, and which prior work informed each protocol.

Image classification. The k-NN evaluation [1] required no training and used labeled ImageNet training features as the reference bank. We evaluated every encoder block on validation at t∈{0,0.05,0.1,0.15,0.2,0.3,0.4,0.5,0.7}t \in \{0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.7\}. For linear probing, we trained each head on ImageNet train on learning rate from {1,2,3,7}×10−n,n∈{2,3,4}\{1, 2, 3, 7\} \times 10^{-n}, n \in \{2,3,4\} , swept every block and tt from the grid used in k-NN by validation top-1 accuracy, and reported top-1 and top-5 on the same validation split.

Object discovery. We follow LOST [2] which required no downstream training. On VOC2012 trainval and COCO20k subset of COCO train2014, models were evaluated at t∈{0,0.05,0.1,0.15,0.2,0.3,0.4,0.5}t \in \{0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5\}. On each dataset, we selected the best block and tt by CorLoc and reported the score on that same split.

Semantic correspondence. We follow the protocal in DIFT [3], with no downstream training. We evaluated t∈{0.1,0.15,0.2,0.25,0.3,0.4}t \in \{0.1, 0.15, 0.2, 0.25, 0.3, 0.4\} for all models, and selected and reported the highest [email protected] on the official test split.

Semantic segmentation. Largely following protocal in [4], we trained a dense linear head on each dataset’s training split and swept every blocks at t∈{0.1,0.15,0.2,0.25,0.3,0.4}t \in \{0.1, 0.15, 0.2, 0.25, 0.3, 0.4\} on learning rate from {1,2,3,7}×10−n,n∈{2,3,4}\{1, 2, 3, 7\} \times 10^{-n}, n \in \{2,3,4\} on all three datasets. For PASCAL VOC 2012, we trained the segmentation head on train and evaluated it on val. For ADE20K, we trained on training and evaluated on validation. For COCO-Stuff 10K, we trained on train and evaluated on test. For multiscale probing, we used the single-block validation results to identify promising encoder depths and concatenated features from three nearby blocks along the channel dimension.

Separated from final evaluation data. To assess how the choice of block and noise level affects the reported gains, we reanalyzed the object discovery and semantic correspondence results for DDT-L/2 (20 encoder and 4 decoder blocks) and DDT-RFE-L/2 (12 encoder and 12 decoder blocks). For each model, we selected the block and tt using only the selection data and reported the score on a separate dataset or split with both settings fixed.

For object discovery, we used the noise-level grid described above and evaluated transfer in both directions. We first selected the block and tt by CorLoc on VOC2012 trainval and reported CorLoc on COCO20k. We then selected on COCO20k and reported on VOC2012 trainval. For semantic correspondence, we selected among all encoder blocks and noise levels by [email protected] on a SPair-71k validation subset whose images do not appear in the official test split. We then reported [email protected] on the official test split using the validation-selected settings.

Task and metricSelected onReported onDDT-L/2 (20/4)DDT-RFE-L/2 (12/12, ours)Gain (pp)
Object discovery, CorLocVOC2012 trainvalCOCO20k31.07%40.80%+9.73
Object discovery, CorLocCOCO20kVOC2012 trainval42.51%58.22%+15.71
Semantic correspondence, [email protected]SPair-71k validationSPair-71k test33.46%37.24%+3.78

For object discovery in DDT-L/2, we selected its 15th block at t=0.05t=0.05 on either dataset. For DDT-RFE-L/2, we selected its 12th block on both datasets, with t=0.10t=0.10 on VOC2012 and t=0.05t=0.05 on COCO20k. For SPair-71k in 'validation', we selected the 15th block at t=0.20t=0.20 for DDT-L/2 and the 10th block at t=0.10t=0.10 for DDT-RFE-L/2.

Weakness 2. It matters because some improvements, such as 0.2 mIoU on XL ADE20K multiscale probing, are small and no repeated-run uncertainty is reported.

The 0.2 mIoU difference on XL ADE20K multiscale probing (29.8 vs. 30.0) is small and may reflect run-to-run variation. We will therefore describe performance in this setting as comparable. Our broader conclusion is supported by larger observed differences at the same XL scale. On VOC2012, DDT-RFE improves multiscale segmentation from 56.9 to 63.1 mIoU and single-block segmentation from 53.8 to 59.8 mIoU (Table 4). For object discovery, CorLoc rises from 41.3% to 57.8% on VOC2012 and from 29.8% to 39.0% on COCO20k (Table 2). On SPair-71k, correspondence [email protected] increases from 35.2% to 39.9% (Table 3).

Notably, DDT-RFE achieves these results with a shallower encoder, which matters for downstream tasks that use only encoder features. At the XL scale, extracting features through the final encoder block requires 14 blocks instead of 22. At the L scale, where we measured inference time directly, reducing the encoder depth from 20 to 12 blocks cut final-block feature extraction time from 101.10 to 58.82 ms per batch, a 41.8% reduction on a single NVIDIA H100 with a batch size of 256.

References

[1] Caron et al. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.
[2] Siméoni et al. Localizing Objects with Self-Supervised Transformers and No Labels. 2021.
[3] Tang et al. Emergent Correspondence from Image Diffusion. NeurIPS 2023.
[4] Xiang et al. DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning. 2025.

Reviewer_urhqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper suggests the use of long connections from each DDT block to a final feature fusion layer before passing it to the velocity decoder, as opposed to the short residual on every self-attention and MLP layers. Furthermore, they introduce an initialization scheme for this architectural change. Experimental results show that this update improves performances on recognition tasks like classification, object discovery, semantic segmentation and correspondence, while achieving slightly better generative performance.

Critical strengths
  1. The paper experimentally shows that the proposed DDT-RFE outperforms the primary baseline, DDT, in object discovery and semantic segmentation while being slightly better at generation. This supports the use of the proposed architectural change.
  2. The feature representations produced by this network appear cleaner compared to the baseline.
Critical weaknesses
  1. The related work is insufficiently researched. There is an extensive literature on representation learning with diffusion models that is not adequately covered. Many relevant papers are missing including [1,2,3,4]. In particular, [1,2] also investigate blockwise features for global and dense prediction tasks, while [2,3] investigate feature aggregation across layers and time steps. Both these directions are relevant to this paper’s motivation and feature fusion. The architecture also seems to have conceptual precedent in earlier architectures like DenseNet [5], Jumping Knowledge Networks [10], and Residual Feature Aggregation Network [11], which are not discussed.
  2. The experimentation is very limited and paper lacks almost any baselines. Several relevant papers are cited but never compared against, including DIFT [6] and DiTF [7] for semantic correspondence, CleanDIFT [8] and DDAE++ [9] for classification and segmentation. USP [4] is another relevant work that is not discussed or compared against. Notably, DDAE++ reports 66.68% ImageNet linear probe accuracy after 100 epochs, compared to DDT-RFE-L’s 60.9% after 80 epochs. Earlier UNet based diffusion models [1] also present stronger linear probing accuracy (64.96%) while [2] reports stronger kNN results (49.04% vs 40.9%). Although these are not controlled comparisons, they bring into question the competitiveness of the proposed architecture compared to existing methods. The paper having improvements over DDT does not establish its broader significance.
  3. Is there a way to test the paper’s expectation that removing the additive pathways encourage progressive semantic abstraction? The current blockwise analysis is limited to kNN classification accuracy and qualitative feature maps, which do not directly establish this hypothesis.

References:

[1] Diffusion Models Beat GANs on Image Classification. arXiv 2023.

[2] Do Text-Free Diffusion Models Learn Discriminative Visual Representations? ECCV 2024.

[3] Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence. NeurIPS 2023.

[4] USP: Unified Self-Supervised Pretraining for Image Generation and Understanding. ICCV 2025.

[5] Densely Connected Convolutional Networks. CVPR 2017.

[6] Emergent Correspondence from Image Diffusion (DIFT). NeurIPS 2023.

[7] Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations (DiTF). NeurIPS 2025.

[8] CleanDIFT: Diffusion Features without Noise. CVPR 2025.

[9] DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning. arXiv 2025.

[10] Representation Learning on Graphs with Jumping Knowledge Networks. ICML 2018.

[11] Residual Feature Aggregation Network for Image Super-Resolution. CVPR 2020.

Questions for authors

See weakness section.

8 comments

Dear reviewer urhq,

We sincerely appreciate your insightful and constructive feedback, which has been instrumental in improving our work. Please find our detailed point-by-point responses below, along with the corresponding revisions in the manuscript (highlighted in blue).

Weakness 1. The related work is insufficiently researched.

Response:

Thank you for highlighting these relevant works. We will expand the related work section to discuss their connections to our approach.

There is an extensive literature on representation learning with diffusion models that is not adequately covered. Many relevant papers are missing including [1,2,3,4]. In particular, [1,2] also investigate blockwise features for global and dense prediction tasks, while [2,3] investigate feature aggregation across layers and time steps. Both these directions are relevant to this paper’s motivation and feature fusion.

Prior work [1–3] takes block outputs as the starting point for feature analysis. These studies examine which outputs are useful for downstream tasks and how features from different layers or timesteps can be combined. For a residual block of the form y=x+f(x)y = x + f(x), the output yy already combines the input with its learned transformation. Our work asks how this residual connections(+x+x) within the encoder block shapes the feature.

USP[4]

USP [4] studies how masked latent pretraining can provide an effective initialization for both image generation and recognition. Our work asks a different architectural question: how do the residual connections around attention and MLP sublayers shape the representations learned by a diffusion encoder, and how can the decoder retain access to shallow features when those connections are removed?

The architecture also seems to have conceptual precedent in earlier architectures like DenseNet [5]

DenseNet [5] also gives each layer direct access to earlier features by concatenating the outputs of all preceding layers within a dense block. In DDT-RFE, each encoder block transforms the output of the preceding block. We aggregate features by addition across depths only at the encoder output, where they condition the velocity decoder.

Jumping Knowledge Networks [10], and Residual Feature Aggregation Network [11]

Jumping Knowledge Networks [10] take the output of each graph layer as a whole and aggregate these layer outputs at the final readout. When the underlying graph layer uses a residual connection, its contribution to the readout is the representation after residual addition. RFANet [11] explicitly exposes features from individual residual branches for aggregation while retaining local residual connections. Both are relevant precedents for cross-layer aggregation.

Our design is motivated by the hypothesis that repeatedly adding shallow features back into the encoder stream may hinder semantic abstraction, even though those features remain useful for denoising. We therefore remove the residual connections around the attention and MLP sublayers while giving the decoder direct access to the input patch embedding and all encoder-block outputs.

Tables 6 and 7 examine a related alternative that retains the local residual connections in the 20/4 DDT-L/2 and adds feature fusion and initialization. This variant improves ImageNet linear-probe accuracy from 59.6% to 60.4%, but its segmentation and object-discovery scores decrease relative to the original 20/4 DDT. It also falls short of the full 12/12 DDT-RFE-L/2 across all four downstream tasks. The gap is small for ImageNet linear probing (60.4% vs. 60.9%) and larger for VOC2012 single-block segmentation (43.1 vs. 57.0 mIoU) and object discovery (42.1% vs. 58.4% CorLoc). These results show that adding fusion and initialization to the residual DDT baseline does not reproduce the downstream performance of the full DDT-RFE design.

Weakness 2.The experimentation is very limited and paper lacks almost any baselines. Several relevant papers are cited but never compared against, including DIFT [6] and DiTF [7] for semantic correspondence, CleanDIFT [8] and DDAE++ [9] for classification and segmentation. USP [4] is another relevant work that is not discussed or compared against. Notably, DDAE++ reports 66.68% ImageNet linear probe accuracy after 100 epochs, compared to DDT-RFE-L’s 60.9% after 80 epochs. Earlier UNet based diffusion models [1] also present stronger linear probing accuracy (64.96%) while [2] reports stronger kNN results (49.04% vs 40.9%). Although these are not controlled comparisons, they bring into question the competitiveness of the proposed architecture compared to existing methods. The paper having improvements over DDT does not establish its broader significance.

Response:

Thank you for raising this question. As you note, these comparisons are not well-controlled because the model architectures and training protocols differ. We argue that these differences make it difficult to isolate the contribution of our encoder design. As discussed in our limitations, our computational budget led us to focus on comparing encoder designs within DDT under a common training schedule. We therefore used DDT [12] as a well-controlled setting to test whether our design improves downstream representations. We trained DDT-RFE and the original DDT under the same protocol and compared their performance on four downstream visual tasks. We agree that it is also worth testing whether this design transfers to other diffusion architectures. Indeed, machine learning offers many examples of techniques that prove useful across different types of models. For example, in a diffusion U-Net (which, like DDT, has an encoder–decoder architecture), one could remove the local residual connections within encoder blocks while retaining the skip connections that carry fine-grained features to the corresponding decoder stages. We will leave it for future work.

Within well-controlled DDT settings, the gains extend across all four downstream tasks we evaluated (Tables 1–4). In particular, object discovery tests whether frozen features can localize objects without downstream training. Under the same 80-epoch training schedule, DDT-RFE-L/2 (12/12) improves CorLoc over the original DDT-L/2 (20/4) from 42.5% to 58.4% on VOC2012 and from 31.1% to 41.4% on COCO20k. We also observe gains in ImageNet-1k linear probing (+1.3 percentage points in top-1 accuracy), SPair-71k correspondence (+3.9 percentage points in [email protected]), and VOC2012 single-block segmentation (+6.5 mIoU). DDT-RFE-L/2 achieves these gains with a 12-block encoder, compared with 20 blocks in the original DDT-L/2. On a single NVIDIA H100 with a batch size of 256, extracting features through the final encoder block takes 58.82 ms for DDT-RFE-L/2 versus 101.10 ms for DDT-L/2, a 41.8% reduction in feature extraction time.

Weakness 3.Is there a way to test the paper’s expectation that removing the additive pathways encourage progressive semantic abstraction? The current blockwise analysis is limited to kNN classification accuracy and qualitative feature maps, which do not directly establish this hypothesis.

Response:

Our paper provides qualitative and quantitative evidence consistent with progressive semantic abstraction in the encoder.

Qualitatively, the PCA visualizations in Figure 1 and Appendix Figures 7-9 show how the feature maps change with depth. With our methodology, foreground objects become more distinct, and background details become less prominent.

Quantitatively, our reported results are not limited to kNN classification. We use performance on four downstream visual tasks as a proxy for how well the encoder captures semantic abstraction. These tasks require features that support category recognition, distinguish objects from their backgrounds, match corresponding parts across instances, and assign semantic labels to pixels. Compared with DDT-L/2 (20/4)(the best baseline model) , DDT-RFE-L/2 (12/12) improves all four metrics (full results in the paper):

  • Image classification: ImageNet-1k linear-probe top-1 accuracy increases from 59.6% to 60.9% (Table 1).
  • Object discovery: VOC2012 CorLoc increases from 42.5% to 58.4% (Table 2).
  • Semantic correspondence: SPair-71k [email protected] increases from 33.3% to 37.2% (Table 3).
  • Semantic segmentation: VOC2012 single-block mIoU increases from 50.5 to 57.0 (Table 4).

We thank you again for reviewing our work. Please let us know if we have misunderstood any of your comments or if you have any follow-up questions. We would be happy to provide further clarification.

Best regards,

Authors

References:

[1] Diffusion Models Beat GANs on Image Classification. arXiv 2023.

[2] Do Text-Free Diffusion Models Learn Discriminative Visual Representations? ECCV 2024.

[3] Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence. NeurIPS 2023.

[4] USP: Unified Self-Supervised Pretraining for Image Generation and Understanding. ICCV 2025.

[5] Densely Connected Convolutional Networks. CVPR 2017.

[6] Emergent Correspondence from Image Diffusion (DIFT). NeurIPS 2023.

[7] Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations (DiTF). NeurIPS 2025.

[8] CleanDIFT: Diffusion Features without Noise. CVPR 2025.

[9] DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning. arXiv 2025.

[10] Representation Learning on Graphs with Jumping Knowledge Networks. ICML 2018.

[11] Residual Feature Aggregation Network for Image Super-Resolution. CVPR 2020.

[12] DDT: Decoupled Diffusion Transformer. CVPR 2026.

Reviewer_HpiROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DDT-RFE, a diffusion transformer architecture that removes residual connections within the condition encoder to encourage progressive semantic abstraction. It introduces a specialized attention initialization scheme for stable training and aggregates features from the input embeddings and all encoder blocks to preserve information needed for image reconstruction. The paper proposes DDT-RFE, a diffusion transformer architecture that removes residual connections within the condition encoder to encourage progressive semantic abstraction. It introduces a specialized attention initialization scheme for stable training and aggregates features from the input embeddings and all encoder blocks to preserve information needed for image reconstruction.

Critical strengths
  • The paper proposes a straightforward architectural modification that separates feature transformation within the encoder from feature aggregation at its output.
  • The method is evaluated across four visual understanding tasks and image generation, demonstrating broad potential beyond synthesis.
  • DDT-RFE improves image classification, object discovery, and semantic correspondence, with particularly notable gains in unsupervised object localization.
Critical weaknesses
  • DDT and DDT-RFE use different encoder–decoder depth allocations. More consistent experiments with matched configurations across downstream tasks are needed to attribute gains specifically to the proposed architecture.
  • The proposed method combines residual removal, a new initialization scheme, and feature fusion. In the ablation study, the paper does not sufficiently isolate their individual contributions and interactions, making it difficult to identify the primary source of improvement.
  • The evaluation protocol is unclear. The paper selects the best encoder blocks and noise levels for several downstream tasks. However, the paper does not clearly specify how these choices are made without introducing test-set selection bias.
  • The efficiency of the proposed method is unclear. Fewer encoder blocks do not necessarily imply lower overall computational cost, since the decoder is deeper. It would be better to have further efficiency analysis.
  • The novelty of the proposed method is unclear. The approach builds on existing residual-free transformer methods. However, its methodological novelty could be better distinguished from prior work.
Questions for authors

Please refer to the weaknesses.

4 comments

Dear reviewer HpiR,

We sincerely appreciate your insightful and constructive feedback, which has been instrumental in improving our work. Please find our detailed point-by-point responses below, along with the corresponding revisions in the manuscript (highlighted in blue).

Weakness 1.DDT and DDT-RFE use different encoder–decoder depth allocations. More consistent experiments with matched configurations across downstream tasks are needed to attribute gains specifically to the proposed architecture.

Response:

We thank you for raising this point. At the L scale, we compared three models with 12 encoder blocks, 12 decoder blocks, and the same training schedule. In addition to the standard DDT-L/2 baseline reported in the appendix, we trained a second residual DDT-L/2 with feature fusion and orthogonal initialization. This new control uses the same fusion rule as DDT-RFE: the input patch embedding and all 12 encoder-block outputs are summed to condition the decoder.

The tables below bring together the results from the main paper and appendix and add the new experiments.

Image classification (ImageNet-1k linear probing, accuracy %)

Encoder featuresMetricDDT-L/2DDT-L/2 + fusion + init (new)DDT-RFE-L/2 (ours)
Best single blockTop-151.047.560.9
Best single blockTop-575.271.783.5
Full encoder outputTop-151.046.659.7
Full encoder outputTop-575.370.982.4

Object discovery (CorLoc %)

DatasetDDT-L/2DDT-L/2 + fusion + init (new)DDT-RFE-L/2 (ours)
VOC201242.138.758.4
COCO20k29.828.141.4

Semantic segmentation (mIoU)

DatasetFeaturesDDT-L/2DDT-L/2 + fusion + init (new)DDT-RFE-L/2 (ours)
VOC2012Single-block39.029.057.0
VOC2012Multi-block42.034.760.5
ADE20KSingle-block20.016.726.8
ADE20KMulti-block21.319.229.7
COCO-Stuff 10KSingle-block20.116.426.0
COCO-Stuff 10KMulti-block21.218.628.2

Semantic correspondence ([email protected] %)

DatasetDDT-L/2DDT-L/2 + fusion + init (new)DDT-RFE-L/2 (ours)
SPair-71k20.118.737.2

Image generation (ImageNet-1k, 256 × 256)

MetricDDT-L/2DDT-L/2 + fusion + init (new)DDT-RFE-L/2 (ours)
FID-50K ↓8.829.537.68
Inception Score ↑108.82102.75118.37

DDT-RFE outperforms both residual baselines across all four downstream tasks at the same 12/12 depth.

Weakness 2.The proposed method combines residual removal, a new initialization scheme, and feature fusion. In the ablation study, the paper does not sufficiently isolate their individual contributions and interactions, making it difficult to identify the primary source of improvement.

Response:

Thank you for raising this point. All models in the table below use 12 encoder blocks and 12 decoder blocks and were trained for 80 epochs. We report ImageNet FID-50K (lower is better). One previously unreported result completes the comparison of initialization and feature fusion within the residual-free encoder. We focus here on image generation. Table 8 in Appendix A.4 also reports how initialization and feature fusion affect visual downstream tasks.

EncoderAttention and MLP initializationFeature fusionFID-50K ↓Source
Residual (DDT-L/2)DefaultOff8.82Table 5
Residual-freeDefaultOff17.72Newly reported
Residual-freeProposedOff16.81Table 8
Residual-freeDefaultOn9.86Table 8
Residual-free (DDT-RFE-L/2, ours)ProposedOn7.68Table 8

Contribution of initialization and fusion. Within the residual-free encoder, feature fusion yields the larger observed reduction in FID. Enabling fusion lowers FID from 17.72 to 9.86 with default initialization and from 16.81 to 7.68 with the proposed initialization. The proposed initialization also improves FID in both settings, with a larger gain when fusion is enabled (2.2 points) than when it is disabled (0.9 points).

Residual and residual-free encoders. The first two rows compare the encoder variants at the same depth, with default attention and MLP initialization and no feature fusion. Under these conditions, the conventional residual encoder achieves a lower FID (8.82 vs. 17.72). With both the proposed initialization and feature fusion enabled, DDT-RFE-L/2 achieves 7.68, improving on the matched DDT-L/2 baseline by 1.14 FID points. In a complementary ablation on the conventional DDT-L/2 with 20 encoder blocks and 4 decoder blocks, adding feature fusion and the proposed initialization together worsened FID by 1.92 points (Appendix A.3, Table 6).

Weakness 3.The evaluation protocol is unclear. The paper selects the best encoder blocks and noise levels for several downstream tasks. However, the paper does not clearly specify how these choices are made without introducing test-set selection bias.

Response:

Thank you for raising this point. We have included this section in our revised verision (A.6 in the Appendix). We kept the diffusion backbone frozen for all downstream tasks. Below, we describe how we selected the encoder block and noise level tt, which split we used to report each score, and which prior work informed each protocol.

Image classification. The k-NN evaluation [1] required no training and used labeled ImageNet training features as the reference bank. We evaluated every encoder block on validation at t∈{0,0.05,0.1,0.15,0.2,0.3,0.4,0.5,0.7}t \in \{0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.7\}. For linear probing, we trained each head on ImageNet train on learning rate from {1,2,3,7}×10−n,n∈{2,3,4}\{1, 2, 3, 7\} \times 10^{-n}, n \in \{2,3,4\} , swept every block and tt from the grid used in k-NN by validation top-1 accuracy, and reported top-1 and top-5 on the same validation split.

Object discovery. We follow LOST [2] which required no downstream training. On VOC2012 trainval and COCO20k subset of COCO train2014, models were evaluated at t∈{0,0.05,0.1,0.15,0.2,0.3,0.4,0.5}t \in \{0, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5\}. On each dataset, we selected the best block and tt by CorLoc and reported the score on that same split.

Semantic correspondence. We follow the protocal in DIFT [3], with no downstream training. We evaluated t∈{0.1,0.15,0.2,0.25,0.3,0.4}t \in \{0.1, 0.15, 0.2, 0.25, 0.3, 0.4\} for all models, and selected and reported the highest [email protected] on the official test split.

Semantic segmentation. Largely following protocal in [4], we trained a dense linear head on each dataset’s training split and swept every blocks at t∈{0.1,0.15,0.2,0.25,0.3,0.4}t \in \{0.1, 0.15, 0.2, 0.25, 0.3, 0.4\} on learning rate from {1,2,3,7}×10−n,n∈{2,3,4}\{1, 2, 3, 7\} \times 10^{-n}, n \in \{2,3,4\} on all three datasets. For PASCAL VOC 2012, we trained the segmentation head on train and evaluated it on val. For ADE20K, we trained on training and evaluated on validation. For COCO-Stuff 10K, we trained on train and evaluated on test. For multiscale probing, we used the single-block validation results to identify promising encoder depths and concatenated features from three nearby blocks along the channel dimension.

Selection on separate data. To assess how the choice of block and noise level affects the reported gains, we reanalyzed the object discovery and semantic correspondence results for DDT-L/2 (20 encoder and 4 decoder blocks) and DDT-RFE-L/2 (12 encoder and 12 decoder blocks). For each model, we selected the block and tt using only the selection data and reported the score on a separate dataset or split with both settings fixed.

For object discovery, we used the noise-level grid described above and evaluated transfer in both directions. We first selected the block and tt by CorLoc on VOC2012 trainval and reported CorLoc on COCO20k. We then selected on COCO20k and reported on VOC2012 trainval. For semantic correspondence, we selected among all encoder blocks and noise levels by [email protected] on a SPair-71k validation subset whose images do not appear in the official test split. We then reported [email protected] on the official test split using the validation-selected settings.

Task and metricSelected onReported onDDT-L/2 (20/4)DDT-RFE-L/2 (12/12, ours)Gain (pp)
Object discovery, CorLocVOC2012 trainvalCOCO20k31.07%40.80%+9.73
Object discovery, CorLocCOCO20kVOC2012 trainval42.51%58.22%+15.71
Semantic correspondence, [email protected]SPair-71k validationSPair-71k test33.46%37.24%+3.78

For object discovery in DDT-L/2, we selected its 15th block at t=0.05t=0.05 on either dataset. For DDT-RFE-L/2, we selected its 12th block on both datasets, with t=0.10t=0.10 on VOC2012 and t=0.05t=0.05 on COCO20k. For SPair-71k in 'validation', we selected the 15th block at t=0.20t=0.20 for DDT-L/2 and the 10th block at t=0.10t=0.10 for DDT-RFE-L/2.

References

[1] Caron et al. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.
[2] Siméoni et al. Localizing Objects with Self-Supervised Transformers and No Labels. 2021.
[3] Tang et al. Emergent Correspondence from Image Diffusion. NeurIPS 2023.
[4] Xiang et al. DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning. 2025.

Weakness 4.The efficiency of the proposed method is unclear. Fewer encoder blocks do not necessarily imply lower overall computational cost, since the decoder is deeper. It would be better to have further efficiency analysis.

Response:

Thank you for raising this point. Our efficiency claim focuses on downstream visual tasks that use only the encoder for feature extraction. We use the original 20/4 DDT as our primary baseline because it outperforms the 12/12 DDT on all four downstream tasks we evaluate.

Both models have 24 transformer blocks in total. On a single NVIDIA H100 with a batch size of 256, we timed two operations separately. For feature extraction, we ran the encoder through its final block. For end-to-end generation, we ran 250 Euler sampling steps followed by a single VAE decoding pass for all 256 images at 256 × 256 resolution. We report the median wall-clock time per batch over seven runs after warm-up.

Model (encoder/decoder blocks)Last-block feature extraction (ms)End-to-end generation (s)
DDT-L/2 (20/4)101.1032.757
DDT-RFE-L/2 (12/12, ours)58.8236.400

Compared with the original 20/4 DDT, DDT-RFE takes 41.8% less time to extract last-block features and 11.1% longer to generate a batch of images. We will clarify in the revision that the measured speedup applies to encoder-only feature extraction for downstream visual tasks.

Weakness 5.The novelty of the proposed method is unclear. The approach builds on existing residual-free transformer methods. However, its methodological novelty could be better distinguished from prior work.

Response:

Many studies have shown that generative models can learn representations that transfer well to downstream visual tasks. Prior work [1] showed that residual-free encoders trained with DINO-style objectives can learn abstract visual representations. Diffusion models, however, have a different training objective. Their decoders need fine-grained information for denoising, and some of that information may be lost as encoder representations become more abstract in the absence of residual connections.

Feature fusion. We give the decoder direct access to features from different depths by summing the input patch embedding and all encoder-block outputs. With the proposed initialization held fixed, enabling feature fusion lowers FID-50K by 9.13 points (from 16.81 to 7.68) and raises the Inception Score by 44.2 points (from 74.2 to 118.4) in Table 8.

Initialization. Our query–key initialization mixes a shared orthogonal matrix with independently sampled query and key weights. A single mixing coefficient ρ\rho replaces the two coefficients α\alpha and β\beta used in prior mimetic initialization [1], reducing the search from two hyperparameters to one. With feature fusion held fixed, replacing the default encoder initialization with our proposed scheme lowers FID-50K by 2.18 points (from 9.86 to 7.68) and raises the Inception Score by 12.7 points (from 105.7 to 118.4) in Table 8.

We thank you again for reviewing our work. Please let us know if we have misunderstood any of your comments or if you have any follow-up questions. We would be happy to provide further clarification.

Best regards,

Authors

[1] Cutting the Skip: Training Residual-Free Transformers. ICLR, 2026.

Dear reviewer HpiR, we thank you very much for your review. Before the discussion period ends, we would like to check if our response has addressed the reviewers concerns or if there are remaining issues we may clarify?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.