- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies information flow in the condition encoder of a Decoupled Diffusion Transformer (DDT). DDT-RFE removes the attention and MLP residual connections inside encoder blocks, introduces orthogonal/mixed query-key initialization and orthogonal MLP initialization, and sums the patch embedding and all encoder-block outputs to condition the velocity decoder. Both DDT and DDT-RFE are trained without REPA, allowing the authors to study representations learned from the generative objective. Experiments use ImageNet-1k at 256 x 256 for 80 epochs at L and XL scales. The method improves several frozen-feature recognition tasks, particularly object discovery and VOC segmentation, and reduces unguided generation FID from 8.98 to 7.68 at L scale and from 7.27 to 6.81 at XL scale. Appendices A.3-A.4 provide residual-baseline and initialization/fusion ablations.
Critical strengths
-
The benefit appears across several distinct representation tasks and two model scales. The evidence extends beyond qualitative PCA images: VOC2012 object-discovery CorLoc rises from 42.5 to 58.4 at L scale and 41.3 to 57.8 at XL scale (Table 2); SPair-71k PCK rises from 33.3 to 37.2 and 35.2 to 39.9 (Table 3). The large VOC segmentation improvements are also consistent across scales (Table 4). These results make the architectural recipe practically interesting even though some scene-segmentation gains are small or absent in the single-block setting.
-
The paper checks the generative role of the representation and includes meaningful controls. The reduced best-block-to-encoder-output classification gap in Table 1 supports the usefulness of the actual decoder condition, while Table 5 shows better FID under a shared training schedule and sampler. The 12/12 residual baseline and Appendices A.3-A.4 strengthen the study: the method is not compared only against one unmatched encoder depth, and the dependence on initialization and fusion is explicitly examined. The reported recall reductions also make clear that generation quality does not improve uniformly on every metric.
Critical weaknesses
-
The ablations support the combined recipe more strongly than they isolate the proposed abstraction mechanism. At L scale, the main method changes residual pathways, initialization, gating, feature fusion, and the encoder/decoder allocation relative to the primary 20/4 DDT. The 12/12 DDT control in Tables 5-7 addresses depth allocation, and the residual DDT with initialization plus fusion in Tables 6-7 is useful, but that latter control is only 20/4. It therefore does not provide the direct 12/12 comparison with identical initialization and fusion that would isolate the residual-pathway choice. Moreover, Table 8's severe no-fusion classification degradation is largely recovered when gates are also removed: top-1 accuracy is 60.7 versus 60.9 for the full method. Fusion still has meaningful benefits: without fusion/gates, FID is 8.34 rather than 7.68 and SPair PCK is 32.87 rather than 37.24. These results reveal an important interaction and task dependence rather than a clean demonstration that aggregation is necessary for semantic abstraction. A matched 12/12 residual-versus-residual-free comparison, together with a more restrained interpretation of the gate/fusion interaction, would materially strengthen the central explanation. This concern does not erase the measured performance of the complete method.
-
The repeated best-block/noise-level reporting is insufficiently separated from final evaluation. Section 5.1.1 explicitly selects the best validation accuracy, while Tables 2-4 report the best over blocks and noise levels; Section 5.1.3 introduces SPair-71k test pairs but does not identify a separate selection set. The manuscript and appendix do not specify the selection/reporting split for each dense task, the full noise-level search, or how multiscale block combinations and probe settings were fixed. This is a protocol ambiguity, not evidence that test labels were actually used for selection. It matters because some improvements, such as 0.2 mIoU on XL ADE20K multiscale probing, are small and no repeated-run uncertainty is reported. Please document a held-out selection protocol and report fixed selections on the reporting split, with uncertainty for the small differences. That would distinguish transferable improvements from optimistic best-of-search summaries.
Questions for authors
- Clarification affecting causal attribution: Is there a 12/12 residual DDT with the same initialization and feature sum as DDT-RFE? If so, what happens to generation, object discovery, and segmentation? If not, how should the abstraction claim be limited given the existing 20/4 control and Table 8's gate/fusion interaction?
- Clarification affecting validity of the reported representation gains: For each task in Tables 1-4, which examples and labels selected the block, noise level, multiscale combination, and probe settings, and which independent split supplied the reported score? Please give the noise-level grid and explain how the SPair-71k test set was reserved.
- Clarification affecting the efficiency claim: The method reduces encoder depth but retains the total block count by deepening the decoder (12/12 versus 20/4 and 14/14 versus 22/6). What measured extraction cost supports the representation-efficiency statement, and what are the end-to-end training/sampling costs? The current evidence directly establishes a shallower encoder; the scope of the broader cost claim should be explicit.
6 comments
Weakness 2. evaluation protocol and further experiments on selection on separate data
Response:
Thank you for raising this point. We kept the diffusion backbone frozen for all downstream tasks. Below, we describe how we selected the encoder block and noise level , which split we used to report each score, and which prior work informed each protocol.
Image classification. The k-NN evaluation [1] required no training and used labeled ImageNet training features as the reference bank. We evaluated every encoder block on validation at . For linear probing, we trained each head on ImageNet train on learning rate from , swept every block and from the grid used in k-NN by validation top-1 accuracy, and reported top-1 and top-5 on the same validation split.
Object discovery. We follow LOST [2] which required no downstream training. On VOC2012 trainval and COCO20k subset of COCO train2014, models were evaluated at . On each dataset, we selected the best block and by CorLoc and reported the score on that same split.
Semantic correspondence. We follow the protocal in DIFT [3], with no downstream training. We evaluated for all models, and selected and reported the highest [email protected] on the official test split.
Semantic segmentation. Largely following protocal in [4], we trained a dense linear head on each dataset’s training split and swept every blocks at on learning rate from on all three datasets. For PASCAL VOC 2012, we trained the segmentation head on train and evaluated it on val. For ADE20K, we trained on training and evaluated on validation. For COCO-Stuff 10K, we trained on train and evaluated on test. For multiscale probing, we used the single-block validation results to identify promising encoder depths and concatenated features from three nearby blocks along the channel dimension.
Separated from final evaluation data. To assess how the choice of block and noise level affects the reported gains, we reanalyzed the object discovery and semantic correspondence results for DDT-L/2 (20 encoder and 4 decoder blocks) and DDT-RFE-L/2 (12 encoder and 12 decoder blocks). For each model, we selected the block and using only the selection data and reported the score on a separate dataset or split with both settings fixed.
For object discovery, we used the noise-level grid described above and evaluated transfer in both directions. We first selected the block and by CorLoc on VOC2012 trainval and reported CorLoc on COCO20k. We then selected on COCO20k and reported on VOC2012 trainval. For semantic correspondence, we selected among all encoder blocks and noise levels by [email protected] on a SPair-71k validation subset whose images do not appear in the official test split. We then reported [email protected] on the official test split using the validation-selected settings.
| Task and metric | Selected on | Reported on | DDT-L/2 (20/4) | DDT-RFE-L/2 (12/12, ours) | Gain (pp) |
|---|---|---|---|---|---|
| Object discovery, CorLoc | VOC2012 trainval | COCO20k | 31.07% | 40.80% | +9.73 |
| Object discovery, CorLoc | COCO20k | VOC2012 trainval | 42.51% | 58.22% | +15.71 |
| Semantic correspondence, [email protected] | SPair-71k validation | SPair-71k test | 33.46% | 37.24% | +3.78 |
For object discovery in DDT-L/2, we selected its 15th block at on either dataset. For DDT-RFE-L/2, we selected its 12th block on both datasets, with on VOC2012 and on COCO20k. For SPair-71k in 'validation', we selected the 15th block at for DDT-L/2 and the 10th block at for DDT-RFE-L/2.
Weakness 2. It matters because some improvements, such as 0.2 mIoU on XL ADE20K multiscale probing, are small and no repeated-run uncertainty is reported.
The 0.2 mIoU difference on XL ADE20K multiscale probing (29.8 vs. 30.0) is small and may reflect run-to-run variation. We will therefore describe performance in this setting as comparable. Our broader conclusion is supported by larger observed differences at the same XL scale. On VOC2012, DDT-RFE improves multiscale segmentation from 56.9 to 63.1 mIoU and single-block segmentation from 53.8 to 59.8 mIoU (Table 4). For object discovery, CorLoc rises from 41.3% to 57.8% on VOC2012 and from 29.8% to 39.0% on COCO20k (Table 2). On SPair-71k, correspondence [email protected] increases from 35.2% to 39.9% (Table 3).
Notably, DDT-RFE achieves these results with a shallower encoder, which matters for downstream tasks that use only encoder features. At the XL scale, extracting features through the final encoder block requires 14 blocks instead of 22. At the L scale, where we measured inference time directly, reducing the encoder depth from 20 to 12 blocks cut final-block feature extraction time from 101.10 to 58.82 ms per batch, a 41.8% reduction on a single NVIDIA H100 with a batch size of 256.
References
[1] Caron et al. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.
[2] Siméoni et al. Localizing Objects with Self-Supervised Transformers and No Labels. 2021.
[3] Tang et al. Emergent Correspondence from Image Diffusion. NeurIPS 2023.
[4] Xiang et al. DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning. 2025.
Dear reviewer wwia,
We sincerely appreciate your insightful and constructive feedback, which has been instrumental in improving our work. Please find our detailed point-by-point responses below, along with the corresponding revisions in the manuscript (highlighted in blue).
Response:
We have now trained a residual DDT-L/2 with 12 encoder and 12 decoder blocks, matching the depth of DDT-RFE-L/2. Both models use the same feature fusion rule: they sum the input patch embedding and all 12 encoder-block outputs to condition the decoder. They also share the same 80-epoch training schedule, optimizer, learning rate, and batch size.
We evaluated the epoch-80 EMA checkpoints using the paper’s within-benchmark protocol. For classification and segmentation, we trained linear probes on the corresponding training splits. Object discovery and semantic correspondence required no downstream training. The table reports the best scores on each reporting split over the configurations evaluated for each model. For classification, Top-5 is reported at the configuration selected by Top-1 accuracy.
trainval)train2014subset)val)val)validation)validation)test)test)test)DDT-RFE achieves better generation scores and outperforms the residual control on all four downstream tasks.
Without feature fusion, removing the gates increases ImageNet linear-probe accuracy and improves FID. Our intuition is that channel-wise gating may suppress directions carrying patch-specific variation. During training, its interaction with attention and MLP transformations may progressively concentrate features into fewer dominant directions, leading to a low-effective-rank representation. Without fusion, earlier features reach the decoder only through the final encoder output, making spatial information lost along this chain difficult to recover. Removing the gates may help the encoder preserve this information.
In Table 8, with feature fusion disabled, removing the gates raises ImageNet linear-probe accuracy from 6.7% to 60.7% and lowers FID-50K from 16.81 to 8.34. One possible explanation is that channel-wise gating attenuates differences between patches. Let ui be the feature of patch i before a gate and hi=(1+g)⊙ui its gated feature. Since patches in the same image share the gate, the squared distance between two patch features becomes
∥hi−hj∥22=∑c(1+gc)2(ui,c−uj,c)2.
If the gate downweights channels that distinguish patches more than it downweights other channels, it suppresses patch-specific variation. Repeated across layers, this could concentrate features in fewer dominant directions and lower their effective rank. Without fusion, the decoder receives encoder features only from the final block, so it has no direct access to spatial detail preserved in earlier blocks. Removing the gates may help preserve that detail through depth.