- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes structured residual connectivity for Diffusion Transformers (DiTs), motivated by an observed preference for symmetric cross-layer dependencies. The method combines adaptive residual aggregation with sparse mirror connections, achieving improved FID and training efficiency across multiple DiT models.
Critical strengths
The paper addresses a meaningful problem in DiT architecture design. The proposed method consistently improves FID across different model scales and achieves a substantial improvement on pretrained REPA (5.90 → 4.34 FID), with negligible additional parameters and computational overhead.
Critical weaknesses
-
Insufficient connection between scientific observations and method design.
My primary concern is that the research motivation, mechanistic analysis, method design, and experimental validation do not form a convincing research loop. The most potentially valuable contribution is the observation of Spontaneous Symmetry Bias in DiTs. However, the evidence mainly consists of learned routing weights and CKA/gradient similarity analyses. These results do not sufficiently explain why symmetric dependencies emerge, what functional role they serve, or whether they are responsible for the performance improvements. Without a convincing validation of this observation, the proposed method largely resembles a combination of existing U-shaped skip connections and adaptive residual aggregation, making its distinctive contribution unclear.
-
Ablation studies do not directly validate the central hypothesis.
The comparisons among Fused-First, Fused-Last, Fused-All, and Fused-Mirror do not adequately isolate the effect of mirror symmetry. For instance, Mirror routing accesses different encoder depths, whereas First/Last routing repeatedly accesses the same representation. The improvement may therefore result from more diverse feature reuse rather than symmetric pairing.Comparisons with random or shifted one-to-one connections would provide more direct evidence. An encoder-only routing ablation is also missing.
-
Inconsistent evaluation results.
Although Fused-Mirror achieves the best FID in Table 3, its sFID (12.91) is the worst among the modified architectures. This discrepancy is not discussed, despite the paper's emphasis on spatial information reuse. While the differences are small and their statistical significance is unclear, the results do not consistently support the claimed advantages.
-
Limited distinctive practical advantages.
Engineering combinations can be valuable when they provide substantial practical improvements or establish generalizable design principles. However, this paper does not convincingly establish either. Compared with Attention Residuals, the FID improvement is modest (62.87 to 62.48), with slightly worse sFID. The substantial REPA gains are encouraging but lack comparisons against alternative routing designs under matched fine-tuning budgets. Furthermore, the reported efficiency benefits are not supported by comprehensive end-to-end training cost measurements.
Questions for authors
Q1. On Spontaneous Symmetry Bias. Can the authors provide more direct evidence that the observed symmetric layer dependencies reflect a meaningful functional organization within DiTs? In particular, how can this preference be distinguished from effects induced by the adaptive routing architecture itself?
Q2. On the necessity of mirror pairing. How does Mirror routing compare with random or shifted one-to-one encoder–decoder connections under identical parameter counts, routing operators, and training budgets? Such comparisons would help isolate the specific contribution of mirror symmetry.
Q3. On the independent contributions of architectural components. What is the performance of encoder-side adaptive routing without decoder-side mirror connections? Additionally, how does the proposed fusion operator compare with simpler gated skip connections?
Q4. On the inconsistent FID and sFID results. Why does Fused-Mirror achieve the best FID but the worst sFID among the Fused variants in Table 3? Are these differences consistent across independent runs, and how do they relate to the claimed benefits of spatial information reuse?
Q5. On practical advantages over existing methods. Can the authors clarify whether the substantial REPA fine-tuning improvements can also be achieved using alternative routing architectures under the same training budget? Could they also provide end-to-end training time and peak memory comparisons?
4 comments
Response to reviewer t2SS (Part 2)
4. FID vs. sFID (W3, Q4)
Table 2. FID-50K and other metrics of DiT-XL/2 at 100K iterations without guidance, shown as mean ± SD over three training seeds.
| Model | FID ↓ | sFID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|---|
| DiT baseline | 46.59 ± 1.08 | 7.33 ± 0.03 | 27.7 ± 0.8 | 0.502 ± 0.008 | 0.599 ± 0.005 |
| Ours | 38.37 ± 2.41 | 6.49 ± 0.15 | 35.0 ± 2.3 | 0.544 ± 0.016 | 0.598 ± 0.005 |
Multiple seeds. Over three seeds at DiT-XL/2, ours improves sFID from 7.33 to 6.49 and is better in every seed (Table 2). This is consistent with the spatial information reuse of mirror routing. At DiT-S/2, our mean sFID over four runs (12.80) is not worse than Fused-All (12.85), Fused-First (12.83) or the baseline (12.93). These gaps are smaller than the run-to-run spread, so the sFID ordering among the Fused variants in the paper's Table 3 is within noise.
FID as the primary metric. In most papers on class-conditional ImageNet generation, FID is the primary metric, while sFID is a complementary measure.
5. Practical advantages (W4, Q5)
(a) Against Attention Residuals
The AttnRes in the paper is already an improved version of the original AttnRes, as it stores the accumulated state after each block. At DiT-S/2 it reaches 63.31 FID, against 66.05 for the original Block AttnRes. We therefore compared against the strongest AttnRes. Ours matches it in FID at DiT-S/2, as you note (63.03 vs 63.31 on average), but is faster, lighter and better at depth. At DiT-XL/2 ours needs 35% less step time and 46% less peak memory (Table 3). It also reaches 38.37 against 43.47 for AttnRes after 100K steps. Fused-All, the other dense variant, even falls behind the baseline at XL/2.
(b) REPA with an alternative routing under the same budget
We fine-tuned the in-order pairing from the released REPA checkpoint with the same recipe and budget as ours. With guidance at = 1.7, it reaches 1.433, level with the step-matched control at 1.431, while ours reaches 1.387.
(c) Training time and peak memory
Table 3. Parameters, FLOPs and training cost of DiT-XL/2, with step time and peak memory measured with torch.compile at batch 16 on one B200 GPU.
| Model | Parameters (M) ↓ | GFLOPs per image ↓ | Step time (ms) ↓ | Peak memory (GB) ↓ |
|---|---|---|---|---|
| DiT baseline (no routing) | 675.13 | 114.46 | 50.1 | 16.4 |
| Mirror pairing, linear fusion (U-ViT-style) | 709.65 (+5.1%) | 123.29 (+7.7%) | 52.1 (1.04×) | 17.0 (1.04×) |
| Ours | 675.26 (+0.02%) | 114.62 (+0.1%) | 79.9 (1.59×) | 25.3 (1.54×) |
| AttnRes | 675.26 (+0.02%) | 114.94 (+0.4%) | 122.8 (2.45×) | 46.5 (2.84×) |
Cost. The U-ViT-style skip adds 5.1% parameters and 7.7% FLOPs but little time, whereas ours adds almost no parameters or FLOPs but takes 1.59× the step time, from stacking and normalising candidate states in unfused code (Table 3). Ours is still faster and lighter than AttnRes (Table 3). Ours also needs 1.33–1.73× fewer steps than the baseline to reach the same FID (reply to Reviewer 3L6x, Table 6).
Response
Thank you for the detailed rebuttal and the substantial additional experiments. I appreciate the authors' efforts to address my concerns.
The new ablation studies provide convincing evidence that mirror pairing and adaptive fusion both contribute to the performance improvements. The additional multi-seed evaluations and training cost measurements also strengthen the empirical results. I acknowledge these improvements.
However, my primary concern regarding the scientific contribution of the paper remains unresolved.
As emphasized in my original review, I consider the observation of Spontaneous Symmetry Bias in DiTs to be the most potentially valuable contribution of this work. Symmetric skip connections have already been extensively explored in architectures such as U-Net and U-ViT, while adaptive residual aggregation has also been studied. Therefore, simply demonstrating that their combination improves generation performance offers relatively limited conceptual novelty.
What could make this work scientifically distinctive is a deeper understanding of why DiTs spontaneously develop symmetric cross-layer dependencies, what functional organization this preference reflects, and how it explains the effectiveness of mirror connections.
Unfortunately, the rebuttal does not substantially advance this understanding. The new experiments demonstrate that mirror pairing outperforms alternative pairing strategies, but they primarily establish the effectiveness of the architectural choice rather than explain the underlying symmetry phenomenon.
Moreover, the authors explicitly acknowledge that the CKA and gradient alignment analyses are partly induced by the direct mirror connections and therefore characterize the resulting model behavior rather than explain why the design works. This further limits the mechanistic evidence supporting the original motivation.
Consequently, while the rebuttal considerably strengthens the empirical justification for the proposed architecture, it does not substantially strengthen the scientific explanation that I regard as the paper's most important potential contribution.
I recognize the practical value of the proposed design and the authors' extensive additional work. Nevertheless, my original concern about the limited conceptual novelty and insufficient mechanistic understanding remains.
For these reasons, I will maintain my original rating of 2 (Weak Rejection).
Response to reviewer t2SS (Part 3)
Dear Reviewer t2SS,
Thank you for clarifying your concern. We fully agree with your insight. In a U-Net, mirror skips are dictated by matching resolutions. A DiT never changes its resolution, yet the symmetry bias still appears, which is well worth investigating. We apologize that our first response focused on the requested experiments and on clarifying the paper, and did not explore this question. Following your clarification, we added a further analysis of a plain DiT to investigate this. A plain DiT has no mirror connection, so what we find there comes from the DiT itself and conceptually supports our design.
1. Reverse-order dependencies in a plain DiT
In the plain DiT-XL/2 baseline at 400K iterations, we test how strongly each decoder block depends on each encoder block. To do so, we remove what one encoder block wrote into the residual stream from the input of one decoder block, and measure how much that decoder block's output changes on real noised ImageNet latents at six timesteps. Every decoder block depends on encoder block 1, which writes a large global component and takes 0.25–0.44 of every decoder group's dependency on the encoder. Table 1 shows how the rest is distributed.
Table 1. Share of the dependency of decoder blocks 1–12 on encoder blocks 2–14 in a plain DiT-XL/2, where the first 14 blocks are the encoder and the last 14 the decoder, averaged over six timesteps.
| Decoder blocks | Encoder 2–5 | Encoder 6–10 | Encoder 11–14 |
|---|---|---|---|
| 1–4 | 0.08 | 0.34 | 0.57 |
| 5–8 | 0.16 | 0.34 | 0.50 |
| 9–12 | 0.53 | 0.27 | 0.21 |
Reverse-order dependency. The deeper a decoder block is, the earlier the encoder blocks it depends on. Decoder blocks 9–12 depend 2.5 times more on encoder blocks 2–5 than on blocks 11–14 (0.53 vs 0.21), although the latter are much closer. Decoder blocks 1–4 show the opposite (0.08 vs 0.57). This ordering holds at every timestep. It also appears in DiT-L/2 and in DiT-XL/2 at 1.3M iterations. Mirror pairing connects the blocks in the same order, pairing deeper decoder blocks with earlier encoder blocks.
Last two blocks. Decoder blocks 13 and 14 depend far less on the encoder blocks than the others do. We conjecture that they mainly rescale the final state for the output layer. Such rescaling depends on the whole state, not on any single block.
2. Why this order arises and why mirror connections help
- The input and output sides of a DiT both handle dense detail. In the plain DiT, the first encoder blocks and the decoder write fine-grained content that differs between neighbouring tokens, while the late encoder blocks write smoother content (fine-grained fraction 0.58 and 0.47 against 0.36 at t = 400). The final prediction is also built mostly from the embedding and what the first four blocks write (85–99% at t ≥ 200). This suggests why late decoder blocks need what early encoder blocks wrote.
- Direct access. In a plain DiT, the early content reaches the late decoder only through the residual stream. A mirror connection hands it over directly, and the adaptive weights let each token decide how much to use.
- This order explains our pairing ablations. Mirror pairing follows the order of the plain DiT and is the best (FID 38.37). Random pairing follows it only partly (41.04). In-order pairing does the opposite and is worse than no routing at all (48.76 vs 46.59).
The dependency is graded rather than one-to-one, and mirror pairing is the simplest wiring that respects it.
Link to Fig. 5. Sec. 4.3 of the paper already hypothesized that early and late layers both handle dense detail while middle layers focus on abstraction. The new analysis supports this in a plain DiT and also explains Fig. 5(a). The accumulated states of a plain DiT look alike across depth, with a mean CKA of 0.63–0.98. What the blocks write is far less alike, with a mean CKA of 0.29–0.51. The organization is therefore hidden in the states. Mirror routing makes it visible, as the anti-diagonal of Fig. 5(b).
3. Conceptual contribution
Our analysis shows that a plain DiT already orders its cross-layer dependencies this way by itself, and our interventions show that respecting this order, together with adaptive fusion, makes the connection effective. We will add this analysis to Sec. 3.4 and Sec. 4.3 as evidence on where the symmetry bias comes from.
Thank you for your insightful comments, which helped us identify the most promising conceptual contribution of this work. We sincerely hope that you will take this new evidence into account and consider raising your score. Thank you very much!
Response to reviewer t2SS (Part 1)
Dear Reviewer t2SS,
We sincerely thank you for your careful review. Your central concern is whether the symmetry observation, the design and the experiments form a closed research loop. We ran the interventions you propose, non-mirrored one-to-one pairings, encoder-only routing and simpler fusion, on DiT-XL/2, and measured the full training cost. Below, we address each of your concerns in turn.
Observation. The mirror preference comes from the routing weights of a freely routed model, which may use every encoder block and still shows an anti-diagonal preference for mirror pairs (Fig. 4). As we also note in our reply to Reviewer 3L6x, the similarity analysis of Fig. 5 is partly induced by the direct connections, so we present it as a characterization of the resulting optimization behavior.
Table 1. Ablations on DiT-XL/2 at 100K iterations in FID-50K without guidance, averaged over seeds.
Causal test. Breaking or removing the mirror correspondence worsens FID from 38.37 to 41.04–51.73 (Table 1). Our improvement therefore depends on the mirror correspondence.
The loop. Observation in a freely routed model, design of mirror routing and interventions on the pairing therefore form a closed loop. We will add the interventions to Sec. 3.4 as direct evidence.
U-shaped skips plus adaptive aggregation. Neither ingredient alone suffices. Linear fusion on the same mirror pairs stays near the baseline at DiT-S/2 (68.06 vs 69.38), and dense adaptive aggregation by AttnRes reaches only 43.47 at DiT-XL/2 in a single run, against 38.37 for ours and 46.59 for the baseline on average, at 1.54× our step time. This suggests that the gain comes from combining stage-matched sources with adaptive fusion.
We ran an in-order pairing, in which the decoder blocks take the encoder blocks in the same order rather than in reverse. Unlike First/Last, it uses every encoder block exactly once, with everything else identical to ours. We also ran a random pairing, in which the decoder blocks take the encoder blocks in a fixed random order.
Pairing. Both pairings reuse exactly the same encoder blocks as ours, yet both are worse than mirror (Table 1). A random pairing loses about a third of our improvement over the baseline (41.04 vs 38.37). The in-order pairing is even worse than the baseline (48.76 vs 46.59). Our improvement therefore depends on which encoder block is paired with which decoder block, not only on source diversity.
Components. Both components are needed (Table 1). Removing the mirror source worsens FID from 38.37 to 51.73, below the baseline. Scalar fusion on the mirror pairs worsens it to 48.06. A timestep-conditioned fusion, which we ran at DiT-S/2, is also slightly worse than ours in both seeds (+0.25 and +1.54 FID).