acceptodds
Withdrawn from ICLR 2027

Structured Residual Connectivity Matters for Diffusion Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion Transformers (DiTs) have established themselves as a scalable backbone for high-fidelity image synthesis. However, unlike U-Net based diffusion models that rely on rigid, hand-crafted skip connections, DiTs predominantly use a uniform residual stream that integrates all preceding layers as a monolithic state. In this work, we rethink residual connections in diffusion transformers and propose to transform them from passive summation into an active retrieval mechanism optimized for image denoising. First, we conduct a systematic analysis of DiT's internal representation, revealing a latent preference for early-layer feature reuse and symmetric layer guidance. Motivated by this, we introduce a structured connectivity design that explicitly integrates local residual connections with long-range pathways. Instead of static skip connections or dense all-layer routing, our method enables each transformer block to selectively “attend” to critical earlier representations, dynamically retrieving spatial and semantic cues through direct, differentiable cross-depth paths. Experiments show that our adaptive connectivity leads to faster convergence, with up to fewer training iterations, and significant gains in FID and visual quality with less than additional parameters, further improving a strong REPA-XL/2 model from to FID without guidance and reaching FID with classifier-free guidance. Our findings suggest that adaptive cross-layer connectivity is a critical yet underexplored factor in diffusion transformers, and that incorporating structured information pathways provides a simple and effective direction for improving scalable generative models.

Primary area:
generative models
Keywords:
information flow, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t2SSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes structured residual connectivity for Diffusion Transformers (DiTs), motivated by an observed preference for symmetric cross-layer dependencies. The method combines adaptive residual aggregation with sparse mirror connections, achieving improved FID and training efficiency across multiple DiT models.

Critical strengths

The paper addresses a meaningful problem in DiT architecture design. The proposed method consistently improves FID across different model scales and achieves a substantial improvement on pretrained REPA (5.90 → 4.34 FID), with negligible additional parameters and computational overhead.

Critical weaknesses
  • Insufficient connection between scientific observations and method design.

    My primary concern is that the research motivation, mechanistic analysis, method design, and experimental validation do not form a convincing research loop. The most potentially valuable contribution is the observation of Spontaneous Symmetry Bias in DiTs. However, the evidence mainly consists of learned routing weights and CKA/gradient similarity analyses. These results do not sufficiently explain why symmetric dependencies emerge, what functional role they serve, or whether they are responsible for the performance improvements. Without a convincing validation of this observation, the proposed method largely resembles a combination of existing U-shaped skip connections and adaptive residual aggregation, making its distinctive contribution unclear.

  • Ablation studies do not directly validate the central hypothesis.

    The comparisons among Fused-First, Fused-Last, Fused-All, and Fused-Mirror do not adequately isolate the effect of mirror symmetry. For instance, Mirror routing accesses different encoder depths, whereas First/Last routing repeatedly accesses the same representation. The improvement may therefore result from more diverse feature reuse rather than symmetric pairing.Comparisons with random or shifted one-to-one connections would provide more direct evidence. An encoder-only routing ablation is also missing.

  • Inconsistent evaluation results.

    Although Fused-Mirror achieves the best FID in Table 3, its sFID (12.91) is the worst among the modified architectures. This discrepancy is not discussed, despite the paper's emphasis on spatial information reuse. While the differences are small and their statistical significance is unclear, the results do not consistently support the claimed advantages.

  • Limited distinctive practical advantages.

    Engineering combinations can be valuable when they provide substantial practical improvements or establish generalizable design principles. However, this paper does not convincingly establish either. Compared with Attention Residuals, the FID improvement is modest (62.87 to 62.48), with slightly worse sFID. The substantial REPA gains are encouraging but lack comparisons against alternative routing designs under matched fine-tuning budgets. Furthermore, the reported efficiency benefits are not supported by comprehensive end-to-end training cost measurements.

Questions for authors

Q1. On Spontaneous Symmetry Bias. Can the authors provide more direct evidence that the observed symmetric layer dependencies reflect a meaningful functional organization within DiTs? In particular, how can this preference be distinguished from effects induced by the adaptive routing architecture itself?

Q2. On the necessity of mirror pairing. How does Mirror routing compare with random or shifted one-to-one encoder–decoder connections under identical parameter counts, routing operators, and training budgets? Such comparisons would help isolate the specific contribution of mirror symmetry.

Q3. On the independent contributions of architectural components. What is the performance of encoder-side adaptive routing without decoder-side mirror connections? Additionally, how does the proposed fusion operator compare with simpler gated skip connections?

Q4. On the inconsistent FID and sFID results. Why does Fused-Mirror achieve the best FID but the worst sFID among the Fused variants in Table 3? Are these differences consistent across independent runs, and how do they relate to the claimed benefits of spatial information reuse?

Q5. On practical advantages over existing methods. Can the authors clarify whether the substantial REPA fine-tuning improvements can also be achieved using alternative routing architectures under the same training budget? Could they also provide end-to-end training time and peak memory comparisons?

4 comments
AuthorsOpenReview ↗

Response to reviewer t2SS (Part 1)

Dear Reviewer t2SS,

We sincerely thank you for your careful review. Your central concern is whether the symmetry observation, the design and the experiments form a closed research loop. We ran the interventions you propose, non-mirrored one-to-one pairings, encoder-only routing and simpler fusion, on DiT-XL/2, and measured the full training cost. Below, we address each of your concerns in turn.

1. The symmetry observation and the research loop (W1, Q1)

Observation. The mirror preference comes from the routing weights of a freely routed model, which may use every encoder block and still shows an anti-diagonal preference for mirror pairs (Fig. 4). As we also note in our reply to Reviewer 3L6x, the similarity analysis of Fig. 5 is partly induced by the direct connections, so we present it as a characterization of the resulting optimization behavior.

Table 1. Ablations on DiT-XL/2 at 100K iterations in FID-50K without guidance, averaged over seeds.

VariantPairingFusionFID ↓
OursMirrorAdaptive38.37
Pairing ablationIn-orderAdaptive48.76
Pairing ablationRandomAdaptive41.04
Source ablationNone, encoder routing only–51.73
Fusion ablationMirrorScalar48.06
DiT baseline––46.59

Causal test. Breaking or removing the mirror correspondence worsens FID from 38.37 to 41.04–51.73 (Table 1). Our improvement therefore depends on the mirror correspondence.

The loop. Observation in a freely routed model, design of mirror routing and interventions on the pairing therefore form a closed loop. We will add the interventions to Sec. 3.4 as direct evidence.

U-shaped skips plus adaptive aggregation. Neither ingredient alone suffices. Linear fusion on the same mirror pairs stays near the baseline at DiT-S/2 (68.06 vs 69.38), and dense adaptive aggregation by AttnRes reaches only 43.47 at DiT-XL/2 in a single run, against 38.37 for ours and 46.59 for the baseline on average, at 1.54× our step time. This suggests that the gain comes from combining stage-matched sources with adaptive fusion.

2. Non-mirrored one-to-one pairings (W2, Q2)

We ran an in-order pairing, in which the decoder blocks take the encoder blocks in the same order rather than in reverse. Unlike First/Last, it uses every encoder block exactly once, with everything else identical to ours. We also ran a random pairing, in which the decoder blocks take the encoder blocks in a fixed random order.

Pairing. Both pairings reuse exactly the same encoder blocks as ours, yet both are worse than mirror (Table 1). A random pairing loses about a third of our improvement over the baseline (41.04 vs 38.37). The in-order pairing is even worse than the baseline (48.76 vs 46.59). Our improvement therefore depends on which encoder block is paired with which decoder block, not only on source diversity.

3. Encoder-only routing and simpler fusion (Q3)

Components. Both components are needed (Table 1). Removing the mirror source worsens FID from 38.37 to 51.73, below the baseline. Scalar fusion on the mirror pairs worsens it to 48.06. A timestep-conditioned fusion, which we ran at DiT-S/2, is also slightly worse than ours in both seeds (+0.25 and +1.54 FID).

AuthorsOpenReview ↗

Response to reviewer t2SS (Part 2)

4. FID vs. sFID (W3, Q4)

Table 2. FID-50K and other metrics of DiT-XL/2 at 100K iterations without guidance, shown as mean ± SD over three training seeds.

ModelFID ↓sFID ↓IS ↑Precision ↑Recall ↑
DiT baseline46.59 ± 1.087.33 ± 0.0327.7 ± 0.80.502 ± 0.0080.599 ± 0.005
Ours38.37 ± 2.416.49 ± 0.1535.0 ± 2.30.544 ± 0.0160.598 ± 0.005

Multiple seeds. Over three seeds at DiT-XL/2, ours improves sFID from 7.33 to 6.49 and is better in every seed (Table 2). This is consistent with the spatial information reuse of mirror routing. At DiT-S/2, our mean sFID over four runs (12.80) is not worse than Fused-All (12.85), Fused-First (12.83) or the baseline (12.93). These gaps are smaller than the run-to-run spread, so the sFID ordering among the Fused variants in the paper's Table 3 is within noise.

FID as the primary metric. In most papers on class-conditional ImageNet generation, FID is the primary metric, while sFID is a complementary measure.

5. Practical advantages (W4, Q5)

(a) Against Attention Residuals

The AttnRes in the paper is already an improved version of the original AttnRes, as it stores the accumulated state after each block. At DiT-S/2 it reaches 63.31 FID, against 66.05 for the original Block AttnRes. We therefore compared against the strongest AttnRes. Ours matches it in FID at DiT-S/2, as you note (63.03 vs 63.31 on average), but is faster, lighter and better at depth. At DiT-XL/2 ours needs 35% less step time and 46% less peak memory (Table 3). It also reaches 38.37 against 43.47 for AttnRes after 100K steps. Fused-All, the other dense variant, even falls behind the baseline at XL/2.

(b) REPA with an alternative routing under the same budget

We fine-tuned the in-order pairing from the released REPA checkpoint with the same recipe and budget as ours. With guidance at ww = 1.7, it reaches 1.433, level with the step-matched control at 1.431, while ours reaches 1.387.

(c) Training time and peak memory

Table 3. Parameters, FLOPs and training cost of DiT-XL/2, with step time and peak memory measured with torch.compile at batch 16 on one B200 GPU.

ModelParameters (M) ↓GFLOPs per image ↓Step time (ms) ↓Peak memory (GB) ↓
DiT baseline (no routing)675.13114.4650.116.4
Mirror pairing, linear fusion (U-ViT-style)709.65 (+5.1%)123.29 (+7.7%)52.1 (1.04×)17.0 (1.04×)
Ours675.26 (+0.02%)114.62 (+0.1%)79.9 (1.59×)25.3 (1.54×)
AttnRes675.26 (+0.02%)114.94 (+0.4%)122.8 (2.45×)46.5 (2.84×)

Cost. The U-ViT-style skip adds 5.1% parameters and 7.7% FLOPs but little time, whereas ours adds almost no parameters or FLOPs but takes 1.59× the step time, from stacking and normalising candidate states in unfused code (Table 3). Ours is still faster and lighter than AttnRes (Table 3). Ours also needs 1.33–1.73× fewer steps than the baseline to reach the same FID (reply to Reviewer 3L6x, Table 6).

Reviewer_t2SSOpenReview ↗

Response

Thank you for the detailed rebuttal and the substantial additional experiments. I appreciate the authors' efforts to address my concerns.

The new ablation studies provide convincing evidence that mirror pairing and adaptive fusion both contribute to the performance improvements. The additional multi-seed evaluations and training cost measurements also strengthen the empirical results. I acknowledge these improvements.

However, my primary concern regarding the scientific contribution of the paper remains unresolved.

As emphasized in my original review, I consider the observation of Spontaneous Symmetry Bias in DiTs to be the most potentially valuable contribution of this work. Symmetric skip connections have already been extensively explored in architectures such as U-Net and U-ViT, while adaptive residual aggregation has also been studied. Therefore, simply demonstrating that their combination improves generation performance offers relatively limited conceptual novelty.

What could make this work scientifically distinctive is a deeper understanding of why DiTs spontaneously develop symmetric cross-layer dependencies, what functional organization this preference reflects, and how it explains the effectiveness of mirror connections.

Unfortunately, the rebuttal does not substantially advance this understanding. The new experiments demonstrate that mirror pairing outperforms alternative pairing strategies, but they primarily establish the effectiveness of the architectural choice rather than explain the underlying symmetry phenomenon.

Moreover, the authors explicitly acknowledge that the CKA and gradient alignment analyses are partly induced by the direct mirror connections and therefore characterize the resulting model behavior rather than explain why the design works. This further limits the mechanistic evidence supporting the original motivation.

Consequently, while the rebuttal considerably strengthens the empirical justification for the proposed architecture, it does not substantially strengthen the scientific explanation that I regard as the paper's most important potential contribution.

I recognize the practical value of the proposed design and the authors' extensive additional work. Nevertheless, my original concern about the limited conceptual novelty and insufficient mechanistic understanding remains.

For these reasons, I will maintain my original rating of 2 (Weak Rejection).

AuthorsOpenReview ↗

Response to reviewer t2SS (Part 3)

Dear Reviewer t2SS,

Thank you for clarifying your concern. We fully agree with your insight. In a U-Net, mirror skips are dictated by matching resolutions. A DiT never changes its resolution, yet the symmetry bias still appears, which is well worth investigating. We apologize that our first response focused on the requested experiments and on clarifying the paper, and did not explore this question. Following your clarification, we added a further analysis of a plain DiT to investigate this. A plain DiT has no mirror connection, so what we find there comes from the DiT itself and conceptually supports our design.

1. Reverse-order dependencies in a plain DiT

In the plain DiT-XL/2 baseline at 400K iterations, we test how strongly each decoder block depends on each encoder block. To do so, we remove what one encoder block wrote into the residual stream from the input of one decoder block, and measure how much that decoder block's output changes on real noised ImageNet latents at six timesteps. Every decoder block depends on encoder block 1, which writes a large global component and takes 0.25–0.44 of every decoder group's dependency on the encoder. Table 1 shows how the rest is distributed.

Table 1. Share of the dependency of decoder blocks 1–12 on encoder blocks 2–14 in a plain DiT-XL/2, where the first 14 blocks are the encoder and the last 14 the decoder, averaged over six timesteps.

Decoder blocksEncoder 2–5Encoder 6–10Encoder 11–14
1–40.080.340.57
5–80.160.340.50
9–120.530.270.21

Reverse-order dependency. The deeper a decoder block is, the earlier the encoder blocks it depends on. Decoder blocks 9–12 depend 2.5 times more on encoder blocks 2–5 than on blocks 11–14 (0.53 vs 0.21), although the latter are much closer. Decoder blocks 1–4 show the opposite (0.08 vs 0.57). This ordering holds at every timestep. It also appears in DiT-L/2 and in DiT-XL/2 at 1.3M iterations. Mirror pairing connects the blocks in the same order, pairing deeper decoder blocks with earlier encoder blocks.

Last two blocks. Decoder blocks 13 and 14 depend far less on the encoder blocks than the others do. We conjecture that they mainly rescale the final state for the output layer. Such rescaling depends on the whole state, not on any single block.

2. Why this order arises and why mirror connections help

  • The input and output sides of a DiT both handle dense detail. In the plain DiT, the first encoder blocks and the decoder write fine-grained content that differs between neighbouring tokens, while the late encoder blocks write smoother content (fine-grained fraction 0.58 and 0.47 against 0.36 at t = 400). The final prediction is also built mostly from the embedding and what the first four blocks write (85–99% at t ≥ 200). This suggests why late decoder blocks need what early encoder blocks wrote.
  • Direct access. In a plain DiT, the early content reaches the late decoder only through the residual stream. A mirror connection hands it over directly, and the adaptive weights let each token decide how much to use.
  • This order explains our pairing ablations. Mirror pairing follows the order of the plain DiT and is the best (FID 38.37). Random pairing follows it only partly (41.04). In-order pairing does the opposite and is worse than no routing at all (48.76 vs 46.59).

The dependency is graded rather than one-to-one, and mirror pairing is the simplest wiring that respects it.

Link to Fig. 5. Sec. 4.3 of the paper already hypothesized that early and late layers both handle dense detail while middle layers focus on abstraction. The new analysis supports this in a plain DiT and also explains Fig. 5(a). The accumulated states of a plain DiT look alike across depth, with a mean CKA of 0.63–0.98. What the blocks write is far less alike, with a mean CKA of 0.29–0.51. The organization is therefore hidden in the states. Mirror routing makes it visible, as the anti-diagonal of Fig. 5(b).

3. Conceptual contribution

Our analysis shows that a plain DiT already orders its cross-layer dependencies this way by itself, and our interventions show that respecting this order, together with adaptive fusion, makes the connection effective. We will add this analysis to Sec. 3.4 and Sec. 4.3 as evidence on where the symmetry bias comes from.

Thank you for your insightful comments, which helped us identify the most promising conceptual contribution of this work. We sincerely hope that you will take this new evidence into account and consider raising your score. Thank you very much!

Reviewer_XkeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adaptive cross-depth feature aggregation in diffusion transformers. The proposed architecture collects differentiable representations from the encoder-side portion of the network and introduces routing operators before the self-attention and MLP sublayers. These operators use learned, content-dependent softmax weights to combine the current representation with selected historical representations. The final design restricts decoder-side historical access to a mirrored encoder representation. Experiments on class-conditional ImageNet-256 generation report improvements across DiT model sizes and during fine-tuning of a pretrained REPA-enhanced SiT model. Representation-similarity and gradient analyses are used to characterize the resulting organization across depth.

Critical strengths
  1. The improvements over the standard residual architecture are substantial relative to the additional parameter count. At 400K iterations, the reported DiT-XL/2 FID improves from 18.85 to 15.07, with less than 0.1% additional parameters. Improvements across the S, B, and XL configurations provide evidence that the benefit is not confined to one model size.
  2. The pretrained-model experiment includes a meaningful continued-training control. Starting from the released REPA checkpoint, the proposed routing reaches 4.34 unguided FID after 350K additional updates, whereas continued fine-tuning without routing reaches 6.01 under the corresponding budget. Appendix A specifies that both settings omit the REPA projection loss during fine-tuning. This comparison supports the usefulness of modifying connectivity beyond simply allocating additional optimization steps.
Critical weaknesses
  1. The contribution relative to directly relevant adaptive-routing work is not sufficiently established. The manuscript acknowledges that its source-scoring mechanism is shared with Attention Residuals and that mirror pairing resembles existing U-shaped connections. More importantly, it does not discuss Rethinking Cross-Layer Information Routing in Diffusion Transformers (DAR; arXiv:2605.20708), whose publicly available May 2026 version already investigates adaptive cross-layer routing in DiTs, representation redundancy, gradient behavior, and compatibility with REPA. DAR is not identical to the proposed mirror architecture, but it is a direct comparator for the broader motivation and claimed benefits. Accordingly, the distinctive contribution should be established as the particular structured routing design and its quality–cost trade-off, rather than adaptive residual retrieval in diffusion transformers generally. This requires comparison with relevant efficient alternatives, including chunked or block-based routing, rather than only dense AttnRes.

  2. The evidence does not yet isolate the importance of mirror correspondence or substantiate the stronger mechanistic interpretation. Table 3 compares several useful variants, but does not compare mirror pairing with other equally sparse pairings under the same fusion operator. It also does not isolate content dependence from simpler scalar or timestep-dependent gates with matched normalization and scaling. The FID difference between Fused-Mirror and AttnRes on DiT-S/2 is relatively small, 62.48 versus 62.87, without reported uncertainty sufficient to establish a reliable quality advantage. Furthermore, Appendix C computes gradient similarity using the squared output norm on four random latents at a single timestep. This characterizes the network under that probe objective; it does not directly establish improved optimization signals under the denoising training loss or the proposed semantic roles of different depths. The main text appropriately acknowledges that mirror connections partly induce gradient alignment, but the stronger claim of a latent structural necessity remains unsupported. Finally, Table 4 measures decoder-side source-stack activations and explicitly excludes encoder-side routing, so it does not establish an equivalent reduction in total training memory.

Questions for authors
  1. Positioning and comparison: What is the specific methodological advantage over DAR, and how does the proposed design compare with efficient chunked or Block AttnRes alternatives under matched training and evaluation settings?

  2. Baseline implementation clarification: In the AttnRes reimplementation, are the stored sources individual sublayer transformation outputs before residual accumulation, or accumulated block states? The original Full AttnRes formulation uses the former. Please provide the implemented recurrence and explain its correspondence to the original method. This distinction could materially affect both the quality and memory comparisons.

  3. Structural attribution: With the fusion operator held fixed, does mirror pairing consistently outperform random or offset pairings? Does content-dependent fusion outperform appropriately normalized learned scalar or timestep-dependent mixing? Verification of the decisive comparisons beyond DiT-S/2 would strengthen the architectural conclusion.

  4. Mechanism and efficiency: Do the gradient findings persist when measured using the actual denoising loss on appropriately corrupted data across timesteps? What are the full-model peak training memory and wall-clock time to a matched FID, including encoder-side routing?

3 comments
AuthorsOpenReview ↗

Response to reviewer XkeN (Part 1)

Dear Reviewer XkeN,

We sincerely thank you for your careful and specific review. We agree with your framing that our distinctive contribution is the structured routing design and its quality–cost trade-off. We compared our model with DAR-style and AttnRes variants under matched settings. We also ran the pairing and fusion controls on DiT-XL/2 and measured the full-model training cost. Below, we address each of your concerns in turn.

1. Relation to DAR and to chunked or Block AttnRes (W1, Q1)

DAR [1] is contemporaneous under the ICLR guidelines and directly relevant. We will discuss it in related work. It differs from ours in four ways.

  • Sources. DAR mixes all earlier sublayer outputs in chunks, with no fixed pairing. In the decoder, ours mixes only the current state and one mirror encoder state.
  • Residual. DAR replaces the residual sum with the routed mix. In the decoder, ours keeps the current state as a candidate and adds the sublayer output.
  • Timestep. DAR adds the timestep embedding to the routing query. Ours sees the timestep only through adaLN-conditioned features.
  • REPA. DAR trains from scratch with the alignment loss. We fine-tune the released REPA checkpoint without it.

Table 1. FID-50K of DiT-S/2 at 400K iterations averaged over training seeds, with compiled step time and peak memory relative to the baseline.

VariantFID ↓Step time ↓Memory ↓
DiT baseline (no routing)69.381×1×
Ours (mirror pairing, adaptive fusion)63.031.84×1.86×
DAR-style, chunk 4¹64.732.49×3.16×
AttnRes, block-state63.312.29×3.46×
Block AttnRes (original definition)66.052.29×3.46×
Full AttnRes (original definition)66.032.89×5.46×

¹ DAR has not released its code, so this row is our re-implementation, as depicted in their paper.

Matched comparison. Ours has the best mean FID among the routing variants at the lowest cost (Table 1). It needs 1.2–1.6× less step time and 1.7–2.9× less memory than the DAR-style and AttnRes variants. This suggests that a sparse, stage-matched source set captures most of the benefit of routing over all earlier states.

2. What our AttnRes implementation stores (Q2)

All three AttnRes variants use our router. At each routing point, one learned query scores the normalised candidates, and a softmax mixes them. They differ only in what the candidates are.

  • Block-state AttnRes, used in the paper. One candidate per earlier block, namely its state after the residual addition.
  • Block AttnRes (original definition). One candidate per earlier block, namely only its output before the residual addition.
  • Full AttnRes (original definition). One candidate per earlier sublayer, namely its output before the residual addition.

Accumulated states work better. The block-state AttnRes in the paper is an improved version of the original AttnRes, as it stores the accumulated state after each block. At DiT-S/2 it reaches better FID 63.31, against 66.05 for the original Block AttnRes (Table 1). The two differ only in what they store. The paper therefore compared against the strongest AttnRes.

Comparison with ours. Full and Block AttnRes are both 3.0 FID worse than ours. Full AttnRes also needs about three times our memory. We will clarify what the AttnRes row stores and add the three definitions to the appendix.

AuthorsOpenReview ↗

Response to reviewer XkeN (Part 2)

3. Structural attribution of pairing and fusion (W2, Q3)

To isolate the roles of mirror pairing and adaptive fusion, we replaced one of them at a time on DiT-XL/2, with everything else identical.

  • In-order pairing. The decoder blocks take the encoder blocks in the same order rather than in reverse.
  • Random pairing. The decoder blocks take the encoder blocks in a fixed random order, with no mirror pair.
  • Scalar fusion. One learned weight per sublayer replaces our adaptive, content-dependent weights.

Table 2. Ablations on DiT-XL/2 at 100K iterations in FID-50K without guidance, for each training seed and on average.

VariantPairingFusionFID ↓, seeds 0 / 1 / 2Mean FID ↓
OursMirrorAdaptive37.91 / 36.23 / 40.9838.37
Pairing ablationIn-orderAdaptive49.18 / 50.25 / 46.8448.76
Pairing ablationRandomAdaptive41.21 / 40.01 / 41.9041.04
Fusion ablationMirrorScalar52.73 / 45.62 / 45.8348.06
DiT baseline––46.09 / 47.84 / 45.8546.59

Mirror pairing. A random pairing loses about a third of our improvement over the baseline (41.04 vs 38.37, Table 2). The in-order pairing is even worse than the baseline (48.76 vs 46.59). Our improvement thus depends on which encoder block feeds which decoder block.

Adaptive fusion. Scalar fusion on the same mirror pairs worsens FID from 38.37 to 48.06 (Table 2). This suggests that the decoder benefits from deciding per token how much of the mirror source to use. A timestep-conditioned fusion, which we ran at DiT-S/2, is also slightly worse than ours in both seeds (+0.25 and +1.54 FID).

Multiple seeds. Ours is better than every other variant in every training seed (Table 2). Our improvement over all these variants is therefore consistent across seeds.

4. Gradients under the actual denoising loss (W2, Q4)

Yes, the gradient alignment of Fig. 5 persists. We recomputed it for DiT-XL/2 at 400K iterations with DiT's training loss on noised ImageNet latents, at six timesteps and at random timesteps. The two outermost mirror pairs have more aligned gradients in ours than in the plain DiT at every timestep.

We agree that this alignment is partly induced by the direct mirror connections. It therefore describes how our model behaves, rather than why the design works. We will revise the corresponding statements.

5. Full-model peak memory and wall-clock time to a matched FID (W2, Q4)

Table 3. Full-model training cost of DiT-XL/2 including encoder-side routing, measured with torch.compile at batch 16 on one B200 GPU.

ModelParameters (M) ↓GFLOPs per image ↓Step time (ms) ↓Peak memory (GB) ↓
DiT baseline (no routing)675.13114.4650.116.4
Ours675.26 (+0.02%)114.62 (+0.1%)79.9 (1.59×)25.3 (1.54×)
AttnRes675.26 (+0.02%)114.94 (+0.4%)122.8 (2.45×)46.5 (2.84×)

Memory. The decoder-side saving of the paper's Table 4 carries over to the full model. Ours needs 46% less peak memory than AttnRes (Table 3). The 1.59× step time comes from stacking and normalising candidate states in unfused code.

Table 4. Training steps and wall-clock time for DiT-XL/2 to reach the same FID, with our wall-clock projected from the step times of Table 3.

Baseline steps (FID)Our steps (FID)Step speed-up ↑Our wall-clock / baseline ↓
400K (18.85)300K (17.78)1.33×1.20×
1M (12.65)600K (12.65)1.67×0.96×
1.3M (11.76)750K (11.74)1.73×0.92×

Wall-clock. Ours needs 1.33–1.73× fewer steps to reach the same FID (Table 4). From about 600K steps on, this offsets its 1.59× step time, at 0.92–0.96× the baseline's wall-clock. For the earliest target, ours still takes 1.20×. Ours is also faster than AttnRes, which takes 1.54× our step time and is still worse after the same 100K steps (43.47 vs 38.37).

References

[1] Xu C, Li M, Li Q, et al. Rethinking Cross-Layer Information Routing in Diffusion Transformers. NeurIPS 2026.

Thank you for the detailed responses. The additional three-seed XL pairing/fusion ablations and full-model cost measurements alleviate my concerns about structural attribution and efficiency evaluation. However, two concerns remain:

  1. The implementation of the key routing baselines requires further clarification. The response primarily specifies which candidate representations are stored, without providing the complete recurrence. For original Block AttnRes, the grouping granularity, within-block accumulation, current partial sum, and treatment of the initial embedding remain unclear. I therefore cannot yet verify correspondence with the original formulation. Similarly, DAR considers multiple query parameterizations, rather than only explicit timestep injection; please specify the configuration used in the DAR-style comparison.

  2. The comparative quality–cost advantage remains insufficiently established. The new ablations support mirror pairing and adaptive fusion, but the mean FID difference from block-state AttnRes is small (63.03 versus 63.31), without uncertainty estimates establishing a reliable quality advantage. The cost measurements are useful but implementation-dependent, and matched-FID training times are projected from per-step measurements. The evidence therefore supports an effective structural choice under the tested implementation more clearly than an overall advantage over related routing methods.

Overall, I acknowledge the value of the additional experiments, but retain reservations about the adequacy of the baseline comparisons and the significance of the incremental contribution. I therefore maintain Weak Rejection.

Reviewer_3L6xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies residual connections in Diffusion Transformers and proposes structured residual routing. Each decoder sublayer mixes its current representation with the feature from its mirrored encoder layer, using a small content-dependent softmax gate. The design comes from two observations in an all-layer routing experiment: DiTs rely heavily on early-layer features, and they tend to prefer mirrored layer pairs. On ImageNet 256x256, the method improves FID for DiT-S/B/XL and reaches the baseline FID with up to 1.73x fewer iterations, with less than 0.1% extra parameters. It also improves a pretrained SiT-XL/2 + REPA model from 5.9 to 4.34 FID with 0.35M extra fine-tuning steps.

Critical strengths
  • The improvements are consistent. They hold for three DiT sizes and also when fine-tuning a strong pretrained REPA model. Continued fine-tuning of the original architecture with the same extra budget resulted in almost no gains but adding the routing did.

  • The method is straightforward. It adds almost no parameters and uses much less decoder-side routing memory.

  • The ablations against a static U-ViT-style skip and other routing choices (all/first/last) help show where the benefit comes from.

Critical weaknesses
  1. It is unclear how much of the improvement comes specifically from mirror pairing, as opposed to adding adaptive skip connections more generally. In the final design, each decoder sublayer uses a two-way softmax to mix its current representation with a single mirrored encoder feature.

  2. The CKA and gradient analyses are useful, but higher similarity between mirrored layers is partly expected, since the method connects them directly. So these analyses describe the result rather than show that the symmetric dependencies cause the FID gains.

  3. All experiments are on ImageNet 256x256, so we don't know if mirror routing is still the best choice for text-to-image or video models.

  4. There are also no results over multiple seeds and statistical significance testing. This matters most for the small gaps between routing variants (e.g., for the guided REPA result, FID goes from 1.42 to 1.39, but IS, precision and recall worsen slightly).

Questions for authors
  1. How much of the improvement comes from mirror pairing versus the adaptive fusion mechanism? Could you compare against (i) non-mirrored one-to-one pairings using the same routing operator and (ii) a simple learnable scalar gate using the same mirrored connections? These controls would help isolate the contributions of the pairing structure and the routing mechanism.

  2. How robust are the differences between mirror routing and the other adaptive variants? Can you report results over several seeds?

  3. Could you report end-to-end training throughput, wall-clock time, and peak GPU memory relative to the baseline and AttnRes? In particular, does the reduction in training iterations translate into comparable wall-clock savings when measured at matched FID?

2 comments
AuthorsOpenReview ↗

Response to reviewer 3L6x (Part 1)

Dear Reviewer 3L6x,

We sincerely thank you for your careful review and constructive suggestions. Your questions ask for exactly the controls that separate mirror pairing from adaptive fusion. We ran both controls on DiT-XL/2, added results over multiple seeds and on ImageNet 512×512512\times512, and measured the full-model training cost. Below, we address each of your concerns in turn.


1. Mirror pairing vs. adaptive fusion (W1, Q1)

We implemented both controls inside our routing operator, with everything else identical.

  • In-order pairing. The decoder blocks take the encoder blocks in the same order rather than in reverse.
  • Scalar fusion. One learned weight per sublayer replaces our adaptive, content-dependent weights.

Table 1. Ablation of the two components on DiT-XL/2 at 100K iterations in FID-50K without guidance, averaged over three training seeds.

VariantPairingFusionFID ↓
OursMirrorAdaptive38.37
Pairing ablationIn-orderAdaptive48.76
Fusion ablationMirrorScalar48.06
DiT baseline––46.59

Mirror pairing. Pairing the blocks in order instead of in reverse worsens FID from 38.37 to 48.76, even worse than the baseline (Table 1). Our improvement over the baseline therefore depends on which encoder block feeds which decoder block, not merely on reusing encoder features.

Adaptive fusion. Scalar fusion on the same mirror pairs worsens FID from 38.37 to 48.06 (Table 1). This suggests that the decoder benefits from deciding per token how much of the mirror source to use.

2. Beyond ImageNet 256×256256\times256 (W3)

To test whether the benefit extends beyond 256×256256\times256, we trained three model sizes, DiT-S/2, DiT-B/2 and DiT-L/2, on ImageNet at 512×512512\times512.

Table 2. FID-50K without guidance on ImageNet 512×512512\times512 at 500K iterations.

FID ↓DiT-S/2DiT-B/2DiT-L/2
Baseline82.6550.1824.43
Ours77.2045.5420.88
Relative gain ↑6.6%9.2%14.5%

Higher resolution. Ours improves FID at all three model sizes, and the relative gain grows from 6.6% for S/2 to 14.5% for L/2 (Table 2). Ours also improves FID at every other evaluated budget. We will add these results to the appendix.

Text-to-image and video. We agree that these are important next settings. Their outputs also rely on fine spatial detail, which our early-to-late pathway is designed to preserve, so we see them as a promising direction for future work.

3. Multiple seeds and statistical significance (W4, Q2)

(a) Multiple seeds at DiT-XL/2

We trained the baseline, ours and the two controls of Section 1 with three seeds at DiT-XL/2.

Table 3. FID-50K of DiT-XL/2 at 100K iterations for each training seed and their mean.

VariantFID ↓, seeds 0 / 1 / 2Mean FID ↓
DiT baseline (no routing)46.09 / 47.84 / 45.8546.59
Ours (mirror pairing, adaptive fusion)37.91 / 36.23 / 40.9838.37
In-order pairing, adaptive fusion49.18 / 50.25 / 46.8448.76
Mirror pairing, scalar fusion52.73 / 45.62 / 45.8348.06
Fused-All50.54 / – / –50.54
AttnRes43.47 / – / –43.47

Significance. Ours is better than the baseline and every other variant in every seed, improving on the baseline by 8.2 FID on average (Table 3), so our improvement does not depend on a particular seed.

(b) The guided REPA result (1.42 → 1.39)

We evaluated ours and the step-matched control with six sampling seeds at the same guidance scale.

Table 4. Guided sampling results at ww = 1.7 of SiT-XL/2 after 0.28M fine-tuning steps from the released REPA checkpoint without the alignment loss, shown as mean ± SD over six sampling seeds.

ModelFID ↓sFID ↓IS ↑Precision ↑Recall ↑
Step-matched control1.431 ± 0.0124.620 ± 0.041294.0 ± 0.80.792 ± 0.0020.648 ± 0.001
Ours1.387 ± 0.0224.457 ± 0.023298.1 ± 0.70.794 ± 0.0030.641 ± 0.003

Multiple seeds. Ours improves FID over the step-matched control from 1.431 to 1.387 at ww = 1.7 and is better on every seed (Table 4).

IS, precision and recall. At the same guidance scale, ours also has a higher IS, on-par precision and recall, so IS and precision do not decrease much once the guidance scale is matched.

AuthorsOpenReview ↗

Response to reviewer 3L6x (Part 2)

4. Throughput, wall-clock time and peak memory (Q3)

Thank you for this question. We measured the training cost of the full model, including encoder-side and output routing, on one B200 GPU.

Table 5. Parameters, FLOPs and training cost of DiT-XL/2, with step time and peak memory measured with torch.compile at batch 16, where throughput is the inverse of step time.

ModelParameters (M) ↓GFLOPs per image ↓Step time (ms) ↓Peak memory (GB) ↓
DiT baseline (no routing)675.13114.4650.116.4
Ours675.26 (+0.02%)114.62 (+0.1%)79.9 (1.59×)25.3 (1.54×)
AttnRes675.26 (+0.02%)114.94 (+0.4%)122.8 (2.45×)46.5 (2.84×)

Cost. Parameters and FLOPs barely change (Table 5), so the per-step overhead of 1.59× comes from stacking and normalising candidate states in unfused code and affects every adaptive variant. Our model is also faster and lighter than AttnRes, needing 35% less step time and 46% less peak memory (Table 5).

Table 6. Training steps and wall-clock time for DiT-XL/2 to reach the same FID, with our wall-clock projected from the compiled step times of Table 5.

Baseline steps (FID)Our steps (FID)Step speed-up ↑Our wall-clock / baseline ↓
400K (18.85)300K (17.78)1.33×1.20×
1M (12.65)600K (12.65)1.67×0.96×
1.3M (11.76)750K (11.74)1.73×0.92×

Wall-clock. Ours needs 1.33–1.73× fewer steps to reach the same FID, which offsets its 1.59× step time from about 600K steps on (0.92–0.96× the baseline's wall-clock), while for the earliest target ours still takes 1.20× (Table 6). Ours is also faster than AttnRes in wall-clock, since AttnRes takes 1.54× our step time and is still worse after the same 100K steps (43.47 vs 38.37, Table 3).


5. CKA and gradient analyses (W2)

Thank you for finding these analyses useful. We agree that higher similarity between mirrored layers is partly expected once they are directly connected, so Fig. 5 should not be presented as independent evidence for the architectural motivation. Instead, it characterizes an optimization effect induced by the proposed connectivity. We will revise the corresponding statements in the paper accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.