acceptodds
Under review as a conference paper at ICLR 2027

Reading and Writing Residual States for Diffusion Transformers

Abstract

Diffusion Transformers have advanced mainly through improved backbones, representations, and objectives, while the residual pathway that carries information across depth remains largely fixed. This work investigates an orthogonal question, i.e., how should a denoiser read from and write to multiple persistent residual states? We present a systematic study of this design in diffusion Transformers, using Gated Residual as a starting point. Controlled experiments show that adaptive access improves generation and that the scale of the mixed feature substantially affects its effectiveness. We introduce a direction-calibrated reader that preserves the direction selected by adaptive gates and matches the mixture's magnitude to an ungated average of normalized states. For writing, we find that modulating the update component orthogonal to each stored state outperforms modulating the entire update. This motivates a noise-conditioned tangent writer that controls this component according to the noise level. Both designs retain a single forward of each attention and MLP sublayer across states. The resulting method improves both pixel- and latent-space diffusion models and remains compatible with representation alignment methods. Code will be made publicly available.

Primary area:
generative models
Keywords:
Diffusion Models, Residual

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b3d3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an orthogonal question, how should a denoiser read from and write to multiple persistent residual states? Introducing a direction-calibrated reader that preserves the direction selected by adaptive gates and matches the mixture’s magnitude to an ungated average of normalized states. For writing, presenting a noise-conditioned tangent writer that controls this component according to the noise level.

Critical strengths

The method modifies internal residual connections, normalization positions and branch activations. The modification is localized and can be plugged into existing DiT-like architectures. Based on controlled comparisons of readout scale and update geometry, they develop a direction-calibrated reader and a noise-conditioned tangent writer, and demonstrate improved generation across pixel- and latent-space diffusion backbones, extending to models trained with representation alignment.

Critical weaknesses
  1. Despite designing a comparatively intricate structure, the work cannot attain the best experimental results, such as, compared with SiT-XL/2 + U-REPA (1.41), + REG (1.40), + REPA (1.42).
  2. The SOTAs have longer training epochs >200, while the method only has 80 epochs, lack of longer training results.
  3. The architecture and hyperparameter choices in this work are largely empirically driven, lacking theoretical analysis and plausible justifications.
  4. Evaluations heavily focus on FID, with insufficient reporting on training stability statistics, inference latency, memory footprint and FLOPs. The residual design introduces extra computation overhead, yet the accuracy-efficiency tradeoff is not quantified.
Questions for authors

Will the method become better when with longer training? What are the theory reason to justify the method better? What is the cost of the structure modification?

5 comments
AuthorsOpenReview ↗

Response to Reviewer b3d3, Weakness 1 and Weakness 2

1. Comparison with 800-Epoch REPA (1.42), U-REPA (1.41), and REG (1.40) (Weakness 1)

REPA, U-REPA, and REG accelerate convergence through external representation-alignment losses (supervising intermediate DiT features with a pretrained vision encoder), whereas our work modifies only the internal residual state read/write pathway while leaving the backbone sublayers and training objective untouched. Because these two directions are orthogonal, our interface can be plugged directly into representation-aligned models under the exact same training budget. In Table 6, we followed REPA's standard 80-epoch (400,000-step) comparison protocol and trained SiT-XL/2 + REPA with and without our method under identical settings: adding our read/write interface improves guided FID from 1.91 to 1.79 at 80 epochs (and already reaches 1.88 at 60 epochs, surpassing the 80-epoch REPA baseline in 25% fewer steps). We observe a similar improvement on representation-aligned PixelDiT-XL/16 at 80 epochs (improving guided FID from 2.36 to 2.08). During limited time in the rebuttal phase, we are not possible to perform a 800 epochs or 1400 epochs run to match REPA [1], U-REPA [2], and REG [3], and we would like to supplement it in the next revision.

2. Scaling to Longer Training Schedules (Weakness 2 & Question 1)

To verify that our gains do not saturate early and continue to improve generation quality under long training schedules, we examine both a newly extended SiT-XL/2 + REPA + Ours training run and our runs from Table 6 and Table 7:

a. New extended training on SiT-XL/2 + REPA + Ours from 80 to 120 epochs (600,000 steps): During the rebuttal period, we resumed our 80-epoch SiT-XL/2 + REPA + Ours checkpoint (1.79 FID, vs. 1.91 for the 80-epoch REPA baseline in Table 6) and continued training to 120 epochs (600,000 steps). As shown below, generation quality continues to improve smoothly across all metrics without early saturation, reaching 1.64 guided FID at 120 epochs:

SiT-XL/2 + REPA + Ours Training Trajectory20 ep (100K)40 ep (200K)60 ep (300K)80 ep (400K, Table 6)100 ep (500K, New)120 ep (600K, New)
Guided FID (CFG=1.8) ↓\downarrow5.312.461.881.791.691.64
Spatial sFID ↓\downarrow5.454.534.494.494.384.37
Inception Score (IS) ↑\uparrow171.9252.8271.5274.0280.1283.6

b. Full 600-epoch training on JiT-B/16 (750,600 steps) and 160-epoch training on PixelDiT-XL/16 (800,000 steps): In Table 6 and Table 7, we trained both the baseline and our method on JiT-B/16 for the full 600-epoch schedule (750,600 steps). Evaluating checkpoints every 20 epochs from epoch 60 to epoch 600 (28 checkpoints in total, Table 7) shows that our method outperforms the baseline at every single checkpoint without crossover, improving 600-epoch unguided FID from 25.46 to 22.86, guided FID from 3.69 to 3.55, and Inception Score from 255.99 to 272.35 (+16.4). Likewise, on PixelDiT-XL/16 trained to 160 epochs (800,000 steps, Table 6), our method continues to improve both unguided FID (14.25 to 14.14, IS 89.40 to 90.99) and guided FID (1.99 to 1.96, Recall 0.61 to 0.62).

3. Theoretical Analysis of Residual State Reading and Writing (Weakness 3 & Question 2)

We will add two formal propositions to the revised manuscript that provide a clear theoretical foundation for our reader and writer designs and directly explain the ablation results in Tables 3–5:

  1. Reader analysis — Decoupling differential feature routing from common-mode gate sparsity: For K=2K=2 normalized streams z1,z2z_1, z_2 and learned sigmoid gates g1,g2∈(0,1)g_1, g_2 \in (0,1), we can express the states and gates in symmetric common-mode and differential coordinates: z+=z1+z22,z−=z1−z22,g+=g1+g22,g−=g1−g22.z_+ = \frac{z_1 + z_2}{2}, \quad z_- = \frac{z_1 - z_2}{2}, \qquad g_+ = \frac{g_1 + g_2}{2}, \quad g_- = \frac{g_1 - g_2}{2}. In these coordinates, the uncalibrated Gated Residual readout rg=12(g1⊙z1+g2⊙z2)r_g = \frac{1}{2}(g_1 \odot z_1 + g_2 \odot z_2) decomposes exactly as: rg=g+⊙z++g−⊙z−.r_g = g_+ \odot z_+ + g_- \odot z_-. This decomposition shows that uncalibrated gating entangles differential feature routing (g−⊙z−g_- \odot z_-) with common-mode gate sparsity (g+⊙z+g_+ \odot z_+). Whenever the active gate density varies across depth or tokens, even constant 2×2\times rescaling (r2=2rgr_2 = 2r_g) suffers layer-to-layer norm drift (in our trained Table 3 checkpoints, sparse gating at the final output layer causes the readout norm of r2r_2 to drop to 16% of the reference scale). By defining the calibrated readout as rdir=r2ρϵ(z+)ρϵ(r2)r_{\text{dir}} = r_2 \frac{\rho_\epsilon(z_+)}{\rho_\epsilon(r_2)} (Equation 5, where r0=z+r_0 = z_+), our Direction Reader becomes invariant to common-mode gate scale fluctuations while preserving both the routed direction of r2r_2 and the natural inter-stream agreement scale ρϵ(z+)≈1+cos⁡(z1,z2)2\rho_\epsilon(z_+) \approx \sqrt{\frac{1+\cos(z_1,z_2)}{2}}.

  2. Writer analysis — Spherical angular velocity, norm inertia, and rank-2 updates from a single sublayer evaluation:

    Because subsequent blocks read each residual state hih_i through RMS normalization (h^i=hi/RMS⁡(hi)\hat{h}_i = h_i / \operatorname{RMS}(h_i)), the directional information passed to later layers evolves on the unit hypersphere SD−1\mathbb{S}^{D-1}, while the radial norm ∥hi∥\|h_i\| acts as inverse step-size inertia for all downstream layers.

    Decomposing a sublayer update ui=aiuu_i = a_i u into its radial component ui∥u_i^\parallel (parallel to hih_i) and tangential component ui⊥=Phi⊥uiu_i^\perp = P_{h_i}^\perp u_i (orthogonal to hih_i), a general modulated write u~i=(1+δr)ui∥+(1+δt)ui⊥\tilde{u}_i = (1+\delta_r)u_i^\parallel + (1+\delta_t)u_i^\perp produces a turning angle θi\theta_i on SD−1\mathbb{S}^{D-1} and post-write norm ∥hi′∥\|h_i'\| given by: tan⁡θi(h^i,h^i′)=(1+δt) ∥ui⊥∥/∥hi∥1+(1+δr) ⟨h^i,ui⟩/∥hi∥,∥hi′∥=∥hi∥(1+(1+δr)⟨h^i,ui⟩∥hi∥+O ⁣(∥ui⊥∥2∥hi∥2)).\tan\theta_i(\hat{h}_i, \hat{h}_i') = \frac{(1+\delta_t)\,\|u_i^\perp\|/\|h_i\|}{1 + (1+\delta_r)\,\langle \hat{h}_i, u_i\rangle/\|h_i\|}, \qquad \|h_i'\| = \|h_i\|\left(1 + (1+\delta_r)\frac{\langle \hat{h}_i, u_i\rangle}{\|h_i\|} + O\!\left(\frac{\|u_i^\perp\|^2}{\|h_i\|^2}\right)\right). This relationship explains two key empirical findings in our paper:

    • Why tangential writing (δr=0\delta_r = 0, Table 4: 4.72 FID) outperforms radial writing (4.95 FID) and whole-update modulation (5.02 FID): because the Jacobian of RMSNorm is proportional to the orthogonal projection Phi⊥P_{h_i}^\perp, radial perturbations along hih_i vanish to first order after normalization and only alter the state norm ∥hi′∥\|h_i'\| (disturbing downstream step-size inertia), whereas tangential modulation (δr=0\delta_r = 0) directly steers the angular update rate θi\theta_i on the hypersphere without perturbing first-order norm inertia.
    • How a single sublayer evaluation produces rank-2 updates across streams: in standard GR, a single shared sublayer output u∈RDu \in \mathbb{R}^D writes WGR=[a1u,a2u]W_{\text{GR}} = [a_1 u, a_2 u], which has rank at most 1 and can only push both states along the same direction uu. Because our tangent writer subtracts each state's own radial projection Phi∥uP_{h_i}^\parallel u, the cross-stream difference contains (Ph2∥−Ph1∥)u(P_{h_2}^\parallel - P_{h_1}^\parallel)u, lifting the two-stream write span to rank 2 whenever h1∦h2h_1 \nparallel h_2 with zero extra sublayer evaluations and zero extra matrix multiplications.

4. Evaluation Metrics Beyond FID, Computational Cost (FLOPs, Latency, Memory), and Training Stability (Weakness 4 & Question 3)

We respectfully clarify that evaluation metrics beyond FID, inference latency, memory footprint, and the accuracy–efficiency trade-off are already reported in the main text (Table 6, Section 5.1) and Appendices B–E (Tables 7–10, Figure 3, Algorithms 1–3) of our submitted manuscript. We will make the main-text pointers to these appendix sections more prominent in the revision, and we summarize the existing manuscript results together with additional FLOPs, stability, and modern-backbone evaluations below:

a. Generation metrics beyond FID (already in Table 6 across all 8 settings, plus sFID and new DINOv2 / CLIP evaluations): To evaluate visual and semantic quality beyond legacy Inception-v3 features, we re-evaluated our saved checkpoints from the main-table backbones and controlled ablations using DINOv2-ViT-L/14 (FD-DINOv2, KID-DINOv2, DINOv2 Precision/Recall [4]) and CLIP-ViT-L/14@336 (CMMD [5]) under identical per-sample initial noise:

Main-Table Backbones (Table 6: Baseline -> + Ours)gFID ↓\downarrow (w/o CFG | w/ CFG)IS ↑\uparrow (w/o CFG | w/ CFG)Precision / Recall ↑\uparrow (Table 6, w/ CFG)FD-DINOv2 [4] ↓\downarrow (w/ CFG)CMMD (CLIP) [5] ↓\downarrow (w/ CFG)KID-DINOv2 (×103\times 10^3) ↓\downarrowDINOv2 Prec. / Rec. ↑\uparrow
JiT-B/16 (200ep, CFG=3.0)31.69 -> 28.69 | 4.61 -> 4.3251.4 -> 55.9 | 233.9 -> 249.60.810 / 0.463 -> 0.821 / 0.464631.1 -> 589.7 (-41.4)62.79 -> 55.48 (-11.6%)202.5 -> 156.2 (-22.8%)0.850 / 0.770 -> 0.859 / 0.817
JiT-B/16 (600ep, CFG=2.7)25.46 -> 22.86 | 3.69 -> 3.5564.1 -> 69.7 | 256.0 -> 272.40.803 / 0.516 -> 0.815 / 0.516592.7 -> 565.0 (-27.7)57.53 -> 49.93 (-13.2%)158.0 -> 129.3 (-18.2%)0.864 / 0.812 -> 0.868 / 0.845
JiT-H/16 (200ep, CFG=2.2)9.20 -> 10.07 | 2.53 -> 2.46123.9 -> 121.3 | 291.1 -> 286.40.810 / 0.580 -> 0.802 / 0.590508.6 -> 514.434.73 -> 34.22 (-1.5%) [CFG=1.0: 57.27 -> 55.73]53.4 -> 57.00.883 / 0.929 -> 0.886 / 0.926
SiT-XL/2 (80ep, CFG=1.5)18.46 -> 13.70 | 5.70 -> 3.88 (sFID 4.81 -> 4.73)72.2 -> 88.2 | 162.3 -> 193.60.799 / 0.532 -> 0.822 / 0.530725.9 -> 626.2 (-99.7)41.02 -> 33.96 (-17.2%)286.8 -> 168.1 (-41.4%)0.761 / 0.656 -> 0.828 / 0.768
SiT-XL/2 + REPA (80ep -> 120ep, CFG=1.8)7.74 -> 7.70 | 1.91 -> 1.79 (120ep: 1.64; sFID 4.53 -> 4.49 -> 4.37)126.1 -> 124.7 | 276.8 -> 274.0 (120ep: 283.6)0.819 / 0.595 -> 0.806 / 0.601521.6 -> 520.7 (-0.8)25.19 -> 25.09 (-0.4%)71.7 -> 69.8 (-2.6%)0.886 / 0.898 -> 0.872 / 0.898
Controlled Ablation Progressions (JiT-B/16, 200 epochs)Inception FID ↓\downarrowFD-DINOv2 [4] ↓\downarrowCMMD (CLIP) [5] ↓\downarrowKID-DINOv2 (×103\times 10^3) ↓\downarrowDINOv2 Precision / Recall ↑\uparrow
Tables 3–5 (b=256, CFG=2.3): GR + Read x2 →\to + Direction Reader →\to + Tangent Writer →\to Full Ours5.01 →\to 4.98 →\to 4.72 →\to 4.65691.9 →\to 678.2 →\to 671.7 →\to 672.6 (-19.3)66.87 →\to 65.83 →\to 64.83 →\to 64.66 (-3.3%)253.5 →\to 239.0 →\to 240.4 →\to 240.7 (-5.0%)0.814/0.710 →\to 0.802/0.712 →\to 0.820/0.724 →\to 0.805/0.728
Table 6 recipe (b=128, CFG=3.0): Base →\to K=1 Gated →\to GR →\to Read x2 →\to Full Ours4.61 →\to 4.66 →\to 4.45 →\to 4.39 →\to 4.32631.1 →\to 603.9 →\to 595.3 →\to 603.0 →\to 589.7 (-41.4)62.79 →\to 60.05 →\to 56.97 →\to 58.18 →\to 55.48 (-11.6%)202.5 →\to 175.5 →\to 162.7 →\to 173.3 →\to 156.2 (-22.8%)0.850/0.770 →\to 0.851/0.800 →\to 0.869/0.803 →\to 0.853/0.801 →\to 0.859/0.817
Reviewer_iX9bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how Diffusion Transformers should read from and write to multiple residual states. It starts from Gated Residual (GR), which keeps K residual states per token. GR feeds each attention/MLP sublayer a gated mixture of the RMS-normalized states, and writes the sublayer output back to each state with input-dependent scalar weights.

Based on controlled ablations on JiT-B/16 (ImageNet 256×256, 200 epochs), the authors propose two changes:

  • a direction-calibrated reader, which rescales the gated mixture to the RMS of the ungated mean;
  • a noise-conditioned tangent writer, which modulates the part of each update that is orthogonal to the destination state, depending on the timestep.

In the controlled setting, FID improves from 7.15 (baseline) to 5.48 (GR) and 4.65 (full method). When added to JiT, PixelDiT, SiT, LightningDiT and SiT+REPA, the method improves guided FID in all reported settings.

Critical strengths
  1. Consistent gains across backbones. Guided FID improves in all eight settings of Table 6, covering pixel- and latent-space models and REPA (e.g., SiT-XL/2: 5.70 → 3.88; LightningDiT-XL/1: 2.11 → 1.88). Table 7 also shows a lower FID than the baseline at every checkpoint from 60 to 600 epochs. Even if much of the gain comes from GR, showing that this kind of residual design transfers to diffusion models is useful. Since the change is simple and works across architectures, this is the main reason for my positive rating.

  2. A systematic and transparent study. The read-only / write-only / shared-write controls and the comparisons of read scale, read direction and update components make the design choices easy to follow. The paper also reports the training and inference overhead (Tables 8–9) and the learned writer coefficients (Table 10). This makes the paper a useful reference on residual-stream design for diffusion models.

Critical weaknesses
  1. The two proposed components add relatively little on top of GR, and the evidence for them is limited.

    In the controlled study, most of the improvement comes from GR itself (7.15 → 5.48), which the paper adopts rather than proposes. Read×2 alone already gives 5.01, so the direction-calibrated reader (4.98) adds little beyond fixing the scale. The tangent writer improves FID by about 0.3 (5.01 → 4.72).

    These differences are reported without seeds or error bars. Table 6 also compares only the baseline with the full method, so it is unclear how much of the cross-backbone gain is due to GR alone.

    In addition, the learned tangent coefficients in Table 10 mostly sit at the ±0.25 bound: 18 of 24 sublayers do so at all three noise levels. The 0.21 gain over a static controller (4.93 → 4.72, Table 4) is therefore hard to interpret without seeds, and the noise conditioning is not very convincing.

    Since these two components are the paper's main contribution, this is my main concern.

  2. The efficiency claim and the ablation setting overstate the gains.

    Fig. 1 (bottleneck 256) reports a 1.67× training-step speedup, but each step takes about 40% longer and uses about 33% more memory (Table 8). If this overhead carries over, the 1.67× becomes about 1.2× in wall-clock time. In Table 7 (bottleneck 128), the baseline at 280 epochs (about the compute of 200 epochs of the method) is slightly better (5.41 vs. 5.52). The method pulls ahead only with longer training (4.17 at 400 epochs vs. 4.30 for the baseline at 560).

    The ablations also use a non-standard patch-embedding bottleneck of 256. With the standard bottleneck of 128 and the same CFG (Table 7), the 200-epoch gain shrinks from 7.15 → 4.65 to 6.29 → 5.52.

    Within a backbone, the gains shrink with longer training or a larger model:

    • JiT-B: from 0.29 (200 epochs) to 0.14 (600 epochs);
    • PixelDiT: from 0.28 (80 epochs) to 0.03 (160 epochs);
    • JiT-H gains only 0.07 (2.53 → 2.46) and is worse without CFG (9.20 → 10.07).

    This matters for whether the extra cost is worth paying. I still lean towards acceptance because the gains are consistent, but I would like to see these points addressed.

Questions for authors

Questions that could change my assessment

  1. Could you report results over multiple seeds for the key ablations, in particular GR vs. Read×2 vs. the direction-calibrated read, and runs with vs. without the tangent writer?
  2. Could you add GR (and GR with Read×2) to Table 6, at least for JiT-B and SiT-XL/2?
  3. What is the actual wall-clock training time of the main runs, and how does the method compare with a baseline trained for the same time?
  4. Do the main conclusions hold with the standard bottleneck of 128? Which CFG values are used for the Table 6 rows where they are not stated?
  5. Read-only GR keeps the two states identical (Sec. 4.1). It is therefore effectively a single-state model with a gated read, yet it obtains about 70% of GR's gain. How does a single-state model with the direction-calibrated read and the tangent writer compare with the full method (4.65)? Does the "single-state interface" in Table 5 include a gated read?

Minor clarifications and suggestions

  • Section 5.1 sets k = D/32, but the parameter increases in Table 6 seem to match k = 24. For example, SiT-XL/2 adds 6.8M parameters, whereas k = 36 would add about 9.9M. Which value is used?
  • Please describe the REPA setup: the encoder, the alignment depth, the loss weight, and which state is aligned.
  • A comparison with Hyper-Connections (Zhu et al., ICLR 2025), which was already evaluated on a DiT, would be helpful, as would some generated samples.
4 comments

Question 1: Could you report results over multiple seeds for the key ablations, in particular GR vs. Read x2 vs. the direction-calibrated read, and runs with vs. without the tangent writer?

Because our GPU budget during the rebuttal period was dedicated to training the new 200-epoch ImageNet controls on Table 6 backbones requested in Q2, Q4, and Q5, running multiple 200-epoch random seeds for all Table 3–5 ablations was computationally infeasible within the rebuttal window. To verify the reliability of each ablation step without multi-seed retraining, we performed two complementary evaluations on the saved checkpoints:

  1. Trajectory consistency across training checkpoints: In Figure 1b, Full Ours (4.65 at 200ep) consistently outperforms both GR (5.48) and GR + Read x2 (5.01) across every logged checkpoint from 140 to 200 epochs without a single crossover (and on standard bottleneck=128 in Table 7, outperforms the baseline across all 28 checkpoints from 60 to 600 epochs).
  2. Re-evaluation under identical initial noise with detail-sensitive modern representations (FD-DINOv2 [1] and CLIP CMMD [2]): As shown below, restoring Read x2's collapsed final-layer readout norm (16% →\to 89%) via Direction Calibration alone reduces FD-DINOv2 by 13.7 points (691.9 →\to 678.2) and CMMD from 66.87 to 65.83, and adding our Tangent Writer further improves Inception FID to 4.65, FD-DINOv2 to 672.6 (-19.3), CMMD to 64.66, and DINOv2 Recall from 0.710 to 0.728:
Controlled Ablation Checkpoints (Tables 3–5, JiT-B 200ep)Final-Layer Readout Norm (vs. reference scale)Inception FID ↓\downarrowFD-DINOv2 [1] ↓\downarrowCMMD (CLIP) [2] ↓\downarrowDINOv2 Recall ↑\uparrow
GR + Read x2 (s=2s=2, constant 2×2\times rescaling)16% (collapsed)5.01691.966.870.710
+ Direction Calibration (without Tangent Writer)89% (restored)4.98678.2 (-13.7)65.830.712
+ Tangent Writer (Full Ours: with Direction Calibration)89% (restored)4.65672.6 (-19.3)64.660.728

Question 2: Could you add GR (and GR with Read x2) to Table 6, at least for JiT-B and SiT-XL/2?

Yes. We newly trained both GR (s=1s=1) and GR + Read x2 (s=2s=2) on Table 6's standard JiT-B/16 (bottleneck=128, 200 epochs) and SiT-XL/2 configurations. As shown in the table below, static 2×2\times rescaling (GR + Read x2) actually performs worse than plain GR on SiT-XL/2 (25.36 vs. 24.12 FID) and on JiT-B/16 across unguided FID, CFG=2.3 FID, FD-DINOv2 [1], and CLIP CMMD [2], whereas our method (Direction Calibration + Tangent Writer) outperforms both GR and GR + Read x2 across every backbone, guidance setting, and metric:

Newly Trained Table 6 ControlsBaseline (K=1K=1)+ GR (s=1s=1)+ GR + Read x2 (s=2s=2)+ Ours (Direction + Tangent)
SiT-XL/2 (200K steps, CFG=1.0 ODE FID ↓\downarrow)28.3824.1225.36 (worse than GR)21.61 (-2.51 vs. GR)
JiT-B/16 (bottleneck=128, 200ep, unguided CFG=1.0 FID ↓\downarrow)31.6928.9529.97 (worse than GR)28.69
JiT-B/16 (bottleneck=128, 200ep, guided CFG=2.3 FID ↓\downarrow, Table 7)6.295.585.68 (worse than GR)5.52
JiT-B/16 (bottleneck=128, 200ep, guided CFG=3.0 FID ↓\downarrow, Table 6)4.614.454.394.32
JiT-B/16 (bottleneck=128, 200ep, CFG=3.0 FD-DINOv2 [1] ↓\downarrow)631.1595.3603.0 (worse than GR)589.7 (-13.4 vs. Read x2)
JiT-B/16 (bottleneck=128, 200ep, CFG=3.0 CMMD [2] ↓\downarrow)62.7956.9758.18 (worse than GR)55.48 (-4.6% vs. Read x2)

Question 3: What is the actual wall-clock training time of the main runs, and how does the method compare with a baseline trained for the same time?

In the actual 8-GPU DDP training logs (B=128B=128 per GPU, global batch 1024) of our 600-epoch JiT-B/16 (bottleneck=128) runs in Table 7, the single-stream baseline runs at 157.1 ms/step (taking 32.8 hours / 262 GPU-hours for 600 epochs) and our method with our fused CuTe/Triton training kernel runs at 193.7 ms/step (+23.3% step time). Comparing generation quality at matched wall-clock time (where the baseline trains for 1.233×1.233\times as many epochs within the same time budget), our method outperforms the baseline at every training-time budget from 13.5 hours (200 epochs) through 32.5 hours:

Matched 8×GPU Training Time (Total GPU-Hours)Ours (JiT-B/16, Table 7) (Epochs completed in this time →\to FID)Baseline (JiT-B/16, Table 7) (Trains 1.233×1.233\times more epochs in the same time →\to FID)Matched-Time Outcome
13.5 hours (108 GPU-h)200 epochs →\to 5.52 FID~247 epochs →\to ~5.78 FID (logged 240ep: 5.85, 260ep: 5.65)Ours wins by -0.26 FID
20.2 hours (162 GPU-h)300 epochs →\to 4.54 FID~370 epochs →\to ~4.73 FID (logged 360ep: 4.76, 380ep: 4.70)Ours wins by -0.19 FID
24.2 hours vs. 32.8 hours360 epochs →\to 4.28 FID (24.2h)600 epochs →\to 4.29 FID (32.8h)Matches 600ep Baseline in 1.35×1.35\times less time (saves 26% GPU-hours)
~32.5 hours (260 GPU-h)480 epochs →\to 3.94 FID (32.3h)600 epochs →\to 4.29 FID (32.8h)Ours wins by -0.35 FID

Question 4: Do the main conclusions hold with the standard bottleneck of 128? Which CFG values are used for the Table 6 rows where they are not stated?

  1. Do the main conclusions hold with the standard bottleneck of 128? Yes. Probing and evaluating our newly trained bottleneck=128 JiT-B/16 models (Baseline, GR, GR + Read x2, Single-State K=1K=1 Gated Control, and K=2K=2 Ours) confirms that all conclusions hold unchanged with bottleneck=128:

    • Uncalibrated Read x2 suffers the exact same final-layer readout collapse on bottleneck=128 (collapsing to 15% of the reference norm at the final output layer, vs. 16% on bottleneck=256), whereas Direction Calibration restores unit-scale readout (100%).
    • Guided FID at CFG=3.0 improves monotonically from Baseline (4.61) →\to GR (4.45) →\to GR + Read x2 (4.39) →\to Ours (4.32), with our Reader and Writer contributing 45% of the total FID gain over Baseline and outperforming GR + Read x2 by -13.4 points on FD-DINOv2 and -4.6% on CLIP CMMD (table under Question 2).
    • Scaling the number of persistent streams KK on bottleneck=128 (Table 8) also improves monotonically from K=1K=1 (4.61) →\to K=2K=2 (4.32) →\to K=4K=4 (4.13).
  2. Exact CFG values, guidance intervals, and samplers for all rows of Table 6 is as below. Please note that these follows the baseline protocol. For JiT, we sweep the best CFG for baseline model then apply for both ours and theirs as they done in JiT paper. For PixelDiT, SiT, LightniingDiT, we use their original configuration.

Table 6 ModelGuided CFG ScaleGuidance IntervalSampler & Steps
JiT-B/16 (200 epochs)3.0σ∈[0,0.88]\sigma \in [0, 0.88]Heun, 50 steps
JiT-B/16 (600 epochs)2.7σ∈[0,0.88]\sigma \in [0, 0.88]Heun, 50 steps
JiT-H/16 (200 epochs)2.2t∈[0.1,1.0]t \in [0.1, 1.0]Heun, 50 steps
PixelDiT-XL/16 (80 & 160 epochs)3.25t∈[0.1,1.0]t \in [0.1, 1.0]FlowDPMSolver, 100 steps
SiT-XL/2 (80 epochs)1.5t∈[0,1.0]t \in [0, 1.0]SDE-Euler, 250 steps
LightningDiT-XL/1 (64 epochs)10.0t∈[0.11,1.0]t \in [0.11, 1.0]Euler, 250 steps
SiT-XL/2 + REPA (80 epochs)1.8t∈[0,0.7]t \in [0, 0.7]SDE-Euler, 250 steps

Question 5: How does a single-state model with the direction-calibrated read and the tangent writer compare with the full method? Does the "single-state interface" in Table 5 include a gated read?

  1. Does "Single-state interface" in Table 5 include a gated read? No. In Table 5, "Single-state interface" applies only our noise-conditioned tangent writer (h′=h+u+δℓ(t)Ph⊥uh' = h + u + \delta_\ell(t)P_h^\perp u) to a standard single-stream (K=1K=1) backbone with no gated read (improving FID from 7.15 to 6.57). Note that in Table 1, Read-only GR (5.85 FID) still maintains K=2K=2 streams with separate learned per-stream RMSNorm gain parameters and a K=2K=2 reader gate that mixes those two distinct normalized views.

  2. New training run of a single-state (K=1K=1) model equipped with the full gated read, direction calibration, and tangent writer: To test your question directly, we trained a new single-state (K=1K=1) control equipped with the full k=24k=24 bottleneck read gate, direction calibration, and noise-conditioned tangent writer (+0.96M parameters) on standard JiT-B/16 (bottleneck=128, 200 epochs, Table 6 recipe). As shown below, adding the entire gated reader, direction calibration, and tangent writer to a single state (K=1K=1) yields 4.66 FID at CFG=3.0—providing no gain over the single-stream baseline (4.61 FID)—whereas our two-state (K=2K=2) model reaches 4.32 FID (and 4.13 FID at K=4K=4, Table 8). This confirms that the improvement fundamentally comes from adaptive reading and writing across multiple persistent residual states (K≥2K \ge 2), rather than from extra gating parameters or nonlinearity:

Stream Count Control (JiT-B/16, bottleneck=128, 200ep)Persistent States KKGated Reader + Direction Calibration + Tangent WriterGuided FID (CFG=3.0, Table 6) ↓\downarrowUnguided FID (CFG=1.0) ↓\downarrowFD-DINOv2 [1] ↓\downarrowCMMD (CLIP) [2] ↓\downarrow
BaselineK=1K=1No4.6131.69631.162.79
Single-State Full Gated Control (New for Q5)K=1K=1Yes (+0.96M params)4.6630.95603.960.05
Ours (Two-State)K=2K=2Yes (+1.95M params)4.3228.69589.755.48

Minor 1: Bottleneck rank k=D/32k = D/32 in Section 5.1 vs. parameter increases in Table 6

Thank you for catching this discrepancy. Section 5.1 inaccurately retained an early draft statement that k=D/32k = D/32; We never used k=36k=36 or k=40k=40, and instead used only two fixed discrete ranks across all experiments:

  • k=24k = 24 for JiT-B/16 (+1.95M params), SiT-XL/2 (+6.76M params), and LightningDiT-XL/1 (+6.69M params).
  • k=64k = 64 for PixelDiT-XL/16 (+16.04M params), JiT-H/16, and SiT-XL/2 + REPA.

Minor 2: Please describe the REPA setup: the encoder, the alignment depth, the loss weight, and which state is aligned

Following the official REPA configuration on SiT-XL/2, we use a frozen DINOv2 ViT-B/14 encoder and align intermediate representations at block 8 (out of 28) with an alignment loss weight of 0.5. To align the two residual states without introducing a second projection head, we take the unweighted arithmetic mean of the two unnormalized residual states hˉ(8)=12(h1(8)+h2(8))\bar{h}^{(8)} = \frac{1}{2}(h_1^{(8)} + h_2^{(8)}) and feed it into a single official 3-layer MLP projection head.

Minor 3: Comparison with Hyper-Connections (Zhu et al., ICLR 2025) and generated samples

Thank you for this suggestion. Because Zhu et al. [3] did not release DiT checkpoints or DiT training code (and trained their DiT-XL/2 model for 1,400 epochs), our compute budget during the rebuttal window was insufficient to re-implement and train Hyper-Connections from scratch. We will expand our discussion comparing static/dynamic Hyper-Connections with our direction-calibrated reader and noise-conditioned tangent writer in Section 2, and include additional side-by-side generated samples under matched random seeds across backbones in next revision.

References

[1] Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models, NeurIPS 2023

[2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024

[3] Hyper-Connections, ICLR 2025

Reviewer_7FsyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates adaptive reading and writing over multiple persistent residual states in Diffusion Transformers. Building on Gated Residual (GR), it introduces a reader that preserves the adaptive mixture’s direction while calibrating its magnitude, and a writer that applies noise-conditioned modulation to the update component orthogonal to each destination state. Experiments cover several pixel- and latent-space diffusion backbones, including a configuration trained with representation alignment.

Critical strengths

The paper provides useful ablations examining readout scaling and different write mechanisms, including whole-update, radial, and tangent modulation. Evaluation across several diffusion backbones broadens the empirical study. The explicit reporting of runtime and memory overhead is also appreciated.

Critical weaknesses
  1. The improvements are modest in several important settings. Table 6 reports guided FID improvements from 1.99 to 1.96 for PixelDiT, from 2.53 to 2.46 for JiT-H, and from 3.69 to 3.55 for JiT-B at 600 epochs. Without repeated runs or uncertainty estimates, it is difficult to assess the reliability of these small differences. Moreover, JiT-H’s unguided FID worsens from 9.20 to 10.07. The benefit of the proposed reader over simple rescaling is particularly limited: direction calibration improves FID from 5.01 to only 4.98 in Table 3. Although the SiT-XL/2 improvement is substantially larger, the results do not establish a consistently meaningful advantage across architectures and training budgets.

  2. The additional computational cost is substantial relative to these gains. In the reported optimized two-state benchmark, training-step time increases from 42.42 to 59.40 ms, approximately 40%, while peak memory increases from 7.65 to 10.15 GiB, approximately 33%. Even with fused inference, throughput falls to 87.37% of the original backbone’s throughput. These are significant overheads for an architectural modification whose quality improvements are often small. Equal-epoch comparisons do not establish whether the method offers a favorable computational trade-off. The reported reduction in training steps also does not directly establish a wall-clock advantage, particularly because the convergence and runtime experiments use different configurations. The authors should compare generation quality at matched measured training time or GPU-hours, allowing the original backbone to train longer within the same budget.

Questions for authors

Addressed in weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7Fsy, Weakness 1

Thank you for your constructive review. Below we address both of your concerns.

1. Weakness 1: Significance of Gains Over Simple Rescaling and Reliability Across Training

(1). Why Direction Calibration is needed when simple 2x rescaling (GR + Read x2) reaches 5.01 vs. 4.98 FID?

In Gated Residual (GR), the reader forms a gated mixture rg=1K∑i=1Kgi⊙zir_g = \frac{1}{K}\sum_{i=1}^K g_i \odot z_i using learned sigmoid gates gi∈(0,1)g_i \in (0,1). Because sigmoid gates are initialized at 0.50.5, simple 2×2\times rescaling (GR + Read x2 in Table 3, r2=2rgr_2 = 2r_g) multiplies the gated readout by a single fixed constant of 22 at every layer, assuming that the gates always shrink the readout norm by roughly half. Although this appears close to Direction Calibration in Table 3 (5.01 vs. 4.98 FID), we inspected the layer-by-layer readout norms of the trained checkpoints and ran new training experiments adding GR and GR + Read x2 to SiT-XL/2 and to our main-table JiT-B/16 (Table 6) setup as below:

a. Why fixed 2×2\times rescaling fails across layers in Table 3 (and why Table 3b introduced Direction Calibration): In Section 4.2 (Table 3b), we compared keeping only Read x2's gate-induced norm (Magnitude: 5.26 FID) against keeping its gated direction while calibrating its norm to the ungated reference r0r_0 (Direction Calibration, rdir=r2ρϵ(r0)ρϵ(r2)r_{\text{dir}} = r_2 \frac{\rho_\epsilon(r_0)}{\rho_\epsilon(r_2)}: 4.98 FID). Inspecting the layer-by-layer readout norms in the Table 3 checkpoints reveals why a single fixed multiplier s=2s=2 is inadequate: learned gate sparsity varies substantially across depth. While multiplying by 22 roughly restores average scale in middle layers, the learned gates become much sparser at the final output readout layer, causing GR + Read x2's signal norm entering the final prediction head to collapse to just 16%16\% of the reference norm—whereas Direction Calibration dynamically normalizes the gated readout at every layer (89%89\% at the final readout) with zero extra parameters. Because attenuating the final output head primarily softens fine-grained details that legacy Inception-v3 features under-weight, fixing this collapse moves Inception FID from 5.01 to 4.98, but produces a much clearer gain on detail-sensitive modern vision backbones (FD-DINOv2 [1] and CLIP CMMD [2]) under identical initial noise, while providing a scale-stabilized reader for our Tangent Writer (Full Ours, Table 5):

Controlled Progression (Table 3 -> Table 5, JiT-B 200ep)Final-Layer Readout Norm (vs. reference scale)Inception FID ↓\downarrowFD-DINOv2 [1] ↓\downarrowCMMD (CLIP) [2] ↓\downarrowDINOv2 Recall ↑\uparrow
GR + Read x2 (constant 2×2\times rescaling)16% (collapsed)5.01691.966.870.710
+ Direction Calibration (our Reader)89% (restored)4.98678.2 (-13.7)65.830.712
+ Tangent Writer (Full Ours: Reader + Writer)89% (restored)4.65672.6 (-19.3)64.660.728

b. Across architectures, GR + Read x2 is unstable and often performs worse than plain GR (SiT-XL/2 & Table 6 JiT-B/16): You noted that our improvement on SiT-XL/2 is much larger than on pixel-space models. To test whether simple 2×2\times rescaling (GR + Read x2) could also capture this gain, we trained new GR and GR + Read x2 baselines on SiT-XL/2 and on the main-table JiT-B/16 (Table 6, bottleneck=128) configuration. Inspecting the trained SiT-XL/2 checkpoints shows that under a fixed 2×2\times multiplier, the readout norm fluctuates widely across depth (from 27%27\% to 108%108\% of the reference norm across middle layers, and collapsing to 24%24\% at the final output layer). As shown below, without per-layer Direction Calibration and Tangent Writing, GR + Read x2 actually performs worse than plain GR on SiT-XL/2 (25.36 vs. 24.12 FID) and on unguided / CFG=2.3 JiT-B/16, whereas our method improves consistently across every backbone and guidance scale:

Newly Trained Backbone ControlsBaseline (K=1K=1)+ GR (s=1s=1)+ GR + Read x2 (s=2s=2)+ Ours (Direction + Tangent)
SiT-XL/2 (200K steps, CFG=1.0 ODE FID ↓\downarrow)28.3824.1225.36 (worse than GR)21.61 (-2.51 vs. GR)
JiT-B/16 (Table 6, 200ep, unguided CFG=1.0 FID ↓\downarrow)31.6928.9529.97 (worse than GR)28.69
JiT-B/16 (Table 7, 200ep, guided CFG=2.3 FID ↓\downarrow)6.295.585.68 (worse than GR)5.52
JiT-B/16 (Table 6, 200ep, guided CFG=3.0 FID ↓\downarrow)4.614.454.394.32
AuthorsOpenReview ↗

Response to Reviewer 7Fsy, Weakness 1 (continued)

(2). Why the converged guided FID gains (3.69 -> 3.55, 2.53 -> 2.46, 1.99 -> 1.96) are reliable? Retraining there models across multiple (at least for 5) seeds is infeasible within the rebuttal window. However, we verified that our converged gains in Table 6 are genuine rather than single-run evaluation noise in two ways:

  • In Table 7, we logged FID every 20 epochs from epoch 60 to epoch 600 (28 evaluation checkpoints in total) for JiT-B/16. If the final 3.69 -> 3.55 difference were random checkpoint fluctuation, the two curves would cross back and forth; instead, our method outperforms the baseline at all 28 checkpoints consecutively without a single crossover.
  • Inception-v3 FID compresses differences nonlinearly. We re-evaluating our converged Table 6 checkpoints (JiT-B/16 at 600 epochs and SiT-XL/2 at 80 epochs / 400K steps) with DINOv2 [1] and CLIP [2] under identical initial noise, to confirm that the perceptual improvement is substantial across both pixel and latent backbones:
Converged Table 6 Models (Baseline -> + Ours)Inception FID ↓\downarrowInception Score (IS) ↑\uparrowFD-DINOv2 [1] ↓\downarrowCMMD (CLIP) [2] ↓\downarrowDINOv2 Recall ↑\uparrow
JiT-B/16 (600 epochs, Table 6)3.69 -> 3.55256.0 -> 272.4 (+16.4)592.7 -> 565.0 (-27.7)57.53 -> 49.93 (-13.2%)0.812 -> 0.845
SiT-XL/2 (80 epochs / 400K steps, Table 6)5.70 -> 3.88215.5 -> 244.4 (+28.9)725.9 -> 626.2 (-99.7)41.02 -> 33.96 (-17.2%)0.656 -> 0.768

(3) Why JiT-H/16's unguided FID increases (9.20 -> 10.07) while guided FID improves (2.53 -> 2.46)

We also investigated why JiT-H/16 improved on guided FID, yet showed a higher unguided FID in Table 6. Our multi-state writer already encourages the model to cover a broader set of image modes (reflected in higher Recall). When sampling without guidance (CFG=1.0), an 200ep early-stage 1B pixel model has not yet tightly concentrated its mode peaks, and Inception FID heavily penalizes this wider spread, even though evaluating those exact same generated samples in CLIP feature space (CMMD [2]) shows that our model actually outperforms the baseline under both unguided (CFG=1.0) and guided (CFG=2.2) sampling. Once standard classifier-free guidance (CFG=2.2) concentrates the distribution around class modes, the penalty from early-stage spread disappears and our model achieves both lower guided FID and higher Recall:

JiT-H/16 (200 epochs, Table 6: Baseline -> + Ours)Inception FID ↓\downarrow (Table 6)Recall ↑\uparrow (Table 6)CMMD (CLIP) [2] ↓\downarrow
Unguided (CFG=1.0)9.20 -> 10.070.649 -> 0.65157.27 -> 55.73 (-2.7%)
Guided (CFG=2.2)2.53 -> 2.460.580 -> 0.59034.73 -> 34.22 (-1.5%)
AuthorsOpenReview ↗

Response to Reviewer 7Fsy, Weakness 2

2. Computational Overhead, Custom Fused Kernels, and Matched Training-Time Comparison (Weakness 2)

(1) What our actual training logs show for step-time overhead, and why we built custom fused kernels

  • Actual training-log overhead (+23.3%) vs. Table 8's single-GPU micro-benchmark (+40%): In the paper, Table 8 reported a +40% step-time increase (42.42 -> 59.40 ms). To perform the matched wall-clock comparison you requested under a consistent configuration, we inspected the actual 8-GPU training logs from our 600-epoch JiT-B/16 (bottleneck=128) runs in Table 7. In those actual training logs, the step time increases by only +23.3% (157.1 ms/step for the baseline vs. 193.7 ms/step for our method, with +18%–28% peak memory), rather than +40%. Why was the actual training overhead in Table 7 nearly half of Table 8's figure? Because Table 8 was a standalone single-GPU benchmark that used a small batch size of 32 per GPU solely so the 8-stream (K=8K=8) ablation (18.97 GiB) could fit on one card without running out of memory (Appendix C.1). In our actual training runs (Appendix A), each GPU runs the standard production batch size of 128 per GPU (1024 global batch across 8 GPUs), where GPU compute is much better utilized and kernel-launch overhead is amortized.
  • Why modern residual-routing architectures require custom fused kernels (Sec. 5.1, Appendix D): We completely agree that runtime overhead is a central concern for architectural work. Mathematically, our method still evaluates Attention and MLP only once per block at width DD, adding just 1.4%–3.6% FLOPs across all models in Table 6 (and zero extra matrix multiplications over plain GR). However, as in other recent multi-stream residual architectures (such as Hyper-Connections [3], mHC [4], and Attention Residuals [5]; Sec. 2), single-stream baselines run atop heavily fused vendor primitives, whereas reading and writing multiple residual streams in eager PyTorch is bottlenecked by GPU memory bandwidth (HBM). Turning low FLOPs into practical speedups therefore requires co-designing the algorithm with custom fused kernels. That is why we implemented custom CuTe/Triton fused forward and backward kernels for both training and inference (Sec. 5.1 & Appendix D, Fig. 3, Algs. 1–3, which we will open-source). As shown in Table 9, our custom kernels speed up inference by +33.9%—retaining 87.4% of single-stream throughput.

(2) Comparing generation quality at matched training time (GPU-hours) on the exact same configuration

You rightly pointed out that a reduction in training steps does not establish a wall-clock advantage if runtime and convergence are measured on different configurations. Using the Table 7 JiT-B/16 (bottleneck=128) training logs—where both models share the exact same backbone configuration and 8-GPU training setup—we can directly compare generation quality at matched wall-clock time (allowing the baseline to train for more epochs within the same time budget):

  • First, even if we conservatively penalized our method by Table 8's +40% (1.40x) micro-benchmark figure, our model reaches the baseline's 600-epoch quality (4.29 FID) in just 360 epochs (4.28 FID)—a 1.67x reduction in training steps that still finishes faster in wall-clock time.
  • Second, using the actual measured step time from the Table 7 training logs (+23.3%, meaning the baseline runs 1.233x as many epochs in the same wall-clock time), our method outperforms the baseline at every single training-time budget:
Matched 8×GPU Training Time (Total GPU-Hours)Ours (JiT-B/16, Table 7) (Epochs completed in this time →\to FID)Baseline (JiT-B/16, Table 7) (Trains 1.233×1.233\times more epochs in the same time →\to FID)Matched-Time Outcome
13.5 hours (108 GPU-h)200 epochs →\to 5.52 FIDapprox. 247 epochs →\to 5.78 FID (logged 240ep: 5.85, 260ep: 5.65)Ours wins by -0.26 FID
20.2 hours (162 GPU-h)300 epochs →\to 4.54 FIDapprox. 370 epochs →\to 4.73 FID (logged 360ep: 4.76, 380ep: 4.70)Ours wins by -0.19 FID
24.2 hours vs. 32.8 hours360 epochs →\to 4.28 FID (24.2h)600 epochs →\to 4.29 FID (32.8h)Matches 600ep Baseline in 1.35× less time (saves 26% GPU-hours)
~32.5 hours (260 GPU-h)480 epochs →\to 3.94 FID (32.3h)600 epochs →\to 4.29 FID (32.8h)Ours wins by -0.35 FID

References

[1] Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models, NeurIPS 2023

[2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation, CVPR 2024

[3] Hyper-Connections, ICLR 2025

[4] mHC: Manifold-Constrained Hyper-Connections, arXiv 2026

[5] Attention Residuals, arXiv 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.