- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates an orthogonal question, how should a denoiser read from and write to multiple persistent residual states? Introducing a direction-calibrated reader that preserves the direction selected by adaptive gates and matches the mixture’s magnitude to an ungated average of normalized states. For writing, presenting a noise-conditioned tangent writer that controls this component according to the noise level.
Critical strengths
The method modifies internal residual connections, normalization positions and branch activations. The modification is localized and can be plugged into existing DiT-like architectures. Based on controlled comparisons of readout scale and update geometry, they develop a direction-calibrated reader and a noise-conditioned tangent writer, and demonstrate improved generation across pixel- and latent-space diffusion backbones, extending to models trained with representation alignment.
Critical weaknesses
- Despite designing a comparatively intricate structure, the work cannot attain the best experimental results, such as, compared with SiT-XL/2 + U-REPA (1.41), + REG (1.40), + REPA (1.42).
- The SOTAs have longer training epochs >200, while the method only has 80 epochs, lack of longer training results.
- The architecture and hyperparameter choices in this work are largely empirically driven, lacking theoretical analysis and plausible justifications.
- Evaluations heavily focus on FID, with insufficient reporting on training stability statistics, inference latency, memory footprint and FLOPs. The residual design introduces extra computation overhead, yet the accuracy-efficiency tradeoff is not quantified.
Questions for authors
Will the method become better when with longer training? What are the theory reason to justify the method better? What is the cost of the structure modification?
5 comments
3. Theoretical Analysis of Residual State Reading and Writing (Weakness 3 & Question 2)
We will add two formal propositions to the revised manuscript that provide a clear theoretical foundation for our reader and writer designs and directly explain the ablation results in Tables 3–5:
-
Reader analysis — Decoupling differential feature routing from common-mode gate sparsity: For normalized streams and learned sigmoid gates , we can express the states and gates in symmetric common-mode and differential coordinates: In these coordinates, the uncalibrated Gated Residual readout decomposes exactly as: This decomposition shows that uncalibrated gating entangles differential feature routing () with common-mode gate sparsity (). Whenever the active gate density varies across depth or tokens, even constant rescaling () suffers layer-to-layer norm drift (in our trained Table 3 checkpoints, sparse gating at the final output layer causes the readout norm of to drop to 16% of the reference scale). By defining the calibrated readout as (Equation 5, where ), our Direction Reader becomes invariant to common-mode gate scale fluctuations while preserving both the routed direction of and the natural inter-stream agreement scale .
-
Writer analysis — Spherical angular velocity, norm inertia, and rank-2 updates from a single sublayer evaluation:
Because subsequent blocks read each residual state through RMS normalization (), the directional information passed to later layers evolves on the unit hypersphere , while the radial norm acts as inverse step-size inertia for all downstream layers.
Decomposing a sublayer update into its radial component (parallel to ) and tangential component (orthogonal to ), a general modulated write produces a turning angle on and post-write norm given by: This relationship explains two key empirical findings in our paper:
- Why tangential writing (, Table 4: 4.72 FID) outperforms radial writing (4.95 FID) and whole-update modulation (5.02 FID): because the Jacobian of RMSNorm is proportional to the orthogonal projection , radial perturbations along vanish to first order after normalization and only alter the state norm (disturbing downstream step-size inertia), whereas tangential modulation () directly steers the angular update rate on the hypersphere without perturbing first-order norm inertia.
- How a single sublayer evaluation produces rank-2 updates across streams: in standard GR, a single shared sublayer output writes , which has rank at most 1 and can only push both states along the same direction . Because our tangent writer subtracts each state's own radial projection , the cross-stream difference contains , lifting the two-stream write span to rank 2 whenever with zero extra sublayer evaluations and zero extra matrix multiplications.
4. Evaluation Metrics Beyond FID, Computational Cost (FLOPs, Latency, Memory), and Training Stability (Weakness 4 & Question 3)
We respectfully clarify that evaluation metrics beyond FID, inference latency, memory footprint, and the accuracy–efficiency trade-off are already reported in the main text (Table 6, Section 5.1) and Appendices B–E (Tables 7–10, Figure 3, Algorithms 1–3) of our submitted manuscript. We will make the main-text pointers to these appendix sections more prominent in the revision, and we summarize the existing manuscript results together with additional FLOPs, stability, and modern-backbone evaluations below:
a. Generation metrics beyond FID (already in Table 6 across all 8 settings, plus sFID and new DINOv2 / CLIP evaluations): To evaluate visual and semantic quality beyond legacy Inception-v3 features, we re-evaluated our saved checkpoints from the main-table backbones and controlled ablations using DINOv2-ViT-L/14 (FD-DINOv2, KID-DINOv2, DINOv2 Precision/Recall [4]) and CLIP-ViT-L/14@336 (CMMD [5]) under identical per-sample initial noise:
| Main-Table Backbones (Table 6: Baseline -> + Ours) | gFID (w/o CFG | w/ CFG) | IS (w/o CFG | w/ CFG) | Precision / Recall (Table 6, w/ CFG) | FD-DINOv2 [4] (w/ CFG) | CMMD (CLIP) [5] (w/ CFG) | KID-DINOv2 () | DINOv2 Prec. / Rec. |
|---|---|---|---|---|---|---|---|
| JiT-B/16 (200ep, CFG=3.0) | 31.69 -> 28.69 | 4.61 -> 4.32 | 51.4 -> 55.9 | 233.9 -> 249.6 | 0.810 / 0.463 -> 0.821 / 0.464 | 631.1 -> 589.7 (-41.4) | 62.79 -> 55.48 (-11.6%) | 202.5 -> 156.2 (-22.8%) | 0.850 / 0.770 -> 0.859 / 0.817 |
| JiT-B/16 (600ep, CFG=2.7) | 25.46 -> 22.86 | 3.69 -> 3.55 | 64.1 -> 69.7 | 256.0 -> 272.4 | 0.803 / 0.516 -> 0.815 / 0.516 | 592.7 -> 565.0 (-27.7) | 57.53 -> 49.93 (-13.2%) | 158.0 -> 129.3 (-18.2%) | 0.864 / 0.812 -> 0.868 / 0.845 |
| JiT-H/16 (200ep, CFG=2.2) | 9.20 -> 10.07 | 2.53 -> 2.46 | 123.9 -> 121.3 | 291.1 -> 286.4 | 0.810 / 0.580 -> 0.802 / 0.590 | 508.6 -> 514.4 | 34.73 -> 34.22 (-1.5%) [CFG=1.0: 57.27 -> 55.73] | 53.4 -> 57.0 | 0.883 / 0.929 -> 0.886 / 0.926 |
| SiT-XL/2 (80ep, CFG=1.5) | 18.46 -> 13.70 | 5.70 -> 3.88 (sFID 4.81 -> 4.73) | 72.2 -> 88.2 | 162.3 -> 193.6 | 0.799 / 0.532 -> 0.822 / 0.530 | 725.9 -> 626.2 (-99.7) | 41.02 -> 33.96 (-17.2%) | 286.8 -> 168.1 (-41.4%) | 0.761 / 0.656 -> 0.828 / 0.768 |
| SiT-XL/2 + REPA (80ep -> 120ep, CFG=1.8) | 7.74 -> 7.70 | 1.91 -> 1.79 (120ep: 1.64; sFID 4.53 -> 4.49 -> 4.37) | 126.1 -> 124.7 | 276.8 -> 274.0 (120ep: 283.6) | 0.819 / 0.595 -> 0.806 / 0.601 | 521.6 -> 520.7 (-0.8) | 25.19 -> 25.09 (-0.4%) | 71.7 -> 69.8 (-2.6%) | 0.886 / 0.898 -> 0.872 / 0.898 |
| Controlled Ablation Progressions (JiT-B/16, 200 epochs) | Inception FID | FD-DINOv2 [4] | CMMD (CLIP) [5] | KID-DINOv2 () | DINOv2 Precision / Recall |
|---|---|---|---|---|---|
| Tables 3–5 (b=256, CFG=2.3): GR + Read x2 + Direction Reader + Tangent Writer Full Ours | 5.01 4.98 4.72 4.65 | 691.9 678.2 671.7 672.6 (-19.3) | 66.87 65.83 64.83 64.66 (-3.3%) | 253.5 239.0 240.4 240.7 (-5.0%) | 0.814/0.710 0.802/0.712 0.820/0.724 0.805/0.728 |
| Table 6 recipe (b=128, CFG=3.0): Base K=1 Gated GR Read x2 Full Ours | 4.61 4.66 4.45 4.39 4.32 | 631.1 603.9 595.3 603.0 589.7 (-41.4) | 62.79 60.05 56.97 58.18 55.48 (-11.6%) | 202.5 175.5 162.7 173.3 156.2 (-22.8%) | 0.850/0.770 0.851/0.800 0.869/0.803 0.853/0.801 0.859/0.817 |
Response to Reviewer b3d3, Weakness 1 and Weakness 2
1. Comparison with 800-Epoch REPA (1.42), U-REPA (1.41), and REG (1.40) (Weakness 1)
REPA, U-REPA, and REG accelerate convergence through external representation-alignment losses (supervising intermediate DiT features with a pretrained vision encoder), whereas our work modifies only the internal residual state read/write pathway while leaving the backbone sublayers and training objective untouched. Because these two directions are orthogonal, our interface can be plugged directly into representation-aligned models under the exact same training budget. In Table 6, we followed REPA's standard 80-epoch (400,000-step) comparison protocol and trained SiT-XL/2 + REPA with and without our method under identical settings: adding our read/write interface improves guided FID from 1.91 to 1.79 at 80 epochs (and already reaches 1.88 at 60 epochs, surpassing the 80-epoch REPA baseline in 25% fewer steps). We observe a similar improvement on representation-aligned PixelDiT-XL/16 at 80 epochs (improving guided FID from 2.36 to 2.08). During limited time in the rebuttal phase, we are not possible to perform a 800 epochs or 1400 epochs run to match REPA [1], U-REPA [2], and REG [3], and we would like to supplement it in the next revision.
2. Scaling to Longer Training Schedules (Weakness 2 & Question 1)
To verify that our gains do not saturate early and continue to improve generation quality under long training schedules, we examine both a newly extended SiT-XL/2 + REPA + Ours training run and our runs from Table 6 and Table 7:
a. New extended training on SiT-XL/2 + REPA + Ours from 80 to 120 epochs (600,000 steps): During the rebuttal period, we resumed our 80-epoch SiT-XL/2 + REPA + Ours checkpoint (1.79 FID, vs. 1.91 for the 80-epoch REPA baseline in Table 6) and continued training to 120 epochs (600,000 steps). As shown below, generation quality continues to improve smoothly across all metrics without early saturation, reaching 1.64 guided FID at 120 epochs:
b. Full 600-epoch training on JiT-B/16 (750,600 steps) and 160-epoch training on PixelDiT-XL/16 (800,000 steps): In Table 6 and Table 7, we trained both the baseline and our method on JiT-B/16 for the full 600-epoch schedule (750,600 steps). Evaluating checkpoints every 20 epochs from epoch 60 to epoch 600 (28 checkpoints in total, Table 7) shows that our method outperforms the baseline at every single checkpoint without crossover, improving 600-epoch unguided FID from 25.46 to 22.86, guided FID from 3.69 to 3.55, and Inception Score from 255.99 to 272.35 (+16.4). Likewise, on PixelDiT-XL/16 trained to 160 epochs (800,000 steps, Table 6), our method continues to improve both unguided FID (14.25 to 14.14, IS 89.40 to 90.99) and guided FID (1.99 to 1.96, Recall 0.61 to 0.62).