- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why mathematically interconvertible prediction targets lead to substantially different learning outcomes in large-patch diffusion transformers. It proposes residual-stream burden: noisy targets require persistent hidden states to retain noise-dependent information for the final readout, potentially interfering with representation learning. Spectral analysis, whitening, representation probes, and architectural interventions examine this account. The paper further introduces Spatially Indexed Hyper-Connections (SiHC), which separate expanded local persistent states from a shared computational workspace, achieving FID 1.71 on ImageNet 256×256 with REPA.
Critical strengths
- The research question is important and clearly motivated. Distinguishing persistent-state bandwidth from computational width provides a useful perspective on the interaction between prediction targets, representation learning, and diffusion architectures.
- The paper develops a coherent empirical argument through complementary measurements and interventions. In particular, four-stream mHC improves pixel-space velocity-prediction FID from 139.83 to 25.36 while only slightly improving clean prediction, and a learned noisy-input skip reaches 9.99. The affine analysis and null-class representation probes further strengthen the evidence.
Critical weaknesses
- Whitening does not isolate spectral concentration from other changes to the learning problem. Appendix B.9 applies standard flow matching in whitened coordinates; after inversion, isotropic noise in that space corresponds to anisotropic pixel-space noise, and the squared-error objective induces a different pixel-space metric. The FID deterioration from 10.19 to 132.95 is striking, but cannot by itself separate spectral concentration from noise geometry, loss weighting, and optimization conditioning.
- The bandwidth experiments do not fully disentangle state capacity, spatial organization, and routing. Table 4 usefully compares copied and spatially indexed inputs at the same four-stream bandwidth. However, mHC also changes cross-depth connections, and increasing SiHC from four to sixteen streams simultaneously increases bandwidth and reduces subpatch size. The remaining contribution of each factor is therefore unclear.
Questions for authors
- Could the authors add partial-whitening and noise/loss controls to distinguish the effects of spectral concentration from changed corruption geometry and optimization conditioning? Orthogonal-transform or global-rescaling controls would also help identify which changes drive the degradation.
- Could the authors vary persistent bandwidth while keeping subpatch size and routing fixed, and compare spatial organizations at a fixed total state dimension? How do these results compare with a wider plain residual backbone at comparable actual compute?
- What are the training throughput, peak memory, and full-sampling latency of fused SiHC versus plain DiT/JiT on the same hardware and at matched batch sizes? Please distinguish the cost of state expansion from the savings obtained by kernel fusion.
4 comments
Rebuttal to Reviewer bAXJ 2/n
Regarding “How do these results compare with a wider plain residual backbone at comparable actual compute?”, making the network wider inevitably increases compute unless we make it shallower or introduce sparsity, which brings in other factors. Given this complication, we provide an informal comparison below, also included in our submitted appendix, as context for how plain width scaling compares.
| Model | Depth × width | Parameters | GFLOPs / forward | FID ↓ | IS ↑ |
|---|---|---|---|---|---|
| DiT-only, reported by DiP | 629M | 5.28 | 243.8 | ||
| DiT-only, reported by DiP | 772M | 4.91 | 251.7 | ||
| DiT-only, reported by DiP | 776M | 4.28 | 249.6 | ||
| DiT-only, reported by DiP | 1.1B | 2.83 | 285.6 | ||
| SiHC-B, no REPA | 131M | 50.9 | 6.17 | 196.3 | |
| SiHC-L, no REPA | 460M | 177.0 | 3.02 | 268.8 | |
| SiHC-H, no REPA | 954M | 368.2 | 2.51 | 274.1 |
Starred FLOPs values are analytical estimates from the published configurations. DiP’s reported results use Euler-100, while our SiHC results use Heun-50.
Regarding hardware efficiency
We provide the requested measurements below. Comparing Torch and fused SiHC at the same stream count shows the effect of fusion, while comparing fused SiHC with plain JiT shows the remaining runtime and memory overhead.
| Model | Streams | Training latency (ms/step) ↓ | Training throughput (images/s) ↑ | Peak training memory (GiB) ↓ | Heun-50 latency (s/128 images) ↓ |
|---|---|---|---|---|---|
| Plain JiT | 1 | 66.4 | 1928.5 | 15.3 | 3.84 |
| Dynamic mHC (TE kernels) | 4 | 113.7 | 1126.0 | 21.7 | 5.39 |
| SiHC, Torch | 4 | 167.8 | 763.0 | 20.4 | 10.19 |
| SiHC, fused | 4 | 111.8 | 1145.0 | 17.9 | 4.43 |
| SiHC, Torch | 16 | 861.2 | 148.6 | 35.4 | 55.48 |
| SiHC, fused | 16 | 118.9 | 1076.4 | 19.3 | 4.85 |
Single H200, batch 128, FP32 parameters with BF16 autocast, no REPA/context tokens/recomputation. Values are medians of three independent process-level means. Training includes backward, optimizer, and EMA updates. Sampling includes all 99 Heun solver evaluations, each with sequential conditional/unconditional forwards. The mHC row uses NVIDIA fused kernels with graph breaks around the Transformer Engine mixing operations.
Though efficiency is not included in the main claim of our submission, we hope this can resolve your concerns.
Thank you for the additional experiments, but my main concerns remain:
-
Whitening also changes the effective noise level and loss weight of each direction, which could degrade any denoiser. The sweep therefore does not show that the degradation comes specifically from residual-stream burden.
-
SiHC takes 1.7–1.8× the training time per step of plain JiT, and its gain has not been shown at matched compute.
I therefore maintain my rating.
We cannot see the logic behind this assessment.
Your follow-up leaves us confused. Which of our central claims do your two objections actually undermine? You maintain rejection without explaining which conclusion is unsupported. We cannot see the logic behind this assessment.
-
Which specific conclusion in our submission does the change in SNR or loss weighting contradict? Why does a change in directional SNR or loss weighting exclude residual-stream burden as an explanation of how the learned representations change? These quantities help determine what the prediction task asks the network to preserve. Your “therefore” assumes a conflict between these explanations that you have not established. Which specific conclusion in our paper does this challenge?
-
Why is a matched-runtime performance advantage necessary for this paper’s contribution? Where do we claim such an advantage? We keep attention/MLP computation fixed to examine the effect of persistent-state bandwidth. Which conclusion does the additional runtime invalidate?
Please explain why the remaining concerns justify rejection given the revised paper and the evidence provided.
Rebuttal to Reviewer bAXJ 1/n
Thank you for recognizing our contribution and proposing insightful experiments.
As partial whitening flattens the spectrum, filtering concentration, class readability, and generation quality decline together.
As suggested in your question, we conducted partial-whitening experiments to test the role of spectral concentration in the patch space. Let p be a clean RGB patch, μ its mean, B an orthogonal matrix whose rows are clean-patch PCA directions, and D=diag(λi). The three new runs use
\qquad \alpha\in\{0.25,0.50,0.75\},\qquad \mathbf B\mathbf B^\top=\mathbf I.$$ All runs apply the same orthogonal rotation and vary the strength of directional rescaling. Their clean-patch covariance eigenvalues are $\lambda_i^{1-\alpha}$. We train clean prediction in each transformed space using the same training and evaluation recipes as the B-size experiments in Section 4.1. Generated patches are mapped back by $\mathbf p=\mathbf B^\top\mathbf D^\alpha\mathbf z_\alpha+\mu$ before FID evaluation. | Whitening strength $\alpha$ | Clean-patch $r_{90}$ | FID ↓ | Probe top-1 (%) ↑ | Top-64 embedding gain (%) | | --------------------------- | -------------------: | -----: | ----------------: | ------------------------: | | Raw RGB baseline | 8 | 10.19 | 27.50 | 61.48 | | 0.25 | 109 | 12.28 | 23.78 | 34.89 | | 0.50 | 350 | 26.23 | 18.88 | 20.71 | | 0.75 | 587 | 64.22 | 12.94 | 11.57 | | 1, fully whitened endpoint | 692 | 132.65 | 5.87 | 10.36 | Regarding noise and loss conditioning, we do not think these changes constitute an alternative explanation that undermines our interpretation, since they may be precisely how spectral concentration affects training. Our experiments connect patch spectral concentration to learned filtering and representation quality. Understanding the gradient dynamics behind this connection would deepen that explanation, though characterizing these dynamics in deep Transformers remains difficult, as discussed in Section 6. We are still unclear why an explanation involving loss weighting or directional SNR would conflict with our interpretation, since these effects could fit within it. Could you clarify which specific claim in our submission such an explanation would challenge, and why separating these effects is necessary to support that claim? >Could the authors vary persistent bandwidth while keeping subpatch size and routing fixed, and compare spatial organizations at a fixed total state dimension? Following your suggestion, we added two static iHC runs with the same static feature-wise read/write maps and identity carry as SiHC, but with every stream assigned the full $16\times16$ patch. Only the number of streams varies between the two runs. **We can see when enlarging the persistent state without changing the subpatch size, the performance still keeps improving, which supports our claims.** | B-size velocity model | Streams | State assignment | FID ↓ | | --------------------- | ------: | ----------------------------- | -----: | | Plain JiT | 1 | Full $16\times16$ patch | 139.83 | | Static iHC | 4 | Copied $16\times16$ patch | 27.81 | | SiHC | 4 | Spatial $8\times8$ subpatches | 10.10 | | Static iHC | 16 | Copied $16\times16$ patch | 22.73 | | SiHC | 16 | Spatial $4\times4$ subpatches | 8.07 | Here copied streams share the full-patch input, while SiHC assigns local inputs and outputs to spatial states. All models use 200 epochs without REPA (the same recipe as in the paper). Comparing the two static iHC rows tests bandwidth expansion, while comparing static iHC with SiHC at each stream count tests spatial assignment. Attention/MLP width remains fixed throughout. We will add these two additional experiments in our new draft and hope them can resolve your concern.