acceptodds
Under review as a conference paper at ICLR 2027

Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers

10 views

Abstract

In diffusion-based generation, a neural network can be trained to predict the clean data, the noise, or the velocity from a noisy input. These prediction targets are interconvertible and describe the same generative process, yet plain Diffusion Transformers operating on large pixel patches succeed with clean prediction and fail with noise or velocity prediction. We argue that this asymmetry arises because noisy targets require the residual stream to preserve noise-dependent input variation through depth for the final readout, forcing subsequent layers to compute on noisy representations. A spectrally concentrated clean target imposes a lighter demand, leaving greater freedom to organize hidden representations for subsequent computation. We call this preservation requirement *residual-stream burden* and show how it shapes representation learning in Diffusion Transformers. Controlled experiments indicate that the exploitable structure is spectral concentration in patch space and that the bandwidth of the persistent residual state is a key resource for noisy prediction. We further show that this account is consistent with recent decoupled pixel-space architectures, whose diverse designs all reduce the residual-stream burden on the main pathway. To examine this understanding from a complementary direction, we expand and reorganize the residual-stream bandwidth directly, introducing Spatially Indexed Hyper-Connections (SiHC) that reach FID 1.71 on ImageNet . Together, these results identify residual-stream burden as a mechanism through which prediction targets and architecture jointly shape representation learning in Diffusion Transformers.

Primary area:
generative models
Keywords:
diffusion models, transformers, inductive bias, generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bAXJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why mathematically interconvertible prediction targets lead to substantially different learning outcomes in large-patch diffusion transformers. It proposes residual-stream burden: noisy targets require persistent hidden states to retain noise-dependent information for the final readout, potentially interfering with representation learning. Spectral analysis, whitening, representation probes, and architectural interventions examine this account. The paper further introduces Spatially Indexed Hyper-Connections (SiHC), which separate expanded local persistent states from a shared computational workspace, achieving FID 1.71 on ImageNet 256×256 with REPA.

Critical strengths
  1. The research question is important and clearly motivated. Distinguishing persistent-state bandwidth from computational width provides a useful perspective on the interaction between prediction targets, representation learning, and diffusion architectures.
  2. The paper develops a coherent empirical argument through complementary measurements and interventions. In particular, four-stream mHC improves pixel-space velocity-prediction FID from 139.83 to 25.36 while only slightly improving clean prediction, and a learned noisy-input skip reaches 9.99. The affine analysis and null-class representation probes further strengthen the evidence.
Critical weaknesses
  1. Whitening does not isolate spectral concentration from other changes to the learning problem. Appendix B.9 applies standard flow matching in whitened coordinates; after inversion, isotropic noise in that space corresponds to anisotropic pixel-space noise, and the squared-error objective induces a different pixel-space metric. The FID deterioration from 10.19 to 132.95 is striking, but cannot by itself separate spectral concentration from noise geometry, loss weighting, and optimization conditioning.
  2. The bandwidth experiments do not fully disentangle state capacity, spatial organization, and routing. Table 4 usefully compares copied and spatially indexed inputs at the same four-stream bandwidth. However, mHC also changes cross-depth connections, and increasing SiHC from four to sixteen streams simultaneously increases bandwidth and reduces subpatch size. The remaining contribution of each factor is therefore unclear.
Questions for authors
  1. Could the authors add partial-whitening and noise/loss controls to distinguish the effects of spectral concentration from changed corruption geometry and optimization conditioning? Orthogonal-transform or global-rescaling controls would also help identify which changes drive the degradation.
  2. Could the authors vary persistent bandwidth while keeping subpatch size and routing fixed, and compare spatial organizations at a fixed total state dimension? How do these results compare with a wider plain residual backbone at comparable actual compute?
  3. What are the training throughput, peak memory, and full-sampling latency of fused SiHC versus plain DiT/JiT on the same hardware and at matched batch sizes? Please distinguish the cost of state expansion from the savings obtained by kernel fusion.
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer bAXJ 1/n

Thank you for recognizing our contribution and proposing insightful experiments.

Could the authors add partial-whitening and noise/loss controls to distinguish the effects of spectral concentration from changed corruption geometry and optimization conditioning? Orthogonal-transform or global-rescaling controls would also help identify which changes drive the degradation.

As partial whitening flattens the spectrum, filtering concentration, class readability, and generation quality decline together.

As suggested in your question, we conducted partial-whitening experiments to test the role of spectral concentration in the patch space. Let p\mathbf p be a clean RGB patch, μ\mu its mean, B\mathbf B an orthogonal matrix whose rows are clean-patch PCA directions, and D=diag⁡(λi)\mathbf D=\operatorname{diag}(\sqrt{\lambda_i}). The three new runs use

\qquad \alpha\in\{0.25,0.50,0.75\},\qquad \mathbf B\mathbf B^\top=\mathbf I.$$ All runs apply the same orthogonal rotation and vary the strength of directional rescaling. Their clean-patch covariance eigenvalues are $\lambda_i^{1-\alpha}$. We train clean prediction in each transformed space using the same training and evaluation recipes as the B-size experiments in Section 4.1. Generated patches are mapped back by $\mathbf p=\mathbf B^\top\mathbf D^\alpha\mathbf z_\alpha+\mu$ before FID evaluation. | Whitening strength $\alpha$ | Clean-patch $r_{90}$ | FID ↓ | Probe top-1 (%) ↑ | Top-64 embedding gain (%) | | --------------------------- | -------------------: | -----: | ----------------: | ------------------------: | | Raw RGB baseline | 8 | 10.19 | 27.50 | 61.48 | | 0.25 | 109 | 12.28 | 23.78 | 34.89 | | 0.50 | 350 | 26.23 | 18.88 | 20.71 | | 0.75 | 587 | 64.22 | 12.94 | 11.57 | | 1, fully whitened endpoint | 692 | 132.65 | 5.87 | 10.36 | Regarding noise and loss conditioning, we do not think these changes constitute an alternative explanation that undermines our interpretation, since they may be precisely how spectral concentration affects training. Our experiments connect patch spectral concentration to learned filtering and representation quality. Understanding the gradient dynamics behind this connection would deepen that explanation, though characterizing these dynamics in deep Transformers remains difficult, as discussed in Section 6. We are still unclear why an explanation involving loss weighting or directional SNR would conflict with our interpretation, since these effects could fit within it. Could you clarify which specific claim in our submission such an explanation would challenge, and why separating these effects is necessary to support that claim? >Could the authors vary persistent bandwidth while keeping subpatch size and routing fixed, and compare spatial organizations at a fixed total state dimension? Following your suggestion, we added two static iHC runs with the same static feature-wise read/write maps and identity carry as SiHC, but with every stream assigned the full $16\times16$ patch. Only the number of streams varies between the two runs. **We can see when enlarging the persistent state without changing the subpatch size, the performance still keeps improving, which supports our claims.** | B-size velocity model | Streams | State assignment | FID ↓ | | --------------------- | ------: | ----------------------------- | -----: | | Plain JiT | 1 | Full $16\times16$ patch | 139.83 | | Static iHC | 4 | Copied $16\times16$ patch | 27.81 | | SiHC | 4 | Spatial $8\times8$ subpatches | 10.10 | | Static iHC | 16 | Copied $16\times16$ patch | 22.73 | | SiHC | 16 | Spatial $4\times4$ subpatches | 8.07 | Here copied streams share the full-patch input, while SiHC assigns local inputs and outputs to spatial states. All models use 200 epochs without REPA (the same recipe as in the paper). Comparing the two static iHC rows tests bandwidth expansion, while comparing static iHC with SiHC at each stream count tests spatial assignment. Attention/MLP width remains fixed throughout. We will add these two additional experiments in our new draft and hope them can resolve your concern.
AuthorsOpenReview ↗

Rebuttal to Reviewer bAXJ 2/n

Regarding “How do these results compare with a wider plain residual backbone at comparable actual compute?”, making the network wider inevitably increases compute unless we make it shallower or introduce sparsity, which brings in other factors. Given this complication, we provide an informal comparison below, also included in our submitted appendix, as context for how plain width scaling compares.

ModelDepth × widthParametersGFLOPs / forwardFID ↓IS ↑
DiT-only, reported by DiP26×115226\times1152629M221.2∗221.2^*5.28243.8
DiT-only, reported by DiP32×115232\times1152772M272.0∗272.0^*4.91251.7
DiT-only, reported by DiP26×128026\times1280776M272.0∗272.0^*4.28249.6
DiT-only, reported by DiP26×153626\times15361.1B389.3∗389.3^*2.83285.6
SiHC-B, no REPA12×76812\times768131M50.96.17196.3
SiHC-L, no REPA24×102424\times1024460M177.03.02268.8
SiHC-H, no REPA32×128032\times1280954M368.22.51274.1

Starred FLOPs values are analytical estimates from the published configurations. DiP’s reported results use Euler-100, while our SiHC results use Heun-50.

Regarding hardware efficiency

We provide the requested measurements below. Comparing Torch and fused SiHC at the same stream count shows the effect of fusion, while comparing fused SiHC with plain JiT shows the remaining runtime and memory overhead.

ModelStreamsTraining latency (ms/step) ↓Training throughput (images/s) ↑Peak training memory (GiB) ↓Heun-50 latency (s/128 images) ↓
Plain JiT166.41928.515.33.84
Dynamic mHC (TE kernels)4113.71126.021.75.39
SiHC, Torch4167.8763.020.410.19
SiHC, fused4111.81145.017.94.43
SiHC, Torch16861.2148.635.455.48
SiHC, fused16118.91076.419.34.85

Single H200, batch 128, FP32 parameters with BF16 autocast, no REPA/context tokens/recomputation. Values are medians of three independent process-level means. Training includes backward, optimizer, and EMA updates. Sampling includes all 99 Heun solver evaluations, each with sequential conditional/unconditional forwards. The mHC row uses NVIDIA fused kernels with graph breaks around the Transformer Engine mixing operations.

Though efficiency is not included in the main claim of our submission, we hope this can resolve your concerns.

Thank you for the additional experiments, but my main concerns remain:

  1. Whitening also changes the effective noise level and loss weight of each direction, which could degrade any denoiser. The sweep therefore does not show that the degradation comes specifically from residual-stream burden.

  2. SiHC takes 1.7–1.8× the training time per step of plain JiT, and its gain has not been shown at matched compute.

I therefore maintain my rating.

AuthorsOpenReview ↗

We cannot see the logic behind this assessment.

Your follow-up leaves us confused. Which of our central claims do your two objections actually undermine? You maintain rejection without explaining which conclusion is unsupported. We cannot see the logic behind this assessment.

  1. Which specific conclusion in our submission does the change in SNR or loss weighting contradict? Why does a change in directional SNR or loss weighting exclude residual-stream burden as an explanation of how the learned representations change? These quantities help determine what the prediction task asks the network to preserve. Your “therefore” assumes a conflict between these explanations that you have not established. Which specific conclusion in our paper does this challenge?

  2. Why is a matched-runtime performance advantage necessary for this paper’s contribution? Where do we claim such an advantage? We keep attention/MLP computation fixed to examine the effect of persistent-state bandwidth. Which conclusion does the additional runtime invalidate?

Please explain why the remaining concerns justify rejection given the revised paper and the evidence provided.

Reviewer_8sWHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why clean-image prediction succeeds where direct velocity prediction struggles in large-patch diffusion Transformers. It proposes “residual-stream burden”: prediction-relevant noise must remain accessible through depth, constraining the representations used by subsequent computation. The authors connect this account to patch covariance spectra, learned input filtering, representation quality, and decoupled architectures. They introduce Spatially Indexed Hyper-Connections (SiHC), which maintain local persistent states accessed by a shared computational workspace, and report FID 1.71 on ImageNet-256 with REPA.

Critical strengths
  1. The authors substantially improve generation quality for direct velocity prediction. In the controlled B-size experiments, SiHC reduces FID from 139.83 for the plain velocity-prediction baseline to 8.07 without REPA.

  2. The paper presents an extensive experimental investigation. The authors explore whitening, patch geometry, multiple residual streams, spatial assignment, learned skip connections, representation quality, and model scaling. This breadth provides useful observations beyond the proposed architecture itself.

Critical weaknesses
  1. My main concern is that the strongest generation result does not isolate the contribution of SiHC from REPA. The headline FID of 1.71 uses DINOv3-based representation alignment, whereas the JiT baselines in Table 5 do not use REPA. Appendix E.4 does provide useful results without alignment, including SiHC-H reaching approximately FID 2.06. However, there is no matched SiHC-XL comparison with and without REPA. Consequently, the headline comparison does not establish how much of the advantage over JiT comes from the architecture and how much comes from the additional pretrained supervision. The authors should provide a same-configuration, same-training-budget REPA ablation and a controlled comparison with JiT under comparable training conditions. Any training-efficiency comparison should also account for the teacher and alignment projector.

  2. The representation analysis should more directly assess the feature properties relevant to generation. Section 5.2 claims that SiHC develops stronger class features than plain velocity prediction. Figure 4 and Appendix H provide meaningful support through linear probes, and these controlled experiments do not use REPA. However, the probes operate on spatially pooled features and primarily establish improved class readability. They do not directly evaluate the spatial organization of the learned features, which is particularly relevant to SiHC’s design and the proposed connection between representation quality and generation.

    At minimum, I would like the authors to supplement these probes with the spatial-structure metrics examined in iREPA—Local vs. Distant Similarity (LDS), Semantic-Region Self-Similarity (SRSS), Correlation Decay Slope (CDS), and RMS Spatial Contrast (RMSC). iREPA shows why global classification accuracy alone can be an inadequate indicator of a representation’s usefulness for generative alignment. These metrics should be compared across SiHC, plain velocity prediction, and clean-image prediction at matched noise levels and comparable network depths. Measuring SiHC both with and without REPA would further distinguish architectural effects from externally supplied representation supervision.

Questions for authors

Addressed in weaknesses

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 8sWH 1/n

Thank you for your thoughtful comments. We especially appreciated your observation that “This breadth provides useful observations beyond the proposed architecture itself.” This captures what we hoped to contribute: an explanation of how prediction targets and architectural choices shape representation learning, bringing several recent pixel-space diffusion designs together through the idea of residual-stream burden. SiHC is an architectural experiment that tests this understanding and explores another design direction.

Your emphasis on representation learning gets directly at the core of our explanation, and we would like to address both concerns from that perspective.

Question 1: the contribution of REPA

My main concern is that the strongest generation result does not isolate the contribution of SiHC from REPA.

Following your suggestion, we trained SiHC-XL without REPA for 200 epochs. The new result is highlighted below, alongside the existing REPA and scaling results.

ModelREPATraining budgetFID ↓
SiHC-XL (new)No200 epochs2.61
SiHC-XLYes200 epochs2.00
SiHC-HNo200 epochs2.51
SiHC-HNoBest within 600 epochs2.06
SiHC-XLYesBest within 600 epochs1.71

Yes, part of SiHC’s best performance comes from REPA, and this fits with the difficulty our paper is trying to explain. DiP, DeCo, and PixelDiT, whose designs we analyze in the paper, also use REPA when scaling up velocity prediction, and we followed the same practice. REPA highlights the difficulty of representation learning in diffusion Transformers, while JiT shows how well clean prediction can work without representation alignment. Understanding how the prediction target contributes to this difference is central to our paper.

Our explanation is that noisy targets require the residual stream to carry noise-dependent information, making it harder for subsequent layers to learn useful representations. SiHC helps by expanding the bandwidth and giving each stream a smaller local prediction task, but the streams still have noise to carry. From this perspective, it makes sense that SiHC benefits from REPA even after these architectural changes. These results show how far the architectural change helps on its own, while our analysis in Section 6 explains why representation learning can remain difficult.

Question 2: spatial representation quality

The representation analysis should more directly assess the feature properties relevant to generation.

We agree that the internal representations deserve a closer look. Following your suggestion, we evaluated LDS, CDS, SRSS, and RMSC using the iREPA framework, measuring both the workspace and the spatially indexed residual states of SiHC.

Model / observation pointLDS ↑CDS ↑SRSS ↑RMSC ↑
Plain clean prediction0.067850.008990.084350.43258
Plain velocity prediction0.074020.007820.090480.41156
SiHC: workspace0.055660.007400.078880.36539
SiHC: spatially indexed residual states0.104120.013210.145850.53978

All models are B-size, trained for 200 epochs without REPA, and evaluated at noise fraction 1−t=0.51-t=0.5 and block boundary 11. SiHC’s residual states retain their native 64×6464\times64 layout, with local and distant distances matched in pixels.

SiHC’s workspace has weaker spatial structure than the plain models, possibly because reading the spatially indexed states into a shared workspace pools away some local detail. At the same time, those residual states retain stronger spatial features. Together with the linear-probe results already reported in Figure 4 and Table 26, this suggests a division of responsibilities: the residual states retain local information for the final prediction, while the workspace provides a class-readable summary for subsequent computation.

In a plain velocity model, the same residual stream must serve the final prediction and subsequent computation, making generation and representation learning harder. The different feature properties we observe in SiHC’s residual states and workspace make this division of responsibilities visible.

We hope this can resolve your concern.

Reviewer_8sWHOpenReview ↗

Response to author rebuttal

Dear authors, thank you for the response and the additional experiments. The comparison with and without REPA and the new spatial-feature analysis address my main concerns. I will increase my score.

Dear reviewer 8sWH, thank you for your constructive review and support again! We have already updated the experiments you suggested into the new version.

Reviewer_VCo1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors provide an in-depth analysis of the issue in flow matching where DiT fails to accurately predict noise or velocity. Through a series of theoretical analyses and visualized measurement representations, the authors attribute the asymmetry of the problem to the residual stream burden caused by noise prediction. The paper proposes Spatial-indexed Hyper Connections to expand the feature representation width of the residual, thereby alleviating the residual stream burden and restoring DiT’s velocity prediction.

Critical strengths
  1. The authors conduct an in-depth investigation of existing issues in flow matching and provide a detailed explanation for why recent work such as JiT achieves better performance by directly predicting images rather than predicting velocity. They further unify the underlying principles of directly predicting clean images and decoupled models effectively predicting velocity.
  2. The evidence analysis is thorough and the visualizations are clear. The authors analyze the variance explained during denoising, employ whitening experiments to verify the plausibility of the manifold hypothesis, and investigate the network’s selective filtering of inputs through correlation matrices. They also use linear probes to conduct readout tests, demonstrating the spectral concentration of semantic endpoints in the decoupled model and further substantiating the applicability of the paper’s hypothesis to decoupled models.
  3. An efficiency metric is provided for the proposed design. The theoretical impact of introducing SiHC on the parameter count is calculated, and the experimental data report the effectiveness of the SiHC's Kernel Fusion implementation.
Critical weaknesses
  1. The residual stream burden currently tends to be explained using variance explained, covariance, and spectral concentration, but lacks theoretical mathematical connections, and there is insufficient explanation of whether these metrics can adequately serve as proxies for the residual stream burden.
  2. The claim that the computational overhead remains essentially unchanged is currently supported only by theoretical analysis. The authors compare the runtime and memory usage before and after SiHC's Kernel Fusion, but do not investigate the costs relative to DiT or decoupled models. This is currently insufficient to substantiate that the essentially unchanged computational overhead confers a practical efficiency advantage.
Questions for authors
  1. The final results do not demonstrate an absolute performance advantage for SiHC. For example, in Table 5, compared with DeCo, SiHC has a larger parameter count and FLOPs but a slightly higher FID. Can the authors clarify the specific contribution of SiHC and whether the model’s advantage lies in computational cost, final quality, or parameter count? A purely mechanistic explanation is insufficient to establish the contribution of SiHC.
  2. Could the authors supplement experiments reporting runtime and peak memory for vanilla DiT, mHC, and SiHC under the same hardware, batch size, precision, and optimization settings, so as to support the theoretically efficient computational performance of SiHC?
  3. The SiHC experiments are affected by multiple factors, and the final results are metrics obtained after REPA alignment. Although supplementary experiments demonstrate results without REPA, the experimental ablation is unclear; for example, the SiHC-H without REPA and SiHC-XL with REPA groups cannot be directly compared quantitatively.
  4. The paper uses the covariance spectrum as a proxy for the burden borne by the model’s residual channels. Could the authors further provide mathematical analysis of whether the covariance spectrum can accurately serve as such a proxy? For example, does the diversity of output prediction directions indicate that the model requires a large hidden state to characterize? The authors have already explained the predictability of random noise variation; could they further clarify how the predictable component of variation represents the actual information burden?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer VCo1 1/n

Before answering your questions, we would like to clarify two points in your review:

employ whitening experiments to verify the plausibility of the manifold hypothesis

In contrast, the whitening experiment challenges the sufficiency of the manifold hypothesis as an explanation for the clean-prediction advantage. Whitening maps the clean-data manifold to a smoothly equivalent manifold, yet training deteriorates sharply, motivating our investigation of spectral concentration.

The final results do not demonstrate an absolute performance advantage for SiHC. […] A purely mechanistic explanation is insufficient to establish the contribution of SiHC.

As we explain in the abstract and introduction, we developed SiHC to test our understanding “from a complementary direction”. This is why much of the paper examines why JiT, PixelDiT, DeCo, and DiP work well, while the SiHC design occupies a relatively small part of the main text. SiHC contributes evidence that accommodating the burden within the residual stream, without a separate decoder, improves velocity prediction and recovers class-readable workspace features. We never claim our proposed method is SOTA from either perspective of efficiency or performance, instead, we described SiHC as a algorithmic probe to verify our understanding.

Regarding the mathematical connection

Could they further clarify how the predictable component of variation represents the actual information burden?

The key point is that even a direction with almost no clean-patch variance can carry noise-dependent variation that is predictable from the input and needed for accurate velocity prediction. We can see this directly in a single clean-PCA direction.

Let pip_i be the centered clean-patch coordinate with variance λi\lambda_i, and let ni∼N(0,1)n_i\sim\mathcal N(0,1) be independent noise. At a fixed t<1t<1, the noisy input is pt,i=tpi+σnip_{t,i}=tp_i+\sigma n_i, with σ=1−t\sigma=1-t, and the velocity target is vi=pi−niv_i=p_i-n_i. For either target yi∈{pi,vi}y_i\in\{p_i,v_i\}, the minimum-MSE affine predictor and its variance are

\operatorname{Var}(\widehat y_i)=\frac{\operatorname{Cov}(y_i,p_{t,i})^2}{\operatorname{Var}(p_{t,i})}.$$ Independence gives $\operatorname{Var}(p_{t,i})=t^2\lambda_i+\sigma^2$, $\operatorname{Cov}(v_i,p_{t,i})=t\lambda_i-\sigma$, and $\operatorname{Cov}(p_i,p_{t,i})=t\lambda_i$. Substituting these expressions yields the predicted variance along this direction for velocity and clean prediction: $$q_i^v(t)=\operatorname{Var}(\widehat v_i)=\frac{(t\lambda_i-\sigma)^2}{t^2\lambda_i+\sigma^2},\qquad q_i^p(t)=\operatorname{Var}(\widehat p_i)=\frac{t^2\lambda_i^2}{t^2\lambda_i+\sigma^2}.$$ As $\lambda_i\to0$, $q_i^v(t)\to1$ while $q_i^p(t)\to0$. The limiting case makes the intuition clear: when $\lambda_i=0$, the input is $\sigma n_i$ and the velocity target is exactly $-p_{t,i}/\sigma$, so the noise-dependent target is fully observable. A plain backbone predicting velocity needs to keep this input information accessible to its final readout, even though the direction contributes almost nothing to the clean estimate. This preservation requirement is the residual-stream burden. Subsequent layers compute on the same persistent state, linking the prediction requirement to the representations they receive. Our filtering and bandwidth experiments examine this connection. The full matrix derivation is given in Eqs. (8)–(11) of the submitted appendix. Our analysis estimates how many directions carry predictable target variation. Determining how much persistent bandwidth representation learning itself requires remains an open problem, as discussed in Section 6.
AuthorsOpenReview ↗

Rebuttal to Reviewer VCo1 2/n

Regarding the REPA comparison

All B-size experiments and representation analyses were conducted without REPA, providing controlled comparisons of the architectural changes. The new XL run adds the matched comparison you requested, with the same model and training budget.

ModelREPATraining budgetFID ↓
SiHC-XL (new)No200 epochs2.61
SiHC-XLYes200 epochs2.00
SiHC-HNo200 epochs2.51
SiHC-HNoBest within 600 epochs2.06
SiHC-XLYesBest within 600 epochs1.71

REPA contributes to SiHC’s best performance, as it does for DiP, DeCo, and PixelDiT, whose designs we analyze. SiHC expands bandwidth and reduces each stream’s local prediction burden, but the streams still carry noise, so representation learning remains difficult. Its benefit from REPA fits the difficulty our paper seeks to explain.

Regarding hardware efficiency

We provide the requested measurements below.

ModelStreamsTraining latency (ms/step) ↓Training throughput (images/s) ↑Peak training memory (GiB) ↓Heun-50 latency (s/128 images) ↓
Plain JiT166.41928.515.33.84
Dynamic mHC (TE kernels)4113.71126.021.75.39
SiHC, Torch4167.8763.020.410.19
SiHC, fused4111.81145.017.94.43
SiHC, Torch16861.2148.635.455.48
SiHC, fused16118.91076.419.34.85

Single H200, batch 128, FP32 parameters with BF16 autocast, no REPA/context tokens/recomputation. Values are medians of three independent process-level means. Training includes backward, optimizer, and EMA updates. Sampling includes all 99 Heun solver evaluations, each with sequential conditional/unconditional forwards. The mHC row uses NVIDIA fused kernels with graph breaks around the Transformer Engine mixing operations.

The extra latency mainly comes from moving the expanded residual state through memory, while Transformer computation remains nearly unchanged. Our fusion targets these repeated transfers. As explained above, we developed the fusion and its analysis along the way to make testing our architectural understanding more practical.


Our concern is that the review asks SiHC to establish a performance or efficiency advantage, while we introduced it to test the mechanism developed throughout the paper. We have now provided the matched REPA comparison and hardware measurements you requested, alongside a fuller explanation of the existing analysis. We hope these results and clarifications give you grounds to reconsider the assessment.

AuthorsOpenReview ↗

Rebuttal to reviewer VCo1 3/n (Update on the mathematical analysis and its connection to the experiments)

According to your comments, we have expanded and highlighted the mathematical analysis in both the main text and Appendix C, and strengthened its connection to the experimental observations. Two propositions now appear directly in Sections 4.2 and 4.3, with intuition before each statement and full proofs in the appendix.

Proposition 1 explains why weak clean directions still matter for velocity prediction. With xt=tx+(1−t)ϵ\mathbf{x}_t=t\mathbf{x}+(1-t)\boldsymbol{\epsilon} and v=x−ϵ\mathbf{v}=\mathbf{x}-\boldsymbol{\epsilon}, consider any unit direction u\mathbf{u} whose clean variance λ=Var⁡(u⊤x)\lambda=\operatorname{Var}(\mathbf{u}^{\top}\mathbf{x}) approaches zero. At fixed t<1t<1, the optimal squared-loss predictions satisfy

\qquad \operatorname{Var}\left(\mathbf{u}^{\top}\mathbb{E}[\mathbf{v}\mid\mathbf{x}_t,t]\right)\to1.$$ The intuition is simple: when clean variation vanishes along a direction, its noisy input reveals the corruption, which velocity prediction must reproduce. Spectral concentration makes this distinction relevant across many patch directions. This connects the existing affine analysis to our observed filtering patterns: clean prediction suppresses weak clean directions, while direct velocity prediction retains a much broader response. **Proposition 2 explains how a separate path can take over this prediction responsibility.** For a predictor $\widehat{\mathbf{v}}=\mathbf{g}_{\theta}(\mathbf{x}_t,t)+\alpha(t)\mathbf{x}_t$, the variance of the optimal main-branch prediction along the same weak direction tends to $[1+(1-t)\alpha(t)]^2$. This vanishes at $\alpha(t)=-1/(1-t)$, leaving the main branch to predict the scaled clean estimate $\mathbb{E}[\mathbf{x}\mid\mathbf{x}_t,t]/(1-t)$. Our learned-skip experiment starts from $\alpha=0$ and approaches this coefficient across most of the time grid by epoch 40. Training therefore approaches the division identified by the analysis, providing a concrete connection between the mathematical result and the experimentally observed benefit of separate prediction paths. We hope this can resolve your concern about the theoretical insight.

We kindly remind you that the discussion period is coming to a close and we would like to ask whether our responses and revised draft have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions (Version 1)

We thank the reviewers for the questions and suggestions that helped us clarify the paper and strengthen its evidence. The revised manuscript incorporates the following changes:

  1. A clearer connection between mathematical analysis and experiments (VCo1). We have organized the derivations into a dedicated appendix section with propositions and proofs, and brought two propositions into Sections 4.2 and 4.3. These explain why weak clean directions still carry predictable variation under velocity prediction and how a separate skip can take over that variation. The accompanying experiments connect these results to learned filtering and the skip coefficient discovered during training.

  2. Spatial representation measurements alongside the class probes (8sWH). We added iREPA-based spatial metrics to the main text, examining both SiHC’s workspace and its persistent states. Together with the existing class probes, these measurements make their division of responsibilities more concrete: spatial information remains in the prediction states, while the workspace develops class-readable features.

  3. Partial whitening and cleaner bandwidth controls (bAXJ). The main text now includes intermediate whitening levels, tracing how filtering, class readability and generation change as the patch spectrum becomes flatter. Matched static iHC and SiHC comparisons at four and sixteen streams further separate the effects of expanding bandwidth and assigning local prediction responsibilities.

  4. A matched comparison with and without REPA (8sWH and VCo1). The appendix now includes SiHC-XL results at the same training budget with and without alignment. This comparison clarifies REPA’s contribution and connects its benefit to the representation-learning difficulty that remains when expanded residual streams still carry noise-dependent information.

  5. More detailed hardware measurements (bAXJ and VCo1). We added matched training and sampling runtime comparisons for plain JiT, mHC and SiHC, together with memory measurements and fused versus unfused SiHC results. These quantify the practical overhead of expanded persistent state and the effect of fusion.

We hope these revisions make the paper’s central contribution clearer: a systematic account of how prediction targets and architecture shape representation learning through residual-stream burden. This account brings together the design principles behind JiT, DeCo, PixelDiT and DiP, and connects them to learning in pretrained representation spaces such as RAE. SiHC examines this understanding from a complementary direction, highlighting residual-stream bandwidth as a resource for accommodating prediction demands and offering a diffusion-model perspective on the benefits of Hyper-Connections. We thank the reviewers for their thoughtful feedback and hope these revisions address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.