acceptodds
Under review as a conference paper at ICLR 2027

On the Mixing Trade-off for Expressive Hyper-Connections: A Geometric Contraction

Abstract

Manifold-constrained Hyper-Connections () extends residual connections to multiple streams and stabilizes residual propagation by constraining stream-mixing residual matrices to the Birkhoff polytope, the set of doubly stochastic matrices. In this work, we investigate *how to effectively mix residual streams within the Birkhoff polytope*. We reveal a **mixing trade-off**: when **under-mixed** residual matrices stay close to the vertices of the Birkhoff polytope, hyper-connections will degenerate into a set of nearly independent streams; when **over-mixed** residual matrices are concentrated around the uniform barycenter of the polytope, streams will become less distinguishable from each other. We theoretically validate our findings by bounding the cumulative residual mixing, stream difference and gradient difference of mixing coefficients across layers. Motivated by these findings, we propose Contraction for Manifold-Constrained Hyper-Connections (), an exact doubly stochastic method which geometrically contracts the Birkhoff polytope toward its interior. This contraction mitigates under-mixing by forcing the residual matrices to explore deeper into the polytope. Moreover, we apply a moderate contraction at the shallow layers to prevent over-mixing. Extensive experiments on GPT-2 and Mixture-of-Experts at both 6B and 300B parameter scales demonstrate the effectiveness and scalability of . We also develop fused operators for on Ascend NPUs, reducing hyper-connection execution latency by up to 44.8%. Code is available at https://anonymous.4open.science/r/cmhc.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hyper-Connections, Multi-Stream Architectures, Birkhoff Polytope, Geometric Contraction, Large Language Model, Pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_br6rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the trade-off between insufficient and excessive residual-stream mixing in manifold-constrained Hyper-Connections. It proposes cmHC, which contracts doubly stochastic mixing matrices toward the uniform matrix and applies moderate contraction to selected shallow layers. The paper analyzes cumulative mixing, stream differences, and coefficient-gradient differences, and evaluates the method using nanoGPT models, 6B/300B MoE training, and fused Ascend operators.

Critical strengths
  1. The intervention is simple and practical. It introduces no additional learnable parameters and preserves double stochasticity in exact arithmetic. Its compatibility with the existing permutation-mixture parameterization makes it straightforward to implement and potentially useful for improving residual-stream mixing at little additional cost.

  2. The paper combines informative mixing diagnostics with substantial empirical and engineering work. The analyses of cumulative mixing and stream differences highlight a useful design trade-off, while three-run nanoGPT results, large-scale MoE experiments, and detailed fused-operator implementations provide evidence of practical feasibility.

Critical weaknesses
  1. The central mechanism and shallow-layer strategy are not sufficiently established. The paper explicitly introduces a local linear approximation and assumes ΠnAlJn≈0\Pi_n A_l J_n\approx 0 for Theorem 3.2. However, the resulting bound does not quantify errors from these simplifications, leaving its applicability to the original network unclear. Theorem 3.3 bounds coefficient-gradient differences, but a smaller bound does not establish entropy-increasing update directions: equal coefficient gradients give zero instantaneous softmax-logit gradients through this path. The observed smoothing is informative, but does not establish the proposed causal explanation. Furthermore, varying the number of contracted shallow layers changes both placement and cumulative contraction strength, so these experiments do not isolate an advantage of shallow placement. These gaps weaken the justification for the proposed design.

  2. The evaluation does not yet establish consistent incremental benefits over the closest alternatives. The default configuration improves over mHC-lite at all three OpenWebText scales. However, its validation-loss gains over the best reported baseline at each scale are 0.0056, 0.0018, and 0.0005, and it does not consistently outperform the best baseline on FineWeb-Edu. Per-seed comparisons would help assess the reliability of the smaller gains. The 300B experiment provides valuable feasibility evidence over 2,000 steps and 33.55B processed tokens, but lacks an mHC-lite control and held-out evaluation. The operator-level speedups over Sinkhorn-based mHC are useful; a comparison with an equally optimized mHC-lite implementation would clarify efficiency relative to the closest predecessor. The main unresolved issue is whether the proposed intervention delivers reproducible model-quality improvements beyond the existing alternatives.

Questions for authors
  1. Can the assumptions of Theorem 3.2 be made exact, or can the bound include the neglected coupling and linearization-error terms? What evidence, beyond the gradient-difference bound, supports the proposed explanation for increased coefficient entropy?

  2. With the same number of contracted layers and the same contraction coefficient, does shallow placement outperform deeper or distributed placement? A matched comparison at nanoGPT scale would help establish whether the placement strategy itself is beneficial.

  3. Can the authors provide held-out evaluations of existing MoE checkpoints and per-seed nanoGPT improvements with uncertainty estimates, using paired comparisons where seeds are matched? For the efficiency contribution, how does cmHC compare with mHC-lite under the same fusion strategy, and do the operator gains translate into improved end-to-end throughput?

5 comments
AuthorsOpenReview ↗

Response to Reviewer br6r (1/3)

We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below.

W1 & Q1. Local linear approximation and coefficient entropy

We use the local linear approximation to isolate the stream-mixing mechanism, an approach also used in theoretical analyses of Transformer mixing and rank collapse. For example, Dovonon et al. [1] analyze a restricted Transformer update without the full LayerNorm, MLP, multi-head and layer-varying dynamics, then examine its predictions in full models. Dong et al. [2] first isolate pure-attention dynamics, then examine the effects of other Transformer components. Besides, Wang et al. [3] separate token representations into a mean component and its complement, analogous to our decomposition across residual streams.

We also provide some insights by carrying out relaxation to the local linear approximation and keeping the nonlinear branch contributions. Let the nonlinear branch at layer ll be Fl(xl)=Alxl+ϵl\mathbf F_l(\mathbf x_l)=\mathbf A_l\mathbf x_l+\epsilon_l where ϵl\epsilon_l is the residual of the local linear approximation. Define rl=ΠAlJxl+Πϵl,\mathbf{r}_l=\mathbf{\Pi} \mathbf{A}_l\mathbf{J}\mathbf{x}_l+\mathbf{\Pi}\epsilon_l,

where ϵl\epsilon_l is the residual of the local linear model. The perturbed recurrence gives

\leq (\lambda'_l+\ell_l) \Vert \widetilde{\mathbf{x}}_l \Vert _F+ \Vert \mathbf{r}_l \Vert _F.$$ Consequently, $$\Vert \widetilde{\mathbf{x}}_k \Vert _F \leq \left(\prod_{l=1}^{k-1}a_l\right) \Vert \widetilde{\mathbf{x}}_1 \Vert _F +\sum_{m=1}^{k-1} \left(\prod_{l=m+1}^{k-1}a_l\right) \Vert \mathbf{r}_m \Vert _F, \qquad a_l=\lambda'_l+\ell_l.$$ Theorem 3.2 is approximately recovered when the weighted additive term is negligible. Otherwise, the nonlinear network has a larger upper bound. However, we make two points here that (1) this nonlinear term mainly affects the existence of the upper bound. We can know from Figure 4 that in a full nonlinear network, the stream-difference norms do not show continued growth, and stronger contraction is associated with smaller norms. These observations imply **the weighted \$ \Vert \mathbf{r}_m \Vert \_F\$ term is not dominant in the inequality above**, and empirically support the extension of Theorem 3.2 to nonlinear networks. (2) The local linear approximation only **affects part of our over-mixing analysis**; Theorem 3.1 bounds the product of residual mixing matrices without the local linear approximation, so the core conclusion of the over-mixing analysis extends to the full nonlinear network. We present the linearized results as a mechanistic explanation of the additional "*butterfly effect*" observations that can amplify over-mixing. Regarding the entropy-increasing update: **we do not claim that Theorem 3.3 implies an entropy-increasing update direction**; this is likely not the actual situation either as you suggest. We only mention *normalized entropy* in Appendix E.3 for measuring how broadly the learned coefficients $\alpha_l$ are distributed over permutation vertices. Figure 10 reports higher entropy for cmHC than mHC-lite in the majority of layers; this is **observational evidence of a smoother coefficient distribution rather than a theoretical evidence of entropy-increasing update of model parameters**. We will clarify this distinction in the revised text. [1] Dovonon et al. Setting the record straight on Transformer oversmoothing. TMLR 2025 [2] Dong et al. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. ICML 2021 [3] Wang et al. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. ICLR 2022 > W1 & Q2. **Advantage of shallow contraction** To examine whether shallow contraction placement outperform deeper placement, we conduct an experiment by keeping $\lambda=0.97$ and the other experimental settings fixed, and applying contraction to the last $m$ layers, denoted by $K=-m$ for $m\in\lbrace1,2,10\rbrace$. Results are shown in the table below. (To be continued in Part 2)
AuthorsOpenReview ↗

Response to Reviewer br6r (2/3)

W1 & Q2. Advantage of shallow contraction (continued)

Table. Shallow and deep contractions on OpenWebText (mean ± standard deviation over three runs) with fixed λ=0.97\lambda=0.97. Best results are in bold.

⟵\longleftarrownanoGPT-S⟶\longrightarrow⟵\longleftarrownanoGPT-M⟶\longrightarrow⟵\longleftarrownanoGPT-L⟶\longrightarrow
Train lossVal. lossGrad. normTrain lossVal. lossGrad. normTrain lossVal. lossGrad. norm
Shallow cmHC (K=2K=2)3.4723 ± 0.00253.4686 ± 0.00270.3774 ± 0.02003.2548 ± 0.00063.2540 ± 0.00070.3726 ± 0.01373.1791 ± 0.00323.1904 ± 0.00320.4353 ± 0.0140
K=−1K=-13.4780 ± 0.00283.4743 ± 0.00270.3770 ± 0.01993.2557 ± 0.00193.2549 ± 0.00200.3728 ± 0.01483.1812 ± 0.00233.1918 ± 0.00260.4360 ± 0.0153
K=−2K=-23.4847 ± 0.00223.4802 ± 0.00220.3940 ± 0.03373.2576 ± 0.00473.2569 ± 0.00460.3738 ± 0.01673.1795 ± 0.00183.1908 ± 0.00180.4368 ± 0.0151
K=−10K=-103.5225 ± 0.04963.5184 ± 0.04921.3792 ± 0.19613.2528 ± 0.00193.2518 ± 0.00170.3761 ± 0.01693.1798 ± 0.00223.1906 ± 0.00210.4449 ± 0.0172

We can observe that shallow contraction yields lower training and validation losses than deep contraction on both nanoGPT-S and nanoGPT-L. The only exception is K=−10K=-10 on nanoGPT-M, but the same setting substantially raises nanoGPT-S validation loss to 3.5184. Deep contraction models also have higher gradient norms in 8/9 cases, suggesting less stable training. These results indicate that deep contraction does not yield a consistent improvement over shallow contraction. We will add the additional results to the revised PDF.

W2 & Q3. Reliability, uncertainty and efficiency analysis

Reliability of the gains. We extended the comparison beyond mHC-lite to all six baselines on nanoGPT-S/M/L and both datasets. The table below shows average validation loss gaps between cmHC and all baselines. We report Δ=Lbaseline−LcmHC\Delta=\mathcal{L}_{\mathrm{baseline}}-\mathcal{L}_{\mathrm{cmHC}}, so a positive value favors cmHC. For each baseline, we also report the one-sided Wilcoxon signed-rank pp-value across the six settings, to assess whether the loss gaps consistently favor cmHC.

Table. Average validation loss gaps between each baseline and cmHC across three runs.

Dataset⟵\longleftarrowOpenWebText⟶\longrightarrow⟵\longleftarrowFineWeb-Edu⟶\longrightarrowcmHC winsWilcoxon pp-value
nanoGPT sizeSMLSML
Residual, n=1n=1+0.0950+0.0823+0.0538+0.0647+0.0742+0.04536/60.016 (<0.05)
Residual, n=4n=4 (no mixing)+0.0056+0.0047+0.0005+0.0012+0.0010+0.00026/60.016 (<0.05)
HC+0.0110+0.0141+0.0452+0.0052+0.0434+0.05466/60.016 (<0.05)
mHC+0.0089+0.0018+0.0016-0.0005+0.0007+0.00215/60.031 (<0.05)
mHC-lite+0.0073+0.0024+0.0045-0.0012+0.0013+0.00295/60.031 (<0.05)
KromHC+0.0080+0.0062+0.0030+0.0001+0.0016-0.00155/60.047 (<0.05)

Across both datasets and six baselines, cmHC obtains a lower average loss in 33/3633/36 (91.7%) comparisons. It outperforms all six baselines for nanoGPT-S and nanoGPT-M on OpenWebText, and for nanoGPT-M on FineWeb-Edu. At the p<0.05p<0.05 threshold, the one-sided Wilcoxon test is significant for each baseline across the six settings, supporting the significance and robustness of the improvements. Furthermore, Figures 6 and 8 provide complementary evidence: despite fluctuations across training steps, the training-loss gaps of mHC-lite and other baselines generally remain above the cmHC zero-gap reference line (pink) in later training.

Per-seed comparisons. We repeat the nanoGPT-S/M experiments with seed 1338, in addition to default seed 1337 used for Table 1 from Section 4.1. We show results of cmHC and other strong baselines below. cmHC uses λ=0.97,K=2\lambda=0.97,K=2.

Table. nanoGPT-S/M results with seed 1338 (mean ±\pm standard deviation for 3 replicate experiments). Best results are in bold.

nanoGPT-S⟶\longrightarrownanoGPT-M⟶\longrightarrow
OpenWebTextTrain lossVal. lossTrain lossVal. loss
mHC3.4677 ± 0.00133.4822 ± 0.00093.2674 ± 0.00213.2699 ± 0.0019
mHC-lite3.4668 ± 0.00023.4808 ± 0.00033.2731 ± 0.00223.2753 ± 0.0018
KromHC3.4672 ± 0.00273.4818 ± 0.00273.2677 ± 0.00163.2699 ± 0.0012
cmHC3.4651 ± 0.00163.4795 ± 0.00143.2626 ± 0.00343.2653 ± 0.0041
Fineweb-EduTrain lossVal. lossTrain lossVal. loss
mHC3.4638 ± 0.00163.4774 ± 0.00173.2402 ± 0.00163.2531 ± 0.0018
mHC-lite3.4644 ± 0.00203.4779 ± 0.00213.2399 ± 0.00223.2526 ± 0.0020
KromHC3.4638 ± 0.00023.4775 ± 0.00043.2430 ± 0.00323.2556 ± 0.0030
cmHC3.4641 ± 0.00073.4777 ± 0.00083.2385 ± 0.00033.2512 ± 0.0007

(To be continued in Part 3)

AuthorsOpenReview ↗

Response to Reviewer br6r (3/3)

W2 & Q3. Reliability, uncertainty and efficiency analysis (continued)

We can tell from the table above that with seed 1338, cmHC has the lowest mean training and validation losses among the evaluated baselines on nanoGPT-S/M + OpenWebText and nanoGPT-M + FineWeb-Edu. Its train/validation loss gaps to the strongest evaluated baseline are 0.0017/0.0013, 0.0048/0.0046 and 0.0014/0.0014, respectively. 5/6 of the gaps exceed cmHC's corresponding standard deviations (with the remaining one 0.0013 vs 0.0014). These are also the same three settings in which cmHC has the lowest mean validation loss in Table 1 with seed 1337, providing evidence that the advantage persists across seeds. On nanoGPT-S + FineWeb-Edu (the setting that mHC-lite outperforms cmHC by 0.0013/0.0012 in Table 1), all four methods perform on par: their mean training and validation losses span only 0.0006 and 0.0005, respectively.

mHC-lite at 300B. Among the baselines considered, mHC is the only one with a public implementation at 300B+ scale available for our training setup (i.e., the DeepSeek series), including the custom kernels and a large amount of other engineering components. Bringing mHC-lite to a comparably optimized 300B implementation would require substantial additional engineering and compute. Given the timeframe and available resources, we have to limit the 300B comparison to mHC.

Uncertainty estimates. Run-to-run variation in our repeated experiments arise from stochastic training in nanoGPT or other potential factors, and such variation is also present across the baselines despite the fixed seed. Thus, we empirically analyze such uncertainty by the reported standard deviations (SD) in Table 1. The table below shows that, averaged over the six settings in Table 1, cmHC has the second-lowest SD for both training and validation loss, better than strong baselines like mHC-lite and KromHC while only behind the standard residual connection with n=1n=1. This provides additional support to the robustness of cmHC's advantage in average loss gaps.

Table. Average loss SD over six settings from Table 1. The lowest is in bold and the second-lowest is underlined‾\underline{\text{underlined}}.

Residual, n=1n=1Residual, n=4n=4HCmHCmHC-liteKromHCcmHC
Train loss SD0.0015\mathbf{0.0015}0.00200.00200.00470.00470.00200.00200.00250.00250.00210.00210.0016‾\underline{0.0016}
Val. loss SD0.0013\mathbf{0.0013}0.00200.00200.00540.00540.00200.00200.00240.00240.00220.00220.0018‾\underline{0.0018}

Efficiency contribution. cmHC and mHC-lite can share the same operator implementation. Our fused kernels therefore benefit both methods, providing an operator-level performance advantage over the fused mHC implementation evaluated in the paper.

For end-to-end throughput, the operator-level experiments in the paper provide a controlled comparison of cmHC and mHC, isolating operator efficiency from system-level factors. End-to-end gains additionally depend on communication overlap, recomputation, and parallelization; lower operator overhead can create headroom for optimizing these aspects.

To provide complementary end-to-end evidence, we compare fused cmHC with fused mHC on our 6B model using 16 Ascend 910C NPUs, a sequence length of 4,096, a global batch size of 64, and forced expert load balancing to control for routing imbalance (which is common in benchmarking the training speed of MoE models). Under the configurations below, cmHC improves throughput by 0.6–34.2%, demonstrating that the operator-level gains can translate into end-to-end benefits, with the magnitude depending on the configuration.

Table. End-to-end training efficiency of fused mHC and cmHC. Step times are medians over 10 measured steps following five warm-up steps.

Activation recomputationMicro-batch size per NPUmHC step time (s)↓\downarrowcmHC step time (s)↓\downarrowcmHC throughput gain$\uparrow$
Full24.9184.668+5.4%
Full44.3633.810+14.5%
None15.5375.502+0.6%
Selective26.6684.969+34.2%
Reviewer_eZdXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how information mixes across residual streams in manifold-constrained Hyper-Connections (mHC). The authors identify two problems: insufficient mixing limits information exchange, while excessive mixing makes streams increasingly similar. To address this, they propose cmHC, which encourages moderate mixing in the first few layers while preserving mHC's stability guarantees. The paper provides theoretical analysis and evaluates the method on language models ranging from GPT-2 to 300B-parameter MoE models.

Critical strengths
  1. The paper addresses a relevant problem with a simple and practical solution that requires no additional learnable parameters.
  2. The theoretical analysis provides useful insights into how mixing accumulates across layers and affects differences between streams.
  3. The experiments cover multiple model sizes, datasets, and mixing configurations, including very large-scale MoE training.
Critical weaknesses
  1. The contribution of the method seems incremental, building on mHC-lite. Under-mixing was previously investigated by Alimaskina et al. (2026), and Zhao et al. (2026) showed that early-layer mixing is more important than late-layer mixing. The paper could better distinguish its contributions and position itself correctly in the literature.
  2. The analysis establishes properties of the mixing matrices but does not directly explain why the proposed approach should improve model quality. For example, it is unclear if reduced differences between streams imply reduced functional diversity (Peng et al. 2026). More insights into this would be helpful.
  3. While cmHC often improves training loss, the gains in validation loss are generally small and not consistent across all configurations. It is unclear whether they translate into meaningful improvements in model capability.
Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Response to Reviewer eZdX (1/2)

We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below.

W1 & W2. Contribution significance, why cmHC should improve model quality, and related work

Simplicity is a strength of the proposed method: a simple geometric contraction addresses the under- and over-mixing problems in Hyper-Connections without introducing additional learnable parameters. This also makes cmHC compatible with large-scale training and fused implementations, which are also important contributions of our work. Additionally, the principles behind this simple solution are nontrivial; we conduct extensive theoretical and empirical analyses to explain why this design is effective. First, mHC-lite optimizes over the convex hull of permutation matrices and can therefore approach its boundary or a permutation vertex, while cmHC contracts the entire feasible polytope toward its barycenter, imposing a strictly positive lower bound on every mixing entry while preserving exact double stochasticity from a geometric perspective, and thus improves the expressiveness of Hyper-Connections. Second, the underlying mixing trade-off paradigm concerns how strongly and across how many layers residual streams are mixed: under-mixing limits information exchange, while over-mixing makes the streams less distinguishable. This trade-off can guide the design of Hyper-Connections more broadly, beyond an incremental change to mHC-lite.

Related work. Here we clarify how our contribution relates to Alimaskina et al. [1], Zhao et al. [2] and Peng et al. [3] you mentioned in W1 & W2.

  • Alimaskina et al. [1] diagnose stream collapse, near-identity residual mixing, etc., and mitigate these effects by breaking symmetry at initialization. Their observations reinforce our motivation that a nominally multi-stream architecture may underutilize cross-stream exchange. However, our contribution is complementary: instead of modifying initialization, cmHC changes the feasible geometry of every residual matrix, guarantees a positive mixing lower bound, and explicitly studies the trade-off between under-mixing and over-mixing.

  • Zhao et al. [2] provide causal evidence that early mixers are more functionally important than late mixers in a trained mHC model. Their work appeared in September 2026 after this work had finished, and therefore should not diminish our novelty. We will cite this as an extra support for our mixing trade-off strategy. Besides, compared to Zhao et al., we further develop a systematic theoretical analysis of both under- and over-mixing, as well as a geometric contraction that encourages moderate early mixing.

    We also conduct an experiment to inspect the early mixers (contraction in shallow layers) and late mixers (resp. deep layers): we kept λ=0.97\lambda=0.97 and other settings fixed, and applied contraction to the last mm layers, denoted by K=−mK=-m for m∈{1,2,10}m\in\lbrace1,2,10\rbrace. Results are shown in the table below.

    Table. Shallow and deep contractions on OpenWebText (mean ± standard deviation over three runs) with fixed λ=0.97\lambda=0.97. Best results are in bold.

    ⟵\longleftarrownanoGPT-S⟶\longrightarrow⟵\longleftarrownanoGPT-M⟶\longrightarrow⟵\longleftarrownanoGPT-L⟶\longrightarrow
    Train lossVal. lossGrad. normTrain lossVal. lossGrad. normTrain lossVal. lossGrad. norm
    Shallow cmHC (K=2K=2)3.4723 ± 0.00253.4686 ± 0.00270.3774 ± 0.02003.2548 ± 0.00063.2540 ± 0.00070.3726 ± 0.01373.1791 ± 0.00323.1904 ± 0.00320.4353 ± 0.0140
    K=−1K=-13.4780 ± 0.00283.4743 ± 0.00270.3770 ± 0.01993.2557 ± 0.00193.2549 ± 0.00200.3728 ± 0.01483.1812 ± 0.00233.1918 ± 0.00260.4360 ± 0.0153
    K=−2K=-23.4847 ± 0.00223.4802 ± 0.00220.3940 ± 0.03373.2576 ± 0.00473.2569 ± 0.00460.3738 ± 0.01673.1795 ± 0.00183.1908 ± 0.00180.4368 ± 0.0151
    K=−10K=-103.5225 ± 0.04963.5184 ± 0.04921.3792 ± 0.19613.2528 ± 0.00193.2518 ± 0.00170.3761 ± 0.01693.1798 ± 0.00223.1906 ± 0.00210.4449 ± 0.0172

    We can observe that shallow contraction yields lower training and validation losses than deep contraction on both nanoGPT-S and nanoGPT-L. The only exception is K=−10K=-10 on nanoGPT-M, but the same setting substantially raises nanoGPT-S validation loss to 3.5184. Deep contraction models also have higher gradient norms in 8/9 cases, suggesting less stable training. These results supports the early mixer as a more robust configuration. We will add the additional results to the revised PDF.

[1] Alimaskina et al. Analyzing stream collapse in Hyper-Connections: from diagnosis to mitigation. CIKM 2026

[2] Zhao et al. How does mHC use its residual streams? selective routing and near-identity mixing. arXiv:2609.05309

(To be continued in Part 2)

AuthorsOpenReview ↗

Response to Reviewer eZdX (2/2)

W1 & W2. Related work (continued)

  • As we have discussed in Appendix B, Peng et al. [3] show that representational similarity does not establish functional interchangeability. Our work has led to different conclusions, which can be explained in the following two aspects. (1) Our work examines a different question: we study stream differences through the convergence of residual mixing and the resulting risk of over-mixing. (2) Our concern is that residual mixing can drive streams toward identical states, rather than merely making their representations similar. Figure 3 illustrates this risk: with λ=0.7\lambda=0.7, the cumulative residual matrix is almost completely uniform. In this case, the residual branch progressively loses its ability to preserve differences among streams, leading to limited expressivity of the multi-stream architecture.

[3] Peng et al. Ablate and rescue: a causal analysis of residual stream Hyper-Connections. ICLR Workshop Sci4DL 2026

W3. Significance of the validation gains

We extend the comparison beyond mHC-lite to all six baselines on nanoGPT-S/M/L and both datasets. The table below shows average validation loss gaps between cmHC and baselines. We report Δ=Lbaseline−LcmHC\Delta=\mathcal{L}_{\mathrm{baseline}}-\mathcal{L}_{\mathrm{cmHC}}, so a positive value favors cmHC. For each baseline, we also report the one-sided Wilcoxon signed-rank pp-value across the six settings, to assess whether the loss gaps consistently favor cmHC.

Table. Average validation loss gaps between each baseline and cmHC across three runs.

Dataset⟵\longleftarrowOpenWebText⟶\longrightarrow⟵\longleftarrowFineWeb-Edu⟶\longrightarrowcmHC winsWilcoxon pp-value
nanoGPT sizeSMLSML
Residual, n=1n=1+0.0950+0.0823+0.0538+0.0647+0.0742+0.04536/60.016 (<0.05)
Residual, n=4n=4 (no mixing)+0.0056+0.0047+0.0005+0.0012+0.0010+0.00026/60.016 (<0.05)
HC+0.0110+0.0141+0.0452+0.0052+0.0434+0.05466/60.016 (<0.05)
mHC+0.0089+0.0018+0.0016-0.0005+0.0007+0.00215/60.031 (<0.05)
mHC-lite+0.0073+0.0024+0.0045-0.0012+0.0013+0.00295/60.031 (<0.05)
KromHC+0.0080+0.0062+0.0030+0.0001+0.0016-0.00155/60.047 (<0.05)

Across both datasets and six baselines, cmHC obtains a lower average loss in 33/3633/36 (91.7%) comparisons. It outperforms all six baselines for nanoGPT-S and nanoGPT-M on OpenWebText, and for nanoGPT-M on FineWeb-Edu. At the p<0.05p<0.05 threshold, the one-sided Wilcoxon test is significant for each baseline across the six settings, supporting the significance and robustness of the improvements. Furthermore, Figures 6 and 8 provide complementary evidence: despite fluctuations across training steps, the training-loss gaps of mHC-lite and other baselines generally remain above the cmHC zero-gap reference line (pink) in later training.

To further show that the observed improvements stem from gains in model capability rather than a lucky seed, we also repeat the nanoGPT-S/M experiments with seed 1338, in addition to default seed 1337 used for Table 1 from Section 4.1. We show results of cmHC and other strong baselines below. cmHC uses λ=0.97,K=2\lambda=0.97,K=2.

Table. nanoGPT-S/M results with seed 1338 (mean ±\pm standard deviation for 3 replicate experiments). Best results are in bold.

nanoGPT-S⟶\longrightarrownanoGPT-M⟶\longrightarrow
OpenWebTextTrain lossVal. lossTrain lossVal. loss
mHC3.4677 ± 0.00133.4822 ± 0.00093.2674 ± 0.00213.2699 ± 0.0019
mHC-lite3.4668 ± 0.00023.4808 ± 0.00033.2731 ± 0.00223.2753 ± 0.0018
KromHC3.4672 ± 0.00273.4818 ± 0.00273.2677 ± 0.00163.2699 ± 0.0012
cmHC3.4651 ± 0.00163.4795 ± 0.00143.2626 ± 0.00343.2653 ± 0.0041
Fineweb-EduTrain lossVal. lossTrain lossVal. loss
mHC3.4638 ± 0.00163.4774 ± 0.00173.2402 ± 0.00163.2531 ± 0.0018
mHC-lite3.4644 ± 0.00203.4779 ± 0.00213.2399 ± 0.00223.2526 ± 0.0020
KromHC3.4638 ± 0.00023.4775 ± 0.00043.2430 ± 0.00323.2556 ± 0.0030
cmHC3.4641 ± 0.00073.4777 ± 0.00083.2385 ± 0.00033.2512 ± 0.0007

With seed 1338, cmHC has the lowest mean training and validation losses among the evaluated baselines on nanoGPT-S/M + OpenWebText and nanoGPT-M + FineWeb-Edu. Its train/validation loss gaps to the strongest evaluated baseline are 0.0017/0.0013, 0.0048/0.0046 and 0.0014/0.0014, respectively. 5/6 of the gaps exceed cmHC's corresponding standard deviations (with the remaining one 0.0013 vs 0.0014). These are also the same three settings in which cmHC has the lowest mean validation loss in Table 1 with seed 1337, providing evidence that the advantage persists across seeds. On nanoGPT-S + FineWeb-Edu (the setting that mHC-lite outperforms cmHC by 0.0013/0.0012 in Table 1), all four methods perform on par: their mean training and validation losses span only 0.0006 and 0.0005, respectively.

AuthorsOpenReview ↗

Follow-up on Our Revisions

Dear Reviewer,

Thank you again for your careful review. We have posted our responses to address the concerns you raised. As the discussion period draws to a close, we would value your thoughts on whether the revisions address your main concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Reviewer_qZ3nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the mixing trade-off in manifold-constrained Hyper-Connections (mHC), where under-mixing limits cross-stream information exchange while over-mixing reduces stream distinguishability. The authors propose cmHC, which geometrically contracts the Birkhoff polytope toward its interior, and apply moderate contraction to shallow layers. Experiments are conducted on GPT-2 and 6B/300B MoE models, along with fused operator implementations on Ascend NPUs.

Critical strengths
  • The paper provides a useful perspective on the trade-off between under-mixing and over-mixing, supported by theoretical analyses of residual mixing and stream differences.
  • The experiments cover models up to 300B parameters, providing preliminary evidence of the scalability of the proposed approach.
  • The fused operators on Ascend NPUs achieve up to 44.8% reduction in hyper-connection execution latency, demonstrating practical engineering value.
Critical weaknesses
  • The proposed method is relatively incremental over mHC-lite, essentially introducing a simple interpolation between the original residual matrix and the uniform matrix.
  • The performance gains over existing baselines are generally small. The proposed method requires tuning hyperparameters such as the contraction strength and the number of contracted layers. Moreover, many of the performance differences in Table 1 are comparable to the reported standard deviations, raising concerns about the significance and robustness of the improvements.
  • The theoretical analysis mainly provides intuition rather than directly justifying the effectiveness of the proposed method. In particular, the results rely on simplified assumptions and do not establish why shallow-layer contraction should improve the performance of the full nonlinear network.
Questions for authors
  • Theorem 3.1 provides a bound on cumulative residual mixing that depends on the product of contraction coefficients, but does not distinguish between contractions applied at different layer positions. Could the authors further justify why shallow-layer contraction is preferred over deep-layer contraction?
  • Theorem 3.2 relies on approximations that neglect certain contributions from the Transformer branch, which may introduce new inter-stream differences. Could the authors further justify these assumptions and clarify whether the theoretical conclusions extend to the full nonlinear network?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qZ3n (1/3)

We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below.

W1. Simple design and contribution significance

Simplicity is a strength of the proposed method: a simple geometric contraction addresses the under- and over-mixing problems in Hyper-Connections without introducing additional learnable parameters. This also makes cmHC compatible with large-scale training and fused implementations, which are also important contributions of our work. Additionally, the principles behind this simple solution are nontrivial; we conduct extensive theoretical and empirical analyses to explain why this design is effective. First, mHC-lite optimizes over the convex hull of permutation matrices and can therefore approach its boundary or a permutation vertex, while cmHC contracts the entire feasible polytope toward its barycenter, imposing a strictly positive lower bound on every mixing entry while preserving exact double stochasticity from a geometric perspective. Second, the underlying mixing trade-off paradigm concerns how strongly and across how many layers residual streams are mixed: under-mixing limits information exchange, while over-mixing makes the streams less distinguishable. This trade-off can guide the design of Hyper-Connections more broadly, beyond an incremental change to mHC-lite.

W2. Significance and robustness of the performance gains

Significance of the improvements. We extend the comparison beyond mHC-lite to all six baselines on nanoGPT-S/M/L and both datasets. The table below shows average validation loss gaps between cmHC and baselines. We report Δ=Lbaseline−LcmHC\Delta=\mathcal{L}_{\mathrm{baseline}}-\mathcal{L}_{\mathrm{cmHC}}, so a positive value favors cmHC. For each baseline, we also report the one-sided Wilcoxon signed-rank pp-value across the six settings, to assess whether the loss gaps consistently favor cmHC.

Table. Average validation loss gaps between each baseline and cmHC across three runs.

Dataset⟵\longleftarrowOpenWebText⟶\longrightarrow⟵\longleftarrowFineWeb-Edu⟶\longrightarrowcmHC winsWilcoxon pp-value
nanoGPT sizeSMLSML
Residual, n=1n=1+0.0950+0.0823+0.0538+0.0647+0.0742+0.04536/60.016 (<0.05)
Residual, n=4n=4 (no mixing)+0.0056+0.0047+0.0005+0.0012+0.0010+0.00026/60.016 (<0.05)
HC+0.0110+0.0141+0.0452+0.0052+0.0434+0.05466/60.016 (<0.05)
mHC+0.0089+0.0018+0.0016-0.0005+0.0007+0.00215/60.031 (<0.05)
mHC-lite+0.0073+0.0024+0.0045-0.0012+0.0013+0.00295/60.031 (<0.05)
KromHC+0.0080+0.0062+0.0030+0.0001+0.0016-0.00155/60.047 (<0.05)

Across both datasets and six baselines, cmHC obtains a lower average loss in 33/3633/36 (91.7%) comparisons. It outperforms all six baselines for nanoGPT-S and nanoGPT-M on OpenWebText, and for nanoGPT-M on FineWeb-Edu. At the p<0.05p<0.05 threshold, the one-sided Wilcoxon test is significant for each baseline across the six settings, supporting the significance and robustness of the improvements. Furthermore, Figures 6 and 8 provide complementary evidence: despite fluctuations across training steps, the training-loss gaps of mHC-lite and other baselines generally remain above the cmHC zero-gap reference line (pink) in later training.

Relatively large standard deviations (SD). Run-to-run variation in our repeated experiments may arise from stochastic training in nanoGPT or other potential factors, and such variation is also present across the baselines. The table below shows that, averaged over the six settings in Table 1, cmHC has the second-lowest SD for both training and validation loss, better than strong baselines like mHC-lite and KromHC while only behind the standard residual connection with n=1n=1. This provides additional support to the robustness of cmHC's advantage in average loss gaps.

Table. Average loss SD over six settings from Table 1. The lowest is in bold and the second-lowest is underlined‾\underline{\text{underlined}}.

Residual, n=1n=1Residual, n=4n=4HCmHCmHC-liteKromHCcmHC
Train loss SD0.0015\mathbf{0.0015}0.00200.00200.00470.00470.00200.00200.00250.00250.00210.00210.0016‾\underline{0.0016}
Val. loss SD0.0013\mathbf{0.0013}0.00200.00200.00540.00540.00200.00200.00240.00240.00220.00220.0018‾\underline{0.0018}

(To be continued in Part 2)

AuthorsOpenReview ↗

Response to Reviewer qZ3n (2/3)

W2. Significance and robustness of the performance gains (continued)

We also repeat the nanoGPT-S/M experiments with seed 1338, in addition to default seed 1337 used for Table 1 from Section 4.1. We show results of cmHC and other strong baselines below. cmHC uses λ=0.97,K=2\lambda=0.97,K=2.

Table. nanoGPT-S/M results with seed 1338 (mean ±\pm standard deviation for 3 replicate experiments). Best results are in bold.

nanoGPT-S⟶\longrightarrownanoGPT-M⟶\longrightarrow
OpenWebTextTrain lossVal. lossTrain lossVal. loss
mHC3.4677 ± 0.00133.4822 ± 0.00093.2674 ± 0.00213.2699 ± 0.0019
mHC-lite3.4668 ± 0.00023.4808 ± 0.00033.2731 ± 0.00223.2753 ± 0.0018
KromHC3.4672 ± 0.00273.4818 ± 0.00273.2677 ± 0.00163.2699 ± 0.0012
cmHC3.4651 ± 0.00163.4795 ± 0.00143.2626 ± 0.00343.2653 ± 0.0041
Fineweb-EduTrain lossVal. lossTrain lossVal. loss
mHC3.4638 ± 0.00163.4774 ± 0.00173.2402 ± 0.00163.2531 ± 0.0018
mHC-lite3.4644 ± 0.00203.4779 ± 0.00213.2399 ± 0.00223.2526 ± 0.0020
KromHC3.4638 ± 0.00023.4775 ± 0.00043.2430 ± 0.00323.2556 ± 0.0030
cmHC3.4641 ± 0.00073.4777 ± 0.00083.2385 ± 0.00033.2512 ± 0.0007

With seed 1338, cmHC has the lowest mean training and validation losses among the evaluated baselines on nanoGPT-S/M + OpenWebText and nanoGPT-M + FineWeb-Edu. Its train/validation loss gaps to the strongest evaluated baseline are 0.0017/0.0013, 0.0048/0.0046 and 0.0014/0.0014, respectively. 5/6 of the gaps exceed cmHC's corresponding SDs (with the remaining one 0.0013 vs 0.0014). These are also the same three settings in which cmHC has the lowest mean validation loss in Table 1 with seed 1337, providing evidence that the advantage persists across seeds. On nanoGPT-S + FineWeb-Edu (the setting that mHC-lite outperforms cmHC by 0.0013/0.0012 in Table 1), all four methods perform on par: their mean training and validation losses span only 0.0006 and 0.0005, respectively.

Hyperparameter tuning. Our search over the contraction strength λ\lambda and the number of contracted layers KK is small. We test only three discrete values for λ:{0.70,0.85,0.97}\lambda: \lbrace0.70,0.85,0.97\rbrace, corresponding to off-diagonal entries {0.10,0.05,0.01}\lbrace0.10,0.05,0.01\rbrace in the contracted permutation matrices. For KK, we test only {2,5,10}\lbrace2,5,10\rbrace and all layers. Because fine-tuning large models is costly, we use the nanoGPT results to guide a direct choice of (λ,K)=(0.97,4)(\lambda,K)=(0.97,4) for 6B-MoE and (0.96,6)(0.96,6) for 300B-MoE. Without a further tuning sweep, cmHC achieves a lower training loss than mHC and mHC-lite at 6B, and than mHC at 300B (Figure 7). We did not conduct a finer hyperparameter search or select seeds. These findings further support the robustness of the reported gains across model scales.

W3 & Q1. Why contract shallow layers?

We provide an intuitive theoretical explanation by examining cumulative mixing at intermediate layers. Theorem 3.1 also applies to any prefix ending at layer kk, where 1≤k<L1\leq k<L:

τ(∏l=0k−1Hk−1−lres′)≤∏l=0k−1λl′.\tau\left(\prod_{l=0}^{k-1} \mathbf{H}^{\mathrm{res}'}_{k-1-l}\right) \leq \prod_{l=0}^{k-1}\lambda'_l.

This means the bound at layer kk depends only on the normalized contraction coefficients λl′\lambda_l' in the first kk layers. For a shallow contraction in the first KK layers, λl<1\lambda_l<1 for l=0,⋯ ,Kl=0,\cdots,K, so the Dobrushin's coefficient has a smaller bound at layer kk. Thus, shallow contraction can propagate its mixing effect through subsequent intermediate layers, whereas deep contraction affects only the final stages of stream interaction.

To examine this explanation in the full nonlinear model, we kept λ=0.97\lambda=0.97 and the other experimental settings fixed, and applied contraction to the last mm layers, denoted by K=−mK=-m for m∈{1,2,10}m\in\{1,2,10\}. Results are shown in the table below.

(To be continued in Part 3)

AuthorsOpenReview ↗

Response to Reviewer qZ3n (3/3)

W3 & Q1. Why contract shallow layers? (continued)

Table. Shallow and deep contractions on OpenWebText (mean ± standard deviation over three runs) with fixed λ=0.97\lambda=0.97. Best results are in bold.

⟵\longleftarrownanoGPT-S⟶\longrightarrow⟵\longleftarrownanoGPT-M⟶\longrightarrow⟵\longleftarrownanoGPT-L⟶\longrightarrow
Train lossVal. lossGrad. normTrain lossVal. lossGrad. normTrain lossVal. lossGrad. norm
Shallow cmHC (K=2K=2)3.4723 ± 0.00253.4686 ± 0.00270.3774 ± 0.02003.2548 ± 0.00063.2540 ± 0.00070.3726 ± 0.01373.1791 ± 0.00323.1904 ± 0.00320.4353 ± 0.0140
K=−1K=-13.4780 ± 0.00283.4743 ± 0.00270.3770 ± 0.01993.2557 ± 0.00193.2549 ± 0.00200.3728 ± 0.01483.1812 ± 0.00233.1918 ± 0.00260.4360 ± 0.0153
K=−2K=-23.4847 ± 0.00223.4802 ± 0.00220.3940 ± 0.03373.2576 ± 0.00473.2569 ± 0.00460.3738 ± 0.01673.1795 ± 0.00183.1908 ± 0.00180.4368 ± 0.0151
K=−10K=-103.5225 ± 0.04963.5184 ± 0.04921.3792 ± 0.19613.2528 ± 0.00193.2518 ± 0.00170.3761 ± 0.01693.1798 ± 0.00223.1906 ± 0.00210.4449 ± 0.0172

We can observe that shallow contraction yields lower training and validation losses than deep contraction on both nanoGPT-S and nanoGPT-L. The only exception is K=−10K=-10 on nanoGPT-M, but the same setting substantially raises nanoGPT-S validation loss to 3.5184. Deep contraction models also have higher gradient norms in 8/9 cases, suggesting less stable training. These results indicate that deep contraction does not yield a consistent improvement over shallow contraction. We will add the additional results to the revised PDF.

W3 & Q2. Scope of the local linear approximation

We use the local linear approximation to isolate the stream-mixing mechanism, an approach also used in theoretical analyses of Transformer mixing and rank collapse. For example, Dovonon et al. [1] analyze a restricted Transformer update without the full LayerNorm, MLP, multi-head and layer-varying dynamics, then examine its predictions in full models. Dong et al. [2] first isolate pure-attention dynamics, then examine the effects of other Transformer components. Besides, Wang et al. [3] separate token representations into a mean component and its complement, analogous to our decomposition across residual streams.

We also provide some insights by carrying out relaxation to the local linear approximation and keeping the nonlinear branch contributions. Let the nonlinear branch at layer ll be Fl(xl)=Alxl+ϵl\mathbf F_l(\mathbf x_l)=\mathbf A_l\mathbf x_l+\epsilon_l where ϵl\epsilon_l is the residual of the local linear approximation. Define rl=ΠAlJxl+Πϵl,\mathbf{r}_l=\mathbf{\Pi} \mathbf{A}_l\mathbf{J}\mathbf{x}_l+\mathbf{\Pi}\epsilon_l,

where ϵl\epsilon_l is the residual of the local linear model. The perturbed recurrence gives

\leq (\lambda'_l+\ell_l) \Vert \widetilde{\mathbf{x}}_l \Vert _F+ \Vert \mathbf{r}_l \Vert _F.$$ Consequently, $$\Vert \widetilde{\mathbf{x}}_k \Vert _F \leq \left(\prod_{l=1}^{k-1}a_l\right) \Vert \widetilde{\mathbf{x}}_1 \Vert _F +\sum_{m=1}^{k-1} \left(\prod_{l=m+1}^{k-1}a_l\right) \Vert \mathbf{r}_m \Vert _F, \qquad a_l=\lambda'_l+\ell_l.$$ Theorem 3.2 is approximately recovered when the weighted additive term is negligible. Otherwise, the nonlinear network has a larger upper bound. However, we make two points here that (1) this nonlinear term mainly affects the existence of the upper bound. We can know from Figure 4 that in a full nonlinear network, the stream-difference norms do not show continued growth, and stronger contraction is associated with smaller norms. These observations imply **the weighted \$ \Vert \mathbf{r}_m \Vert \_F\$ term is not dominant in the inequality above**, and empirically support the extension of Theorem 3.2 to nonlinear networks. (2) The local linear approximation only **affects part of our over-mixing analysis**; Theorem 3.1 bounds the product of residual mixing matrices without the local linear approximation, so the core conclusion of the over-mixing analysis extends to the full nonlinear network. We present the linearized results as a mechanistic explanation of the additional "*butterfly effect*" observations that can amplify over-mixing. [1] Dovonon et al. Setting the record straight on Transformer oversmoothing. TMLR 2025 [2] Dong et al. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. ICML 2021 [3] Wang et al. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. ICLR 2022
AuthorsOpenReview ↗

Follow-up on Our Revisions

Dear Reviewer,

Thank you again for your careful review. We have posted our responses to address the concerns you raised. As the discussion period draws to a close, we would value your thoughts on whether the revisions address your main concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.