- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the trade-off between insufficient and excessive residual-stream mixing in manifold-constrained Hyper-Connections. It proposes cmHC, which contracts doubly stochastic mixing matrices toward the uniform matrix and applies moderate contraction to selected shallow layers. The paper analyzes cumulative mixing, stream differences, and coefficient-gradient differences, and evaluates the method using nanoGPT models, 6B/300B MoE training, and fused Ascend operators.
Critical strengths
-
The intervention is simple and practical. It introduces no additional learnable parameters and preserves double stochasticity in exact arithmetic. Its compatibility with the existing permutation-mixture parameterization makes it straightforward to implement and potentially useful for improving residual-stream mixing at little additional cost.
-
The paper combines informative mixing diagnostics with substantial empirical and engineering work. The analyses of cumulative mixing and stream differences highlight a useful design trade-off, while three-run nanoGPT results, large-scale MoE experiments, and detailed fused-operator implementations provide evidence of practical feasibility.
Critical weaknesses
-
The central mechanism and shallow-layer strategy are not sufficiently established. The paper explicitly introduces a local linear approximation and assumes for Theorem 3.2. However, the resulting bound does not quantify errors from these simplifications, leaving its applicability to the original network unclear. Theorem 3.3 bounds coefficient-gradient differences, but a smaller bound does not establish entropy-increasing update directions: equal coefficient gradients give zero instantaneous softmax-logit gradients through this path. The observed smoothing is informative, but does not establish the proposed causal explanation. Furthermore, varying the number of contracted shallow layers changes both placement and cumulative contraction strength, so these experiments do not isolate an advantage of shallow placement. These gaps weaken the justification for the proposed design.
-
The evaluation does not yet establish consistent incremental benefits over the closest alternatives. The default configuration improves over mHC-lite at all three OpenWebText scales. However, its validation-loss gains over the best reported baseline at each scale are 0.0056, 0.0018, and 0.0005, and it does not consistently outperform the best baseline on FineWeb-Edu. Per-seed comparisons would help assess the reliability of the smaller gains. The 300B experiment provides valuable feasibility evidence over 2,000 steps and 33.55B processed tokens, but lacks an mHC-lite control and held-out evaluation. The operator-level speedups over Sinkhorn-based mHC are useful; a comparison with an equally optimized mHC-lite implementation would clarify efficiency relative to the closest predecessor. The main unresolved issue is whether the proposed intervention delivers reproducible model-quality improvements beyond the existing alternatives.
Questions for authors
-
Can the assumptions of Theorem 3.2 be made exact, or can the bound include the neglected coupling and linearization-error terms? What evidence, beyond the gradient-difference bound, supports the proposed explanation for increased coefficient entropy?
-
With the same number of contracted layers and the same contraction coefficient, does shallow placement outperform deeper or distributed placement? A matched comparison at nanoGPT scale would help establish whether the placement strategy itself is beneficial.
-
Can the authors provide held-out evaluations of existing MoE checkpoints and per-seed nanoGPT improvements with uncertainty estimates, using paired comparisons where seeds are matched? For the efficiency contribution, how does cmHC compare with mHC-lite under the same fusion strategy, and do the operator gains translate into improved end-to-end throughput?
5 comments
Response to Reviewer br6r (2/3)
W1 & Q2. Advantage of shallow contraction (continued)
Table. Shallow and deep contractions on OpenWebText (mean ± standard deviation over three runs) with fixed . Best results are in bold.
| nanoGPT-S | nanoGPT-M | nanoGPT-L | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Train loss | Val. loss | Grad. norm | Train loss | Val. loss | Grad. norm | Train loss | Val. loss | Grad. norm | |
| Shallow cmHC () | 3.4723 ± 0.0025 | 3.4686 ± 0.0027 | 0.3774 ± 0.0200 | 3.2548 ± 0.0006 | 3.2540 ± 0.0007 | 0.3726 ± 0.0137 | 3.1791 ± 0.0032 | 3.1904 ± 0.0032 | 0.4353 ± 0.0140 |
| 3.4780 ± 0.0028 | 3.4743 ± 0.0027 | 0.3770 ± 0.0199 | 3.2557 ± 0.0019 | 3.2549 ± 0.0020 | 0.3728 ± 0.0148 | 3.1812 ± 0.0023 | 3.1918 ± 0.0026 | 0.4360 ± 0.0153 | |
| 3.4847 ± 0.0022 | 3.4802 ± 0.0022 | 0.3940 ± 0.0337 | 3.2576 ± 0.0047 | 3.2569 ± 0.0046 | 0.3738 ± 0.0167 | 3.1795 ± 0.0018 | 3.1908 ± 0.0018 | 0.4368 ± 0.0151 | |
| 3.5225 ± 0.0496 | 3.5184 ± 0.0492 | 1.3792 ± 0.1961 | 3.2528 ± 0.0019 | 3.2518 ± 0.0017 | 0.3761 ± 0.0169 | 3.1798 ± 0.0022 | 3.1906 ± 0.0021 | 0.4449 ± 0.0172 |
We can observe that shallow contraction yields lower training and validation losses than deep contraction on both nanoGPT-S and nanoGPT-L. The only exception is on nanoGPT-M, but the same setting substantially raises nanoGPT-S validation loss to 3.5184. Deep contraction models also have higher gradient norms in 8/9 cases, suggesting less stable training. These results indicate that deep contraction does not yield a consistent improvement over shallow contraction. We will add the additional results to the revised PDF.
W2 & Q3. Reliability, uncertainty and efficiency analysis
Reliability of the gains. We extended the comparison beyond mHC-lite to all six baselines on nanoGPT-S/M/L and both datasets. The table below shows average validation loss gaps between cmHC and all baselines. We report , so a positive value favors cmHC. For each baseline, we also report the one-sided Wilcoxon signed-rank -value across the six settings, to assess whether the loss gaps consistently favor cmHC.
Table. Average validation loss gaps between each baseline and cmHC across three runs.
| Dataset | OpenWebText | FineWeb-Edu | cmHC wins | Wilcoxon -value | ||||
|---|---|---|---|---|---|---|---|---|
| nanoGPT size | S | M | L | S | M | L | ||
| Residual, | +0.0950 | +0.0823 | +0.0538 | +0.0647 | +0.0742 | +0.0453 | 6/6 | 0.016 (<0.05) |
| Residual, (no mixing) | +0.0056 | +0.0047 | +0.0005 | +0.0012 | +0.0010 | +0.0002 | 6/6 | 0.016 (<0.05) |
| HC | +0.0110 | +0.0141 | +0.0452 | +0.0052 | +0.0434 | +0.0546 | 6/6 | 0.016 (<0.05) |
| mHC | +0.0089 | +0.0018 | +0.0016 | -0.0005 | +0.0007 | +0.0021 | 5/6 | 0.031 (<0.05) |
| mHC-lite | +0.0073 | +0.0024 | +0.0045 | -0.0012 | +0.0013 | +0.0029 | 5/6 | 0.031 (<0.05) |
| KromHC | +0.0080 | +0.0062 | +0.0030 | +0.0001 | +0.0016 | -0.0015 | 5/6 | 0.047 (<0.05) |
Across both datasets and six baselines, cmHC obtains a lower average loss in (91.7%) comparisons. It outperforms all six baselines for nanoGPT-S and nanoGPT-M on OpenWebText, and for nanoGPT-M on FineWeb-Edu. At the threshold, the one-sided Wilcoxon test is significant for each baseline across the six settings, supporting the significance and robustness of the improvements. Furthermore, Figures 6 and 8 provide complementary evidence: despite fluctuations across training steps, the training-loss gaps of mHC-lite and other baselines generally remain above the cmHC zero-gap reference line (pink) in later training.
Per-seed comparisons. We repeat the nanoGPT-S/M experiments with seed 1338, in addition to default seed 1337 used for Table 1 from Section 4.1. We show results of cmHC and other strong baselines below. cmHC uses .
Table. nanoGPT-S/M results with seed 1338 (mean standard deviation for 3 replicate experiments). Best results are in bold.
| nanoGPT-S | nanoGPT-M | |||
|---|---|---|---|---|
| OpenWebText | Train loss | Val. loss | Train loss | Val. loss |
| mHC | 3.4677 ± 0.0013 | 3.4822 ± 0.0009 | 3.2674 ± 0.0021 | 3.2699 ± 0.0019 |
| mHC-lite | 3.4668 ± 0.0002 | 3.4808 ± 0.0003 | 3.2731 ± 0.0022 | 3.2753 ± 0.0018 |
| KromHC | 3.4672 ± 0.0027 | 3.4818 ± 0.0027 | 3.2677 ± 0.0016 | 3.2699 ± 0.0012 |
| cmHC | 3.4651 ± 0.0016 | 3.4795 ± 0.0014 | 3.2626 ± 0.0034 | 3.2653 ± 0.0041 |
| Fineweb-Edu | Train loss | Val. loss | Train loss | Val. loss |
| mHC | 3.4638 ± 0.0016 | 3.4774 ± 0.0017 | 3.2402 ± 0.0016 | 3.2531 ± 0.0018 |
| mHC-lite | 3.4644 ± 0.0020 | 3.4779 ± 0.0021 | 3.2399 ± 0.0022 | 3.2526 ± 0.0020 |
| KromHC | 3.4638 ± 0.0002 | 3.4775 ± 0.0004 | 3.2430 ± 0.0032 | 3.2556 ± 0.0030 |
| cmHC | 3.4641 ± 0.0007 | 3.4777 ± 0.0008 | 3.2385 ± 0.0003 | 3.2512 ± 0.0007 |
(To be continued in Part 3)
Response to Reviewer br6r (3/3)
W2 & Q3. Reliability, uncertainty and efficiency analysis (continued)
We can tell from the table above that with seed 1338, cmHC has the lowest mean training and validation losses among the evaluated baselines on nanoGPT-S/M + OpenWebText and nanoGPT-M + FineWeb-Edu. Its train/validation loss gaps to the strongest evaluated baseline are 0.0017/0.0013, 0.0048/0.0046 and 0.0014/0.0014, respectively. 5/6 of the gaps exceed cmHC's corresponding standard deviations (with the remaining one 0.0013 vs 0.0014). These are also the same three settings in which cmHC has the lowest mean validation loss in Table 1 with seed 1337, providing evidence that the advantage persists across seeds. On nanoGPT-S + FineWeb-Edu (the setting that mHC-lite outperforms cmHC by 0.0013/0.0012 in Table 1), all four methods perform on par: their mean training and validation losses span only 0.0006 and 0.0005, respectively.
mHC-lite at 300B. Among the baselines considered, mHC is the only one with a public implementation at 300B+ scale available for our training setup (i.e., the DeepSeek series), including the custom kernels and a large amount of other engineering components. Bringing mHC-lite to a comparably optimized 300B implementation would require substantial additional engineering and compute. Given the timeframe and available resources, we have to limit the 300B comparison to mHC.
Uncertainty estimates. Run-to-run variation in our repeated experiments arise from stochastic training in nanoGPT or other potential factors, and such variation is also present across the baselines despite the fixed seed. Thus, we empirically analyze such uncertainty by the reported standard deviations (SD) in Table 1. The table below shows that, averaged over the six settings in Table 1, cmHC has the second-lowest SD for both training and validation loss, better than strong baselines like mHC-lite and KromHC while only behind the standard residual connection with . This provides additional support to the robustness of cmHC's advantage in average loss gaps.
Table. Average loss SD over six settings from Table 1. The lowest is in bold and the second-lowest is .
| Residual, | Residual, | HC | mHC | mHC-lite | KromHC | cmHC | |
|---|---|---|---|---|---|---|---|
| Train loss SD | |||||||
| Val. loss SD |
Efficiency contribution. cmHC and mHC-lite can share the same operator implementation. Our fused kernels therefore benefit both methods, providing an operator-level performance advantage over the fused mHC implementation evaluated in the paper.
For end-to-end throughput, the operator-level experiments in the paper provide a controlled comparison of cmHC and mHC, isolating operator efficiency from system-level factors. End-to-end gains additionally depend on communication overlap, recomputation, and parallelization; lower operator overhead can create headroom for optimizing these aspects.
To provide complementary end-to-end evidence, we compare fused cmHC with fused mHC on our 6B model using 16 Ascend 910C NPUs, a sequence length of 4,096, a global batch size of 64, and forced expert load balancing to control for routing imbalance (which is common in benchmarking the training speed of MoE models). Under the configurations below, cmHC improves throughput by 0.6–34.2%, demonstrating that the operator-level gains can translate into end-to-end benefits, with the magnitude depending on the configuration.
Table. End-to-end training efficiency of fused mHC and cmHC. Step times are medians over 10 measured steps following five warm-up steps.
| Activation recomputation | Micro-batch size per NPU | mHC step time (s) | cmHC step time (s) | cmHC throughput gain$\uparrow$ |
|---|---|---|---|---|
| Full | 2 | 4.918 | 4.668 | +5.4% |
| Full | 4 | 4.363 | 3.810 | +14.5% |
| None | 1 | 5.537 | 5.502 | +0.6% |
| Selective | 2 | 6.668 | 4.969 | +34.2% |
Response to Reviewer br6r (1/3)
We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below.
We use the local linear approximation to isolate the stream-mixing mechanism, an approach also used in theoretical analyses of Transformer mixing and rank collapse. For example, Dovonon et al. [1] analyze a restricted Transformer update without the full LayerNorm, MLP, multi-head and layer-varying dynamics, then examine its predictions in full models. Dong et al. [2] first isolate pure-attention dynamics, then examine the effects of other Transformer components. Besides, Wang et al. [3] separate token representations into a mean component and its complement, analogous to our decomposition across residual streams.
We also provide some insights by carrying out relaxation to the local linear approximation and keeping the nonlinear branch contributions. Let the nonlinear branch at layer l be Fl(xl)=Alxl+ϵl where ϵl is the residual of the local linear approximation. Define rl=ΠAlJxl+Πϵl,
where ϵl is the residual of the local linear model. The perturbed recurrence gives
\leq (\lambda'_l+\ell_l) \Vert \widetilde{\mathbf{x}}_l \Vert _F+ \Vert \mathbf{r}_l \Vert _F.$$ Consequently, $$\Vert \widetilde{\mathbf{x}}_k \Vert _F \leq \left(\prod_{l=1}^{k-1}a_l\right) \Vert \widetilde{\mathbf{x}}_1 \Vert _F +\sum_{m=1}^{k-1} \left(\prod_{l=m+1}^{k-1}a_l\right) \Vert \mathbf{r}_m \Vert _F, \qquad a_l=\lambda'_l+\ell_l.$$ Theorem 3.2 is approximately recovered when the weighted additive term is negligible. Otherwise, the nonlinear network has a larger upper bound. However, we make two points here that (1) this nonlinear term mainly affects the existence of the upper bound. We can know from Figure 4 that in a full nonlinear network, the stream-difference norms do not show continued growth, and stronger contraction is associated with smaller norms. These observations imply **the weighted \$ \Vert \mathbf{r}_m \Vert \_F\$ term is not dominant in the inequality above**, and empirically support the extension of Theorem 3.2 to nonlinear networks. (2) The local linear approximation only **affects part of our over-mixing analysis**; Theorem 3.1 bounds the product of residual mixing matrices without the local linear approximation, so the core conclusion of the over-mixing analysis extends to the full nonlinear network. We present the linearized results as a mechanistic explanation of the additional "*butterfly effect*" observations that can amplify over-mixing. Regarding the entropy-increasing update: **we do not claim that Theorem 3.3 implies an entropy-increasing update direction**; this is likely not the actual situation either as you suggest. We only mention *normalized entropy* in Appendix E.3 for measuring how broadly the learned coefficients $\alpha_l$ are distributed over permutation vertices. Figure 10 reports higher entropy for cmHC than mHC-lite in the majority of layers; this is **observational evidence of a smoother coefficient distribution rather than a theoretical evidence of entropy-increasing update of model parameters**. We will clarify this distinction in the revised text. [1] Dovonon et al. Setting the record straight on Transformer oversmoothing. TMLR 2025 [2] Dong et al. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. ICML 2021 [3] Wang et al. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. ICLR 2022 > W1 & Q2. **Advantage of shallow contraction** To examine whether shallow contraction placement outperform deeper placement, we conduct an experiment by keeping $\lambda=0.97$ and the other experimental settings fixed, and applying contraction to the last $m$ layers, denoted by $K=-m$ for $m\in\lbrace1,2,10\rbrace$. Results are shown in the table below. (To be continued in Part 2)