acceptodds
Under review as a conference paper at ICLR 2027

oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions

36 views

Abstract

Hyper-Connections (HC) replace the single residual stream of a Transformer with parallel ones, mixing them at every layer with a learned n x n residual matrix. Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training. Manifold-constrained Hyper-Connections (mHC) address this by restricting the matrix to the doubly stochastic matrices. That caps the factor at one, so the mixing can no longer amplify any direction, but nothing bounds it from below. We prove that inside this set the mixing step can reduce the norm of the residual streams only by shrinking the differences between the streams, while their mean is left unchanged; and since this reduction compounds across layers, the streams grow increasingly alike and lose their diversity with depth. We therefore propose orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group , so that the mixing step can neither amplify nor attenuate the residual streams in any direction, which keeps training stable and no longer contracts the differences between streams. Specifically, at the four streams used by recent HC models we parameterize the group in closed form by a pair of unit quaternions, which adds no parameters over mHC, replaces the iterative projection with a fixed pattern of signed additions, and is faster to construct. We evaluate oHC on a broad set of downstream tasks against a range of HC variants. oHC outperforms all of them, and its gain over mHC widens as model size increases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Residual Connection, Hyper Connection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WpF1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Paper proposes Orthogonal Hyper connections for Transformer-Based architecture . For it uses a learned matrix at each layer, using doubly stochastic. That stops the streams from blowing up (σ_max = 1), but nothing stops them from shrinking (σ_min can be close to 0).

Paper goes on to define in some detail, how is this achieved mathematically and looks to substantiate the results both empirically and through the experimentation on different tasks such as MoE scaling and Dense-S comparisons to prove the efficacy of the proposed method.

Critical strengths

Strengths of the Paper:

  1. Paper provides a clear and succint analysis that explains the whole HC family in one picture.
  2. Paper provides an exact, cheap and well-behaved SO(4) parameterization.
  3. Discusses in some detail the mathematical intuition and empirical equations to bolster the idea of using quaternions
  4. Provides a relevant experiments to support the validity of the method proposed
Critical weaknesses

Weakness of the work:

  1. Stability and depth. Train unconstrained HC and much deeper models (e.g., 48+ layers) to show the stability benefit directly, and check whether the oHC gain grows with depth as the theory predicts.
  2. Iterating scale vs. baselines. Can you report iHC (and ideally JPmHC) at MoE-M? If iHC is close to oHC at M, the benefit of the SO(n) constraint itself is much less clear. 3.Seeds. Can you give at least 3 seeds for Dense-S (all variants in Table 3) and MoE-S, with standard deviations? Is the oHC vs c = 0 gap (2.2137 vs 2.2223) larger than the seed noise? 4.Benchmark regressions. Why does oHC lose on GSM8K and MATH at M and L while gaining on knowledge MC tasks? Are the results stable under log-likelihood scoring for the MC benchmarks?

Clarifications:

  1. Which model is in Fig. 1 and Fig. 8? Fig. 1 says MoE-S, but the composite 1.9 × 10⁻¹⁴ also appears in Fig. 8, which seems to use the 12-layer dense model. How many layers does MoE-S have? What is the “entry sublayer” excluded in Fig. 8? Also, which test set is used for HumanEval? Values like 1.85, 5.56 and 55.56 are multiples of 1/54, not 1/164.
Questions for authors

Have mostly put it under the Weakness section

5 comments

We thank you for recognizing the method's effectiveness. We address your questions point by point below.

Clarifications

Figures and layer counts. Figures 1 and 8 are both MoE-S (12 layers); the layer counts in the relevant captions and the appendix will be corrected in the revision.

Entry sublayer. "Sublayer" here means each attention / MLP module; MoE-S has 12 layers and therefore 24 sublayers. The phrase "entry sublayer excluded" in the Figure 8 caption is an erroneous leftover; every sublayer is in fact included in the computation. Thank you for catching this.

HumanEval / MBPP item counts. When execution-judging the generated outputs, silent defects (e.g., sandbox errors) left some items without a judgeable result, so they were dropped from the denominator: HumanEval used 162 items (full set 164, 2 dropped) and MBPP used 370 (full set 378, 8 dropped). We re-evaluated on the full sets (164 / 378); the relative ordering across arms is unchanged (overall oHC ≥ mHC, RC lowest). Results:

Tab 3-1: HumanEval (%, n: 162→164)

scaleRCmHCoHC
S1.85 → 1.228.64 → 7.935.56 → 4.88
M17.90 → 17.6822.70 → 23.7825.93 → 25.00
L46.30 → 47.5650.00 → 51.2255.56 → 56.71

Tab 3-2: MBPP (%, n: 370→378)

scaleRCmHCoHC
S17.57 → 15.8726.76 → 26.4625.68 → 26.46
M38.38 → 37.8349.19 → 48.6852.43 → 54.23
L62.97 → 62.7064.59 → 65.3462.70 → 64.29

W1: Stability and depth

Unconstrained HC. We reproduced HC [1]. On Dense-S we obtain RC < HC ≈ mHC < oHC; at small scale HC's stability issue is not severe, and the gain of mHC's [2] manifold constraint over HC is small; its value lies mainly in stability at large scale/depth, as shown in [2]. oHC, by contrast, already shows a clear improvement at small scale.

Tab 3-3: HC train loss (Dense-S)

stepRCHCmHCiHCoHCoHC−HC
10002.52272.50012.49622.49472.4858−0.0144
20002.33332.31242.30902.30742.2993−0.0130
30002.29332.27312.27062.26962.2593−0.0138
40002.22762.20682.20592.20492.1944−0.0124
final2.24702.22612.22522.22422.2137−0.0124

Stability benefit. The instability of unconstrained HC at larger scale/depth is thoroughly discussed in the mHC paper [2]: HC is stable at ~1B, but at larger scale/depth (e.g., 27B, 30 layers) it exhibits training-loss spikes and residual-stream-norm blow-up, and HC's advantage over the baseline narrows with depth, even falling below plain residual connections at great depth [2,3]. Given compute and time constraints, we cite these conclusions here rather than re-establishing them.

oHC's gain. Figure 5 partly answers this: as model size/depth grows, oHC's accuracy gain over mHC increases.

W2: Scaling ladder and baselines

Given time and compute constraints, we could not train iHC and JPmHC to the full step count on MoE-M. But we trained the first 1000 steps against mHC and oHC, which already reflects the trend.

Tab 3-4: MoE-M (L18) train loss

itermHCoHCiHCJPmHC
2003.35393.31763.34583.3395
5002.20972.18532.20312.1942
10001.98701.96811.98361.9717

Early on MoE-M (step 1000) the conclusion agrees with Dense-S: the ordering is oHC < JPmHC < iHC < mHC. That is, oHC is best; iHC is only slightly better than mHC; and JPmHC [4] is slightly worse than oHC.

W3: Random seeds

On Dense-S we extended all five variants in the paper's Table 3 (mHC, iHC, oHC(c=0), JPmHC, oHC) to 3 random seeds each.

Tab 3-5: Per-variant train loss over 3 seeds (Dense-S)

variantmeanstd (n=3)oHC − *p
oHC2.21280.0025——
JPmHC2.21470.0028−0.00190.36
oHC(c=0)2.22010.0021−0.00730.0042
iHC2.22110.0031−0.00830.0018
mHC2.22380.0012−0.01100.0002

oHC's improvement over mHC, iHC, and oHC(c=0) all clearly exceeds the seed noise; relative to JPmHC [4] it is slightly better.

W4: Benchmark regressions

We attribute this to oHC's insufficient math-format-following ability. First, we rule out a deficit in math knowledge: on multiple-choice math datasets (mmlu-math / mmlu_pro-math; see Tab 3-6) oHC performs better at M/L; moreover, computing BPB on the gold answers of gsm8k and math still gives oHC ≤ mHC (lower is better); and on other open-ended QA oHC is not generally worse. Hence we attribute the regression to insufficient math-format-following.

In addition, for the multiple-choice evaluation format we also tested log-likelihood scoring: on M and L oHC is still better than mHC, and the gap widens with scaling: M +0.06, L +0.58.

Tab 3-6: MC math accuracy (%)

scalebenchmarkoHCmHCΔ(oHC−mHC)
M (L18)mmlu-math38.3332.67+5.66
M (L18)mmlu_pro-math13.7711.40+2.37
L (L24)mmlu-math51.1845.75+5.43
L (L24)mmlu_pro-math28.5025.46+3.04

Tab 3-7: gsm8k / math BPB (lower is better)

scalebenchmarkoHCmHCΔ(oHC−mHC)
M (L18)gsm8k0.32410.3287−0.0046
M (L18)math0.39370.3980−0.0043
L (L24)gsm8k0.27470.2862−0.0115
L (L24)math0.34600.3459+0.0001

References

[1] Zhu et al. Hyper-Connections.

[2] Xie et al. mHC: Manifold-Constrained Hyper-Connections.

[3] Liu et al. Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections (sHC).

[4] Sengupta et al. JPmHC: Dynamical Isometry via Orthogonal Hyper-Connections. arXiv:2602.18308.

Dear Reviewer WpF1,

Thank you again for recognizing the effectiveness of our method and for your careful, concrete feedback. We have uploaded a revised PDF with the changes highlighted in blue: we have added multi-seed results with standard deviations, and corrected the Figure 8 caption and stated the model layer counts. We are especially grateful for your help in making the paper more rigorous. We greatly appreciate your time and consideration, and please feel free to share any further thoughts you may have.

Best regards,

The Authors

Reviewer_F7pXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes orthogonal Hyper-Connections (oHC), which impose an orthogonality constraint on the residual mixing matrix of Hyper-Connections to promote training stability and preserve inter-stream diversity. For the four-stream setting, the mixing matrix is parameterized using a pair of unit quaternions. The authors evaluate the method on language models at multiple scales and report improvements in average downstream task performance over existing methods.

Critical strengths
  • In the four-stream setting, oHC achieves lower inter-stream cosine similarity than the evaluated baselines and improves average downstream performance over mHC. These performance gains are consistently observed across multiple model scales.
  • Parameterizing four-stream residual mixing using a pair of unit quaternions is an elegant design that constructs norm-preserving mixing matrices without iterative procedures or matrix inversion. It accelerates mixing-matrix construction without increasing the parameter count relative to mHC.
  • The method is simple and computationally efficient. The quaternion-pair construction requires neither iteration nor matrix inversion, initializes with H=IH=I, and introduces no additional parameters relative to mHC.
Critical weaknesses
  • The authors use lower inter-stream cosine similarity as evidence of increased diversity. However, cosine similarity measures directional similarity between representations and does not directly establish that individual streams retain complementary, task-relevant information. Further investigation would help clarify how the observed reduction in similarity relates to informational diversity and performance improvements.

  • Although the theoretical analysis covers general stream counts, the proposed quaternion construction and empirical evaluation are limited to four streams. It therefore remains unclear whether the benefits of orthogonal mixing extend to other stream counts and how the proposed construction can be adapted to those settings.

Questions for authors
  • The RQ4 ablation supports the effectiveness of the mean-to-difference pathway. However, is there additional evidence that the reduction in cosine similarity reflects complementary, task-relevant information rather than merely changes in feature directions? For example, have you analyzed the performance impact of removing individual streams?
  • How would you extend the method to n≠4n\neq4? Can the mechanism identified by the theoretical analysis and the computational advantages be retained when the quaternion construction is not directly applicable? For n=2,3n=2,3, have you considered zero-padding the unused input components and constraining the quaternion pair to preserve the corresponding subspace? For example, when n=3n=3, conjugation of a pure imaginary quaternion, qxqˉqx\bar q, provides a possible construction. Would you expect such constructions to retain the benefits of the proposed orthogonal mixing?
  • The relationship to existing orthogonal-matrix parameterizations, such as expRNN, scoRNN, and Householder-based approaches, as well as prior work on quaternion rotation representations, is insufficiently discussed. Please cite the relevant literature and clarify the novelty and advantages of the proposed construction relative to existing parameterizations.
2 comments

We especially thank you for the constructive suggestions. We address your comments and questions below.

W1 & Q1: Does a lower cosine correspond to complementary, task-relevant information?

Following your suggestion, we added two tests on the trained Dense-S.

Stream removal. At inference we replace one stream with the mean of the four streams and measure the increase in next-token loss. oHC's increase is far larger than mHC's and iHC's, and far larger than a random perturbation of the same size; if we keep only a single stream for readout and discard the other three, oHC's cost is an order of magnitude higher still (Tab 2-1). Thus, as inter-stream similarity decreases, each stream carries more information of unique predictive value.

Predictability. We linearly reconstruct one stream from the other three; a larger 1−R² means the stream is less linearly recoverable from the others (less redundant). mHC is almost fully recoverable, oHC far from recoverable (Tab 2-2). Thus, as inter-stream similarity decreases, the complementarity of inter-stream information increases.

Tab 2-1: Stream removal (nats/token; mean ± std, n=3 seeds)

armremoval Δlossrandom-perturbation controlsingle-stream readout
oHC0.274 ± 0.0560.064 ± 0.0105.11 ± 0.74
iHC0.035 ± 0.0070.0072 ± 0.00060.65 ± 0.10
c=00.029 ± 0.0140.0065 ± 0.00150.52 ± 0.20
mHC0.008 ± 0.0010.0022 ± 0.00020.13 ± 0.01

Tab 2-2: Linear predictability 1−R² (larger = more complementary; mean ± std, n=3 seeds)

arm1−R²
oHC0.449 ± 0.027
iHC0.201 ± 0.005
c=00.185 ± 0.009
mHC0.022 ± 0.006

Notes:

  1. Removal: replacing stream k with the four-stream mean m̄ equals a move of ‖x_k − m̄‖ along the "deviation-from-mean" direction.
  2. Random control: adding to stream k a random-direction vector of the same norm ‖x_k − m̄‖, i.e., the same move distance as removal.
  3. Single-stream readout: reading out from one stream while discarding the other three, averaged over the four choices.

In summary, as the inter-stream cosine decreases (oHC lowest), removing a stream costs more and the streams become harder to reconstruct from one another, showing that a lower cosine corresponds to more complementary, more diverse inter-stream information, not merely a change of feature direction.

W2 & Q2: Extension to n≠4

We genuinely appreciate your concrete and thoughtful suggestions for the n=2,3 constructions, especially the pure-imaginary quaternion conjugation for n=3. The matrix exponential of a skew-symmetric matrix is an exact orthogonal construction that works for any n, and the quaternion is its efficient closed-form special case at n=4; so for any n≠4 we can use the matrix-exponential construction directly, and when n is a multiple of 4 we can use the block-quaternion construction. As for x↦qx q̄ (pure-imaginary quaternion), it is precisely a closed-form, inverse-free SO(3) that is the identity at zero logits, and is the r=q slice of the four-stream map, a viable construction.

We trained mHC / matrix-exponential / block-quaternion at n=8:

Tab 2-3

method (n=8)train lossper-step time vs mHC
mHC2.2197—
matrix exponential2.1987−2.8%
block quaternion (ours)2.2002−11.8%

The benefit of orthogonal mixing persists at n≠4.

Q3: Related work

Thank you for pointing this out. The revision will add the missing citations [1,2,3,4,5] and clarify our relationship to existing orthogonal-matrix parameterization methods and to prior work on quaternion rotation representations.

First, a comparison with other orthogonal constructions:

Tab 2-4

constructioncovers all of SO(4)?eager µs (fwd+bwd)build memory @256kpotential issue
matrix exponentialyes1070803 MBslightly slow
SVD / polaryes3749195 MBslowest, and needs one sign flip to represent SO(4)
Cayleyno400150 MBcannot represent 180° planar rotations
Householderno432204 MBsome rotations are unreachable
Givensyes465159 MBnumerically unstable at large angles
quaternion (ours)yes82108 MB—

Here the matrix exponential is the general form of the quaternion and extends to arbitrary n; the quaternion is its efficient closed-form solution at n=4. expRNN [1] uses the matrix exponential of a skew matrix; scoRNN [3] uses Cayley; the Householder-based parametrization [4] constrains the matrix with a product of Householder reflections (the Householder row in Tab 2-4). In addition, Cayley requires a matrix inverse: being exact requires solve/LU (slow), whereas a pure-GEMM implementation can only be iterative and approximate.

Relative to existing parameterization work (mHC [6] / sHC [7]), our oHC guarantees non-decaying inter-stream norm via orthogonality; relative to iHC, oHC's H_res lets mean energy flow into difference energy, yielding larger inter-stream differences and better loss; relative to JPmHC [8], our quaternion is an exact closed-form solution (JPmHC is an iterative approximation) and is slightly better in train loss.

References

[1] Lezcano-Casado & Martínez-Rubio. Cheap Orthogonal Constraints in Neural Networks (expRNN). ICML 2019.

[2] Arjovsky et al. Unitary Evolution Recurrent Neural Networks. 2016.

[3] Maduranga et al. Complex Unitary Recurrent Neural Networks Using Scaled Cayley Transform (scoRNN). 2019.

[4] Mhammedi et al. Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections. ICML 2017.

[5] Zhou et al. On the Continuity of Rotation Representations. 2019.

[6] Xie et al. mHC: Manifold-Constrained Hyper-Connections.

[7] Liu et al. Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections (sHC).

[8] Sengupta et al. JPmHC: Dynamical Isometry via Orthogonal Hyper-Connections. arXiv:2602.18308.

Reviewer_93MMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes oHC, which replaces the doubly stochastic matrices in manifold-constrained hyperconnection (mHC) with SO(4) matrices parameterized in the quaternion form. The paper theoretically analyzes the advantage of SO(n) residual mixing over mHC in terms of norm and difference gains, showing that mHC tends to decrease the difference between the streams. This mechanism is verified with ablation RQ4. The advantage of orthogonal hyperconnection is verified up to 30B model.

Critical strengths

While this work is not the first to propose using SO(n) in as the residual for hyperconnection, to my knowledge it is the first to validate its effectiveness in LLM pretraining. Previous work, JPmHC, only verified its effectiveness only in ARC-AGI tasks with 7M parameter model. The analysis makes sense intuitively, and the hypothesis is partially supported with the inter-stream cosine similarity measure in table 3. Therefore, while oHC is not the first to propose SO(n) for hyperconnection, I still find its contribution novel and meaningful.

Critical weaknesses

[Concern 1] The advantage of quaternion parameterization is unclear. The authors argue that oHC is 2.9x faster than JPmHC. However, both are microsecond-level. Thus, optimizing the SO(n) parameterization will have very tiny impact on the overall model throughput. The authors also argue that JPmHC is not exact. However, it is unclear whether this approximation has really undermined the performance. It could be due to different initialization or optimization dynamics from different parameterization. More importantly, there exists a massive body of literature on paramterizing orthogonal matrices for neural networks, e.g., SVD, matrix exponential, Householder etc. These are also exact, and unlike quaternion parameterization, these generalize to arbitrary number of dimensions. The paper does not compare against these alternatives within the same framework, nor does it report any experiment with n≠4n \neq 4. All these methods are trivially cheap for SO(4), which is just a 4x4 matrix. The computational cost is only relevant in much higher dimensions. Therefore, I disagree that the quaternion parameterization is a significant contribution.

[Concern 2] The theoretical insight is limited, and the writing is unnecessarily complicated compared to what it actually delivers. Section 4.1 discusses the norm gain in terms of singular values, which I believe is unnecessarily complicated. The conclusion is already well-known: orthogonal matrices preserve norm by definition, and doubly stochastic matrices are contractive. For the difference-gain, JPmHC already provides similar intuition.

For instance, if AnA_n is bistochastic with eigenvalues {1,λ2,⋯ ,λn}\{1, \lambda_2, \cdots , \lambda_n\} where ∣λi∣<1|λ_i| < 1 for i≥2i ≥ 2, the scalar theory sees only the average behavior, not the sector-wise contraction that drives vanishing gradients.

The writing could be improved if the authors move all the less important mathematical details to the appendix and leave only the core lemma, theorem or propositions.

Questions for authors

The related section discusses that JPmHC's hypothesis is wrong. However, I could not find an experiment that supports this claim. Could the authors clarify on this?

JPmHC (Sengupta et al., 2026) likewise notes the attenuation, argues that it leads to vanishing gradients, and proposes replacing the constraint with orthogonality. In our measurements, however, the gradient magnitudes barely change; what vanishes is the difference between the gradients the streams receive.

6 comments

We thank you for clearly recognizing the novelty and significance of the work. We address your concerns and questions below.

W1: Value of the quaternion construction

Thank you for helping us better position our contribution. We agree that the quaternion pair is only one of several exact orthogonal constructions; but we believe its value rests on two points: (a) it has no theoretical shortcoming; (b) at n=4 (the stream count commonly used by industrial HC [1] models: DeepSeek-V4 [2], Hunyuan Hy4 [3], and Qwen3.8-Flash-Next [4] all use four streams) it is the simplest exact construction: closed-form, purely element-wise arithmetic, cheapest in the forward pass and also simplest in the backward pass (no backprop through any decomposition / linear solve / matrix function). It is therefore worth adopting.

1. Exact vs. approximate

To analyze the effect of exact vs. approximate orthogonality more cleanly, we raise JPmHC's [5] own Cayley iteration from s=2 to s=8 (s=8 can be regarded as approximately exact orthogonal), ruling out "the difference comes from a different parameterization or dynamics"; we list our quaternion method (oHC) for reference.

Tab 1-1

metrics=2 (approx.)s=8 (≈exact)oHC (quaternion)
composite σ0.941.001.00
inter-stream similarity0.510.400.46
train loss2.21772.21522.2137

Here composite σ is the product over the 24 sublayers of each layer's minimum stream-norm gain. As orthogonality approaches exactness, composite σ clearly rises to 1, and inter-stream similarity and train loss improve slightly; this reflects the mechanistic advantage of exact orthogonality.

2. Exact orthogonal methods

Following your recommendation, we add a comparison with other exact orthogonal constructions:

Tab 1-2

constructioncovers all of SO(4)?eager µs (fwd+bwd)build memory @256kpotential issue
matrix exponentialyes1070803 MBslightly slow
SVD / polaryes3749195 MBslowest, and needs one sign flip to represent SO(4)
Cayleyno400150 MBcannot represent 180° planar rotations
Householderno432204 MBsome rotations are unreachable
Givensyes465159 MBnumerically unstable at large angles
quaternion (ours)yes82108 MB—

Here the matrix exponential is the general form of the quaternion and can extend to arbitrary n; the quaternion is its efficient closed-form solution at n=4. In addition, Cayley requires a matrix inverse: being exact requires solve/LU (slow), whereas a pure-GEMM implementation can only be iterative and approximate.

From Tab 1-1, exact JPmHC's loss and similarity are close to the quaternion's; the exact-orthogonal methods thus behave consistently, and as models scale up the quaternion may hold further advantages in training stability and dynamics.

3. Supplementary experiments at n≠4

Our method extends to arbitrary n in two ways: (1) the quaternion's general form, the matrix exponential; (2) a block-structured quaternion, splitting the n streams into several n=4 blocks, each rotated, plus one inter-block rotation (R⊗I₄); method 2's loss is close to the matrix exponential's while being faster. We ran experiments at n=8:

Tab 1-3

method (n=8)train lossper-step time vs mHC
mHC2.2197—
matrix exponential2.1987−2.8%
block quaternion (ours)2.2002−11.8%

Specifically, as the number of streams increases from 4 to 8, oHC's gain is larger than mHC's, where oHC at n=8 is the block-quaternion construction (Tab 1-4):

Tab 1-4

n=4n=8Δ(8−4)
mHC2.22522.2197−0.0055
oHC2.21372.2002−0.0135
gap (mHC−oHC)0.01150.0195+0.0080

In summary, we agree that presenting the quaternion alone as a contribution is somewhat thin. We revise Contribution 2 to: the first validation and analysis of oHC on LLMs, together with one efficient exact solution at n=4 (the quaternion). We will adjust the wording accordingly in the revision.

W2: Theoretical insight and §4.1 writing

Thank you for the writing suggestion.

On writing. We agree that §4.1 is currently too dense. The revision will streamline it, moving secondary derivations to the appendix so the main line is clearer and easier to read.

On the theoretical contribution.

(a) We acknowledge that JPmHC [5] already provides a spectral-side intuition related to difference-gain, but that analysis is indirect. JPmHC notes that for a doubly-stochastic mixer the non-mean eigen-sectors contract (eigenvalues |λᵢ|<1, i≥2) while the mean sector (λ=1) is preserved, a spectral-level characterization of "the inter-stream difference directions are compressed," which we will cite and credit in the revision. But this characterization is indirect: it acts on the asymptotic singular-value density of the end-to-end Jacobian (operator-valued free probability), stating its conclusion in terms of "eigen-sectors" and "spectral collapse," without directly answering "which part of the stream information the contraction is charged to," and it reads rather abstractly.

(b) We give a more precise and direct result: for mHC [6], the contraction comes entirely from the inter-stream differences. In the (mean, inter-stream-difference) basis we exactly decompose H_res and prove that, within the Birkhoff polytope, any contraction of the residual-stream norm by the mixing step can only be achieved by shrinking the inter-stream differences, while the mean component of the streams is left exactly unchanged.

(c) The block analysis of H_res explains a phenomenon JPmHC cannot: why oHC has better diversity than iHC. Both oHC and iHC avoid collapsing the inter-stream differences; spectral analysis alone (both are |λ|=1, dynamical isometry) cannot distinguish them, and hence cannot explain oHC's diversity advantage. In one unified framework (mean→mean, the mean↔difference cross channels, difference→difference) we precisely locate each HC variant: both iHC and oHC avoid stream contraction, but oHC's decomposition has a nonzero mean→difference injection channel (c≠0), letting it convert mean energy into difference energy and actively create inter-stream differences, so its inter-stream diversity exceeds iHC's.

(d) Our analysis is more precise and does not rely on assumptions inconsistent with the actual model. JPmHC's spectral analysis relies on mean-field isotropy and asymptotic assumptions, and models the sublayer as a full-width isotropic random matrix, which does not match the actual hyper-connection sublayer structure; our decomposition is an exact algebraic identity that holds per token, makes none of those assumptions, and applies directly to the actual model.

(e) We validate the mechanism experimentally. We systematically report the inter-stream similarity of each HC variant, matching the theoretical analysis; and we perform a causal ablation on the trained oHC: after turning off the mean→difference injection channel, oHC's inter-stream similarity and its "remove-one-stream" cost both fall back to iHC's level, confirming that the channel in (c) is indeed the source of oHC's diversity advantage.

Q1: The gradient claim in related work

Thank you for pointing this out. Our earlier wording was indeed not clear enough; we clarify below and add experimental support.

(1) JPmHC's [5] vanishing-gradient result is self-consistently derived under its theoretical assumption, but that assumption does not match the actual model. JPmHC does argue that a doubly-stochastic mixer leads to vanishing gradients, but this derivation rests on its modeling of the sublayer: it models the read-compute-write path as a full-width N×N isotropic operator, i.e., the n streams pass through the module at full rank. The actual hyper-connection sublayer is n→1→n: h_pre first aggregates the n streams into 1, the module computes once on that single stream, and h_post then expands back to n. Hence JPmHC's vanishing gradient is self-consistent within its own full-width model, but does not correspond to the actual sublayer structure.

(2) On the actual model, the inter-stream gradient difference shrinks with inter-stream similarity. On the trained model, the gradient magnitude is of the same order across variants and does not decay (the cumulative amplification of the gradient RMS from the last layer to the first is similar across variants: mHC 285×, iHC 379×, oHC 247×, i.e., the backward pass does not vanish); what varies with inter-stream similarity is the difference between the gradients the streams receive. Using the pairwise cosine between streams to measure both forward (stream state) and backward (gradient) similarity, the per-variant results are (Dense-S):

Tab 1-5

variantforward inter-stream cosgradient inter-stream cos
mHC0.820.64
iHC0.670.63
oHC0.380.42

Across variants, the more similar the streams, the more similar their gradients (i.e., the smaller the gradient difference). Extrapolating this trend to the limit where the streams fully coincide, the inter-stream gradient difference vanishes. We will adjust this wording in the revision.

References

[1] Zhu et al. Hyper-Connections.

[2] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence.

[3] Tencent Hunyuan. Hy4.

[4] Qwen Team. Qwen3.8-Flash-Next. https://qwen.ai/blog?id=qwen3.8-flash-next

[5] Sengupta et al. JPmHC: Dynamical Isometry via Orthogonal Hyper-Connections. arXiv:2602.18308.

[6] Xie et al. mHC: Manifold-Constrained Hyper-Connections.

Comments on the paper

1 comment

Dear Reviewer F7pX,

We thank you for the careful reading and for maintaining a positive evaluation. Your suggestions were especially constructive, and we appreciate your time and engagement.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.