- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Paper proposes Orthogonal Hyper connections for Transformer-Based architecture . For it uses a learned matrix at each layer, using doubly stochastic. That stops the streams from blowing up (σ_max = 1), but nothing stops them from shrinking (σ_min can be close to 0).
Paper goes on to define in some detail, how is this achieved mathematically and looks to substantiate the results both empirically and through the experimentation on different tasks such as MoE scaling and Dense-S comparisons to prove the efficacy of the proposed method.
Critical strengths
Strengths of the Paper:
- Paper provides a clear and succint analysis that explains the whole HC family in one picture.
- Paper provides an exact, cheap and well-behaved SO(4) parameterization.
- Discusses in some detail the mathematical intuition and empirical equations to bolster the idea of using quaternions
- Provides a relevant experiments to support the validity of the method proposed
Critical weaknesses
Weakness of the work:
- Stability and depth. Train unconstrained HC and much deeper models (e.g., 48+ layers) to show the stability benefit directly, and check whether the oHC gain grows with depth as the theory predicts.
- Iterating scale vs. baselines. Can you report iHC (and ideally JPmHC) at MoE-M? If iHC is close to oHC at M, the benefit of the SO(n) constraint itself is much less clear. 3.Seeds. Can you give at least 3 seeds for Dense-S (all variants in Table 3) and MoE-S, with standard deviations? Is the oHC vs c = 0 gap (2.2137 vs 2.2223) larger than the seed noise? 4.Benchmark regressions. Why does oHC lose on GSM8K and MATH at M and L while gaining on knowledge MC tasks? Are the results stable under log-likelihood scoring for the MC benchmarks?
Clarifications:
- Which model is in Fig. 1 and Fig. 8? Fig. 1 says MoE-S, but the composite 1.9 × 10⁻¹⁴ also appears in Fig. 8, which seems to use the 12-layer dense model. How many layers does MoE-S have? What is the “entry sublayer” excluded in Fig. 8? Also, which test set is used for HumanEval? Values like 1.85, 5.56 and 55.56 are multiples of 1/54, not 1/164.
Questions for authors
Have mostly put it under the Weakness section
5 comments
W3: Random seeds
On Dense-S we extended all five variants in the paper's Table 3 (mHC, iHC, oHC(c=0), JPmHC, oHC) to 3 random seeds each.
Tab 3-5: Per-variant train loss over 3 seeds (Dense-S)
| variant | mean | std (n=3) | oHC − * | p |
|---|---|---|---|---|
| oHC | 2.2128 | 0.0025 | — | — |
| JPmHC | 2.2147 | 0.0028 | −0.0019 | 0.36 |
| oHC(c=0) | 2.2201 | 0.0021 | −0.0073 | 0.0042 |
| iHC | 2.2211 | 0.0031 | −0.0083 | 0.0018 |
| mHC | 2.2238 | 0.0012 | −0.0110 | 0.0002 |
oHC's improvement over mHC, iHC, and oHC(c=0) all clearly exceeds the seed noise; relative to JPmHC [4] it is slightly better.
W4: Benchmark regressions
We attribute this to oHC's insufficient math-format-following ability. First, we rule out a deficit in math knowledge: on multiple-choice math datasets (mmlu-math / mmlu_pro-math; see Tab 3-6) oHC performs better at M/L; moreover, computing BPB on the gold answers of gsm8k and math still gives oHC ≤ mHC (lower is better); and on other open-ended QA oHC is not generally worse. Hence we attribute the regression to insufficient math-format-following.
In addition, for the multiple-choice evaluation format we also tested log-likelihood scoring: on M and L oHC is still better than mHC, and the gap widens with scaling: M +0.06, L +0.58.
Tab 3-6: MC math accuracy (%)
| scale | benchmark | oHC | mHC | Δ(oHC−mHC) |
|---|---|---|---|---|
| M (L18) | mmlu-math | 38.33 | 32.67 | +5.66 |
| M (L18) | mmlu_pro-math | 13.77 | 11.40 | +2.37 |
| L (L24) | mmlu-math | 51.18 | 45.75 | +5.43 |
| L (L24) | mmlu_pro-math | 28.50 | 25.46 | +3.04 |
Tab 3-7: gsm8k / math BPB (lower is better)
| scale | benchmark | oHC | mHC | Δ(oHC−mHC) |
|---|---|---|---|---|
| M (L18) | gsm8k | 0.3241 | 0.3287 | −0.0046 |
| M (L18) | math | 0.3937 | 0.3980 | −0.0043 |
| L (L24) | gsm8k | 0.2747 | 0.2862 | −0.0115 |
| L (L24) | math | 0.3460 | 0.3459 | +0.0001 |
References
[1] Zhu et al. Hyper-Connections.
[2] Xie et al. mHC: Manifold-Constrained Hyper-Connections.
[3] Liu et al. Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections (sHC).
[4] Sengupta et al. JPmHC: Dynamical Isometry via Orthogonal Hyper-Connections. arXiv:2602.18308.
Dear Reviewer WpF1,
Thank you again for recognizing the effectiveness of our method and for your careful, concrete feedback. We have uploaded a revised PDF with the changes highlighted in blue: we have added multi-seed results with standard deviations, and corrected the Figure 8 caption and stated the model layer counts. We are especially grateful for your help in making the paper more rigorous. We greatly appreciate your time and consideration, and please feel free to share any further thoughts you may have.
Best regards,
The Authors
We thank you for recognizing the method's effectiveness. We address your questions point by point below.
Clarifications
Figures and layer counts. Figures 1 and 8 are both MoE-S (12 layers); the layer counts in the relevant captions and the appendix will be corrected in the revision.
Entry sublayer. "Sublayer" here means each attention / MLP module; MoE-S has 12 layers and therefore 24 sublayers. The phrase "entry sublayer excluded" in the Figure 8 caption is an erroneous leftover; every sublayer is in fact included in the computation. Thank you for catching this.
HumanEval / MBPP item counts. When execution-judging the generated outputs, silent defects (e.g., sandbox errors) left some items without a judgeable result, so they were dropped from the denominator: HumanEval used 162 items (full set 164, 2 dropped) and MBPP used 370 (full set 378, 8 dropped). We re-evaluated on the full sets (164 / 378); the relative ordering across arms is unchanged (overall oHC ≥ mHC, RC lowest). Results:
Tab 3-1: HumanEval (%, n: 162→164)
Tab 3-2: MBPP (%, n: 370→378)
W1: Stability and depth
Unconstrained HC. We reproduced HC [1]. On Dense-S we obtain RC < HC ≈ mHC < oHC; at small scale HC's stability issue is not severe, and the gain of mHC's [2] manifold constraint over HC is small; its value lies mainly in stability at large scale/depth, as shown in [2]. oHC, by contrast, already shows a clear improvement at small scale.
Tab 3-3: HC train loss (Dense-S)
Stability benefit. The instability of unconstrained HC at larger scale/depth is thoroughly discussed in the mHC paper [2]: HC is stable at ~1B, but at larger scale/depth (e.g., 27B, 30 layers) it exhibits training-loss spikes and residual-stream-norm blow-up, and HC's advantage over the baseline narrows with depth, even falling below plain residual connections at great depth [2,3]. Given compute and time constraints, we cite these conclusions here rather than re-establishing them.
oHC's gain. Figure 5 partly answers this: as model size/depth grows, oHC's accuracy gain over mHC increases.
W2: Scaling ladder and baselines
Given time and compute constraints, we could not train iHC and JPmHC to the full step count on MoE-M. But we trained the first 1000 steps against mHC and oHC, which already reflects the trend.
Tab 3-4: MoE-M (L18) train loss
Early on MoE-M (step 1000) the conclusion agrees with Dense-S: the ordering is oHC < JPmHC < iHC < mHC. That is, oHC is best; iHC is only slightly better than mHC; and JPmHC [4] is slightly worse than oHC.