- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ITO-HC, which modifies the residual mixer in Hyper-Connections by adding static, signed off-diagonal corrections to the identity matrix. It provides conditional stability analysis and experiments on small GPT-2-style models.
Critical strengths
- The modification is simple and straightforward to implement.
Critical weaknesses
-
Limited contribution. ITO-HC modifies only the residual mixer of the existing HC architecture. Given that iHC performs competitively without residual mixing, the paper does not demonstrate that this modification constitutes a substantial contribution. The theoretical analysis also largely follows standard perturbation bounds for near-identity matrix products.
-
The experiments are limited. HC demonstrated meaningful benefits at 7B scale, while mHC evaluated scaling up to 27B MoE models and demonstrated advantages over HC. Here, the experiments remain confined to tiny- to small-level models, which can not fully support the effectiveness of the proposed method.
-
The improvements over iHC are marginal at the conventional 4-stream setting. For example, 3.0461 v.s. 3.0473 on the L model. Moreover, on the L model with n=4, mHC’s training loss is approximately 0.01 higher than iHC’s, making it difficult to assess whether the experimental setup is convincing or not.
-
The main improvements appear at n=32 or 64, but such stream counts are not practical. Multi-stream residual states introduce substantial memory-access and communication overhead on LLM. Practical HC configurations typically use 4 streams, or at most 8. Results at very large n therefore provide limited evidence of practical value.
-
The analysis assumes conditions required for stability, but provides no guarantee that these conditions hold during training. Consequently, it does not establish training stability for the proposed method.
Questions for authors
- What substantial benefit does modifying the residual mixer provide over simply using iHC at n=4?
3 comments
follow-up
Thank you again for your thoughtful review. We have addressed your concerns by adding a 10,000-step Llama 2 7B comparison at the practical four-stream setting, reporting layerwise residual perturbations and cumulative-product diagnostics, and clarifying that our larger-width OpenWebText results are capacity studies rather than evidence that very large stream counts are practical. We also agree that the OpenWebText gain over iHC at is small, and that our single-run residual-path measurements do not establish a general training-stability or performance advantage. We have narrowed the claims accordingly. We would welcome further discussion and would be grateful to hear whether these clarifications address your main concern about ITO-HC’s practical benefit over iHC.
Thanks for the rebuttal. However, my concerns largely remain.
The authors response to W2, W3, W4 with a new Llama 2 7B experiment. However, it is unclear how many tokens are consumed under the 10,000 training steps. From figure 2 of the revised paper, we could observe that the training loss is around 3.5, which is overly high for a 7B model. This indicates that the model is far from converge, and it is likely that the trained tokens may < 7B, i.e., the model size. Therefore, this result can not support the claims and W2, W3, W4 remain.
For W1 and W5, the authors provide clarification which is helpful and appreciated. However, it does not fully resolve the contribution and stability concerns.
We thank the reviewer for the careful assessment. We address each critical weakness below. We added a 10,000-step Llama 2 7B pretraining.
1. Limited contribution and benefit over iHC
We agree that the architectural change is intentionally simple. The motivation is the gap between learned residual mixing and the identity baseline: in the settings reported in the manuscript, mHC and mHC-lite do not consistently improve on iHC, and the cited Qwen3 ablation post-ranks iHC above them (the source provides partial curves rather than numerical losses). This suggests that, in these settings, learning a doubly stochastic mixer can cost more than it gains. The open question motivating ITO-HC is whether a non-identity mixer can improve on identity by restoring useful cross-stream communication while keeping the cross-layer composite residual map well-conditioned.
The analysis further explains one possible cost of doubly stochastic mixing. Under exact doubly stochastic mixing with a uniform positive lower bound ε on every entry, the manuscript derives ∥GL−Jn∥2≤(1−nε)L, so stream-difference directions contract toward zero as depth increases. This is a conditional result: double stochasticity alone does not imply such decay, and mHC's finite Sinkhorn iterations only approximately satisfy the constraint. ITO-HC addresses the open question by keeping an exact unit diagonal and adding signed, bounded off-diagonal corrections. Its stability statement is likewise conditional: for ηl=∥Hlres−I∥2<1, the residual-product minimum singular value is bounded below by ∏l(1−ηl), with a corresponding upper bound controlled by exp(∑lηl). This is a sufficient guarantee under a controlled perturbation budget, not an unconditional property of the unit diagonal.
2. Experimental scale
We have added a separate Llama 2 7B pretraining experiment on streamed English C4, using 10,000 steps, sequence length 2,048, and n=4. It compares HC, iHC, mHC, mHC-lite, and ITO-HC, and reports training and validation trajectories through step 10,000. The manuscript reports that ITO-HC has lower loss than the existing methods in this experiment. This larger-backbone result directly addresses the concern that the evidence was limited to small GPT-2-style models, while we agree that one 7B run does not establish performance at 27B MoE scale.
3. Marginal OpenWebText gains at n=4
We agree that the OpenWebText difference between ITO-HC and iHC at n=4 is small; for the L model, the reported validation losses are 3.0810 and 3.0842, respectively. We do not present this gap as evidence of a large gain in that setting. The new Llama 2 7B experiment evaluates the practical four-stream configuration against four relevant alternatives (HC, iHC, mHC, and mHC-lite) on a substantially larger backbone; its loss curves and endpoint comparison are reported in the manuscript. This gives evidence beyond the small OpenWebText comparison, although the results remain single-run measurements.
4. Practicality of larger stream counts
We agree that results at n=32 or 64 alone would have limited practical relevance given the memory-access and communication costs of wide multi-stream states. The new Llama 2 7B comparison uses n=4, a conventional HC configuration, and thus tests ITO-HC in the setting most relevant to the reviewer’s concern. The OpenWebText width sweep is retained as a capacity and width study; our practical-scale evidence is the added four-stream Llama experiment, not a claim that very large stream counts are necessary.
5. Whether the stability conditions hold during training
We agree that the general perturbation bound is conditional and by itself does not show that its assumptions hold during training. We now report checkpoint measurements from the 7B Llama ITO-HC run. Across its 64 attention and feed-forward residual mixers, the largest learned γl is approximately 3.96×10−3, and the largest measured spectral perturbation ηl=∥Hlres−I∥2 is 0.00848 (block 0 attention). Thus every measured mixer satisfies ηl<1. Using this maximum, the manuscript gives the conservative residual-product guarantee σmin≥(1−0.00848)64≈0.580.
We directly measure the final residual-only products across all 64 sub-layers. The ITO-HC product has singular values (1.0214,1.0133,1.0082,0.9583) and condition number 1.0659. For mHC, the token-mean cumulative product has singular values (1.0000,0.4450,0.3436,0.1314) and condition number 7.608; the median singular values of the per-token products at final depth are (1.0000,0.7723,0.4181,0.1642). These are residual-path measurements for the recorded checkpoint, not a claim about full-network or optimizer stability, but they show that the stated perturbation condition holds in this run and provide direct evidence about residual-path conditioning.