acceptodds
Under review as a conference paper at ICLR 2027

ITO-HC: Identity-Centered Tanh Off-Diagonal Hyper-Connections

Abstract

Hyper-Connections (HC) widen the residual state and learn dynamic read, write, and stream-mixing maps. Manifold-Constrained Hyper-Connections (mHC) stabilize these maps by constraining the residual mixer to the Birkhoff polytope, but repeated nonnegative mixing can suppress stream-difference directions and finite Sinkhorn iterations introduce projection error. We propose ITO-HC, an identity-centered, input-independent residual mixer that adds bounded, signed off-diagonal corrections while preserving an exact unit diagonal. ITO-HC retains the mHC-style dynamic pre/post routers and nonlinear Transformer branches, combining an explicit residual highway with controlled cross-stream interactions. We derive a layer-attention formulation that exposes the low-rank dense routing structure of the identity-residual baseline, and we establish conditional perturbation and spectral stability bounds for ITO-HC. Experiments on OpenWebText with GPT-2-style models compare ITO-HC with identity, mHC, mHC-lite, and a single-stream residual baseline across multiple stream widths. The results support the proposed stability–capacity trade-off while highlighting the need for further scale and routing-diagnostic studies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hyper-Connections, pretraining, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bS3WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ITO-HC, which modifies the residual mixer in Hyper-Connections by adding static, signed off-diagonal corrections to the identity matrix. It provides conditional stability analysis and experiments on small GPT-2-style models.

Critical strengths
  1. The modification is simple and straightforward to implement.
Critical weaknesses
  1. Limited contribution. ITO-HC modifies only the residual mixer of the existing HC architecture. Given that iHC performs competitively without residual mixing, the paper does not demonstrate that this modification constitutes a substantial contribution. The theoretical analysis also largely follows standard perturbation bounds for near-identity matrix products.

  2. The experiments are limited. HC demonstrated meaningful benefits at 7B scale, while mHC evaluated scaling up to 27B MoE models and demonstrated advantages over HC. Here, the experiments remain confined to tiny- to small-level models, which can not fully support the effectiveness of the proposed method.

  3. The improvements over iHC are marginal at the conventional 4-stream setting. For example, 3.0461 v.s. 3.0473 on the L model. Moreover, on the L model with n=4, mHC’s training loss is approximately 0.01 higher than iHC’s, making it difficult to assess whether the experimental setup is convincing or not.

  4. The main improvements appear at n=32 or 64, but such stream counts are not practical. Multi-stream residual states introduce substantial memory-access and communication overhead on LLM. Practical HC configurations typically use 4 streams, or at most 8. Results at very large n therefore provide limited evidence of practical value.

  5. The analysis assumes conditions required for stability, but provides no guarantee that these conditions hold during training. Consequently, it does not establish training stability for the proposed method.

Questions for authors
  1. What substantial benefit does modifying the residual mixer provide over simply using iHC at n=4?
3 comments

We thank the reviewer for the careful assessment. We address each critical weakness below. We added a 10,000-step Llama 2 7B pretraining.

1. Limited contribution and benefit over iHC

We agree that the architectural change is intentionally simple. The motivation is the gap between learned residual mixing and the identity baseline: in the settings reported in the manuscript, mHC and mHC-lite do not consistently improve on iHC, and the cited Qwen3 ablation post-ranks iHC above them (the source provides partial curves rather than numerical losses). This suggests that, in these settings, learning a doubly stochastic mixer can cost more than it gains. The open question motivating ITO-HC is whether a non-identity mixer can improve on identity by restoring useful cross-stream communication while keeping the cross-layer composite residual map well-conditioned.

The analysis further explains one possible cost of doubly stochastic mixing. Under exact doubly stochastic mixing with a uniform positive lower bound ε\varepsilon on every entry, the manuscript derives ∥GL−Jn∥2≤(1−nε)L\|G_L-J_n\|_2\leq(1-n\varepsilon)^L, so stream-difference directions contract toward zero as depth increases. This is a conditional result: double stochasticity alone does not imply such decay, and mHC's finite Sinkhorn iterations only approximately satisfy the constraint. ITO-HC addresses the open question by keeping an exact unit diagonal and adding signed, bounded off-diagonal corrections. Its stability statement is likewise conditional: for ηl=∥Hlres−I∥2<1\eta_l=\|H_l^{\mathrm{res}}-I\|_2<1, the residual-product minimum singular value is bounded below by ∏l(1−ηl)\prod_l(1-\eta_l), with a corresponding upper bound controlled by exp⁡(∑lηl)\exp(\sum_l\eta_l). This is a sufficient guarantee under a controlled perturbation budget, not an unconditional property of the unit diagonal.

2. Experimental scale

We have added a separate Llama 2 7B pretraining experiment on streamed English C4, using 10,000 steps, sequence length 2,048, and n=4n=4. It compares HC, iHC, mHC, mHC-lite, and ITO-HC, and reports training and validation trajectories through step 10,000. The manuscript reports that ITO-HC has lower loss than the existing methods in this experiment. This larger-backbone result directly addresses the concern that the evidence was limited to small GPT-2-style models, while we agree that one 7B run does not establish performance at 27B MoE scale.

3. Marginal OpenWebText gains at n=4n=4

We agree that the OpenWebText difference between ITO-HC and iHC at n=4n=4 is small; for the L model, the reported validation losses are 3.0810 and 3.0842, respectively. We do not present this gap as evidence of a large gain in that setting. The new Llama 2 7B experiment evaluates the practical four-stream configuration against four relevant alternatives (HC, iHC, mHC, and mHC-lite) on a substantially larger backbone; its loss curves and endpoint comparison are reported in the manuscript. This gives evidence beyond the small OpenWebText comparison, although the results remain single-run measurements.

4. Practicality of larger stream counts

We agree that results at n=32n=32 or 6464 alone would have limited practical relevance given the memory-access and communication costs of wide multi-stream states. The new Llama 2 7B comparison uses n=4n=4, a conventional HC configuration, and thus tests ITO-HC in the setting most relevant to the reviewer’s concern. The OpenWebText width sweep is retained as a capacity and width study; our practical-scale evidence is the added four-stream Llama experiment, not a claim that very large stream counts are necessary.

5. Whether the stability conditions hold during training

We agree that the general perturbation bound is conditional and by itself does not show that its assumptions hold during training. We now report checkpoint measurements from the 7B Llama ITO-HC run. Across its 64 attention and feed-forward residual mixers, the largest learned γl\gamma_l is approximately 3.96×10−33.96\times10^{-3}, and the largest measured spectral perturbation ηl=∥Hlres−I∥2\eta_l=\lVert H_l^{\mathrm{res}}-I\rVert_2 is 0.008480.00848 (block 0 attention). Thus every measured mixer satisfies ηl<1\eta_l<1. Using this maximum, the manuscript gives the conservative residual-product guarantee σmin⁡≥(1−0.00848)64≈0.580\sigma_{\min}\geq(1-0.00848)^{64}\approx0.580.

We directly measure the final residual-only products across all 64 sub-layers. The ITO-HC product has singular values (1.0214,1.0133,1.0082,0.9583)(1.0214,1.0133,1.0082,0.9583) and condition number 1.0659. For mHC, the token-mean cumulative product has singular values (1.0000,0.4450,0.3436,0.1314)(1.0000,0.4450,0.3436,0.1314) and condition number 7.608; the median singular values of the per-token products at final depth are (1.0000,0.7723,0.4181,0.1642)(1.0000,0.7723,0.4181,0.1642). These are residual-path measurements for the recorded checkpoint, not a claim about full-network or optimizer stability, but they show that the stated perturbation condition holds in this run and provide direct evidence about residual-path conditioning.

Thank you again for your thoughtful review. We have addressed your concerns by adding a 10,000-step Llama 2 7B comparison at the practical four-stream setting, reporting layerwise residual perturbations and cumulative-product diagnostics, and clarifying that our larger-width OpenWebText results are capacity studies rather than evidence that very large stream counts are practical. We also agree that the OpenWebText gain over iHC at n=4n=4 is small, and that our single-run residual-path measurements do not establish a general training-stability or performance advantage. We have narrowed the claims accordingly. We would welcome further discussion and would be grateful to hear whether these clarifications address your main concern about ITO-HC’s practical benefit over iHC.

Thanks for the rebuttal. However, my concerns largely remain.

The authors response to W2, W3, W4 with a new Llama 2 7B experiment. However, it is unclear how many tokens are consumed under the 10,000 training steps. From figure 2 of the revised paper, we could observe that the training loss is around 3.5, which is overly high for a 7B model. This indicates that the model is far from converge, and it is likely that the trained tokens may < 7B, i.e., the model size. Therefore, this result can not support the claims and W2, W3, W4 remain.

For W1 and W5, the authors provide clarification which is helpful and appreciated. However, it does not fully resolve the contribution and stability concerns.

Reviewer_EoydOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ITO-HC, a variant of Hyper-Connections (HC) in which the n×n residual-stream mixer is parameterized as H_res = I + γ_l (11ᵀ − I) ⊙ tanh(S_l) with a learned, input-independent logit matrix S_l and a learned per-layer scale γ_l, while the pre/post read/write routers remain dynamic as in mHC. The diagonal is fixed to one and off-diagonal entries are signed and bounded by γ_l; no doubly-stochastic (Sinkhorn/Birkhoff) constraint is imposed. The paper (i) derives a layer-attention view showing that the identity-residual baseline (iHC) induces a rank-≤n dense routing kernel, with explicit caveats that this does not bound the causal/nonlinear network; (ii) shows that products of entrywise-positive doubly stochastic mixers contract the stream-difference subspace toward the rank-one averaging matrix J_n at rate (1−nε)^L (Appendix A.7), motivating signed identity-centered mixing; (iii) gives Weyl-type conditional bounds on the singular spectrum of the ITO-HC residual product under a cumulative perturbation budget Σ‖E_l‖₂ (Eqs. 19–24); and (iv) reports final train/val losses after 10,000 iterations (≈2.6B tokens) of GPT-2-style S/M/L models on OpenWebText for iHC and ITO-HC at n ∈ {4,8,16,32,64}, and for mHC, mHC-lite, KromHC and a single-stream residual baseline at n = 4 only (Table 2), plus a single-token diagnostic of the 48-mixer residual product for mHC vs. ITO-HC (Fig. 2) and a small static/dynamic router ablation (Table 4).

Critical strengths
  1. A correct and clearly argued limitation of Birkhoff-constrained mixers (Sec. 2.4, Appendix A.7). The invariant-subspace decomposition and the bound ‖G_L − J_n‖₂ ≤ (1−nε)^L are proved cleanly, and the single-checkpoint diagnostic in Fig. 2 (σ_min ≈ 0.004, cond ≈ 236 for mHC vs. 0.50 / 3.3 for ITO-HC) gives a concrete, if anecdotal, illustration. This is the one place where the paper contributes an idea beyond the parameterization itself, and it is why I do not consider the paper a clear reject.
  2. Honest scoping. The paper consistently states what its derivations do and do not establish (rank bounds are "capacity diagnostics", the stability result is "a sufficient conditional guarantee"), and reports missing runs explicitly rather than imputing them.
Critical weaknesses
  1. The headline claim—training stability—has no experimental support, and the paper's own checkable sufficient condition fails at its best configurations (Sec. 4, Eq. 19; Table 2). The only verifiable stability condition is γ_l √(n(n−1)) < 1. With the stated initialization γ_l = 0.05 this gives 0.17 / 0.37 / 0.77 at n = 4 / 8 / 16 but 1.57 at n = 32 and 3.17 at n = 64, i.e., the guarantee does not cover the two widths at which ITO-HC shows its only clear gains (val 3.0559 / 3.0310 vs. iHC 3.0635 / 3.0606 for L). Sec. 4 says this "motivates monitoring both η_l … and the learned γ_l during training", but neither quantity is ever reported. No experiment exhibits any instability in any baseline (no loss spikes, divergence rates, gradient norms, or learning-rate sensitivity); every method trains to similar loss. The conclusion that "the results support the proposed training stability of ITO-HC" therefore rests on a Weyl bound whose premise is not checked and a single-token spectrum (Fig. 2) that the paper itself says "illustrates, but does not prove" the claim. This is consequential because stability is the stated reason to prefer ITO-HC over the simpler iHC and over mHC.

  2. The empirical comparison cannot separate ITO-HC from existing methods: differences at matched width are within single-run noise, the only large gains are not parameter-matched, and the closest baseline (original HC) is absent (Sec. 5, Table 2). All numbers are single runs without seeds or confidence intervals. At n = 4—the only width at which all methods are compared—final validation loss for the L model is 3.0810 (ITO-HC) vs. 3.0823 (KromHC), 3.0842 (iHC), 3.0898 (mHC-lite), 3.0916 (mHC): gaps of 0.001–0.01 nats after 10k iterations, far below what a single run can resolve. The only sizeable advantage over iHC (≈0.03 nats at n = 64) comes with 2dn² router parameters per hyper-connection, i.e., by the paper's Eq. 67 roughly 406M added parameters across 48 hyper-connections—more than the 355M backbone—and the paper acknowledges these are not parameter- or memory-matched runs; mHC is not run beyond n = 4, so the claimed width sweep for mHC (Sec. 5, first paragraph) does not exist in Table 2. Finally, ITO-HC differs from the original Hyper-Connections (Zhu et al., 2024)—static matrix initialized at I plus a tanh-bounded dynamic term—only by dropping the dynamic term, fixing the diagonal, and passing the static logits through tanh; yet original HC is not a baseline, and sHC (signed mixing) is mentioned only in related work. Without the HC baseline, multiple seeds, and parameter-matched comparisons, the paper does not establish that its parameterization is better than, or even different in effect from, what already exists. No downstream evaluation is reported, and the promised anonymous code supplement was not uploaded.

Questions for authors

Clarifications that could change my assessment:

  1. Please report the learned γ_l and η_l = ‖E_l‖₂ per layer at the end of training for all widths, and the resulting Π(1−η_l) and Σ η_l, so that Eqs. 22–24 can actually be checked—especially for n = 32 and 64, where the initialization violates γ_l √(n(n−1)) < 1.
  2. Is there any configuration (e.g., higher learning rate, longer training, larger n) in which iHC, mHC or HC exhibits instability that ITO-HC avoids? If not, what evidence supports the stability claim beyond Fig. 2?
  3. Can you provide ≥3 seeds (or at least loss curves with run-to-run variability) for the n = 4 comparison, and a parameter-matched comparison at n = 32/64 (e.g., iHC/mHC with the same total parameter count or the same throughput)?
  4. Why is the original Hyper-Connections parameterization (Zhu et al., 2024) not included as a baseline, given that ITO-HC is a restriction of it? Please add it, ideally at the same widths.
  5. Sec. 5 states mHC is swept over n ∈ {4,…,64}, but Table 2 contains mHC only at n = 4. Which is correct? If mHC runs at larger n failed or were not run, please say so.

Suggestions (not affecting the score): report at least one downstream evaluation; fix the "Appendix 3" reference and the leaked "qquad" tokens in Appendix A; upload the promised code.

2 comments

We thank the reviewer for identifying the limits of our stability and empirical claims. We clarify them below and distinguish measured results from evidence that remains unavailable. We also submitted the code as promised

1. Learned stability quantities

The general result uses ηl=∥El∥2\eta_l=\lVert E_l\rVert_2: if every ηl<1\eta_l<1, then σmin⁡(∏l(I+El))≥∏l(1−ηl)\sigma_{\min}(\prod_l(I+E_l))\geq\prod_l(1-\eta_l), while ∥∏l(I+El)∥2≤exp⁡(∑lηl)\lVert\prod_l(I+E_l)\rVert_2\leq\exp(\sum_l\eta_l). The estimate ηl≤γln(n−1)\eta_l\leq\gamma_l\sqrt{n(n-1)} is only a parameterization-specific sufficient bound.

For the separate Llama 2 7B ITO-HC run, we record γl\gamma_l and ηl\eta_l for all 64 attention and feed-forward mixers. The largest γl\gamma_l is approximately 3.96×10−33.96\times10^{-3}, and the largest ηl=∥Hlres−I∥2\eta_l=\lVert H_l^{\mathrm{res}}-I\rVert_2 is 0.008480.00848 (block 0 attention). Thus every measured mixer has ηl<1\eta_l<1, with the conservative envelope ∑lηl≤0.54272\sum_l\eta_l\leq0.54272 and ∏l(1−ηl)≥(1−0.00848)64≈0.580\prod_l(1-\eta_l)\geq(1-0.00848)^{64}\approx0.580. We do not have corresponding per-layer logs for every OpenWebText width, so this check applies only to this checkpoint. The bound is conditional rather than a claim that the initialization or all training steps satisfy it. It also uses the measured spectral perturbation directly, which can be sharper than the elementwise gamma bound.

2. Evidence for empirical stability

We did not run higher-learning-rate or longer-training stress tests and therefore cannot claim that ITO-HC prevents divergence in another method. We added a 10,000-step Llama 2 7B pretraining run on C4 at n=4n=4, comparing HC, iHC, mHC, mHC-lite, and ITO-HC; training and validation trajectories are logged.

Figures 6--8 give residual-only diagnostics from this run. Figure 6 shows cumulative residual maps across all 64 sub-layers for ITO-HC, the token-mean mHC product, and the mHC static-logit component. Figure 7 reports final singular values: ITO-HC (1.0214,1.0133,1.0082,0.9583)(1.0214,1.0133,1.0082,0.9583), condition number 1.06591.0659; the token-mean mHC product has (1.0000,0.4450,0.3436,0.1314)(1.0000,0.4450,0.3436,0.1314), condition number 7.6087.608, and final-depth median per-token values (1.0000,0.7723,0.4181,0.1642)(1.0000,0.7723,0.4181,0.1642). Figure 8 tracks these spectra with depth and shows stronger attenuation of smaller mHC directions. These measurements support residual-path conditioning for this checkpoint, not full-network or optimizer stability. The Llama run also records the learned layerwise gamma and eta values, so the premise of the residual-product bound is directly checkable at the final checkpoint, although not throughout optimization.

3. Seeds and parameter matching

Compute limits prevented running all methods with three or more seeds, so we make no seed-level significance claim. We added a parameter-matched comparison of ITO-HC n=32n=32 and mHC n=12n=12 for each S/M/L GPT-2 configuration. Total parameters differ by only 0.57%, 0.60%, and 0.57% (Small/Medium/Large): 57.99M vs. 57.66M, 162.84M vs. 161.86M, and 457.16M vs. 454.56M.

The corresponding training/validation losses are 3.3774/3.3612 vs. 3.3851/3.3724 (Small), 3.1234/3.0982 vs. 3.1579/3.1325 (Medium), and 3.0187/3.0559 vs. 3.0495/3.0813 (Large), for ITO-HC/mHC. This removes the main total-parameter confound because the totals include the shared backbone and all method-specific hyper-connection parameters. ITO-HC has lower reported validation loss in all three parameter-matched configurations. Since the stream widths differ, the comparison matches total parameter count, not runtime or memory; it remains single-run evidence and does not replace multi-seed experiments.

4. Original HC baseline and scale

We added original HC at n=4n=4 to both the OpenWebText and Llama 2 7B comparisons. The wider OpenWebText sweep remains limited to iHC and ITO-HC; the Llama experiment uses n=4n=4, 10,000 C4 steps, and sequence length 2,048. It provides a larger-backbone comparison, but does not isolate the parameterization with multiple seeds or establish superiority over HC statistically. The Llama curves use validation logging every 100 steps and training logging every optimizer step with time-weighted EMA. The run uses a global batch of 8 sequences; validation is performed on the local c4_mini split with 50 micro-batches. This complements the GPT-2-style OpenWebText study with a larger backbone and a common four-stream setting, while remaining a single-run comparison. ITO-HC disables the input-dependent residual generator, fixes the residual diagonal to one, and retains dynamic pre/post routers; the comparison therefore tests the proposed residual parameterization at a larger backbone while keeping the practical four-stream setting.

5. Width-sweep scope

The correct scope is that iHC and ITO-HC were swept over n∈{4,8,16,32,64}n\in\{4,8,16,32,64\}, while mHC, mHC-lite, and KromHC were evaluated only at n=4n=4. We corrected the manuscript wording accordingly.

Thank you again for the thoughtful review. In response to your central concerns, we added an HC baseline and a 10,000-step Llama 2 7B/C4 comparison at the practical four-stream setting, reported a parameter-matched ITO-HC/mHC comparison, and added layerwise residual-scale and cumulative-product diagnostics. These additions provide larger-backbone and residual-path evidence, while we agree that the small n=4 OpenWebText differences and single-run design do not establish a statistically reliable advantage or optimizer-level stability. We have narrowed our claims and clarified which width sweeps and measurements are available. We would welcome further discussion, especially if any remaining point would change your assessment.

Reviewer_tbNxOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ITO-HC, a residual-mixing parameterization for Hyper-Connections:

Hlres=I+γl (11⊤−I)⊙tanh⁡(Sl),H^{\mathrm{res}}_l = I + \gamma_l \,(\mathbf{1}\mathbf{1}^\top - I) \odot \tanh(S_l),

where SlS_l is a static, learned, layer-specific logit matrix and γl\gamma_l a learned scale. The diagonal is fixed to one and the off-diagonal entries are signed and bounded by γl\gamma_l. Dynamic pre/post routers are retained as in mHC, but the dynamic residual generator is removed. This reduces per-connection parameters from O(dn3)O(dn^3) for mHC to O(dn2)O(dn^2).

The paper makes three theoretical points:

  • Exactly doubly stochastic mixers with a uniform entrywise lower bound ε\varepsilon contract the stream-difference subspace 1⊥\mathbf{1}^\perp exponentially with depth.
  • A layer-attention view shows that the iHC (identity-residual) routing kernel satisfies rank⁡(C)≤n\operatorname{rank}(C) \le n, and ITO-HC does not change that bound.
  • Weyl-type perturbation bounds keep the product of ITO-HC mixers well-conditioned when the cumulative perturbation budget is small.

Experiments pretrain GPT-2-style S/M/L models on OpenWebText for 10k iterations, comparing ITO-HC against iHC for n∈{4,…,64}n \in \{4, \dots, 64\} and against mHC, mHC-lite, KromHC, and a single-stream residual at n=4n = 4. A single-token diagnostic compares the conditioning of the product of 48 trained residual mixers for mHC and ITO-HC.

Critical strengths
  1. The method is simple, cheap, and easy to implement. Removing the O(dn3)O(dn^3) dynamic residual generator and the Sinkhorn iterations is a practical advantage, and the parameter accounting (Table 1, Figure 1, Appendix A.6) is clear.
  2. The doubly stochastic contraction argument (Section 2.4, Appendix A.7) is stated carefully. It notes that double stochasticity alone is insufficient, that a uniform lower bound ε\varepsilon is needed, and that finite Sinkhorn iterations break the exact invariant-subspace argument.
  3. The paper is generally careful not to overclaim in its theory. It explicitly says the rank bound on CC does not transfer through the causal mask, and that invertible ITO-HC mixers can be absorbed into the read/write factors (Eq. 31–32).
  4. Running iHC and ITO-HC across nn up to 64 at three model sizes is a useful width sweep. The output-scale issue with sum reducers (Appendix A.5) is a sensible point to raise.
Critical weaknesses
  1. The empirical gains are very small and lack any measure of variance. Table 2 reports single last-iteration losses, apparently from one seed. At n=4n = 4, ITO-HC improves on iHC by 0.003–0.007 validation loss across S/M/L. The largest gap, about 0.03 at L with n=64n = 64, is plausible but is not established as significant without multiple seeds or confidence intervals. Train loss also exceeds validation loss in many S-model rows (for example, the residual baseline at 3.4177 train vs. 3.4598 val, and the reverse pattern elsewhere). This suggests last-iteration train loss is a noisy single-batch estimate. Please report multiple seeds, or at least averaged final losses, before drawing conclusions from differences of this size.

  2. The baselines are incomplete, and the main comparisons are confounded.

    • mHC, mHC-lite, and KromHC are evaluated only at n=4n = 4, although Section 5 says mHC is swept over n∈{4,…,64}n \in \{4, \dots, 64\}.
    • Original HC, with an unconstrained static plus dynamic residual, is not compared at all. That is the most direct baseline for a method whose novelty is constraining HC's static residual.
    • ITO-HC differs from mHC in two ways at once: static versus dynamic residual, and signed/identity-centered versus doubly stochastic. A static DS mixer and a dynamic ITO-HC mixer would separate the two.
    • There are no ablations of the method's own design choices. Missing variants include free diagonal versus fixed unit diagonal, the tanh⁡\tanh bound versus unbounded static off-diagonals, and the γl\gamma_l initialization or a fixed γ\gamma.

3.The central stability claims are not supported by the experiments. The abstract and conclusion state that results support the proposed training stability and the stability–capacity trade-off. However, the paper reports no stability measurements: no loss spikes, gradient norms, divergence rates, or learning-rate sensitivity. The theory's sufficient condition, γln(n−1)<1\gamma_l \sqrt{n(n-1)} < 1 together with a bounded ∑lηl\sum_l \eta_l where ηl=∥El∥2\eta_l = \|E_l\|_2, is never checked. Section 4 says this motivates monitoring ηl\eta_l and the learned γl\gamma_l, but no learned values are reported. γl\gamma_l is learned through an unconstrained log-scale, so nothing enforces the condition. The one diagnostic shows a cumulative product with σmax⁡(GITO)=1.665\sigma_{\max}(G_{\mathrm{ITO}}) = 1.665, so the n=4n = 4 product is already far from identity. At n=64n = 64, the condition requires γl<1/64⋅63≈0.016\gamma_l < 1/\sqrt{64 \cdot 63} \approx 0.016.

Questions for authors
  1. Can you report mean ±\pm s.d. over at least 3 seeds for the key comparisons, at minimum iHC vs. ITO-HC vs. mHC at n=4n = 4 and n=64n = 64 for M and L?
  2. Why is original HC not included as a baseline? How does an unconstrained static residual compare to the tanh⁡\tanh-bounded, unit-diagonal version?
  3. What values do the learned γl\gamma_l and ηl=∥El∥2\eta_l = \|E_l\|_2 take after training, per layer and per nn? Does the sufficient condition γln(n−1)<1\gamma_l \sqrt{n(n-1)} < 1 with bounded ∑lηl\sum_l \eta_l hold in practice, especially at n=64n = 64?
  4. Can you separate the "static" and "signed, identity-centered" factors, for example with a static DS mixer and a dynamic ITO-HC residual?
  5. Do you observe any stability difference between methods, such as loss spikes, divergence, or learning-rate sensitivity, that would support the stability claims in the abstract?
  6. What are the throughput and peak memory for iHC and ITO-HC at n∈{4,16,64}n \in \{4, 16, 64\}, compared with the single-stream baseline?
  7. Can the mixer-product diagnostic in Section 5.3 be computed over many tokens and sequences, with the distribution of σmin⁡(GL)\sigma_{\min}(G_L) reported for both methods?
  8. Where are the mHC results for n>4n > 4 that Section 5 describes?
3 comments

We thank the reviewer and clarify our work through the following parts:

1. Mean and standard deviation over at least three seeds

Seed-level uncertainty is important for these small gaps. The current OpenWebText results are single runs, so we cannot report means or standard deviations for iHC, ITO-HC, and mHC at n=4 and n=16 for M and L. The n=4 validation losses for iHC / ITO-HC / mHC are S: 3.3833 / 3.3765 / 3.3817, M: 3.1348 / 3.1293 / 3.1472, and L: 3.0842 / 3.0810 / 3.0916. These are descriptive point estimates, not evidence of statistical significance; multi-seed replication is needed. At n=16, the manuscript reports iHC/ITO-HC point estimates for M (3.1128/3.1085) and L (3.0712/3.0657), but no mHC result at that width; neither requested three-method means nor standard deviations can be calculated.

2. Original HC and an unconstrained static residual baseline

Original HC is included at n=4, with validation losses 3.4013 / 3.1807 / 3.1583 for S / M / L. We did not run the requested matched baseline: a static, unconstrained mixer with ITO-HC's routers. HC is not a substitute because its residual generator and parameterization differ. Thus we cannot isolate the bounded, unit-diagonal constraint; this ablation is future work.

3. Learned γ and η across layers and widths

The GPT-2 runs lack per-layer γ and η for every width, so we cannot report them for n=4, n=16, or n=64. In the separate Llama 2 7B/C4 ITO-HC run, across 64 attention and MLP mixers, the maximum learned γ is approximately 3.96 × 10⁻³, and the maximum η=‖Hʳᵉˢ−I‖₂ is 0.00848 (block 0 attention). Hence every measured mixer has η<1, and the conservative bound is σ_min ≥ (1−0.00848)⁶⁴ ≈ 0.580. This verifies the condition only for that checkpoint, not every OpenWebText width or step.

4. Separating static/dynamic and signed/identity-centered/doubly-stochastic factors

The ITO-HC versus mHC comparison confounds these factors. We did not run a static doubly stochastic mixer or a dynamic ITO-style residual mixer, so we cannot attribute the difference to staticity or to signed identity-centering separately. The appendix's four-way ablation varies only dynamic/static pre- and post-routers at n=4; every ITO-HC row retains the proposed static residual matrix. We will avoid causal attribution; a factorial residual-mixer ablation remains future work.

5. Loss spikes, divergence, and learning-rate sensitivity

We did not test loss spikes, divergence, gradients, or learning-rate sensitivity, so we cannot claim optimizer-level stability. Our evidence is narrower: conditional spectral bounds and one final-checkpoint Llama residual diagnostic. The ITO-HC residual-only product has singular values (1.0214, 1.0133, 1.0082, 0.9583) and condition number 1.0659. The mHC token-mean product has (1.0000, 0.4450, 0.3436, 0.1314) and condition number 7.608; its final-depth per-token median is (1.0000, 0.7723, 0.4181, 0.1642). These measure residual-path conditioning only; we will narrow the claims.

6. Throughput and peak memory at n=4

We lack matched throughput or peak-memory measurements for iHC, ITO-HC, and single-stream at n=4. The only available throughput measurement is a separate small-model stress test: static/static iHC at n=70 achieved 4.61 × 10⁵ tokens/s versus 4.85 × 10⁵ for dynamic-router iHC at n=4. This is neither an ITO-HC nor memory comparison; no n=4 efficiency claim follows. A matched benchmark is needed.

7. Mixer-product diagnostics over many tokens and sequences

The Llama analysis reports the token-mean mHC cumulative product and per-token singular-value statistics at final depth; the figure shows a median and 5th–95th percentile range across evaluated tokens. The median is (1.0000, 0.7723, 0.4181, 0.1642). It lacks σ_min quantiles aggregated over independent sequences. ITO-HC is token-independent because its mixers are static. Sequence-level quantiles would be more informative; we will label this a token-level, single-checkpoint diagnostic.

8. Missing mHC results for n>4

The reviewer is correct that the wording could be ambiguous. The width sweep over n ∈ {4, 8, 16, 32, 64} is only for iHC and ITO-HC; mHC, mHC-lite, and KromHC are reported at n=4. We did not complete an mHC sweep at larger widths and will correct the text. The appendix's ITO-HC n=32 versus mHC n=12 experiment is parameter-matched (total parameters differ by less than 0.61%) but is a single comparison, not a width sweep.

These questions identify the main limits: single-seed losses, missing matched ablations and efficiency measurements, and residual-path-only stability diagnostics. We will align the claims accordingly.

Reviewer_tbNxOpenReview ↗

Reply to Rebuttal

I appreciate the efforts of the authors, and my questions have mostly been addressed. I will keep my positive scores.

We are grateful for the reviewer's feedback that has meaningfully improved our exposition and empirical evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.