acceptodds
Under review as a conference paper at ICLR 2027

One Marginal Is Enough: What Doubly Stochastic Routing Buys Hyper-Connections, and What It Costs

Abstract

Hyper-Connections widen the residual pathway into parallel streams and put a learned matrix in front of every layer to mix them. Those matrices multiply along depth, so they have to be constrained, and Manifold-Constrained Hyper-Connections (mHC) constrain them to the Birkhoff polytope of doubly stochastic matrices. That is two constraints, one on the row sums and one on the column sums, and we ask what each marginal is worth. Fixing either one alone already bounds the spectral norm of the whole product by , at any depth, so a single marginal buys the depth-uniform stability the constraint was introduced for. The second only tightens that bound from to , and it is not free: the mixer can then neither replicate a source nor aggregate at a destination, and it can only preserve or dissipate the differences among the streams, never enlarge them. We therefore keep one marginal and release the other, giving One-Sided Hyper-Connections (osHC). We instantiate four variants — two that fix the same marginal at every layer, and two that switch marginals partway through the network — and prove that all four are stable at every depth. Across ten routing constructions, two pretraining corpora and three scales, osHC attains the best validation loss in five of the six settings and beats every exactly doubly stochastic construction in all six, and at the largest scale it takes the best and the second-best zero-shot perplexity on each of five held-out corpora, under either pretraining corpus. Trained osHC routers use the released capability at almost every site, while their depth-composed gain stays inside the proved ceiling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hyper-Connections, doubly stochastic routing, residual streams, language model pre-training, spectral stability, neural architecture

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uQ67OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits whether the doubly stochastic constraint used in mHC is necessary for stabilizing residual mixing. The main idea is that fixing only one marginal--either the row sums or the column sums--is already sufficient to obtain depth-independent control of the residual-routing operator, which motivates the proposed One-Sided Hyper-Connections (osHC) with greater routing flexibility. Theoretically, the authors show that one-sided stochastic matrix families are closed under multiplication, yielding spectral-norm bounds of n\sqrt{n} for pure osHC schedules and nn for a single row/column switch, while also arguing that the doubly stochastic constraint limits source replication, destination aggregation, and the growth of disagreement across streams. Empirically, across two pretraining corpora, three model scales (up to 354M parameters), three random seeds, and multiple routing baselines, osHC achieves the best validation loss in five of six in-domain settings and also performs strongly on five Paloma out-of-distribution corpora. Routing diagnostics further show that the learned routers actively exploit the released replication/aggregation capability.

Critical strengths

The paper develops a clean theoretical account of why one marginal can be sufficient for stable residual routing. A coherent set of empirical results support the theoretical predicitions.

Critical weaknesses

The empirical evaluation is relatively limited in scale, which leaves the practical effectiveness of osHC at larger model and stream sizes uncertain. All experiments use n=4n=4 streams, models up to 354M parameters, and relatively short pretraining runs, so it remains unclear whether the observed gains persist in more realistic large-scale settings. This limitation is partially offset by the paper’s clean theoretical analysis, which provides a principled justification for the method beyond the current empirical regime.

Questions for authors
  1. Appendix A states that the proof of Theorem 3.2 does not rely on H≥0H\ge 0, yet the proof in Appendix B explicitly uses non-negativity to obtain ∥H∥1=∥H∥∞=1\lVert H\rVert_1=\lVert H\rVert_\infty=1. Is the intended argument for sHC instead based on its additional spectral-norm constraint? Clarifying this would resolve an apparent inconsistency in the theoretical presentation.

  2. Can the authors provide uncertainty estimates for the main validation-loss comparisons against mHC, mHC-lite, and KromHC? Since several differences are small and each setting uses only three seeds, paired differences or confidence intervals would help assess whether the reported rankings reflect robust gains.

  3. Do the authors have results for stream counts other than n=4n=4? Since the theoretical bounds scale as n\sqrt n for pure osHC and nn for a one-switch schedule, even a limited sweep over nn would help assess whether the empirical advantage persists as the number of streams increases.

A clear response to these questions would help strengthen my confidence in the paper and may lead to a higher rating.

3 comments
AuthorsOpenReview ↗

Theorem 3.2 Clarification, Paired Confidence Intervals, and n=8 / 1.1B Scaling

Thank you for the careful reading and the three precise questions. All were answerable with new analysis or new runs; short answers below, details in the revised appendix.

Q1. H≥0H\ge0 in the proof of Theorem 3.2. You are right, and your reading is the intended one. The proof has two steps: (a) the two marginals alone give ΠH=HΠ\Pi H=H\Pi; (b) ∥H∥2≤1\lVert H\rVert_2\le1, which on Bn\mathcal B_n comes from non-negativity (Riesz–Thorin). sHC keeps both marginals, so (a) holds, and it imposes ∥H∥2=1\lVert H\rVert_2=1 by construction, which replaces (b). The sentence in Appendix A ("never uses H≥0H\ge0") was wrong as written and is corrected. No theorem or conclusion changes.

Q2. Uncertainty estimates. All methods share seeds, data and schedule, so we report paired per-seed differences over the 18 (corpus, scale, seed) pairs: mean Δ in nats [95% tt-interval], pairs won. Negative favours osHC.

vs mHCvs mHC-litevs KromHC
osHC-C−0.0054 [−0.0083, −0.0026], 17/18−0.0069 [−0.0101, −0.0036], 16/18−0.0076 [−0.0099, −0.0053], 17/18
osHC-R−0.0041 [−0.0059, −0.0024], 16/18−0.0055 [−0.0081, −0.0030], 17/18−0.0063 [−0.0077, −0.0049], 18/18
osHC-C→R−0.0067 [−0.0092, −0.0042], 16/18−0.0081 [−0.0106, −0.0056], 18/18−0.0088 [−0.0108, −0.0068], 18/18
osHC-R→C−0.0038 [−0.0075, −0.00003], 14/18−0.0052 [−0.0092, −0.0012], 15/18−0.0059 [−0.0090, −0.0028], 15/18

Eleven of twelve intervals clearly exclude zero; the one for osHC-R→C vs mHC ends at zero. Pooled over the four variants, osHC wins 63/72 pairs against mHC, 66/72 against mHC-lite and 68/72 against KromHC (sign test p<10−10p<10^{-10} each). Bootstrap intervals, Wilcoxon tests and per-cell differences are in new appendix tables.

Q3. Other stream counts. We trained every method at n=8n=8 on FineWeb-Edu (Small and Medium, seed 0; validation loss, same protocol; best in bold). mHC-lite is omitted at n=8n=8 because it needs 8!=403208!=40320 permutation coefficients.

n=8n=8SmallMedium
HC3.51323.3650
mHC3.48333.2837
KromHC3.47593.2844
sHC3.48963.2848
osHC-C3.47103.2692
osHC-R3.46583.2676
osHC-C→R3.46203.2715
osHC-R→C3.46463.2693

Every osHC variant is below every other method at both scales, by 0.005–0.014 nats (Small) and 0.012–0.016 (Medium) against the best baseline. The one-switch schedules, whose ceiling is n=8n=8 rather than 8\sqrt 8, land in the same band as the pure ones.

Weakness: scale. We scaled up the model to 1.1B parameters (36 layers, width 1536; FineWeb-Edu, seed 0, same training protocol; best in bold):

1.1BHCmHCmHC-liteKromHCsHCosHC-CosHC-RosHC-C→RosHC-R→C
val loss3.34793.22763.17613.16453.42583.12153.12963.11133.1296

All four osHC variants are below every baseline, by 0.035–0.053 nats against the best one (KromHC). Together with the n=8n=8 results, this indicates that the advantage persists at larger model and stream sizes. Both tables are added to the paper.

Thank you for the detailed response. Most of my concerns have been addressed in the rebuttal, and I therefore plan to raise my score. However, the currently uploaded revised PDF does not appear to incorporate the promised updates. I look forward to seeing these corrections and additional results reflected in the final revision.

AuthorsOpenReview ↗

Revised PDF Uploaded — Exact Locations of Updates (Appendix A, §B.9, and Tables 3–7)

Thank you very much for reading our response and for raising your score! We have now uploaded the revised PDF incorporating all promised corrections and additional experiments. For convenience, here are the exact locations in the updated manuscript corresponding to each of your points:

  • Q1 (Role of H≥0H \ge 0 in Theorem 3.2):

    • Appendix A ("Why the three relaxations do not reach the released mechanism", page 12, lines 607–622): Corrected the discussion to distinguish the role of the two marginals (ΠH=HΠ\Pi H = H\Pi) from that of H≥0H \ge 0 (∥H∥2≤1\lVert H\rVert_2 \le 1, which sHC enforces directly via ∥H∥2=1\lVert H\rVert_2 = 1).
    • Appendix §B.9, Proposition B.9 ("Balanced mixers without the sign condition", page 18): Formal statement and proof separating the two marginals from the spectral-norm bound, together with {∥H∥2≤1}=conv O(n)\{\lVert H\rVert_2 \le 1\} = \mathrm{conv}\,\mathcal O(n).
  • Q2 (Uncertainty estimates and paired statistics):

    • Appendix §C.3, Table 3 (page 19): Complete per-seed validation losses for all 180 runs.
    • Appendix §C.4, Tables 4 and 5 (pages 19–20): Paired per-seed differences across all 18 (corpus, scale, seed) pairs, 95% tt-intervals, 95% bootstrap intervals, win counts, and sign-test / Wilcoxon pp-values (Table 4), as well as per-corpus, per-scale paired differences with standard errors (Table 5).
  • Q3 (Higher stream count, n=8n=8):

    • Appendix §C.8, Table 6 (page 22): Full-split validation losses at n=8n=8 streams on FineWeb-Edu (Small and Medium).
  • Weakness (Larger model scale, 1.1B parameters):

    • Appendix §C.9, Table 7 (page 22): Full-split validation losses at 1.1B parameters (36 layers, width 1536) on FineWeb-Edu.

(In addition, Appendix §B.7–B.8 on pages 16–17 and Appendix §C.10–C.12, Tables 8–10 on pages 22–24 record the single-step Sinkhorn baseline, full-network Jacobian bounds and probes, and forward–backward stream diversity analyses added in response to the other reviews.)

Thank you again for your constructive feedback, which helped us improve the paper!

Reviewer_aNfjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose One-Sided Hyper-Connections (osHC). It only constrain one marginal of routing matrix. This can keep bounded stability and allow more flexible routing. Authors prove theoretical bound. Experiments on two datasets and three model sizes show better validation loss and perplexity than mHC baselines.

Critical strengths

Simple method with clear theoretical guarantee. Only one softmax is needed. This make implementation easy.

Experiments compare ten methods. Results show consistent improvement on validation loss and OOD perplexity.

Critical weaknesses
  1. Theoretical stability is not fully convincing for real network. The paper only prove bounded norm for residual mixing matrices. But actual Transformer also have attention, MLP and input-dependent routing. Their gradient effect is not covered by this proof. Especially router itself depends on input, so its derivative also affect stability. Also, switching marginal increase theoretical upper bound from (\sqrt{n}) to (n). This may become problem when number of streams is large. Current gradient norm experiments cannot fully prove full model stability.

  2. Experimental evidence and mechanism analysis still not sufficient.The paper claim one-sided routing can better preserve stream diversity. But no direct experiment to measure actual representation diversity. Only showing replication and aggregation factors is not enough. Appendix also mention one-sided routing can still have over-smoothing. So why performance improve is still not very clear.

Questions for authors

1Since routing matrix depend on input, can authors analyze full Jacobian instead of only matrix product? Can the stability guarantee still hold when including attention, MLP and routing derivative? This is important for main theoretical claim.

2Can authors directly show stream diversity across different layers? For example, pairwise cosine similarity, effective rank, or disagreement energy (E(X)). The paper show routing weights become more flexible, but does this actually prevent stream collapse?

3Can authors provide ablation to show the improvement really come from releasing one marginal?

2 comments
AuthorsOpenReview ↗

Full-Network Jacobian, Stream Diversity & Duality, and Controlled Ablations

Thank you for the constructive review and three concrete questions. Short answers below; proofs and tables are in the revised appendix.

Q1 / W1. Full-network Jacobian & larger nn (Appendix §B.7, §C.8, §C.11).

  • Theory: For T(X)=H(X)XT(X)=H(X)X, Prop. B.6 gives ∥DT(X)∥≤∥H∥2+K\lVert DT(X)\rVert \le \lVert H\rVert_2 + K (K=12∣α∣∥φ∥2nd(1+max⁡i∣γi∣)K = \frac{1}{2}\lvert\alpha\rvert\lVert\varphi\rVert_2\sqrt{nd}(1+\max_i\lvert\gamma_i\rvert), scale/depth-free as RMSNorm cancels ∥X∥\lVert X\rVert). With layer Jacobian Hl⊗I+ElH_l\otimes I + E_l (ElE_l: branch + router derivatives), closure of one-sided matrices gives Prop. B.7: ∥JL∥2≤n∏l(1+n∥El∥2)\lVert J_L\rVert_2 \le \sqrt{n}\prod_l (1 + \sqrt{n}\lVert E_l\rVert_2)—matching ∏l(1+∥El∥2)\prod_l(1+\lVert E_l\rVert_2) up to n\sqrt{n} per factor, vs. κL\kappa^L unconstrained.
  • Measurement: Forward-mode AD (jvp) on all 180 models gives ∥Jv∥/∥v∥\lVert Jv\rVert/\lVert v\rVert (Large, FineWeb-Edu, 3 seeds):
MethodFrozen-router gainFull JacobianGrad ratio (1st/last)
Residual / Bn\mathcal B_n25.3 / 28.1–29.725.3 / 26.7–28.438 / 37–59
HC / sHC88.8 / 181192.2 / 1660154 / 73
osHC (4 variants, mean)25.1 (17.7–33.9)24.4 (17.3–33.7)38 (25–55)

osHC's full Jacobian (mean 24.4) and grad ratio (mean 38) match the residual network and are slightly lower than Bn\mathcal B_n baselines; enabling router derivatives slightly reduces gain (25.1 to 24.4).

  • Switch & n=8n=8: One switch raises the ceiling from n\sqrt{n} to nn, still depth-free. At n=8n=8 streams (FineWeb-Edu S/M, seed 0, Appendix §C.8), all four osHC variants are stable and beat all baselines (HC, mHC, KromHC, sHC) by 0.005–0.014 nats (S) and 0.012–0.016 nats (M).

Q2 / W2. Stream diversity & forward–backward duality (Appendix §C.12). Probing all 48 sites at Large (8×10248\times 1024 tokens, 3 seeds) on forward states XlX_l and backward gradients Gl=∂L/∂XlG_l = \partial\mathcal L/\partial X_l for relative disagreement E(⋅)/∥⋅∥F2\mathcal E(\cdot)/\lVert\cdot\rVert_F^2 and per-site amplification rates (E(HX)\mathcal E(HX) > E(X)\mathcal E(X) forward; E(H⊤G)\mathcal E(H^\top G) > E(G)\mathcal E(G) backward):

Large, FineWeb-EduHH / H⊤H^\topFwd Amp.Fwd Disagr. (Final)Bwd Amp.Bwd Disagr.
mHC-lite / KromHCBn\mathcal B_n / Bn\mathcal B_n0.0%0.43–0.44 (0.21–0.22)0.0%0.28
mHC (T=20T=20)Approx. Bn\mathcal B_n29.6%0.42 (0.36)0.9%0.31
osHC-CCn\mathcal C_n / Rn\mathcal R_n64.6%0.50 (0.56)3.4%0.33
osHC-RRn\mathcal R_n / Cn\mathcal C_n0.4%0.36 (0.13)57.5%0.36
osHC-C→RCn→Rn\mathcal C_n\to\mathcal R_n / Rn←Cn\mathcal R_n\leftarrow\mathcal C_n34.6%0.43 (0.14)36.1%0.34
osHC-R→CRn→Cn\mathcal R_n\to\mathcal C_n / Cn←Rn\mathcal C_n\leftarrow\mathcal R_n16.2%0.37 (0.29)29.9%0.31
  • No collapse: Unconstrained HC collapses mid-depth (mean disagreement E(X)/∥X∥F2=0.16\mathcal E(X)/\lVert X\rVert_F^2 = 0.16, dropping as low as 0.040.04). All constrained variants prevent collapse (2.2–3.1×2.2\text{--}3.1\times higher disagreement across depth).
  • Forward–backward duality (Xl↦HXl+…X_l \mapsto HX_l+\dots, Gl←H⊤Gl+…G_l \gets H^\top G_l+\dots): Transposition swaps Rn↔Cn\mathcal R_n \leftrightarrow \mathcal C_n. Bn\mathcal B_n cannot amplify disagreement in either pass (Thm. 3.2; 0.0% measured; final 0.21–0.220.21\text{--}0.22). A row-stochastic operator fixes consensus (H1=1H\mathbf 1=\mathbf 1), so it only rescales existing disagreement (at most n/2n/2 in energy, Thm. B.8; 0.4% / 3.4%); a column-stochastic one also turns the shared component into disagreement. Thus osHC-C amplifies forward disagreement (64.6%; 0.56 final), osHC-R mirrors it in backward gradients (57.5%; 0.36), both hybrids amplify in both passes, and all four osHC variants sustain higher gradient disagreement than exact Bn\mathcal B_n (0.31–0.36 vs. 0.28). Since Rep(H⊤)=Agg(H)\mathrm{Rep}(H^\top)=\mathrm{Agg}(H), every osHC model uses one released capability forward and its adjoint backward in each step.

Q3. Ablation isolating the released marginal (Appendix §C.4, §C.9, §C.10).

  1. 1-to-1 control (Table 1): osHC-R/C and mHC share identical parameters, Hpre/HpostH^{\mathrm{pre}}/H^{\mathrm{post}}, RMSNorm, logits ZlZ_l, and init (H0≈InH_0 \approx I_n); only the marginal normalization on ZlZ_l changes. All three Bn\mathcal B_n baselines train identically and trail osHC (63/72, 66/72, 68/72 seed wins, pp < 10−1010^{-10}; by 0.035–0.053 nats at 1.1B).
  2. Sinkhorn TT-sweep in mHC (Fig. 2 & Appendix §C.10): One Sinkhorn iteration (T=1T=1) already produces an exactly one-sided stochastic matrix. Loss worsens with TT, while mHC (T=1T=1) (3.4577 / 3.2394 / 3.1792 on FWE S/M/L) beats mHC (T=20T=20) (3.4663 / 3.2412 / 3.1816) and mHC-lite (3.4630 / 3.2426 / 3.1829) at every scale (8/9 seed pairs each) and lands in the osHC band.
AuthorsOpenReview ↗

Revised PDF Uploaded — Exact Page and Table Pointers for Your Questions

Dear Reviewer aNfj,

As the discussion period progresses, we wanted to briefly follow up to let you know that the revised manuscript PDF has been uploaded, and to provide a quick index of where each of your questions is addressed in the updated paper:

  • Q1 / W1 (Full-network Jacobian stability & n=8n=8 streams):

    • Appendix §B.7, Propositions B.6 and B.7 (pages 16–17): Scale-free bound on the input-dependent router derivative and depth-uniform bound on the full network Jacobian.
    • Appendix §C.11, Table 9 (page 23): Forward-mode AD full-Jacobian gains and backward gradient ratios across all 180 checkpoints.
    • Appendix §C.8, Table 6 (page 22): Validation losses at n=8n=8 streams (Small and Medium).
  • Q2 / W2 (Layerwise stream diversity & forward–backward adjoint duality):

    • Appendix §B.8, Theorem B.8 (page 17): Sharp one-step disagreement ceiling under one marginal.
    • Appendix §C.12, Table 10 (page 24): Direct forward (XlX_l) and backward (GlG_l) stream disagreement and amplification rates across all 48 mixing sites at Large.
  • Q3 (Ablations isolating the released marginal):

    • Appendix §C.10, Table 8 (pages 22–23): Single-iteration Sinkhorn mHC (T=1) across Small, Medium, and Large.
    • Appendix §C.4, Tables 4–5 (pages 19–20) & Appendix §C.9, Table 7 (page 22): Paired per-seed confidence intervals and 1.1B1.1\text{B} scaling results.

Please let us know if our response and revisions have addressed your concerns, or if there are any further questions we can clarify. Thank you again for your time and thoughtful review!

Reviewer_kSbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the doubly stochastic constraint in Manifold-Constrained Hyper-Connections (mHC) and proposes One-Sided Hyper-Connections (osHC), which enforces only row- or column-stochasticity. The authors show that a single marginal constraint is sufficient to ensure depth-uniform stability of the residual routing path, while allowing more flexible information routing. Experiments across different model scales and pretraining corpora demonstrate consistent but modest improvements in validation loss and zero-shot generalization.

Critical strengths
  • The proposed approach replaces iterative SK normalization with a single softmax operation, making it straightforward to implement and integrate into existing mHC architectures.
  • The paper provides a clear analysis showing that enforcing both stochasticity constraints is not necessary for depth-uniform stability.
  • The authors evaluate multiple routing variants across two pretraining corpora, three model scales, and multiple random seeds.
Critical weaknesses
  • The theoretical novelty is limited. The main theoretical results largely follow from standard properties of stochastic matrices and matrix norms. While their application to HC is meaningful, the mathematical contribution itself appears relatively incremental.
  • The paper attributes the improvements to the additional replication and aggregation capabilities enabled by relaxing one marginal constraint. However, the reported routing statistics mainly demonstrate that these capabilities are available and used, rather than establishing that they cause better performance. Direct measurements of stream disagreement or targeted ablation studies would strengthen this claim.
  • The paper shows in Figure 2 that reducing the number of SK iterations improves performance, with a single iteration performing best in the tested settings. Since one iteration already produces a row-stochastic matrix under the adopted normalization order, including mHC with one SK iteration as a baseline in the main experiments is important for isolating the actual benefit of osHC.
Questions for authors

How well does osHC scale to larger models and higher stream counts, where routing stability may become more challenging?

2 comments
AuthorsOpenReview ↗

Theoretical Contribution, Stream Diversity & Duality, 1-Step mHC Baseline, and n=8 / 1.1B Scaling

Thank you for the constructive review; proofs and tables are in the appendix.

W1. Theoretical novelty. We agree the tools are classical, as throughout this line of work: mHC rests on Sinkhorn's theorem and the Birkhoff polytope Bn\mathcal B_n, mHC-lite on Birkhoff–von Neumann, KromHC on Kronecker closure of Bn\mathcal B_n. In each case the contribution is identifying which constraint residual routing needs. Every follow-up (mHC-lite, KromHC, go-mHC, TBP-mHC) took both marginals as required for stability and competed on how to reach Bn\mathcal B_n. We ask if both are needed, and prove that one marginal already gives depth-uniform stability (≤n\le\sqrt{n}, Thm. 4.2), while the second forbids replication, aggregation and disagreement growth (Thm. 3.2)—turning an iterative or n!n!-term projection into one exact softmax.

W2. Disagreement measurements & causal link (Appendix §C.12). All 48 sites at Large (FineWeb-Edu, 8×10248\times 1024 tokens, 3 seeds): relative disagreement E(⋅)/∥⋅∥F2\mathcal E(\cdot)/\lVert\cdot\rVert_F^2 of states XlX_l and gradients Gl=∂L/∂XlG_l=\partial\mathcal L/\partial X_l, and the share of sites where HH or H⊤H^\top amplifies it.

(a) Forward (Xl+1=HXl+…X_{l+1}=HX_l+\dots):

MethodForward HHAmplifying sitesDisagreement (mean)Disagreement (final)
mHC-lite / KromHCBn\mathcal B_n0.0%0.43–0.440.21–0.22
mHC (T=20T=20)≈Bn\approx\mathcal B_n29.6%0.420.36
osHC-CCn\mathcal C_n64.6%0.500.56
osHC-RRn\mathcal R_n0.4%0.360.13
osHC-C→RCn→Rn\mathcal C_n\to\mathcal R_n34.6%0.430.14
osHC-R→CRn→Cn\mathcal R_n\to\mathcal C_n16.2%0.370.29

(b) Backward (Gl=H⊤Gl+1+…G_l=H^\top G_{l+1}+\dots; transposition swaps Rn↔Cn\mathcal R_n\leftrightarrow\mathcal C_n):

MethodBackward H⊤H^\topAmplifying sitesDisagreement (mean)
mHC-lite / KromHCBn\mathcal B_n0.0%0.28
mHC (T=20T=20)≈Bn\approx\mathcal B_n0.9%0.31
osHC-CRn\mathcal R_n3.4%0.33
osHC-RCn\mathcal C_n57.5%0.36
osHC-C→RRn←Cn\mathcal R_n\leftarrow\mathcal C_n36.1%0.34
osHC-R→CCn←Rn\mathcal C_n\leftarrow\mathcal R_n29.9%0.31
  • Measurements. HC collapses mid-depth (mean 0.16, min 0.04); all constrained variants avoid this. Bn\mathcal B_n never amplifies disagreement (Thm. 3.2; 0.0% sites). osHC-C amplifies it forward, osHC-R backward, the hybrids in both, and all four have higher gradient disagreement than exact Bn\mathcal B_n (0.31–0.36 vs. 0.28).
  • Causality. Usage statistics alone do not prove cause; the targeted ablation is W3: mHC (T=1T=1) and mHC (T=20T=20) share code, parameters and training and differ only in how far the free marginal is released, and the less balanced one wins at all scales. Conversely, the three Bn\mathcal B_n constructions (mHC, mHC-lite, KromHC) use three distinct parameterizations of Bn\mathcal B_n, yet train almost identically and all trail osHC (Table 1).

W3. mHC (T=1T=1) baseline (Appendix §C.10). You are right that one Sinkhorn iteration already produces an exactly one-sided stochastic matrix. We added mHC (T=1T=1) on FineWeb-Edu across all three scales (3 seeds, best in bold):

KromHCmHC-litemHC (T=20T=20)mHC (T=1T=1)osHC-RosHC-R→CosHC-CosHC-C→R
Small3.46633.46303.46633.45773.45953.45733.45793.4559
Medium3.24263.24263.24123.23943.23723.23703.23743.2369
Large3.18513.18293.18163.17923.17953.17913.17373.1716

(1) mHC (T=1T=1) beats mHC (T=20T=20), mHC-lite and KromHC at all scales (8/9 seed pairs each). As a one-marginal method itself, it directly supports our claim: stopping at one marginal beats converging to Bn\mathcal B_n. (2) osHC matches or improves on mHC (T=1T=1) while replacing the two-step Sinkhorn pass with a single softmax and no iteration hyperparameter.

Q1. n=8n=8 streams and 1.1B parameters (Appendix §C.8–C.9). FineWeb-Edu, seed 0 (mHC-lite omitted at n=8n=8: 8!=403208!=40320; 1.1B: 36L, d=1536d=1536, n=4n=4):

HCmHCmHC-liteKromHCsHCosHC-CosHC-RosHC-C→RosHC-R→C
n=8n=8, Small3.51323.4833—3.47593.48963.47103.46583.46203.4646
n=8n=8, Medium3.36503.2837—3.28443.28483.26923.26763.27153.2693
1.1B3.34793.22763.17613.16453.42583.12153.12963.11133.1296

All four osHC variants train stably and beat all baselines in all rows (0.035–0.053 nats at 1.1B). The bound stays depth-free (n=2.83\sqrt{n}=2.83 at n=8n=8; full Jacobian: §B.7, §C.11).

AuthorsOpenReview ↗

Revised PDF Uploaded — Exact Page and Table Pointers for Your Questions

Dear Reviewer kSbn,

As the discussion period progresses, we wanted to briefly follow up to let you know that the revised manuscript PDF has been uploaded, and to provide a quick index of where each of your points is addressed in the updated paper:

  • W1 (Theoretical scope, sharpness, and single-switch schedules):

    • Appendix §B.8, Theorem B.8 (page 17): Sharp one-step disagreement ceiling (n/2\sqrt{n/2} on Rn\mathcal R_n vs. common-mode conversion on Cn\mathcal C_n).
    • Appendix §C.6, Figure 9 (pages 20–21): Single-layer and composite spectral norms for the switch schedules.
  • W2 (Stream diversity mechanism & forward–backward adjoint duality):

    • Appendix §C.12, Table 10 (page 24): Direct forward (XlX_l) and backward (GlG_l) stream disagreement and amplification rates across all 48 mixing sites at Large, demonstrating the Rn↔Cn\mathcal R_n \leftrightarrow \mathcal C_n transpose duality.
  • W3 (Single-iteration Sinkhorn mHC (T=1) across scales):

    • Appendix §C.10, Table 8 (pages 22–23): Full-split validation losses of mHC (T=1) on FineWeb-Edu across Small, Medium, and Large (winning 8/9 seed pairs against mHC (T=20), mHC-lite, and KromHC).
  • Q1 (Scaling to n=8n=8 streams and 1.1B1.1\text{B} parameters, plus full-network Jacobians):

    • Appendix §C.8, Table 6 (page 22) & Appendix §C.9, Table 7 (page 22): Validation losses at n=8n=8 streams (Small and Medium) and at 1.1B1.1\text{B} parameters (36 layers, width 1536).
    • Appendix §B.7, Propositions B.6–B.7 (pages 16–17) & Appendix §C.11, Table 9 (page 23): Full-network Jacobian bounds and AD measurements across all 180 checkpoints.

Please let us know if these new results and clarifications address your concerns, or if there is anything else we can discuss. Thank you again for your constructive feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Manuscript Revisions

We thank all three reviewers for their feedback. The reviews converged on four questions: (1) scaling to larger models and higher stream counts, (2) direct layerwise measurements of stream diversity and mechanism analysis, (3) full-network Jacobian stability including branches and input-dependent routers, and (4) statistical uncertainty and single-iteration Sinkhorn baselines. All new experiments and proofs are in the revised paper:

  1. Scaling to 1.1B1.1\text{B} parameters and n=8n=8 streams (Reviewers uQ67, aNfj, kSbn; Appendix §C.8–C.9):

    • 1.1B1.1\text{B} parameters (36 layers, width 1536, FineWeb-Edu, seed 0): All four osHC variants (3.1113–3.12963.1113\text{--}3.1296) beat every baseline (KromHC 3.16453.1645, mHC-lite 3.17613.1761, mHC 3.22763.2276, HC 3.34793.3479, sHC 3.42583.4258) by 0.035–0.0530.035\text{--}0.053 nats.
    • n=8n=8 streams (Small & Medium, FineWeb-Edu, seed 0): All four osHC variants beat HC, mHC, KromHC, and sHC at both scales (by 0.005–0.0140.005\text{--}0.014 nats at Small and 0.012–0.0160.012\text{--}0.016 nats at Medium), while mHC-lite (8!=403208! = 40320 permutations) is intractable.
  2. Direct layerwise stream diversity & Forward–Backward Adjoint Duality (Reviewers aNfj, kSbn; Appendix §C.12): We measured relative disagreement E(⋅)/∥⋅∥F2\mathcal E(\cdot)/\lVert\cdot\rVert_F^2 and per-site amplification rates (E(HX)\mathcal E(HX) > E(X)\mathcal E(X) in forward; E(H⊤G)\mathcal E(H^\top G) > E(G)\mathcal E(G) in backward) across all 48 mixing sites at Large (8×10248\times 1024 tokens, 3 seeds) on both forward states XlX_l and backward stream gradients Gl=∂L/∂XlG_l = \partial\mathcal L/\partial X_l.

    • Preventing collapse: Unconstrained HC collapses in middle-to-deep layers (mean forward disagreement E(X)/∥X∥F2=0.16\mathcal E(X)/\lVert X\rVert_F^2 = 0.16, dropping as low as 0.040.04 in mid-layers); all constrained methods prevent this (2.2–3.1×2.2\text{--}3.1\times higher disagreement across depth).
    • Forward–Backward Adjoint Duality (Xl↦HXlX_l \mapsto HX_l vs. Gl←H⊤GlG_l \gets H^\top G_l): Because transposition swaps Rn↔Cn\mathcal R_n \leftrightarrow \mathcal C_n, exact Bn\mathcal B_n (mHC-lite, KromHC) never amplifies disagreement in either pass (0.0% sites; forward disagreement decays from 0.580.58 early to 0.21–0.220.21\text{--}0.22 at the final layer, backward mean 0.280.28). osHC-C (H∈CnH\in\mathcal C_n) amplifies forward disagreement at 64.6% of sites (0.50 mean, 0.56 final), osHC-R (H∈RnH\in\mathcal R_n) mirrors it in backward gradients (H⊤∈CnH^\top\in\mathcal C_n, 57.5% of sites, 0.36 mean), both hybrids amplify in both passes (osHC-C→R: 34.6% / 36.1%; osHC-R→C: 16.2% / 29.9%, final 0.29), and all four osHC variants sustain higher gradient disagreement than exact Bn\mathcal B_n (0.31–0.36 vs. 0.280.28). Every osHC network thus exercises one released capability forward and its adjoint backward within each step.
  3. Full-network Jacobian stability—theory and AD measurements (Reviewers aNfj, kSbn; Appendix §B.7, §C.11): We proved a scale-free bound on the input-dependent router derivative (Prop. B.6, where RMSNorm cancels activation scale) and showed that closure of one-sided matrices gives ∥JL∥2≤n∏l(1+n∥El∥2)\lVert J_L\rVert_2 \le \sqrt{n}\prod_l (1+\sqrt{n}\lVert E_l\rVert_2) for the full network Jacobian (Prop. B.7)—matching a plain residual network. Forward-mode AD across all 180 checkpoints confirms that every osHC model's full gain (mean 24.424.4 on FWE, 25.925.9 on OWT) and backward gradient ratio (mean 3838) match the residual network (25.3–26.125.3\text{--}26.1, 37–3837\text{--}38) and are slightly lower than exact Bn\mathcal B_n baselines (26.7–30.426.7\text{--}30.4, 36–5936\text{--}59), with router derivatives reducing full gain by 0.3–4.7%.

  4. Paired uncertainty statistics & mHC (T=1) baseline (Reviewers uQ67, aNfj, kSbn; Appendix §C.4, §C.10):

    • Paired seed statistics: Across the 18 (corpus, scale, seed) pairs per variant, 11 of 12 95% tt- and bootstrap CIs exclude zero; pooled across variants, osHC wins 63/7263/72 pairs vs. mHC, 66/7266/72 vs. mHC-lite, and 68/7268/72 vs. KromHC (sign-test pp < 10−1010^{-10}).
    • mHC (T=1) baseline: One Sinkhorn iteration already produces an exactly one-sided stochastic matrix (preceded by one preliminary normalization along the other axis). On FineWeb-Edu (S/M/L, 3 seeds), mHC (T=1) (3.45773.4577 / 3.23943.2394 / 3.17923.1792) beats mHC (T=20) (3.46633.4663 / 3.24123.2412 / 3.18163.1816), mHC-lite (3.46303.4630 / 3.24263.2426 / 3.18293.1829), and KromHC (3.46633.4663 / 3.24263.2426 / 3.18513.1851) at every scale (8/9 seed pairs each), while osHC matches or improves on T=1 with one softmax and no iteration count.
    • Sign condition in Theorem 3.2 (Reviewer uQ67 Q1; Appendix §B.9): We added Prop. B.9 separating the role of the two marginals (ΠH=HΠ\Pi H = H\Pi) from ∥H∥2≤1\lVert H\rVert_2 \le 1 (supplied via H≥0H \ge 0 in Bn\mathcal B_n and directly by the spectral sphere in sHC).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.