- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits whether the doubly stochastic constraint used in mHC is necessary for stabilizing residual mixing. The main idea is that fixing only one marginal--either the row sums or the column sums--is already sufficient to obtain depth-independent control of the residual-routing operator, which motivates the proposed One-Sided Hyper-Connections (osHC) with greater routing flexibility. Theoretically, the authors show that one-sided stochastic matrix families are closed under multiplication, yielding spectral-norm bounds of for pure osHC schedules and for a single row/column switch, while also arguing that the doubly stochastic constraint limits source replication, destination aggregation, and the growth of disagreement across streams. Empirically, across two pretraining corpora, three model scales (up to 354M parameters), three random seeds, and multiple routing baselines, osHC achieves the best validation loss in five of six in-domain settings and also performs strongly on five Paloma out-of-distribution corpora. Routing diagnostics further show that the learned routers actively exploit the released replication/aggregation capability.
Critical strengths
The paper develops a clean theoretical account of why one marginal can be sufficient for stable residual routing. A coherent set of empirical results support the theoretical predicitions.
Critical weaknesses
The empirical evaluation is relatively limited in scale, which leaves the practical effectiveness of osHC at larger model and stream sizes uncertain. All experiments use streams, models up to 354M parameters, and relatively short pretraining runs, so it remains unclear whether the observed gains persist in more realistic large-scale settings. This limitation is partially offset by the paper’s clean theoretical analysis, which provides a principled justification for the method beyond the current empirical regime.
Questions for authors
-
Appendix A states that the proof of Theorem 3.2 does not rely on , yet the proof in Appendix B explicitly uses non-negativity to obtain . Is the intended argument for sHC instead based on its additional spectral-norm constraint? Clarifying this would resolve an apparent inconsistency in the theoretical presentation.
-
Can the authors provide uncertainty estimates for the main validation-loss comparisons against mHC, mHC-lite, and KromHC? Since several differences are small and each setting uses only three seeds, paired differences or confidence intervals would help assess whether the reported rankings reflect robust gains.
-
Do the authors have results for stream counts other than ? Since the theoretical bounds scale as for pure osHC and for a one-switch schedule, even a limited sweep over would help assess whether the empirical advantage persists as the number of streams increases.
A clear response to these questions would help strengthen my confidence in the paper and may lead to a higher rating.
3 comments
Thank you for the detailed response. Most of my concerns have been addressed in the rebuttal, and I therefore plan to raise my score. However, the currently uploaded revised PDF does not appear to incorporate the promised updates. I look forward to seeing these corrections and additional results reflected in the final revision.
Revised PDF Uploaded — Exact Locations of Updates (Appendix A, §B.9, and Tables 3–7)
Thank you very much for reading our response and for raising your score! We have now uploaded the revised PDF incorporating all promised corrections and additional experiments. For convenience, here are the exact locations in the updated manuscript corresponding to each of your points:
-
Q1 (Role of in Theorem 3.2):
- Appendix A ("Why the three relaxations do not reach the released mechanism", page 12, lines 607–622): Corrected the discussion to distinguish the role of the two marginals () from that of (, which sHC enforces directly via ).
- Appendix §B.9, Proposition B.9 ("Balanced mixers without the sign condition", page 18): Formal statement and proof separating the two marginals from the spectral-norm bound, together with .
-
Q2 (Uncertainty estimates and paired statistics):
- Appendix §C.3, Table 3 (page 19): Complete per-seed validation losses for all 180 runs.
- Appendix §C.4, Tables 4 and 5 (pages 19–20): Paired per-seed differences across all 18 (corpus, scale, seed) pairs, 95% -intervals, 95% bootstrap intervals, win counts, and sign-test / Wilcoxon -values (Table 4), as well as per-corpus, per-scale paired differences with standard errors (Table 5).
-
Q3 (Higher stream count, ):
- Appendix §C.8, Table 6 (page 22): Full-split validation losses at streams on FineWeb-Edu (Small and Medium).
-
Weakness (Larger model scale, 1.1B parameters):
- Appendix §C.9, Table 7 (page 22): Full-split validation losses at 1.1B parameters (36 layers, width 1536) on FineWeb-Edu.
(In addition, Appendix §B.7–B.8 on pages 16–17 and Appendix §C.10–C.12, Tables 8–10 on pages 22–24 record the single-step Sinkhorn baseline, full-network Jacobian bounds and probes, and forward–backward stream diversity analyses added in response to the other reviews.)
Thank you again for your constructive feedback, which helped us improve the paper!
Theorem 3.2 Clarification, Paired Confidence Intervals, and n=8 / 1.1B Scaling
Thank you for the careful reading and the three precise questions. All were answerable with new analysis or new runs; short answers below, details in the revised appendix.
Q1. H≥0 in the proof of Theorem 3.2. You are right, and your reading is the intended one. The proof has two steps: (a) the two marginals alone give ΠH=HΠ; (b) ∥H∥2≤1, which on Bn comes from non-negativity (Riesz–Thorin). sHC keeps both marginals, so (a) holds, and it imposes ∥H∥2=1 by construction, which replaces (b). The sentence in Appendix A ("never uses H≥0") was wrong as written and is corrected. No theorem or conclusion changes.
Q2. Uncertainty estimates. All methods share seeds, data and schedule, so we report paired per-seed differences over the 18 (corpus, scale, seed) pairs: mean Δ in nats [95% t-interval], pairs won. Negative favours osHC.
Eleven of twelve intervals clearly exclude zero; the one for osHC-R→C vs mHC ends at zero. Pooled over the four variants, osHC wins 63/72 pairs against mHC, 66/72 against mHC-lite and 68/72 against KromHC (sign test p<10−10 each). Bootstrap intervals, Wilcoxon tests and per-cell differences are in new appendix tables.
Q3. Other stream counts. We trained every method at n=8 on FineWeb-Edu (Small and Medium, seed 0; validation loss, same protocol; best in bold). mHC-lite is omitted at n=8 because it needs 8!=40320 permutation coefficients.
Every osHC variant is below every other method at both scales, by 0.005–0.014 nats (Small) and 0.012–0.016 (Medium) against the best baseline. The one-switch schedules, whose ceiling is n=8 rather than 8, land in the same band as the pure ones.
Weakness: scale. We scaled up the model to 1.1B parameters (36 layers, width 1536; FineWeb-Edu, seed 0, same training protocol; best in bold):
All four osHC variants are below every baseline, by 0.035–0.053 nats against the best one (KromHC). Together with the n=8 results, this indicates that the advantage persists at larger model and stream sizes. Both tables are added to the paper.