- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the coupling between state mixing and residual writes in multi-stream residual networks through Equilibrium-coordinate Hyper-Connections (eHC), which reparameterize residual writes in terms of conditional equilibrium states. The formulation preserves additive accumulation of the stream mean while allowing mixing to regulate inter-stream differences around the current target. The theoretical analysis characterizes contraction toward a conditional target and tracking errors arising from target variation across depth. It also establishes local expressive equivalence to signed write vectors for a fixed, strictly positive doubly stochastic mixing matrix. Experiments on 125M- and 355M-parameter language models use three paired random seeds to compare eHC with mHC and an initialization-matched Signed-Direct baseline. The reported results show consistent but modest improvements across the evaluated settings.
Critical strengths
- The paper addresses a meaningful question as how state retention and residual writes jointly determine conditional equilibrium states. The theoretical analysis and experiments provide useful insights into coordinating retention and writing, with potential value for future residual architecture design.
- eHC couples residual writes with state mixing through a straightforward reparameterization, without changing the computational sublayers or residual width. The additional computation is confined to low-dimensional connection coefficients, increasing training step time by only 0.7% over mHC.
Critical weaknesses
- The conditional target contraction property is not unique to eHC as under the assumptions of Proposition 2, Signed-Direct and mHC can also be assigned conditional equilibrium targets and satisfy the same form of contraction relation. These theoretical results therefore do not by themselves explain eHC’s performance advantage, and comparisons of target variation and tracking errors across methods are needed to substantiate the proposed mechanism.
- Section 5.4 compares the two intervention strategies only within trained eHC models, without corresponding experiments on the baselines. The same equilibrium-preserving intervention can be applied directly to Signed-Direct and mHC by adjusting their write coefficients, without retraining. The current results therefore do not establish whether the reduced predictive sensitivity reflects an advantage specific to eHC’s learning dynamics.
- The theoretical analysis assumes exactly doubly stochastic mixing matrices, whereas the implementation uses a fixed 20 Sinkhorn iterations, which do not automatically guarantee the required contraction property. Although the appendix discusses finite-iteration errors, it does not report normalization residuals or contraction measurements for the matrices encountered during training. The applicability of the theoretical guarantees to the implementation therefore remains insufficiently verified, particularly regarding the sensitivity of the main results to the number of Sinkhorn iterations.
Questions for authors
- The scope of the theoretical claims should be clarified, particularly which conclusions distinguish eHC from mHC and Signed-Direct, since both baselines admit conditional equilibrium targets with the same contraction form.
- Section 5.4 should clarify whether the observed sensitivity reduction demonstrates a general benefit of write compensation or an advantage specific to eHC.
- The paper should clarify whether the matrices produced by 20 Sinkhorn iterations satisfy the contraction conditions in practice, supported by normalization residuals or spectral measurements.
- eHC-Detached removes gradients to both the mixing-head parameters and the input state through the mixing head. Separating these contributions while preserving identical forward computation would clarify whether the improvement primarily comes from mixing-parameter optimization.
3 comments
Response to Reviewer M2Vg (2/2): Finite Sinkhorn Verification, Gradient Routing, and Clarity
[W3 & Q3: Finite 20-Iteration Sinkhorn Residuals, Contraction Norms, and Precision Sensitivity]
Following mHC (Xie et al., ICML 2026, Sec. 5.4), our standard runs use 20 FP32 Sinkhorn iterations, which do not by themselves guarantee exact doubly stochastic margins or a uniform strict contraction bound. To verify how closely the implementation satisfies these conditions and whether iteration precision affects our results, we:
- Clarified finite-iteration expressions (Appendix A.13): Eq. 57 expresses the realized disagreement update as with row residual , and mean residual with .
- Measured marginal residuals and operator norms (Table 13): Across evaluated mixing matrices in trained models, we recorded row/column sum residuals and the transverse operator norm (the singular/operator norm on disagreement).
- Ran a higher-precision Sinkhorn control at 125M (Table 14):
| Method | Row Resid. P99 / Max () | Col Resid. P99 / Max () | P99 | Max | Matrices (%) | High-Prec. 125M NLL (vs. 20-iter) |
|---|---|---|---|---|---|---|
| mHC | 1.19 / 3.58 | 1.84 / 91.7 | 0.9826 | 1.00034 | 0.023% | 3.3466 (3.3467) |
| Signed-Direct | 1.19 / 2.38 | 1.61 / 82.6 | 0.9808 | 1.00029 | 0.018% | 3.3436 (3.3435) |
| eHC | 1.19 / 2.38 | 1.47 / 76.3 | 0.9789 | 1.00024 | 0.014% | 3.3393 (3.3394) |
- Practical contraction: holds on 99.977% (mHC), 99.982% (
Signed-Direct), and 99.986% (eHC) of evaluated mixing matrices, with maxima at most ( for eHC), P99 row residuals of , and P99 column residuals . - Sensitivity to Sinkhorn precision: Higher-precision Sinkhorn scaling (+8.7% relative eHC step time, vs. ) leaves 125M test NLLs within nats/token of the 20-iteration runs, with eHC's gains over mHC and
Signed-Directat 0.0073 / 0.0043 nats/token versus 0.0073 / 0.0041 nats/token at 20 iterations.
[Q4: Separating Mixing-Parameter vs. Input-State Gradients Through ]
In eHC, the write branch contributes to the mixing-coefficient adjoint (Eqs. 16, 21), while ordinary state transport remains active across all variants. Because depends on both mixing-head parameters and the routing input, we gated write-branch gradients to each pathway independently while keeping the exact same forward function at identical weights (Section 5.3, Tables 4, 17):
| Variant (125M) | Write Grad Mixing Params | Write Grad Input | Test NLL (nats/token) | Paired Reduction vs. Detached ( nats/token) |
|---|---|---|---|---|
| Signed-Direct | — | — | 3.3435 | — |
Detached (00) | Off | Off | 3.3410 | 0.0 (reference) |
InputOnly (01) | Off | On | 3.3406 | 0.4 |
ParamOnly (10) | On | Off | 3.3399 | 1.1 |
Full (11) | On | On | 3.3394 | 1.6 |
Forward coordinate adaptation alone (Signed-Direct Detached) lowers test NLL by 0.0025 nats/token. Relative to Detached, restoring write-branch gradients to the mixing-head parameters (ParamOnly, 1.1e-3 nats/token) yields a larger individual reduction than the input-state pathway (InputOnly, 0.4e-3 nats/token), and keeping both pathways active (Full, 1.6e-3 nats/token) achieves the lowest test NLL.
[Presentation]
To make the paper substantially easier to follow, we rewrote the abstract and main prose—adding a concrete two-stream circuit walkthrough in Section 2, separating fixed- expressivity from differential mixing sensitivity in Sections 4.1–4.3, clarifying finite-iteration formulas in Appendix A.13, and presenting the new diagnostics, cross-method interventions, Sinkhorn measurements, gradient-routing ablations, and scaling runs (up to ≈1.3B parameters / ≈26B tokens, 0.0171 nats/token over mHC) in Sections 5.2–5.5 and Appendix D.
Follow-up Note to Reviewer M2Vg
Dear Reviewer M2Vg,
We apologize for the late timing of our reply. Running additional language model pretraining experiments for our controls and ablations was computationally intensive and took time to complete.
We greatly appreciate your careful technical comments, particularly your observation that equilibrium-preserving write compensation can be tested directly on the trained baselines without retraining. As soon as your review arrived, we reconsidered the presentation and scope of our theoretical claims, reviewed our earlier experimental records, and ran the cross-method write compensation tests, finite Sinkhorn normalization checks, and ablations of the gradient paths to the mixing parameters and to the input state. Your feedback has been instrumental in making the paper much clearer and more thorough.
Please let us know if you have any further questions or suggestions.
Response to Reviewer M2Vg (1/2): Theoretical Scope, Cross-Method Diagnostics, and Interventions
We thank Reviewer
M2Vgfor the incisive technical critique and for appreciating both our analysis of coordinating stream retention with writing and eHC's minimal runtime overhead. Below we address each theoretical question, cross-method diagnostic, and ablation in the revised manuscript.[W1 & Q1: Scope of Theoretical Claims & Cross-Method Target Diagnostics]
At a fixed strictly positive doubly stochastic P, any baseline write w=g1+b (with centered b) admits an implicit target q∗=(I−P+J)−1b on the centered subspace and T∗=q∗f satisfying the contraction identity D+−T∗=P(D−T∗). Sections 4.1–4.3 state clearly which properties are shared at fixed P and what distinguishes eHC:
Signed-Direct(whereas mHC's componentwise wi=2σ(zi)∈(0,2) excludes negative entries and entries >2).Table 10 in Appendix D compares target variation and tracking error across the three methods' trained checkpoints:
In trained models, measured similarity between adjacent sublayer outputs ranges from Cf=0.214 to 0.226, and eHC achieves lower target variation (VT=0.611) and lower tracking error (ET=0.779) while sustaining larger target magnitude (MT=0.918) and larger stream disagreement energy (ED=0.0487).
[W2 & Q2: Equilibrium-Preserving Interventions Across Baselines]
Under Pη=(1−η)P+ηI (Eq. 22), I−Pη=(1−η)(I−P). Scaling any baseline's centered write via bη=(1−η)b while preserving mean gain g satisfies (I−Pη)q∗=(1−η)b, changing the local state by η(I−P)(D−T∗) rather than η(I−P)D (Eq. 23) without a native q head or retraining. Section 5.4 and Table 11 report output predictive KL and ΔNLL on the same corpus, positions, and intervention across trained Base-355 checkpoints at η=0.025 (common KL scale sets eHC fixed = 1.000):
This directly answers Q2: (1) write compensation provides a general benefit across all three methods, lowering output KL and ΔNLL for mHC,
Signed-Direct, and eHC alike; and (2) trained eHC checkpoints show the lowest baseline sensitivity and the largest observed relative reduction (37.3% in output KL, alongside a 0.607 local perturbation-energy ratio in Eq. 24 & Table 12), consistent with their lower learned tracking error ET.