acceptodds
Under review as a conference paper at ICLR 2027

Equilibrium Updates for Residual Learning

Abstract

Multi-stream residual networks combine mechanisms for retaining intermediate state and injecting sublayer output. When mixing and writing are independently parameterized, changing retention also changes the state maintained by repeated input. This paper introduces Equilibrium-coordinate Hyper-Connections (eHC), which parameterize residual writes via this conditional equilibrium using the existing write head. Given a doubly stochastic mixing matrix , eHC writes , sharing a gain and zero-mean coordinates . This separates additive accumulation of the stream mean from target-centered disagreement dynamics, making the direct sensitivity to mixing dependent on target error. For fixed positive mixing, equilibrium coordinates are in one-to-one correspondence with signed affine writes, so eHC changes the learning of equivalent writes without changing their local expressive capacity. Controlled experiments with language models indicate improved language modeling and zero-shot transfer compared to mHC and a matched signed-write baseline with little additional training cost. Gradient ablations identify benefits of adaptive write coordinates and their gradient coupling to mixing, and retention interventions reveal that preserving the target reduces predictive sensitivity. These results suggest that parameterizing residual writes by the states they should sustain, rather than learning injection independently of retention, may improve residual networks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
residual connections, hyper-connections, equilibrium parameterization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M2VgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the coupling between state mixing and residual writes in multi-stream residual networks through Equilibrium-coordinate Hyper-Connections (eHC), which reparameterize residual writes in terms of conditional equilibrium states. The formulation preserves additive accumulation of the stream mean while allowing mixing to regulate inter-stream differences around the current target. The theoretical analysis characterizes contraction toward a conditional target and tracking errors arising from target variation across depth. It also establishes local expressive equivalence to signed write vectors for a fixed, strictly positive doubly stochastic mixing matrix. Experiments on 125M- and 355M-parameter language models use three paired random seeds to compare eHC with mHC and an initialization-matched Signed-Direct baseline. The reported results show consistent but modest improvements across the evaluated settings.

Critical strengths
  1. The paper addresses a meaningful question as how state retention and residual writes jointly determine conditional equilibrium states. The theoretical analysis and experiments provide useful insights into coordinating retention and writing, with potential value for future residual architecture design.
  2. eHC couples residual writes with state mixing through a straightforward reparameterization, without changing the computational sublayers or residual width. The additional computation is confined to low-dimensional connection coefficients, increasing training step time by only 0.7% over mHC.
Critical weaknesses
  1. The conditional target contraction property is not unique to eHC as under the assumptions of Proposition 2, Signed-Direct and mHC can also be assigned conditional equilibrium targets and satisfy the same form of contraction relation. These theoretical results therefore do not by themselves explain eHC’s performance advantage, and comparisons of target variation and tracking errors across methods are needed to substantiate the proposed mechanism.
  2. Section 5.4 compares the two intervention strategies only within trained eHC models, without corresponding experiments on the baselines. The same equilibrium-preserving intervention can be applied directly to Signed-Direct and mHC by adjusting their write coefficients, without retraining. The current results therefore do not establish whether the reduced predictive sensitivity reflects an advantage specific to eHC’s learning dynamics.
  3. The theoretical analysis assumes exactly doubly stochastic mixing matrices, whereas the implementation uses a fixed 20 Sinkhorn iterations, which do not automatically guarantee the required contraction property. Although the appendix discusses finite-iteration errors, it does not report normalization residuals or contraction measurements for the matrices encountered during training. The applicability of the theoretical guarantees to the implementation therefore remains insufficiently verified, particularly regarding the sensitivity of the main results to the number of Sinkhorn iterations.
Questions for authors
  1. The scope of the theoretical claims should be clarified, particularly which conclusions distinguish eHC from mHC and Signed-Direct, since both baselines admit conditional equilibrium targets with the same contraction form.
  2. Section 5.4 should clarify whether the observed sensitivity reduction demonstrates a general benefit of write compensation or an advantage specific to eHC.
  3. The paper should clarify whether the matrices produced by 20 Sinkhorn iterations satisfy the contraction conditions in practice, supported by normalization residuals or spectral measurements.
  4. eHC-Detached removes gradients to both the mixing-head parameters and the input state through the mixing head. Separating these contributions while preserving identical forward computation would clarify whether the improvement primarily comes from mixing-parameter optimization.
3 comments
AuthorsOpenReview ↗

Response to Reviewer M2Vg (1/2): Theoretical Scope, Cross-Method Diagnostics, and Interventions

We thank Reviewer M2Vg for the incisive technical critique and for appreciating both our analysis of coordinating stream retention with writing and eHC's minimal runtime overhead. Below we address each theoretical question, cross-method diagnostic, and ablation in the revised manuscript.

[W1 & Q1: Scope of Theoretical Claims & Cross-Method Target Diagnostics]

At a fixed strictly positive doubly stochastic PP, any baseline write w=g1+bw = g\mathbf{1} + b (with centered bb) admits an implicit target q∗=(I−P+J)−1bq^* = (I - P + J)^{-1}b on the centered subspace and T∗=q∗fT^* = q^* f satisfying the contraction identity D+−T∗=P(D−T∗)D^+ - T^* = P(D - T^*). Sections 4.1–4.3 state clearly which properties are shared at fixed PP and what distinguishes eHC:

  • Shared at fixed PP (Props. 1 & 2): Proposition 1 establishes contraction for a fixed target. Section 4.1 then gives the moving-target identity el+1=Plel+Tl−Tl+1e_{l+1} = P_l e_l + T_l - T_{l+1} (Eq. 12) and tracking bound under uniform contraction ∥Pl−J∥2≤ρ<1\\\|P_l - J\\\|_2 \le \rho < 1 and bounded target drift (Eq. 13). Proposition 2 (Eq. 14) gives the fixed-PP bijection between equilibrium coordinates and signed writes of mean g∈(0,2)g \in (0,2)—spanning the exact same signed-write space as Signed-Direct (whereas mHC's componentwise wi=2σ(zi)∈(0,2)w_i = 2\sigma(z_i) \in (0,2) excludes negative entries and entries >2>2).
  • What distinguishes eHC (Prop. 3): When PP varies along an admissible direction (dP1=0,1⊤dP=0dP\mathbf{1}=0, \mathbf{1}^\top dP=0) while a baseline's centered write bb is held fixed, (I−P)q∗=b(I-P)q^* = b implies (I−P)dq∗=dP,q∗(I-P)dq^* = dP\\,q^*, and the one-step state variation is dP,DdP\\,D, which can be nonzero even at D=T∗D = T^*. In eHC, predicting qq and setting b=(I−P)qb = (I-P)q induces the compensating write differential db=−dP,qdb = -dP\\,q for fixed qq, yielding the innovation-weighted variation dP(D−T)dP(D - T) (Eq. 15), which is zero at D=TD = T for every admissible dPdP (even when the ambient coefficient adjoint is nonzero).

Table 10 in Appendix D compares target variation and tracking error across the three methods' trained checkpoints:

MethodAdj.-Output Sim. CfC_fTarget Mag. MTM_TTarget Var. VTV_TTracking Err. ETE_TDisagreement Energy EDE_D
mHC0.2140.8420.6760.8730.0376
Signed-Direct0.2190.8750.6490.8460.0419
eHC0.2260.9180.6110.7790.0487

In trained models, measured similarity between adjacent sublayer outputs ranges from Cf=0.214C_f = 0.214 to 0.2260.226, and eHC achieves lower target variation (VT=0.611V_T = 0.611) and lower tracking error (ET=0.779E_T = 0.779) while sustaining larger target magnitude (MT=0.918M_T = 0.918) and larger stream disagreement energy (ED=0.0487E_D = 0.0487).

[W2 & Q2: Equilibrium-Preserving Interventions Across Baselines]

Under Pη=(1−η)P+ηIP_\eta = (1-\eta)P + \eta I (Eq. 22), I−Pη=(1−η)(I−P)I - P_\eta = (1-\eta)(I-P). Scaling any baseline's centered write via bη=(1−η)bb_\eta = (1-\eta)b while preserving mean gain gg satisfies (I−Pη)q∗=(1−η)b(I-P_\eta)q^* = (1-\eta)b, changing the local state by η(I−P)(D−T∗)\eta(I-P)(D-T^*) rather than η(I−P)D\eta(I-P)D (Eq. 23) without a native qq head or retraining. Section 5.4 and Table 11 report output predictive KL and Δ\DeltaNLL on the same corpus, positions, and intervention across trained Base-355 checkpoints at η=0.025\eta = 0.025 (common KL scale sets eHC fixed = 1.000):

Method (Base-355, η=0.025\eta=0.025)Fixed KLCompensated KLWithin-Method KL Ratio (Reduction)Fixed Δ\DeltaNLLCompensated Δ\DeltaNLL
mHC1.1120.8320.748 (25.2%)0.00610.0044
Signed-Direct1.0610.7440.701 (29.9%)0.00560.0037
eHC1.0000.6270.627 (37.3%)0.00490.0028

This directly answers Q2: (1) write compensation provides a general benefit across all three methods, lowering output KL and Δ\DeltaNLL for mHC, Signed-Direct, and eHC alike; and (2) trained eHC checkpoints show the lowest baseline sensitivity and the largest observed relative reduction (37.3% in output KL, alongside a 0.607 local perturbation-energy ratio in Eq. 24 & Table 12), consistent with their lower learned tracking error ETE_T.

AuthorsOpenReview ↗

Response to Reviewer M2Vg (2/2): Finite Sinkhorn Verification, Gradient Routing, and Clarity

[W3 & Q3: Finite 20-Iteration Sinkhorn Residuals, Contraction Norms, and Precision Sensitivity]

Following mHC (Xie et al., ICML 2026, Sec. 5.4), our standard runs use 20 FP32 Sinkhorn iterations, which do not by themselves guarantee exact doubly stochastic margins or a uniform strict contraction bound. To verify how closely the implementation satisfies these conditions and whether iteration precision affects our results, we:

  1. Clarified finite-iteration expressions (Appendix A.13): Eq. 57 expresses the realized disagreement update as D+=T+ΠP~Π(D−T)+ΠrxˉD^+ = T + \Pi\tilde{P}\Pi(D-T) + \Pi r\bar{x} with row residual r=P~1−1r = \tilde{P}\mathbf{1}-\mathbf{1}, and mean residual c⊤(X−T)c^\top(X-T) with c⊤=1⊤(P~−I)/nc^\top = \mathbf{1}^\top(\tilde{P}-I)/n.
  2. Measured marginal residuals and operator norms (Table 13): Across evaluated mixing matrices in trained models, we recorded row/column sum residuals and the transverse operator norm ρ⊥=∥ΠP~Π∥2\rho_\perp = \\\|\Pi\tilde{P}\Pi\\\|_2 (the singular/operator norm on disagreement).
  3. Ran a higher-precision Sinkhorn control at 125M (Table 14):
MethodRow Resid. P99 / Max (10−710^{-7})Col Resid. P99 / Max (10−510^{-5})ρ⊥\rho_\perp P99ρ⊥\rho_\perp MaxMatrices ρ⊥≥1\rho_\perp \ge 1 (%)High-Prec. 125M NLL (vs. 20-iter)
mHC1.19 / 3.581.84 / 91.70.98261.000340.023%3.3466 (3.3467)
Signed-Direct1.19 / 2.381.61 / 82.60.98081.000290.018%3.3436 (3.3435)
eHC1.19 / 2.381.47 / 76.30.97891.000240.014%3.3393 (3.3394)
  • Practical contraction: ρ⊥<1\rho_\perp < 1 holds on 99.977% (mHC), 99.982% (Signed-Direct), and 99.986% (eHC) of evaluated mixing matrices, with maxima at most 1.000341.00034 (1.000241.00024 for eHC), P99 row residuals of 1.19×10−71.19\times 10^{-7}, and P99 column residuals ≤1.84×10−5\le 1.84\times 10^{-5}.
  • Sensitivity to Sinkhorn precision: Higher-precision Sinkhorn scaling (+8.7% relative eHC step time, 1.087×1.087\times vs. 1.000×1.000\times) leaves 125M test NLLs within 10−410^{-4} nats/token of the 20-iteration runs, with eHC's gains over mHC and Signed-Direct at 0.0073 / 0.0043 nats/token versus 0.0073 / 0.0041 nats/token at 20 iterations.

[Q4: Separating Mixing-Parameter vs. Input-State Gradients Through (I−Pl)ql(I-P_l)q_l]

In eHC, the write branch (I−Pl)ql(I-P_l)q_l contributes −(Gf⊤)q⊤-(Gf^\top)q^\top to the mixing-coefficient adjoint (Eqs. 16, 21), while ordinary state transport PlXlP_l X_l remains active across all variants. Because PlP_l depends on both mixing-head parameters and the routing input, we gated write-branch gradients to each pathway independently while keeping the exact same forward function at identical weights (Section 5.3, Tables 4, 17):

Variant (125M)Write Grad →\to Mixing ParamsWrite Grad →\to InputTest NLL (nats/token)Paired Reduction vs. Detached (10−310^{-3} nats/token)
Signed-Direct——3.3435—
Detached (00)OffOff3.34100.0 (reference)
InputOnly (01)OffOn3.34060.4
ParamOnly (10)OnOff3.33991.1
Full (11)OnOn3.33941.6

Forward coordinate adaptation alone (Signed-Direct →\to Detached) lowers test NLL by 0.0025 nats/token. Relative to Detached, restoring write-branch gradients to the mixing-head parameters (ParamOnly, 1.1e-3 nats/token) yields a larger individual reduction than the input-state pathway (InputOnly, 0.4e-3 nats/token), and keeping both pathways active (Full, 1.6e-3 nats/token) achieves the lowest test NLL.

[Presentation]

To make the paper substantially easier to follow, we rewrote the abstract and main prose—adding a concrete two-stream circuit walkthrough in Section 2, separating fixed-PP expressivity from differential mixing sensitivity in Sections 4.1–4.3, clarifying finite-iteration formulas in Appendix A.13, and presenting the new diagnostics, cross-method interventions, Sinkhorn measurements, gradient-routing ablations, and scaling runs (up to ≈1.3B parameters / ≈26B tokens, 0.0171 nats/token over mHC) in Sections 5.2–5.5 and Appendix D.

AuthorsOpenReview ↗

Follow-up Note to Reviewer M2Vg

Dear Reviewer M2Vg,

We apologize for the late timing of our reply. Running additional language model pretraining experiments for our controls and ablations was computationally intensive and took time to complete.

We greatly appreciate your careful technical comments, particularly your observation that equilibrium-preserving write compensation can be tested directly on the trained baselines without retraining. As soon as your review arrived, we reconsidered the presentation and scope of our theoretical claims, reviewed our earlier experimental records, and ran the cross-method write compensation tests, finite Sinkhorn normalization checks, and ablations of the gradient paths to the mixing parameters and to the input state. Your feedback has been instrumental in making the paper much clearer and more thorough.

Please let us know if you have any further questions or suggestions.

Reviewer_WR8bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends Hyper-Connections, where Xl∈Rn×CX_l \in \mathbb{R}^{n\times C} is the llth layer hidden state for one token with nn lanes and CC as hidden dimension, and Xl+1=PlXl+wlFl(rl⊤Xl).X_{l+1} = P_l X_l + w_l F_l(r_l^\top X_l). Here, rlr_l writes nn lanes into RC\mathbb{R}^C, FlF_l is the standard Transformer layer, wlw_l writes the output into nn lanes again, PlP_l mixes the nn lanes in llth layer. Building on mHC, the paper further proposes a constraint on mixing PlP_l and writing wlw_l, such that the equilibrium of Xl (l→∞)X_l~(l \to \infty) will be the same if Fl(rl⊤Xl)F_l(r_l^\top X_l) is the same across ll.

Without the constraint, the equilibrium of Xl (l→∞)X_l~(l \to \infty) can be different by changing PlP_l and wlw_l independently even when Fl(rl⊤Xl)F_l(r_l^\top X_l) are all the same. The new method, called Equilibrium-coordinate Hyper-Connections (eHC), shows a better validation loss when compared to mHC and other baselines.

Critical strengths
  1. The paper is well-motivated, well-organized, and clearly written. Following the motivation, the proposed method is introduced and then tested empirically. The proposal is reasonable and intuitive, with a small but statistically significant improvement in validation loss.
Critical weaknesses
  1. The theoretical motivation/justification for eHC is weak. In theory, eHC will work better if the injections from different layers to the residual are the same, and the hidden state does want to reach an equilibrium that only depends on the repeated injection. In reality, at least from this work, there is no justification that injections are similar across many layers; there is also no justification that under the same injection, it is better to have the same equilibrium. Furthermore, as far as I understand, mHC has a higher degree of freedom and can, in principle, reduce to eHC under some parameter. To conclude, the advantage of eHC is not guaranteed in theory.
  2. The empirical improvement is weak. The empirical improvement from mHC to eHC is around 0.01 nats/token, at steps around 10410^4. Based on my experience, this is still at a relatively early stage of pre-training. The value 0.01 is very small compared to the reducible part of the loss.

If the empirical improvement is significant, even without any theory, it is important and should be reported. If the theory is strong, even if the observed improvement is small, we can be confident that this improvement will be persistent and generalizable; then it is also fine. The current paper has both theory and experiments, but neither side seems to be strong enough for acceptance. I hold a positive view of this paper, but I think that on both theory and experiment side, there may need more evidence.

Questions for authors
  1. Why do you consider the situation to be persistent inputs/injections to the residual stream? Have you or previous work demonstrated this is very often?
  2. Why does convergence to an equilibrium independent of writing and mixing become theoretically better in the presence of persistent inputs?
  3. Can mHC learn a parametrization such that it is equivalent to eHC? If so, does eHC constrain the search space to enable faster training?
  4. In your pre-training experiments, have you scanned hyperparameters for eHC and mHC independently and compared the optimal performance of eHC with the optimal performance of mHC? I am asking this because the hyperparameters you used can be good for eHC but not optimal for mHC, leading to an unfair comparison.
  5. Can you provide some criteria for improvement? Like how different in nats/token will be considered as significant?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WR8b (1/2): Theoretical Justification, Expressivity, and Non-Persistent Inputs

We thank Reviewer WR8b for the thoughtful review and for highlighting the clarity, organization, and intuitive motivation of our formulation. Below we address your theoretical questions on non-persistent sublayer injections and expressive capacity, followed in Post 2/2 by the new experiments on independent tuning, longer horizons, larger scale, and evaluation criteria.

[W1, Q1, Q2: Why We Analyze Frozen Inputs, and Why Decoupling Helps Under Changing Sublayers]

  1. Why we examine persistent inputs (Q1): In the trained checkpoints, adjacent sublayer output similarity is Cf=0.226C_f = 0.226 for eHC and 0.2140.214 for mHC (Table 10). Much as recurrence analyses examine repeated-input scaling (e.g., Orvieto et al., ICML 2023), holding the input fixed in Section 1 is an analytic probe of a single layer's parameterization: it reveals that when retention PlP_l and differential write blb_l are parameterized independently, changing PlP_l can change the effective scale (I−Pl+J)−1bl(I-P_l+J)^{-1}b_l of that layer's write.
  2. Actual update across distinct sublayers (W1): In the network, incoming state Xl=1xˉl+DlX_l = \mathbf{1}\bar{x}_l + D_l splits into the stream mean xˉl\bar{x}_l, which accumulates xˉl+1=xˉl+glfl\bar{x}_{l+1} = \bar{x}_l + g_l f_l additively, and zero-mean disagreement DlD_l, for which untied sublayer ll sets its own local target Tl=qlflT_l = q_l f_l and computes Dl+1=Tl+Pl(Dl−Tl)D_{l+1} = T_l + P_l(D_l - T_l) (Eq. 7). Across any sequence of distinct targets TlT_l, tracking error el=Dl−Tle_l = D_l - T_l obeys the exact identity el+1=Plel+Tl−Tl+1e_{l+1} = P_l e_l + T_l - T_{l+1} (Eq. 12). When ∥Pl−J∥2≤ρ<1\\\|P_l - J\\\|_2 \le \rho < 1 uniformly and ∥Tl+1−Tl∥F≤δ\\\|T_{l+1}-T_l\\\|_F \le \delta, ∥eL∥F≤ρL∥e0∥F+δ1−ρL1−ρ\\\|e_L\\\|_F \le \rho^L \\\|e_0\\\|_F + \delta\frac{1-\rho^L}{1-\rho} (Eq. 13; persistent targets δ=0\delta=0 are one special case). Empirically (Table 10), eHC learns lower target variation (VT=0.611V_T = 0.611 vs. 0.6760.676 in mHC) and lower tracking error (ET=0.779E_T = 0.779 vs. 0.8730.873) while maintaining larger stream disagreement energy (ED=0.0487E_D = 0.0487 vs. 0.03760.0376).
  3. Why decoupling is advantageous even in one step (Q2, Proposition 3): Consider a single sublayer with local inputs and target held fixed while PlP_l is perturbed along any admissible doubly stochastic direction (dPl1=0,1⊤dPl=0dP_l\mathbf{1}=0, \mathbf{1}^\top dP_l=0). With an independent write held fixed, the updated state shifts by dPlDldP_l D_l, which can be nonzero even when Dl=TlD_l = T_l—so adjusting mixing disturbs an already-attained target. With eHC's write bl=(I−Pl)qlb_l = (I-P_l)q_l, the induced write change dbl=−dPlqldb_l = -dP_l q_l makes the state shift innovation-weighted: dPl(Dl−Tl)dP_l(D_l - T_l) (Eq. 15), which is zero at Dl=TlD_l = T_l for every admissible dPldP_l. Consequently, PlP_l controls how rapidly prior mismatch decays without rescaling the target itself, and the write-branch gradient directly informs PlP_l optimization (tested in Table 4).

[W1 & Q3: Can mHC Reduce to eHC, and Does eHC Constrain the Search Space?]

eHC does not constrain local write capacity, and mHC does not subsume eHC:

  • Identical parameter count: mHC, Signed-Direct, and eHC all predict the exact same n2+2nn^2+2n connection scalars per sublayer.
  • Larger signed-write space than mHC (Proposition 2): mHC's componentwise sigmoid wi=2σ(zi)w_i = 2\sigma(z_i) restricts every stream entry to (0,2)(0, 2), so all streams receive same-sign writes. In eHC (with mean gain g=2σ(zˉ)∈(0,2)g = 2\sigma(\bar{z}) \in (0,2) and zero-sum qq), individual entries of w=g1+(I−P)qw = g\mathbf{1} + (I-P)q can be negative or above 2 while keeping the stream mean in (0,2)(0, 2). At any fixed strictly positive doubly stochastic PP, Proposition 2 (Eq. 14) gives a one-to-one bijection q∗=(I−P+J)−1(w−g1)q^* = (I-P+J)^{-1}(w - g\mathbf{1}) between eHC's coordinates and all signed writes of mean g∈(0,2)g \in (0, 2).
  • Pointwise matching vs. training dynamics: At a fixed PP, mHC can match individual writes whose entries all lie in (0,2)(0, 2), and our Signed-Direct baseline spans the exact same full signed-write space as eHC with matched initialization and initial write-head Jacobian. However, pointwise agreement at one PP does not reproduce eHC's coupled derivative db=−dP,qdb = -dP\\,q as PP updates during training. Thus eHC improves optimization by changing the coordinate geometry linking ww and PP, rather than by shrinking the search space.
AuthorsOpenReview ↗

Response to Reviewer WR8b (2/2): Independent Tuning, Extended Scaling, and Significance Criteria

[Q4: Independent Hyperparameter Tuning]

Following your suggestion, we independently tuned hyperparameters for mHC, Signed-Direct, and eHC at 355M (2.147B tokens; Tables 2, 16). Method-specific tuning improved both baselines (mHC: 3.1527→3.15123.1527 \to 3.1512; Signed-Direct: 3.1488→3.14713.1488 \to 3.1471) and eHC (3.1436→3.14193.1436 \to 3.1419), with eHC maintaining gains of 0.0093 nats/token over mHC and 0.0052 nats/token over Signed-Direct.

[W2: Longer Pretraining Horizons and Larger Model Scale]

To test whether eHC's advantage persists well past early pretraining, we ran independently tuned 355M models for 4x longer (8.590B tokens, ≈24.2 tokens/parameter) and trained a ≈1.3B-parameter configuration for ≈26B tokens (≈20 tokens/parameter, comparable to the compute-optimal allocation of Hoffmann et al., NeurIPS 2022) (Tables 2, 16):

ConfigurationTokens (B)mHC NLLSigned-Direct NLLeHC NLLGain vs. mHC (nats)Gain vs. Signed-Direct (nats)
125M1.0743.34673.34353.33940.00730.0041
355M2.1473.15273.14883.14360.00910.0052
355M (tuned)2.1473.15123.14713.14190.00930.0052
355M (tuned, long)8.5902.77192.76782.75920.01270.0086
≈1.3B≈26.02.60162.59482.58450.01710.0103

Extending tuned 355M training from 2.147B to 8.590B tokens widens eHC's held-out NLL gain over mHC from 0.0093 to 0.0127 nats/token (and from 0.0052 to 0.0086 over Signed-Direct), and the ≈1.3B / ≈26B comparison shows a 0.0171 nats/token gain over mHC (0.0103 over Signed-Direct).

[Q5: Criteria for Statistical and Practical Significance]

Because there is no single universal cutoff in nats/token—especially for a stream-space reparameterization adding zero parameters and +0.67% step time at standard 355M—we evaluate significance on three concrete criteria:

  1. Statistical precision (paired 95% confidence intervals, Table 16): Every primary paired interval excludes zero across all five configurations:
    • 125M: 0.0073 nats/token vs. mHC; 0.0041 vs. Signed-Direct
    • 355M (original): 0.0091 vs. mHC; 0.0052 vs. Signed-Direct
    • 355M (tuned): 0.0093 vs. mHC; 0.0052 vs. Signed-Direct
    • 355M (tuned, 8.59B): 0.0127 vs. mHC; 0.0086 vs. Signed-Direct
    • ≈1.3B (≈26B): 0.0171 [0.0149, 0.0193] vs. mHC (1 - \exp(-0.0171) \approx \mathbf{1.70\\%} relative perplexity reduction); 0.0103 vs. Signed-Direct
  2. Training compute to a shared loss target (Table 5): On the 8.590B-token 355M runs (throughput 214,030 / 214,591 / 212,607 tokens/s for mHC / Signed-Direct / eHC), eHC reaches the target development NLL in 9.57 GPU-hours vs. 9.84 (Signed-Direct) and 10.04 (mHC)—saving 4.7% GPU-hours—and achieves lower common-time test NLL (2.7597 vs. 2.7678 and 2.7722).
  3. Downstream transfer and zero-shot accuracy: On standard 355M checkpoints (Table 3), eHC improves held-out perplexity on WikiText-103 and C4 and increases accuracy on each of four zero-shot tasks (mean 39.1% vs. 38.6% Signed-Direct and 38.2% mHC). On the 8.590B-token 355M checkpoints (Table 5), 4-task zero-shot accuracy improves from 43.9% (mHC) and 44.4% (Signed-Direct) to 45.1% (eHC), a +1.2 percentage-point gain over mHC.
AuthorsOpenReview ↗

Follow-up Note to Reviewer WR8b

Dear Reviewer WR8b,

We sincerely apologize for replying so late in the rebuttal period. Pretraining language models requires substantial compute, and finishing the new runs took time.

We greatly appreciate your constructive review and your clear framing of what was needed on both the theoretical and empirical sides. Right after your comments arrived, we reconsidered how we explain non-persistent sublayer injections and local write capacity, reviewed our earlier experimental records, and carried out the independent hyperparameter tuning and longer-horizon pretraining comparisons you suggested. Your feedback has been invaluable in strengthening the paper.

We warmly welcome any follow-up questions or suggestions.

Reviewer_qtNnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a variant to manifold-constrained Hyper-Connections (mHC) based on stabilizing the target magnitude of the residual stream compared to past approaches. The authors motivate their approach through theoretical analysis and small-scale pretraining experiments.

Critical strengths
  • Principled approach: The method stems from an interesting perspective on mHC geometry in terms of mean contribution and "disagreement". The proposed decoupling is a relevant solution to this concern. Documenting this perspective is interesting in itself.
  • Timely topic: Improving Hyper-Connections and related method is a promising avenue for architectural improvements.
Critical weaknesses
  • Small-scale experiments and marginal improvements: The experiments consist in 125M and 355M models trained on 2B tokens. This is a tiny scale, and does not clearly validate the relevance of eHC in general. The observed benefits are marginal (<5e-3 nats), for a method that introduces architectural complexity. Moreover, some technical details are not ideal: the chosen batch size is unusually small for pretraining experiments (~100,000 tokens vs. 500,000-4M for usual runs), and warm-up is relatively short compared to SmolLM2 runs for instance.
  • Missing empirical analysis: The authors diagnose a theoretical issue with HC/mHC/Signed-Direct methods, but they never observe its consequences empirically in existing models or in models they trained. They also do not verify the effect of eHC on the geometry of the residual stream. It makes it unclear whether the theoretical issues and benefits are observable in practice in the models.
Questions for authors
  • Could you provide a high-level intuition of fundamental steps in the method/theory? Qualitatively, what does eHC entail from a "circuit" view of hyper-connections?
  • By my understanding, the equilibrium you are seeking is relevant only if the sublayers have a strong similarity, and a form of fixed-point can be reached. Although this makes the framework interesting for looped models, it is not clear to me how close to reality this general assumption is? In other words, should we really expect the residual updates to reach an equilibrium even when sub-layers perform vastly different operations?
  • How do you justify the initial/frozen coefficients chosen for the different methods? Did you experiment with several different initial setups?
3 comments
AuthorsOpenReview ↗

Response to Reviewer qtNn (1/2): Scale, Training Recipe, and Empirical Geometry

We thank Reviewer qtNn for recognizing the value of analyzing mHC through its stream-mean and disagreement geometry and for viewing eHC as an appropriate architectural remedy for retention-write coupling. Below we address each point using the new experiments and rewritten exposition in the revised manuscript.

[W1: Model Scale, Training Horizon, and Effect Size]

To evaluate eHC at longer horizons and larger model scale, Tables 2 and 16 add independently tuned 355M runs, 4x longer independently tuned 355M runs (8.590B tokens, ≈24.2 tokens/parameter), and a ≈1.3B-parameter / ≈26B-token comparison (≈20 tokens/parameter, comparable to the compute-optimal allocation of Hoffmann et al., NeurIPS 2022):

ConfigurationTokens (B)mHC NLLSigned-Direct NLLeHC NLLGain vs. mHC (10−310^{-3} nats)Gain vs. Signed-Direct (10−310^{-3} nats)
125M1.0743.34673.34353.33947.34.1
355M2.1473.15273.14883.14369.15.2
355M (tuned)2.1473.15123.14713.14199.35.2
355M (tuned, long)8.5902.77192.76782.759212.78.6
≈1.3B≈26.02.60162.59482.584517.110.3

eHC's test-NLL reduction over mHC increases from 0.0093 nats/token on tuned 355M (2.147B tokens) to 0.0127 nats/token at 8.590B tokens, and reaches 0.0171 nats/token (≈1.70% relative perplexity reduction) in the ≈1.3B/26B comparison. On the 8.590B-token 355M runs (Table 5), eHC reaches the shared validation NLL target in 9.57 vs. 10.04 GPU-hours (4.7% time saving), lowers equal-time test NLL (2.7597 vs. 2.7722), and increases 4-task zero-shot accuracy from 43.9% (mHC) and 44.4% (Signed-Direct) to 45.1% (+1.2 pp over mHC). Doubling depth at matched parameters (Deep-125 vs. Base-125, Table 2) also increases eHC's lead over Signed-Direct from 4.1e-3 to 6.6e-3 nats/token.

[W1: Architectural Complexity, Batch Size, and Warmup]

  • Computational complexity: mHC, Signed-Direct, and eHC share the exact same backbone, hidden width, 4 streams, and n2+2nn^2+2n connection-head outputs. Forming w=g1+q−Pqw = g\mathbf{1} + q - Pq adds one 4×44\times 4 matrix-vector product and O(n)O(n) elementwise operations per token in stream space, independent of feature width. On standard 355M (Table 9), step time is 616.5 ms vs. 612.4 ms (+0.67%) and allocated memory is 31.9 GB vs. 31.8 GB.
  • Batch size & warmup: Our reference runs use a shared batch of 131,072 prediction tokens/update (context 2,048), 256 warmup steps (33.6M tokens), and matched schedules and token ordering within each seed. Prior pretraining studies show that critical batch size scales primarily with data horizon rather than model size—characterizing diminishing returns to data parallelism (Zhang et al., ICLR 2025)—and that tuned smaller batches can match or exceed larger-batch per-FLOP performance (Marek et al., NeurIPS 2025). For warmup, SmolLM2-1.7B (Ben Allal et al., COLM 2025) used 2,000 warmup steps at 2M tokens/batch over an ≈11T-token horizon (≈0.036% of total tokens), whereas 256 warmup steps in our shorter-horizon controls correspond to 3.125% (125M) and 1.5625% (355M) of total updates.

[W2: Empirical Observations of Stream Geometry and Coupling]

To observe these effects directly in trained models, we added internal stream-geometry metrics (Appendix D, Table 10) and retention-perturbation experiments across all three methods (Section 5.4, Table 11):

MethodAdj.-Output Sim. CfC_fTarget Mag. MTM_TTarget Var. VTV_TTracking Err. ETE_TDisagreement Energy EDE_D
mHC0.2140.8420.6760.8730.0376
Signed-Direct0.2190.8750.6490.8460.0419
eHC0.2260.9180.6110.7790.0487
  1. Stream geometry (Table 10): Trained eHC checkpoints exhibit lower target variation (VT=0.611V_T = 0.611) and lower tracking error (ET=0.779E_T = 0.779) than Signed-Direct and mHC, alongside larger target magnitude (MT=0.918M_T = 0.918) and larger stream disagreement energy (ED=0.0487E_D = 0.0487).
  2. Retention-write coupling in trained baselines (Table 11): When test-time retention in Base-355 checkpoints is perturbed via Pη=(1−η)P+ηIP_\eta = (1-\eta)P + \eta I (η=0.025\eta=0.025), fixed-write predictive KL is 1.112 (mHC), 1.061 (Signed-Direct), and 1.000 (eHC) on a common scale normalized to the eHC fixed-write value, whereas rescaling centered writes to preserve each layer's conditional target reduces those values by 25.2% (mHC), 29.9% (Signed-Direct), and 37.3% (eHC) (and lowers Δ\DeltaNLL across all three), directly demonstrating the diagnosed coupling in trained models.
AuthorsOpenReview ↗

Response to Reviewer qtNn (2/2): Circuit Intuition, Untied Sublayers, and Initialization

[Q1: High-Level Circuit Intuition]

Section 2 now includes a two-stream circuit walkthrough alongside Figure 1 and Eqs. (6)–(7). Suppose two streams carry mean xˉ\bar{x} plus or minus offset dd:

  • Mean path (xˉ\bar{x}): Symmetric doubly stochastic mixing with cross-weight α∈(0,1/2]\alpha \in (0, 1/2] leaves xˉ\bar{x} unchanged, and the shared gain gg adds gfgf into xˉ\bar{x} just as in a standard residual connection.
  • Offset path (dd): Mixing retains (1−2α)d(1-2\alpha)d and erases 2αd2\alpha d. With an independent differential write ±βf\pm\beta f, the update d′=(1−2α)d+βfd' = (1-2\alpha)d + \beta f has conditional equilibrium βf/(2α)\beta f/(2\alpha), which can change when retention 1−2α1-2\alpha changes—altering the stream offsets read out by unequal downstream weights rr.
  • eHC's update: eHC predicts a target offset t=qft = qf and writes ±2αt\pm 2\alpha t, giving d′=t+(1−2α)(d−t)d' = t + (1-2\alpha)(d - t). Mixing controls how much prior mismatch d−td-t carries over, while the write restores the amount 2αt2\alpha t that mixing would otherwise erase from the chosen target. Everything runs in a single explicit forward pass, and attention and the following MLP freely choose different targets.

[Q2: Relevance When Sublayers Perform Vastly Different Operations]

eHC is designed for standard untied Transformers where attention and MLP sublayers perform distinct operations (Cf≈0.22C_f \approx 0.22 between adjacent sublayer outputs in Table 10), without expecting the hidden state to settle to a global fixed point:

  1. Local single-step reference: For incoming Xl=1xˉl+DlX_l = \mathbf{1}\bar{x}_l + D_l, the mean xˉl\bar{x}_l accumulates additively, while sublayer ll specifies its own local disagreement target Tl=qlflT_l = q_l f_l and computes Dl+1=Tl+Pl(Dl−Tl)D_{l+1} = T_l + P_l(D_l - T_l) (Eq. 7). "Equilibrium" refers to the fixed point of sublayer ll's affine disagreement map, which calibrates that single layer's write scale against its own mixing matrix PlP_l.
  2. Moving-target dynamics across distinct layers: When consecutive sublayers choose different targets Tl≠Tl+1T_l \neq T_{l+1}, the tracking mismatch el=Dl−Tle_l = D_l - T_l satisfies the exact identity el+1=Plel+Tl−Tl+1e_{l+1} = P_l e_l + T_l - T_{l+1} (Eq. 12). Under uniform contraction ∥Pl−J∥2≤ρ<1\\\|P_l - J\\\|_2 \le \rho < 1 and target increments ∥Tl+1−Tl∥F≤δ\\\|T_{l+1}-T_l\\\|_F \le \delta, ∥eL∥F≤ρL∥e0∥F+δ1−ρL1−ρ\\\|e_L\\\|_F \le \rho^L \\\|e_0\\\|_F + \delta \frac{1-\rho^L}{1-\rho} (Eq. 13).
  3. Single-layer gradient decoupling: Even at a single sublayer in isolation, holding local inputs and target fixed while perturbing PlP_l along an admissible doubly stochastic direction (dPl1=0,1⊤dPl=0dP_l\mathbf{1}=0, \mathbf{1}^\top dP_l=0) changes an independent-write output by dPlDldP_l D_l, whereas in eHC the induced write change dbl=−dPlqldb_l = -dP_l q_l gives dPl(Dl−Tl)dP_l(D_l - T_l), which vanishes when Dl=TlD_l = T_l (Prop. 3, Eq. 15).

[Q3: Justification of Initial/Static Coefficients and Initialization Experiments]

  • Why 0.1 in Signed-Direct: In Signed-Direct (w=g1+0.1qw = g\mathbf{1} + 0.1q, Eq. 18), qq, PP, and gg are all learned; only the scalar factor 0.1 mapping centered logits qq to the centered write is fixed. At our standard initialization (P0=0.9I+0.1JP_0 = 0.9I + 0.1J, r0=1/nr_0 = \mathbf{1}/n, g0=1g_0 = 1, and rotating zero-sum q0q_0 offsets), I−P0=0.1ΠI - P_0 = 0.1\Pi on the zero-mean subspace. Using 0.1 gives Signed-Direct the exact same initial effective writes and initial write-head Jacobian (0.5J+0.1Π0.5J + 0.1\Pi) as eHC (Section 5.1, Eqs. 18–20, Appendix B.3), isolating the effect of adapting I−PlI - P_l during training.
  • Initialization sweep: We evaluated five one-factor 125M settings varying initial retention λ0∈0.8,0.9,0.95\lambda_0 \in \\{0.8, 0.9, 0.95\\} (at amplitude 1.0) and write amplitude ∈0.5,1.0,2.0\in \\{0.5, 1.0, 2.0\\} (at λ0=0.9\lambda_0 = 0.9) (Tables 2, 15):
Setting (λ0,amplitude)(\lambda_0, \text{amplitude})mHC NLLSigned-Direct NLLeHC NLLGain vs. mHC (nats)Gain vs. Signed-Direct (nats)
(0.90, 1.0) [Standard]3.34673.34353.33940.00730.0041
(0.80, 1.0)3.34593.34273.33950.00640.0032
(0.95, 1.0)3.34873.34513.33980.00890.0053
(0.90, 0.5)3.34823.34493.34110.00710.0038
(0.90, 2.0)3.35063.34603.34020.01040.0058

eHC outperforms mHC by 0.0064–0.0104 nats/token and Signed-Direct by 0.0032–0.0058 nats/token across all five initializations.

AuthorsOpenReview ↗

Follow-up Note to Reviewer qtNn

Dear Reviewer qtNn,

Please accept our sincere apologies for posting our response so late in the discussion period. Language model pretraining is computationally demanding, and completing the larger-scale runs and initialization comparisons took considerable wall-clock time.

We greatly appreciate your thoughtful comments, especially your questions on the circuit intuition across untied sublayers and whether the predicted stream geometry actually appears in trained models. As soon as your review arrived, we reconsidered those aspects of our work, reviewed our earlier experimental records, and compared models at larger scales, varied the initialization, and measured the geometry of the trained residual streams. Your feedback has been very valuable in improving both the clarity and empirical grounding of the paper.

We welcome any further questions or suggestions you may have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Note to AC: Rebuttal Summary

Dear Area Chair,

We thank the three reviewers for their evaluations of Submission #58753, Equilibrium Updates for Residual Learning. Reviewer qtNn commended our geometric perspective on mHC and called eHC a "relevant solution"; Reviewer WR8b described the approach as "reasonable and intuitive" and the paper as "well-motivated, well-organized, and clearly written"; and Reviewer M2Vg noted our "useful insights into coordinating retention and writing" along with eHC's minimal overhead (+0.67% step time at standard 355M).

In the revised manuscript, we rewrote the abstract and main prose for clarity and added the scaling, diagnostic, and ablation experiments requested in the reviews:

  • Larger Scale, Longer Horizon, and Independent Tuning (qtNn, WR8b): We added independently tuned 355M runs at 2.147B tokens, 4x longer tuned 355M runs at 8.590B tokens, and a ≈1.3B-parameter / ≈26B-token comparison (Tables 2, 16). eHC's test-NLL reduction over mHC reaches 0.0127 nats/token on long 355M—reaching a shared validation NLL in 4.7% fewer GPU-hours and raising 4-task zero-shot accuracy by +1.2 percentage points (Table 5)—and 0.0171 nats/token (≈1.70% relative perplexity reduction) at ≈1.3B.
  • Single-Step Theory & Changing Sublayer Targets (qtNn, WR8b, M2Vg): Sections 2 and 4.1–4.3 add a concrete two-stream circuit and clarify how eHC operates across untied, changing sublayers. While mHC and Signed-Direct admit implicit fixed-point targets at a frozen strictly positive PP (Prop. 2), perturbing PP with an independent write held fixed shifts the state by dP,DdP\\,D, whereas eHC's coupled write b=(I−P)qb=(I-P)q yields the innovation-weighted variation dP(D−T)dP(D-T), which vanishes on target (Prop. 3).
  • Stream Geometry & Cross-Method Interventions (qtNn, M2Vg): Trained eHC models exhibit lower target variation and lower tracking error alongside higher stream disagreement energy than both baselines (Table 10). Applying equilibrium-preserving write compensation without retraining reduces sensitivity to retention perturbations across all three methods, with eHC showing the largest relative output-KL reduction (37.3%, Table 11).
  • Gradient Routing, Finite Sinkhorn, & Initialization (qtNn, M2Vg): Gating write-branch gradients through (I−Pl)ql(I-P_l)q_l separates the benefit of forward coordinate adaptation (+0.0025 nats vs. Signed-Direct) from backward coupling, where the mixing-parameter path accounts for the clearer individual reduction (ParamOnly +1.1e-3 nats vs. Detached; Full +1.6e-3 nats, Table 4). Evaluated 20-iteration Sinkhorn matrices satisfy ρ⊥<1\rho_\perp < 1 in >99.97% of cases, higher-precision scaling yields nearly identical 125M NLLs (Tables 13–14), and eHC outperforms both baselines across five initialization settings (Table 15).

Full technical details appear in our individual responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.