acceptodds
Under review as a conference paper at ICLR 2027

Gated Delta Hyper-Connections: Separating Residual Mixing from State Retention

33 views

Abstract

Hyper-Connections (HC) expand Transformer residual pathways into multiple streams, offering a scaling dimension beyond model depth and width. Manifold-Constrained Hyper-Connections (mHC) stabilize cross-stream mixing, but mixing alone does not control how much of the carried state is retained: under its doubly stochastic constraint, the stream-average component is left unattenuated as layers add new outputs. Our measurements on trained models show that this component grows faster across depth than the remaining state, while write gates become more concentrated and lose mass during training. These observations motivate Gated Delta Hyper-Connections (GDHC), which adds explicit retention control alongside the existing read/write maps by combining global decay with write-aligned erasure. Decay regulates overall retention, while erasure selectively attenuates old content along the incoming write direction, independently of write strength. Experiments with mHC and identity-mixer iHC show lower training loss at both 4B- and 7B-parameter mixture-of-experts models, with 0.55B and 0.91B active parameters, respectively, and higher average downstream scores at the larger scale. Component ablations show further gains from combining decay and erasure. GDHC also flattens activation profiles and increases median write-gate mass; common-mode RMS decreases across depth on A1B mHC and becomes nearly flat on iHC. Together, these results support explicit retention as a complementary control in multi-stream residual architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hyper-connections, Residual connections, Transformer architectures, State retention, Gated delta rule

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DdiTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper distinguishes retention of the existing residual state from mixing across streams in Hyper-Connections. A doubly stochastic mixer preserves the stream-average component, so stable mixing alone does not attenuate that component between successive writes. Gated Delta Hyper-Connections (GDHC) augments either mHC or identity-mixer iHC with input-dependent scalar decay and rank-one erasure aligned with the existing write direction, while allowing erase and write strengths to differ. The paper analyzes the carried-state operator conditional on realized gates, evaluates two mixture-of-experts model sizes, and reports lower training loss, improved average downstream performance, and flatter activation profiles. Component ablations examine decay, erasure, and the frequency of decay placement.

Critical strengths
  1. The proposed interface is simple and technically well motivated. It preserves the existing read/write maps and mixer while adding explicit control over the old state. Equations 3 and 10–12 and Appendix A.1 explain the common-mode behavior, the selected-direction update, and the conditional norm bound clearly. I particularly appreciate the explicit distinction between a bound on the realized carried-state operator and a guarantee for the full nonlinear block, as well as the treatment of approximate Sinkhorn normalization and the numerical key-normalization floor.
  2. The evaluation provides meaningful evidence that this integration is useful at the tested scales. The paired runs use the same data order and training settings; both base architectures improve at both reported A1B budgets, with iHC gaining 0.86–1.30 points in the ten-task average (Table 1). The training-loss trajectory and A500M component experiments provide complementary evidence (Tables 2, 6, and 8). The accompanying state diagnostics are informative, and the paper is unusually explicit about their limitations, same-seed execution variability, mixed task-level results, and differences too small to resolve.
Critical weaknesses
  1. The reliability and generalization of the smaller improvements remain uncertain. The 0.13–0.25-point mHC downstream gains are modest, and the additional dense-placement benefit of combining the controls is 0.004–0.011 training-PPL units over the available single-control alternatives (Tables 2 and 8). Appendix A.7 reports approximately 0.008 PPL variation between two same-seed mHC executions. This is not a significance test for the other comparisons, but it makes the precision of the component conclusions difficult to assess. The paired trajectories are useful evidence, and the paper correctly does not present their temporal variation as independent-run uncertainty. Nevertheless, held-out language-model loss and uncertainty estimates for the downstream differences are missing from the reported evidence. Bootstrap intervals based on paired evaluation examples, together with a limited smaller-scale repeat of the most important component comparison, would help establish which effects are reproducible. I do not regard the absence of repeated full 4B/7B training runs, by itself, as a reason to reject the paper.
  2. The experiments establish the usefulness of the combined modification more directly than the benefit of its particular write-aligned, independently gated formulation. Depth-wise rank-one rewriting and independently controlled erase/write operations already have close precedents, which the paper acknowledges. The ablations show that adding the two controls helps, but do not isolate whether the tied write direction or independently chosen strengths matter relative to a simpler matched retention update. The DDL comparison is informative but does not resolve this issue: Appendix A.10 changes its read-side convolution to kernel size 1 and correctly acknowledges other interface and cost differences. Thus, the DDL scores cannot by themselves identify the benefit of stream-space editing or the proposed coupling choices. One targeted control, such as a fixed common-mode erase direction or a matched coupled erase/write gate, would make the distinctive design insight more convincing. This limitation does not invalidate the matched comparisons against mHC and iHC.
Questions for authors
  1. Can you provide paired uncertainty estimates for the existing downstream predictions and held-out language-model losses for the evaluated checkpoints? For component complementarity, a limited repeat at the smaller scale would be especially useful; please distinguish evaluation uncertainty, execution variation, and variation across training seeds.
  2. Which DDL source version and compressor variant were reproduced, and was its embedding-convolution expansion retained? Appendix A.10 describes a width-1 causal compressor, whereas the reference also distinguishes token-axis and channel-axis variants. Please explain the choice and whether the main conclusion is robust to the reference configuration or a small compressor control.
  3. Is there evidence comparing the tied write direction with a fixed common-mode direction, or comparing independent strengths with a coupled gate under the same residual interface? Even one such controlled comparison would help clarify which part of the proposed parameterization is essential.
  4. Please clarify the exact expert-width convention and number of MoE layers underlying Table 4, so the reported total and active parameter counts can be reconstructed.
Reviewer_pVtMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Gated Delta Hyper-Connections (GDHC), which adds explicit retention control to multi-stream residual architectures. The motivation is that doubly stochastic mixing preserves the stream-average component, allowing previous writes to persist without independent attenuation. GDHC combines input-dependent global decay with rank-one erasure aligned to the incoming write direction, while retaining the existing mixer and read/write interface. Experiments on two mixture-of-experts model scales show consistently lower training loss. At the larger scale, downstream improvements are more substantial on iHC and smaller on mHC. Activation and gate diagnostics support the intended change in state propagation. The distinction between mixing and retention is useful, and the mathematical formulation is sound.

Critical strengths
  1. Clear and valid architectural motivation. Equation (3) correctly identifies the component preserved by an exactly doubly stochastic mixer. Separating this preservation property from control over state retention provides a useful perspective on multi-stream residual networks. Appendix A.1 appropriately clarifies that preservation permits accumulation without making growth inevitable.
  2. Simple formulation with appropriately scoped analysis. Global decay, directional erasure, and writing are cleanly separated in Equations (5)-(10). The conditional operator bound is correct, and the appendix addresses finite Sinkhorn normalization and numerical protection of the erase key. The manuscript explicitly distinguishes this bound from a guarantee about the full nonlinear Jacobian.
  3. Encouraging results on the identity mixer. GDHC improves the iHC downstream average by approximately 1.30 and 0.86 percentage points at the two evaluated budgets, winning on eight and seven of ten tasks, respectively. All four exam tasks improve at both budgets. Both model scales and both mixers also show lower training loss. The downstream budgets are checkpoints from the same runs, rather than independent replications.
  4. Informative diagnostics and transparent qualifications. The common-mode decomposition, gate statistics, and component ablations provide evidence beyond aggregate benchmark scores. The paper carefully distinguishes net state revision from pure write magnitude, token-averaged gate allocation from token-wise selectivity, and trajectory fluctuations from uncertainty across independent runs.
Critical weaknesses
  1. Uncertainty limits the interpretation of the smallest gains. Each variant is trained once. The mHC downstream gains are 0.13 and 0.25 percentage points, and its later training-loss advantage is approximately 0.0014 nats. Paired trajectories and the repeated same-seed baseline are informative but do not estimate independent-run variation. Held-out language-model loss would complement the existing downstream evaluation. These limitations reduce confidence in precise effect sizes, particularly on mHC.
  2. Mechanistic attribution and marginal component gains remain partly unresolved. The iHC ablation reports PPL of 5.876 for the base, 5.863 for decay alone, 5.860 for erasure alone, and 5.856 for both. This is suggestive of complementarity; the additional 0.004 over erasure alone warrants cautious interpretation without replication. Diagnostics show changed dynamics, while their causal role in improved quality remains open. Alternative erase-key controls would sharpen attribution, rather than being necessary to establish that the complete GDHC configuration can help.
  3. The DDL comparison has limited scope. Appendix A.10 discloses a read-side convolution of kernel size one, differing from the reference implementation, and notes that parameters and compute need not match. Conclusions should therefore remain specific to the reproduced configuration. This limitation weakens the DDL comparison while leaving the matched iHC/mHC comparisons supporting the central retention contribution intact.
  4. Reproducibility and practical cost need fuller documentation. Exact evaluation configurations, prompt templates, decoding settings, judge instructions, added-head AdamW settings, and the complete learning-rate schedule would improve reproducibility. The expert projection shapes and parameter-counting convention also need clarification. Measured throughput and memory overhead would help establish the practical quality-cost tradeoff; these measurements would strengthen the contribution rather than substitute for its existing evidence.
Questions for authors
  1. Reliability and generalization: Can the authors provide held-out loss for existing checkpoints and paired uncertainty estimates for downstream differences, especially on mHC? A focused independent-seed replication would strengthen confidence in the smaller effects.
  2. Component interpretation: Can the claims about combining the two controls be qualified to reflect the small additional gains? Any available repeatability evidence or matched alternative-key results would help distinguish component effects from the benefit of the complete update.
  3. DDL implementation: What reference convolution width was changed to one, and why? Please keep comparative conclusions explicitly tied to the reproduced implementation unless its sensitivity has been checked.
  4. Reproducibility and parameter accounting: Can the authors provide exact evaluation and optimizer configurations, expert projection shapes, and total/active parameter-counting conventions, including the meaning of "expert width 512" in Table 4?
  5. Practical overhead: What training throughput, inference latency, and memory overhead does GDHC add relative to each base? Reporting these would make its practical tradeoff easier to judge.
Reviewer_QnrQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GDHC adds scalar decay and write-aligned directional erasure to multi-stream residual connections while retaining the HC read/write interface. The contribution is an HC-compatible retention mechanism with a conditional non-expansion bound and diagnostics of state accumulation across depth. Validation uses 4B/7B-total-parameter MoE models with 0.55B/0.91B active parameters, comparing iHC and mHC bases, component ablations, downstream tasks, and activation profiles.

Critical strengths
  1. The retention mechanism addresses a real limitation of stable mixing. Doubly stochastic mixing preserves the stream mean, but repeated writes can still accumulate state. GDHC allows explicit attenuation while preserving the HC interface. The conditional operator bound in Eq. 12 appears sound and is distinguished from a bound on the full nonlinear Jacobian. Unlike sHC[4], the mechanism can attenuate the common mode. This makes the integration technically meaningful, even though its basic operations have precedents.

  2. The iHC results provide positive evidence for the combined controls. Appendix A.9 reports training PPL of 5.876 for the base, 5.863 with decay, 5.860 with erasure, and 5.856 with both. Downstream gains on iHC are +1.30/+0.86 points at 52.43B/83.89B tokens (Table 1). Section 5.4 also carefully distinguishes flatter depth profiles from uniformly lower activation magnitude.

[1] https://proceedings.iclr.cc/paper_files/paper/2025/file/4904fad153f6434a7bcf04465d4be2cc-Paper-Conference.pdf | Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.

[2] https://arxiv.org/pdf/2605.22791v1 | Hatamizadeh et al. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. Accepted at NeurIPS 2026; cited equation from v1. Acceptance: https://github.com/NVlabs/GatedDeltaNet-2

[3] https://arxiv.org/pdf/2601.00417v1 ; https://arxiv.org/pdf/2601.00417v5 | Zhang et al. Deep Delta Learning. Preprint, 2026; v1 (January) and v5 (September) distinguished above. No formal publication verified.

[4] https://arxiv.org/pdf/2603.20896v1 | Liu et al. Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections. March 2026 version; revised as Spectral-Sphere-Constrained Hyper-Connections, accepted at NeurIPS 2026. Venue record: https://6zhaoyi.github.io/publication/5-s2hc/

Critical weaknesses
  1. The small incremental gains are not yet shown to be reliable. The paper acknowledges the underlying operations from Gated Delta Networks[1], Gated DeltaNet-2[2], and DDL[3]. The algebraic relationship does not make the complete architectures equivalent; the contribution lies in the HC integration. Its ablations give positive point evidence, but the combined iHC gain over erasure alone is only 0.004 PPL, and the corresponding erase-only comparison is missing on mHC. Each variant is trained once; the repeated same-seed mHC baseline measures execution variability, not independent-seed uncertainty. At 80K steps, GDHC reduces training loss relative to mHC by 0.0014, with downstream gains of +0.13/+0.25 points at the two token budgets. The incremental value of the write-aligned key is also unresolved. This limits confidence that the added design consistently improves on its simpler component controls, despite the encouraging iHC results.

  2. The practical gain cannot yet be weighed against implementation cost. Parameter counts do not establish runtime overhead. Matched throughput and peak-memory measurements against the same four-stream base are needed to judge whether the small incremental gains justify the FP32 gates, reductions, and state accesses. The DDL comparison also needs a precise configuration: Appendix A.10 specifies kernel size 1, while DDL v5[3] distinguishes token compression (TC) from current-token value-channel compression (CC), recommending CC. CC and TC with kernel size 1 are not interchangeable. Clarifying the version, compressor, and embedding convolution would establish which reference configuration is being compared; position-wise reading alone is not evidence of an unfair baseline.

[1] https://proceedings.iclr.cc/paper_files/paper/2025/file/4904fad153f6434a7bcf04465d4be2cc-Paper-Conference.pdf | Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.

[2] https://arxiv.org/pdf/2605.22791v1 | Hatamizadeh et al. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. Accepted at NeurIPS 2026; cited equation from v1. Acceptance: https://github.com/NVlabs/GatedDeltaNet-2

[3] https://arxiv.org/pdf/2601.00417v1 ; https://arxiv.org/pdf/2601.00417v5 | Zhang et al. Deep Delta Learning. Preprint, 2026; v1 (January) and v5 (September) distinguished above. No formal publication verified.

[4] https://arxiv.org/pdf/2603.20896v1 | Liu et al. Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections. March 2026 version; revised as Spectral-Sphere-Constrained Hyper-Connections, accepted at NeurIPS 2026. Venue record: https://6zhaoyi.github.io/publication/5-s2hc/

Questions for authors
  1. Is an erase-only mHC result available, and what uncertainty estimates support the incremental gain from combining both controls? Independent-seed results would be particularly informative if available.
  2. Which DDL version and compressor (TC or CC) does Appendix A.10 reproduce, and is embedding convolution enabled? How does it differ from that version's reference configuration?
  3. Can matched throughput and peak-memory measurements be provided against the same four-stream base, with hardware and batch settings held fixed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.