acceptodds
Under review as a conference paper at ICLR 2027

uHC: Uncoupled Hyper-Connections

Abstract

Hyper-Connections (HC) extend Transformer residual representations into multiple streams, while Manifold-Constrained Hyper-Connections (mHC) stabilize multi-stream propagation through constrained residual mixing. However, existing HC architectures maintain coupled interfaces between residual streams and the transformation blocks: readout patterns are shared across feature dimensions, and write updates remain aligned across streams, limiting the benefits of residual-stream expansion. We identify this transformation-interface coupling as a structural bottleneck. To alleviate this limitation, we propose Uncoupled Hyper-Connections (uHC), which redesigns the transformation interfaces without modifying the underlying Transformer blocks. uHC introduces Group-Wise Feature Readout for feature-dependent stream aggregation and Low-Rank Directional Write-Back for stream-specific update directions with lightweight overhead. Across three model scales ranging from 46M to 363M parameters, uHC consistently improves validation loss and perplexity over residual connections, mHC, and mHC-lite. At the largest scale, uHC achieves the best performance across all reported metrics, reducing validation loss from 2.9470 for mHC to 2.8188. Further analyses reveal that uHC better utilizes the expanded residual-stream space and achieves more stable optimization, highlighting the importance of decoupling transformation interfaces for scaling multi-stream residual architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hyper-Connections, residual connections, Transformer architectures, language modeling, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KuJuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors propose Uncoupled Hyper-Connections (uHC) which is an extension of manifold constrained Hyper-Connections (mHC) that targets "transformation-interface coupling." In current multi-stream architectures, the MLP block reads from all streams using a single routing pattern and writes back the same output vector to every stream. uHC addresses this with a group-wise feature readout which lets different feature subsets blend at the same parameter cost as mHC. uHC also uses a low-directional write-back which adds a stream specific correction, which gives each stream a distinct update direction. The authors validate the method by pretraining models from 46M-363M on OpenWebText. They report lower validation loss and generally better perplexity than residual, mHC, and mHC-lite baselines. However, all results come from single runs without any reported variance. The headline result is measured against an mHC baseline that underperformed the plain residual baseline so the validation does not convincingly support the improvements.

Critical strengths
  1. The authors have a fairly precise reason for a real limitation. Hyper-Connection readouts use a single routing pattern which is shared across features and the write back sends every stream a scaled copy of the same vector. Which means the write matrix is rank one no matter how many streams you add. Mathematically elegant and nice to verify which helps provide an explanation for why adding more residual streams may be diminishing. Which I believe is a useful insight in further work.

  2. The proposed fix is minimal and cheap. The Group-wise readout keeps the same number of parameters as mHC's readout. The low-rank write back only adds a small number of parameters back to the layer. The initialization of uHC starts the same as mHC so any deviation is just learned. That means for any codebase already using mHC an easy drop in uHC is possible and any changes would be attributable to the method.

Critical weaknesses
  1. I have some doubts on the statistical support. There appears to be no seeds or error bars, and I do not see multiple runs reported so I believe this all may be from a singular run. At S and M (Table 1) uHC only has a small advantage and a worse perplexity score on LAMBADA at M. This could just be noise or run-to-run variation. So the claims may be overstated.

  2. I worry that there may be an issue with the reported baselines. At the largest scale the plain residual outperforms mHC and mHC-lite which might be pointing to an unstable or undertuned baseline. The core claim of this paper is tied to that baseline. It is entirely possible that uHC may just have a more stable training under the hyperparameters.

  3. The experiments may be too small to support scaling claims. All models used are under 1B and trained on approx 8B tokens at most (Can you double check that number?). Architectural gains at this size frequently matter less under larger sizes or with better tuning. The lack of runs larger than 363M or even additional seed runs do not leave me convinced of the claim.

Overall as it stands, none of the evidence provided can support the claimed improvements. This could all be alleviated with additional runs.

Questions for authors

At L scale, plain residual outperforms both mHC and mHC-lite, contradicting prior work. Did you tune the learning rate or other hyperparameters separately for each method, or reuse one configuration?

Can you show an mHC run at L that is at least competitive with the residual baseline, and report uHC’s gain against it?

How much does validation loss vary across random seeds in your setup?

Do you have any results at 1B parameters or above, even a single run? If not, why should readers expect the gains to persist where mHC’s original instability problem actually arises?

Is uHC’s benefit at L mainly that it avoids the gradient spikes mHC suffers?

Would mHC with gradient clipping, a lower learning rate, or longer warmup close most of the gap?

3 comments

We thank the reviewer for recognizing our structural insight and lightweight design, and for the critical feedback on experimental reliability.

W3 / Q4. Larger-scale validation and training budget.

We have extended our evaluation to 1.07B parameters with a 10B-token budget, beyond the original L-scale experiments (7.86B tokens).

At this scale, uHC reduces validation loss from 2.6556 to 2.6433, improves LAMBADA perplexity from 18.86 to 17.32, and outperforms mHC on 4 of 5 Paloma corpora. Importantly, both models exhibit stable gradient dynamics.

Complete results are provided in the Common Response, Sec. 4.2, and Appendix C.7.

W2 / Q1 / Q2 / Q6. Baseline fairness and hyperparameter tuning.

We acknowledge the weaker performance of the original L-scale mHC baseline. All main experiments use shared scale-specific configurations, including learning rate, warmup, and gradient clipping (1.0), without method-specific tuning.

To further address this concern, we retrained matched L-scale mHC and uHC models with a different random seed (1338), using the same L-scale architecture and optimization settings with a 5B-token budget (19,073 steps). Both models exhibit stable gradient dynamics.

Additional L-scale results (5B tokens, seed 1338):

MetricmHCuHC
Val Loss ↓2.87122.8702
OWT-val PPL ↓17.748917.6644
WT103 PPL ↓26.465824.6335
LAMBADA PPL ↓31.237732.5365

Paloma evaluation (↓):

DatasetmHC Word PPLuHC Word PPLmHC BPBuHC BPB
C4112.25106.201.13501.1217
Dolma223.15205.231.19991.1813
Falcon158.01144.201.22841.2062
RedPajama788.99676.241.33611.3052
WikiText59.4356.031.10141.0855

The new mHC run achieves substantially lower monitored validation loss than the original run, despite a shorter training budget. Against this stronger, stable mHC baseline, uHC improves OWT-val, WT103, and all five Paloma corpora, with LAMBADA being the exception.

These results provide further evidence that uHC's gains do not depend on the original unstable mHC baseline. We also provide the anonymous source code and training configurations in the supplementary materials to facilitate independent reproduction.

W2 / Q5. Are the improvements primarily due to avoiding gradient spikes?

We observed recurring gradient instability in L-scale mHC training and further investigated its potential connection to write-back coupling and optimizer update allocation in Appendix C.5.

Both the new L-scale and 1.07B experiments show that uHC retains performance advantages even when mHC trains stably, suggesting that its benefits extend beyond mitigating gradient instability.

W1 / Q3. Statistical reliability and random-seed variability.

Our main experiments use matched configurations and random seeds for controlled comparisons. The additional L-scale comparison with seed 1338 provides complementary evidence across training runs.

Given the computational cost of large-scale training, we prioritized matched comparisons with a stronger, stable mHC baseline and validation at the 1.07B scale. Although these experiments do not establish statistical significance or directly quantify seed variance, they provide additional evidence that uHC's improvements are not solely attributable to the instability of the original baseline.

We appreciate the reviewer's careful scrutiny and constructive suggestions. The additional experiments and analyses have strengthened our empirical evaluation and provided broader evidence for uHC's architectural benefits.

Thank you for the additional experiments. I have three remaining points:

  1. The 1338 seed L-scale rerun shows the gap against a stable mHC baseline shrinks from 0.128 to 0.001 in val loss. Will the abstract, intro, and Table 1 be revised to report this run alongside, or instead of, the original L-scale run?

  2. In the original L-scale results, the plain residual outperformed both mHC variants. Could you add the residual baseline under the same 5B token L-scale setting, so readers can see whether uHC improves over a standard residual connection when all methods train stably?

  3. The ablation conclusions and several main table gaps rest on loss differences of 0.00003-0.012. Could you report 2-3 seeds at the S or M scale so readers can judge which differences exceed run to run variance?

We thank the reviewer for these constructive questions. We have conducted further experiments and updated the manuscript accordingly.

Q1. Revising the original L-scale results

We acknowledge the weaker performance of the original L-scale mHC baseline. In the independent 5B-token experiment, the validation-loss gap between mHC and uHC decreases from 0.1282 to 0.0011.

We retain the original L-scale results in Table 1 for transparency, rather than replacing them with results from a different seed and training budget. Meanwhile, we have revised the Abstract and Introduction to emphasize the 1.07B-scale results, where mHC trains stably and uHC retains its advantage.

The independent L-scale comparison is provided in Appendix C.2.1.

Q2. Residual baseline under stable optimization

We additionally trained residual and mHC-lite baselines under the same L-scale configuration (seed 1338, 5B tokens, 19,073 steps).

MethodOWT-val ↓WT103 ↓LAMBADA ↓
Residual18.3731.7737.20
mHC17.7526.4731.24
mHC-lite17.8026.1032.09
uHC17.6624.6332.54
Paloma Word-level Perplexity (↓)
MethodC4DolmaFalconRedPajamaWikiText
Residual107.89267.20154.531046.9363.85
mHC112.25223.15158.01788.9959.43
mHC-lite113.70226.99158.61801.0663.37
uHC106.20205.23144.20676.2456.03

All three multi-stream variants outperform residual connections on OWT-val, WT103, and LAMBADA. uHC achieves the best results on seven of eight perplexity evaluations, including all five Paloma corpora, with LAMBADA being the only exception.

The full-training and late-stage gradient trajectories further confirm that both mHC and uHC train stably without pronounced late-stage spikes. These results demonstrate that uHC retains its advantages against a stable, competitive mHC baseline.

Complete results and gradient diagnostics are provided in Appendix C.2.1 (Table 7 and Figure 5).

Q3. Random-seed variability

We extended the M-scale rank and grouping ablations to two seeds (1337 and 1339) under matched 7.5K-step budgets.

G=4 achieves the lowest validation loss under both seeds. Ranks 8 and 16 show comparable mean performance (3.278019 ± 0.005134 vs. 3.277639 ± 0.004557), supporting our choice of r=8 for parameter efficiency.

While these two-seed ablations provide additional evidence on run-to-run variability, we acknowledge that they do not establish statistical significance for all main-table comparisons. Given the computational cost and limited rebuttal timeline, we prioritized the matched L-scale comparison and 1.07B-scale validation. We plan to further expand the multi-seed evaluation in subsequent revisions.

The results are reported in Sec. 4.5 and Appendix C.2 (Table 6).

Together with the stable 1.07B-scale comparison, these experiments strengthen the evidence that uHC's benefits are not solely attributable to the original L-scale mHC instability.

Reviewer_V6upOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Uncoupled Hyper-Connections (uHC), combining Group-Wise Feature Readout and Low-Rank Directional Write-Back to enable feature-dependent stream aggregation and stream-specific updates with minimal overhead. Experiments on language models ranging from 46M to 363M parameters demonstrate improved validation loss and perplexity over residual connections, mHC, and mHC-lite, supported by ablation studies, scaling analyses, and gradient stability measurements.

Critical strengths
  1. Clear motivation and well-designed architectural improvements. The paper identifies transformation-interface coupling as a structural bottleneck in existing Hyper-Connections and provides a clear mathematical motivation for addressing it.

  2. Comprehensive experiments with consistent performance improvements. The paper evaluates uHC across three model scales (46M–363M parameters) and multiple benchmarks, demonstrating improvements in validation loss and perplexity over existing methods.

Critical weaknesses

Limited experimental scale and generalizability. The experiments are restricted to relatively small language models (up to 363M parameters) trained exclusively on OpenWebText, leaving the effectiveness of uHC at larger scales and in more realistic training settings unclear. Since the proposed method targets the scalability of multi-stream residual architectures, the lack of large-scale validation weakens the practical significance of the reported improvements.

Questions for authors

none

1 comment

We thank the reviewer for recognizing our clear motivation, architectural design, and consistent empirical improvements.

W1a. Limited experimental scale.

We extended our experiments to 1.07B parameters with a 10B-token budget, where uHC reduces validation loss from 2.6556 to 2.6433 and LAMBADA perplexity from 18.86 to 17.32, outperforming mHC on 4 of 5 Paloma corpora.

Notably, these gains persist even when mHC exhibits stable gradient dynamics. Full results are provided in the Common Response and Sec. 4.2.

W1b. Generalizability beyond OpenWebText.

The original submission already includes cross-domain evaluations on five Paloma corpora, alongside WikiText-103 and LAMBADA (Table 1), providing evidence of generalization beyond OpenWebText.

We used controlled OpenWebText pretraining to isolate architectural effects under matched settings. We agree that pretraining on broader data distributions would further strengthen our conclusions and consider this an important direction for future work.

We appreciate your encouraging assessment and constructive suggestions, which have helped strengthen our evaluation of uHC's scalability and generalizability.

Reviewer_KzjXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes uHC, a variant of mHC that seeks to expand the flexibility of mHC. The paper starts from the observation that the transformation pathway in mHC (i.e., the part that takes the residual stream, projects it into a CC-dimensional vector, processes it with Attention/MLP, then uses the result to update the residual stream) is overly restrictive in two ways:

  1. Read-out: The input to the transformation (i.e., MLP/Attention) is a linear combination of each of the parallel streams (so all dimensions in each parallel stream get mixed in the exact same way). Instead, we’d ideally like to have separate mixing coefficients across the dimensions of the stream, for flexibility.
  2. Write-back: When written out, one can see that the update applied to the residual stream after the transformation is a rank-one matrix (since we are taking the transformation output and re-scaling it differently for each stream)

The authors propose to make these more flexible in two ways:

  1. Read-out: Dimensions in each channel of the residual stream are grouped into gg-dimensional blocks, whose mixing coefficients can be different. This allows some degree of dimension-specific mixing when computing the input to the transformation.
  2. Write-back: The update is a context-, stream- and layer- dependent rank-rr matrix. This breaks the symmetry along streams in the residual update.

The authors pretrain models at 45M, 120M, and 360M scales, at varying tok/param budgets (~20x at 300M). They report PPL on a series of benchmarks. Overall, the results are positive, suggesting uHC improves upon mHC and mHC-lite. They also conduct ablations and find that both changes (to the read-out and write-back) are necessary to recover the improvements, though each in isolation is helpful. Finally, they conduct an interpretability study arguing that uHC has higher stream entropy, effective number of streams, etc.

Critical strengths

Positive results. The results are overall positive, with uHC showing (varying degrees of) improvements at all scales in most benchmarks.

Critical weaknesses

Methodological contribution is small. While the results are positive, the essence of the contribution is fairly small. This is not a problem, but given the methodological change is relatively small (slight tweak to some aspects of mHC) I think the paper could be more substantial on the empirical axis by ablating different aspects of the architecture, e.g., different parameterizations for the write-back low-rank update, different grouping strategies, varying the group/rank hyperparameters, etc.

Writing unclear at times. At times it is hard to understand exactly what the method is doing, and what that corresponds to (e.g., in terms of implementation). I think a figure would be very beneficial to explain the specific changes to both the read-out and write-back. I also found it hard to follow the metrics definitions in the analysis/interpretability section, which makes it hard to understand the conclusions made there.

Experiments conducted at small scale. Even the largest scale experiments are fairly small (~360M) which makes it hard to gauge if these improvements would actually hold at larger scales.

Questions for authors

Could you clarify the metrics you use in the analysis section?

Is there any way you could report Paloma bits-per-byte? (The numbers on the table seemed a bit high to me at first glance, but I imagine this could be because of the choice of tokenizer. Having BPB would be helpful to compare against numbers in the literature at these model scales)

1 comment

We thank the reviewer for recognizing our positive results and for the constructive suggestions on ablations and clarity.

W1a. Methodological contribution.

Our central contribution is identifying transformation-interface coupling as a structural bottleneck and alleviating it through lightweight architectural modifications.

While residual-stream expansion increases representational capacity, feature-shared readout and rank-one write-back constrain how this capacity is utilized. Our key insight is that expanding residual streams should be accompanied by more flexible transformation interfaces, without necessarily modifying the underlying Transformer blocks. The stronger improvements with increasing stream count (Sec. 4.4) support this perspective.

W1b. Additional architectural ablations.

The original submission already included component ablations (Sec. 4.5) and four normalization variants (Appendix C.2), with MidNorm achieving the best validation loss (2.8188).

We agree that more thorough ablations are particularly valuable for lightweight architectural modifications. Following your suggestion, we additionally evaluated different readout group numbers and write-back ranks. Results are summarized in the Common Response, with further details in Sec. 4.5 and Appendix C.2.

W2 / Q1. Method illustration and metric definitions.

Following your suggestion, we added a detailed method illustration in Sec. 3.1 to clarify the readout and write-back data flows.

We also refined the metric definitions and computation procedures in Sec. 4.3 and Appendix C.6, improving the clarity and reproducibility of our mechanistic analyses.

Q2. Paloma bits-per-byte (BPB).

Your observation is correct: the original Paloma results use word-level perplexity rather than token-level perplexity, explaining their relatively large values.

Following your suggestion, we additionally report BPB for easier cross-tokenizer comparisons, with its definition, conversion procedure, and complete results provided in Appendix C.1.

Paloma BPB (↓)

ScaleMethodC4DolmaRefinedWebRedPajamaWikiText
SResidual1.31291.40491.43021.56061.3089
SmHC1.30701.41451.43991.58941.3001
SmHC-lite1.30171.41601.43001.59731.3281
SuHC1.29731.41731.41471.59061.2937
MResidual1.24251.34521.35091.52231.2093
MmHC1.21731.34421.33021.52981.1920
MmHC-lite1.21931.31991.34051.48601.1846
MuHC1.20141.30621.30291.45991.1738
LResidual1.13601.24221.24021.41391.1072
LmHC1.13961.22621.21561.36551.1530
LmHC-lite1.13481.23911.22581.40631.1440
LuHC1.11831.18981.20871.31901.0941
XLmHC1.08671.16451.19931.29881.0266
XLuHC1.08181.15791.18691.29041.0441

W3. Limited experimental scale.

We extended our evaluation to 1.07B parameters with a 10B-token budget, where uHC improves validation loss from 2.6556 to 2.6433, reduces LAMBADA perplexity from 18.86 to 17.32, and outperforms mHC on 4 of 5 Paloma corpora. Full results are provided in the Common Response and Sec. 4.2.

We appreciate your constructive suggestions, which have helped strengthen our experimental evidence and improve the clarity of our presentation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

Common Response — Additional Experiments and Revisions

We thank all reviewers for their thoughtful feedback and constructive suggestions. We have conducted additional experiments and revised the manuscript to address the shared concerns regarding experimental scale, architectural ablations, and analysis clarity.

1. Scaling to 1B Parameters

We additionally trained 1.07B-parameter mHC and uHC models with a 10B-token budget under matched training configurations.

Importantly, the mHC baseline exhibits stable gradient dynamics at this scale, yet uHC retains an advantage across most evaluation metrics.

MetricmHCuHC
Val Loss ↓2.65562.6433
OWT-val PPL ↓14.420414.3724
WT103 PPL ↓20.413720.0178
LAMBADA PPL ↓18.864017.3176
LAMBADA Acc. ↑39.98%40.83%

Paloma Word-level Perplexity (↓)

MethodC4DolmaRefinedWebRedPajamaWikiText-103
mHC91.81190.24140.11655.0545.03
uHC89.96184.64133.14628.0548.05

uHC outperforms mHC on 4 of 5 Paloma corpora. Notably, these improvements persist even when the mHC baseline no longer exhibits the pronounced gradient instability observed at the L scale, suggesting that uHC's benefits are not solely attributable to mitigating gradient spikes.

These results have been added to Sec. 4.2 (Main Results), including the main performance table and gradient diagnostics. Detailed training configurations and full XL-scale gradient trajectories are provided in Appendix A and Appendix C.7, respectively.

2. Additional Architectural Ablations

We further conducted ablations on the readout group number G∈{2,4,8}G\in\{2,4,8\} and write-back rank r∈{4,8,16}r\in\{4,8,16\} at the M scale, using a 7.5K-step training budget with otherwise identical settings.

ConfigurationSetting 1Setting 2Setting 3
Rank rr, with G=4G=4r=4r=4: 3.276178r=8r=8: 3.274388r=16r=16: 3.274416
Groups GG, with r=8r=8G=2G=2: 3.278666G=4G=4: 3.274388G=8G=8: 3.283081

Values denote validation loss (↓) at 7.5K training steps.

The results show that G=4G=4 achieves the best performance, while increasing the rank from 8 to 16 provides negligible improvement. This supports our default choice of G=n=4G=n=4 and r=8r=8, balancing performance and parameter efficiency.

We have incorporated these findings into Sec. 4.5 (Ablation Studies), with detailed results and training curves provided in Appendix C.2.

3. Clarifications and Other Manuscript Revisions

Beyond the additional experiments, we have revised the experimental analysis and clarified the definitions and computation procedures of several metrics:

  • Sec. 4.3: Clarified readout-utilization statistics and write-back diversity analysis.
  • Appendix C.1: Expanded evaluation protocols and added Paloma bits-per-byte (BPB) results.
  • Appendix C.6: Formalized the write-back directional metrics, layer-wise routing divergence, and AdamW update-allocation statistics.
  • Appendix C.7: Added full gradient-norm trajectories for the 1.07B models.

We have also added a detailed architectural diagram and improved the presentation of the proposed readout and write-back mechanisms. To further improve reproducibility, we have also included the source code and configuration files in the supplementary materials, enabling independent verification and reproduction of our experiments.

We sincerely appreciate the reviewers' constructive feedback. Their suggestions motivated these targeted experiments and helped us clarify both the empirical evidence and the scope of our claims. We hope these additions provide a more comprehensive and transparent evaluation of uHC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.