- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The authors propose Uncoupled Hyper-Connections (uHC) which is an extension of manifold constrained Hyper-Connections (mHC) that targets "transformation-interface coupling." In current multi-stream architectures, the MLP block reads from all streams using a single routing pattern and writes back the same output vector to every stream. uHC addresses this with a group-wise feature readout which lets different feature subsets blend at the same parameter cost as mHC. uHC also uses a low-directional write-back which adds a stream specific correction, which gives each stream a distinct update direction. The authors validate the method by pretraining models from 46M-363M on OpenWebText. They report lower validation loss and generally better perplexity than residual, mHC, and mHC-lite baselines. However, all results come from single runs without any reported variance. The headline result is measured against an mHC baseline that underperformed the plain residual baseline so the validation does not convincingly support the improvements.
Critical strengths
-
The authors have a fairly precise reason for a real limitation. Hyper-Connection readouts use a single routing pattern which is shared across features and the write back sends every stream a scaled copy of the same vector. Which means the write matrix is rank one no matter how many streams you add. Mathematically elegant and nice to verify which helps provide an explanation for why adding more residual streams may be diminishing. Which I believe is a useful insight in further work.
-
The proposed fix is minimal and cheap. The Group-wise readout keeps the same number of parameters as mHC's readout. The low-rank write back only adds a small number of parameters back to the layer. The initialization of uHC starts the same as mHC so any deviation is just learned. That means for any codebase already using mHC an easy drop in uHC is possible and any changes would be attributable to the method.
Critical weaknesses
-
I have some doubts on the statistical support. There appears to be no seeds or error bars, and I do not see multiple runs reported so I believe this all may be from a singular run. At S and M (Table 1) uHC only has a small advantage and a worse perplexity score on LAMBADA at M. This could just be noise or run-to-run variation. So the claims may be overstated.
-
I worry that there may be an issue with the reported baselines. At the largest scale the plain residual outperforms mHC and mHC-lite which might be pointing to an unstable or undertuned baseline. The core claim of this paper is tied to that baseline. It is entirely possible that uHC may just have a more stable training under the hyperparameters.
-
The experiments may be too small to support scaling claims. All models used are under 1B and trained on approx 8B tokens at most (Can you double check that number?). Architectural gains at this size frequently matter less under larger sizes or with better tuning. The lack of runs larger than 363M or even additional seed runs do not leave me convinced of the claim.
Overall as it stands, none of the evidence provided can support the claimed improvements. This could all be alleviated with additional runs.
Questions for authors
At L scale, plain residual outperforms both mHC and mHC-lite, contradicting prior work. Did you tune the learning rate or other hyperparameters separately for each method, or reuse one configuration?
Can you show an mHC run at L that is at least competitive with the residual baseline, and report uHC’s gain against it?
How much does validation loss vary across random seeds in your setup?
Do you have any results at 1B parameters or above, even a single run? If not, why should readers expect the gains to persist where mHC’s original instability problem actually arises?
Is uHC’s benefit at L mainly that it avoids the gradient spikes mHC suffers?
Would mHC with gradient clipping, a lower learning rate, or longer warmup close most of the gap?
3 comments
Thank you for the additional experiments. I have three remaining points:
-
The 1338 seed L-scale rerun shows the gap against a stable mHC baseline shrinks from 0.128 to 0.001 in val loss. Will the abstract, intro, and Table 1 be revised to report this run alongside, or instead of, the original L-scale run?
-
In the original L-scale results, the plain residual outperformed both mHC variants. Could you add the residual baseline under the same 5B token L-scale setting, so readers can see whether uHC improves over a standard residual connection when all methods train stably?
-
The ablation conclusions and several main table gaps rest on loss differences of 0.00003-0.012. Could you report 2-3 seeds at the S or M scale so readers can judge which differences exceed run to run variance?
We thank the reviewer for these constructive questions. We have conducted further experiments and updated the manuscript accordingly.
Q1. Revising the original L-scale results
We acknowledge the weaker performance of the original L-scale mHC baseline. In the independent 5B-token experiment, the validation-loss gap between mHC and uHC decreases from 0.1282 to 0.0011.
We retain the original L-scale results in Table 1 for transparency, rather than replacing them with results from a different seed and training budget. Meanwhile, we have revised the Abstract and Introduction to emphasize the 1.07B-scale results, where mHC trains stably and uHC retains its advantage.
The independent L-scale comparison is provided in Appendix C.2.1.
Q2. Residual baseline under stable optimization
We additionally trained residual and mHC-lite baselines under the same L-scale configuration (seed 1338, 5B tokens, 19,073 steps).
| Method | OWT-val ↓ | WT103 ↓ | LAMBADA ↓ |
|---|---|---|---|
| Residual | 18.37 | 31.77 | 37.20 |
| mHC | 17.75 | 26.47 | 31.24 |
| mHC-lite | 17.80 | 26.10 | 32.09 |
| uHC | 17.66 | 24.63 | 32.54 |
| Paloma Word-level Perplexity (↓) |
| Method | C4 | Dolma | Falcon | RedPajama | WikiText |
|---|---|---|---|---|---|
| Residual | 107.89 | 267.20 | 154.53 | 1046.93 | 63.85 |
| mHC | 112.25 | 223.15 | 158.01 | 788.99 | 59.43 |
| mHC-lite | 113.70 | 226.99 | 158.61 | 801.06 | 63.37 |
| uHC | 106.20 | 205.23 | 144.20 | 676.24 | 56.03 |
All three multi-stream variants outperform residual connections on OWT-val, WT103, and LAMBADA. uHC achieves the best results on seven of eight perplexity evaluations, including all five Paloma corpora, with LAMBADA being the only exception.
The full-training and late-stage gradient trajectories further confirm that both mHC and uHC train stably without pronounced late-stage spikes. These results demonstrate that uHC retains its advantages against a stable, competitive mHC baseline.
Complete results and gradient diagnostics are provided in Appendix C.2.1 (Table 7 and Figure 5).
Q3. Random-seed variability
We extended the M-scale rank and grouping ablations to two seeds (1337 and 1339) under matched 7.5K-step budgets.
G=4 achieves the lowest validation loss under both seeds. Ranks 8 and 16 show comparable mean performance (3.278019 ± 0.005134 vs. 3.277639 ± 0.004557), supporting our choice of r=8 for parameter efficiency.
While these two-seed ablations provide additional evidence on run-to-run variability, we acknowledge that they do not establish statistical significance for all main-table comparisons. Given the computational cost and limited rebuttal timeline, we prioritized the matched L-scale comparison and 1.07B-scale validation. We plan to further expand the multi-seed evaluation in subsequent revisions.
The results are reported in Sec. 4.5 and Appendix C.2 (Table 6).
Together with the stable 1.07B-scale comparison, these experiments strengthen the evidence that uHC's benefits are not solely attributable to the original L-scale mHC instability.
We thank the reviewer for recognizing our structural insight and lightweight design, and for the critical feedback on experimental reliability.
W3 / Q4. Larger-scale validation and training budget.
We have extended our evaluation to 1.07B parameters with a 10B-token budget, beyond the original L-scale experiments (7.86B tokens).
At this scale, uHC reduces validation loss from 2.6556 to 2.6433, improves LAMBADA perplexity from 18.86 to 17.32, and outperforms mHC on 4 of 5 Paloma corpora. Importantly, both models exhibit stable gradient dynamics.
Complete results are provided in the Common Response, Sec. 4.2, and Appendix C.7.
W2 / Q1 / Q2 / Q6. Baseline fairness and hyperparameter tuning.
We acknowledge the weaker performance of the original L-scale mHC baseline. All main experiments use shared scale-specific configurations, including learning rate, warmup, and gradient clipping (1.0), without method-specific tuning.
To further address this concern, we retrained matched L-scale mHC and uHC models with a different random seed (1338), using the same L-scale architecture and optimization settings with a 5B-token budget (19,073 steps). Both models exhibit stable gradient dynamics.
Additional L-scale results (5B tokens, seed 1338):
Paloma evaluation (↓):
The new mHC run achieves substantially lower monitored validation loss than the original run, despite a shorter training budget. Against this stronger, stable mHC baseline, uHC improves OWT-val, WT103, and all five Paloma corpora, with LAMBADA being the exception.
These results provide further evidence that uHC's gains do not depend on the original unstable mHC baseline. We also provide the anonymous source code and training configurations in the supplementary materials to facilitate independent reproduction.
W2 / Q5. Are the improvements primarily due to avoiding gradient spikes?
We observed recurring gradient instability in L-scale mHC training and further investigated its potential connection to write-back coupling and optimizer update allocation in Appendix C.5.
Both the new L-scale and 1.07B experiments show that uHC retains performance advantages even when mHC trains stably, suggesting that its benefits extend beyond mitigating gradient instability.
W1 / Q3. Statistical reliability and random-seed variability.
Our main experiments use matched configurations and random seeds for controlled comparisons. The additional L-scale comparison with seed 1338 provides complementary evidence across training runs.
Given the computational cost of large-scale training, we prioritized matched comparisons with a stronger, stable mHC baseline and validation at the 1.07B scale. Although these experiments do not establish statistical significance or directly quantify seed variance, they provide additional evidence that uHC's improvements are not solely attributable to the instability of the original baseline.
We appreciate the reviewer's careful scrutiny and constructive suggestions. The additional experiments and analyses have strengthened our empirical evaluation and provided broader evidence for uHC's architectural benefits.