acceptodds
Withdrawn from ICLR 2027

MV-HC: Multi-View Hyper-Connections for Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hyper-Connections (HC) improve residual routing by expanding the hidden state into multiple interacting streams, while recent variants such as mHC, mHC-lite, and KromHC improve the stability and scalability of this multi-stream formulation. However, these methods typically initialize all streams from the same representation, diversifying routing but not the source information carried by each stream. We introduce Multi-View Hyper-Connections (MV-HC), which instead initializes dedicated hyper-streams from complementary tokenization views of the same input. An auxiliary representation is causally aligned to the main-token sequence and propagated through KromHC-style manifold-constrained residual mixing, while a lightweight gated reduction combines the heterogeneous streams before the LM head. We evaluate MV-HC from M to B parameters and observe consistent improvements over KromHC in validation and zero-shot downstream performance. At B, MV-HC reduces BPB from to , improves CORE from to , and improves the broader language-modeling/BBH evaluation average from to . These gains require only additional parameters, while training throughput remains within of KromHC (k vs. k tokens/s). These results show that complementing multi-path residual routing with multi-view source representations provides a simple and efficient way to enrich hyper-connections.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hyper-Connections, auxiliary representation, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tbe9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MV-HC initializes separate KromHC residual streams from BPE embeddings and causally aligned auxiliary-token embeddings, then combines the final streams through a learned gate. It retains the original residual mixer and shared Transformer branch. Experiments from 60M to 1.5B parameters report improved validation and downstream performance, with component, tokenizer-choice, and stream-similarity analyses and approximately 1% throughput overhead relative to KromHC.

Critical strengths

S1. Simple and compatible construction. The main tokenizer retains the prediction interface, auxiliary alignment explicitly respects the observed prefix, and each layer executes the expensive Transformer branch only once.

S2. Reported improvements across model scales. MV-HC improves validation BPB at 60M, 186M, and 1.5B. At 1.5B, BPB decreases from 0.628 to 0.617, while commonsense and broader evaluation averages improve from 55.3 to 56.7 and 35.1 to 36.7, respectively.

S3. Useful existing controls. Table 7 includes parameter matching, gating, and auxiliary-fusion baselines. Tokenizer comparisons and late-layer stream similarities provide additional diagnostics beyond aggregate benchmark results.

S4. Small incremental cost over KromHC. At 1.5B, the reported additional parameters increase from 1.36M to 2.05M, while throughput decreases from 63.5k to 62.9k tokens/s. This makes the extension inexpensive within the existing HC architecture.

Critical weaknesses

W1. Limited methodological novelty. The residual mixer and layer update are inherited from KromHC; the changes are auxiliary-embedding initialization and final gating. Combining character/subword representations also has substantial precedent, including the cited CharBERT. The described lookup/projection/pooling path offers a compositional token representation, leaving the conceptual advance relatively modest.

W2. The decisive routing control is missing. Table 7's KromHC+AuxFuse reaches 0.699 BPB/48.2 CORE versus 0.696/48.6 for full MV-HC, but lacks the latter's final gate. An AuxFuse+gate control is needed to attribute this remaining gain to keeping views in separate streams. Lower stream cosine similarity alone does not establish useful information diversity.

W3. Baseline provenance and CORE reporting need reconciliation. The small-model baseline rows in Tables 2–3 largely match the published KromHC tables, despite the statement that all baselines were trained from scratch. Meanwhile, Table 1 reports KromHC CORE of 42.4/47.7, versus 9.018/16.872 in the cited paper. This may reflect different runs or metric definitions, but their relationship must be documented. See KromHC, Tables 3–5.

W4. Essential reproduction details are missing. The paper omits auxiliary embedding dimensions, concrete tokenizer configurations, the selected pooling rule, and optimization settings for the new parameters and 1.5B model. ParamMatch and the “shared tokenizer” variant are insufficiently defined; describing tokenizer sharing “across layers” is also unclear when auxiliary tokenization occurs only at initialization.

W5. Scaling beyond small dense models is unverified. Experiments cover 60M–1.5B dense models with four residual streams. Do the gains and approximately 1% incremental throughput overhead persist at larger scales, particularly a 30B MoE? Evidence or analysis of the scaling trend, compatibility with expert routing, and training/inference memory costs would strengthen the practical significance.

Questions for authors
  1. Which baseline results were rerun versus taken from KromHC? Please reconcile the differing CORE values and specify the exact metric computation and evaluation protocol.

  2. How does KromHC+AuxFuse perform with the same final gate as MV-HC, and is the remaining difference stable across training seeds?

2 comments

Summary. We thank the reviewer. Both questions are answered with new results: AuxFuse + gate (W2, Q2) is run with 3 seeds, and we revised our claims; baseline provenance and CORE (W3, Q1) are reconciled below. We commit to releasing reproduction details (W4), add a scaling analysis (W5), and address novelty (W1). Some points overlap with Reviewer bj7q; we refer to that reply to save space. We hope these responses address your main concerns and encourage you to raise your score.

W2/Q2 (AuxFuse + gate). This was the decisive control. Over 3 seeds at 186M (BPB, mean ± std):

MethodBPB↓
KromHC.7043±.0015
KromHC+AuxFuse+gate.6983±.0021
MV-HC.6967±.0021

AuxFuse+gate beats KromHC in every seed and recovers ~80% of MV-HC's gain. MV-HC is lower in 2 of 3 seeds, but the remaining 0.0016 BPB is within seed noise (seed-1 CS Avg.: 49.0 vs. 49.3). The gain therefore comes mainly from the causally aligned auxiliary view, not from keeping it in separate streams; we revised Sec. 3.6, App. A.8 and the Conclusion accordingly. We agree that lower cosine similarity alone does not show useful diversity. The controls in our reply to 9Fpt (Q1) test this: noise-induced diversity gives no gain, an unrelated byte view is down-weighted by the gate, and corrupting the aligned view removes the gain.

W3/Q1 (provenance, CORE). All models were trained by us, and the training loss and BPB in Table 1 come from our runs. However, the 60M/186M baseline rows of Tables 2–3 were copied from the KromHC paper instead of being evaluated on our checkpoints. We re-evaluated all eight rows with the same pipeline as MV-HC (table in our reply to bj7q, M1); MV-HC remains best on average at every scale. Our "CORE" column was the unweighted mean accuracy over the 11 commonsense tasks of Table 2, not DCLM's centered CORE, so it is not comparable to 9.018/16.872. It is now named CS Avg. and defined in Sec. 3.3. ΔParams was also recounted; our KromHC overhead now equals KromHC's exactly (240K/959K).

W4 (reproduction). We will add a full configuration table to the appendix (tokenizers, auxiliary dimensions, pooling, gate, optimization, the 1.5B setup, and the ParamMatch and shared-tokenizer variants) and release the code.

W5 (scaling). We have not trained beyond 1.5B or MoE models, and the Limitations now state this. The analysis is favorable. With the same n, MV-HC's per-layer computation and residual-stream memory are identical to KromHC's; MV-HC only changes what fills the streams. Its extra cost sits at the input (byte lookup, d_a×d projection, pooling) and the output (gate) and does not grow with depth, so its relative overhead shrinks as models grow. The parameter overhead beyond KromHC is 192(256+d)+d+1, i.e. 0.20M at d = 768. Extra activation memory is B×S_aux×(192+d), at the input only. MV-HC does not change the attention/FFN branch, so expert routing receives the same aggregated input as in KromHC.

W1 (novelty). We agree the change to KromHC is small; see our reply to bj7q (M4). Unlike CharBERT, which fuses character and subword channels in a bidirectional encoder, MV-HC targets autoregressive LMs: its causal alignment admits only auxiliary units completed by the current position, and the views enter as hyper-connection streams rather than through a fusion module. Our analyses also show which views help (causal coverage, App. A.13) and that diversity alone does not.

Thanks for the response from the authors. I am still concerned with the novelty thus I keep my rating.

Reviewer_bj7qOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Multi-View Hyper-Connections (MV-HC), an extension of KromHC that replaces the identical initialization of residual streams with heterogeneous initializations derived from BPE and auxiliary byte-level embeddings. The authors additionally introduce a gated reduction over the final residual streams. They train models ranging from 60M to 1.5B parameters and demonstrate consistent, but modest, improvements in validation loss and downstream language modeling benchmarks.

Critical strengths

The method is simple and intuitive, requiring minimal modifications to the existing KromHC architecture. I also appreciate the large scale evaluations.

Critical weaknesses

The paper is unnecessarily verbose and repetitive, introducing substantial terminology to describe relatively simple architectural modifications. The authors should consider a substantial rewrite for clarity.

The proposed method is extremely incremental

  • The auxiliary byte-level representation is simply another embedding of the same input token. Under the proposed byte-level alignment scheme, the auxiliary representation is a deterministic function of the BPE token ID. Thus, the method does not introduce additional information.
  • It is unclear whether the improvements even arise from the proposed heterogeneous residual streams or simply from providing the network with an additional learned embedding function. I would have liked to see a baseline with independently parameterized BPE embeddings instead of byte-level embeddings.

Insufficient experimental evidence and inconsistencies in baseline reporting.

  • Appendix A.8 compares MV-HC with KromHC + AuxFuse, which already achieves 0.699 BPB versus 0.696 for the full method. However, this baseline does not include the proposed final gated reduction. The most relevant comparison, KromHC + AuxFuse + gated reduction, is missing.
  • The paper does not report multi-seed experiments or uncertainty estimates, and the gains are modest, so i cannot recommend acceptance without
  • Concerningly, the 60M and 186M baseline downstream results in Tables 2–3 almost exactly reproduce those from the original KromHC paper, despite the authors claiming that all baselines were retrained from scratch. Meanwhile, the corresponding training losses, BPB, CORE scores, and parameter counts differ substantially. For example, the original 186M KromHC model reports 2.966 training loss, 0.862 BPB, and 16.872 CORE, compared with 2.936, 0.704, and 47.7 here.
Questions for authors
  • Can the authors produce multi-seed evaluations?
  • Can the authors clarify the novelty concerns?
  • Can the authors clarify the reported results mismatch with KromHC?
2 comments

Summary. We thank the reviewer, especially for catching the baseline inconsistency. Mismatch with KromHC (M1): the 60M/186M baseline rows of Tables 2–3 are now re-evaluated on our checkpoints, and the CORE label and ΔParams are corrected. Multi-seed (M2): 3 seeds at 60M/186M, including both controls you asked for. AuxFuse + gate (M3): run with 3 seeds; it closes most of the gap, and we revised our claims. Independent BPE embedding / novelty (M4): run with 3 seeds (B3). Verbosity (M5). We hope these responses address your main concerns and encourage you to raise your score.

M1 (mismatch). (a) All baselines were trained by us, but the 60M/186M baseline rows of Tables 2–3 were copied from the KromHC paper instead of being evaluated on our checkpoints. We re-evaluated all eight rows with the same pipeline as MV-HC. MV-HC remains best on average at every scale and beats KromHC on 19/20 tasks at 60M and 20/20 at 186M. (b) Our "CORE" column was the mean accuracy over the 11 commonsense tasks of Table 2, not DCLM's centered CORE, so it is not comparable to KromHC's 16.872. It is now named CS Avg. and defined in Sec. 3.3. (c) ΔParams was miscounted. Recounted, our KromHC overhead equals KromHC's exactly (240K/959K), confirming the same architecture; MV-HC adds 0.12M/0.20M beyond it, still less than mHC and mHC-lite. (d) Training loss and BPB come from our own runs and evaluation pipeline. All comparisons in our paper use a single pipeline.

ModelMethodBPB↓CS Avg↑LM/BBH↑ΔP
60MResidual.78641.516.60
mHC.77942.317.6.46M
mHC-lite.78142.217.6.61M
KromHC.77242.817.9.24M
MV-HC.76443.518.9.36M
186MResidual.72147.123.10
mHC.71547.823.71.84M
mHC-lite.71347.823.82.43M
KromHC.70448.224.2.96M
MV-HC.69649.326.21.16M

M2 (multi-seed). Yes; details are in our reply to Reviewer 9Fpt (C3) and App. A.11. MV-HC beats KromHC in every seed at both scales, by ~ 0.0075 BPB (~ 3× std), and improves mean CS Avg. by 0.7–1.0. 1.5B remains a single run, as stated in the new Limitations paragraph. BPB, mean ± std over 3 seeds:

ModelMethodBPB↓
60MKromHC.7717±.0025
MV-HC.7643±.0025
186MKromHC.7043±.0015
B3: 2nd BPE embedding.7003±.0025
KromHC+AuxFuse+gate.6983±.0021
MV-HC.6967±.0021

M3 (AuxFuse + gate). This was the right control. KromHC+AuxFuse+gate beats KromHC in every seed and recovers ~80% of MV-HC's gain. MV-HC is lower in 2 of 3 seeds, but the remaining 0.0016 BPB is within seed noise. We revised Sec. 3.6, App. A.8 and the Conclusion accordingly: the gain comes mainly from the causally aligned auxiliary view, and keeping it in separate streams adds at most a small further gain. AuxFuse uses our alignment module and byte view; it differs from MV-HC only in where the view enters.

M4 (independent BPE embedding; novelty). We agree that the byte view is a deterministic re-encoding of the input, and have revised the text to describe it as a complementary representation. Our motivation is that different granularities and parameter-sharing structures can provide useful representations even when they encode the same text.

B3 (App. A.12) is the proposed control: a second, independently learned BPE embedding in the auxiliary streams, with the same final gate. It recovers about half of the mean BPB improvement over KromHC (table above). MV-HC has a lower mean BPB with 33× fewer auxiliary-path parameters (0.20M vs. 6.44M), although three runs give limited evidence about the remaining difference. These results suggest a parameter-efficiency advantage for the byte representation; they do not establish its strict superiority. The single-run controls in our reply to 9Fpt (Q1) are consistent with input alignment being what matters: noise-based stream diversity gives no improvement, an unrelated byte view is down-weighted by the gate, and corrupting the aligned byte view degrades performance.

MV-HC is a lightweight extension of KromHC. Its central contribution is a causal alignment that lets a second tokenization view initialize hyper-connection streams; as M3 shows, this aligned view, more than its placement in separate streams, provides most of the gain. The coverage and diversity analyses (App. A.12–A.13) investigate when such a view is useful. Across scales up to 1.5B, the validation improvements come at modest cost: about 0.11% additional parameters beyond KromHC at 186M and a 0.95% throughput reduction at 1.5B.

M5 (verbosity). We will condense Sec. 2, move derivations to the appendix, and remove repeated terminology.

Thank you for the detailed response; however, I'm still concerned about novelty and will maintain my score.

Reviewer_9FptOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Recap:

The paper presents an alternative axis of improvement on KromHC and other recent Hyper Connection methods. They point out that prior work had focused on building attractive topologies, namely, by ensuring that the hyper connection was parameterized by a doubly-stochastic matrix. Their method, MV-HC, for Multi-View Hyper Connections, uses alternative tokenization views of the main input sequence as initializations of hyper connection streams. They show that aligning and gating auxiliary tokenizations allows them to recover the standard language model shape. Their method is not a novel approach to hyper connection solvers or algorithms – just a different perspective on what they should be doing and where to start from. They choose to implement MV-HC on top of KromHC, with only an additional ~1% overhead. They show that by training loss (CE), BPB, and CORE metrics their MV-HC is marginally better than KromHC, mHC, and mHC-lite. They also show attractive results on downstream benchmarks by similar margins of about 1-2%.

Critical strengths

Strengths:

(1) The authors identify a genuinely new perspective illuminating a limitation of prior HC methods, namely, the initialization of the HC.

(2) The method is well motivated, claims are generally appropriately scoped by the evidence, the experiments are clear and well-motivated, and the paper is well written.

Critical weaknesses

Weaknesses:

(1) Figure 2 makes an unsupported claim w.r.t the convergence loss of MV-HC vs KromHC and others. Figure 2 does not show any of the methods at convergence, and no link to this evidence is present. I find this claim believable, but it must be supported.

(2) The auxiliary tokenizer used isn’t clearly stated, and no method for choosing the auxiliary tokenizer(s) seems to be presented, so long as it is different from the primary source. Clearly link this to Appendix A.5 where it is discussed.

(3) All results shown appear to be for n=1n=1 runs. I appreciate that larger sample sizes are expensive additions for experiments with LLMs, but this fact nevertheless lessens the strength of the conclusions that can be reached.

Questions for authors

Questions:

(1) This question may have been explored in prior work – but are token or residual based initializations of the HC actually better than random initializations? ie, is MV-HC good just because it offers diversity, or because the diversity has some structural advantage? This could potentially be studied by noising the auxiliary and/or primary initializations or their duplicates per eqn 17. If prior work explores this question, clearly cite it.

(2) What makes which auxiliary tokenizer choices good or bad? Is there some way to understand this other than simply going out and measure it? I believe this is connected to my Question 1.

(3) What might happen if more than two tokenizers are used? I don’t see the source of the two tokenizer limitation and am assuming it is self-imposed. One can imagine eqn 17 working very nicely for 3 tokenizers at n=6, for example.

Conclusion:

I am an emergency reviewer, and as such have had extremely limited time to complete my initial review, and I have not done research on Hyper Connections myself in the past. I will be updating my review during the response window. That said, I think these authors’ manuscript would likely be a solid addition to the ICLR conference.

My score of Weak Accept can improve if:

• The authors can supply some results with sample sizes greater than one, reporting variance on benchmarks.

• The convergence claims are supported or tempered.

• Identify if MV-HCs advantage is from diversity alone or from some structural component, or if this is a false difference (or both?).

2 comments

Summary. We thank the reviewer. All three score conditions are addressed: variance: 3 seeds at 60M/186M (C3); convergence claims: tempered and linked to end-of-training results (C1); diversity vs. structure: new controls show diversity alone does not help, and the gain needs input-aligned auxiliary content (Q1). Q2 and Q3 are answered with new analyses. We hope these responses address your main concerns and encourage you to raise your score.

C1 (convergence). Agreed: Fig. 2 shows steps 5k–7k, not convergence. The caption and Sec. 3.3 now claim only that MV-HC has the lowest loss in this window, and point to Table 1 for end-of-training results (1.5B vs. KromHC: train loss 2.452 vs. 2.481, BPB 0.617 vs. 0.628). The appendix tokenizer figure now cites Table 4. Full training curves will be added.

C2 (auxiliary tokenizer). All main experiments use BPE (primary) and byte-level (auxiliary); this is now stated in the Introduction, Sec. 3, Sec. 3.1 and Table 1, with a direct link to the tokenizer comparison (App. A.7). Sec. 2.2 now gives the selection criteria: auxiliary units no coarser than primary tokens, full coverage, small vocabulary. Q2 shows these criteria predict the measured ranking.

C3 (n=1). We trained 2 more seeds of KromHC and MV-HC at 60M and 186M. MV-HC wins on BPB in every seed, by 0.0075 ( 3× std), and improves mean CS Avg. by 0.7–1.0 (App. A.11). 1.5B remains a single run, now stated in Sec. 3.1 and a new Limitations paragraph; claims resting on ≤0.003 BPB ablation gaps are softened.

ModelMethodBPB↓CS Avg↑
60MKromHC.7717±.002542.82±0.35
MV-HC.7643±.002543.55±0.21
186MKromHC.7043±.001548.17±0.25
MV-HC.6967±.002149.17±0.16

Q1 (diversity vs. structure). To our knowledge, prior HC work (HC, mHC, KromHC) initializes all streams identically and does not test this. Following your suggestion, we fixed the setup (186M, n=4, 2 main + 2 aux streams, gate) and changed only what fills the auxiliary streams (App. A.12). Diversity alone does not help: noise variants (B1, B2, B5c) match or exceed MV-HC's stream diversity with no gain; an unrelated byte view (B4, from another document) is ignored by the gate; noising the byte view removes the gain monotonically (B5a–c). A second BPE embedding (B3: same tokens, no sub-token structure) recovers only part of it. The advantage is structural: it needs input-aligned sub-token content.

VariantBPB↓CS Avg↑Cos↓Aux gate
KromHC.70448.2.889–
B0 Main copies+gate.70248.3.895.49
B1 Main+noise.70348.1.780.35
B2 All noised.70647.9.690.50
B3 2nd BPE emb..70048.9.820.46
B4 Unrelated byte.70448.2.750.23
B5a Byte+0.5σ.69849.1.740.44
B5b Byte+1σ.70148.4.680.33
B5c Byte+2σ.70547.9.590.17
B6 MV-HC.69649.3.786.48

σ = RMS of byte view; Cos = mean pairwise stream cosine, last 4 layers, 4K context; single seed.

Q2 (what makes a tokenizer good). At 1.5B (App. A.13), the gain follows causal coverage, i.e. the share of positions where an auxiliary unit is already available. Byte/char units are available at every position; word units arrive only after the whole word is read, leaving 28% of positions empty. Redundancy with the primary embedding (R²) does not predict the gain, and the gate down-weights coarse views. This links Q1 and Q2: useful views carry input-aligned content available at every position.

AuxBPB↓ΔBPBCS Avg↑Cov.↑Delay↓R²GateAux→main
None.628–55.3–––––
Word.625−.00355.772%.48.68.31.09
Morph.622−.00656.084%.25.55.37.12
Char.619−.00956.499.7%.02.70.43.15
Byte.617−.01156.7100%0.72.48.17

Cov. = % text positions with an available aux unit; Delay = mean positions until available; Gate = aux-stream softmax mass (.5 = uniform); Aux→main = mean row-sum of H^{ma}.

Q3 (more tokenizers). The two-view limit is self-imposed: Eq. 17 extends to V views, X₀ = [Repeat_{n₁}(Z⁽¹⁾), …, Repeat_{n_V}(Z⁽ⱽ⁾)], now Eq. 18 in Sec. 2.2. At 186M (App. A.14), a third (morphology) view gives a small gain at n=6 (−.003 BPB, +0.5 CS Avg.) and a negligible change at n=4, for 4–5% throughput. Notably, 2-view MV-HC at n=4 beats KromHC at n=6 (.696 vs. .702) with half the ΔP and is faster.

nMethodMain/Byte/MorphBPB↓CS Avg↑Tok/sΔP
4KromHC4/0/0.70448.2250k.96M
MV-HC2/2/0.69649.3247k1.16M
MV-HC2/1/1.69549.2239k2.09M
6KromHC6/0/0.70248.9239k2.32M
MV-HC4/2/0.69449.4235k2.52M
MV-HC2/2/2.69149.9226k3.45M

Edit: the column previously labeled CORE is the mean accuracy over the 11 commonsense tasks of Table 2, not DCLM's centered CORE; it is now named CS Avg. These values and the ΔP column were recomputed. BPB values are unchanged.

Reviewer_9FptOpenReview ↗

Response to Authors

I appreciate the efforts the authors have gone through to support their claims by providing additional results at n=3n=3, as well as the interesting investigations of alternative tokenizer selections.

I trust that the results shared in the responses, which do address many of my weaknesses and questions, will be reflected in a reuploaded manuscript, but they do not appear to be at this time. Ultimately, I do not feel that my score has changed. I still think that MV-HC would probably be a reasonable addition to ICLR. I will revisit my score in light of the conclusions of other reviewers and the resubmitted revised manuscript.

The other reviewers make good points, particularly that the measured gains are small (1-2%) and may not be beyond noise levels. I do not find the fact that MV-HC must be implemented on top of an existing HC method a serious weakness, as the authors claims are well scoped and they are very honest about the fact that MV-HC is built on an existing backbone (primarily KromHC in the text).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.