acceptodds
Under review as a conference paper at ICLR 2027

DAHC: Differentiation-Augmented Hyper-Connections via Structured Transformations

Abstract

Hyper-Connections (HC) expand the Transformer residual state into multiple parallel streams, yet the resulting streams need not become correspondingly differentiated representations. We show that, in mHC, stream differentiation emerges gradually during training and is reorganized non-monotonically across network depth. Motivated by these observations, we propose Differentiation-Augmented Hyper-Connections (DAHC), which introduces structured sources of stream differentiation both at stream creation and throughout depth. DAHC combines Differentiated Stream Expansion (DSE), which replaces replicated initialization with stream-indexed causal transformations, and Layerwise Differentiation Augmentation (LDA), which constructs depth-adapted stream candidates and integrates them through state-conditioned write-back while preserving the native mHC pathway. Empirically, DAHC reaches the final mHC training loss and validation perplexity after 7.3k and 7.5k updates, requiring 27% and 25% fewer updates, respectively. At the matched 10k-update budget, DAHC further reduces validation perplexity from 29.34 to 28.49, corresponding to a 2.90% improvement over mHC.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Hyper-Connections, Residual Connection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8PxBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DAHC augments four-stream manifold-constrained Hyper-Connections (mHC). Differentiated Stream Expansion replaces identical initial streams with causal depthwise convolutions with receptive fields 1, 2, 3 and 4, preserving an identity branch. Layerwise Differentiation Augmentation reprojects the original token embeddings at each layer, constructs stream-specific candidates, and adds them after the native attention-side update using state-conditioned gates. The motivation comes from training-time and depth-wise analyses using cosine similarity, linear CKA and an effective-width statistic. On DCLM training, the main eight-layer comparison reports validation perplexity of 28.49 versus 29.34 for mHC and reaches the baseline's final perplexity after about 7.5k rather than 10k updates. Additional experiments vary width and depth and include a 0.6B-scale training-loss comparison. The intended contribution is to make stream differentiation explicit at initialization and throughout depth.

Critical strengths
  1. The proposed intervention is concrete and preserves autoregressive causality. Different causal receptive fields provide a simple source of input-dependent differences, while LDA can inject depth-specific information with a bounded gate. Section 4.2 explicitly acknowledges that the augmented update is not constrained to the original mHC manifold. Section 4.3 also describes the extra projection and convolution costs. These details make the design understandable and avoid claiming an unproved inheritance of the original constraint.
  2. The paper includes both modeling outcomes and representation observations, with some component separation. The main comparison contains native mHC, mHC+DSE and full DAHC under approximately matched parameter counts, and Table 1 extends the perplexity gains to four depth and width settings. Appendix A provides geometry trajectories for all eight layers rather than only selected layers. The paper also acknowledges that DAHC does not produce the most separated streams at every layer. These observations give a useful initial case for investigating structured multi-stream residual states.
Critical weaknesses
  1. The study does not adequately isolate the proposed contribution or substantiate efficiency. Learned Stream Scaling (Alimaskina et al., 2026), which the paper already cites, directly addresses initial stream symmetry but is not evaluated. The current three-way comparison also lacks an LDA-only variant and controls for repeated access to the original embedding, identical versus different convolution kernels, and state-dependent versus constant gates. Adjusting FFN widths to match total parameters is useful, but does not match FLOPs or establish that the gain comes from differentiation rather than altered local context or skip pathways. LDA adds one d-by-d projection per layer; fewer updates alone do not demonstrate lower training cost. A 25% reduction in updates yields a time saving only if the per-update overhead is less than about 33%, which is not measured. All runs use seed 42, and the 0.6B result reports training loss without corresponding validation perplexity. Missing exact FFN widths, parameter counts, learning rates, gate scale alpha, and held-out split construction further limit replication. These limitations are consequential for both incremental novelty and practical benefit.
  2. The geometry analysis supports descriptive differences but not the proposed mechanism of increased useful capacity. Equation (5) computes participation ratio after averaging token-level Gram matrices. This need not equal the average effective dimension at a token: two streams can be collinear at every token, with their relative signs varying so that the averaged Gram matrix is the identity and the reported width is 2. For example, scalar stream activations [1,-1,1,-1] and [1,1,-1,-1] both have zero mean and exhibit exactly this behavior. Thus, the per-token direction interpretation in Section 3 is too strong. Also, lower cosine or CKA may reflect the introduced filtering and embedding injection without implying more useful information or explaining the perplexity gain. The authors appropriately call these diagnostics rather than direct quality measures, but controlled interventions or stream-removal/readout probes are still needed to link them to the claimed architectural rationale.
Questions for authors
  1. Evidence that could change the contribution assessment: How does DAHC compare with the cited Learned Stream Scaling method under the same backbone and training data? Please provide an LDA-only result and at least a control that repeats the projected original embedding identically across streams. These comparisons would distinguish differentiated representations from the benefit of an added embedding skip path.
  2. Evidence that could change the efficiency assessment: What are the actual parameter counts, FFN widths, per-update FLOPs, throughput and peak memory for mHC, mHC+DSE and DAHC? Does DAHC reach the baseline validation target with less wall-clock time or accelerator compute when the additional projections are counted?
  3. Clarification of reliability and evaluation: Can the main perplexity comparison be repeated with paired random seeds, and can validation perplexity be reported for the 0.6B run? Please state the exact held-out DCLM construction, shard lists or identifiers, data exposure, learning-rate schedule, gate scale alpha and initialization. What setup difference explains the eight-layer d=768 mHC perplexity of 29.34 in the main comparison versus 30.32 in Table 1?
  4. Clarification of the geometry interpretation: Do you intend Equation (5) to measure diversity over the token population rather than dimension within each token state? Please report participation ratio before token averaging, or qualify the interpretation. Does a functional probe, such as removing or restricting stream reads, show that the additional differences are used by the model and predict the perplexity benefit?
  5. Clarification of stability and reproduction: Since the added candidates bypass Sinkhorn normalization, how do activation and gradient norms behave across depth and training? Please identify the exact pretrained checkpoint and provide analysis code for the 33-sequence depth study, including CKA token sampling and centering. These would support the stated observations without requiring pretraining a production-scale DAHC model.
5 comments
AuthorsOpenReview ↗

Responses to Reviewer 8PxB (Part 1)

Thanks for the detailed comments and suggestions. Responses are below:

Q1: Architectural Contribution and Mechanism Isolation. How does DAHC compare with Learned Stream Scaling under the same backbone and training data? Additional controls, including LDA-only and shared embedding injection across streams, are needed to distinguish the benefit of stream differentiation from that of an additional embedding skip pathway.

A1: We thank the reviewer for these constructive suggestions. We conduct additional comparisons with Learned Stream Scaling (LSS), LDA-only, direct embedding injection, and shared-transformation variants under matched parameter budgets and identical training configurations. The results demonstrate the effectiveness of DAHC beyond learned stream scaling and simple input reinjection, while supporting the additional value of differentiated transformations.

  • Comparison with LSS: Following the reviewer's suggestion, we incorporate LSS into mHC using the same backbone and training data. LSS reduces the validation loss from 3.374776 to 3.372401, whereas DAHC achieves a substantially lower validation loss of 3.344909, demonstrating the effectiveness of our architectural design beyond learned stream scaling in the evaluated setting.
  • LDA-only and Embedding Skip: Retaining only LDA without DSE reduces the validation loss to 3.353499, demonstrating its effectiveness independently of differentiated initialization. We further evaluate a simpler embedding skip pathway that directly injects the original token embeddings identically across streams, retaining DSE but removing the LDA projection, convolution, and gating operations. This variant achieves a validation loss of 3.360227, which is higher than that of DAHC. These results suggest that repeated access to input embeddings alone does not fully account for the benefits of the complete LDA design.
  • Additional Benefits of Differentiated Transformations: We additionally introduce a fully shared variant that retains local contextual processing and layerwise input injection while sharing convolutional features and auxiliary gates across streams. This variant achieves a validation loss of 3.344220, comparable to the original DAHC. More importantly, introducing stream-specific multi-scale convolutions with kernel sizes (1, 2, 3, 4) into the shared design, while keeping the gating mechanism, parameter budget, and training configurations unchanged, further reduces the validation loss to 3.330977. This controlled comparison supports the additional value of differentiated transformations beyond shared contextual processing and repeated input access. The resulting variant also outperforms the original DAHC, suggesting that a well-structured shared starting representation and subsequent differentiated augmentation can be complementary.

Table A. Additional architectural comparisons and ablations under matched parameter budgets.

MethodTrain Loss ↓Validation Loss ↓
mHC3.4412653.374776
mHC + LSS3.4407883.372401
mHC + LDA3.4176633.353499
mHC + DSE + Direct Embedding Injection3.4258173.360227
mHC + Shared-DSE + Shared-LDA3.4076643.344220
DAHC (Original)3.4091183.344909
mHC + Shared-DSE + Differentiated-LDA3.3952243.330977
AuthorsOpenReview ↗

Responses to Reviewer 8PxB (Part 2)

Q2: Computational Cost and Training Efficiency. Please provide exact parameter counts, FFN widths, per-update FLOPs, throughput, and peak memory for mHC, mHC+DSE, and DAHC, and evaluate whether DAHC reduces actual wall-clock time or accelerator compute to reach a matched validation target.

A2: We thank the reviewer for highlighting the importance of practical training efficiency. We provide detailed parameter accounting, theoretical FLOPs estimates, and actual training-cost measurements for mHC, mHC+DSE, and DAHC. The results show that DAHC achieves lower cumulative training FLOPs and shorter elapsed training time to the matched validation target under nearly identical parameter and per-update FLOPs budgets, despite the additional runtime overhead of the current unoptimized implementation.

  • Parameter and Compute Accounting: For the representative 8-layer model with hidden size 768, DAHC reduces the SwiGLU FFN intermediate width from 2308 to 2048 to compensate for the additional projection, convolution, and gating parameters. We further estimate training FLOPs independently, accounting for the complete LDA injection pathway, including layerwise dense projections, depthwise convolutions, gate computation, and gated write-back. The injection branch adds approximately 30.345 TFLOPs per global update, largely offset by the 30.151 TFLOPs saved through FFN narrowing. As shown below, all three models are closely matched in both parameter counts and estimated per-update model FLOPs.
  • Practical Training Efficiency: Using historical two-device training logs, we compare the computational cost of reaching the same validation target, defined by the final validation loss of mHC. Although DAHC incurs higher per-update runtime and memory consumption, it reduces estimated cumulative training FLOPs by 19.98% and 11.10% relative to mHC and mHC+DSE, respectively. More importantly, these improvements translate into observed elapsed training-time reductions of 9.74% and 4.32%, respectively. Since all three models use the same number of devices, these reductions also correspond to the same relative savings in accelerator-hours.
  • Implementation Optimization: Importantly, the current DAHC implementation has not undergone dedicated performance optimization, leaving room for further improvement. In particular, the embedding-dependent candidate construction in LDA is independent of the evolving residual-stream states, providing opportunities for precomputation, kernel fusion, and overlapping candidate construction with the main Transformer pathway. We will further investigate these optimization opportunities and empirically evaluate their runtime benefits.

Table B. Computational cost and training efficiency comparison (8 layers, hidden size 768).

MetricmHCmHC+DSEDAHC
FFN Width230823082048
Exact Parameters296,112,112296,119,024296,121,008
Estimated FLOPs/Update (TF)1170.141170.181170.38
Mean Step Time (s) ↓11.79312.33813.299
Throughput (k tokens/s) ↑88.9084.3678.80
Peak Memory (GiB) ↓46.9247.3648.88
Updates to Target ↓10,0009,0008,000
Estimated Cumulative FLOPs (EF) ↓11.70110.5329.363
Elapsed Time to Target (h) ↓33.05331.18029.834
AuthorsOpenReview ↗

Responses to Reviewer 8PxB (Part 3)

Q3: Experimental Reliability and Reproducibility. Additional paired-seed experiments, held-out validation of the 0.6B model, and detailed training configurations are needed, including DCLM data construction, shard identifiers, data exposure, learning-rate schedule, gate scale, and initialization. The perplexity discrepancy between the main experiment and Table 1 also requires clarification.

A3: We thank the reviewer for highlighting the importance of experimental reliability and reproducibility. We provide additional paired-seed experiments, held-out and downstream evaluation results, detailed data and training configurations, and clarification of the metric-labeling error in Table 1. The additional results consistently favor DAHC across the evaluated settings and support the robustness of the reported improvements across random seeds.

  • Larger-scale Validation: Using the 0.6B checkpoints at iteration 30,000, we evaluate both models on 10,240 held-out DCLM validation samples, where DAHC reduces perplexity from 19.82 to 19.49 (1.70%). We further evaluate five downstream benchmarks and two independent test distributions under identical protocols. DAHC improves zero-shot accuracy across all five benchmarks by 0.49–2.21 percentage points and reduces perplexity on WikiText-103 and PTB by 2.11% and 13.82%, respectively.
  • Data and Training Configurations: We use DCLM-Baseline-1.0, specifically global-shard_01_of_10/local-shard_{0,...,7}_of_10. The held-out validation set is constructed using Megatron's dataset partitioning and sampling mechanism with a train/validation/test split of 949:50:1. Both mHC and DAHC use identical data and training settings at each model scale. The main 8-layer experiment and the 0.6B experiment process approximately 10.49B and 125.83B training token positions, respectively. Both scales adopt cosine learning-rate decay with a maximum learning rate of 1e-3, a minimum learning rate of 1e-4, and a 10% warmup ratio.
  • Gate Scale and Initialization: The write-back scale α is set to 10 at both model scales. The LDA projection matrices and gate vectors are initialized from a zero-mean normal distribution with a standard deviation of 0.02, while the depthwise convolution kernels in both DSE and LDA are initialized as uniform averaging filters, with kernel-size-1 branches implemented as identity mappings.
  • Multi-seed Robustness: Following the reviewer's suggestion, we repeat the main 8-layer experiment with two additional paired seeds (1 and 2), keeping all other settings unchanged. Across the three seeds (42, 1, and 2), DAHC achieves lower training loss (3.406732 ± 0.002193 vs. 3.438400 ± 0.003325) and validation loss (3.343098 ± 0.001688 vs. 3.372790 ± 0.002231) than mHC (mean ± standard deviation). The mean loss reductions substantially exceed the observed standard deviations, supporting the robustness of DAHC across random seeds.
  • Clarification of Table 1: We apologize for the incorrect metric labeling in Table 1, which reports perplexity calculated from training loss, whereas Fig. 3 reports validation perplexity. For the 8-layer, hidden-size-768 configuration, the training perplexities are 30.32 and 29.73 for mHC and DAHC, respectively, compared with validation perplexities of 29.34 and 28.49. This discrepancy reflects a metric-labeling error rather than different training configurations. We will correct the label to training perplexity in the revised manuscript. Table C. Additional evaluation of the 0.6B models.
EvaluationMetricmHCDAHCImprovement
DCLM Held-out ValPPL ↓19.8219.49−1.70%
HellaSwagAcc ↑34.7235.56+0.84 pp
ARC-EasyAcc ↑55.8156.31+0.50 pp
ARC-ChallengeAcc ↑22.1023.21+1.11 pp
WinoGrandeAcc ↑52.0154.22+2.21 pp
CommonsenseQAAcc ↑35.4635.95+0.49 pp
WikiText-103 TestPPL ↓14.6014.29−2.11%
PTB TestPPL ↓195.03168.08−13.82%

Table D. Data and training configurations.

ConfigurationMain Experiment0.6B Experiment
Transformer Layers812
Sequence Length10244096
Global Batch Size10241024
Training Steps10,00030,000
Warmup Steps1,0003,000
Training Token Positions10.49B125.83B
Shared ConfigurationSetting
DatasetDCLM-Baseline-1.0
Train/Val/Test Split949:50:1
Optimizer / (β₁, β₂)Adam / (0.9, 0.95)
Max/Min Learning Rate1e-3 / 1e-4
LR Schedule / Warmup RatioCosine / 10%
Weight Decay / Gradient Clipping0.1 / 1.0
Write-back Scale α10
DSE / LDA Kernel Sizes(1, 2, 3, 4)
Convolution InitializationUniform averaging
Projection / Gate InitializationNormal (mean=0, std=0.02)
Random Seed42
Reviewer_HcsSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DAHC, an extension of mHC using different causal convolutions to initialize residual streams and gated input transformations at each layer. In the main 8-layer experiment, validation perplexity decreases from 29.34 to 28.49, and the baseline’s final validation performance is reached in approximately 25% fewer steps. The paper analyzes stream geometry during training and across depth. However, the value of the stream-specific design remains unclear, and missing settings limit assessment of the matched-budget comparisons. These limitations leave important questions about the contribution unresolved.

Critical strengths
  1. Validation ppl improves across several depths and hidden sizes under reportedly approximately matched parameter budgets. DSE alone helps, and adding LDA improves the main result further.

  2. The analysis shows how stream geometry evolves during training and varies across depth in a pre-trained model.

Critical weaknesses
  1. The benefit of the stream-specific design is not isolated. DSE adds local context processing, while LDA provides direct access to input representations at each layer. A simpler baseline retaining these features but sharing convolutional features and auxiliary gates across streams would help establish whether the stream-specific choices provide a meaningful advantage.

  2. The matched-budget comparisons are insufficiently specified. FFN widths are adjusted to compensate for added parameters, but exact widths and parameter counts are missing. The different perplexities in the main 8-layer experiment and Table 1 also require an explanation of the configuration differences.

Questions for authors

Can the authors provide a matched-budget comparison with a simpler variant that retains local convolutions and layerwise input injection but shares their features and auxiliary gates across streams?

  1. Please provide exact FFN widths, parameter counts, and training settings, including the write-back scale, and explain the different results in the main experiment and Table 1.

  2. What is the main new architectural contribution relative to xHC and Engram, and what evidence supports its usefulness beyond the existing components?

  3. Сould the authors provide results from additional seeds, held-out evaluation of the 0.6B checkpoints, and training-time measurements? These would strengthen the empirical evidence.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer HcsS (Part 1)

Q1: Contribution of the Stream-specific Design. The current experiments do not sufficiently isolate the benefit of stream-specific differentiation from additional local contextual processing and layerwise input injection. A shared-feature and shared-gate baseline is needed to establish the contribution of the stream-specific design.

A1: We thank the reviewer for this constructive suggestion. We conduct additional ablations with shared convolutional features and auxiliary gates under matched parameter budgets and identical training configurations. The results demonstrate that shared contextual processing is beneficial, while differentiated transformations provide further improvements beyond the fully shared design.

  • Shared-feature and Shared-gate Baseline: Following the reviewer's suggestion, we first replace the stream-specific transformations in DSE with a shared convolutional kernel of size 3, reducing the validation loss from 3.374776 to 3.353439. Further incorporating LDA with shared convolutional features and auxiliary gates reduces the validation loss to 3.344220, comparable to that of the original DAHC (3.344909). These results highlight the importance of constructing a well-structured starting representation and acknowledge the contribution of shared contextual processing and layerwise input access.
  • Additional Benefits of Differentiated Transformations: More importantly, introducing stream-specific multi-scale convolutions with kernel sizes (1, 2, 3, 4) into the shared design, while keeping the gating mechanism, parameter budget, and training configurations unchanged, further reduces the validation loss from 3.344220 to 3.330977, outperforming both the fully shared variant and the original DAHC. This controlled comparison provides evidence that differentiated transformations offer additional benefits beyond shared contextual processing and layerwise input injection. Interestingly, these results further suggest that a well-structured shared starting representation and subsequent differentiated augmentation can be complementary, refining and supporting our original motivation.

Table A. Comparison of shared and stream-specific designs under matched parameter budgets.

MethodTrain Loss ↓Validation Loss ↓
mHC3.4412653.374776
mHC + Shared-DSE3.4200943.353439
mHC + Shared-DSE + Shared-LDA3.4076643.344220
DAHC (Original)3.4091183.344909
mHC + Shared-DSE + Differentiated-LDA3.3952243.330977

Q2: Specification of Matched-budget Comparisons. The matched-parameter comparisons lack exact FFN widths and parameter counts. The configuration differences between the main 8-layer experiment and Table 1 also require clarification.

A2: We thank the reviewer for pointing out these ambiguities. We provide the exact model configurations, parameter counts, and training settings below, and clarify that the discrepancy between Fig. 3 and Table 1 arises from a metric-labeling error rather than different training configurations.

  • Parameter and Training Configurations: In the main approximately 296M-parameter experiment, DAHC reduces the SwiGLU FFN intermediate width from 2308 to 2048 to compensate for the additional projection, convolution, and gating parameters, resulting in nearly identical parameter counts (296,112,112 vs. 296,121,008). Both models are trained under identical settings for 10,000 steps, corresponding to approximately 10.49B training tokens. The detailed configurations are provided below.
  • Clarification of Table 1: We apologize for the incorrect metric labeling in Table 1. The perplexity values in Table 1 were calculated from training loss, whereas Fig. 3 reports validation perplexity. Specifically, for the 8-layer, hidden-size-768 configuration, the training perplexities are 30.32 and 29.73 for mHC and DAHC, respectively, while the corresponding validation perplexities are 29.34 and 28.49. We will correct the metric label in Table 1 to training perplexity and clarify this distinction in the revised manuscript.

Table B. Model configurations and training settings for the main 8-layer experiment.

ConfigurationSetting
Layers / Hidden Size / Heads / Streams8 / 768 / 16 / 4
FFN ActivationSwiGLU
FFN Width (mHC / DAHC)2308 / 2048
Parameters (mHC / DAHC)296,112,112 / 296,121,008
DSE / LDA Kernel Sizes(1, 2, 3, 4)
Write-back Scale α10
Dataset / Train-Val-Test SplitDCLM / 949:50:1
Sequence Length / Global Batch / Micro Batch1024 / 1024 / 16
Training Steps / Tokens10,000 / 10.49B
Optimizer / (β₁, β₂)Adam / (0.9, 0.95)
Learning Rate / Schedule / Warmup1e-3 → 1e-4 / Cosine / 1000 steps
Weight Decay / Gradient Clipping0.1 / 1.0
Attention / Hidden Dropout0.1 / 0.1
Random Seed42
AuthorsOpenReview ↗

Responses to Reviewer HcsS (Part 2)

Q3: Can the authors provide a matched-budget comparison with a simpler variant that retains local convolutions and layerwise input injection but shares their features and auxiliary gates across streams?

A3: Following the reviewer's suggestion, we evaluate a matched-budget variant that retains local convolutions and layerwise input injection while sharing convolutional features and auxiliary gates across streams. This variant achieves a validation loss of 3.344220, comparable to the original DAHC (3.344909). More importantly, introducing stream-specific multi-scale transformations while keeping the gating mechanism, parameter budget, and training configurations unchanged further reduces the validation loss to 3.330977, supporting the additional value of differentiated transformations beyond the fully shared design. Please refer to A1 for the complete ablation results and discussion.

Q4: Please provide exact FFN widths, parameter counts, and training settings, including the write-back scale, and explain the different results in the main experiment and Table 1.

A4: We thank the reviewer for requesting these details. As provided in A2, the FFN intermediate widths are 2308 and 2048 for mHC and DAHC, respectively, resulting in nearly identical parameter counts (296,112,112 vs. 296,121,008). Both models share identical training settings, and the write-back scale α is set to 10 for DAHC. Regarding the discrepancy between Fig. 3 and Table 1, this results from an incorrect metric label in Table 1, where perplexity was calculated from training loss rather than validation loss. We will correct the label to training perplexity in the revised manuscript. Please refer to A2 for the complete model configurations, training settings, and detailed clarification.

Q5: What is the main new architectural contribution relative to xHC and Engram, and what evidence supports its usefulness beyond the existing components?

A5: We thank the reviewer for raising this important question. The main architectural contribution of DAHC lies in explicitly introducing structured sources of stream differentiation at both stream creation and throughout network depth, rather than in the individual transformation operators themselves. Although DAHC shares certain architectural components with xHC and Engram, their design motivations, information sources, and architectural roles are different.

  • Architectural Distinction: xHC primarily addresses the scalability of residual-stream expansion by enriching block-output write-back signals and introducing sparse residual updates. Engram introduces conditional memory through N-gram lookup to decouple knowledge retrieval from neural computation. In contrast, DAHC addresses a complementary architectural question: how to better realize the representational potential of existing residual streams without increasing the stream count or introducing an external memory table.
  • Differentiation-oriented Design: Our design is directly motivated by two empirical observations: stream differentiation emerges progressively during optimization and evolves non-monotonically across network depth (Fig. 1). Accordingly, DSE introduces structured differentiation at stream creation through stream-specific causal transformations, while LDA provides layer-adaptive sources of differentiation by repeatedly constructing stream-specific representations from the shared pre-expansion representation. This coordinated design explicitly supports stream differentiation from initialization throughout layerwise evolution, while preserving the native mHC pathway.
  • Empirical Evidence: Under approximately matched parameter budgets, DSE alone reduces validation perplexity from 29.34 to 28.86 (1.64%), and incorporating LDA further reduces it to 28.49 (2.90% relative to mHC). Moreover, the representation-level analysis (Fig. 4) reveals changes in cross-stream geometry consistent with our design objectives, while Table 1 demonstrates consistent performance improvements across different model depths and hidden dimensions. These results support the usefulness of the proposed architectural interventions from both performance and representation perspectives.
AuthorsOpenReview ↗

Responses to Reviewer HcsS (Part 3)

Q6: Could the authors provide results from additional seeds, held-out evaluation of the 0.6B checkpoints, and training-time measurements? These would strengthen the empirical evidence.

A6: We thank the reviewer for these suggestions. To strengthen the empirical evidence, we conduct additional held-out validation, downstream evaluations, multi-seed experiments, and training-cost analyses. The results consistently favor DAHC across the evaluated benchmarks and random seeds, while demonstrating practical training-time savings in the representative 8-layer configuration despite the additional overhead of the current unoptimized implementation.

  • Larger-scale Validation: Using the 0.6B checkpoints at iteration 30,000, we evaluate both models on 10,240 held-out DCLM validation samples, where DAHC reduces perplexity from 19.82 to 19.49 (1.70%). To further assess its broader utility, we additionally evaluate both models on five downstream benchmarks and two independent test distributions under identical evaluation protocols. DAHC consistently improves zero-shot accuracy across all five benchmarks, with gains ranging from 0.49 to 2.21 percentage points, and reduces perplexity on WikiText-103 and PTB by 2.11% and 13.82%, respectively. These results demonstrate that the improvements extend beyond training loss and persist across diverse evaluation settings.
  • Training Efficiency: For the representative 8-layer configuration, mHC and DAHC have nearly identical parameter counts and estimated model FLOPs per training step. Although the current DAHC implementation has not undergone dedicated performance optimization and incurs additional per-step overhead, the historical training logs show a 9.74% reduction in observed wall-clock time to the matched validation target.
  • Multi-seed Robustness: We repeat the main 8-layer experiment with two additional random seeds (1 and 2), keeping all other settings unchanged. Across the three seeds (42, 1, and 2), DAHC achieves lower training loss (3.406732 ± 0.002193 vs. 3.438400 ± 0.003325) and validation loss (3.343098 ± 0.001688 vs. 3.372790 ± 0.002231) than mHC (mean ± standard deviation). These consistent improvements provide further evidence of the robustness of DAHC across random seeds.

Table C. Additional empirical evaluation and training-cost comparison.

Evaluation / MetricmHCDAHCImprovement
0.6B-scale Evaluation
HellaSwag (Acc ↑)34.7235.56+0.84 pp
ARC-Easy (Acc ↑)55.8156.31+0.50 pp
ARC-Challenge (Acc ↑)22.1023.21+1.11 pp
WinoGrande (Acc ↑)52.0154.22+2.21 pp
CommonsenseQA (Acc ↑)35.4635.95+0.49 pp
DCLM Held-out Val (PPL ↓)19.8219.49−1.70%
WikiText-103 Test (PPL ↓)14.6014.29−2.11%
PTB Test (PPL ↓)195.03168.08−13.82%
Training Cost (8 layers, d=768)
Parameters (M)296.112296.121—
Estimated FLOPs/Step (TF)1170.1401170.378—
Mean Step Time (s) ↓11.79313.299—
Throughput (k tokens/s) ↑88.978.8—
Peak Memory (GiB) ↓46.9248.88—
Time to Matched Validation Target (h) ↓33.0529.83−9.74%
Reviewer_PnrDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Differentiation-Augmented Hyper-Connections (DAHC), extending mHC with two components: differentiated stream expansion through causal depthwise convolutions, and layerwise injection of projected input representations through state-conditioned gates. The motivation is that replicated residual streams acquire differentiation gradually during training and exhibit non-monotonic changes in geometry across depth. The paper reports improved validation perplexity and fewer updates to reach the baseline’s final loss.

Critical strengths
  1. The paper proposes a concrete architectural modification and reports consistent validation perplexity improvements across the four configurations in Table 1.
  2. The DSE-only ablation provides some evidence about the contributions of the proposed components.
  3. The analysis of stream geometry provides a useful descriptive view of how the representations change during training and across layers.
Critical weaknesses
  1. The empirical evaluation is too limited to establish the method’s broader value. The performance evaluation primarily relies on training loss and validation perplexity, with no downstream task evaluation or independent evaluation distribution. Additional diagnostics, such as predictive entropy, could help characterize how the method changes model predictions, although these would not replace broader quality evaluation. The larger 0.6B experiment reports training-loss improvements without corresponding validation results. Furthermore, the experiments use a single stated random seed, leaving the robustness of the relatively modest gains unclear. Overall, the experimental coverage is insufficient to support a strong conclusion about the method’s utility.

  2. The additional computational cost is not adequately evaluated. DAHC introduces layerwise dense projections, convolutions, and gating. Although the authors reduce FFN widths to approximately match parameter budgets, this does not establish comparable FLOPs, memory consumption, or runtime. The reported 25–28% reduction in optimization steps therefore does not demonstrate a corresponding reduction in actual training cost. The paper should report throughput, peak memory, and wall-clock time to reach a matched validation target, together with exact parameter counts and a compute comparison.

  3. The ablations do not isolate the proposed mechanism. DSE adds local contextual processing, while LDA introduces repeated access to input representations, additional projections, and multiplicative gating. These changes could improve performance independently of stream differentiation. Essential missing controls include LDA without DSE, a shared contextual transformation replicated across streams, and an auxiliary pathway that injects the same content into every stream. The current results show that the architectural package improves the tested models, but do not establish that correcting insufficient stream differentiation explains the gains.

Questions for authors
  1. Can you provide LDA-only and shared-content injection ablations to separate the benefit of stream differentiation from additional contextual processing and input access?
  2. Can you evaluate DAHC beyond loss and perplexity to better characterize the reported improvements?
3 comments
AuthorsOpenReview ↗

Responses to Reviewer PnrD (Part 1)

Thanks for the detailed comments and suggestions. Responses are below:

Q1: Limited Empirical Evaluation and Broader Utility. The empirical evaluation is insufficient to establish the broader utility and reliability of DAHC, as it lacks downstream evaluations, independent-distribution testing, validation results at the 0.6B scale, and multi-seed experiments.

A1: We thank the reviewer for highlighting the need for broader empirical evaluation. To address these concerns, we conduct additional evaluations using the 0.6B-scale checkpoints at iteration 30,000 reported in the paper, including five downstream benchmarks, two independent test distributions, held-out DCLM validation, and predictive-distribution diagnostics. The results consistently favor DAHC over mHC, providing additional evidence beyond the original training-loss comparison.

  • Broader Evaluation: Under identical zero-shot evaluation protocols, DAHC consistently improves accuracy across all five downstream benchmarks using total token log-likelihood for answer selection, with gains of up to 2.21% on WinoGrande. Moreover, we evaluate the models on 10,240 held-out DCLM validation samples and the WikiText-103 and PTB test sets using identical token-level evaluation protocols. DAHC reduces perplexity by 1.70% on the held-out DCLM validation split and by 2.11% and 13.82% on the WikiText-103 and PTB test sets, respectively. These results demonstrate that the improvements extend beyond training loss and persist on the evaluated independent text distributions.
  • Predictive behavior: Following the suggestion, we additionally examine predictive entropy and calibration. DAHC slightly reduces predictive entropy on WikiText-103 (2.70869 → 2.70626 nats/token) and PTB (4.71365 → 4.66592 nats/token), while achieving lower 15-bin token-level ECE on both datasets. Although lower entropy alone does not imply higher prediction quality, the calibration results provide complementary evidence characterizing changes in predictive behavior.
  • Multi-seed Robustness: To assess the reliability of the reported improvements, we repeat the main 8-layer experiment with two additional random seeds (1 and 2), replacing the original seed of 42 while keeping all other settings unchanged. Across the three seeds, DAHC achieves lower training loss (3.406732 ± 0.002193 vs. 3.438400 ± 0.003325) and validation loss (3.343098 ± 0.001688 vs. 3.372790 ± 0.002231) than mHC (mean ± standard deviation). The improvements in mean losses substantially exceed the observed standard deviations, providing further evidence of the robustness of DAHC across random seeds.

Table A. Additional evaluation of the 0.6B models. Acc is reported in %, and pp denotes percentage points. PPL is computed at the token level using the Qwen3 tokenizer. ECE denotes 15-bin token-level top-label expected calibration error.

EvaluationMetricmHCDAHCImprovement
HellaSwagAcc ↑34.7235.56+0.84 pp
ARC-EasyAcc ↑55.8156.31+0.50 pp
ARC-ChallengeAcc ↑22.1023.21+1.11 pp
WinoGrandeAcc ↑52.0154.22+2.21 pp
CommonsenseQAAcc ↑35.4635.95+0.49 pp
DCLM Held-out ValPPL ↓19.8219.49−1.70%
WikiText-103 TestPPL ↓14.6014.29−2.11%
PTB TestPPL ↓195.03168.08−13.82%
WikiText-103 TestECE ↓0.008120.00747−8.00%
PTB TestECE ↓0.016170.00897−44.53%
AuthorsOpenReview ↗

Responses to Reviewer PnrD (Part 2)

Q2: Computational Cost and Training Efficiency. The additional computational overhead of DAHC is not sufficiently evaluated. Matching parameter budgets does not guarantee comparable FLOPs, memory usage, or runtime, and the reported reduction in optimization steps does not necessarily translate into lower actual training cost.

A2: We thank the reviewer for highlighting the importance of practical training efficiency. We agree that fewer optimization steps do not necessarily translate into proportional wall-clock savings. To clarify this distinction, we provide detailed parameter and computational cost analyses, together with actual training-time measurements, using the 8-layer configuration as a representative example.

  • Parameter and Compute Accounting: For the representative 8-layer model with hidden size 768, DAHC reduces the SwiGLU intermediate width from 2308 to 2048 to offset the additional injection parameters, resulting in nearly identical parameter counts (296.112M vs. 296.121M). We further estimate training FLOPs independently, accounting for the complete LDA injection pathway, including layerwise dense projections, depthwise convolutions, gate computation, and gated write-back. The injection branch requires approximately 30.345 TFLOPs per global training step, largely offset by the 30.151 TFLOPs saved through FFN narrowing. Including the differentiated input expansion, the estimated full-model costs are approximately 1170.140 and 1170.378 TFLOPs/step for mHC and DAHC, respectively. Thus, the two models are closely matched in both parameter count and theoretical model FLOPs, although this does not imply identical runtime.
  • Practical Training Efficiency: As shown in Table B, although the current DAHC implementation incurs higher per-step runtime and memory consumption, it reduces the observed wall-clock time to the matched validation target by 9.74% compared with native mHC. This demonstrates that the faster convergence can translate into actual training-time savings, although the reduction is smaller than the reported 25% decrease in optimization steps.
  • Implementation Optimization: Importantly, the current implementation has not undergone dedicated performance optimization, leaving substantial room for improvement. In particular, the LDA injection branch constructs its candidates from the shared input representation independently of the evolving residual-stream computations, providing opportunities for precomputation, kernel fusion, and overlapping injection computations with the main Transformer pathway. Such optimizations could reduce the current runtime overhead without changing the proposed architecture.

Table B. Computational cost comparison (8 layers, hidden size 768).

MetricmHCDAHC
FFN Width23082048
Parameters (M)296.112296.121
Estimated FLOPs/Step (TF)1170.1401170.378
Mean Step Time (s) ↓11.79313.299
Throughput (k tokens/s) ↑88.978.8
Peak Memory (GiB) ↓46.9248.88
Time to Matched Validation Target (h) ↓33.0529.83
Wall-clock Time Reduction—9.74%
AuthorsOpenReview ↗

Responses to Reviewer PnrD (Part 3)

Q3: Insufficient Ablation of Stream Differentiation. The current ablations do not sufficiently isolate the contribution of stream-specific differentiation, as the improvements may arise from additional contextual processing, input access, or gating mechanisms. Missing controls, including LDA-only, shared contextual transformations, and shared-content injection, make it unclear whether stream differentiation itself is responsible for the observed gains.

A3: We thank the reviewer for this insightful suggestion. To isolate the contribution of stream-specific differentiation, we conduct additional ablations under matched parameter budgets and identical training configurations. The results highlight the importance of both the starting representation and subsequent differentiated augmentation, demonstrating that differentiated transformations provide additional benefits beyond shared contextual processing and layerwise input access.

  • LDA-only: Retaining only LDA without DSE reduces the validation loss from 3.374776 to 3.353499, demonstrating that layerwise differentiated augmentation is effective independently of differentiated initialization.
  • Shared Contextual Processing: Following the reviewer's suggestion, we first apply a shared convolutional kernel of size 3 at stream creation, reducing the validation loss to 3.353439. Further incorporating shared convolutional features and auxiliary gates in LDA reduces the validation loss to 3.344220, comparable to the original DAHC (3.344909). These results highlight the importance of constructing a well-structured starting representation and confirm that shared contextual processing and layerwise input access contribute to the improvements. More importantly, introducing stream-specific multi-scale convolutions with kernel sizes (1, 2, 3, 4) into the shared design, while keeping the gating mechanism, parameter budget, and training configurations unchanged, further reduces the validation loss to 3.330977, outperforming both the fully shared variant and the original DAHC. This controlled comparison supports the additional value of differentiated transformations beyond shared contextual processing and layerwise input access. These findings further refine our original motivation: a well-structured shared starting representation and subsequent differentiated augmentation can be complementary in improving multi-stream representations.

Table C. Additional architectural ablations under matched parameter budgets.

MethodTrain Loss ↓Validation Loss ↓
mHC3.4412653.374776
mHC + LDA3.4176633.353499
mHC + Shared-DSE3.4200943.353439
mHC + Shared-DSE + Shared-LDA3.4076643.344220
DAHC (Original)3.4091183.344909
mHC + Shared-DSE + Differentiated-LDA3.3952243.330977

Q4: Can you provide LDA-only and shared-content injection ablations to separate the benefit of stream differentiation from additional contextual processing and input access?

A4: Following the reviewer's suggestion, we conduct additional LDA-only and shared-content injection ablations, as detailed in A3 (Table C). The results demonstrate that LDA is effective independently of DSE, and that introducing differentiated multi-scale transformations provides further improvements beyond shared contextual processing and layerwise input access. Please refer to A3 for the detailed experimental results and discussion.

Q5: Can you evaluate DAHC beyond loss and perplexity to better characterize the reported improvements?

A5: We thank the reviewer for this suggestion. As detailed in A1 (Q1), we additionally evaluate the 0.6B models on five downstream benchmarks and two independent test distributions. DAHC consistently improves accuracy across all five downstream tasks, with gains of up to 2.21 percentage points on WinoGrande. We further analyze predictive entropy and calibration on WikiText-103 and PTB, where DAHC achieves lower token-level ECE on both datasets, providing complementary evidence of improved predictive calibration. These additional results extend our evaluation beyond loss and perplexity. Please refer to Table A in A1 for the complete results and evaluation protocols.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.