- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DAHC augments four-stream manifold-constrained Hyper-Connections (mHC). Differentiated Stream Expansion replaces identical initial streams with causal depthwise convolutions with receptive fields 1, 2, 3 and 4, preserving an identity branch. Layerwise Differentiation Augmentation reprojects the original token embeddings at each layer, constructs stream-specific candidates, and adds them after the native attention-side update using state-conditioned gates. The motivation comes from training-time and depth-wise analyses using cosine similarity, linear CKA and an effective-width statistic. On DCLM training, the main eight-layer comparison reports validation perplexity of 28.49 versus 29.34 for mHC and reaches the baseline's final perplexity after about 7.5k rather than 10k updates. Additional experiments vary width and depth and include a 0.6B-scale training-loss comparison. The intended contribution is to make stream differentiation explicit at initialization and throughout depth.
Critical strengths
- The proposed intervention is concrete and preserves autoregressive causality. Different causal receptive fields provide a simple source of input-dependent differences, while LDA can inject depth-specific information with a bounded gate. Section 4.2 explicitly acknowledges that the augmented update is not constrained to the original mHC manifold. Section 4.3 also describes the extra projection and convolution costs. These details make the design understandable and avoid claiming an unproved inheritance of the original constraint.
- The paper includes both modeling outcomes and representation observations, with some component separation. The main comparison contains native mHC, mHC+DSE and full DAHC under approximately matched parameter counts, and Table 1 extends the perplexity gains to four depth and width settings. Appendix A provides geometry trajectories for all eight layers rather than only selected layers. The paper also acknowledges that DAHC does not produce the most separated streams at every layer. These observations give a useful initial case for investigating structured multi-stream residual states.
Critical weaknesses
- The study does not adequately isolate the proposed contribution or substantiate efficiency. Learned Stream Scaling (Alimaskina et al., 2026), which the paper already cites, directly addresses initial stream symmetry but is not evaluated. The current three-way comparison also lacks an LDA-only variant and controls for repeated access to the original embedding, identical versus different convolution kernels, and state-dependent versus constant gates. Adjusting FFN widths to match total parameters is useful, but does not match FLOPs or establish that the gain comes from differentiation rather than altered local context or skip pathways. LDA adds one d-by-d projection per layer; fewer updates alone do not demonstrate lower training cost. A 25% reduction in updates yields a time saving only if the per-update overhead is less than about 33%, which is not measured. All runs use seed 42, and the 0.6B result reports training loss without corresponding validation perplexity. Missing exact FFN widths, parameter counts, learning rates, gate scale alpha, and held-out split construction further limit replication. These limitations are consequential for both incremental novelty and practical benefit.
- The geometry analysis supports descriptive differences but not the proposed mechanism of increased useful capacity. Equation (5) computes participation ratio after averaging token-level Gram matrices. This need not equal the average effective dimension at a token: two streams can be collinear at every token, with their relative signs varying so that the averaged Gram matrix is the identity and the reported width is 2. For example, scalar stream activations [1,-1,1,-1] and [1,1,-1,-1] both have zero mean and exhibit exactly this behavior. Thus, the per-token direction interpretation in Section 3 is too strong. Also, lower cosine or CKA may reflect the introduced filtering and embedding injection without implying more useful information or explaining the perplexity gain. The authors appropriately call these diagnostics rather than direct quality measures, but controlled interventions or stream-removal/readout probes are still needed to link them to the claimed architectural rationale.
Questions for authors
- Evidence that could change the contribution assessment: How does DAHC compare with the cited Learned Stream Scaling method under the same backbone and training data? Please provide an LDA-only result and at least a control that repeats the projected original embedding identically across streams. These comparisons would distinguish differentiated representations from the benefit of an added embedding skip path.
- Evidence that could change the efficiency assessment: What are the actual parameter counts, FFN widths, per-update FLOPs, throughput and peak memory for mHC, mHC+DSE and DAHC? Does DAHC reach the baseline validation target with less wall-clock time or accelerator compute when the additional projections are counted?
- Clarification of reliability and evaluation: Can the main perplexity comparison be repeated with paired random seeds, and can validation perplexity be reported for the 0.6B run? Please state the exact held-out DCLM construction, shard lists or identifiers, data exposure, learning-rate schedule, gate scale alpha and initialization. What setup difference explains the eight-layer d=768 mHC perplexity of 29.34 in the main comparison versus 30.32 in Table 1?
- Clarification of the geometry interpretation: Do you intend Equation (5) to measure diversity over the token population rather than dimension within each token state? Please report participation ratio before token averaging, or qualify the interpretation. Does a functional probe, such as removing or restricting stream reads, show that the additional differences are used by the model and predict the perplexity benefit?
- Clarification of stability and reproduction: Since the added candidates bypass Sinkhorn normalization, how do activation and gradient norms behave across depth and training? Please identify the exact pretrained checkpoint and provide analysis code for the 33-sequence depth study, including CKA token sampling and centering. These would support the stated observations without requiring pretraining a production-scale DAHC model.
5 comments
Responses to Reviewer 8PxB (Part 2)
Q2: Computational Cost and Training Efficiency. Please provide exact parameter counts, FFN widths, per-update FLOPs, throughput, and peak memory for mHC, mHC+DSE, and DAHC, and evaluate whether DAHC reduces actual wall-clock time or accelerator compute to reach a matched validation target.
A2: We thank the reviewer for highlighting the importance of practical training efficiency. We provide detailed parameter accounting, theoretical FLOPs estimates, and actual training-cost measurements for mHC, mHC+DSE, and DAHC. The results show that DAHC achieves lower cumulative training FLOPs and shorter elapsed training time to the matched validation target under nearly identical parameter and per-update FLOPs budgets, despite the additional runtime overhead of the current unoptimized implementation.
- Parameter and Compute Accounting: For the representative 8-layer model with hidden size 768, DAHC reduces the SwiGLU FFN intermediate width from 2308 to 2048 to compensate for the additional projection, convolution, and gating parameters. We further estimate training FLOPs independently, accounting for the complete LDA injection pathway, including layerwise dense projections, depthwise convolutions, gate computation, and gated write-back. The injection branch adds approximately 30.345 TFLOPs per global update, largely offset by the 30.151 TFLOPs saved through FFN narrowing. As shown below, all three models are closely matched in both parameter counts and estimated per-update model FLOPs.
- Practical Training Efficiency: Using historical two-device training logs, we compare the computational cost of reaching the same validation target, defined by the final validation loss of mHC. Although DAHC incurs higher per-update runtime and memory consumption, it reduces estimated cumulative training FLOPs by 19.98% and 11.10% relative to mHC and mHC+DSE, respectively. More importantly, these improvements translate into observed elapsed training-time reductions of 9.74% and 4.32%, respectively. Since all three models use the same number of devices, these reductions also correspond to the same relative savings in accelerator-hours.
- Implementation Optimization: Importantly, the current DAHC implementation has not undergone dedicated performance optimization, leaving room for further improvement. In particular, the embedding-dependent candidate construction in LDA is independent of the evolving residual-stream states, providing opportunities for precomputation, kernel fusion, and overlapping candidate construction with the main Transformer pathway. We will further investigate these optimization opportunities and empirically evaluate their runtime benefits.
Table B. Computational cost and training efficiency comparison (8 layers, hidden size 768).
| Metric | mHC | mHC+DSE | DAHC |
|---|---|---|---|
| FFN Width | 2308 | 2308 | 2048 |
| Exact Parameters | 296,112,112 | 296,119,024 | 296,121,008 |
| Estimated FLOPs/Update (TF) | 1170.14 | 1170.18 | 1170.38 |
| Mean Step Time (s) ↓ | 11.793 | 12.338 | 13.299 |
| Throughput (k tokens/s) ↑ | 88.90 | 84.36 | 78.80 |
| Peak Memory (GiB) ↓ | 46.92 | 47.36 | 48.88 |
| Updates to Target ↓ | 10,000 | 9,000 | 8,000 |
| Estimated Cumulative FLOPs (EF) ↓ | 11.701 | 10.532 | 9.363 |
| Elapsed Time to Target (h) ↓ | 33.053 | 31.180 | 29.834 |
Responses to Reviewer 8PxB (Part 3)
Q3: Experimental Reliability and Reproducibility. Additional paired-seed experiments, held-out validation of the 0.6B model, and detailed training configurations are needed, including DCLM data construction, shard identifiers, data exposure, learning-rate schedule, gate scale, and initialization. The perplexity discrepancy between the main experiment and Table 1 also requires clarification.
A3: We thank the reviewer for highlighting the importance of experimental reliability and reproducibility. We provide additional paired-seed experiments, held-out and downstream evaluation results, detailed data and training configurations, and clarification of the metric-labeling error in Table 1. The additional results consistently favor DAHC across the evaluated settings and support the robustness of the reported improvements across random seeds.
- Larger-scale Validation: Using the 0.6B checkpoints at iteration 30,000, we evaluate both models on 10,240 held-out DCLM validation samples, where DAHC reduces perplexity from 19.82 to 19.49 (1.70%). We further evaluate five downstream benchmarks and two independent test distributions under identical protocols. DAHC improves zero-shot accuracy across all five benchmarks by 0.49–2.21 percentage points and reduces perplexity on WikiText-103 and PTB by 2.11% and 13.82%, respectively.
- Data and Training Configurations: We use DCLM-Baseline-1.0, specifically
global-shard_01_of_10/local-shard_{0,...,7}_of_10. The held-out validation set is constructed using Megatron's dataset partitioning and sampling mechanism with a train/validation/test split of 949:50:1. Both mHC and DAHC use identical data and training settings at each model scale. The main 8-layer experiment and the 0.6B experiment process approximately 10.49B and 125.83B training token positions, respectively. Both scales adopt cosine learning-rate decay with a maximum learning rate of 1e-3, a minimum learning rate of 1e-4, and a 10% warmup ratio. - Gate Scale and Initialization: The write-back scale α is set to 10 at both model scales. The LDA projection matrices and gate vectors are initialized from a zero-mean normal distribution with a standard deviation of 0.02, while the depthwise convolution kernels in both DSE and LDA are initialized as uniform averaging filters, with kernel-size-1 branches implemented as identity mappings.
- Multi-seed Robustness: Following the reviewer's suggestion, we repeat the main 8-layer experiment with two additional paired seeds (1 and 2), keeping all other settings unchanged. Across the three seeds (42, 1, and 2), DAHC achieves lower training loss (3.406732 ± 0.002193 vs. 3.438400 ± 0.003325) and validation loss (3.343098 ± 0.001688 vs. 3.372790 ± 0.002231) than mHC (mean ± standard deviation). The mean loss reductions substantially exceed the observed standard deviations, supporting the robustness of DAHC across random seeds.
- Clarification of Table 1: We apologize for the incorrect metric labeling in Table 1, which reports perplexity calculated from training loss, whereas Fig. 3 reports validation perplexity. For the 8-layer, hidden-size-768 configuration, the training perplexities are 30.32 and 29.73 for mHC and DAHC, respectively, compared with validation perplexities of 29.34 and 28.49. This discrepancy reflects a metric-labeling error rather than different training configurations. We will correct the label to training perplexity in the revised manuscript. Table C. Additional evaluation of the 0.6B models.
| Evaluation | Metric | mHC | DAHC | Improvement |
|---|---|---|---|---|
| DCLM Held-out Val | PPL ↓ | 19.82 | 19.49 | −1.70% |
| HellaSwag | Acc ↑ | 34.72 | 35.56 | +0.84 pp |
| ARC-Easy | Acc ↑ | 55.81 | 56.31 | +0.50 pp |
| ARC-Challenge | Acc ↑ | 22.10 | 23.21 | +1.11 pp |
| WinoGrande | Acc ↑ | 52.01 | 54.22 | +2.21 pp |
| CommonsenseQA | Acc ↑ | 35.46 | 35.95 | +0.49 pp |
| WikiText-103 Test | PPL ↓ | 14.60 | 14.29 | −2.11% |
| PTB Test | PPL ↓ | 195.03 | 168.08 | −13.82% |
Table D. Data and training configurations.
| Configuration | Main Experiment | 0.6B Experiment |
|---|---|---|
| Transformer Layers | 8 | 12 |
| Sequence Length | 1024 | 4096 |
| Global Batch Size | 1024 | 1024 |
| Training Steps | 10,000 | 30,000 |
| Warmup Steps | 1,000 | 3,000 |
| Training Token Positions | 10.49B | 125.83B |
| Shared Configuration | Setting |
|---|---|
| Dataset | DCLM-Baseline-1.0 |
| Train/Val/Test Split | 949:50:1 |
| Optimizer / (β₁, β₂) | Adam / (0.9, 0.95) |
| Max/Min Learning Rate | 1e-3 / 1e-4 |
| LR Schedule / Warmup Ratio | Cosine / 10% |
| Weight Decay / Gradient Clipping | 0.1 / 1.0 |
| Write-back Scale α | 10 |
| DSE / LDA Kernel Sizes | (1, 2, 3, 4) |
| Convolution Initialization | Uniform averaging |
| Projection / Gate Initialization | Normal (mean=0, std=0.02) |
| Random Seed | 42 |
Responses to Reviewer 8PxB (Part 1)
Thanks for the detailed comments and suggestions. Responses are below:
A1: We thank the reviewer for these constructive suggestions. We conduct additional comparisons with Learned Stream Scaling (LSS), LDA-only, direct embedding injection, and shared-transformation variants under matched parameter budgets and identical training configurations. The results demonstrate the effectiveness of DAHC beyond learned stream scaling and simple input reinjection, while supporting the additional value of differentiated transformations.
Table A. Additional architectural comparisons and ablations under matched parameter budgets.