- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper distinguishes retention of the existing residual state from mixing across streams in Hyper-Connections. A doubly stochastic mixer preserves the stream-average component, so stable mixing alone does not attenuate that component between successive writes. Gated Delta Hyper-Connections (GDHC) augments either mHC or identity-mixer iHC with input-dependent scalar decay and rank-one erasure aligned with the existing write direction, while allowing erase and write strengths to differ. The paper analyzes the carried-state operator conditional on realized gates, evaluates two mixture-of-experts model sizes, and reports lower training loss, improved average downstream performance, and flatter activation profiles. Component ablations examine decay, erasure, and the frequency of decay placement.
Critical strengths
- The proposed interface is simple and technically well motivated. It preserves the existing read/write maps and mixer while adding explicit control over the old state. Equations 3 and 10–12 and Appendix A.1 explain the common-mode behavior, the selected-direction update, and the conditional norm bound clearly. I particularly appreciate the explicit distinction between a bound on the realized carried-state operator and a guarantee for the full nonlinear block, as well as the treatment of approximate Sinkhorn normalization and the numerical key-normalization floor.
- The evaluation provides meaningful evidence that this integration is useful at the tested scales. The paired runs use the same data order and training settings; both base architectures improve at both reported A1B budgets, with iHC gaining 0.86–1.30 points in the ten-task average (Table 1). The training-loss trajectory and A500M component experiments provide complementary evidence (Tables 2, 6, and 8). The accompanying state diagnostics are informative, and the paper is unusually explicit about their limitations, same-seed execution variability, mixed task-level results, and differences too small to resolve.
Critical weaknesses
- The reliability and generalization of the smaller improvements remain uncertain. The 0.13–0.25-point mHC downstream gains are modest, and the additional dense-placement benefit of combining the controls is 0.004–0.011 training-PPL units over the available single-control alternatives (Tables 2 and 8). Appendix A.7 reports approximately 0.008 PPL variation between two same-seed mHC executions. This is not a significance test for the other comparisons, but it makes the precision of the component conclusions difficult to assess. The paired trajectories are useful evidence, and the paper correctly does not present their temporal variation as independent-run uncertainty. Nevertheless, held-out language-model loss and uncertainty estimates for the downstream differences are missing from the reported evidence. Bootstrap intervals based on paired evaluation examples, together with a limited smaller-scale repeat of the most important component comparison, would help establish which effects are reproducible. I do not regard the absence of repeated full 4B/7B training runs, by itself, as a reason to reject the paper.
- The experiments establish the usefulness of the combined modification more directly than the benefit of its particular write-aligned, independently gated formulation. Depth-wise rank-one rewriting and independently controlled erase/write operations already have close precedents, which the paper acknowledges. The ablations show that adding the two controls helps, but do not isolate whether the tied write direction or independently chosen strengths matter relative to a simpler matched retention update. The DDL comparison is informative but does not resolve this issue: Appendix A.10 changes its read-side convolution to kernel size 1 and correctly acknowledges other interface and cost differences. Thus, the DDL scores cannot by themselves identify the benefit of stream-space editing or the proposed coupling choices. One targeted control, such as a fixed common-mode erase direction or a matched coupled erase/write gate, would make the distinctive design insight more convincing. This limitation does not invalidate the matched comparisons against mHC and iHC.
Questions for authors
- Can you provide paired uncertainty estimates for the existing downstream predictions and held-out language-model losses for the evaluated checkpoints? For component complementarity, a limited repeat at the smaller scale would be especially useful; please distinguish evaluation uncertainty, execution variation, and variation across training seeds.
- Which DDL source version and compressor variant were reproduced, and was its embedding-convolution expansion retained? Appendix A.10 describes a width-1 causal compressor, whereas the reference also distinguishes token-axis and channel-axis variants. Please explain the choice and whether the main conclusion is robust to the reference configuration or a small compressor control.
- Is there evidence comparing the tied write direction with a fixed common-mode direction, or comparing independent strengths with a coupled gate under the same residual interface? Even one such controlled comparison would help clarify which part of the proposed parameterization is essential.
- Please clarify the exact expert-width convention and number of MoE layers underlying Table 4, so the reported total and active parameter counts can be reconstructed.