- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The standard Manifold-Constrained Hyper-Connections (mHC) use Sinkhorn–Knopp to achieve doubly stochasticity. However, several mixers with different cross-stream interactions can satisfy this constraint. To address this, the authors introduce Structured mHC, deriving a differentiable primal-dual method. Experimental results show lower late-stage training loss compared to the original mHC in 64M and 198M models, and structured mHC provides better accuracy in downstream tasks.
Critical strengths
- The paper introduces a novel mHC approach which provides lower loss and higher downstream accuracy.
Critical weaknesses
- The experimental setup is generally limited. Firstly, the experiments are done on smaller models compared to the original mHC, which uses 3B, 9B, and 27B models. It is not discussed why different models are chosen compared to mHC. Secondly, the scalability and efficiency of the proposed method is not discussed and evaluated. It is not clear if the primal-dual optimization can have additional overhead compared to Sinkhorn for equal number of iterations. Additionally, the effect of number of primal-dual iterations K is not ablated. K is set to 20, but it is not clear how different K values can affect performance (This is studied in the original mHC paper, for the number of Sinkhorn iterations). A comprehensive comparison with different number of iterations for Sinkhorn and Primal-dual could be helpful. This can include timing and final performance comparisons. Finally, the proposed method is only compared to mHC in Table 1, while the authors mention several of other recent and related papers in the Related Work section, and emphasize that their method is complementary to those. If applicable, adding additional baselines could strengthen the paper.
Questions for authors
Can the authors provide clarifications on their choice of models, as well as the choice of K? Also, can efficiency comparisons with Sinkhorn and additional baselines be included?