acceptodds
Under review as a conference paper at ICLR 2027

Structured mHC: Objective-Guided Residual Mixing over the Birkhoff Polytope

Abstract

Manifold-Constrained Hyper-Connections (mHC) use Sinkhorn–Knopp normalization to approximate doubly stochastic residual mixers; exactly feasible mixers in the Birkhoff polytope compose non-expansively across depth. Doubly stochasticity, however, does not determine how information is exchanged: identity, uniform, and permutation mixers satisfy the same constraint while inducing different cross-stream interactions. We introduce structured mHC, which treats the objective used to select a mixer within the Birkhoff polytope as a design variable. We instantiate the framework with a token-dependent linear objective derived from learned mixer scores and compute its mixers with a fixed-depth, differentiable primal–dual method tailored to the row- and column-sum constraints. Under matched protocols and token budgets within each model scale, our method attains lower late-stage training loss than mHC-Sinkhorn in 64M dense and 198M-A64M MoE MiniMind-3 models. In a separate 7B comparison after 15B training tokens, it attains mean accuracy gains of , , and percentage points over mHC-Sinkhorn across eight tasks under zero-, five-, and ten-shot evaluation.

Primary area:
optimization
Keywords:
large language model, mHC

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KaDmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The standard Manifold-Constrained Hyper-Connections (mHC) use Sinkhorn–Knopp to achieve doubly stochasticity. However, several mixers with different cross-stream interactions can satisfy this constraint. To address this, the authors introduce Structured mHC, deriving a differentiable primal-dual method. Experimental results show lower late-stage training loss compared to the original mHC in 64M and 198M models, and structured mHC provides better accuracy in downstream tasks.

Critical strengths
  • The paper introduces a novel mHC approach which provides lower loss and higher downstream accuracy.
Critical weaknesses
  • The experimental setup is generally limited. Firstly, the experiments are done on smaller models compared to the original mHC, which uses 3B, 9B, and 27B models. It is not discussed why different models are chosen compared to mHC. Secondly, the scalability and efficiency of the proposed method is not discussed and evaluated. It is not clear if the primal-dual optimization can have additional overhead compared to Sinkhorn for equal number of iterations. Additionally, the effect of number of primal-dual iterations K is not ablated. K is set to 20, but it is not clear how different K values can affect performance (This is studied in the original mHC paper, for the number of Sinkhorn iterations). A comprehensive comparison with different number of iterations for Sinkhorn and Primal-dual could be helpful. This can include timing and final performance comparisons. Finally, the proposed method is only compared to mHC in Table 1, while the authors mention several of other recent and related papers in the Related Work section, and emphasize that their method is complementary to those. If applicable, adding additional baselines could strengthen the paper.
Questions for authors

Can the authors provide clarifications on their choice of models, as well as the choice of K? Also, can efficiency comparisons with Sinkhorn and additional baselines be included?

Reviewer_SmP6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits manifold-constrained Hyper-Connections (mHC), which use doubly stochastic residual mixers to control signal amplification across depth. The authors argue that the Birkhoff constraint specifies mixer feasibility but not which feasible stream interactions should be preferred.

They propose Structured mHC, which explicitly defines an optimization objective over the Birkhoff polytope. Their main instantiation uses token-dependent linear costs derived from learned mixer scores and approximately solves the resulting assignment problem using a fixed number of differentiable primal–dual iterations.

The method is evaluated on 64M dense and 198M-A64M MoE models, where it achieves lower late-stage training loss than mHC-Sinkhorn, and on a separate 7B model trained for 15B tokens, where it improves average downstream accuracy over mHC-Sinkhorn.

Critical strengths
  • The distinction between enforcing doubly stochasticity and choosing which doubly stochastic mixer to use is conceptually useful. The proposed framework provides a systematic way to explore this design space rather than treating Sinkhorn normalization as fixed.

  • Structured–Score achieves lower late-stage loss than mHC-Sinkhorn in both dense and MoE experiments, and also improves average downstream accuracy in the separate 7B experiment. This suggests that mixer construction can meaningfully affect training.

Critical weaknesses
  • The paper's stability motivation relies on mixers lying in the Birkhoff polytope, but after a fixed number of primal–dual iterations the actual mixer need not have unit row/column sums, and therefore does not inherit the claimed non-expansive guarantee. Moreover, the chosen step sizes violate the stated sufficient convergence condition. This makes it unclear whether the gains arise from objective-guided selection within the Birkhoff polytope or simply from the behavior of the finite-step, potentially infeasible optimizer. This concern is reinforced by Structured–Trace: its exact optimum is simply the identity, yet its finite-step implementation behaves differently from the explicit identity baseline.

  • No multi-seed results or uncertainty estimates are provided. The 7B average gains are also uneven across tasks: the zero- and five-shot improvements are substantially influenced by large gains on CommitmentBank, while several other tasks slightly degrade and the ten-shot mean gain falls to only 0.59 points. The smaller-scale experiments report only training loss, while the 7B experiment reports downstream accuracy without corresponding training-loss analysis. This makes it difficult to establish a consistent scaling story.

Questions for authors
  • What are the actual row/column constraint violations, spectral norms, and cumulative residual-product norms of the finite-step Structured–Score mixers during training? Does the improvement remain if the final mixer is explicitly projected back onto the Birkhoff polytope?

  • How sensitive are the results to the number of primal–dual steps (K) and the step-size parameters (r,s)? In particular, what happens when using parameters that satisfy the stated convergence condition, or when increasing (K) toward the exact linear-program solution?

  • Can the authors report multiple training seeds or uncertainty estimates, particularly for the 7B evaluation? It would also be useful to report the average gain excluding CommitmentBank to show how broadly the downstream improvement is distributed.

Reviewer_s2wDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes structured mHC, which treats residual-mixer construction in mHC as an optimization over the Birkhoff polytope, with an explicit objective that decides which doubly stochastic mixer is selected. Its main instance builds a token-dependent linear cost from the learned mixer scores. It replaces SK normalization with a fixed number of unrolled primal-dual iterations and trains through them end to end. In MiniMind-3 runs at 64M dense and 198M-A64M MoE, it reaches lower late-stage training loss than mHC-Sinkhorn. A 7B model trained on 15B tokens shows higher mean few-shot accuracy across eight tasks.

Critical strengths
  • The paper asks a valuable question. Instead of only guaranteeing the mixer is stable, it is also important to ask which point of the Birkhoff polytope a mixer construction selects. Figure 1 illustrates this point clearly.
  • The algorithm is simple. It uses only reductions, broadcasts and elementwise operations, runs in parallel across tokens, and has the same per-iteration complexity as SK. The derivations are correct, based on my understanding, and the authors are candid that the finite-step output need not be feasible and that the exact solution has zero gradient.
  • On the 64M dense model, the roughly 0.02-nat improvement over the other mixers holds throughout the run, and the MoE run shows the same ordering.
Critical weaknesses
  • The stated contribution is to separate the stability constraint from the objective that selects a mixer. But the only learnable objective is the same linear term that SK already optimizes implicitly, with the entropic regularizer removed. Its exact minimizer is a permutation with zero gradient, so what is actually trained is the truncated K-step map, which, with a convergent step size, behaves like an approximate Euclidean projection onto the polytope. The paper would be more convincing if it were framed as entropic versus Euclidean projection and compared accordingly.
  • The method weakens the stability property that motivates mHC, and never checks it. The step size is below the convergence condition the paper itself states, and even below 0.75×2n0.75 \times 2n (6 for n = 4), the constant below which [1] give a divergence example. The truncated SK used in the original mHC at least always returns a nonnegative, exactly row(or column)-stochastic mixer.
  • All experiment results appear to be single runs measured by training loss, and the gaps shrink over training. Under reasonable step sizes, Structured-Trace is numerically almost the identity, yet in Figure 4 it ends close to Structured-Score on MoE, so much of the gain over Sinkhorn may come from not mixing at all. Methods the paper already cites (mHC-lite, KromHC) are not compared, no learned mixers are analyzed, and key hyperparameters (ρ\rho, rr, ss, initialization) are not reported.
  • The 7B evaluation is rather weak. The model sees only 15B tokens with a 6400-token vocabulary, and many tasks are near chance. More than half of the zero- and five-shot mean gain comes from CB, which has only 56 examples, and the sample-weighted gain is under one point in every shot setting.

[1] A generalized primal-dual algorithm with improved convergence condition for saddle point problems

Questions for authors

I wonder if the authors have tried Sinkhorn with a lower or learned temperature, or an exact Euclidean projection? Also, with a convergent step size and larger K, so the output gets closer to the ideal permutation, does performance improve or degrade?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.