- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BRANCH-MoE, a hierarchical MoE router that places experts at the leaves of a binary tree and uses arrival-weighted EMA anchors to center each internal routing decision. The goal is to obtain balanced expert utilization without an auxiliary load-balancing loss while simultaneously giving expert indices a meaningful tree topology that can support communication locality.
Critical strengths
-
Coherent routing design that jointly addresses balance and topology. BRANCH-MoE does more than replace one balancing heuristic with another: the tree structure, arrival-weighted EMA anchoring, and prefix-based expert addresses are coupled in a single routing mechanism. This is consequential because the method maintains competitive predictive quality and reasonably balanced utilization without an auxiliary balancing loss, while also producing locality that flat routers do not exhibit on the three UCI benchmarks.
-
The paper provides theoretical support for the routing mechanism. The analysis connects EMA anchoring to a noise lag trade-off, establishes a positive soft routing-mass bound under stated assumptions, relates expert execution frequency to SGD convergence, and derives explicit communication-locality bounds under tree-aligned placement. This strengthens the contribution beyond an empirical routing heuristic and provides useful insight into why the proposed architecture may avoid starvation and induce structured expert assignments.
Critical weaknesses
-
The empirical evaluation is too small-scale to substantiate the paper’s systems-oriented motivation. All experiments use a single MoE layer with only 16 experts on tabular benchmarks, and the paper does not evaluate a multilayer language/embedding model or an actual distributed, topology-aware implementation. The claimed communication benefit is assessed through tree-distance proxies rather than measured cross-device traffic, throughput, or end-to-end latency. The authors themselves note that multilayer interactions and large-scale topology-aware implementations remain future work. This is important because reduced communication and suitability for large embedding/MoE models are central motivations of the method, yet they are not directly demonstrated in the setting where they would matter most.
-
The strongest theoretical guarantee does not directly cover the actual sparse routing behavior, and the locality effect is not robust across datasets. The anti-collapse result concerns soft routing mass for linear routing under log-concave arrival distributions, the paper explicitly notes that this does not guarantee positive execution probability after deterministic top-K selection. Moreover, the proposed locality disappears on Criteo: the linear BRANCH-MoE tree distance is statistically indistinguishable from random, and the authors have not isolated the cause. This weakens the connection between the theoretical motivation and the behavior of the actual sparse router, particularly because locality is one of the paper’s principal claimed advantages.
Questions for authors
-
Can the authors provide experiments on a multilayer MoE model with substantially more experts, ideally including an actual multi-device setup? In particular, how much do the observed tree-distance improvements translate into reductions in communication volume and end-to-end latency?
-
Can the authors empirically quantify how well the theoretical soft leaf-mass bound correlates with actual expert execution frequency after top-K truncation? Are there settings where a leaf has non-negligible soft mass but is nevertheless rarely or never executed?
-
Why does training remove the locality that is already present in the untrained BRANCH-MoE tree on Criteo?
-
The paper observes that changing the node map can affect predictive loss more strongly than changing the routing rule itself, and nonlinear node maps can also weaken BRANCH-MoE locality. How sensitive are the claimed benefits to this architectural choice, and is there a principled way to choose the node-map complexity?
4 comments
Question 3 and Question 4
Q3. Why Does Training Remove Initial Untrained Locality () on Criteo?
- Why untrained BRANCH-MoE starts at : At random Glorot initialization, unit-variance centered logits produce moderate depth- splits (). Sibling leaves share identical ancestor probabilities up to depth , so whenever their parent has large arrival mass, both siblings enter together (), yielding initial .
- Plausible mechanisms behind the training shift on Criteo: While we did not vary batch size or training passes independently on Criteo, two diagnostics shed light on this shift: first, in a 25-seed single-expert dense control (
base, Appendix G.5) directly on Criteo, one SwiGLU expert ( logloss, AUC) already matches or slightly exceeds all 17 MoE arms on AUC (–) and comes within in logloss (–), whereas on Covertype 16-expert MoE beats single-expertbaseby accuracy. Second, Criteo concatenates 39 anonymized categorical fields whose effects combine additively; in a controlled synthetic comparison of cluster-partitioned vs. 4-block additive targets from identical initial trees (, Appendix G.5), additive multi-field tasks suppress root margins relative to leaf margins ( vs. , ratio ), causing tokens to select specialists across distinct subtrees in accordance with Theorem 4.5.
Q4. Sensitivity to Node-Map Complexity & Principled Selection Guideline
We added both theoretical intuition and a direct linear vs. MLP ablation in our 2-layer causal Transformer LM in Section 5.2 ("Node maps", Pages 9–10) and Appendix G.5 (Table 11):
- Why MLPs loosen balance and locality (Section 5.2): Anchoring enforces zero mean score () for any node map, whereas balanced mass splitting depends on the median. Linear maps link mean and median via Grünbaum's halfspace inequality; curved MLP level sets permit skewed tails. Even so, on UCI moving from linear to shifts only modestly from – to –, still – below random () and every flat router (–).
- New multilayer Transformer linear-vs.-MLP comparison (Appendix G.5, Table 11): Comparing
BRANCH-MoE (linear)againstBRANCH-MoE (MLP, h=8)across 5 seeds on the 2-layer Shakespeare Transformer LM shows statistically indistinguishable task quality ( vs. Val CE, Welch ), while MLPs widen Layer 0 Load CV (), loosen Layer 0/1 (), raise group fanout (), and add more router parameters. - Principled selection rule (Section 5.2 & Appendix G.5): Per-node MLPs improve task loss only when the MoE sits directly on raw standardized features with no upstream hidden layer (Covertype, MSD). Whenever an upstream learned representation layer precedes the MoE—such as Criteo's
Dense(256)->SiLU->LN() or self-attention blocks in multilayer Transformers (, Table 11)—use linear node maps, which match MLP quality while preserving exact Grünbaum guarantees, tighter Load CV, strongest prefix locality, and minimal router cost. (Where nonlinear router features are needed on raw inputs, applying linear node maps after a shared projection also preserves halfspace splits in -space; Appendix C.2.)
A gentle reminder as the deadline is approaching
Thank you again for your constructive feedback, which has meaningfully strengthened our work.
As the discussion phase closes soon, we wanted to let you know that we have addressed all the points raised in your review with new experiments and analyses. If any of your concerns remain, please let us know so we can discuss them further.
If our response has resolved your questions, we would be grateful if you would consider updating your score to reflect these improvements. Thank you for your time and guidance!
Response to authors
Dear Authors,
Thank you for your effort. However, you response haven't addressed my concern.
Q1. The author report 23.2% latency reduction, but this is CPU gather/scatter time without routing computation, expert execution and real communication with GPUs. Need to evaluate end-to-end inference and training (because your motivation is system oriented method)
Moreover, the author need to compare communication savings with scenario that flat router place frequently co-selected experts on the same device. If this work, then your contribution (hiearachial routing) is not necessary.
Finally, the 2-layer character-level Transformer experiment is too small to demonstrate the method's effectiveness on realistic large-scale MoE language models.
Q2. The lower bound in Proposition 4.4 is useless when , which includes balanced soft routing for , . Moreover, the condition does not establish that top-K starvation occurs only when the router weight norms approach zero. This misleading and overstates what Proposition 4.4 actually guarantees.
The authors need to clarify whether non-negligible weight norms can also lead to expert starvation, and why the reported correlation decreases from 0.972 at to 0.696 at .
Q3. The single-expert results do not explain why training removes locality, and the synthetic additive-task experiment does not establish that the same mechanism occurs on Criteo. Moreover, Theorem 4.5 provides a sufficient condition for locality, not a causal explanation for its disappearance. The authors need to provide direct evidence from Criteo, such as root/leaf margin evolution and tree distance throughout training, to substantiate the proposed explanation.
Q4. The proposed node-map selection guideline remains empirical rather than principled. Moreover, applying linear routing after a nonlinear learned representation does not necessarily preserve log-concavity, which is required by Corollary 4.2. Therefore, the claimed preservation of theoretical guarantees is not justified.
I believe this work still requires substantial improvements to meet the acceptance bar for ICLR.
Question 1 and Question 2
We thank Reviewer
F8mMfor the positive assessment of our coherent balance-and-topology design, and four insightful technical questions. Below we address all three weaknesses and questions. We have also uploaded a revised 10-page manuscript (all revisions highlighted in blue) to reflect these changes.Q1. Multilayer MoE, Larger Expert Counts (E>16), and Reductions in Communication Volume / Latency
We added three sets of experiments in Section 5.3 and Appendix G.5 (Figures 3–4, Tables 10–11):
float32, G=8,K=4, median of 25 trials after warmup per seed; distinct from distributed multi-node GPU/TPU network collectives, which remain future systems work) reduces gathered/scattered payload volume from 113.5MB to 83.1MB (−26.8) and CPU gather/scatter latency from 77.5ms to 59.5ms (−23.2).Q2. Correlation Between Soft Leaf Mass qe and Sparse Top-K Execution pe (and When qe>0 Rarely Executes)
We address both parts via new Proposition 4.4 (Section 4.3) and empirical measurements (Table 10, Figure 4b):
Mathematical bridge & exact failure regime: Because any leaf e∈/SK(x) is exceeded by at least K other entries of m(D)(x)∈ΔE−1, me(D)(x)≤K+11 on e∈/SK(x). Taking expectations gives
pe≥qe−δe,tail≥K(K+1)qe−1,δe,tail:=E![me(D)(X)1e∈/SK(X)].
In the sharp-split limit (router weight norms β:=∣Wj(d)∥2→∞ at exact anchors on log-concave inputs), halfspace indicators preserve log-concavity at every depth, δe,tail→0, and Grünbaum's inequality yields pe=qe≥e−D for every K≥1. Conversely, qe>0 can coexist with pe=0 only if me(D)(X)≤K+11 almost surely—i.e., in the near-zero weight-norm regime (β=∣Wj(d)∥2≈0) where all branch splits hover near 1/2 (me(D)(x)≈1/E<K+11) and small static offsets rank K leaves ahead.
Empirical quantification: In trained BRANCH-MoE models, qe and pe/K correlate strongly (Pearson r=0.972 at E=16, 0.859 at E=32, with 0 dead experts across all depths and seeds). Scaling the router weight norm Wj(d)=βWˉj(d) (∣Wˉj(d)∥2=1) across β∈[0.02,4.0] at E=16,K=4 (Figure 4b) confirms Proposition 4.4: for β∈[0.25,4.0], minepe∈[0.19,0.24] (near uniform K/E=0.25), and minepe drops near zero (0.0006 despite mineqe=0.060) only when β is compressed to 0.02 so that every split hovers near 1/2 (1−E[RK]=0.743).