- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduzes ProGram, a layer-aware multi-prime n-gram hash routing method that treats shallow MoE routing as depth-aware routing-key design. It uses unigram, bigram, and trigram hashing in the first three MoE layers, respectively, while retaining learned mixture weights and standard learned routing in deeper layers. The contributions include the progressive routing schedule, a multi-prime hashing procedure with deduplication and fallback probing, and an analysis connecting routing-key concentration to expected load imbalance under idealized hashing assumptions. Validation uses a 5B-parameter MoE with 412M activated parameters, pretrained on approximately 100B tokens and evaluated on 12 benchmarks. PROGRAM improves over the matched learned-routing baseline on 11 benchmarks, ties on one, and performs best on nine benchmarks among the evaluated hash schedules.
Critical strengths
- The paper presents a simple, well-motivated way to incorporate local context into shallow expert selection while retaining adaptive weighting and learned routing in deeper layers.
- The method is well specified at the algorithmic level, and the reported results are promising.
- The paper is well written and easy to follow, and the authors conducted comprehensive experiments.
Critical weaknesses
- For unigram routing, Eq. (8) makes the selected expert set depend only on , so token IDs separated by multiples of 192 will receive identical routing under this configuration. This limits the unigram implementation to at most 192 distinct expert sets. Can author provide more explaination about this setting?
- Tables 1 and 2 report point estimates without variation across runs or confidence intervals for downstream scores.
- The explanation that unigram routing establishes a lexical anchor and higher orders subsequently enable contextual specialization is not supported by direct analysis of expert behavior. I hope the author can explain this in detail.
- Appendix B argues that routing overhead is negligible using operation counts, but the author do not provide throughput, latency, or peak-memory measurements. I hope the author can provide more details about the computational costs.
Questions for authors
- Please refer to the weakness.
- I don't really understand the sequence-level balancing-loss, it will be helpful if the authors can provide a more detailed explaination.
6 comments
[Response to Reviewer GhZS (2/2)]
W4: Computational Cost
We thank the reviewer for pointing out the need for empirical measurements of computational overhead.
To address this concern, we have added end-to-end training latency and throughput measurements in Appendix B (Page 14). Specifically, Figure 4 reports the absolute iteration time and throughput of ProGram-123 and Learned-MoE, while Figure 5 compares the relative computational costs of ProGram-123, ProGram-234, Dense-First-1, and Learned-MoE. Over steps 101–3000, ProGram-123 achieves an average iteration time of 1248.48 ms and a throughput of 420,291 tokens/s, compared with 1249.47 ms and 420,076 tokens/s for Learned-MoE.
These measurements show that ProGram-123 not only introduces no measurable end-to-end training overhead compared with Learned-MoE, but also achieves slightly lower iteration latency and higher throughput in our experiments.
Q1: Please Refer to the Weaknesses
We have addressed the individual concerns above, including the modular limitation of unigram hashing, pretraining seed sensitivity, the distinction between routing-key statistics and expert specialization, and the computational cost of ProGram.
Q2: Explanation of Sequence-Level Load-Balancing Loss
Thank you for asking us to clarify the purpose of this metric. We report it because all routing variants use the same load-balancing objective during pretraining. It lets us compare how well the variants satisfy that objective as training progresses. Computing the loss separately for each sequence captures concentration within individual sequences, which can be obscured when expert usage is aggregated across a batch.
For each expert, the loss multiplies its normalized selection frequency in the auxiliary routing map by its mean router score over the sequence, then sums these products across experts. A lower value indicates less overlap between frequently selected experts and experts receiving high average router scores. In the hash-routed layers, the hash function determines the experts that actually process each token. We therefore report MaxVio on the actual expert assignments in Appendix D.1 as a complementary measurement.
In Figure 2(b), ProGram-123 achieves lower sequence-level balancing loss than both fixed 2-gram hashing and ProGram-234 at the 30B, 60B, and 100B checkpoints. This shows that ProGram-123 better satisfies the shared auxiliary objective at the reported checkpoints. Its stronger overall downstream performance in Table 2 provides separate evidence in favor of the progressive (1→2→3) schedule.
Update on W2 — Variation across training seeds
Thank you for raising the concern about variation across runs. To further assess robustness to random initialization, we have now completed the downstream evaluation of three independent ProGram-123 pretraining runs with random seeds 42, 43, and 44, using the same evaluation protocol across all 12 benchmarks.
| Seed | MMLU-Pro | MMLU-Redux | MLQA-Zh-Zh | INCLUDE | GPQA | GSM8K | MGSM | BBH | MBPP | MBPP+ | HumanEval | RepoBench |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 42 | 9.99 | 26.94 | 15.90 | 32.38 | 29.24 | 5.16 | 8.80 | 21.62 | 10.60 | 16.14 | 13.41 | 6.60 |
| 43 | 10.27 | 22.42 | 14.72 | 28.39 | 26.56 | 4.62 | 8.00 | 22.07 | 12.20 | 17.20 | 12.20 | 6.19 |
| 44 | 8.93 | 23.45 | 15.98 | 27.59 | 26.12 | 4.88 | 8.80 | 20.65 | 11.40 | 15.61 | 11.59 | 7.38 |
| Mean | 9.73 | 24.27 | 15.53 | 29.45 | 27.31 | 4.89 | 8.53 | 21.45 | 11.40 | 16.32 | 12.40 | 6.72 |
| Std | 0.58 | 1.93 | 0.58 | 2.10 | 1.38 | 0.22 | 0.38 | 0.59 | 0.65 | 0.66 | 0.76 | 0.49 |
The standard deviation is below one score point on 9 of 12 benchmarks, with larger variation on MMLU-Redux, INCLUDE, and GPQA. Rankings vary across tasks, and each seed achieves the highest score on at least one benchmark. These downstream results complement the closely aligned pretraining LM-loss trajectories in Appendix B, Figure 6, and provide evidence of robustness to random initialization across the three tested seeds.
[Response to Reviewer GhZS (1/2)]
We thank the reviewer for the thoughtful and constructive evaluation. We appreciate the positive assessment of the method and its experimental results. We address the concerns regarding the expressiveness of unigram hashing, experimental variance, expert specialization, computational overhead, and sequence-level load balancing below.
W1: Modulo collisions in unigram routing.
The reviewer is correct. In Eq. (8), two token IDs that are congruent modulo 192 produce the same sequence of candidate expert IDs. Deterministic deduplication and fallback probing therefore give them the same selected expert set. Each unigram layer can consequently produce at most 192 distinct expert sets. The mixture weights over a selected set can still vary with the token's hidden state. We will make this bound explicit and describe the first layer as a coarse, deterministic partition of token IDs. In layers 2 and 3, adding preceding tokens to the routing key allows different occurrences of the same token to select different expert sets.
W2: Variance Across Runs and Confidence Intervals
Tables 1 and 2 report point estimates on the benchmark test sets. We use greedy decoding, which removes variation from stochastic generation for a fixed checkpoint and prompt. Uncertainty due to the finite test sets and variation across independent pretraining runs are separate questions. For MGSM-Zh, we average over four evaluation seeds for each checkpoint.
W3: Evidence for Lexical Anchoring and Contextual Specialization
We thank the reviewer for pointing out the need for direct evidence supporting the proposed transition from lexical anchoring to contextual specialization.
To address this concern, we provide complementary evidence from routing-schedule ablations and direct expert-behavior analysis. First, as reported in Section 4.3 (Page 8, Table 2) and Figure 3 (Page 9), we compare ProGram-123 with ProGram-321 and ProGram-223. ProGram-123 achieves the best performance on 9 out of 12 benchmarks, indicating that the performance advantage depends not merely on introducing higher-order n-gram routing, but also on how routing granularity is organized across layers.
More importantly, Appendix D.2 (Pages 19–20, Table 5) provides a direct analysis of layer-wise expert behavior. For the same token appearing in different contexts, the routed-output cosine similarities are 0.9980, 0.1586, and 0.1207 in the first, second, and third layers, respectively, indicating progressively stronger context dependence in the routed computations. We further conduct controlled expert-swap interventions. Replacing first-layer experts with those selected for different tokens increases next-token NLL by 0.4926 nats (95% CI: [0.4254, 0.5584]), supporting the functional role of the token-level lexical anchor. In the second layer, replacing experts with those selected for the same token under different preceding contexts increases NLL by 0.0369 nats (95% CI: [0.0222, 0.0525]), demonstrating a measurable functional contribution from contextual information.
Taken together, these results support the interpretation that ProGram-123 establishes a functionally meaningful token-level routing anchor and progressively incorporates local contextual information into subsequent layers. We have included these analyses and clarified their implications in Section 4.3 (Page 8), Figure 3 (Page 9), and Appendix D.2 (Pages 19–20).